Last Wednesday was the 18th anniversary of the 9/11 attacks, the deadliest terror attack in history. The attacks were the beginning of a new era and dramatically changed U.S. foreign policy.  It became the start of a 'Global War on Terrorism', that prompted the invasion of Afghanistan and was used as justification for the Iraq invasion. In the last few years Al-Qaeda has been largely overshadowed by its offshoot the Islamic State, which became more successful than its parent group. It has also not executed a terrorist attack comparable to 9/11 in the last decade. These metrics paint Al-Qaeda as a group in decline, or even defeat, but it has also benefited from various events that will be explained later on. The Al-Qaeda that existed on 9/11 is not the Al-Qaeda that exists today, the organization has transformed out of necessity but remains a threat. 
Al-Qaeda was formed in Afghanistan during the Soviet invasion.  Many Muslims had joined the war to fight against the Soviets, in what they viewed as a 'holy war' (jihad). The war ended with the withdrawal of the Soviets and it was seen as a major win. But some of the fighters had bigger ambitions, among them Osama Bin Laden, and wanted to create a vanguard group to fight future 'holy wars'. Such a group was formed in August 1988 and was named Al-Qaeda (which is Arabic for 'The Base').  After the Soviets withdrew, Osama Bin Laden returned to his home country Saudi Arabia. When Iraq invaded Kuwait in 1990 Osama Bin Laden offered the Saudi monarch to remove Iraq from Kuwait with his 'army'. His offer was rejected by the Saudi government, who preferred U.S. support.  This outraged him because he viewed it as an occupation, and it fueled his anti-American hate. In the following years multiple attacks would be linked to Al-Qaeda, like the 1993 Battle of Mogadishu  and the World Trade Center Bombing in the same year.  After Osama Bin Laden was ousted from Sudan after pressure from the international community, he returned to Afghanistan.  In 1996 he would formally announce his war against America in a document called 'Declaration of Jihad against the Americans Occupying the Land of the Two Holiest Sites'.  The groups' first major attack came in 1998, in a coordinated car bomb attack at U.S. embassies in Kenya and Tanzania, killing over 200 and wounding more than 4000.  Two weeks later, America launched cruise missiles at training camps in Afghanistan and Sudan in retaliation.  Despite those strikes, the attacks continued: a series of coordinated plots were foiled around 2000 , they bombed the U.S.S. Cole that was refueling in Yemen in 2000  and ultimately the attacks against the Twin Towers, the Pentagon and an attempted attack against the White House on 11 September 2001.  Osama Bin Laden wanted the U.S. to withdraw its support for Arab government and thought that continued attacks against it would achieve that goal. He had seen how the U.S. had retreated from Somalia and Lebanon following attacks against its forces and wished to replicate that. 
One month after 9/11 the U.S. invaded Afghanistan to dislodge the Taliban from power and to get Osama Bin Laden. Within weeks the Taliban only controlled a small portion of Afghanistan and Osama Bin Laden was cornered in Tora Bora before fleeing to tribal areas in neighboring Pakistan.  In the subsequent years many (senior) Al-Qaeda operatives were arrested, but despite that pressure Al-Qaeda continued terrorist attacks across the world, most notably the 2004 Madrid train bombings and 2005 London bombings. The continued pressure eventually took its toll on Al-Qaeda's 'external operations', and attacks started to reduce after 2006. In the ensuing years, Al-Qaeda focused on building affiliates across the globe. Its Yemeni affiliate (AQAP, Al-Qaeda in the Arabian Peninsula) was responsible for most of the (foiled) terror attacks from 2009 to 2015.  Some predicted that the Arab Spring and killing of Osama Bin Laden in 2011 was the nail in the coffin for Al-Qaeda , this turned out to be false. The Arab Spring resulted in several civil wars that Al-Qaeda profited off, and while the death of Osama Bin Laden was a major blow, it didn't result in the organization falling apart. ISIS breaking away from Al-Qaeda was also a massive challenge, but besides some defectors, Al-Qaeda has lost no affiliates to them. Currently, Al-Qaeda has many affiliates deeply rooted in local conflicts eschewing 'external operations', but this is a deliberate strategy  in which Al-Qaeda is trying to expand its influence while remaining under the counter-terrorism radar and positioning itself as a 'moderate' alternative to the Islamic State.  And while Al-Qaeda had roughly 300 members on the eve of 9/11, it currently has thousands in various countries across the world. ISIS has also proven that the 'Jihadi-Salafism' ideology still appeals to many. Despite many challenges facing them the organization has proven resilient, and while its current focus is more local, Al-Qaeda still repeatedly threatens the U.S.