Security experts say CrowdStrike's routine update of its widely used cybersecurity software, which caused a global internet crash, apparently did not undergo adequate quality checks before it was deployed.
The latest version of its Falcon Sensor software was meant to make the systems of CrowdStrike's clients more secure against hacking by updating the threats it defends against.
Faulty code in the update files resulted in one of the most widespread tech outages in recent years for companies using Microsoft's Windows operating system. Global banks, airlines, hospitals and government offices were disrupted.
CrowdStrike released information to fix affected systems, but experts said it would time to manually weed out the flawed code.
"What it looks like is, potentially, the vetting or the sandboxing they do when they look at code, maybe somehow this file was not included in that or slipped through," said Steve Cobb, chief security officer at Security Scorecard, which also had some systems impacted by the issue.
Problems came to light quickly after the update was rolled out, and users posted pictures on social media of computers with blue screens displaying error messages, known in the industry as "blue screens of death."
Industries from travel to finance were crippled for hours, highlighting the risks of a global shift towards digital, interconnected technologies.
CrowdStrike CEO George Kurtz said on social media platform X that a defect was found "in a single content update for Windows hosts" that affected Microsoft's customers and that a fix was being deployed.
"We're deeply sorry for the impact that we've caused to customers, to travellers, to anyone affected by this, including our company," Kurtz told NBC News' Today program.
CrowdStrike shares plunged as much as 14.5 per cent shortly after the Wall Street open before paring losses to trade down 8.5 per cent. Its cyber rivals were up, with SentinelOne 3.6 per cent higher and Palo Alto Networks up 1.7 per cent.
Microsoft was down 0.2 per cent.
"Earlier today, a Crowdstrike update was responsible for bringing down a number of Windows systems globally. We are actively supporting customers to assist in their recovery," Microsoft chief communications officer Frank Shaw said in a post on X.
Patrick Wardle, a security researcher, said it was very common that security products updated their signatures, as often as once a day.
"Because they're continually monitoring for new malware and because they want to make sure that their customers are protected from the latest threats," he said.
The frequency of updates was "probably the reason why (CrowdStrike) didn't test it as much," he said.
While largely fixed, the cyber outage revealed the risks of an increasingly online world.
"This is a very, very uncomfortable illustration of the fragility of the world’s core internet infrastructure," said Ciaran Martin, professor at Oxford University's Blavatnik School of Government and former head of the UK National Cyber Security Centre.
Accelerated by the COVID-19 pandemic, governments and businesses alike have become increasingly dependent on a handful of interconnected technology companies.
Airports from Los Angeles to Singapore, Hong Kong, Amsterdam and Berlin experienced problems including planes being grounded, flight delays and staff having to check in passengers manually.
Banks and financial services companies from Australia to India and Germany warned customers of disruptions and traders across markets spoke of problems executing transactions.
In Britain, booking systems used by doctors were offline, multiple reports posted on X by medical officials said.
Government agencies were also affected with the Dutch and United Arab Emirates' foreign ministries reporting some disruptions.
"IT security tools are all designed to ensure that companies can continue to operate in the worst-case scenario of a data breach, so to be the root cause of a global IT outage is an unmitigated disaster," said Ajay Unni, CEO of StickmanCyber, one of Australia's largest cybersecurity services companies.
US-based CrowdStrike, with a market value of about $US83 billion ($A124 billion), is among leading cybersecurity companies, counting more than 20,000 subscribers around the world, its website showed.
The global impact of the outage reflects CrowdStrike's dominance. Over half of Fortune 500 companies and many government bodies such as the top US cybersecurity agency, use the company's software.
