A botched update by one of the world’s leading software security companies has caused more havoc in one day for businesses around the world than all but the worst hacking groups have caused.
CrowdStrike built its reputation by capturing and publicizing malicious electronic attacks by Russian and Chinese spies and organized crime groups, building a market capitalization of more than $70 billion.
But the company relies on deep access to millions of computers to defend them against new attacks, and the commands CrowdStrike sent overnight to computers running Microsoft’s Windows operating system rendered them unavailable by Friday morning.
As banks, airlines and 911 emergency systems struggled to restore services, CrowdStrike apologized and said the problem was the result of an internal system error rather than a hacking attack.
“This was not a cyberattack,” CrowdStrike said in a blog post. The Austin-based company said it had identified the issue and provided fixes to customers to get employees back to work.
But the failure was so widespread and its impact so severe that not all security experts were convinced it was simply human error.CrowdStrike has grown rapidly in the last year and just last month joined the S&P 500, an index of the 500 largest publicly traded companies. But the company has made enemies around the world by blaming hacking operations such as the 2016 hacking operation by Russian intelligence that stole emails from the Democratic National Committee and Hillary Clinton’s campaign chairman.
“I don’t think this is a coincidence. There are just too many flaws,” said Matthew Hickey, founder of training firm Hacker House, who said the files contained random data, were not digitally signed and had not been extensively tested.
A U.S. federal official, speaking on condition of anonymity regarding national security matters, said there was no evidence of sabotage or foreign involvement.
Get caught up in
Stories to keep you up to date
Some analysts said they were waiting for more information from CrowdStrike and that the complexity of cutting-edge hacking defenses makes them dangerously vulnerable.
Jake Williams, a former National Security Agency hacker, said “endpoint detection” products like CrowdStrike’s Falcon tool often send not just updated identifiers to block malicious programs, but also active lines of code to thwart more complex attack scenarios. He said CrowdStrike’s systems for testing code before installing it everywhere may not have been “diverse enough” to catch mistakes.
Computer network outages are not uncommon, but experts were stunned Friday that one company’s error could spread to so many systems.
“We’ve never seen a cascading outage like this, probably never have seen it before,” said Chuck Herrin, an executive at digital security firm F5.
The sheer scale of tech crashes across the globe on Friday exposed inherent risks in security software that many consider essential for businesses to ward off ransomware and other devastating hacks.
To be effective, such a program needs to know everything that’s going on on a machine. But that access could have caused a disruption to be catastrophic, and because it was a Friday, the fix the company offered afterward was complicated; many organizations would have had to manually reboot each machine, one by one, to remove the problematic update.
That privileged access also makes security programs a prime target for spies and everyday hackers: Last month, U.S. authorities barred Russian antivirus software company Kaspersky Lab from doing new business in the United States after it was accused of helping NSA officials steal secrets.
Friday’s troubles canceled or delayed thousands of flights and forced hospitals to postpone surgeries.The worst cyber attacks, such as Russia’s NotPetya attack on Ukrainian companies and North Korea’s WannaCry virus, have caused more lasting damage by inflicting permanent damage on computers, but even those have never spread so quickly and widely.
The extent of the economic damage from the outages and who will pay for them will not be known for some time. Most software providers are protected from legal liability for damage caused by their programs because they license them, not sell them. But most have service contracts with large customers that could require assistance with repairs, discounts and other compensation.
What makes the CrowdStrike failure so striking is that company executives have been some of the most prominent voices in the industry blaming Microsoft for repeated security failings. The software giant has been blamed for recent major intrusions into U.S. government agencies, including the theft of emails from employees, including Commerce Secretary Gina Raimondo, last year. A scathing report released in April by a cybersecurity review board led by officials from the Cybersecurity and Infrastructure Security Agency cited “a corporate culture that disregards both corporate security investments and rigorous risk management.”
Beyond these missteps by Microsoft, CrowdStrike says the company’s dominant market position in operating systems and productivity software means any weakness could have devastating effects.
Some experts have said similar things about CrowdStrike, one of a handful of top-tier security companies with widespread influence and power.
“Obviously this is very serious and it’s going to take weeks and weeks and we’re going to have to touch keyboards,” said Brian Palma, chief executive of rival security firm Trellix. “This speaks to the need for redundancy and defense in depth.”
The Cybersecurity and Infrastructure Security Agency said it was assisting with recovery efforts and warned that criminals posing as CrowdStrike were trying to trick customers into downloading malicious programs or giving up access to their computers.
Marie Basek, an associate professor in the computer science department at University College London, said the widespread computer meltdown showed how dependent the world’s technology systems are on software from a small number of companies, including Microsoft and CrowdStrike.
“The problem here is that Microsoft is the standard software that everyone uses, and the CrowdStrike bug was deployed to every system,” she said.
Vasek said technology networks have become so widespread, complex and interconnected that a mistake in a single line of software code is increasingly likely to take down an entire computer network.
The flaw only affected hundreds of millions of personal computers and computers that use Windows, which powers many back-end systems for airlines, digital payments, emergency services, call centers and more.
CrowdStrike said in a statement that it is “working with all affected customers to ensure that our systems are restored and we can provide the service they expect.”
Some businesses affected by the CrowdStrike glitch, including banks and emergency service centers, said on Friday they were deploying CrowdStrike’s fixed software and beginning to recover.
Vasek said both Microsoft and CrowdStrike needed to review their procedures to prevent such widespread technology outages from happening again.
He said CrowdStrike should consider how to securely update software across its network of millions of computers, and that Microsoft should do more to prevent updates to other companies’ software from crippling Windows machines.
“Microsoft needs to figure out how to check that their software is working as it should,” she said.
Microsoft did not directly address the criticism but said in a statement that the company was “actively working to help our customers recover.”
The company also reported outages with some of its popular web-connecting software for business and government technology networks.
It was not immediately clear how much of Friday’s computer network outages were due to flaws in CrowdStrike’s software updates and how much was due to problems that began Thursday with Microsoft’s online services and its enterprise cloud-computing service, Azure.
A Microsoft spokesman said the company doesn’t believe the CrowdStrike software bug was related to the outage that affected “a small percentage of Azure customers.” The issue has been resolved, he said.
Fixes
An earlier version of this article incorrectly listed Brian Palma’s first name as Ryan. The article has been corrected.
