A week after an ill-fated update from cybersecurity giant CrowdStrike took down an estimated 8.5 million Windows computers and caused a range of issues from taking down medical systems at healthcare facilities to delaying flights for many airlines, some organizations are still trying to restore access to the remaining affected systems.
Healthcare companies were among the hardest hit, with corrupted files affecting about half of the Health Information Sharing and Analysis Center’s members, said Errol Weiss, chief security officer at Health-ISAC. As of July 25, only 18% of affected organizations had fully restored their systems, and three-quarters of companies had up to 25% of their systems still needing attention, Weiss said.
With many organizations adopting Windows-based medical devices, Weiss said he expects a long-term recovery.
“We estimate that there were a lot of automated fixes shared on Friday and Saturday, and those methods likely contributed significantly to the majority of completions,” he said of tools and scripts provided by Microsoft, CrowdStrike and other companies. “But some of those scripts and automated fixes may not work on the types of devices we’re talking about, and they will need to be manually reviewed by healthcare providers.”
Microsoft has released a USB Recovery Tool that allows administrators to recover affected systems from WinPE or Safe Mode using a USB drive. The tool can recover from Safe Mode even if the device has BitLocker enabled and the recovery key is not available. It also has detailed recovery instructions for affected Windows 365 cloud PCs and Azure Virtual Machines, as well as Windows clients, servers and OS hosted on Hyper-V.
Measuring the impact of power outages
CrowdStrike estimated on July 25 that 97% of affected computers had returned to an active state, based on the state of its Falcon software, which was at the center of the outage. Quest Software, a managed security services provider, has clients of all sizes and is still offering help to those still working to resolve the issues. The remaining companies are likely a small number of systems at larger companies that will be harder to patch, and a large number of smaller companies that don’t have the technical expertise to easily recover, said Kent Fayed, senior director of product management at Quest Software.
“That 3% actually represents the number of devices, which means there’s a significant number of small and medium-sized businesses that still don’t really know how to respond to this attack,” he said. “Small and medium-sized businesses tend to leverage IT generalists and may not have in-house IT specialists.”
The full impact of the outage has yet to be tallied, but insurance services company Parametric Solutions estimates that it affected a quarter of Fortune 500 companies, causing losses of $5.4 billion, including nearly $2 billion in the healthcare sector and more than $1.1 billion in the banking sector.
Even with the tools, many companies still work on weekends
While the recovery process is fairly simple for the most part, tech experts estimate that it takes an average of 15 minutes to recover each system because administrators must have physical access to each system. Additionally, companies that have encrypted their hard drives using BitLocker (a cybersecurity best practice, especially for laptop systems) must find and enter the encryption key at the beginning of the process.
“There’s no way to do this remotely, because you have to run it in safe mode, so networking doesn’t work and you can’t connect to the machine remotely,” said Vadim Vladimirskiy, CEO of virtual desktop management company Nerdio.
At least 700 outages occurred during the same period as CrowdStrike’s update glitch, 39% of which were rated “critical.” Source: Parametrix Solutions
Nerdio, which is in the business of providing virtual desktops to customers, said the failed update had minimal impact on its customers and that its cloud desktop systems were easily repaired by restoring them to a previous image. Many customers use Windows computers to connect to Nerdio’s service, but only systems that remained on during the 78 minutes that CrowdStrike’s faulty update was distributed were affected. Affected customers were able to access their virtual desktops simply by switching to another system, Vladimirskiy said, so the impact was minimal.
Ironically, healthcare companies recovered by relying on the measures they put in place to protect themselves against the very threat CrowdStrike is putting in place to defend against: ransomware. Health-ISAC’s Weiss compiled a list of systems affected by the attack, including patient services, lab result collection, secure file transfer, dictation and transcription services, shipping, electronic medical records, and Medicaid and insurance billing.
“I started hearing about the impacts to these organizations and I saw the list and I thought this looked like another ransomware incident,” he said. “And that’s exactly what happened in healthcare on Friday. Affected organizations said, ‘Okay, our systems are down. We’re going to switch to manual backup procedures and we’re going to back them up on paper,’ and they knew what to do because they had been trained beforehand. [their response to ransomware] In the past.”
Prevent the next big mistake
The bad update also came after a major outage of Azure services affected a higher-than-average number of companies, according to Parametrix Solutions. (On average, Fortune 500 companies experience roughly 300 outages each day, according to the company. On Thursday, July 18, 419 outages coincided with the Azure outage, and on Friday, at least 700 outages occurred as the company addressed a bad update from CrowdStrike.)
Although CrowdStrike is currently feeling the wrath of the market, it isn’t likely to be down for long because businesses need the type of services it and companies like it offer, Quest Software’s Feid said.
“No software development company is perfect, and we’re not one of them,” he said. “What’s difficult, especially in the security industry, especially for a company like CrowdStrike, is that you’re looked to and relied on by a large portion of the market to protect their endpoints… and your products are specifically designed to be as ahead of the curve as possible. So for consumers, you can’t have both. There’s always an inherent risk.”
