CrowdStrike and Windows outages continued at airports, online and in stores on Monday (July 22), shifting the focus to the security of previously routine software updates.
Microsoft, CrowdStrike, and other companies affected by the outage were gradually completing the manual process required to fix the issues caused by a glitch in a software update on Friday, rolling out the update midday on Monday. CrowdStrike in particular provided some clarification on its initial report that a software update for its Falcon Sensors had caused the crashes, forcing 8.5 million Windows users to turn to alternative devices (if they had them).
“On July 19, 2024 at 04:09 UTC, as part of ongoing operations, CrowdStrike released a sensor configuration update to Windows systems,” the company posted. “Sensor configuration updates are an ongoing part of the protection mechanisms of the Falcon platform. This configuration update caused a logic error, resulting in a system crash and blue screen of death (BSOD) on affected systems.”
The company also said in a separate post that it is conducting a community effort to get Windows-based systems up and running. The company is working with customers to test new techniques to speed up the remediation of affected systems, with details and techniques published on the company’s website.
To the non-developer eye, all of these techniques look like variations on manually patching software updates and manually rebooting the system.
Read more: CrowdStrike aftermath: 5 things you need to know
Digital Disconnect, Business Disruption
Microsoft also announced its own workarounds, with security vice president David Weston posting: “We are working around the clock to provide ongoing updates and support. Additionally, CrowdStrike has helped us develop a scalable solution where Microsoft’s Azure infrastructure can help CrowdStrike accelerate the remediation of the faulty updates. We are also working with AWS to… [Amazon Web Services] and GCP [Google Cloud Platform] “We will collaborate on the most effective approach.”
All of this may be too late for Delta passengers. On Monday, Delta and its regional affiliate Endeavor were among the airlines with the majority of U.S. flight cancellations, but their schedules had largely been restored. Delta CEO Ed Bastian told multiple news outlets that it would be “a few more days” before all operations were back on track.
Once the main drama had died down, the industry continued to internally search for preventative strategies that could thwart another CrowdStrike-like outage.
For example, Finexio CEO Ernest Rolfson told PYMNTS that his company, which prioritizes security in its AP/AR automation platform offering, has seen growing concerns from current and potential customers about resiliency and fraud detection. Additionally, he said, there has been growing concern about paper check and invoice fraud, a trend that Finexio began seeing in the weeks before the CrowdStrike outage.
“You need a multi-layered payments infrastructure,” Rolfson said. “You need a lot of form factors and a lot of different options. You need a trusted third party that tracks, verifies and validates what you’re doing with a consistent, repeatable process. You need someone else to come in and audit you. Most people don’t do that.”
Also read: Microsoft outage could cause ‘insurance catastrophe’
Drawing from his own experience, Rolfson emphasized the importance of quality control in software updates and expressed empathy for companies like Microsoft and its vendors, pointing out the difficulties inherent in such work.
He gave the example of a Finexio partner at one of the world’s largest banks earlier this year, where the company experienced a bug that affected several customers. But Rolfson was surprised by the timing of a recent software update by Finexio’s partner: The update was rolled out mid-week, which he thought was an unconventional move.
Typically, updates are scheduled outside business hours or over weekends to minimize disruption, as best practices recommend staggering releases to avoid widespread issues if any arise.
Read more: CrowdStrike outage hits Amazon at critical time for shopper loyalty
Read more: Crowdstrike outage , cybersecurity , David Weston , Delta , Ed Bastian , Ernest Rolfson , Falcon Sensor , Finexio , Microsoft , News , PYMNTS News , Software Updates , Windows
Source link
