Last updated: 2024-07-22 1528 UTC
Updated 2024-07-22 1528 UTC
As noted in a social media post on July 21, 2024 at 21:06 UTC, CrowdStrike has been working with customers to test a new technique to accelerate the remediation of affected systems. We are in the process of operationalizing an opt-in for this technique. We encourage customers to follow our Tech Alerts to stay up to date and be notified when action is required.
We have published a video outlining the steps required to self-repair an affected remote Windows laptop.
We will continue to provide updates here as information becomes available and new fixes are deployed.
CrowdStrike is actively assisting customers who were affected by the recent content update glitch on Windows hosts. Mac and Linux hosts were not affected. The issue was identified, isolated, and a fix was deployed. This was not a cyber attack.
We encourage customers to check our support portal for updates, and we will continue to provide updates here and on our blog as they become available. We encourage organizations to ensure they are in contact with CrowdStrike representatives through official channels.
CrowdStrike is operating normally and this issue does not impact Falcon platform systems. If your system is operating normally, protection will not be affected even if Falcon sensors are installed.
We understand the seriousness of this situation and sincerely apologize for any inconvenience or trouble it may have caused. Our teams are committed to ensuring the security and stability of CrowdStrike customers.
overview
Statement from the CEO
Date sent 2024-07-19 1930 UTC
Dear valued customers and partners,
I would like to personally and sincerely apologize to you for the outage you experienced. Everyone at CrowdStrike understands the severity and impact of this situation. We were able to quickly identify the issue, apply a fix, and focus on restoring our customers’ systems as our top priority.
This outage is caused by a flaw found in a Falcon content update for Windows hosts. Mac and Linux hosts are not affected. This is not a cyber attack.
We are working closely with affected customers and partners to ensure that all systems are restored and we can provide the services you rely on.
CrowdStrike is operating normally and this issue does not impact Falcon Platform systems. If Falcon sensors are installed, protection is unaffected. Falcon Complete and Falcon OverWatch services are not interrupted.
We will provide ongoing updates via our support portal: https://supportportal.crowdstrike.com/s/login/.
CrowdStrike is committed to helping you and your team, so if you have any questions or need additional assistance, please contact your CrowdStrike representative or technical support.
We know that adversaries and malicious actors will attempt to exploit events such as these, so we urge everyone to remain vigilant and stay in touch with official CrowdStrike representatives. Our blog and technical support will continue to be the official channels for updates.
Nothing matters to me more than the trust and confidence our customers and partners have in CrowdStrike, and as we resolve this incident, I am committed to providing full transparency about how this issue occurred and the steps we are taking to ensure that this never happens again.
George Kurtz
Founder and CEO of CrowdStrike
Technical details
Technical details about the outage can be found here. Read the blog Published: 2024-07-19 0100 UTC CrowdStrike is operating normally and assures customers that this issue does not impact Falcon platform systems. If your system is operating normally, protection will not be affected even if the Falcon Sensor is installed. Falcon Complete and OverWatch services are not interrupted by this incident. CrowdStrike determined that the cause of this issue was the deployment of Windows sensor-related content and has reverted those changes. The content is channel files located in the %WINDIR%\System32\drivers\CrowdStrike directory. Channel files “C-00000291*.sys” with timestamps later than 2024-07-19 0527 UTC are the reverted (good) versions. The affected version is the channel file “C-00000291*.sys” with a timestamp of 2024-07-19 0409 UTC. NOTE: It is normal for multiple “C-00000291*.sys” files to be present in the CrowdStrike directory. If one of the files in the folder has a timestamp later than 05:27 UTC, it will be active content. Symptoms include hosts experiencing bugcheck\blue screen errors related to the Falcon Sensor. For unaffected Windows hosts, the affected channel file has been reverted and no action is required.
Unaffected Hosts
Windows hosts that were brought online after 2024-07-19 0527 UTC are not affected. Windows hosts that were installed and provisioned after 2024-07-19 0527 UTC are not affected. Updated to 2024-07-21 1435 UTC. This issue does not affect Mac or Linux based hosts.
How do I identify affected hosts?
How can I identify affected hosts using advanced event search queries?
Updated 2024-07-22 0139 UTC
The queries used in the dashboards are listed at the bottom of the applicable dashboard KB article.
How do I identify affected hosts via the Dashboard?
Updated 2024-07-22 0139 UTC
An updated, detailed dashboard is available showing Windows hosts affected by the content update failures described in this technical alert. See Detailed Status Dashboard Identifying Windows Hosts Affected by Content Issues (v8.6) (pdf) or log in to view it in the Support Portal. Note that the queries used in the dashboard are listed at the bottom of the applicable dashboard KB article.
If your host continues to crash and is unable to stay online to receive channel file updates, you can use the repair steps below.
How do I remediate an individual host?
Updated 2024-07-21 0932 UTC
Reboot the host so it can download the reverted channel files. It is highly recommended to connect the host to a wired network (not WiFi) before rebooting. The host has a much faster Internet connection over Ethernet. If the host crashes again upon reboot: Option 1 – Manually See this Microsoft article for detailed steps. Note: Bitlocker encrypted hosts may require a recovery key. Check out the following video on CrowdStrike host self-remediation for remote users. Follow the steps in the video if instructed to do so by your organization’s IT department. Updated: 2024-07-22 1510 UTC Option 2 – Automatic via bootable USB key
How do I recover my Bitlocker key?
Updated 2024-07-21 1810 UTC
How to recover cloud-based environment resources
Cloud Environment Guidance
Amazon
AWS Article
Azure
Microsoft article
(PDF) or log in to the Support Portal to view
Public Cloud/Virtual Environment
Option 1:
Detach the operating system disk volume from the affected virtual server. As a precaution against unexpected changes, take a snapshot or backup of the disk volume before proceeding. Attach/mount the volume to the new virtual server. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory. Find and delete files matching “C-00000291*.sys”. Detach the volume from the new virtual server. Reattach the pinned volume to the affected virtual server.
Option 2:
Roll back to a snapshot before 2024-07-19 0409 UTC