By now, just about everyone has heard of the CrowdStrike outage that shut down a good portion of the internet on July 19, 2024. From dentists to airlines, every business running the software was affected—if their Windows machines were turned on at the time of the update.
But what does the CrowdStrike outage mean for midmarket companies? Are there takeaways to avoid this type of issue?
Here’s everything you need to know.
What happened? Why was the CrowdStrike outage so catastrophic?
CrowdStrike released a corrupt driver for their Falcon software, which is an endpoint protection solution. That driver got pushed out to 8.5 million Windows devices around the world that were running the software. The driver is designed to load as the operating system boots, and because it was corrupted, the operating system couldn’t boot properly. Windows detected this and tried to reboot automatically, creating an infinite boot loop.
The fix for this issue is simple, and anyone can do it by following basic instructions. The impact was catastrophic not due to technical complexity, but due to the number of machines that were affected worldwide.
How do you fix this issue?
The fix is simple—you delete the driver that caused the boot loop.
You can do this via command prompt, or you can use CrowdStrike’s updated fix, which works over the internet. Note, however, that you’ll need a wired connection to use CrowdStrike’s fix. Wireless capabilities are often tied to your operating system, so if the OS can’t boot, you may not be able to use Wi-Fi.
How did Corsica Technologies respond to this issue?
Like other MSPs, we had no way to foresee or prevent this issue. It originated with CrowdStrike and their QA (quality assurance) processes for their Falcon software. It’s always best practice to keep cybersecurity software up to date, so we would never recommend not updating a solution like CrowdStrike Falcon.
Although we couldn’t foresee this problem, our team has worked closely with affected clients to resolve it. We initially prioritized clients in critical care services like hospitals, police departments, EMS and fire, and so on. Through our team’s dedication, we had remediated all clients in critical care services by 8AM on Friday, July 19. This ensured that hospitals could go on treating patients and dispatchers could handle 911 calls.
As of this writing, 80% of workstations and 95% of servers that we manage have been remediated. We are actively working with clients to restore the remaining devices.
Does this mean you should avoid third-party endpoint protection software?
In a word… no.
Almost anything installed on a PC has the potential to cause this type of issue. Endpoint protection from a trusted vendor is the least of your worries. This type of software provides far more benefits than risks, although nothing can eliminate the risks associated with third-party software entirely.
Think of it like this. The best way to secure your house is to lock the doors. There’s a risk that you could lose your keys and get locked out of your house. But the benefit of locked doors far outweighs the risk of losing your keys.
How can you better prepare for outages like this?
Unfortunately, there’s no way to prevent issues like this completely. Third-party software is a reality in today’s technology environment. This means there will always be potential conflict between operating systems and other software.
Rather than seeking total prevention, which is unattainable, companies should focus on resilience. Every business should take certain steps to protect revenue, employees, and customers during any outage, regardless of its cause.
If you already know those steps, and you’ve put them into practice, you may be in good shape. However, most companies benefit from engaging a true technology partner to build resilience. If you haven’t experienced these issues before, or if it’s been a while, it makes a world of difference to have a seasoned team advising you on how to prepare—and responding when the unthinkable happens.
Here’s what we recommend, implement, and manage for our clients to help them become more resilient.
- Solutions and services for backup and disaster recovery
- Regular cybersecurity risk assessments
- Adherence to a cybersecurity framework
- Regular evaluation and application of appropriate patches
- Regular consulting with a dedicated vCIO/vCISO (virtual CIO and CISO)
Moving forward: Finding a true technology partner
In today’s complex technology environment, it’s not enough for an MSP to handle break/fixes and support requests. These things are essential, but they’re not sufficient. Companies also need next-level strategic guidance from a partner who’s focused on mutual success. This takes dedication and experience—including visibility into what has and hasn’t worked for many different clients.
Here at Corsica Technologies, our focus is to act as a true partner to every client, offering unmetered technology services, one predictable monthly price, and C-level expertise to help you run better and build resilience. Contact us today to learn more.
Need help with disaster preparedness and backups?
Reach out to schedule a consultation with our security specialists.
