You know, any of these software platforms or or systems that we depend on could potentially lead to, you know, in in event like this. How should businesses be thinking about, is this gonna happen again? What should we do if it does? Hello, everyone, and welcome to Unraveling IT: Expert Tech Talks. I’m your host, Brian Harmison, CEO at Corsica Technologies. I have with me today, Ross Filipek, our chief information security officer here at Corsica. Welcome, Ross. Thanks, Brian. Glad to be here. Alright. Well, it’s no coincidence that we’re here, with our CISO, and going to talk about CrowdStrike. So, we had an event that happened here in the last week. It started actually just over a week ago and, was pretty impactful around the world. Ross, you want to kind of just give us the quick breakdown of what happened? Yeah. Yeah. Sure. So in this particular, incident, CrowdStrike released a content update for the Falcon EDR slash MDR platform. So that is CrowdStrike’s endpoint security agent, and they release content updates all the time. This was nothing, you know, particularly fancy- Couple times a day. Right? Yeah. In fact. So pretty routine stuff. Unfortunately, in this case, the content update included a faulty driver, which, unfortunately, as soon as it was deployed to really any Windows endpoint that was running CrowdStrike, basically, blue screened the device and threw it into a reboot loop. And, of course, you know, that essentially, ruined availability for everybody’s machine. Yeah. And so it impacted those primarily that had their machines on- Yes. Over last Thursday night and required, you know, physical- Yeah. -intervention on the machine, which is what makes this even more impactful. It wasn’t- Yeah. It wasn’t something we could deploy live a fix for- Right. Until this week, and even that is not a hundred percent reliable. Yeah. Yeah. That’s true. So, you know, unfortunately, the reality is, you know, if the machine’s not online, you know, your IT department really doesn’t have a way to get to get to that remotely to be able to fix it. So, yeah, highly impactful even though the fix itself is, you know, pretty straightforward. Right. Right. It was boot in safe mode, get a command prompt- Yeah. Delete a file, reboot. Yeah. Yeah. So let’s talk a little bit about, you know, kind of the mechanics behind this. So, you know, I think a lot of us have run into Windows issues where the machine’s able to repair itself. In this particular case, you know, all the machines went through a kind of a repair cycle, but were not able to. And that’s because of where this CrowdStrike Falcon driver, and I noticed you called it a driver, lives. Can you talk a little bit about that? Yeah. So CrowdStrike Falcon, really all modern endpoint security software, runs in what’s called kernel mode on machines. And that gives the software itself very high privileges, to be able to protect even the core of that machine, operating system wise. So, you know, unfortunately, we’ve seen, you know, many times in the past, the more privileges something has, the more potential for there to be, an undesirable effect. So, you know, that’s why, you know, not to sidetrack, but we talked about the principle of least privilege so often- Right. -In cybersecurity. So usually that is geared toward restricting, user account privileges. You don’t wanna give an account more privileges than it really needs to get his or her work done. So- Right. This is, you know, sort of, an example of, you know, that, you know, taken to an extreme where- Yeah. You know, if you’ve got your end users running around with domain admin rights, for instance, you know, very high, privileges, the potential for them to, you know, cause something that you don’t want is, you know, increased. Right. But in the case of, of endpoint protection, EDR specifically, it really needs to live there. That’s- As you said. Yeah. That’s you know, in order for that software to do what it needs to be able to do, as effectively as it can, it only needs to be able to get as deep inside that operating system as it can. So, hence, kernel mode. Right. So it’s loaded as a driver. Typically, drivers go through a certification process- Mhmm. That make sure that they don’t blue screen the system, which the CrowdStrike driver Falcon driver has gone through. Yes. But what makes it unique is because it’s not hardware, it loads software dynamically as needed, which is those updates, right, that you were talking about. So, really, we could be vulnerable that any endpoint protection software- Sure. Operates under similar principles. Yeah. I mean, even any operating system update, you know, just doesn’t necessarily have to be, you know, something that’s related to, endpoint protection specifically. I mean, really, you know, any, you know, Windows hotfix could potentially do the same thing. Right. And we’ve certainly seen that in the past. You know, one of one of the things that comes up a lot is, you know, patch delays. So the way we get past the issue that that you brought up, which is Microsoft releases an update that blue screens the machine. You know, for us as a company that patches thousands of systems, you know, we typically introduce a delay where we let, you know, the rest of the world and our test team figure out Yeah. Is this going to cause a problem or not? You know, I’ve heard people say things like, well, why wasn’t this delayed? Why didn’t providers like ourselves delay that? Can you talk a little bit about that? Yeah. Yeah. So in the case of endpoint protection software, you, as a general rule, are going to have many more problems by intentionally delaying content updates than you will by allowing this to be applied immediately. Because if you think about what the purpose of endpoint protection content updates are, Yeah. They’re generally to address, you know, new vulnerabilities that are being, you know, exploited, be able to detect new types of attacks that could be targeting those machines where we’re protected. So, you know, whereas operating system hotfixes, you know, a lot of times those do address vulnerabilities, but a lot of times it’s, you know, patching, you know, buggy code, things like that. So we think it’s very important to maintain, you know, as soon as those content updates are available for endpoint production specifically to get those installed. Yeah. That makes sense. And often, we’re counting on the endpoint protection helping give us a time window on- Yeah. On a lot of those other patches because it has that more timely information and inability to protect, you know, both the kernel and the rest of the machine. And I mean, that that’s not just Corsica saying that either. I mean, if you look at any reputable cybersecurity framework, you know, these things invariably include elements about making sure that endpoint protection software updates automatically, in a timely fashion. So, in in the spirit of that, you know, it’s really critical to allow the updates to proceed. Yeah. That makes sense. So as we think about this event, it gets me thinking, well, what could happen next? And I think it’s opened a lot of eyes, that you know, any of these software, platforms, or systems that we depend on could potentially lead to, you know, an event like this. And, you know, fortunately, even though CrowdStrike’s in the news for some data leaks, you know, those are kind of old news compared to this event. We don’t think there’s been any actual cybersecurity breach. It’s just a coding error, that had a really big impact. How should business be thinking about, is this going to happen again? What should we do if it does? Well, so I think the safe bet will be that, yes, it will happen again, not necessarily through CrowdStrike, but, you know, sooner or later, there’s going to be something else. So, you know, this occurrence was Crowd Strike. I mean, the next occurrence could be, you know, another EDR product, and we’ve seen that- Sure. -In the past. You know, like we talked about, it could be an operating system hotfix. I mean, the fact that, we are reliant on computing technology, I just we can never suppress that vulnerability entirely, so that’s always going to be there. But I think this particular situation really underscores the importance of business continuity. So, being able to sit down and figure out from a business process, you know, what are the, critical functions that we need to be able to provide? And then what are the underlying technologies that support that? So business continuity, disaster recovery. Conceptually, it just you know, I talked to a lot of clients who are kind of behind, where they, really should be on that. So definitely worth the time investment to, get those plans developed. Yeah. I think my experience has been, much like a lot of areas of life. Mhmm. You know, people tend to not take action until there’s, you know, that that tragedy or that event. And so I’ve been using this as an opportunity to encourage businesses. Look at this as, you know, one of the types of things that could happen that impact your business. Ultimately, you know, those of us in technology, our goal is to protect the revenue generating capabilities of a business, which is what business continuity disaster recovery is all about. How do we get that business to a point where the recovery is seamless? You know, it’s not going to be without pain because, you know, this this was a great example. You know, we had just about everyone in the company assisting people and walking them through those manual steps. But knowing how is my business going to run if we come in tomorrow and our systems are down. You know, car dealers recently experienced it a big, long outage due to some ransomware and a SaaS application. Mhmm. So while we’re talking about kind of the security piece that there’s a bigger dependency now on more of these cloud-based systems. And when they go down, it’s not just your server room that’s down anymore. It’s probably everybody. So getting those plans in place is really the right thing to do regardless of what that kind of next event is going to be. Absolutely. Yep. Yeah. So well, thanks, Ross. This has been really helpful. I’m sure we’ll talk more about this event and how to plan in the future. It’s been one of those opportunities for us to have a chance to educate, you know, folks that that maybe aren’t as interested in in that or don’t understand some of the mechanics of how the software that’s protecting them works. And that’s an area that I find really interesting. Yeah. Me too. Because it’s the core that a lot of our services are built upon. So thanks for joining me today. Appreciate your insight and your thoughts. Yeah. Thanks for having me.
