CrowdStrike recently caused a widespread Blue Screen of Death (BSOD) issue on Windows PCs, disrupting various sectors. However, this was not an isolated incident, CrowdStrike affected Linux PCs also.
In April, a CrowdStrike update caused all Debian Linux servers in a civic tech lab to crash simultaneously and refuse to boot.
And then, you boot their servers from a Linux Live USB, run TimeShift to restore the last system snapshot, refuse the latest patch from Cloudstrike and they all lived happily ever after.
If I ran a computer lab that wasn’t already net booted, I’d use this as the motivating factor to put that in place. Net booting to a repair image, or just reinstalling the whole OS either from scratch or a known good disk image, is where anybody who manages a fleet of computers should be.
There was a point in time where I had a pxe boot server vm set up on my laptop that I used to reload servers in our little row of racks at 365 main, because it let me quickly swap out the boot iso, and was faster than usb sticks were at the time.
Anybody who doesn’t already have ipmi serial console access set up needs to put that on their list of acceptance criteria for remediation of this incident.
In April, a CrowdStrike update caused all Debian Linux servers in a civic tech lab to crash simultaneously and refuse to boot.
And then, you boot their servers from a Linux Live USB, run TimeShift to restore the last system snapshot, refuse the latest patch from Cloudstrike and they all lived happily ever after.
And on Windows you booted in safe mode and removed one file. What’s the point of your post?
And it’s not much more difficult to fix on Windows, except for the scale of the problem.
If I ran a computer lab that wasn’t already net booted, I’d use this as the motivating factor to put that in place. Net booting to a repair image, or just reinstalling the whole OS either from scratch or a known good disk image, is where anybody who manages a fleet of computers should be.
There was a point in time where I had a pxe boot server vm set up on my laptop that I used to reload servers in our little row of racks at 365 main, because it let me quickly swap out the boot iso, and was faster than usb sticks were at the time.
None of these things are used in actual server operations.
Good luck doing that remotely. Which is the sole problem with this most recent CrowdStrike bug.
Anybody who doesn’t already have ipmi serial console access set up needs to put that on their list of acceptance criteria for remediation of this incident.