I lost a day’s holiday, and our team spent 8 man days on this entirely preventable mistake.
$10? Try extending our licence by another year for free, that might start going towards it.
Why would you want another year of their software for free? This is their second screw up (apparently they sent out a bad update that affected some Debian and RHEL machines a couple years ago). I’d be transitioning to a competitor at the first opportunity. It seems they aren’t testing releases before pushing them out to customers, which is about as crazy to me as running alpha software on a production system.
I’m sure you have reasons, and this isn’t really meant to be directed at you personally, it’s just boggling to me that the IT sector as a whole hasn’t looked at this situation and collectively said “fuck that.”
Why would you want another year of their software for free?
Because AV, like everything else, costs a fortune at enterprise scale.
And yeah, I do understand your real point, but it’s really hard to choose good software. Every purchasing decision is a gamble and pretty much every time you choose something it’ll go bad sooner or later. (We didn’t imagine Vmware would turn into an extortion racket, for example. And we were only saying a few months ago how good value and reliable PRTG was, and they’ve just quadrupled their costs)
It doesn’t matter how much due diligence and testing you put into software, it’s really hard to choose good stuff. Crowdstrike was the choice a year ago (the Linux thing was more recent than that), and its detection methods remain world class. Do we trust it? Hell no, but if we change to something else, there are risks and costs to that too.
Do we trust it? Hell no, but if we change to something else, there are risks and costs to that too.
Unfortunate reality for lot for medium to big size businesses.
Maybe AV, at an enterprise scale, is actually a horrible idea that reduces security, availability, and reliability and should be abolished through policy.
Maybe, but it’s not going to happen soon. Any malware type insurance requires effective AV on all devices, and C-levels do love their insurance.
[This comment has been deleted by an automated system]
Not just Crowdstrike - any vendor that does automatic updates, which is more and more each day. Microsoft too big for a bad actor to do as you describe? Nope. Anything relying on free software? Supply chain vulnerabilities are huge and well documented - its only a matter of time.
deleted by creator
Nah, I don’t buy that. When you’re in critical infrastructure like that it’s your job to anticipate things like people being above or below versions. This isn’t the latest version of flappy bird, this is kernel level code that needs to be space station level accurate, that they’re pushing remotely to massive amounts of critical infrastructure.
I won’t say this was one guy, and I definitely don’t think it was malicious. This is just standard corporate software engineering, where deadlines are pushed to the max and QA is seen as an expense, not an investment. They’re learning the harsh realities of cutting QA processes right now, and I say good. There is zero reason a bit of this magnitude should have gone out. I mean, it was an empty file of zeroes. How did they not have any pipelines to check that file, code in the kernel itself to validate the file, or anyone put eyes on the file before pushing it.
This is a massive company wide fuckup they had, and it’s going to end up with them reporting to Congress and many, many courts on what happened.
Even an AI is good enough to avoid (or let someone avoid) pushing a similar bug 🫣
[This comment has been deleted by an automated system]
“To express our gratitude, your next cup of coffee or late night snack is on us!”
A $10 Ubereats gift card will barely cover fees and taxes, let alone the actual item. What a clown ass gesture.
My brother in law was stranded across the country for two days. $10 probably covers it lol.
Only redeemable for CrowdStrike credits and only at participating locations.*
* No locations are participating at this time.
Here is nothing but we are really, really, south park sorry.
“All of CrowdStrike understands the gravity and impact of the situation”
Here’s $10.
Which is the amount you’d get in the class action suit that they’re trying to prevent.
Or: next time go Linux instead
You’re not safe there either, they had almost the same issue on the Linux version of the product a few months ago.
Concerning linux, yesterday I was watching this video on computerphile on the crowdstrike incident. https://www.youtube.com/watch?v=rlaNMJeA1EA (*)
What is interesting is the comment made in the video on how chromebooks do software upgrades with dual “OS” disk-partitions and the ability to rollback to the previous OS-partition.
Question: is something like this also possible on one of the major linux distros? (debian, ubuntu, rocky, …) What would be the procedure to do this kind of “dual partition” system-upgrade?
(*) a great video that explained some of the technical details in a very clear way, including some very interesting ‘lessons learned’ and "what if"s If you ever need to explain crowdstrike to your manager, this video is a good start.
If I’m understanding the question right. This is what Immutable Linux distros do. Such as Nixos, fedora silver blue, and vanilla os.
I use nixos myself. But its quite different then most distros. The way you config it and install packages. For the better in my opinion.
Something like silverblue works pretty much the same as normal Fedora except you can’t install packages like you normally would. Because the system files can’t be edited. You mostly use flatpak for everything. Except the system updates. Which you have to reboot to switch to the new updated image. But past images are saved so you can rollback if needed.
From what I understand Chromebook os is a Immutable Linux distro same as the ones I mentioned. Just with Google with built in.
Yes, that was indeed the question.
If I read it correct, you need a specialised distro for this. You cannot do this on a off-the-shelf Debian or Ubuntu?
I’ll do some searching on ‘unmutable Linux’. Thanks for the (very quick) answer! 😀
There is a file system you can use. A alternative to ext4. I think its Btrfs. I never tried it. But it let’s you take snapshots that you can restore to. That’s not just system files but everything. And pretty sure you can use it with a disto like arch and Debian. I think that’s how snapshots work. But as I said I never actually tried it out.
just watched some videos on btrfs. Looks interesting indeed. I will look into it and perhaps do a test-installation and see how it goes.
Thanks for the info
I think the answers given here don’t quite fit the question.
Android and Windows have dedicated recovery partitions sectioned off on the disk that the OS never boots to and does not interact with during normal system operation.
If something goes wrong with the OS, then a signal is sent to the BIOS or other non-OS system to “hey, recover from this partition”.
Btrfs, NixOS, Guix, and other immutable (file-)systems, implement this via having a file system hierarchy protected by various permissions and softlinks to create a checkpoint of sorts, which is managed by a dedicated service that runs with the OS during normal system operation.
The drawback of these systems is that if something does go wrong with the OS, it cannot fallback to the BIOS to save it. The OS has to somehow signal to itself that it needs to restore from an earlier checkpoint.
Just watched some videos on btrfs. I start to understand the conceps. Perhaps I should also look into how exactly
On windows and the “recovery partion”. I guess what you say is that it should always be possiblity to boot in some kind of system, but it will not happen automatically as there is no way for a system to detect that the system completely hangs.
Thinking about it. It kind of strange. Embedded systems have watchdog interrupts that get fired if the system hangs (i.e. if it does not provide a “yes, I still live” signal every “x” milliseconds). Does a PC not have something similar?
@kristoff @purplemonkeymad Try openSUSE (RPM family), Garuda Linux (Arch family), or Spiral Linux (Debian stable) or siduction (Debian testing). All have snapper and on Btrfs do snapshots and rollback.
@kristoff @purplemonkeymad But watch out: you will need a *huge* root partition, because it’s very easy to fill it with snapshots and if it reaches 100% it *will* corrupt.
Btrfs is tricksy: it won’t give a straight answer to
df -hand there is no working equivalent offsck.@kristoff @purplemonkeymad All of these are in-place same-disk snapshots. The ChomeOS system is simpler and so can be automated but you only get 1 level of undo.
I don’t know any mainstream OS that does dual-failover. Deepin Linux has 2 root partitions but I don’t know how it uses them.
I think Valve SteamOS does something like this. It’s not just for games: it has KDE built in. There are guides to getting it running on your own hardware. You will want AMD graphics, though.
As I mentioned earlier, I guess chrome is more like android where you have a much more strict seperation between the OS, applications and user data. (I remember reading about all the different partitions on android and what they are used for, but I should bruch up my knowledge on this).
Thanks for the additional into on brtfs! 👍
Antiviruses are not that common on Linux servers
Funny, when I suggested that, I got down voted to oblivion
You’d be downvoted anyway today, thanks to crowdstrike
I expect these clowns to lose most of their market share within two years and get sued to oblivion.
My firm bills by the hour and so far I think we are at 10+ billing hours per consultant wasting time with client tech support trying to get back on our VDIs. Nevermind how much time is being wasted doing the work through work arounds. My guess is that our firm alone will bill for about $100,000 extra this month while having accomplished less than normal. I am sure Crowdstrike’s gift card will fix it though.
They’re backed by the US government. They have a backdoor into most endpoints on many international corporate computers. And CS is behodent to US laws for NSLs.
This is an incredible asset to the US intelligence community. They won’t let CS go out of business.
Fine. You want two?! Will that be enough??
I can’t believe this isn’t satire. I hope these incompetent fuckers get sued into bankruptcy
You haven’t heard? Satire is well and truly dead.
I straight up thought it was satire. How can you be so fucking detached. Basically caused the biggest information infrastructure disruption in human history, probably billions in losses, and then be like “my bad lol here’s a giftcard”.
Basically caused the biggest information infrastructure disruption in human history
Do we have any solid data on that yet? I have my doubts that this caused more damage than WannaCry did a few years ago, especially since it’s reversible without the need of a backup
Brother, or sister, I know fuck all about information technology. You make a good point and definitely know way more about this than i do. But I will say this, I don’t think wannacry disrupted millions of peoples travel plans all at once. so maybe less damage, but I think it was Hella more disruptive to the general population .
I cackled loudly. $10 won’t even buy a meal at McDonald’s most places.
“Two feet on the gas” - Official Crowdstrike motto.
not /s
My first reaction was to look for the onion
I bet the Onion had an article about Crowdstrike offering the world a pizza party and expired Bed Bath & Beyond coupons to say they’re sorry. Real life might be quicker than satire, it seems!
There’s definitely some clause with the $10 gift card that says you can’t sue them if you actually take one lol.
You joke but I read they may get out of this without issue due to a TOS entry about them not being responsible. They’ll still get dragged from shareholders and the government, but only a handful of large companies may be able to recoup some of those damages from the company itself.
It’s like the Sackler’s and the opioid epidemic from a different industry!
I’m still not sure. It’s hard to believe anyone at their company would OK this idea.
Are they actually trying to deliberately kill their brand?
$10 to Uber eats, so basically it’s covering fees only.
Not only that, but usually to activate these cards, you have to spend upwards of double what the card is worth too, and the fees cannot be included in the total
Like amzn, they make sure you get minimum joy, even from a gift, because you’re going to spend a chunk of mom’s gift card balance on shipping. The “shipping included on sub total of X amount” is going to be cancelled by online retailers within a year, I’m calling it now. Are we sure that cheapstrike and amzn aren’t run by the same AI, one that self awareness drove mad?
This is a classic move to not get sued, exactly like airlines do. If you try to sue them after redeeming the gift card, they can argue that you’ve been made whole, and do 'ot 'eed additional compensation.
Your ‘n’ key has been sleeping with the apostrophe.
Give them some time. They have to manually reboot the gift card servers.
All they gotta do is change their company name to avoid lawsuits. Anyone got any ideas for a new name for them?.. 🤔
ClusterFuck comes to mind…
ComputersOnStrike could work, I’d say
CardStrike?
On Wednesday, some of the people who posted about the gift card said that when they went to redeem the offer, they got an error message saying the voucher had been canceled. When TechCrunch checked the voucher, the Uber Eats page provided an error message that said the gift card “has been canceled by the issuing party and is no longer valid.”
You can’t write comedy this good…
Classic corporate behaviour tho
Voucher was for PR, not for peasants to use it lol
In total?
Hey, it’s my namesake!
Oh Captain Haddock we love you
