I've said this elsewhere but the enabling of instant auto-updates on software relied on by a mission critical system is a much bigger problem than kernel drivers.
Just imagine that there's a proprietary firewall that everyone uses on their production servers. No kernel-level drivers necessary. A broken update causes the firewall to blindly reject any kind of incoming or outgoing request.
Easier to rollback because the system didn't break? Not really, you can't even get into the system anymore without physical access. The chaos would be just as bad.
A firewall is an easy example, but it can be any kind of application. A broken update can effectively bring the system down.
There sure are a lot of mission-critical systems and companies hit by this. I am surprised that auto-updates are enabled. I read about some large companies/services in my country being affected, but also a few which are unaffected. Maybe they have hired a good IT provider.
A k8s variety. By Canonical. Screams production, no one is using this for their gaming PC. Comes with.. auto-updates enabled through snap.
Yup, that once broke prod at a company I worked at.
Should our DevOps guy have prevented this? I guess so, though I don't blame him. It was a tiny company and he did a good job given his salary, much better than similar companies here. The blame goes to Canonical - if you make this the default it better come with a giant, unskippable warning sign during setup and on boot.
One thing to consider with security software, though, is that time is of essence when it comes to getting protection again 0day vulnerabilities.
Gotta think that the pendulum might swing into the other direction now and enterprises will value gradual, canary deployments over instant 100% coverage.
Just imagine that there's a proprietary firewall that everyone uses on their production servers. No kernel-level drivers necessary. A broken update causes the firewall to blindly reject any kind of incoming or outgoing request.
Easier to rollback because the system didn't break? Not really, you can't even get into the system anymore without physical access. The chaos would be just as bad.
A firewall is an easy example, but it can be any kind of application. A broken update can effectively bring the system down.