We had a microwave generator that was used to cook cancers in living patients. We'd ask for a given power, and we had the ability to read back how much power we actually got. But we didn't check that the power we read back was something reasonable. When an op amp failed, the generator produced full power whenever we asked for any power at all. The patient literally got hot enough to emit smoke.
Thank God, the patient was a pig. We hadn't made it into clinical use yet.
Although that is true, the point is that the software could have done something (in this particular failure case) since it had the means to monitor the actual power. Like "Oh shit, there is too much power; something is wrong; shut everything down".
No. We added code that, if the power was too far off from what we asked for, tried to kill power three different ways, plus alerted the operator. It was a bit tricky, because the power is never exactly what you ask for (variable impedance match, plus noise).
The specific issue that we encountered was that the power was measured correctly, but was out of control. At that point, not being able to kill the power is a very real concern.
If we measured wrong, we could either be high or low. If we measured high (that is, the reading is higher than reality), we would either turn down the power until it read right, or else kill power completely. If we read low, though, IIRC we would limit how high we'd turn up the gain to try to get the power we were asking for.
There was also a feedback loop based on temperature. If the power was double what we asked for, the temperature would quickly climb, and we'd reduce power. It would have worked, even with inaccurate power readings, though not as smoothly as it should with accurate power readings. But when we got 20 times the power we asked for (due to the power control failure), it was too much too fast.
Thank God, the patient was a pig. We hadn't made it into clinical use yet.