If you’ve been working in IT for as long as I have been, you’ll maybe remember this wonderful example of legalese:
NOTE ON JAVA SUPPORT: THE SOFTWARE PRODUCT MAY CONTAIN SUPPORT FOR PROGRAMS WRITTEN IN JAVA. JAVA TECHNOLOGY IS NOT FAULT TOLERANT AND IS NOT DESIGNED, MANUFACTURED, OR INTENDED FOR USE OR RESALE AS ONLINE CONTROL EQUIPMENT IN HAZARDOUS ENVIRONMENTS REQUIRING FAIL-SAFE PERFORMANCE, SUCH AS IN THE OPERATION OF NUCLEAR FACILITIES, AIRCRAFT NAVIGATION OR COMMUNICATION SYSTEMS, AIR TRAFFIC CONTROL, DIRECT LIFE SUPPORT MACHINES, OR WEAPONS SYSTEMS, IN WHICH THE FAILURE OF JAVA TECHNOLOGY COULD LEAD DIRECTLY TO DEATH, PERSONAL INJURY, OR SEVERE PHYSICAL OR ENVIRONMENTAL DAMAGE.
Windows NT4 License agreement
It’s a pretty good example of where our minds tend to go when you think of “safety critical” systems. I tend to also think about things like complex train automation systems or the Therac-25 radiation therapy machine.
All things that are complicated, but are generally grounded in physical interactions with machinery, machinery that has high energy, or that interacts with other humans.
This came to mind because once again the Post Office Horizon Scandal, one of the biggest miscarriages ever in justice in the UK, is in the news.
If you weren’t aware, the system was buggy, could cause the branch to have massive shortfalls, giving postmasters three options.
- Make up the loss up themselves, and hope the problem didn’t happen again
- Report that their accounts balance, which was an act of fraud
- Try to report to the Post Office which would be unhelpful at best, or began an investigation at worse
The results of this were bleak:
- People were wrongly convicted of fraud/stealing from the Post Office
- People were wrongly imprisoned
- Some people ended their lives in the immense shame of being someone who stole from their local community
In hindsight, that looks pretty safety critical… lives were materially changed, damaged, or extinguished.
What’s worse is that people from the software vendor, and the post office claimed that the system was robust, that remote access wasn’t possible – at the same time as planning remote access to resolve issues caused by known bugs.
The latest BBC Radio 4 program on this (after an amazing series), had an instance where a Post Master lost his branch due to these bugs, a new owner bought the shop, only to then experience the same bugs. The helpline gave the same line “Nobody else is reporting these problems” which sounds highly unlikely to be true.
Sure some senior people at the time have stepped down from their non-exec directorships.
In my view this is either negligence as they should have done the due diligence to ascertain that the system was generally robust.
Software is everywhere.
Ovens have Wifi, cars have highly complex computer vision, human bodies have attachments controlling insulin flows. People had artificial eye implants to help them see, that the manufacturer no longer supports.
Whistleblowing is a painful and sacrificial act for the person who does it.But if you see people from your company, testifying in courts of law that “there are no problems with the software” (an impossible situation in all by the smallest of programs), we need to provide better ways to help this information surface.
Maybe if defence teams were better briefed, a statement like that could be countered with “No problems? Cool, we’ll verify that with an extract from your Jira instance” or “a third-party code review wouldn’t be a problem”?
I don’t know the solution.
I’m not a lawyer, I’m not an ethicist, I’m not someone who typically works in these kinds of environments – but I do know that lives were lost due to an accounting system being buggy.
And that doesn’t sit right with me.