Today, software applications are the backbone of almost all industries. We create systems to manage, automate, and enhance the most straightforward to complex processes.
However, despite the meticulous planning and expertise in our creation, software failures have occurred and will continue to occur, and it does not matter how many testing techniques you include.
I promise today I will not talk much about the day I ran a T-SQL script DELETE statement without a WHERE clause in a production database, deleting all entries in the systems configuration table that put out a 401K bank system for the entire morning; I almost lost my job, but that one (thank God) is not well documented so.
Instead, in today’s article, I will try to create a chronological summary list of those well-documented and most notable software failures in history to gather the lessons from those failures and recall in this corner of the internet the essential lessons they have taught us.
1962
Mariner 1 (July 28, 1962).
Mariner 1, the first American spacecraft intended for Venus, met its fiery demise just 256 seconds after launch due to a simple typo in the software code. A missing hyphen caused the spacecraft to lose control and send incorrect data, leading to its destruction. This error cost around $135 Million, which taught NASA a valuable lesson about the importance of meticulous code verification.
1983
The Vancouver Stock Exchange (1983).
The exchange index was undervalued by approximately 50% after 22 months of accumulated roundoff error. The obvious algorithm is to add up all the stock prices afterward. Instead, an analyst decided it would be more efficient to recompute the index by adding the net change of stock after each trader.
This computation used float decimal places and then truncated (not rounding) the result to three.
At its inception in 1982, the index was given a value of 1000.000. After 22 months of recomputing the index and truncating it to three decimal places at each change market value, the index stood at 524.881, even though its “true” value should have been 1009.811.
Be careful when dealing with decimal values; some rounder errors will cancel out, but generally, one should safely assume that the limits of this rounding error will grow.
1985
Thorac-25 Radiation Therapy Machine (1985-1987)
In the mid-1980s, the Therac-25 was a radiation therapy machine designed to treat cancer. However, due to software bugs, it sometimes delivers massive overdoses of radiation, leading to severe injuries and even deaths. The failures were attributed to programming errors and inadequate safety checks, highlighting the critical need for thorough testing in medical technology and the significance of rigorous testing, validation, and safety measures in mission-critical software systems, especially those used in healthcare.
1987
Rounding error costs DHSS 100 million pounds (1987)
The British government has underestimated inflation by 0.1% for over a year due to a programming error.
More than nine million pensioners received a tax-free lump sum bonus of 7.50 and 12.00 pounds.
After the government admitted that computer error had led to the publication of incorrect inflation figures for the past 21 months, the cumulative effect of this error on payments was around 100 million pounds.
The lesson here is that the compounding principle also works for small quantities.
1988
Morris Worm (1988): The Teenager Who Brought the Internet to its Knees.
In 1988, a 23-year-old Cornell University student named Robert Tappan Morris unleashed the Morris Worm, the first ever self-replicating worm to infect the nascent internet.
The worm clogged networks, brought down critical systems, and caused millions of dollars in damage. This event was a wake-up call for cybersecurity measures in the rapidly evolving digital world.
1991
American patriot missile failure (February 25, 1991).
An American patriot missile failed to track and destroy an Iraqi scud missile. Instead, it hit an army barrack, killing 26 people.
The cause was later determined to be an inaccurate calculation caused by measuring time in tenths of a second.
In binary computers, 1/10 cannot be exactly represented using binary floating-point format.
Again, be careful when dealing with decimal values.
1993
London Stock Exchange’s Taurus Project (1993-1994)
The project to modernize the stock exchange was scrapped after substantial cost overruns and technical issues, eventually leading to its failure.
1994
Pentium FDIV Bug (1994)
Intel’s Pentium chip, released in 1994, was plagued by a math error in its floating-point division unit. This bug caused inaccurate calculations, leading to concerns about its reliability in scientific and financial applications.
Intel was forced to replace millions of faulty chips, damaging its reputation significantly.
And again, be careful when dealing with decimal values.
1996
Ariane 5 Rocket Explosion (June 4, 1996)
The European Space Agency’s Ariane 5 rocket, designed to compete with American launchers, met its fiery end on its maiden flight due to a software error.
A conversion from 64-bit to 16-bit integers in the guidance system caused the rocket to veer off course and self-destruct. This setback delayed Europe’s space program for several years.
The rocket self-destructed 40 seconds after liftoff due to a software issue caused by data conversion from 64-bit floating-point to 16-bit signed integer value. It is understanding the impact of CPU architecture in the chosen language routines and definition types.
1998
Mars Climate Orbiter (1998)
The failure of the Mars Climate Orbiter in 1999 remains a prominent example of miscommunication and oversight in software development.
Discrepancies between metric and imperial units were not accounted for during the spacecraft’s navigation, leading to a trajectory deviation and subsequent loss of the mission.
This failure emphasized the necessity of standardized communication protocols, standardized units, and thorough cross-checks in software interfaces, particularly in high-stakes environments such as space exploration.
1999
NASA’s Mars Climate Orbiter (1999)
A navigation error occurred due to software inconsistencies between metric and imperial unit systems, causing the spacecraft to burn up in the Martian atmosphere.
Y2K Bug (1999-2000)
As the world transitioned into a new millennium, in South Korea, the Y2K bug ( The Millennium Meltdown (That Never Happened)), a potential software glitch caused by the two-digit date format, was widely feared to cause widespread chaos as the clock struck midnight on December 31st, 1999.
Many computer systems used a two-digit date code to represent years, causing concerns that when the year changed from 1999 to 2000, these systems would interpret “00” as 1900 instead of 2000.
There were fears of potential failures in banking, telecommunications, and transportation systems, leading to extensive preparation efforts; businesses and governments spent billions preparing for the worst, but the dreaded meltdown never materialized. While some minor disruptions occurred, Y2K serves as a reminder of the potential consequences of outdated software systems.
It also serves as a reminder of the critical importance of forward-thinking design and the potential risks of overlooking seemingly minor details in software development.
2000
Microsoft’s Windows ME (Millennium Edition) (2000)
It was plagued by numerous bugs, crashes, and stability issues, receiving widespread criticism from users and tech experts.
I discussed the reasons in one of my previous articles, elite teams can also fail.
2003
Norwegian Tax Administration’s ELSA Project (2003-2008):
An attempt to modernize the tax system led to cost overruns, delays, and, ultimately, the project’s cancellation.
2004
Mars Spirit Rover File System Error (2004)
The Spirit rover experienced a file system error that caused it to reboot repeatedly, hampering its operations.
2007
Malaysian man Phone bill for $218 trillion (2007)
A Malaysian man received a $218 trillion phone bill and was ordered to pay up within ten days or face prosecution.
He had disconnected his late father’s phone line in January after he died and settled the $23 bill. Still, Telekom Malaysia later sent him a bill for recent telephone calls and an order to settle within ten days or face legal proceedings.
The error, reportedly, was the failure of some software to account for a “leap” second.
2008
Heathrow Airport Terminal 5 Opening (2008)
A baggage system meltdown marred the opening of Heathrow Airport’s Terminal 5 in 2008 due to software issues; thousands of bags went missing, causing chaos and frustration for passengers.
This debacle exposed the complexities of integrating large-scale software systems and the need for thorough testing before critical launches.
2009
Herschel Space Observatory Software Issue (2009)
A software error in the cooling system led to the depletion of coolant, rendering the telescope unusable for its primary mission.
T-Mobile Sidekick Data Loss (2009)
A server failure at Microsoft/Danger resulted in the loss of personal data, including contacts, calendars, and photos, for numerous T-Mobile Sidekick users.
Toyota Acceleration Issues (2009-2010)
While not solely a software issue, reports of unintended acceleration in certain Toyota vehicles were initially attributed to faulty software systems, though the ultimate cause was a combination of factors.
2011
Mt. Gox Bitcoin Exchange Hack (2011)
Mt. Gox, once the world’s largest Bitcoin exchange, suffered a series of security breaches between 2011 and 2014, resulting in the loss of millions of $460 million.
Cybersecurity has become more critical than ever before in recent years.
The Fukushima Daiichi Nuclear Disaster (2011)
The nuclear disaster, triggered by a massive earthquake and tsunami in Japan, was aggravated by failures in safety software systems, leading to reactor meltdowns and radiation leaks.
Sony PlayStation Network Outage (2011)
A cyberattack resulted in the PlayStation Network being offline for 23 days, compromising the personal information of millions of users and disrupting online gaming services.
2012
Knight Capital Group Trading Loss (2012)
A software glitch at Knight Capital Group led to erroneous trades, resulting in a loss of approximately $440 million in just 45 minutes.
The issue stemmed from a flawed software update that caused the system to execute trades based on outdated testing code.
This incident underscored the importance of robust change management processes, proper code review, and fail-safe mechanisms in financial software systems.
SpaceX CRS-1 Mission Failure (2012)
During the first Commercial Resupply Services (CRS) mission to the International Space Station (ISS), a software issue with the Dragon spacecraft’s thrusters led to a loss of primary mission objectives.
Olympic Games Ticketing System (2012)
The ticketing system for the London 2012 Olympic Games experienced technical glitches and crashes, causing frustration for users attempting to purchase tickets.
2013
Healthcare .gov Launch Issues (2013)
The rollout of the Affordable Care Act Healthcare.gov website in 2013, intended as a portal for Americans to enroll in health insurance plans, faced numerous technical problems upon its launch.
Issues such as slow loading times, system crashes, and usability challenges plagued the platform, hindering user access and functionality.
This debacle emphasized the necessity of scalability, rigorous testing and stringent quality assurance, continuous monitoring, robust contingency under realistic conditions, and user-centric design in large-scale government software projects.
2015
Volkswagen’s Dieselgate (2015)
While not solely a software issue, Volkswagen faced a major scandal in 2015 when it was discovered that the company had intentionally manipulated software in its diesel engines to cheat emissions tests.
The software could detect when the vehicle was being tested and alter the performance to meet emission standards. This led to significant legal consequences and financial penalties.
The New York Stock Exchange (NYSE) Trading Halt (2015)
A technical glitch caused a trading halt on the NYSE, stopping all trading on the exchange for nearly four hours.
2016
Samsung Galaxy Note 7 Battery Issues (2016)
Software and hardware issues with the battery led to multiple incidents of the phone catching fire or exploding, resulting in a global recall of the device.
Google’s April Fools’ Gmail Mic Drop (2016)
A prank feature in Gmail caused users to accidentally send GIFs of a Minion character, dropping a mic in their emails, leading to complaints and causing problems in professional communication.
Delta Airlines System Outage (2016)
A system failure at Delta Airlines caused a global shutdown, leading to flight cancellations and delays affecting thousands of passengers.
2017
British Airways IT Failure (2017)
An IT system failure led to the cancellation of flights, affecting thousands of passengers, and causing significant disruptions to the airline’s operations.
Equifax Data Breach (2017)
Equifax, one of the major credit reporting agencies, experienced a massive data breach that exposed the personal information of approximately 147 million consumers due to vulnerabilities in their software systems.
NHS WannaCry Ransomware Attack (2017)
A widespread ransomware attack affected the UK’s National Health Service (NHS), disrupting healthcare services and causing the cancellation of appointments and operations due to unpatched systems.
Amazon Web Services (AWS) Outage (2017):
AWS experienced a widespread outage that affected several popular websites and services relying on their infrastructure, including Netflix, Slack, and Reddit.
2018
Facebook’s Cambridge Analytica Data Scandal (2018)
While not solely a software failure, the scandal involved a third-party app exploiting Facebook’s platform, accessing and misusing millions of users’ data without proper consent.
Intel’s Spectre and Meltdown Vulnerabilities (2018)
Significant security vulnerabilities were discovered in Intel processors, known as Spectre and Meltdown, allowing attackers to potentially access sensitive information stored in the computer’s memory.
The Singapore Health System Cyberattack (2018)
Due to vulnerabilities in the system’s software, Singapore’s health system experienced a massive cyberattack resulting in the theft of personal information, including medical records of 1.5 million patients.
Google+ Data Exposure (2018)
A software glitch in the Google+ social network exposed the private data of hundreds of thousands of users, leading to the acceleration of Google+ shutdown plans.
Boeing 737 MAX Software Issues (2018-2019)
In 2018 and 2019, two Boeing 737 MAX airplanes crashed, killing 346 people.
A Maneuvering Characteristics Augmentation System (MCAS) failure contributed to two fatal crashes, leading to a global grounding of the aircraft.
This failure led to the grounding of the 737 MAX fleet worldwide and raised serious concerns about the safety of automated flight systems.
2019
The Boeing Starliner Orbital Flight Test (2019)
During an uncrewed test flight to the International Space Station, Boeing’s Starliner spacecraft encountered software issues that prevented it from reaching the intended orbit, leading to a premature end of the mission.
The U.S. National Weather Service (NWS) Data Loss (2019)
A server outage led to a loss of weather data dissemination for a significant period, affecting the availability of critical weather information.
The Pentagon’s Joint Strike Fighter (F-35) Software Issues (2019-Today).
The F-35 fighter jet program faced recurring software issues, delays, and cost overruns due to complexities integrating various systems and functionalities.
The U.S. National Weather Service (NWS) Data Loss (2019): A server outage led to a loss of weather data dissemination for a significant period, affecting the availability of critical weather information.
2020
NHS Test and Trace App Issues (2020)
The United Kingdom’s NHS COVID-19 contact tracing app faced technical glitches and compatibility issues, causing problems in accurately tracing and notifying individuals exposed to the virus.
Citigroup’s $900 Million Mistake (2020)
Due to a software error, Citigroup mistakenly sent $900 million in payments to Revlon’s lenders, intending to send only $7.8 million in interest payments.
Microsoft’s Azure Cloud (2020)
Microsoft’s Azure cloud services faced a widespread outage that affected multiple Microsoft services for several hours.
An error caused the outage during a routine code update, which resulted in servers becoming overwhelmed and causing disruptions to services like Office 365, Teams, and Azure.
2021
Facebook’s Outage (2021)
In October 2021, Facebook and its other platforms, like Instagram and WhatsApp, experienced a global outage for several hours. The outage was caused by a configuration change with unintended consequences, making the platforms inaccessible to users worldwide.
Summary of lessons
Dates: Deal with date values and date zone conversion carefully.
Rounding error: Some rounder errors will cancel out, but generally, one should safely assume that the limits of this rounding error will grow.
Importance of Thorough Testing: Comprehensive and rigorous testing of software systems is crucial to identify and rectify potential issues before deployment. Testing should cover various scenarios and edge cases to minimize the chances of critical failures.
Robustness of Redundancy Systems: Implementing redundant systems or fail-safes can mitigate the impact of software failures. Backup systems or failover mechanisms can help maintain functionality or minimize disruptions during unexpected events.
Attention to Security: Prioritizing cybersecurity measures is paramount. Regular security audits, timely software updates, and adherence to best practices can prevent vulnerabilities that might lead to breaches, data loss, or unauthorized access.
Clear Communication and Transparency: Transparent communication during software failures is essential. Providing timely updates, acknowledging issues, and offering solutions or workarounds can help manage user expectations and maintain trust.
Adherence to Standards and Protocols: Following industry standards and protocols can minimize interoperability issues and ensure compatibility among different systems, reducing the risk of failures due to integration problems.
User Experience and Feedback Incorporation: Considering user feedback and focusing on user experience during software development can help identify potential issues before they become widespread problems.
Proper Risk Assessment and Contingency Planning: Conducting thorough risk assessments and developing contingency plans can aid in preparing for potential failures, enabling a swift response to minimize disruptions and mitigate damages.
Continuous Monitoring and Maintenance: Continuous monitoring of software systems is crucial to detect anomalies or vulnerabilities promptly. Regular maintenance, updates, and patches are essential to address evolving threats and issues.
Learning from Failures: Analyzing and learning from past failures is vital to prevent similar incidents in the future. Conducting post-mortems, identifying root causes, and implementing corrective measures based on these findings is essential.
Regulatory Compliance and Accountability: Adhering to regulatory requirements and maintaining accountability for software integrity and security are fundamental in preventing catastrophic failures and ensuring user safety and data protection.