Reliability targets for software are written as numbers, and some of them are blunt. Bell Laboratories set the No. 1 ESS ADF store-and-forward message switch a limit of 2 hours of downtime in 40 years, according to the Bell System Technical Journal (December 1970).
The idea underneath is simple enough. Software reliability is a quality attribute that expresses the probability of failure-free operation over a specified period in a specified environment. Engineers measure it as failure intensity or mean time to failure and test it against an objective fixed before release, following practice such as IEEE Std 1633-2016.
What Is Software Reliability?
Strip any condition out of that definition and the number stops meaning anything. The required function is the behavior the specification promises. The period can be execution time or calendar time, and it has to be fixed before measurement starts. The environment covers the hardware and data the software sees, plus how people actually use it.
The environment part trips people up most, at least in my experience. The same build can be dependable under one usage pattern and shaky under another, because different inputs reach different faults.
Standards bodies don’t word the idea identically. NIST’s glossary carries the IEEE dictionary version, which calls reliability the ability to function under stated conditions for a specified period, and sets it beside a probability version from NIST IR 8323r1.
In practice the ability wording ends up in requirements documents, while a test team can only measure against the probability one.
Reliability counts as a non-functional requirement, so it says how well the software does its jobs and leaves the question of which jobs to the functional side.
The IEEE Technology Navigator treats it as a design and engineering concern. A software defect gets built in at design time, not produced on a factory line.
For assessing and predicting software reliability across the life cycle, the recommended practice is IEEE Std 1633-2016. It came out on 18 January 2017 as a revision of the 2008 edition. Prediction models cover early development, and growth models take over for test and operation.
Fault, error and failure
Start with the error, which is the mistake a developer makes, like forgetting to handle an empty shopping cart. That mistake leaves a fault in the code (an unguarded division in the checkout routine, say). Nothing visible happens until the failure, the observable departure from specified behavior. Here that would be a crash when a customer checks out with zero items.
Pai’s survey of reliability models describes a failure as a fault activated by a particular input. So a fault that no real input ever reaches adds nothing to the failure rate.
How ISO/IEC 25010 Breaks Down Software Reliability
In ISO/IEC 25010, reliability is a product quality characteristic split into faultlessness, availability, fault tolerance and recoverability.
The characteristic itself restates the classic definition, the degree to which a system performs specified functions under specified conditions for a specified period.
The wider ISO 25010 quality model files reliability next to security, maintainability and performance efficiency, among others.
The second edition, published in November 2023, replaced the 2011 text. Its foreword swaps maturity for faultlessness, and arc42’s comparison of the two editions notes that the model grew from 8 characteristics to 9 with the addition of safety.
- Faultlessness is performing specified functions without fault under normal operation. It replaces maturity from the 2011 edition.
- Availability means the system is operational and accessible when required for use.
- Fault tolerance, operating as intended despite hardware or software faults.
- For recoverability the question is whether the affected data comes back and the desired state is re-established after an interruption or failure.
Reliability and availability
ISO/IEC 25010:2023 puts availability inside reliability. arc42’s reading of the standard splits it into frequency of failure, governed by faultlessness, and length of downtime, governed by fault tolerance and recoverability.
Operations writing usually treats the two as separate measures. Antimetal’s glossary, for one, frames reliability as whether the service responded correctly.
Pick one framing and state it in the requirements, because an uptime target and a correctness target need different evidence.
Redundancy and failover, the core of architectures built for high availability, shorten outages without touching the fault content of the code.
Reliability and correctness
Correct and reliable are different claims.
ISO/IEC 25010 files functional correctness, the degree to which functions return correct results at the needed precision, under functional suitability. Reliability asks how long that behavior holds under stated conditions.
A function that is wrong only for an input real users never send adds almost nothing to the failure rate. A program can be incorrect and still reliable in practice.
Software Reliability vs Hardware Reliability
Hardware mostly fails because parts wear out, which is why its failure rate climbs with age. Software has no such decay. Its faults are in the design from the day of release, and the failure rate only moves when the code or the way people use it changes.
| Aspect | Hardware | Software |
|---|---|---|
| Failure cause | Wear, manufacturing defects | Design faults present from release |
| Failure curve | Bathtub curve with early failures, a flat middle and a wear-out phase | Flat after debugging, spikes after changes |
| Effect of repair | Restores the original state | Produces a new version with its own faults |
| Effect of redundancy | Independent copies fail independently | Identical copies repeat the same fault |
Software does not wear out. Every source checked agrees on that point.
Where the sources disagree is the shape of the software curve. Dalhousie University course notes draw an idealized curve that flattens after debugging and spikes with each change. A 2023 arXiv paper by Sharma and Sadam (NIT Warangal) goes with a revised bathtub that runs from test and debug through upgrade to stable or obsolescence, citing Hartz, Walker and Mahar’s 1996 review.
Both drop the wear-out phase, and both tie the spikes to changes instead of age.
In practice that means every release resets part of the curve, since the fix is itself new code with new faults.
The redundancy row matters more than it looks. Two replicas of the same build share every fault in it, so duplication protects against hardware loss and does nothing for a logic error.
How Software Reliability Is Measured

Measuring reliability comes down to counting failures against time or against demand. The usual metrics are MTTF, MTBF, MTTR, failure intensity (rate of occurrence of failures), probability of failure on demand and availability.
| Metric | What it measures | Formula or unit | Fits |
|---|---|---|---|
| MTTF | Average operating time until failure | Total operating time / failures | Non-repairable items, single-client long-use systems such as disks |
| MTBF | Average time between failures of a repairable system | MTTF + MTTR | Systems that recover after failure |
| MTTR | Average time to restore service | Total repair time / repairs | Any repairable service |
| Failure intensity (ROCOF) | Failures per unit time | Failures / observation time, equal to 1 / MTTF in stable systems | Continuous services, transaction processing |
| POFOD | Chance that a request fails | Failed requests / total requests | Rarely used systems, such as airliner oxygen masks |
| Availability | Share of time the system is operational | MTTF / (MTTF + MTTR) | Continuous online services |
Choose the metric from how the system is used, not from whichever one is easiest to compute. The University of Edinburgh’s 2024 lecture notes on reliability metrics follow the same logic. A system that is called rarely and costs a lot when it fails suits POFOD. One that runs continuously, where each failure matters, suits failure intensity. And if it simply has to stay up, use availability, with MTTR to show how fast service returns.
Availability depends on both sides. The IEEE Technology Navigator describes it as a function of reliability, how often failures occur, and maintainability, how quickly service returns after one.
The reliability function R(t) is the probability of running without failure from time 0 to t. Under a constant failure rate it takes the form e^(-lambda * t), a result Sharma and Sadam (2023) adapt from Lyu’s 1996 Handbook of Software Reliability Engineering.
Software rarely holds a constant rate, since each fix and release changes it. R(t) has to use the current failure intensity, and an old value gives an old answer.
An INFLIBNET course chapter shortcuts the calculation with R = MTBF / (1 + MTBF). That expression carries no time period, so it falls outside the definition above.
Defect density, defects per thousand lines of code, belongs in a different bucket. It describes the product, not behavior over time, so two programs with the same density can differ in reliability when usage decides which faults fire.
Which Models Estimate Software Reliability?
Reliability growth models estimate how failure intensity falls as testing removes faults. The foundational ones are Jelinski-Moranda (1972), Littlewood-Verrall (1973), Goel-Okumoto (1979) and Musa-Okumoto (1984).
All of them are black-box models. They read failure data only and ignore the structure of the code, as Ganesh Pai’s 2002 survey at the University of Virginia points out.
| Model | Year | Failure assumption | Data input |
|---|---|---|---|
| Jelinski-Moranda | 1972 | Fixed number of faults, equal hazard per fault, instant perfect fixes | Times between failures |
| Littlewood-Verrall | 1973 | Bayesian, fix size is random, a fix can make things worse | Times between failures |
| Goel-Okumoto | 1979 | Non-homogeneous Poisson process, finite expected total failures | Failure counts per test interval |
| Musa-Okumoto | 1984 | Logarithmic Poisson, intensity falls exponentially with failures seen, no cap on total | Execution time per failure |
Jelinski-Moranda and Goel-Okumoto produce the same mean value function. Where they part ways is the fault count. Jelinski-Moranda fixes it at a single value, while Goel-Okumoto only expects one.
A workable rule is to reach for a finite-failure model (Jelinski-Moranda or Goel-Okumoto) when the software is approaching a stable fault pool. Musa-Okumoto fits better where new failures keep arriving as releases ship. Littlewood-Verrall is the one for sparse data or for fixes that add faults, since it accepts a prior and allows negative growth.
Every model starts with failure data, either times between failures or counts per test interval. Most teams already hold it in defect tracking records and test logs, though the timestamps have to be trustworthy.
No model is trusted everywhere. Brocklehurst and Littlewood, quoted in Pai’s survey from Lyu’s 1996 handbook, state that no universally acceptable model exists, and that models with similar assumptions can disagree on the same data or make the same wrong prediction together.
A 2006 analysis from the Korea Atomic Energy Research Institute, run on Goel and Okumoto’s published failure data, shows the practical cost. After 34 failures, Jelinski-Moranda estimated 34.003 total faults.
Estimates from both models dropped sharply after long gaps between failures. The authors suggest that at least 20 failures are needed before either model gives a usable result.
Littlewood and Sofer’s 1983 NASA report adds a sharper case. On Musa’s System 3 data, the Jelinski-Moranda estimate of total faults equaled the number of failures already seen for failures 25 through 38. The model declared the program perfect while it kept failing.
Taken together, those sources point to a habit worth adopting. Fit at least two models and report the current failure intensity with the spread between them. Remaining-fault counts stay out of the headline. Littlewood argued in 1980 that failure rate, not bug count, is the metric that matters.
What Affects Software Reliability

Reliability moves with fault introduction and removal rates, complexity, change volume, requirement quality, test realism and the operational profile.
Faults enter with each change and leave with each verified fix, so failure intensity follows whichever rate is winning. Complexity takes the blame in Glenford Myers’s 1976 book, and James Cusick’s 2019 history of the field notes later work linking complexity to reliability. Then there’s change volume. The more code a release touches, the more new faults it brings.
Microsoft Research data backs up the change volume point. Nagappan and Ball (ICSE 2005) found that relative code churn, meaning change normalized by component size and time, separated fault-prone from not fault-prone Windows Server 2003 binaries with 89.0 percent accuracy.
Absolute churn was a poor predictor.
Requirements decide what counts as a failure at all. Myers’s book tells of a missile-warning system that reported the rising moon as an incoming rocket, a case where only the requirements can say whether that output is a failure.
A software requirement specification that states the operating envelope and the failure classes gives test and field data something to be measured against.
Test realism is a separate matter from test volume. Code coverage reports which statements ran, not which faults a user will hit, and Pai’s survey notes that full coverage can still miss faults while failures keep occurring without any gain in coverage.
Purdue researchers Chen, Garg, Mathur and Rego did find coverage and reliability related in their experiments. That makes coverage a useful signal and a poor target.
The operational profile is the set of operations users perform, with the probability of each. Failure intensity depends on it, according to Musa’s 1994 ISSRE paper on adjusting field failure intensity, so a shift in usage moves reliability without a line of code changing.
Profile errors carry into estimates. In the Purdue case study, the error in Musa-Okumoto and Goel-Okumoto reliability estimates grew nonlinearly with the error in the profile.
How to Set and Verify a Software Reliability Target
In practice the target is a failure intensity objective for each failure class. Testing against the operational profile and fitting failure data shows when the current failure intensity meets it.
- Define what counts as a failure, and sort failures by severity.
- Set a failure intensity objective for each class: the maximum failure rate acceptable at release.
- Pick the reliability strategies, from fault avoidance to fault tolerance, that can reach the objective.
- Build the operational profile, the usage pattern the tests will imitate.
- Prepare and run tests that follow the profile, with each operation exercised in proportion to its expected use.
- Record failure data and fit a growth model to track current failure intensity against the objective.
- Decide: release when the objective is met, keep testing when it is not, and rework the design if growth stalls.
The steps follow John Musa’s software-reliability-engineered testing process, as laid out in his ISSRE 2001 tutorial outline.
The number itself depends on severity. FAA Advisory Circular 25.1309 treats catastrophic failure conditions in civil aircraft as extremely improbable, on the order of 1 in 109 per flight hour or less, and John Rushby of SRI International summarizes looser values for the lesser classes: 10-7, 10-5 and 10-3.
Growth testing and certification testing are different modes. Growth testing fixes faults as failures appear and tracks failure intensity. Certification testing runs without debugging and accepts or rejects the build against the objective.
Write the objective into the release’s acceptance criteria so the release decision depends on it.
How to Improve Software Reliability
Faults can be dealt with at several points. Avoidance in design comes first, since the best fault is the one that never gets written. Verification removes more, and fault tolerance has to absorb whatever survives.
Michael Lyu’s 2007 roadmap adds fault and failure forecasting to that list, which is the model work covered above.
Fault avoidance
Avoidance means preventing faults by construction, through specification review, design review, static analysis and language-level protection against range and type errors.
Dijkstra argued at the 1968 NATO conference that product quality cannot be established afterwards, a point recorded in Cusick’s history of the field.
Reviews have a documented limit, though. They check what reviewers think to ask, and the Ariane 5 case below is the recorded example.
Fault removal
Removal runs on unit, integration and system testing, plus operational profile testing, which spends test effort in proportion to use. Root cause analysis on every field failure belongs here too.
Each fix is a change, so regression testing re-runs earlier tests to catch faults the fix introduced.
Dijkstra’s remark at the 1969 NATO conference still stands: testing shows the presence of bugs, not their absence. It exposes faults. It cannot show there are none.
Fault tolerance
Design diversity
N-version programming runs independently built versions in parallel and votes on the output. It earns its reliability gain only if the versions fail independently.
Knight and Leveson tested that assumption in 1986, with 27 versions built independently from one specification at two universities and run against one million tests. The versions were individually extremely reliable, but the number of tests where more than one version failed ran substantially above what independence predicts.
The sources split on what that proves. Knight and Leveson’s own reply to critics quotes the opposing claim, that later empirical work showed independence can hold under a best-practice development paradigm, and disputes it, noting that the UCLA/H study never tested independence.
Fault injection and chaos engineering
Netflix runs Chaos Monkey, which randomly terminates virtual machine instances hosting production services so that engineers design for instance loss. The practice rests on a steady-state hypothesis. Define a measurable output as normal, then try to disprove that it holds under injected failure.
It has a blind spot, though. Killing instances exercises recovery from lost capacity, but a logic fault triggered by one specific input stays untouched, so injection complements profile-based testing and does not replace it.
Containment in production
Google’s SRE book defines the error budget as the gap between the SLO and measured uptime. Releases continue while budget remains and slow or stop when it runs out.
The SRE Workbook defines canarying as a partial, time-limited deployment of a change and its evaluation. A canary deployment limits how many users meet a new fault, and the SRE book names canary duration and size as a standing trade-off between risk and effort.
Order of spend matters. Specification review and operational profile testing come before design diversity, whose independence assumption stays contested and whose price is a second development effort.
What Software Failures Show About Reliability
Ariane 5 flight 501 and the Therac-25 overdoses are two documented cases of software reliability breaking down. Design faults that testing never exercised did the damage in both, along with safeguards that assumed the software was right.
Ariane 5 flight 501
The Inquiry Board report of 19 July 1996 places the loss of the inertial reference systems at 36.7 seconds after H0 on 4 June 1996, about 30 seconds after lift-off.
The fault was an unprotected conversion of the horizontal bias variable from a 64-bit floating point value to a 16-bit signed integer. It sat inside alignment code that serves no purpose after lift-off. That code came from Ariane 4 and kept running for about 40 seconds of flight, and because Ariane 5 built up horizontal velocity five times more rapidly, the value went out of range.
The standby inertial unit failed first. The active unit, running identical software, followed about 0.05 seconds later. The on-board computer then read diagnostic data as flight data and commanded full nozzle deflection, and the launcher broke up at about 39 seconds.
The Board called it a systematic software design error, not a random failure. Programme culture addressed only random hardware failures, so the specification told the processor to stop on any exception, a rule that suits a hardware fault and was fatal here.
No test had run the inertial reference system against the Ariane 5 trajectory, and the trajectory data was missing from its specification. In the terms of the definition, the specified environment changed and nobody had written the change down.
Therac-25
Therac-25 shows what happens when reliability is assumed instead of measured.
Between June 1985 and January 1987, six known accidents involved massive overdoses. A physicist at the first site estimated one or two doses of 15,000 to 20,000 rads, against about 200 rads for a typical single therapeutic dose (Leveson and Turner, IEEE Computer, 1993).
The investigation traced the overdoses to software. At Tyler, Texas, an operator finished editing within about 8 seconds while the magnet-setting routine ran for about 8 seconds, so the edit never reached the setup logic (a race condition on shared variables). At Yakima in 1987, a one-byte counter overflowed to zero on every 256th pass through a setup routine and skipped the collimator position check.
The machine also dropped hardware interlocks that the Therac-20 kept, so software checks stood alone. Leveson and Turner infer that unit and software testing was minimal, with most effort going to integrated system testing.
Their verdict is that chasing individual bugs misses the point. The basic mistakes were poor software engineering practice and a machine that relied on software for safe operation.
When Software Reliability Models Do Not Apply
Growth models stop working in some fairly predictable situations.
The first is data that shows no growth. Littlewood and Sofer’s 1983 NASA report shows Jelinski-Moranda returns nonsensical output when failure data shows no reliability growth, a condition they tie to the slope of inter-failure times against failure number. The same report found the model optimistic on almost every data set it analysed.
Changing code is another. Every model assumes some homogeneity of growth, and a change in testing procedure or the integration of more code breaks it (Littlewood and Sofer).
Too few failures cause trouble as well, and Pai’s survey warns that maximum likelihood estimates from sparse failure data can be unstable or wrong.
Then there are targets beyond what testing can show. Butler and Finelli of NASA Langley conclude that statistical quantification of life-critical software reliability is infeasible, because classical estimation needs exorbitant testing and growth models cannot remove that need. They add that the independence assumption behind fault-tolerant claims cannot be justified by experiment at ultrareliability levels.
A model also fits only failures. A fault that no test or field use has triggered leaves nothing to fit, which is why the Ariane 5 fault gave no trend before the first flight.
Quantified safety claims built on weak assumptions carry the same risk. AECL’s fault tree for Therac-25 excluded residual software errors by assumption, then gave the event “computer selects wrong energy” a probability of 10-11 with no justification, according to Leveson and Turner.
When a model cannot carry the claim, the evidence shifts to assurance of correctness, meaning inspections and formal verification along with testing. John Rushby of SRI International argues that these support a probability of possible perfection, not a failure rate.
Software Reliability FAQ
Does software reliability mean the same as software quality?
No. Quality is the umbrella and reliability is one characteristic under it.
A program can fail rarely and still be hard to change or slow to run, which are maintainability and performance problems, not reliability problems.
Can software be 100% reliable?
No. Reliability is a probability, and Google’s SRE book calls 100% probably never the right target, because it cannot be reached and exceeds what users notice.
Teams set an objective below it and spend the difference on change.
Which standards set reliability requirements for safety-critical software?
DO-178C governs airborne software and ISO 26262 covers road vehicles, while IEC 61508 reaches programmable safety systems across industries. Each ranks software by an integrity or assurance level tied to failure severity.
John Rushby of SRI International notes that correctness-based assurance does not appear to support a quantitative failure rate claim.
How much downtime does 99.9% availability allow?
About 8.76 hours a year, or 43.2 minutes in a 30-day month.
Google’s SRE book gives 52.56 minutes a year for 99.99%, and each added nine cuts the allowance tenfold.
Is software reliability engineering the same as site reliability engineering?
No, even though both abbreviate to SRE. Software reliability engineering is the statistical side, predicting and tracking reliability from failure data, which is the work IEEE Std 1633-2016 describes.
Site reliability engineering comes out of Google’s operations practice and manages service reliability through SLOs and error budgets.
When Extra Reliability Stops Paying for Itself
Software reliability stops paying for itself at the point where the next nine of availability costs more than the revenue it protects.
Google’s SRE book puts a price on one step. Raising availability from 99.9% to 99.99% is worth $900 on $1M of revenue.
The next nine returns $90, a tenfold drop, while the same book warns that an increment can cost 100 times more than the previous one. Combining the two figures, the cost-to-value ratio worsens by up to 1,000 times with each added nine.
Users set a second ceiling. The book notes that someone on a smartphone with 99% reliability cannot tell 99.99% service from 99.999%.
Set the target at the lower of the two limits, and accept the failures that fall inside it.
Settling that number leads to the software quality assurance process that reviews and tests the build to meet it.
- Who Answers Your Users After Launch? - November 18, 2026
- How to Create a Folder in GitHub From the Browser - November 17, 2026
- 6 Best Cookie Consent Apps for Shopify in 2027: Features and Setup Compared - November 17, 2026



