Software Reliabilty Overview

When you buy a car, often the dealership will ask you if you're interested in purchasing a warranty. However, a warranty simply means it's affordable to fix, not that the vehicle is reliable.

When we discuss the reliabilty of software, we can do so from different roles. As a manager, I might want to know is the software good enough to roll out to production and put in the hands of users. As a consumer, I may be more interested in how long the software is going to work correctly until the next failure.

Typically it works like this:
  Step 1. Keep a log of past failures.
  Step 2. Plot the failures.
  Step 3. Determine a curve that best fits the observations.
  Step 4. Measure how accurate your curve model is.
  Step 5. Predict the future using the model.

Estimation [of the model, not future behavior] is finding the right curve for historical data. Prediction is just that, intelligently guessing what will happen in the future.

Complex software should be less reliable than less complex software. High level languages should be more reliable than low level languages. Experienced staff should produce more reliable code than less experienced staff. Proven technologies and processes should be more reliable than untested technologies. All these factors, and many more, go into twiddling the shape of the curve.

Software and hardware don't behave the same way. The longer software is used and maintained, the higher the degree of confidence it obtains as defects are found and weeded out. Hardware, on the other hand, ages. Heat, oxidation, and wear take their tolls on hardware, and even a perfectly functioning device (like your old VCR) will eventually fail, despite how well it's treated.

A software reliabilty curve flattens out over time. A hardware reliability curve is more U-shaped. Since a system consists of running software on hardware, both need to be taken into account concurrently, and for that we have the system reliability model.

Measuring Software Reliabilty is more of an art than a hard science. The tricky part is finding a realistic curve. This isn't a matter of making a spline that goes through all points, because what's really being mapped is a general trend over time. Luckily for us the field of statistics provides many tools for generating curves and measuring their fitness to a collection of raw data points.

Luckily for people who dislike statistics, are terrible at math, and/or have a phobia of greek letters, there's a wonderful tool called SMERFS. SMERFS will take collected raw data, analyze it, try various models, tell us which is the best, and predict future behavior based on past trends.

Note that while a model allows for incremental improvements (such as weeding out defects), it typically does not handle major rewrites of code or process. Curves therefore are much better at evolutionary changes than revolutionary changes.

However, the real world does have revolutionary steps such as massive code change, new versions, and new platforms. In those cases, a model needs to take data from the latest version and weight it more than historical data of prior versions. Sure the historical trends can provide an indication of what may happen, but they may not provide a prediction of what may happen.

The real world can be even harsher. Sometimes there is no data, or it's inaccurate, or it's just plain missing. For this we have to fall back to the original factors that went into the design and make scientific wild-ass guesses from there.

There are lots of models to choose from, and what this really means is that there are lots of equations that express different shaped trend curves. The different models take different things into accounts with various weightings.

Commonly, it's useful to collect things like time of failure (or the interval between failures) and the intensity of a failure. Failure intensity is how bad things went wrong. This can be through some arbitrary ranking, or it can be an aggregate total number of failures over a given time period.

A failure is when the software and/or hardware doesn't do what it should when it should. This is due to a fault.

There's a difference between a failure and a fault. A failure is the a crashing into a tree. The fault is the steering wheel wasn't screwed on at the factory, the driver fell asleep while driving, or the roads were covered in ice. Faults can be product defects, user related, or environment related. Failure is what happens when the fault shows itself during operation.

Note that we're trying to track new, unexpected failures. If we notice the steering wheel isn't screwed on tightly, that doesn't count. If we noticed it before and opt not to fix it right now, we don't count the failure again. Because software shops tend to fix things in batches and then release them, the actual raw data appears as a stair-step. The model's curve, however, demonstrates what an instanteous fix would look like the moment a fault is discovered.

By measuring the whats and whens of a failure, it possible to calculate things like the mean time to failure, the number of failures at a given point at time, or at what point in time there will be some specific number of failures left to occur.

It's doubtful any model or methodology is going to be 100% accurate, however it is possible to get a statistically strong confidence level by which make a sound decision.