The Testing Uncertainty Principle

On Thursday's SIGIST meeting, it was great to have such a positive reaction to my workshop and closing talk.

The Fallibility axiom (p41) tells us our sources of knowledge are undependable. The tester is a human being and prone to error. The system is being tested because we are uncertain of its behaviour or reliability. As a consequence, the plan for any test worth running cannot be relied upon to be accurate before we follow it.
Predictions of test status (e.g. coverage achieved or test pass- rate) at any future date or time are notional. The planning quandary is conveniently expressed in the testing uncertainty principle:

• One can predict test status, but not when it will be achieved;
• One can predict when a test will end, but not its status.

Consequently, if a plan defines completion of testing using test exit criteria to be met at a specified date (expressed in terms of tests run and the status of those tests) it is wise to regard them as planning assumptions, rather than hard targets.

• If exit criteria are met on time or earlier, our planning assumptions are sound: We are where we want to be.
• If exit criteria are not met or not met on time, our plan was optimistic: Our plan needs adjustment, or we must relax the criteria.

Whichever outcome arises, we still need to think very carefully about what they actually mean in our project.