The Testing Uncertainty Principle

On Thursday’s SIGIST meeting, it was great to have such a positive reaction to my workshop and closing talk.

The Fallibility axiom (p41) tells us our sources of knowledge are undependable. The tester is a human being and prone to error. The system is being tested because we are uncertain of its behaviour or reliability. As a consequence, the plan for any test worth running cannot be relied upon to be accurate before we follow it.
Predictions of test status (e.g. coverage achieved or test pass- rate) at any future date or time are notional. The planning quandary is conveniently expressed in the testing uncertainty principle:

  • One can predict test status, but not when it will be achieved;
  • One can predict when a test will end, but not its status.

Consequently, if a plan defines completion of testing using test exit criteria to be met at a specified date (expressed in terms of tests run and the status of those tests) it is wise to regard them as planning assumptions, rather than hard targets.

  • If exit criteria are met on time or earlier, our planning assumptions are sound: We are where we want to be.
  • If exit criteria are not met or not met on time, our plan was optimistic: Our plan needs adjustment, or we must relax the criteria.

Whichever outcome arises, we still need to think very carefully about what they actually mean in our project.

Advancing Testing Using Axioms

Last week I presented a talk called “Advancing Testing Using Axioms” at the First IIR Testing Forum in Helsinki, Finland.

Test Axioms have been formulated as a context-neutral set of rules for testing systems. Because they represent the critical thinking processes required to test any system, there are clear opportunities to advance the practice of testing using them.

The talk introduces “The First Equation of Testing” and discusses opportunities to use the Axioms to support test strategy development, test assessment and improvement and suggests that a tester skills framework could be an interesting by-product of the Axioms. Finally, “The Quantum Theory of Testing” is introduced.

Go to the web page for this talk.

Test Axioms as Thinking Tools

Is it possible to define a set of axioms that provide a framework for software testing that all the variations of test approach currently being advocated align with or obey? In this respect, an axiom would be an uncontested principle; something self-evidently and so obviously true and not requiring proof. What would such test axioms look like?

This paper summarises some preliminary work on defining a set of Test Axioms. Some applications of the axioms that would appear useful are suggested for future development. It is also suggested the work of practitioners and researchers is on very shaky ground unless we refine and agree these Axioms. This is a work in progress.

Registered users can download the paper from the link below. If you aren’t registered, you can register here.