Specification by Example is not Enough

A couple of weeks ago I gave a talk that included a couple of slides that focused on the idea of Specification by Example and how it cannot be relied upon to fully define the functionality of a software solution. I thought I'd summarise it here while the thought was fresh in my mind and also because Martin Fowler recently re-posted a blog originally published some time ago.

Martin provides a broader perspective and significantly, he says 'Specification By Example only works in the context of a working relationship where both sides are collaborating and not fighting'. Quite. He quotes a Cedric Beust post that critiques TDD (and Agile projects in general) that promote the use of tests as specifications.

Clearly, SBE can work nicely in an Agile environment where the scenarios are there to capture some key examples of the feature in use. The more general business rules to be implemented in code are (presumably) discussed and captured elsewhere - specifically in the code and exampled in tests. The examples and automated tests based on the conversations are retained to provide evidence that the software 'works' and stays working after changes elsewhere. One obvious, valuable outcome of SBE, Behaviour-Driven or Test-Driven approaches is a set of automated tests that are a quite effective anti-regression measure for use in projects that practice a continuous delivery approach. But what about non-Agile? Can SBE work in all contexts?

The questions is, "can examples alone be trusted to fully describe some system behaviour?" The answer is occasionally yes, but usually - no. Here's an example of why not.

The table below shows some scenarios associated with a feature. Call it SBE, BDD or just a shorthand for some TDD tests. Whatever.

given a, b, c are real numbers
when a=<a>
  and b=<b>
  and c=<c>
then r1=<r1> and r2=<r2>

| a |  b | c | r1 | r2    |
| 1 | -2 | 1 |  1 |  1    |
| 1 |  3 | 2 |  1 |  2    |
| 1 |  3 | 2 |  1 |  2    |
|12 |-28 |15 | 1.5|  0.833|

It doesn't give much away does it? "Do you know what it is yet?" (as Rolf Harris might ask).

Now, I could keep giving you new examples that are correct from the point of view of the requirement (that I'm not yet sharing with you). Maybe you'd spot the pattern of the inputs and outputs and guess that a b c are the coefficients of a quadratic and r1 r2 are the roots. Aha. The programmer could easily implement the code as follows:

r1 = (-b + sqrt(b*b - 4*a*c))/(2*a)
r2 = (-b - sqrt(b*b - 4*a*c))/(2*a)

Sorted. But is it...?

Suppose I then gave you an example that could NOT be processed by the quadratic formulae? The example below would probably cause an exception in the code:

| a |  b | c |  r1 | r2  |
| 4 |  3 | 2 | ... | ... |

You can't take square roots of negative numbers. So you could argue that there's a validation rule (not yet specified) that rejects inputs that cause this exception and change the code accordingly. But in fact, one CAN derive the square root of negative numbers. They are just called 'complex numbers' that's all. (Mathematicians can be a bit slippery). Have we got it right yet? We'd have to look at the expected outcomes in the examples provided, and generate them in code and hope for the best. Whatever. That's enough maths for one day.

The principle must be that examples on their own do not provide enough information to formulate a general solution. It is always possible to code a solution that will satisfy the examples provided. But that is not a solution - it is mimicry. A coded table of pre-defined answers can mimic the general solution. But the very next example, when used to test our solution will fail if the example is not in our coded table. Our model of the solution is incomplete - it's wrong. In fact, to be certain we have the perfect solution we would effectively need an infinite number of examples that when tested generate the required outcomes. Specification by Example ALONE cannot provide a complete specification.

Where does this leave us? Specifications are models of systems. All models are wrong (or at least incomplete) but some are useful. But having a specification is a necessary (but probably not sufficient) condition for building a solution.

Perhaps Specification by Example is mis-named. It should called be Specification AND Example.

The question remains, "How much software is out there that just mimics a solution?"