Specification by Example is not Enough
A couple of weeks ago I gave a talk that included a couple of slides that focused on the idea of Specification by Example and how it cannot be relied upon to fully define the functionality of a software solution. I thought I'd summarise it here while the thought was fresh in my mind and also because Martin Fowler recently re-posted a blog originally published some time ago.
Martin provides a broader perspective and significantly, he says 'Specification By Example only works in the context of a working relationship where both sides are collaborating and not fighting'. Quite. He quotes a Cedric Beust post that critiques TDD (and Agile projects in general) that promote the use of tests as specifications.
Clearly, SBE can work nicely in an Agile environment where the scenarios are there to capture some key examples of the feature in use. The more general business rules to be implemented in code are (presumably) discussed and captured elsewhere - specifically in the code and exampled in tests. The examples and automated tests based on the conversations are retained to provide evidence that the software 'works' and stays working after changes elsewhere. One obvious, valuable outcome of SBE, Behaviour-Driven or Test-Driven approaches is a set of automated tests that are a quite effective anti-regression measure for use in projects that practice a continuous delivery approach. But what about non-Agile? Can SBE work in all contexts?
The questions is, "can examples alone be trusted to fully describe some system behaviour?" The answer is occasionally yes, but usually - no. Here's an example of why not.
The table below shows some scenarios associated with a feature. Call it SBE, BDD or just a shorthand for some TDD tests. Whatever.
given a, b, c are real numbers when a=<a> and b=<b> and c=<c> then r1=<r1> and r2=<r2> | a | b | c | r1 | r2 | | 1 | -2 | 1 | 1 | 1 | | 1 | 3 | 2 | 1 | 2 | | 1 | 3 | 2 | 1 | 2 | |12 |-28 |15 | 1.5| 0.833|
It doesn't give much away does it? "Do you know what it is yet?" (as Rolf Harris might ask).
Now, I could keep giving you new examples that are correct from the point of view of the requirement (that I'm not yet sharing with you). Maybe you'd spot the pattern of the inputs and outputs and guess that a b c are the coefficients of a quadratic and r1 r2 are the roots. Aha. The programmer could easily implement the code as follows:
r1 = (-b + sqrt(b*b - 4*a*c))/(2*a) r2 = (-b - sqrt(b*b - 4*a*c))/(2*a)
Sorted. But is it...?
Suppose I then gave you an example that could NOT be processed by the quadratic formulae? The example below would probably cause an exception in the code:
| a | b | c | r1 | r2 | | 4 | 3 | 2 | ... | ... |
You can't take square roots of negative numbers. So you could argue that there's a validation rule (not yet specified) that rejects inputs that cause this exception and change the code accordingly. But in fact, one CAN derive the square root of negative numbers. They are just called 'complex numbers' that's all. (Mathematicians can be a bit slippery). Have we got it right yet? We'd have to look at the expected outcomes in the examples provided, and generate them in code and hope for the best. Whatever. That's enough maths for one day.
The principle must be that examples on their own do not provide enough information to formulate a general solution. It is always possible to code a solution that will satisfy the examples provided. But that is not a solution - it is mimicry. A coded table of pre-defined answers can mimic the general solution. But the very next example, when used to test our solution will fail if the example is not in our coded table. Our model of the solution is incomplete - it's wrong. In fact, to be certain we have the perfect solution we would effectively need an infinite number of examples that when tested generate the required outcomes. Specification by Example ALONE cannot provide a complete specification.
Where does this leave us? Specifications are models of systems. All models are wrong (or at least incomplete) but some are useful. But having a specification is a necessary (but probably not sufficient) condition for building a solution.
Perhaps Specification by Example is mis-named. It should called be Specification AND Example.
The question remains, "How much software is out there that just mimics a solution?"
The UK TMF
|Gerrard Consulting founded the UK Test Management Forum in January 2004. Over 2500 delegates have attended the quarterly sessions.
The Forums provide lively debate between practitioners, consultants and vendors and cover both Test Management and Technical Testing topics.
Recent Blog Posts
- Will the Test Leaders Stand Up?
- Continuous Delivery of Long Term requirements
- Top 10 Books for Testers
- Story Based Acceptance Test Automation Using Free Tools
- Story Based Testing and Automation
- New Tutorial: Test Strategy in a Day
- Live Specifications: From Requirements to Automated Tests and Back
- Business Story Method and Platform Awareness
- Business Story Manager Tutorial: Analysis Module Part I
- Redistribution of Testing: A Brief Summary
Business Story Pocketbook
The Business Story Method is supported by and largely documented in a book called "The Business Story Pocketbook" which is available for FREE DOWNLOAD or can be purchased from Gerrard Consulting (or other good booksellers).
The Business Story Pocketbook
Testing Axioms and Pocketbook
The Testing Axioms identify the critical thinking processes for testing. As a set of context-neutral rules they also provide a universal testing framework.