Focus on Test Strategy

*** See our ‘Test Strategy in a Day‘ workshop ***

Test Strategy is a Thought Process, Not a Document

Test strategy is one of those nebulous terms for an activity that everyone has to undertake for their development projects. The problem with test strategy is that there isn’t a single agreed definition of what it is. Some people treat Test Strategy as a document: “A high-level description of the test levels to be performed and the testing within those levels for an organization or programme”. So – a strategy could range from one to hundreds of pages of text. All we need is a document template!

Not quite.

Over the last twenty years, almost all of the Test Strategy documents Paul has reviewed have been “copy and edits” of documents for previous projects. All that changed was the names and the dates. And all of these document were read by … no one.

We suggest that rather than being a document, test strategy is a thought process. The outcome of the thinking might be a short or a long document, but really, the strategy must address the needs of the participants inside the project as well as the customers of the product to be built. It needs to be appropriate to a short agile project or to a 1000 man-year development. It has to have the buy-in of stakeholders but most importantly, it must have value and be communicated. That’s quite an ambitious goal. but we think it is achievable.

A Strategy Answers Questions

Before your organisation, your team or you as an individual test a system, you need some questions answered. These questions are pretty fundamental and yet, we have met many teams who could not answer them – even when they were in the thick of the testing. Here is a sample of the type of questions to be answered?

  • Who are your testing stakeholders? What do they want from testing? Why do they want it? In what format? How often?
  • What is the scope of testing? What is out of scope? How will you manage scope?
  • How much testing is enough? What models should be used to guide the testing? How will you assess coverage?
  • and so on…

One way of looking at strategy is to regard it as a way of answering these questions. But not every question can be answered up-front. Some questions cannot be answered now, but perhaps later when information come to light. So the strategy must provide guidance for answering these questions and making decisions. It does this in three ways:

  1. Some decisions can be made now, directly and with confidence.
  2. Some decisions cannot be made now, but the strategy will provide a process, method or mechanism for reaching a decision.
  3. Some decisions cannot be made now, and no method could be formulated, but the strategy can provide some guiding principles to be followed in reaching a decision.

In this way, a strategy written early in a project, before all the information required is available, can flex and support a project that is heading into unknown territory. 

Is Agile Test Strategy an oxymoron? We don’t think so. If you look at the three levels of decision making above, structured or waterfall projects aim to answer every questin ahead of time. So, in these projects, most of the quesitons can be directly answered so the emphasis is on ‘type 1’ decisions. In Agile or less certain projects, the lack of knowledge, or the need to remain flexible demands that the emphasis will shift more towards ‘type 2’ or ‘type 3’ decisions.

The Trick is to Ask the Right Questions

The challenge in test strategy is not providing perfect answers to these questions. The secret to success is in asking the right questions in the first place. Over twenty years, we have found oruselves asking the same questions again and again. The Test Axioms are an attempt to arrange these questions into a more usable structure. The Axioms represent context-neutral aspects of testing to think about. The 16 axioms are organised into three Axiom groups: Stakeholder, Design and Delivery. Each Axiom has 5-10 questions associated with the, the Axioms therefore provide a checklist of questions to ask, a a structure for your strategy document, should you coose to write one.

Our Test Strategy workshop

We have created a workshop titled ‘Test Strategy in a Day‘ that you might find useful. In this workshop, we present a practical definition of a Test Strategy, provide a simple template for creating one and describe a systematic approach to thinking the right way. It’s an interactive session where we invite you to bring your test strategy problems with you – we’ll try and address them during the day.

  • Test strategy is a thought process, not a document
  • The goal is to acquire and communicate knowledge of achievement
  • How a valuable test strategy can be formulated and communicated


To view the workshop brochure click here.

Testing is in a mess

Got your attention? I’m triggered into posting what follows because Rob Lambert has published an ‘ebook’ The Problems of Testing. Now it reminded me of a blog I wrote 18 months or so ago – but never published. I never pressed the ‘publish’ button because I thought it was rather too negative. But Rob’s frustration is shared and here’s my take on it. I don’t agree with all he says, but I think there’s a certain amount of complacency, self-delusion and over capacity in our business. Here what I wrote … I was going to call this post “Do you engage your brain before you test?” but as I wrote it, I got more and more upset by the state of our discipline. (I can hardly bring myself to call it a discipline any more). It seems to me that the major ‘trends’ occurring in our discipline are being promoted and adopted at a terrifying rate. I’m not worried about the rate of change, or being left behind. I am worried that the people who are being sold these approaches and implementing them are taking on so-called ‘solutions’ without understanding what the underlying problem is, what they are trying to achieve, or how to evaluate ‘success’. People have no time and no frame of reference in which to think or consider the pros and cons of available courses of action. Often, they don’t even know who they are testing for. For example:

  • tools are used to automate tests that have unknown or limited value
  • ‘crowds’ might help us test but what is the objective, where is the control or accountability?
  • test design techniques are promoted as ‘good practice’ without people understanding the concept of test models, their value and limitations
  • coverage is discussed endlessly without any understanding of it’s meaning, subjectivity and interpretation
  • the language and terminology we use is riddled with duplication, inconsistency and ambiguity
  • outsourcing/offshoring are promoted as being effective and economic – without any discussion of its value
  • the exploratory/ad-hoc v planned/documented testing debate is a yah-boo-sucks schoolyard shouting match – how is an interested observer to understand the issues?
  • most debate is about software testing, but we test systems, don’t we? Why don’t we adopt a Systems Approach?
  • certification schemes are embedding many of these flawed ideas and are strangling competitive, viable and often better alternatives.

I could go on. If I suggested that 50% of the testers currently in our business shouldn’t be – how would you argue against that? Why are YOU in this business? For example – can you string two words together? One of the reasons I wrote the TESTER’S POCKETBOOK was to drill down to the fundamentals that underpin all testing (if I could find them). It’s been a struggle, but I’ve come up with some and they are useful. I call these fundamentals TEST AXIOMS and you can see them here. They make sense to me as a starting point for discussion on testing and test practices, but they also underpin much of the thinking about test strategy and improvement. Ask them of yourself or your organisation or the next person you interview. They work for me. Am I too pessimistic about the state of our industry? Is anything going well out there? If/as/when the economies of our various countries squeezes testers out of businesses – will you still be in a job? How will you justify your role? Developers, analysts and users combined can do most of what you do. Can you do their job? Who is indispensable now? Think about it.