The Future of Generative AI for Software Testing (IEEE Computer magazine)
I was asked in May (2023) to take part in an online panel session on the use of AI in Testing. I was one of five prominent bloggers/speakers on the subject. Given a range of questions we were to provide answers from our varying backgrounds. We were asked to provide separate responses of less than 100 words to each of seven questions. So far so good.
Here's the overall process. There are some interesting aspects to this.
All panellists were asked to provide answers to the questions independently by end of June
The editor would put together the article for review by the panellists, by end July
Panellists would submit their 'final edits' and submit the article for processing and typesetting by professional staff editors at the Computer Society.
Final paper to appear in the January 2024 issue of IEEE Computer.
IEE have a formal process that takes time and this is understandable. But the speed of development of the AI landscape – free and proprietary products, applications, reviews and so on mean the half-year timescale will make much of what the panellists say out of date, won't it? We'll all have gained another several months experience and the tools will have moved on.
There doesn't seem to be much opportunity to actually have a conversation. I note the format of past panel articles implies there is interaction, but I don't think that will be so easily achieved in this instance.
I’m writing it here to record the original independent pitches and edits to the content for publication, but also as a stake in the ground on what I believe to be the future of AI in testing mid-2023. Perhaps there'll be some interesting retrospectives to document, later in this process.
Anyway, here is the content I provided on Wednesday, 21 June 2023:
What are the most exciting opportunities Generative AI creates for software testing? Can generative AI accelerate testing activity and improve test quality?
The short answer is “yes, but...”
I have used ChatGPT to scan HTML code to identify form fields, test data and boundary values, create covering test cases and Python code to automate tests of simple transactions. But there are limitations in accuracy and comprehensiveness in such test design. Random/statistically based outputs mean responses are inconsistent, and the tool can forget what it has previously reported earlier in the same conversation.
The tool can generate 'ideas' for tests, but needs careful prompting and supervision to check that it doesn't stray from the mission. It is almost human in its frailties.
How do you see Generative AI integrating into software testing processes 5 years in the future? 10 years?
A tester uses their knowledge and experience, communication and analytical skills to model usage patterns, failure modes (risks), required and conventional behaviour and scenarios to demonstrate software 'works' to enable testing stakeholders to make better-informed decisions.
AI tools for testers will require integrated training data from code, changes, the old system, real-world data, usage patterns, test and defect histories. AI could become a trusted partner of testers who explore knowledge sources and direct AI to perform much of the legwork of testing. But these tools need 'Integrated Systems Intelligence' and a focus on the thought processes of testers.
How will privacy and confidentiality concerns change when Generative AI services are integrated into software testing?
The training data that AI requires to deliver meaningful, reliable services to testers would need to include much proprietary data (code, usage patterns, architectural models, defect histories etc.) collected across many organisations and systems. It's unlikely this will happen of course. It may be possible that some products appear that are trainable and usable within the realm of single organisations. But it seems unlikely a global 'AI Test Model' could be created.
Organisations sensitive to exposing their IP and commercial activity to the outside world, will probably insist tools and models are for internal use only, within their own cloud infrastructure.
What are the current barriers to Generative AI adoption for software testing? What is required to address these challenges?
For too long, tool vendors have focused on the logistics of testing – test case management, test execution, defect reporting and management and so on. With AI, vendors see low-hanging opportunities to, for example, make it easier to generate test automation code or perhaps test data. Help with such logistics is useful of course, but this doesn't help with the intellectual challenge of building test models from varying source of knowledge, defining coverage measures, balancing test utility, coverage and cost. We need to understand how testers think to identify requirements for true AI-based test assistants.
How will the skills required of software testers and quality assurance engineers change as Generative AI tools integrate into the software engineering process? Will software testers and QA engineers become nonexistent?
With the right tools, the skills profile of testers will change. They will become more valuable to software teams but that will mean fewer testers. The best testers will develop a collaborative relationship with their AI partner.
Testers will shift-left to build relationships with stakeholders to refine system requirements and stakeholder needs for information from testing. New tools will capture models ands data across the technical stack, the test team, test outcomes – for all time. The tools will develop both exploratory and advisory capabilities. They will make recommendations and with permission, run tests autonomously, when they see opportunities.
What are the ethical considerations that need to be addressed when deploying generative AI for software testing? How can organizations ensure fairness, transparency, and accountability in the testing process?
Setting aside the obvious challenges of using, for example, production or personal data for testing – AI may have a role in protecting sensitive data. The bigger effect will be how we measure and improve effectiveness of testers, our developers and our processes. The product of testing is information and only testing captures evidence of achievement. If 'Integrated Systems Intelligence' becomes available, AI can evaluate the performance of the test process against stakeholder needs for high quality information. The value of testing is how insightful and actionable the product of testing – information – is to stakeholders.
Aside from software testing, in what other ways do you anticipate Generative AI will change how we engineer software?
Improvements in the logical analysis of text will enable more effective critical evaluation of textual requirements whether written in natural language or domain-specific languages like Gherkin. They will use examples to illustrate feature gaps, ambiguities, conflicts and missing behaviours. These tools know more about business and application domains as those models will emerge over time. Security, reliability, availability and failover testing will be supported or even performed by AI-based tools using model-based approaches.
AI may also offer trustworthy guidance to stakeholders on the documentation, prioritisation and even repair of defects, and potentially the release-readiness of whole systems.