Testing Needs Evidence, Not Ritual
Part 14 of the Software Development in the Age of Al series
Software testing has accumulated a large vocabulary over several decades. Unit testing, integration testing, system testing, regression testing, TDD, BDD, shift-left, shift-right, continuous testing, service virtualisation, model-based testing, exploratory testing and, more recently, AI-augmented testing all describe techniques, practices or organisational approaches that emerged in response to real engineering problems. The terminology can be useful, but it also creates a risk that the adoption of a named approach becomes confused with the quality of the outcome it is supposed to produce.
Beneath that vocabulary, the purpose of testing is relatively straightforward. Testing is the disciplined production of evidence that software behaves correctly against its requirements under relevant operating conditions, and that a change has not introduced unintended behaviour elsewhere in the system. Establishing this requires considerably more than demonstrating that a new function produces the expected result on a handful of normal inputs. It requires evidence across representative data, boundaries, exceptional conditions, failure paths, dependencies and the existing behaviour that may be affected by the change.
This distinction is important because a piece of software can satisfy the immediate requirement and still create a defect elsewhere. A new calculation may be correct while altering downstream reconciliation. An API may return the right result while handling retries incorrectly. A user interface may implement the requested workflow while producing unexpected state in another system. Testing therefore has two related responsibilities: to establish that the intended behaviour has been implemented correctly and to provide confidence that existing behaviour has not been damaged in the process.
The relevant question is consequently not whether a test suite is green or whether an organisation has reached a particular automation percentage. It is whether the evidence produced by the testing process is sufficient to support confidence in the software that will actually be operated.
Testing as a discipline
Formal software-testing disciplines developed because answering that question consistently is difficult. ISO/IEC/IEEE 29119 provides an internationally recognised family of standards covering software-testing concepts, processes, documentation and test-design techniques [1][2][3]. The standards are deliberately independent of a particular software-development lifecycle and therefore do not require an organisation to use Waterfall, Agile, DevOps or any other named delivery model.
That is useful because testing itself should not depend on the methodology currently in fashion. The standards describe a disciplined process in which testing is governed, planned, analysed, designed, implemented, executed, monitored and completed against defined criteria. They also formalise techniques for deriving tests rather than leaving test selection entirely to intuition or precedent.
The standards should not be treated as an answer in themselves. They provide a useful external reference against which testing practices can be examined. Many large organisations have developed substantial internal testing procedures over years or decades. These may include dedicated test teams, automated regression suites, integration environments, defect-management processes, performance testing, user acceptance activities and formal release gates. Such organisations may perform a great deal of testing without explicitly governing their processes against ISO/IEC/IEEE 29119.
That distinction is more significant than it first appears. A process can be highly structured while still containing weaknesses that have evolved unnoticed because the organisation measures itself against its own procedures rather than against an independent model. Automation may be excellent while environment fidelity is poor. Testers may work carefully against stories and expected outcomes while the available data does not represent production behaviour. Failure scenarios may be simulated while the wider architecture on which those failures depend is materially different from the live environment.
The problem is therefore not necessarily a lack of testing or even a lack of formality. It is that internally evolved procedures can create confidence in the process without necessarily establishing whether the process is producing sufficiently strong evidence.
AI provides a useful reason to revisit this. Before asking how existing testing activities can be automated, an organisation can first ask whether those activities are the right ones.
From methodology to evidence
Software engineering has a tendency to turn successful practices into organisational doctrine. A technique is found to work well in a particular context, training and tooling develop around it, and eventually the language of the technique can become a substitute for examining whether it continues to serve the intended purpose.
Testing has experienced this repeatedly.
Test-driven development is one example. TDD remains useful where a developer can describe the expected behaviour of a sufficiently bounded element before implementing it. It can improve local reasoning, encourage explicit contracts and provide rapid feedback during development. The difficulty begins when the technique is treated as a universal design principle rather than one available engineering practice.
Complex systems do not always present themselves as a series of small, fully understood behaviours waiting to be encoded. Requirements may be incomplete. The technical decomposition may not yet exist. Dependencies may shape behaviour in ways that cannot be known until implementation progresses. The most useful boundary for testing may emerge through design rather than precede it.
A strict response that every ambiguous behaviour should simply be decomposed further also has limits. Beyond a certain point, additional decomposition can increase indirection, multiply interfaces and spread a coherent business rule across so many small abstractions that the implementation becomes harder to understand. Testability is an important characteristic of good software, but it is not the only characteristic. Readability, cohesion, operational simplicity, performance and the ability of another engineer to reason about the system remain equally important.
The same caution applies to newer testing terminology. Shift-left, shift-right, continuous testing, testing pyramids and similar models all contain useful ideas, but none removes the need to understand the system being built. A mature testing function is not one that can demonstrate adherence to a particular vocabulary. It is one that can explain what evidence is required, how that evidence will be produced and why it is sufficient for the risks associated with the change.
AI makes that distinction increasingly important because it can apply a methodology with extraordinary consistency while still applying it poorly. If instructed to maximise coverage, it can generate enormous numbers of tests. If instructed to pursue isolated testability, it can continue decomposing code. If instructed to make a suite green, it can modify tests and implementation until they agree. None of those outcomes necessarily demonstrates that the software is correct.
The objective has to remain external to the method.
Testing begins with understanding
At the developer level, testing is closely connected to software construction. Unit tests rarely constitute independent evidence that a business requirement has been delivered. Their primary value is as engineering instruments that help a developer examine whether an implementation behaves according to the technical understanding used to create it, expose gaps in logic, exercise boundaries and protect behaviour as the code evolves.
This becomes particularly relevant when AI is introduced into development.
A well-written story can provide the initial statement of intent, but it will not normally define every internal class, method, parameter, state transition or technical contract that will emerge during implementation. Those are products of engineering design. Attempting to generate a complete set of low-level tests directly from the story therefore confuses business intent with implementation structure.
A more useful starting point is a structured interaction between the developer and AI in which the story is examined before implementation begins. The purpose of that discussion is not to force every test into existence before code is written, but to establish what is known, what remains ambiguous and what evidence will eventually be needed.
The discussion might explore the intended behaviour, dependencies, data types, valid ranges, failure conditions, state transitions and assumptions that have not yet been resolved. It can also help determine which behaviours are suitable for isolated testing and which depend on a larger component or interaction.
As the design becomes clearer, AI can then provide substantial practical value. Once a component contract and its data constraints are understood, AI can generate candidate cases across boundaries, invalid values, empty states, malformed inputs, unusual combinations and error paths far more quickly than a developer could construct them manually. This does not remove the need for engineering judgement; it shifts that judgement away from repetitive test construction and towards deciding whether the underlying contract and expected behaviour are correct.
Recent research into AI-generated testing demonstrates both the opportunity and the limitation. Large language models can generate significant numbers of candidate tests and can derive cases from natural-language requirements, code and other artefacts, but their effectiveness remains dependent on the quality and completeness of the available context [4]. AI can automate an incomplete assumption just as effectively as it can automate a correct one.
The useful role for AI at this level is therefore not simply to write more tests. It is to participate in the reasoning process through which meaningful tests become possible.
The test ecosystem is part of the architecture
Testing becomes progressively more demanding as software moves beyond an isolated component. Modern systems depend on databases, APIs, message brokers, identity platforms, external services, asynchronous processing, batch workloads and other components whose behaviour may vary according to state, timing, transaction volume and failure conditions.
For that reason, the test environment cannot be treated merely as infrastructure supplied after development has started. The architecture of the production system should always be considered alongside the architecture of the test ecosystem required to validate it.
In practice this is frequently neglected. A component is designed and built, and only later does someone ask for a UAT environment, staging environment or performance environment. What is then produced may simply be another instance of the component surrounded by simplified stubs. That may answer some questions, but it may be inadequate for establishing how the component will behave when connected to the systems, data and operating conditions it will encounter in production.
A large banking platform illustrates the difficulty. It would be impractical to reproduce every branch, ATM, payment participant and external interface physically. A credible test ecosystem does not need to duplicate production in every detail. It needs to reproduce the production behaviours, conditions and dependencies that are relevant to the change.
That may include a core banking platform with materially equivalent behaviour, realistic account and transaction states, representative volumes, controllable API and message responses, delays, retries, duplicate transactions, batch cycles and failure conditions. Simulators may represent branch systems, ATMs or third-party services, but the simulated interfaces must behave closely enough to production that the evidence produced remains meaningful.
The critical requirement is therefore not physical similarity but behavioural fidelity.
This should be determined during architecture and design. Every significant architectural decision creates corresponding test requirements. A new event stream creates requirements for message ordering, replay, duplication and failure handling. A stateful service creates requirements for representative state transitions. An external API creates requirements for both normal and abnormal responses. A high-volume component creates requirements for realistic workloads and meaningful performance observation.
If those requirements are considered only after development is complete, the organisation may discover that the software cannot be tested credibly using the infrastructure that exists. The architecture of the test ecosystem must therefore evolve with the architecture of the software itself.
AI can make this relationship easier to maintain. Given a sufficiently documented production architecture and test architecture, it can help identify the dependencies introduced by a proposed change, determine which are represented adequately in the available environments, distinguish between real and simulated behaviour, and highlight production characteristics that cannot currently be reproduced. This moves AI beyond test generation and into testability analysis.
Data fidelity and the limits of conventional environments
Environment fidelity is only one part of the problem. Test data can be equally important.
Organisations operating in sensitive environments have legitimate obligations to protect personal, financial and other confidential information. Production data may therefore be masked, tokenised, obfuscated or replaced with synthetic data before it enters a conventional test environment. These controls are necessary, but they can also alter the relationships and distributions that drive real system behaviour.
If a transformation changes the characteristics that determine how software responds, the resulting test may satisfy a privacy requirement while weakening the evidence that the test was intended to produce.
This becomes particularly significant when validating replacement processing engines or other high-risk changes. In such cases, one of the strongest approaches may be to run the candidate implementation in parallel with the incumbent system for an extended period. The same production events are processed by both implementations, but only the existing system remains authoritative. The candidate path produces a non-authoritative result that can be compared with the production result.
Over several weeks or months, this exposes the new implementation to the actual distribution of production activity, including unusual combinations of data and state that may never have been anticipated when conventional test cases were designed.
Event-streaming platforms such as Kafka make this practical in some architectures because production events can be consumed by an additional processing path without changing the authoritative transaction. Similar patterns can be implemented through mirrored traffic, temporary production code or other forms of shadow processing.
Organisations often describe this as a parallel run rather than testing. From an engineering perspective, however, its role is clearly part of the testing discipline because its purpose is to produce evidence about whether the candidate implementation behaves correctly.
This example exposes an important limitation in conventional testing terminology. Testing should not be defined by whether an activity occurs inside an environment called TEST, UAT or STAGING. In some situations, the strongest available evidence can only be obtained alongside production, provided that the candidate implementation is prevented from creating production consequences.
Testing therefore has to be defined by its purpose rather than by the label attached to the environment in which it is performed.
What AI changes
The examples above point to a broader change. AI should not be treated merely as another source of test automation. Its more significant effect is that activities which were previously too expensive to perform comprehensively — systematic test derivation, large-scale data variation, comparison of environments, maintenance of traceability and continuous examination of gaps in evidence — can increasingly become part of ordinary delivery.
This creates an opportunity to review the testing discipline itself rather than simply accelerate existing procedures. The starting question for an organisation is therefore not which testing tasks can be handed to AI, but whether its current testing process produces sufficient evidence that software behaves correctly against its requirements, under representative operating and failure conditions, without introducing unintended behaviour elsewhere.
AI can then be applied where it strengthens that evidence or materially reduces the effort required to obtain it. Existing processes that remain useful do not need to be discarded, but neither should they be preserved simply because they have become institutionalised. Recognised standards provide an external reference, established organisational practices provide experience, and AI provides new capability. The opportunity lies in combining those elements around the required outcome rather than creating another prescribed methodology.
There is already substantial research activity in AI-augmented software testing, while ISTQB now provides a specialist certification specifically concerned with applying generative AI across software-testing activities [5][6]. The significance of these developments is not that another testing doctrine has appeared, but that rigorous analysis, test derivation and evidence management can increasingly be performed at a scale that was previously too labour-intensive.
The objective remains unchanged. Software must behave correctly against its requirements, including under realistic abnormal and failure conditions, and changes must not create unforeseen behaviour elsewhere in the system. What has changed is the amount and breadth of evidence that can now be produced, analysed and maintained economically.
Testing is not Quality Assurance
One final distinction is important because the terms are frequently confused.
Testing and Quality Assurance are not interchangeable. Testing produces evidence about software behaviour. Quality Assurance is the broader oversight function that determines whether the evidence produced across the delivery is sufficient to justify confidence in the intended outcome.
That evidence may come from analysis, architecture, development, testing, risk assessment, security, compliance, operational readiness and other activities. Testing is therefore one contributor to assurance rather than another name for it.
The widespread organisational practice of referring to test teams as “QA” obscures that distinction. A team can execute tests extremely well and still not perform the broader assurance role.
AI may eventually make continuous, evidence-based assurance considerably easier because it can maintain visibility across the full delivery process. That is a larger subject in its own right. For the purposes of testing, however, the distinction is enough: testing establishes evidence about behaviour; assurance determines whether the total evidence is sufficient.
A reason to review testing now
The introduction of AI into software delivery creates an unusual opportunity. Organisations do not need to preserve every existing testing ritual merely because it has become institutionalised, nor should they abandon the discipline developed through decades of software engineering.
The useful task is to separate the purpose from the procedure. Recognised standards provide a reference point, existing organisational practice provides experience, modern techniques provide additional options, and AI reduces the cost of analysis, test derivation, execution and evidence management.
The resulting question for any organisation is therefore not whether it follows the latest testing methodology. It is whether its current testing approach can produce credible evidence that software behaves correctly against its requirements, across representative operating conditions and failures, without introducing unexpected behaviour elsewhere.
If the answer cannot be demonstrated convincingly, increasing the amount of automation will not solve the underlying problem. AI gives organisations the opportunity to improve the testing discipline itself before accelerating it.
References
[1] ISO (2022), ISO/IEC/IEEE 29119-1:2022 — Software and systems engineering — Software testing — Part 1: General concepts, International Organization for Standardization.
https://www.iso.org/standard/81291.html
[2] ISO (2021), ISO/IEC/IEEE 29119-2:2021 — Software and systems engineering — Software testing — Part 2: Test processes, International Organization for Standardization.
https://www.iso.org/standard/79428.html
[3] ISO (2021), ISO/IEC/IEEE 29119-4:2021 — Software and systems engineering — Software testing — Part 4: Test techniques, International Organization for Standardization.
https://www.iso.org/standard/79430.html
[4] Folorunsho, O. and Reza, H. (2026), AI-Driven Test Case Generation from Natural Language Requirements: A Survey of Techniques and Research Gaps, arXiv.
https://arxiv.org/abs/2606.06563
[5] A Taxonomy for AI-Augmented Software Testing (2025), arXiv.
https://arxiv.org/abs/2506.14640
[6] ISTQB (2026), Certified Tester — Testing with Generative AI (CT-GenAI), International Software Testing Qualifications Board
https://istqb.org/certifications/gen-ai/