We are at a point in the evolution of artificial intelligence where software developers and engineers are adopting it in various capacities. Developers use it to write code, explain stack traces, summarize technical documents, and come up with test scenarios. Quality engineering teams are now using it to review requirements, triage failed tests, scan logs, and identify patterns across test runs.
This is an enormous opportunity, as we are already sitting on mountains of test data that take quality engineers hundreds of hours to analyze and draw insights from
AI can be leveraged to do this faster, but it needs to be guided by humans. A pragmatic approach could be that the automation does the executing, AI does the interpreting, and quality engineers make the final decisions about code, infrastructure, issues, and software releases.
Why Conventional Test Automation Falls Short
Conventional test automation has been great at executing predefined sets of steps. We can write a script that launches the application, invokes an API, enters some data, compares expected and actual results, and marks the test as pass or fail.
However, execution is only part of the equation
We still need to understand and draw inferences from the results, which can be extremely time-consuming if we have to comb through hundreds of thousands of log lines to find out what really went wrong in a failed test case. That is where an AI-powered middle layer could help.
We can keep the existing test automation framework as is and let the AI layer consume the logs, error messages, stack traces, and any other relevant information to group related failures together to give us a meaningful explanation.
Where Can AI Add Value to Quality Teams?
Interpretation of test results is not the only area where AI can help. One of the pain points for quality engineers is reviewing requirements. Requirements are often ambiguous and incomplete, and there are often corner cases that may not be apparent at first blush.
For example, a requirement that says ‘Send a notification when the transaction is complete’ may seem straightforward, but it does not cover what should happen when the notification service is unavailable, if the notification was delivered but the event was processed twice, or if part of the transaction succeeded while another failed.
We can use AI to infer such corner cases and help the requirement reviewers, such as Quality Engineers, ask the right questions before development starts. Similarly, we can leverage an AI assistant to identify the most relevant tests to execute when a specific set of code changes lands, based on the files changed, service dependencies, test coverage, and relevant defects.
We can also use AI to help us find relevant tests when we have to maintain our test suite due to changing UIs or APIs and help us identify potential flaky tests that may need closer scrutiny.
The Most Common Sense and Most Valuable Use Case for AI
For many quality teams, understanding what went wrong in a failure is more valuable than writing new tests. When a test fails, we often have to trawl through application logs, browser logs, network logs, screenshots, API responses, and infrastructure metrics, deployment information, and the commits made in the last few days before we can pinpoint the exact root cause.
An AI assistant can consume all this information and help us get to the bottom of the issue much faster.
Instead of saying ‘Checkout test failed’, it could say something like ‘Checkout failed during the payment processing step. Twelve other tests have also failed with a gateway timeout after the latest deployment. The failures occurred between the same time window, and the payment service was slow during the same period. You may want to review the payment service connection setup before rerunning the entire regression suite.’
That is the power of inference that can supercharge our quality engineering efforts.
AI Needs to Understand the Context
A common-sense example will illustrate why context is crucial for AI: If I say I am going to the mountains, you could assume I am going skiing. However, if I say I am going to the mountains to ski, you know I mean snow skiing. On the other hand, if I said I am going to the mountains to summer, you might think I meant a mountain bike.
Context makes all the difference.
Similarly, a general-purpose AI model will not fully understand the domain-specific context of our software unless we teach it. It may not know which microservice owns a specific piece of functionality or the exact meaning of an error code. It may not know which environment our latest build was deployed to or if this type of issue occurred a month ago and what the resolution was. This lack of context will create noise in its responses.
We can help an AI model get the context it needs by exposing it to trusted data sources such as our test reports, architecture diagrams, API definitions, runbooks, deployment pipelines, defect reports, and telemetry data. This is known as Retrieval-Augmented Generation (RAG). In quality engineering, it can be extremely powerful because we are not looking for generic answers; we want answers that are relevant to the product or service being tested.
Trust Is Earned Through Evidence
Trust is an essential aspect of any interaction with AI, especially in engineering contexts.
Consider these two responses: “The database is overloaded.” vs. “There are fourteen checkout failures with database timeout errors. Database memory jumped to 98% during the same test window, and the failures began after the latest deployment.” The second response gives us concrete evidence. We can verify for ourselves whether the database was indeed overloaded by looking at the memory metrics during that time.
AI systems used in quality engineering should be able to show us what they used as evidence to arrive at their conclusion. If they cannot find relevant evidence, they should say so.
Another advantage of this approach is that it reduces the risk of hallucination. Hallucination occurs when an AI provides answers or information that is not accurate or cannot be supported by the data. By forcing the system to rely on evidence, we can reduce the risk of it making things up.
We can further reduce the risk of hallucination by adopting what is called an evidence-first design. In this approach, the AI systems are designed to ask questions and request more information when they cannot find relevant evidence. This enables transparency and accountability, and it goes a long way in reducing the risk of providing misleading information.
Prioritize Based on Risk
Finally, while every feature of an AI-powered quality engineering system should be available for inspection, certain high-risk operations should require manual approval. Some tasks are relatively low risk and can be automated, such as log summarization, test result analysis, test idea generation, and defect description. Others, such as recommending a smaller regression suite, test script changes, or a different UI locator, may fall into a middle ground and require quality engineer approval. On the other hand, critical operations, such as changing essential assertions, making production configuration changes, or closing a defect automatically, should probably require explicit human approval before taking any action.
This risk-based stratification can allow us to reap the benefits of automation while ensuring that sufficiently risky operations continue to require human oversight.
A Sample Human-Guided AI Process
A simple example of how a human-guided AI process could work in quality engineering could have the following five steps.
First, the system identifies and understands the engineering question. The question could be something like ‘Investigate failed regression tests.’
Second, we gather trusted data and evidence, such as logs, test reports, telemetry data, code changes, and deployment information.
Third, we analyze the evidence and information and draw inferences.
Fourth, we can validate the inferences to ensure accuracy before presenting the conclusions and recommendations to the quality engineer or specialist.
Finally, we take action using whatever tools are available to address the root cause, update the defect report, or execute a subset of relevant tests.
This approach keeps the quality engineer in the loop and makes the most of the combined strengths of AI and automation.
How Quality Engineering Roles and Responsibilities Will Evolve
Human-guided AI will change the responsibilities and roles of quality engineers. While they will still be responsible for quality gatekeeping, they will have more opportunities to focus on higher-value activities, such as risk analysis, operational observability, and test strategy design. They will have to learn to work more closely with AI and operationalize its insights within the constraints and limitations of existing tools.
Software quality is no longer about individual test cases passing or failing. It is about interactions between services, the impact of a release on the customer, and the ability to operate and understand complex systems. Quality Engineers are uniquely positioned to evolve into this new paradigm as they already understand the nuances of risk, operational complexity, customer impact, and release readiness.
Focus on Measuring Outcomes, Not AI Contributions
While executives and managers may be tempted to measure the success of AI initiatives in terms of how much it contributes to test automation, such metrics are not always easy to come by. It may be much more useful to measure the engineering outcomes that AI enables.
Did it reduce the time spent on failure triage?
Are related failures being grouped together correctly?
How often are its recommendations rejected by engineers?
Did it make flaky test investigations faster?
Did it reduce the effort required for script maintenance?
Are its inferences backed by sufficient evidence?
These outcome-based metrics will be much more useful in determining whether AI is genuinely enhancing our quality engineering efforts.
The Bottom Line
The future of AI in software quality is not about letting an AI model take charge of everything. It is about finding the right balance between reliable automation, AI-powered information synthesis, and human expertise, insight, and judgment. Leverage AI to identify issues and speed up repetitive tasks, but make sure there is always a human in the loop for complex decisions. Used this way, AI can indeed supercharge our quality engineering initiatives while keeping the software quality professionals at the center of the process.
