In most game development teams, automated testing begins only after a feature has already been designed and implemented. At Age of Magic, Playkot’s long-running fantasy RPG, the QA team is trying to move that point much earlier in the development process.
The new pipeline starts with the game design document itself. The system can analyze the GDD, extract and validate requirements, compare them against the broader product context, generate testing strategies and test suites, run them through several rounds of agent-based review, prepare the test environment, determine whether execution is necessary or whether resources can be saved, execute the tests when needed, and produce a final report.
For simple features, this process can already run fully autonomously. The decision to release, however, remains with a human.
And automation for its own sake was never the goal. “We approached this from the perspective of business metrics from the very beginning,” says Alexey, QA Engineer at the Age of Magic studio. “We needed to ship features faster. That meant testing and configuring them faster while also improving stability.” In a live-service game, those two objectives are closely connected. Age of Magic regularly ships a large number of content changes through server-side updates. A single incorrect number in an offer can lead to player frustration, refunds, and direct financial losses.
For Alexey, QA automation therefore became more than a way to save a few hours of work. It became a way to protect both the player experience and the game’s economy.
It started with small deterministic checks
The system did not begin as one large AI project. On the QA side, engineers started building small checkers – scripts written with the help of LLMs that validate configuration files used in regular content updates. Structural validation of configs already existed and was handled at the architecture level. The new checkers addressed a different problem: finding errors inside the configs themselves, such as broken relationships, anomalies, and other inconsistencies.
Previously, much of this work relied on manually comparing hundreds of parameters line by line, making the process vulnerable to ordinary human errors and missed details.
Gradually, the team began combining individual checkers into larger “gates” capable of validating entire features and relationships between different parts of the game.
Once that worked, a more ambitious question emerged: Why wait for the feature itself to exist before starting full-scale QA work?
“If the GDD has already been approved by the product team, then we already have enough information to start doing useful work,” Alexey says. That led to a pipeline that starts with documentation and ends with executable tests. Step one: turn the GDD into testable requirements
Requirements analysis was an obvious candidate for automation because it was both important and time-consuming.
QA was already involved relatively early in feature development. Engineers read GDDs, prepared questions, and participated in discussions with other teams.
But the depth of that preparation was not always consistent. Sometimes there was enough time to analyze a document thoroughly; sometimes the meeting happened before QA had a chance to check every dependency.
The new pipeline makes it possible to do that work in advance.
An LLM-driven script analyzes the GDD using rules and approaches drawn from established requirements-testing practices. It extracts testable specifications, looks for contradictions within the document, and identifies potential dependencies on adjacent or related features.
As a result, QA can come into a discussion with structured material already prepared instead of starting the analysis during the meeting itself. “This is one of the cheapest changes we’ve made, but it has delivered a lot of value,” Alexey says. “If we resolve contradictions at this stage, it speeds up not only testing but development of the feature itself.”
The system does not treat the GDD in isolation.
It can compare textual requirements against configuration data, LiveOps timelines, Asana tasks, and changes in GitLab. If a new mini-game is launching alongside several other features, the pipeline can take their context into account as well. That matters because game systems rarely exist in isolation. The UI may be changing in the same release. Another event may be competing for the same resource. A configuration may depend on a mechanic that is not even described directly in the new feature’s GDD. The QA system gathers context from multiple sources – Confluence, Asana, internal timeline spreadsheets, and GitLab diffs – to surface these relationships much earlier.
From requirements to a test matrix
Once the requirements have been extracted, the next stage looks much more like traditional test design. Each specification is mapped against a matrix of relevant testing categories. The system considers, for example:
- logic;
- UI and visual behavior;
- balance and configuration;
- negative scenarios;
- edge cases;
- front-end and back-end behavior.
Not every requirement needs every possible type of test. A configuration value may not require a UI check. Visual behavior may not need a back-end test. Instead of mechanically generating every possible combination, the system selects categories based on the nature of each individual specification. The result is a structured set of test cases with standardized labels, priorities, and other metadata. This standardization has another benefit: the documentation becomes easier for another QA Engineer to read and execute, even if they have never worked with that particular feature before. Before the pipeline was introduced, designing a test suite for a large new feature could require around six hours of focused QA work. With LLM-assisted generation, Alexey says that in some cases this stage can be reduced to roughly 30 minutes. But generation is not the final step.
Why one agent reviews another
At Age of Magic, the team deliberately avoids relying on a single agent to create a test plan and then declare its own work correct. Generated test suites go through cross-agent review.
According to Alexey, there is a systemic problem with the “one model creates and reviews its own work” approach: the agent tends to assume that it has already performed the task well.
There is also a practical limitation related to large contexts. Even a strong model may compress or omit information in ways that make the output less useful for human review.
Instead, the team uses independent agents to challenge and validate the result. A single test suite may go through three to five rounds of review, with feedback coming from both agents and humans. Importantly, review does not happen only once at the very end. It takes place at different stages throughout the process. “Cross-review at every stage gives us faster and more consistent feedback,” Alexey says. “We catch errors and inaccuracies earlier and more cheaply than if we tried to find everything in one final review.” Once approved, the test cases are stored in the test-management system. This gives the team both human-readable documentation and historical data that can be revisited later if the feature changes.
Only then does the pipeline move from describing tests to actually executing them.
Giving the agent the skills of a regular QA Engineer
One of the more unusual parts of the system is automated environment setup.
The team essentially studied the routine sequence of actions a QA Engineer performs before testing a feature and turned those actions into reusable skills.
Given a link to an Asana or GitLab task, the system can identify the required branch and locate the related documentation and balance data. It then checks the available test environments, finds a free one if necessary, deploys the appropriate branch, checks it out locally, and launches Unity. From there, test scripts can begin executing the required cases through the game editor. Alexey emphasizes, however, that this part sounds more “AI-driven” than it really is. Much of the environment orchestration is simply conventional deterministic automation.
“In reality, this is one of the simplest parts of the entire pipeline,” he says. “It’s just a set of small automated actions performed by a script. You don’t need an LLM for everything.”
That distinction is fundamental to the architecture of the entire solution.
Age of Magic does not try to solve every problem with a language model. LLMs are used where interpretation, contextual understanding, and reasoning are required. Scripts are used where the expected result can be defined deterministically. The system is therefore better understood not as an “AI tester,” but as an orchestration layer combining LLM reasoning, engineering rules, and traditional automation.
Some features can already go from GDD to report
The level of autonomy depends on risk. For a simple, familiar content feature – a set of new offers, for example – the team can already run the pipeline fully autonomously.
The input is documentation. The output is a completed testing report.
For larger or fundamentally new features, human checkpoints remain. Alexey highlights three in particular. First, a QA Engineer reviews the extracted requirements. Second, a human reviews the final set of test cases after agent-based cross-validation. Third, QA manually performs checks that automation cannot yet evaluate reliably.
These include things like game feel and certain visual aspects. “A machine can’t tell you whether the game is fun enough or how it feels,” Alexey says. And if the system determines that it does not have the tools required to execute a particular test, it is expected to say so explicitly rather than pretending the check was successful. That case is returned to QA for manual execution.
The hardest problem is defining the source of truth
AI-generated tests introduce another interesting problem: the test itself may be logically correct, while the requirement it is based on has already become outdated.
Age of Magic is developed iteratively.
A Game Designer may change a feature during implementation without immediately updating the GDD. The code changes, while the test cases continue to reflect the previous version of the specification.
The tests then fail – not because the feature is broken, but because the testing oracle, the source of truth, has changed.
The team is developing a separate mechanism to address this problem.
A dedicated debugger compares the current codebase against the GDD and test documentation and attempts to identify where they diverge. In the future, the team plans to run this check on every commit and automatically flag documentation that may have become outdated. This illustrates one of the fundamental limitations of autonomous QA particularly well. A system can reason very effectively about the truth it has been given. Making sure that truth is still current is a separate engineering problem.
From a full working day to a few hours
The team is already seeing measurable results.
Preparing testing for a medium-sized feature – reading the GDD, extracting requirements, designing cases, and resolving questions – previously took approximately six to eight hours.
With the new pipeline, Alexey estimates that producing a final, agreed-upon test suite takes around two to three hours.
The execution process has changed in a similar way. Preparing environments and test data, configuring profiles, running cases, testing on devices, and documenting defects could take another full working day. For a comparable feature, the automated workflow can reduce that stage to less than three hours. Overall, Alexey estimates that the testing contribution to a feature’s delivery timeline can be reduced by roughly half. But this is not simply a matter of running the same tests faster. Automation makes it economically viable to perform checks that the team previously could not afford to run as frequently. After a small code change, the pipeline can rerun the entire relevant regression set instead of waiting for changes to accumulate and performing one large check near the end of development.
Automation can find what a human physically cannot test often enough
One production example emerged while testing a dice-based mini-game.
The system performed around 3,000 simulated rolls and discovered that a bonus roll was occurring too frequently. A human playing through the mini-game several times would almost certainly never generate a large enough sample to detect the statistical bias.
Testing also uncovered configurations that could potentially be exploited, as well as a broader LiveOps dependency: offers for the new mini-game conflicted with offers from another concurrent event because both relied on the same competing resource.
The team adjusted the balance as a result. At this point, automation is changing more than productivity. It makes entire classes of testing possible that would be practically impossible to perform manually at sufficient scale.
QA stops being “a config validator in human form”
For Alexey, the most important consequence of implementing this kind of pipeline is a change in what a QA Engineer actually does. “We’ve stopped being config validators in human form,” he says. Instead of manually checking large volumes of data, the QA team increasingly designs and maintains systems that perform those checks automatically.
The work shifts toward test architecture, risk analysis, reliability, agent behavior, and the design of quality systems that allow new features to reach players faster and more safely.
That creates a new engineering challenge of its own: the QA system itself cannot become fragile or dependent on one specific LLM model, subscription, or provider. The team now has to think about the architecture and resilience of the automation itself just as seriously as it once thought about individual test cases.
From a QA pipeline to a product pipeline
Alexey expects the boundaries to move even further over time. Today, the system begins with an already approved GDD. In the future, the starting point could be the product hypothesis itself. An idea could pass through systems that help structure the concept, configure it, analyze the business context, formulate requirements – and then seamlessly hand all of that over to testing. The result would not be a single AI-generated verdict, but several different perspectives: a QA report, a playtest report, product analytics, and potentially other assessments of whether the idea actually works. In such a system, the boundaries between QA, Game Design, and other product roles may become less rigid. But Alexey does not expect humans to disappear from the process. Instead, their role may shift closer to producing and directing the system. Agents can analyze requirements, perform thousands of checks, and report whether a feature meets the conditions it was given. But a human still has to decide whether the resulting feature should actually be released. “The human checkpoint will remain at the level of feeling and taste,” Alexey says. “Does it feel right to a person? Is it balanced? Is it boring or fun? Too long or too short?” That distinction may ultimately define the next stage of QA at Age of Magic.
Machines are becoming increasingly capable of determining whether a feature works correctly. But deciding whether it works as a game is still a human responsibility.
