AI & Technology

From Test Execution to Engineering: How SuperCity Is Building AI into QA

For many game studios, AI adoption begins with isolated experiments: someone tries a coding assistant, uses an LLM to summarise a document or generates a small internal script.  

At Playkot’s SuperCity studio, the more important shift began when these experiments started turning into tools embedded directly into production workflows. 

SuperCity is a 15-year-old live-service game. Over that time, it has accumulated a large codebase, interconnected systems, recurring events and extensive product knowledge. Even a seemingly small change can affect game configurations, server logic, player-facing interfaces and habits formed over years. For the studio, AI is therefore not primarily about replacing specialists or delegating decisions to a model. It is about reducing routine work, connecting information spread across different systems and giving people more time for investigation, product thinking and judgment.  

“AI is not concentrated in one team at SuperCity,” says  Yury Zinkevich, QA Lead at the studio. “Different specialists use it in different ways and integrate it into their own workflows.” 

 Yury dates the move towards more systematic adoption to mid-2024, when the team began looking beyond individual productivity and considering how AI-assisted tools could benefit a wider part of production.    

Starting with a practical QA problem 

One of the first questions was deliberately simple: could a substantial part of a routine QA check be reduced to something close to a single action?  

Some configuration checks could take up to eight hours. They also depended heavily on the knowledge and habits of individual specialists, who might approach the same task differently. Besides consuming time, this created a concentration of expertise in particular people. The initial answer was not to ask an LLM to validate everything itself. Instead, the team used AI to help design and write deterministic scripts capable of performing repeatable checks reliably. 

This distinction became central to SuperCity’s approach. LLMs are valuable when a task involves unstructured text, collecting context, generating tooling or coordinating several systems. But when something can be checked using a precise rule, the rule itself should remain deterministic. Once the team saw a measurable effect from conventional tooling, it began exploring the next layer: analysing requirements, assembling information from different sources, interpreting player reports and helping specialists reach decisions faster. 

A checklist that reads both intent and implementation 

One of the studio’s newer services automatically prepares test documentation. 

A QA engineer sends the agent a link to a task through Slack. The system collects the description, comments, agreements recorded during discussion, subtasks and links to the relevant merge request or feature branch.  

It then examines the code diff, identifies the functions affected by the change and looks at where those functions are used. Because SuperCity has a large repository, the depth of the analysis is deliberately limited rather than allowed to expand indefinitely. The agent combines its findings with context prepared for the feature, module and game, then generates a checklist containingpositive and negative scenarios, possible edge cases and a risk map. 

The important part is that the agent analyses both the original task and the implementation. 

The task explains what the developer was expected to build. The merge request shows what was actually changed. Comparing the two can reveal an incomplete implementation, an ambiguity in the requirements or a difference between the acceptance criteria and the resulting code.  

Instead of producing only a generic list of checks, the agent flags potential inconsistencies so that the team can clarify them before testing continues. It also structures the checklist across different layers, from game configurations to connected systems and server logic.  

This supports an important QA principle inside the studio: a specialist should not only verify the change described in the task, but also consider what else that change might affect. The generated checklist is not treated as a final authority. A QA specialist reviews it, removes irrelevant points and adds product context that the model may not possess.  Yury estimates that approximately 75–80% of the generated material is useful; the remainder may be too broad or unrelated to the particular change. 

The service is still relatively new, so the studio is cautious about presenting a single headline efficiency figure. Its immediate value is broader coverage and a more consistent process. For recurring configuration checks supported by the team’s deterministic tools, however, SuperCity estimates that preparation and validation time has fallen by around 40–50%, depending on the workflow.       

Connecting player feedback with releases 

A second service focuses on the period immediately after a release. 

SuperCity operates across four different networks, each with its own users and infrastructure considerations. Previously, release monitoring required QA specialists to switch between player-support reports, error-monitoring platforms and other technical systems, compare changes over time and determine whether a new signal was related to the latest deployment.  

The agent now performs much of this observation and initial context gathering. 

After a deployment, a QA specialist asks it to monitor a particular release during a defined period. The system watches the sources it is permitted to access and alerts the team when it detects a meaningful rise in errors or player reports. Rather than simply announcing that something has changed, it groups the evidence and suggests a concrete problem or hypothesis for the team to investigate.  

To distinguish a genuine spike from ordinary fluctuations, the studio worked with its support and customer-experience teams. Historical reports, existing tags and examples were used to define categories and calibrate thresholds. 

The first versions were too sensitive and sometimes reported incidents when the game was operating normally. The team therefore tested and adjusted the thresholds against historical patterns and real releases. According to  Yury, the current service can identify a developing cluster of related reports around two hours earlier than a person would typically notice and investigate it manually. Once the system detects a spike, the QA team can receive the relevant context within minutes instead of beginning with a manual search across several platforms, languages and monitoring tools.     

In one case, players received a seasonal reward but were then shown the previous season by default. The feature itself was working as designed, yet some players did not realise that they could switch seasons and interpreted the interface as a problem. The agent detected the pattern in their reports. This helped the studio understand that the issue was not a broken reward mechanic but unclear UI, allowing the team to plan an interface improvement based on the real source of player frustration. 

For a live-service game, linking feedback to release history is essential. The team cannot pause a rollout in response to every individual complaint or unrelated fluctuation. It needs to understand whether the signal was caused by the current release, whether deployment should stop or whether the issue can be investigated separately without disrupting the content schedule. 

Deterministic checks, AI-assisted engineering 

SuperCity’s configuration-validation tooling demonstrates another principle: adopting AI does not mean making every part of a system probabilistic. 

“If something can be checked with a deterministic rule, we will check it with a deterministic rule,”  Yury explains. 

An LLM may help create a script, select a tool or coordinate a sequence of checks. The validation itself, however, remains predictable. Sending large volumes of configuration data directly to a model would be expensive, difficult to verify and less reliable than applying a defined rule. What began as a separate collection of scripts has since been integrated into SuperCity’s internal administration platform, which game designers and other specialists use to work with game configurations. 

As more QA engineers began contributing AI-assisted scripts, the team also created a common process for writing, reviewing and reusing them. Instead of every specialist developing tools differently, the studio introduced shared standards for scripts, reviews, data handling and existing rules 

This has changed more than the speed of individual checks. QA specialists without deep programming expertise have learned that they can build internal tooling, formalise repeated knowledgeand improve workflows used by the wider team. 

The role of QA is consequently moving beyond executing a prepared list of requirements. With greater access to context and better tools, QA can become involved earlier, identify formal gaps during feature planning and ask product teams better questions.     

Giving agents access without giving up control 

The studio is also experimenting with an MCP layer connected to its internal administration platform. 

Selected API capabilities are exposed as tools. A specialist can describe a multi-step task in natural language, after which the agent interprets the request, identifies the appropriate tools, asks clarifying questions when necessary and executes the permitted sequence of actions. The greatest value appears in repetitive workflows that would otherwise require moving through multiple screens and making dozens of related changes manually. 

However, the agent does not receive unrestricted access. Read-only permissions are the default for systems where it only needs to retrieve and interpret information. Actions are logged, and potentially critical operations can be reviewed or rolled back. 

High-level actions such as publishing or deleting content are not currently delegated to the agent. The studio has chosen not to expose them until the workflow has demonstrated sufficient valueand additional safeguards are in place. 

The longer-term ambition is to create a connected agent environment in which different internal systems provide tools and context through MCP servers. Such an environment could support active-event monitoring, player-feedback analysis, test-data preparation and other multi-stage workflows without requiring specialists to coordinate every system manually.     

The next version of QA is still human 

SuperCity is not trying to automate the entire testing process. 

Full agent-driven interaction with the current game client would require substantial visual processing, repeated screenshots and automated interface actions. For now, the studio sees greater practical value in automating the work around testing: preparing environments, reviewing documentation, clarifying requirements, analysing production signals and assembling context.  

Human judgment remains essential whenever product knowledge, ambiguous player behaviour or high-impact decisions are involved. AI can suggest a checklist, identify a pattern or execute a permitted workflow, but a specialist must decide whether the result is relevant and what should happen next. 

“The role of QA has shifted from execution towards engineering,” says Yury. “We formalise processes, build our own tools, collect data about quality and the testing cycle, participate in requirements analysis, and create services that operate both before and after a release.” 

In a mature live-service game, this broader perspective matters. The purpose of AI is not to remove people from the workflow, but to reduce the repetitive work that prevents them from seeing the product as a whole. 

Author

Related Articles

Back to top button