Large tech companies run thousands of A/B tests a year, many at the same time and on the same users, request paths, and caches. When those experiments interact, they can bring production systems down, and the code behind them can outlive the teams that wrote it. This column argues that a central experimentation platform’s first job is to keep a live record of every experiment, and that AI agents cleaning up old experiment code need that record as much as engineers do.
When a team wants to try a new feature, it often runs an A/B test: it splits users into a control group and one or more treatments, each seeing a different version of the feature, and compares the results. The team starts with a small allocation of traffic and dials the experiment up gradually, often one location at a time. To get the experiment running quickly, the team may take a shortcut in a system that other teams also use. Host teams, which own shared infrastructure and high-level customer experiences such as search and product pages, often allow such shortcuts on the understanding that the changes will be removed if the idea is rejected. That’s a reasonable trade: a team should get a market response before investing in a full implementation.
The trouble starts when the shortcut outlives the experiment. A successful experiment can lead to a launch, but launched code isn’t necessarily ready for long-term use, and if another team adopts the workaround, it becomes part of someone else’s implementation. These temporary compromises, along with the dependencies and unfinished work they leave behind, are experimentation debt. It degrades the codebase and creates operational risk.
In large engineering organizations, away teams build specialized customer experiences, such as selling luxury goods, displaying ads, or delivering groceries, by changing host teams’ systems. A handful of experiments can be coordinated directly between teams. Thousands a year need a central platform and a team to run it.
That platform’s most important job is to keep a current record of every experiment: what it changes, who owns it, and where and how widely each of its treatments is running. Preventing collisions, enforcing host teams’ rules, and cleaning up old code, whether by engineers or AI agents, all depend on that record. At Amazon, I was a founding engineer and technical lead of such a platform, and later led the onboarding of multiple organizations onto it.
Where Experiments Collide
One user can be in many experiments at once, and one request can pass through code that several of them have changed. Each change may pass review on its own and still fail in combination. Collisions take two forms: experiments compete for shared resources, and they come to depend on each other’s changes.
Resource competition shows up first in latency. Each experiment on a request path may add only a little work, such as another service call or database lookup, and its cost looks small in its own results. But the request has a single latency budget, and no experiment owns the total. Caches pose a dilemma of their own. If a cached response ignores which treatment a user is in, people can be served the wrong version. If cache keys include treatments, overlapping experiments fragment the cache, lowering hit rates and sending more traffic to the services behind it. Reviewing each experiment in isolation misses both problems.
Dependencies are harder to see. Team A runs an experiment that makes a fundamental change to a shared service, and Team B builds on that change, assuming it will remain. When Team A dials its experiment down, or launches a different treatment from the one Team B built on, Team B’s feature breaks, and the risk grows when no one on Team A is left to warn Team B.
A platform can prevent some resource collisions by construction. Google described one approach in 2010: parameters are grouped into layers, and a request can participate in experiments from different layers but in at most one experiment within each layer. Layers don’t stop experiments from adding latency or splitting a cache, so a platform should also monitor latency, error rates, and cache hit rates as experiments are dialed up, helping teams catch regressions before they reach a larger share of traffic.
Neither safeguard catches a dependency like Team B’s. For that, the platform can require every code change to a host team’s system to reference the experiment it belongs to. It can then show which experiments touch the same code, and Team B can check the status and owner of Team A’s experiment before building on it.
Keep Ownership Current
Ownership goes stale first, because it changes with every reorganization. Experiments become orphaned when an away team is disbanded or reorganized, or when the engineers who worked on them move on. The host team still owns the service, but no one clearly owns finishing or retiring the experiment. Keeping ownership current can’t depend on people updating records after every reorganization.
A better mechanism attributes each experiment to a team and resolves that team through its management chain: manager, senior manager, and so on up to the CEO. When a team dissolves or its manager leaves, ownership moves automatically to the next level up, so no experiment is ever unassigned. Whoever the experiment resolves to has two choices: find a better owner willing to claim it, or handle it themselves.
Host Teams Set the Rules, the Platform Enforces Them
Host teams set the rules for their systems, including review processes, coding standards, and deadlines for cleaning up experimental changes. They run those systems, answer for their incidents, and know their limits. A central team writing rules for thousands of systems would either produce rules too generic to help or become a bottleneck for every change.
The platform puts those rules where away teams will see them and checks that they’re followed. When an away team sets up an experiment on a system, the platform shows the host team’s requirements and asks for an owning team and a cleanup deadline, and it later flags experiments that miss the deadline. Enforcement matters most across organizational boundaries, because a host team usually has no authority over an away team in another organization. The platform applies the same policy to every team on the system, and the strongest lever ties compliance to new work: an organization with unresolved violations, such as stale or abandoned experiments, can’t start new experiments on that system until it clears them.
Product, user experience, and technical decisions stay with the teams. The platform provides controls for changing allocations, such as dialing a treatment up in one location while holding it in another, and host and away teams decide when to use them. Dialing an experiment down automatically when it goes stale would be risky: if another team has built on it, as Team B did, that team’s feature breaks.
Automate the Follow-Through
Apart from those decisions, the platform can do the rest automatically. It watches every experiment and opens a ticket for the owner as soon as one goes stale, for example when a treatment has launched in every location but the experiment code is still in place, or when an experiment passes its cleanup deadline. The ticket lists the code that still references the experiment, and AI agents can prepare the cleanup change. If the owner doesn’t act within a set window, the ticket escalates, and once the cleanup change deploys, it closes on its own. When an experiment is closed, the platform lists the code it touched and checks that it is turned off. That leaves the owner with one decision: retire the experiment or make its code fit for long-term use.
Leaders can review a few metrics monthly to see whether the debt is shrinking: active, inactive, and newly created experiments, experiments overdue for retirement, and policy violations, broken down by category and away team.
AI Agents Need the Record Too
AI agents can take on much of the cleanup, but only with access to the system that stores each experiment’s flags, treatments, and allocations. Source code can’t tell an agent whether an experiment is live: the same flag check looks identical whether the experiment is running at 1% in one location, has launched everywhere, or was abandoned months ago. An experiment may also have several treatments, each at a different allocation in each location, and one treatment may have launched in one location while the experiment still runs in another. Along with that live state, an agent needs the experiment’s status and its current owner.
Stale feature flags are the most visible form of experimentation debt. In 2026, US food delivery company DoorDash reported managing more than 60,000 flags, over a thousand of which were no longer needed. According to DoorDash, manually removing a single stale flag could take an engineer one to two hours. Because a flag’s name, the function that reads it, and the conditional logic can live in separate files behind layers of wrappers that pattern-based tools struggle to follow, DoorDash built an LLM-based multi-agent system.
Before making changes, the agents query DoorDash’s experimentation platform through MCP for each flag’s current rollout percentage and target value. An engineer confirms which behavior should remain after cleanup, which catches ambiguous cases such as a partial rollout or an abandoned experiment. The agents then update the code and tests and open pull requests assigned to the flag’s owner. Of the 50 most recent stale flags processed, the system produced usable pull requests for 45.
Automated cleanup existed before AI agents: by 2020, ride-hailing company Uber had removed around 2,000 stale flags with Piranha, an open-source static analysis tool. Piranha sent each change to the flag’s original author for review, but that person might have moved to another team or left the company. Routing pull requests is only as reliable as the owner record, and agents inherit the same gap unless ownership resolves automatically.
AI code reviewers depend on the same record. In my experience, they are useful because they can check proposed changes against host teams’ policies written in plain language, and they work best when they can also see which experiment a change belongs to, who owns it, and whether it’s past its cleanup deadline.
