Interview

Ayush Singh on Proving a Ranking Change Actually Made Things Better

When a platform says its feed or search results got better, proving it usually takes more work than making the change. Ayush Singh has seen that problem from several sides. He is a research scientist who works on the quality side of a large-scale ads ranking system at a major social media platform. Earlier in his career he was a software engineer at SoFi, and at the AI startup RapidKen.ai he led a team that built keyword-extraction and document-similarity tools. He holds an M.S. in Electrical and Computer Engineering from the University of Washington. 

In this conversation, Singh explains how teams test ranking changes before launch and where those checks tend to miss things. He also covers what a “hide ad” click can tell a company about its system, and how much he would trust LLMs to grade content. He ends with advice for smaller companies running their own recommendation models, and it starts with measurement. 

Platforms announce they’ve “improved quality” all the time. If you were reading one of those announcements, what would you want to see before you believed it? 

Platforms use “improved quality” claims to build trust and bring users back, so the first thing I look for is what the company actually means by quality. If the evidence is that users stayed longer or increased transactions on the platform, that shows engagement went up, but on its own it doesn’t show quality improved. For example, Microsoft published research about a Bing bug which regressed the quality of search results, but increased queries per user by over 10% and revenue per user by over 30%. Basically, people were searching more because they couldn’t find what they wanted. 

I look for four signals: user surveys, negative report/hide button clicks, human-rated content and LLM-labelled content. Each tells part of the story, and each can mislead on its own. 

User surveys capture what behavior can’t. Someone can watch a video to the end and still feel it wasted their time, which is why YouTube added satisfaction surveys on top of watch time. If quality really improved, I’d expect a higher share of positive answers to questions like “Was this worth your time?”, not just more engagement. The caveat with the signal is that its coverage is low, and we can’t be certain that all user groups have been correctly represented in it. 

Hides, reports, and “not interested” clicks are the clearest sign that something went wrong for a user, but each click on its own is noisy. Some people use these buttons constantly and others never touch them, so a single click tells you as much about the person as about the content. At SoFi, we saw this firsthand. A user might report an ad from a company because of one bad personal experience, even though the company had a good reputation overall. Acting on that one report would mean penalizing a good advertiser. The signal only becomes meaningful when you aggregate it across many users and over time for the same advertiser or piece of content, and look at the rate per impression rather than raw counts. So before we believe a quality claim, we should check if the aggregated rate is going down, not just fewer total clicks. 

With human rating, trained reviewers score content against written guidelines. This is usually treated as the gold standard, since reviewers apply consistent, carefully defined criteria and their labels tend to be the most accurate. The limit is volume. Large platforms add millions of new items every day and people can’t label all of them, so in practice teams label a small sample and assume the results hold for everything else. 

LLM labelling is the newest option and it’s catching on fast. Models can follow a platform’s written guidelines the way human labelers do, at a far larger scale. The LLM can output the final label, along with an explanation. The open question is trust: a model can be wrong and still sound certain, so it is important to have the labels checked against human ones. 

So, a credible quality announcement defines what quality means, reports how each of these signals moved instead of only the best-looking one, and shows the improvement lasted for months after launch. One launch rarely moves every signal, but none of them should get worse. 

Why does figuring out whether a ranking change helped sometimes take more effort than building the change in the first place? 

Building a ranking change can be seen as a bounded task: write the code, test it, and train the model. Evaluating the impact of a given change isn’t. At large tech companies, a single launch candidate is tested against dozens of metrics, if not hundreds. Microsoft has written that most of its teams compute hundreds of metrics per experiment, and LinkedIn says it computes over 1,000. Only a handful decide whether a change ships. The rest are there to catch side effects. With that many metrics, something important often moves the wrong way, and before the change can ship the team has to explain why and show how it will manage the risk. 

When a metric moves unexpectedly, someone has to work out whether it’s real or a bug, and why it happened. When LinkedIn redesigned its search page, many key metrics tanked, and the team had to add the changes back one at a time to find the culprit. Airbnb once observed that a new model increased long views of listings by a large margin, but bookings didn’t move. The team had to inspect listings by hand to learn that people were spending time on expensive or unusual places they never booked. 

Getting to the bottom of this generally takes several kinds of expertise at once. Data scientists check whether a movement is statistically real. Research and ML scientists work out whether the model is behaving as intended. Data engineers dig into logs and data pipelines to rule out bugs. At large companies, tracking down a single unexpected result can take several people weeks, sometimes months. 

Smaller companies often have the opposite problem: no metrics at all. During my time at the startup RapidKen.ai, we built a new technique to find keywords in a wiki and link them to the most relevant related wiki page. The roadblock came when we started iterating: we shipped several improvements, but had no way of knowing which ones actually made users engage more with the feature. We resolved it by adding measurement to track how many people clicked the inserted links, how long they used the tool, and whether they came back to the original page. That let us compare different techniques for picking the most relevant page against each other, keep the ones that worked, and drop the ones that didn’t. Getting accurate measurements took more effort than the ranking changes themselves. 

Whether at a large platform or a small startup, the code change might take a week, but proving what it actually did can take much longer. 

Walk us through how a team tests a change before it goes live. Where does that process tend to miss things? 

At most large tech companies, a ranking change goes through a series of gates before reaching everyone. Getting a change from idea to full launch can easily take more than a month. 

It usually starts offline, with the new model scored against historical data. This is fast and cheap. Airbnb’s search team has written that it takes only an hour or two, but “lacks accuracy”. Some companies add a faster online check, such as interleaving, before a full test. 

Changes that pass then go into an A/B test. A small percentage of users gets the new version, while a matching group stays on the current one, and the test needs to run at least a couple of weeks to collect enough data and smooth out day-of-week patterns. 

Next comes review, where the team goes through the full set of metrics and the hypotheses behind any surprising movement, and talks through the trade-offs with stakeholders across engineering, data science, and product. 

If the change is approved, it rolls out with a holdback: a small group of users is kept on the old version, usually for at least a couple of weeks, and longer when the decision was borderline or the team needs to understand long-term effects. The holdback results get the same review as the A/B test, and if they hold, the change goes out to everyone. 

Even with all of these gates, some things slip through. 

  1. Holidays and seasonal shifts in how people use a platform can produce test results that don’t hold the rest of the year. Teams that expect this run the A/B test and holdback longer. 
  2. Outside events that have nothing to do with the change, like a competitor’s launch or a big news cycle, can move the metrics during a test and make a change look better or worse than it is. 
  3. People often click on something simply because it’s new. A change can look like a clear win in the first two weeks and fade once the novelty wears off, and the reverse happens too, when users need time to get used to a change they initially resist. That’s a big part of why the holdback exists: it shows whether the effect still holds after a few weeks. 
  4. An overall win can hide a real loss for a smaller group, such as new users, people in a particular country, or users on older devices. If the review only looks at the topline number, that loss ships to everyone. Good teams break results down by key segments before deciding, but there are always more possible segments than anyone can check. At SoFi, for example, we looked at loan application results separately for students applying themselves and for parents applying on their behalf. Something like a cosigner step can make sense for a student with a limited credit history, but just adds friction for a parent who doesn’t need one, so a single combined number could easily hide both effects. 
  5. Each of these steps involves a lot of settings, and one misconfigured experiment can produce a wrong read or break something for users. As AI coding tools speed up how fast changes ship, there are more chances for that kind of mistake. 

Tracking quality over several years seems even trickier, since the ads, the audience and the measurement tools all keep changing underneath you. How do teams deal with that? 

Teams use a few approaches, and two of them do most of the work. 

The most direct is a long-running holdout: a small group of users stays on an older version of the system for years, and the team compares them with everyone else. Because both groups live through the same holidays and audience shifts and are measured with the same tools, those outside effects cancel out. In this way, holdouts can also show the true total of many launches combined. Google used long-running experiments for understanding how ads affect user behavior over years. There’s a real cost. Users in the holdout miss every new feature, which gives up some engagement and revenue, and large companies accept that price for a clean read on whether years of launches actually added up.  

The second is human rating against a fixed guideline. Engagement can shift for reasons that have nothing to do with quality, like a competitor’s launch, new regulation, or changing trends, so teams need a measure that stays stable across years.  Google publishes its Search Quality Rater Guidelines and reports running over 120,000 side-by-side rater experiments in 2023. If a team keeps the same guidelines and re-rates a fixed sample of content at regular intervals, it gets a quality reading that can be compared across years, regardless of who’s using the platform or how the logging has changed. 

Most people have hit “hide ad” without giving it a second thought. How much can those clicks actually tell a team about the system as a whole? 

People hit “hide ad” for all kinds of reasons: to clear the screen, because they dislike the brand, or by accident. Ranking systems still treat each hide as a strong signal and react hard to it, because they can’t tell a deliberate click from an accidental one. For one user, an accidental click can make the experience worse. For example, if the user hid a car ad accidentally, but was actually looking to buy a car, they might stop seeing deals near them. On the system level, Twitter’s open-sourced ranking algorithm gave a predicted “show less often,” block, or mute a weight of −74, against +13.5 for a reply, so a single negative signal can outweigh dozens of positive ones. That means a handful of offhand hides can push down an advertiser or a whole category that most people found perfectly useful. 

The buttons also do less than users assume.  For example, when Mozilla studied YouTube’s controls in 2022, it found that “Dislike” and “Not interested” prevented only about 12% and 11% of unwanted recommendations. And most people who lose interest in a platform never click anything. They just leave, so a low hide rate can mask real dissatisfaction. 

Where hides earn their keep is in aggregate. The long-term trend in the hide rate says a lot about whether the system is improving, and alerts on sharp jumps or drops in that rate can catch something broken before anyone files a bug.  

I think of hide clicks as a smoke detector: they tell a team quickly that something is wrong and roughly where to look, and taking the temperature of overall quality takes hides combined with user surveys and human labels. 

A lot of companies have started using LLMs to grade content quality. How far would you trust them, and what do you still want human feedback for? 

Given the recent advances with LLMs, I’d trust them to grade content far more than I would have a few years ago. The condition is that we measure how closely the LLM’s grades match human ratings of the same content. 

The biggest change is scale. Human rating is slow and expensive, so teams could only label small samples and detect large effects. LLM graders have changed that. Bing deployed LLMs for large-scale relevance labelling and found them as accurate as human labellers, and better than third-party workers for a fraction of the cost. Cheaper measurement lets teams compare more candidates and pick the best one faster. LLM labels can also be used to train other models, which wasn’t practical at this volume before. For example, Pinterest fine-tuned an LLM on about 2.6 million human-rated examples and used it to score search relevance in A/B tests. That opens up new ways to improve the whole product. 

Once an LLM grader matches human labels closely, I’d trust it for most everyday labelling. Getting there takes real work, though. A person has to define clearly what “good” means, the LLM’s labels need regular audits against human ones to make sure they aren’t drifting, and someone has to find the edge cases where the model gets it wrong. Because this approach is so new, not much has been published on it yet, but what has been published shows why the human work matters. OpenAI’s approach to content moderation starts with policy experts labelling a small “golden set” of examples. The model then labels the same set, and the experts study where the two disagree to refine the guidelines, repeating until they’re satisfied. Even then, GPT-4’s labels were on par with lightly trained moderators, but still outperformed by experienced, well-trained ones, and OpenAI’s conclusion was to use the model to free people up for “the complex edge cases” rather than to remove them. Hence, I’d treat an LLM grader as only as trustworthy as the last time it was compared with people, and keep a human in the loop for high-stakes decisions, where a confident but wrong label has real consequences. 

If a smaller company running its own recommendation model could borrow one habit from the way large platforms measure quality, what should it be? 

Larger companies have much more complex recommendation models, and much bigger teams to build them. The habit worth borrowing is investing early in a strong measurement system: the ability to tell, quickly and reliably, whether a change actually made things better.  

In my view, there are three main reasons it’s worth it. The first is speed of iteration as most ideas don’t work. Microsoft has published that only about a third of the ideas it tested improved their target metric, so the teams that move fastest are the ones that find out quickly which ideas work, instead of debating for weeks. With a good measurement system, nothing blocks a new change: you launch a test, get a trustworthy answer, and move on to the next idea. The second is knowing how the system is doing over time. That’s what shows a team its work is actually improving customer satisfaction, not just moving a dashboard. It also builds confidence, both inside the team and with leadership, that the progress is real. The third is replacing opinions with evidence and catching mistakes early. Even a simple A/A test, where both groups get the same experience, can reveal a broken setup. Microsoft called it “our most useful tool in identifying issues”. 

I saw the difference firsthand at SoFi and at the startup RapidKen.ai. At SoFi, we could set up A/B tests quickly with third-party tools that also tracked metrics across the loan application funnel. At the startup, we didn’t have that kind of measurement in place. We relied mostly on offline analysis of each new feature, had less confidence in what we launched, and had a harder time choosing the best launch candidate. 

If I were back at a startup like RapidKen, I’d set up three things before shipping the next improvement. First, basic usage tracking on every new feature: clicks, time spent, and whether people come back. Second, a simple A/B testing setup. Third-party tools make this quick, as I saw at SoFi, and it should run an A/A test first to make sure the numbers can be trusted. Third, a small set of a few hundred examples labeled by hand, so every new model can be scored against the same standard before it reaches users. None of that needs a big team, and it’s what turns “we think this is better” into “we know it is.” 

Author:

Related Articles

Back to top button