Every day, online platforms make millions of decisions about what content people can see, what gets flagged, and what gets removed. Behind those decisions are machine-learning systems tasked with identifying spam and harmful activity at a scale no human moderation team could manage alone. But as bad actors adapt their tactics and AI models become more sophisticated, detecting problematic content is only half the challenge. The other half is knowing when those systems get it wrong—and building the infrastructure to make them better.
Software engineer Nishant Guarav has worked on this challenge firsthand. During his time at LinkedIn, he developed backend infrastructure that connected automated content classification, human review, and machine-learning model evaluation. His work helped create a common foundation for testing and improving different models without requiring engineers to rebuild the supporting systems each time. He is also a named inventor on the published patent application Spam Processing with Continuous Model Training, which explores an approach to using classification results and human feedback to continuously improve spam detection.
In this interview with AI Journal, Guarav discusses the engineering challenges behind large-scale content moderation, why human feedback remains critical to effective machine learning, and what the rise of large language models means for the next generation of trust and safety systems.
What are some of the biggest challenges in developing machine-learning systems that can accurately identify spam, low-quality content, and harmful online activity?
One challenge is that spam keeps evolving. Once you start detecting certain patterns, people find new ways around them.
The harder problem, in my experience, is knowing whether your system is actually improving as spam patterns evolve. A model might catch more spam but also start blocking legitimate content. Looking at accuracy alone doesn’t tell you enough. Precision and recall help reveal that tradeoff, but those numbers are only meaningful when you’re evaluating models against reliable, representative data.
I’ve always felt that a good classification model is only part of the solution. You also need the infrastructure to collect reliable feedback, investigate incorrect decisions, and measure improvements consistently.
Can you tell us about your experience building machine-learning infrastructure for content moderation? What were some of the more complex engineering problems you had to solve?
During my time at LinkedIn, I built backend infrastructure supporting content classification, human review, and machine-learning model evaluation. The platform was model-agnostic, allowing data scientists to develop, evaluate, and improve different ML models without having to rebuild the underlying systems.
That meant building systems to collect content and classification results, identify potentially incorrect decisions, route content for human review, and produce reliable labeled datasets.
Another important part was supporting experimentation. Different model versions needed to be evaluated against consistent data so engineers could compare their precision, recall, and overall performance.
The challenge was making all of this work as a reusable platform rather than building a separate pipeline for every model.
Many content moderation systems rely on a combination of automated detection and human review. How can these two approaches work together to improve accuracy and efficiency?
I think the two work best when human review is built into the system rather than added as an afterthought.
Automated classifiers can handle large volumes of content, but they will always make some questionable decisions. Reviewing everything manually isn’t realistic either.
One area I worked on was deciding which classifications needed another look. We could consider the model’s confidence and other signals about the content or its source, then route potentially incorrect decisions to reviewers.
Those reviewers provided something extremely valuable: reliable labeled data.
That data could then be used by different ML teams to train and evaluate improved models. Human review wasn’t just correcting individual mistakes. It was helping the entire system learn.
You’ve worked on technology that allows machine-learning models to continuously improve. What limitations did you identify in existing approaches, and how did your work address them?
One limitation I saw was that model development could become disconnected from what was happening in production. A data scientist might have a better model idea, but testing it required good training data, reliable human-reviewed labels, and a consistent way to measure performance. Building that infrastructure separately for every model would be wasteful.
My work focused on creating a common foundation for those activities.
The platform could collect different samples of content, incorporate reviewer feedback, and support evaluation of new model versions against existing ones.
The key idea was to make continuous improvement repeatable. Model developers could focus on their algorithms while the infrastructure handled much of the data collection and evaluation workflow.
You’re a named inventor on a published patent application involving machine learning. Can you explain the technology behind it and what makes your approach different?
I’m one of the named inventors on Spam Processing with Continuous Model Training, a patent application published during my time at LinkedIn.
The architecture connects several parts of spam detection that are often treated separately: classification, confidence assessment, human review, data collection, and model improvement.
For example, a model could classify something as spam with high confidence, but other information about the source might suggest a possible mistake. Instead of blindly filtering it, the system could send it for expert review.
The reviewed content then became useful for future model training and evaluation.
My contribution was primarily on the infrastructure side. We wanted a system where different ML models could be developed and compared without rebuilding the underlying review and evaluation machinery each time.
As online platforms grow, they need to process enormous amounts of content while responding to new forms of spam and abuse. What does it take to build infrastructure that can keep up with those demands?
The first thing is separating what needs to happen immediately from what can happen later.
Content classification has to be fast. Human review, data sampling, and model evaluation have different requirements.
You can’t send everything for manual assessment, so the infrastructure needs to identify which cases are worth reviewing. You also need to collect representative samples, including content the current model might be missing.
Another important consideration is making the platform reusable. When data scientists introduce a new model, they shouldn’t have to rebuild the entire pipeline around it.
I think that separation is essential to making machine-learning systems sustainable as both traffic and the number of models grow.
One of the challenges with automated content moderation is avoiding false positives while still catching harmful content. How have you approached that balance in the systems you’ve developed?
This was an important part of the infrastructure I worked on.
A false positive means legitimate content gets blocked. But if you loosen detection too much, harmful content gets through.
We approached this by looking beyond the classification label itself. The model’s confidence mattered, but so did information suggesting that its decision might be questionable.
The infrastructure could direct those cases to human reviewers rather than acting on every prediction automatically.
We also needed a reliable way to evaluate this tradeoff across model versions. Precision and recall helped show whether a proposed change improved detection or simply shifted the mistakes elsewhere.
Looking ahead, how do you see advances in AI and large language models changing content moderation? What challenges do you think engineers will still need to address?
Large language models can help with content that requires more context. They may recognize subtle forms of abuse that traditional classifiers struggle with.
But they also introduce challenges around latency, cost, inconsistent predictions, and explainability.
I think the underlying infrastructure will remain important. Whether you’re using a traditional classifier or an LLM, you still need trustworthy labeled data, human feedback, and a way to compare model performance.
That’s one reason I’ve always liked the idea of model-agnostic infrastructure. The models will keep changing, but the need to evaluate them carefully and learn from mistakes won’t.
