One Idiom, Five Failures: What Happens When You Run the Same Translation Through Every GPT Model at Once

The benchmark problem 

The AI industry runs on benchmarks. MMLU, HumanEval, HellaSwag, the list grows longer every year. What these frameworks consistently skip is a deceptively simple test: can a model distinguish between a French idiom about insects and a statement about a person’s emotional state? Run one phrase through every available GPT model at once and that question stops being rhetorical. Some versions answer it correctly. Others return output that is grammatically clean and contextually wrong. All from the same provider, all under the same product name. 

The gap in evaluation frameworks sits at the intersection of two fields that rarely speak to each other: AI benchmarking and applied natural language processing. Bring them together and a question emerges that most AI coverage ignores: not which provider is best at translation, but which version within a provider is reliable enough, and whether anyone building workflows on top of these tools actually knows which version they are routing through. 

A one-sentence diagnosis of the real risk 

Here is the practical problem: most users who reach for an AI translation tool select a provider, not a model version. They open a product, type in text, and receive output. The routing logic is invisible. The model version is invisible. The quality variation between versions is invisible. 

This is not a hypothetical concern. Research into production LLM deployments has documented that version drift, the behavioral changes that occur when a provider updates or retunes a model, can introduce breaking changes in output format, reasoning style, and linguistic register without any public changelog entry. A workflow that returns reliable idiomatic translation on Monday may degrade silently by Wednesday, with no signal to the user that anything changed. 

What model version opacity actually costs 

The version gap problem is well documented in performance benchmarks. A 2026 analysis of LLM production deployments found significant variance in model behavior even within the same provider family, with researchers observing that “a model that leads in speed during Monday’s testing may experience severe degradation by Wednesday.” Performance volatility is not unique to latency: it extends to output quality in ways that tool vendors rarely surface. 

For translation specifically, the cost of this opacity is asymmetric. Latency degradation is observable, users wait longer and notice. Idiomatic failure is not. The output looks like a plausible English sentence. A word-for-word rendering of a French idiom passes spell-check, passes quality gates, and gets published. The person who commissioned the translation never finds out it meant something else entirely. 

Multiply that across high-volume translation pipelines and the invisible failure rate becomes a significant operational risk, particularly in sectors where language precision carries legal or reputational weight. 

The test: one idiom, every GPT model, run simultaneously 

The phrase tested was avoir le cafard, a common French idiom meaning to feel deeply down or melancholic. Literal translation: “to have the cockroach.” The phrase has nothing to do with insects. It is a well-documented expression used in everyday French conversation, journalism, and literature. Any model trained on a representative French corpus has encountered it. The question is not recognition. The question is whether the model understands that it is an idiom at all, and what it does when it does not. 

The results: 

Two models returned the same literal output. Three produced something idiomatically defensible, though not identical to each other. All five are from the same provider. All five are available through the same ChatGPT interface. 

The two literal outputs, GPT-4o-MINI and GPT-4.1-NANO, are both efficiency-optimized, smaller-parameter versions. They did not fail to process the sentence. They processed it perfectly and translated each word individually, which is exactly the wrong approach for an idiomatic phrase. The result reads like English. It means nothing. 

That is what makes this failure category genuinely dangerous. “To have the cockroach” is not an obvious error. A French speaker would catch it immediately. A non-French speaker reviewing a translated HR memo, support article, or product description probably would not. The document ships. The meaning is lost. 

When correct still means disagreement 

The three models that produced idiomatic output did not agree with each other. “To feel down,” “to be in low spirits,” and “to feel blue” are all defensible English renderings of avoir le cafard. They are not the same. 

“To feel down” is general and conversational, the kind of phrasing that fits a casual email but reads as thin in literary or clinical contexts. “To be in low spirits” carries a slightly formal register, better suited to a business communication or editorial piece. “To feel blue” lands as idiomatic but distinctly American in flavor, which creates its own localization question when the target audience is international. 

The original French phrase carries a specific weight, something closer to a settled, heavy gloom than a passing dip in mood. None of the three English outputs fully captures that weight. Standard translation quality frameworks are not built to measure this kind of register variance. BLEU scores compare against a reference, but who chose the reference, and which of these three would they have picked? 

What a multi-model view changes 

The most direct response to version-layer opacity is transparency at the output level. The emerging pattern in AI tool design, documented across image generation, coding assistants, and other domains, is that multi-model platforms are replacing single-tool workflows precisely because the performance gap between specialized models is wide enough that routing all tasks to a single model is a quality compromise. 

In translation, this approach takes the form of running multiple models simultaneously and comparing outputs at the sentence level. Multi-engine AI translation platforms that aggregate 22 or more AI models surface exactly the kind of divergence the test above exposed: when two models return the same literal rendering and three others produce different idiomatic versions that disagree on register, the pattern of disagreement is itself the signal. It tells you that idiomatic content is in play, that the sentence requires judgment, and that routing it blindly to a single model is a risk. 

MachineTranslation.com, an AI translator built by Tomedes, applies this logic through its SMART system: rather than routing a job to a single model and returning a single output, SMART runs jobs across its full model ensemble and surfaces the translation that the majority of models agree on at the sentence level. When models diverge, as they clearly do on idiomatic content, SMART flags the disagreement rather than hiding it. The point is not to replace human judgment on high-stakes content; it is to make the version gap visible, so that judgment can be applied where it is actually needed. 

The governance question nobody is asking 

There is a broader question here that extends well beyond translation. Enterprise AI deployments increasingly depend on consistent, predictable output from model APIs, and model providers do not publish translation-specific changelogs. A model version that handles French idioms competently today may be updated, fine-tuned, or silently swapped next quarter. The user has no way of knowing, no changelog to check, and no alert when output quality shifts. 

For teams building translation workflows into legal, compliance, or customer-facing operations, this is a governance risk that most current frameworks do not cover. Evaluation typically happens at deployment, a one-time quality assessment that does not account for ongoing version drift. The result is a class of quality failure that is systematic, invisible, and never attributed to the underlying cause. 

The AI industry has invested heavily in measuring which provider is best. The more useful question in 2026 is whether the model version your workflow is currently routing to is the same one you tested, and whether it still knows that “avoir le cafard” has nothing to do with cockroaches. 

“When several AI engines align on the same sentence, the probability of an invented or fabricated segment drops significantly. We built SMART to make that convergence visible, not to replace translators, but to show teams exactly where the version gap is creating risk.”, Ofer Tirosh, CEO, Tomedes