Skyfall AI’s Morpheus Benchmark Reveals LLMs Aren’t Actually Learning

Enterprise AI is shifting from static chatbots to autonomous always-on agents. Today’s supply chains, warehouse operations, and production planners don’t just want systems that can answer questions; they want systems that can make sequential decisions in environments that are constantly shifting.

But according to a new research paper from AI lab Skyfall AI, the current crop of frontier models is hiding a massive structural flaw: they aren’t actually learning anything new.

In the paper, Skyfall AI unveiled Morpheus, a persistent enterprise simulation platform designed to test continual reinforcement learning (CRL). The lab’s findings suggest that when enterprise conditions inevitably change, models like GPT-5.5 and Gemini 3.1 Pro don’t adapt. Instead, they rely on massive pre-training data until they hit a wall, and then they crash.

“Stable absolute performance across configuration intervals is not evidence of robustness. It is evidence of a fixed policy operating within its coverage boundary,” the Skyfall AI team noted in the report.

Enter Morpheus: Testing the Big World Hypothesis

Standard AI benchmarks typically reset the world at the end of each episode. If a model makes a mistake, the slate is wiped clean.

Morpheus does not play by these rules. Grounded in the “Big World Hypothesis,” the platform simulates real enterprise environments where past decisions compound, objectives shift over time, and the consequences of an action might not be visible for several simulated days.

Skyfall evaluated GPT-5.5 and Gemini 3.1 Pro across two primary tasks:

• Task 1 (Dynamic Resource Allocation): Managing fulfillment capacity amid supply chain spikes and capacity drops.
• Task 2 (Scheduling Under Drift): Handling long-horizon dispatch scheduling where the rewards for a good decision are severely delayed.

The expectation was that Large Language Models (LLMs), which are fundamentally fixed-weight systems, would show measurable degradation. The reality was much more concerning.

The Four Core Failures of Frontier LLMs

Skyfall AI’s research surfaced four key findings that should serve as a wake-up call for anyone deploying LLMs in non-stationary enterprise environments:

1. Pre-training coverage is masquerading as adaptation.

On Task 1, both GPT-5.5 and Gemini 3.1 Pro maintained incredibly stable scores (averaging 0.918 and 0.864, respectively) even when injected with simulated data failures and capacity drops. While this looks like resilience, it’s an illusion. The models weren’t adapting to the crisis; they were just executing a fixed heuristic that happened to still work. The conditions simply hadn’t pushed them outside their pre-training distribution yet.

2. The context window bottleneck is very real.

When tasks got harder and consequence chains grew longer (Task 2 Inbound), the wheels came off. GPT-5.5 failed to detect that operational conditions had changed within its available context window, continuing to execute a broken strategy. Because the reward signal was delayed past the model’s memory threshold, it was effectively flying blind.

3. Models rely on generic heuristics, not reward signals.

The data revealed that Gemini and GPT-5.5 pursue entirely different allocation strategies. Gemini tends to concentrate its resources, while GPT-5.5 diversifies them. Both sound plausible on paper, but neither is derived from actually learning what works best in the Morpheus environment. Because they aren’t learning from the benchmark’s reward signals, they cannot optimize their performance to close the gap to a perfect score.

4. When LLMs fail, they fail opaquely.

This is the nail in the coffin for enterprise deployment. In Task 2, GPT-5.5 repeatedly collapsed to zero reward with no discernible pattern. With traditional Reinforcement Learning (RL), performance drops leave a breadcrumb trail, forgetting, slow adaptation, or policy oscillation, that engineers can fix. With LLMs, the cause is an unresolvable black box.

What This Means for Enterprise AI

The takeaway here isn’t that LLMs are useless for business. It is simply that their performance in evolving, chaotic enterprise environments is strictly bounded by their pre-training.

Right now, an LLM might seamlessly run a warehouse simulation because that exact scenario exists somewhere in its vast training data. But the moment a supply chain experiences an unprecedented anomaly, the agent has no underlying mechanism to detect the shift, learn from the new data, or recover its operations.

The industry’s focus must shift toward algorithms capable of genuine, reward-signal learning, and away from static models masquerading as adaptable agents. For researchers and founders looking to build those next-generation systems, Skyfall AI’s Morpheus benchmark is now available to put them to the test.

Read the full paper here: https://skyfall.ai/blog/llms-are-not-continual-learners.