Monte Carlo Tree Search has become one of the best-known approaches for making decisions in environments where examining every possible future is computationally impossible. Instead of exhaustively exploring an entire decision space, MCTS selectively searches the futures that appear most promising while balancing exploration of alternatives.
That model works particularly well when decisions are sequential: one agent acts, the environment changes, another decision is made, and the process continues. Multi-agent robotics introduces a different problem. Multiple autonomous agents may need to commit to actions during the same control cycle, without knowing what the others will select beforehand.
ZoumMax is an experimental MCTS variant designed around that setting. Rather than maintaining one tree containing every combination of actions from every agent, it separates the search into one tree per agent and synchronizes those trees through a shared simulator.
Why Conventional MCTS Becomes Difficult With Multiple Agents
Traditional MCTS is commonly described through four stages: selection, expansion, simulation and backpropagation. A search tree represents possible futures, and statistics collected from simulations gradually steer the search toward stronger decisions.
The UCT approach introduced by Levente Kocsis and Csaba Szepesvári combines MCTS with an Upper Confidence Bound selection strategy. It provides a practical mechanism for balancing actions that already appear promising against actions that have not yet been sufficiently explored.
The structure becomes more complicated when several agents act simultaneously. Suppose each robot has eight possible actions during a control cycle and four robots must choose at the same time.
Representing every joint action requires examining up to:
8 × 8 × 8 × 8 = 4,096
possible combinations at a single level of the tree. Adding another robot increases that number to 32,768.
This exponential growth is the central difficulty. With a real-time control budget measured in milliseconds, the search may spend most of its computation exploring the first one or two levels instead of looking meaningfully into the future.
Decoupling the Search Trees
ZoumMax approaches the problem differently by assigning each agent its own search tree.
If four robots each have eight possible actions, each individual tree still branches into only eight choices rather than representing all 4,096 joint combinations. The interaction between agents is therefore removed from the topology of the tree and moved into the simulator.
A node in an agent’s tree does not represent the complete state of the world. Instead, it represents that agent’s own sequence of actions from the root.
For example, a branch might represent:
Move forward → rotate left → slow down
The actions selected by the other agents are not stored inside that node. Their influence appears indirectly through the rewards produced when all selected actions are simulated together.
This makes ZoumMax an open-loop search method: the tree primarily represents action sequences rather than an explicit graph of future states.
Lockstep Descent Keeps the Trees Connected
Completely independent search trees would create another problem: each robot could optimize against a future that has nothing to do with the futures considered by the others.
ZoumMax therefore descends the trees in lockstep.
At depth one, every agent selects one candidate action from its tree. Those actions are combined into a joint action and executed inside the simulator.
The resulting simulated state becomes the starting point for depth two. Each agent then chooses another action from its own tree, another joint action is constructed, and the simulator advances again.
This continues until a predetermined search depth is reached.
The trees never directly share nodes or statistics, but their decisions interact during every simulated step. Depth five in one agent’s tree therefore corresponds to the same simulated control cycle as depth five in every other agent’s tree.
Let the Simulator Carry the Interaction
This creates an important separation between search and interaction.
Each agent searches only its own action space. The simulator is responsible for calculating what happens when those independently selected actions meet in the same environment.
Consider several mobile robots approaching the same constrained area. One robot choosing to accelerate may be beneficial if the others turn away, but dangerous if another robot selects a conflicting trajectory.
ZoumMax does not need to explicitly place every possible opponent trajectory underneath that robot’s tree node. Instead, repeated simulations expose the node to different combinations of behavior from the other agents.
The resulting reward becomes an estimate of how useful that action sequence is against the behaviors currently being produced by the other agents’ searches.
Replacing Long Random Rollouts With Bounded Evaluation
Classical MCTS often continues from a leaf node using a rollout or simulation policy until it reaches a terminal outcome. That works well when terminal states can be reached relatively quickly and random play still provides meaningful information.
Many robotics and continuous-control problems have long horizons. Randomly controlling a robot for hundreds of simulated steps may generate little useful information while consuming most of the available computation budget.
ZoumMax instead treats the search depth itself as the planning horizon.
If the configured depth is eight, every iteration simulates up to eight future control cycles. At the final state, a heuristic evaluator estimates how favorable that state is for each agent.
The evaluator could consider factors such as distance to a target, collision risk, progress toward a task, resource usage or any other domain-specific objective.
There is therefore no separate random rollout phase. Computation is concentrated on expanding and evaluating the actual search tree.
Backpropagating a Reward Vector
Multi-agent control is not necessarily zero-sum.
One robot succeeding does not always mean another robot must fail. Agents may cooperate on some objectives, compete for resources on others and maintain entirely independent goals elsewhere.
For this reason, ZoumMax evaluates the simulated state separately for every agent. Instead of producing one reward, the evaluator produces a vector:
(r₁, r₂, … , rₙ)
Each component is then backpropagated only through the tree belonging to that agent.
Robot one updates its search using its own evaluation. Robot two does the same with its evaluation, and so on.
This allows the method to operate in general-sum environments without assuming that one agent’s reward must simply be the negative of another’s.
Keeping UCB Exploration Meaningful
Another practical issue appears when heuristic evaluations replace simple win-or-loss results.
A UCB exploration term behaves differently if rewards fall between zero and one than if an evaluator returns values in the thousands. If raw scores are extremely large, exploitation can dominate exploration; if they are extremely small, the opposite can happen.
ZoumMax therefore normalizes observed child values locally before applying its UCB-based selection rule.
The objective is not to change which actions are considered better. It is to keep the exploitation and exploration components operating at comparable numerical scales throughout different parts of the tree.
This becomes especially useful when the meaning and magnitude of the evaluation function change as the search moves deeper into the simulated future.
Why Open-Loop Search Can Help
Representing nodes as action prefixes rather than exact states introduces both benefits and costs.
The immediate benefit is simplicity. The search does not require every simulated world state to become a unique persistent tree node.
This can be useful in stochastic systems, where executing the same action sequence may lead to slightly different future states. Rather than creating separate branches for every possible transition, those variations contribute to the statistics of the same action-prefix node.
It can also reduce memory requirements and make the tree structure compact enough for repeated real-time planning.
The cost is state aggregation. Two meaningfully different world states may have arrived through the same sequence of actions and will therefore share statistics.
That trade-off means open-loop search is not automatically appropriate for every robotics problem.
Known Limitations
ZoumMax should not be treated as a Nash-equilibrium solver.
Research into simultaneous-move MCTS has shown why deterministic selection strategies can fail to converge to equilibrium behavior in certain games. Approaches based on regret matching or algorithms such as Exp3 provide stronger theoretical properties in particular two-player zero-sum settings.
ZoumMax makes a different trade-off. Its goal is to obtain useful decisions under a restricted simulation and timing budget rather than guarantee equilibrium convergence.
A second limitation comes from the synchronized searches themselves. As each agent’s tree gradually concentrates around its preferred actions, the behavior observed by the other agents is increasingly generated by those same search policies.
The trees can therefore reinforce assumptions about one another. More exploration and larger simulation budgets can reduce this effect, but they do not remove the underlying limitation.
State aggregation is another important concern. If two futures reached by the same action sequence differ in a way that strongly affects the correct next action, an open-loop representation can hide information that a state-based search would preserve.
Where This Approach May Be Useful
The design is most relevant when several conditions appear together: multiple agents make decisions simultaneously, the available computation time is short, an accurate simulator exists, and the joint action space grows too quickly to search directly.
Multi-robot coordination is one example. Autonomous vehicles negotiating shared environments, fleets of warehouse robots, simulated drone coordination and multi-agent industrial control can create similar planning structures.
The approach becomes less attractive when accurate state-dependent policies are essential, when only a few agents and actions exist, or when equilibrium guarantees are more important than short-horizon decision speed.
The action representation also matters. Continuous robotic controls must typically be discretized into a manageable set of representative actions before tree search can be applied effectively.
A Different Point in the MCTS Design Space
ZoumMax is best viewed as a combination of design choices within the broader MCTS family rather than as a replacement for Monte Carlo Tree Search itself.
The central changes are straightforward: separate the tree by agent, index nodes using action prefixes rather than complete states, synchronize those trees through a simulator, stop the search at a fixed horizon, evaluate that state directly, and return each agent’s reward only through its own tree.
None of those ideas eliminates the fundamental difficulty of multi-agent planning. What they do is move complexity away from an exponentially growing joint-action tree and toward repeated simulation of smaller, synchronized searches.
For robotics systems operating under strict computational deadlines, that trade-off may be worth exploring.
As autonomous systems increasingly move from isolated decision-making toward environments containing many simultaneously acting agents, search algorithms will need to account for more than the quality of their decisions. They will also need to decide which parts of the future are computationally affordable to represent.
