VLA vs World Model: Two Bets on How Robots Should Think

Bet on a vision-language-action model if you need a robot doing useful work soon, and on world models if you are funding research. Demonstrations exist for the first and not the second, which decides most projects. The bet flips when your task cannot be demonstrated by a human teleoperator at all, because then a model that predicts consequences is the only route. The rest of this post is for people who train models and now have to decide what runs under one, whether that is ROS 2 or HORUS.

You have read both stacks of papers. One camp shows a single network taking a camera image and a sentence and emitting the arm motions that follow, trained on thousands of hours of people driving robots by hand. The other camp shows a model that watches video, learns to predict what happens next, and then plans by imagining several futures and picking one. Both camps are confident. Both have a lab behind them that you respect.

On video the demos look the same. A robot picks up a thing it was asked for, a little slowly, in a room with good lighting, and the caption says the object was never seen in training. You cannot tell from watching which approach produced which clip, and neither can anyone else in the meeting where you are asked which one the company should bet on.

What bothers you is not the papers. It is that neither camp tells you what happens when the model is wrong at the exact moment the gripper is closing, and that question is the one your robot will ask you first.

Should you bet on a vision-language-action model or a world model?

Bet on a vision-language-action model if you need a robot doing a job within the year, and on a world model if you are funding a research programme rather than a product. The deciding factor is data, not elegance. A vision-language-action model eats demonstrations, so it works for tasks a person can perform through a teleoperation rig, and there are open checkpoints you can fine-tune rather than train from nothing. A world model eats observation, learns how the world responds, and lets a planner search through imagined futures, which is the only route left for tasks nobody can demonstrate: motions faster than a human can drive, contact-heavy searching, anything where the useful behaviour was never in a human's repertoire. The honest framing is that these are two bets rather than two products. One has a shorter path to a robot doing something in your building this quarter. The other has a better story about a robot handling situations nobody anticipated, and a longer wait before that story is testable on hardware.

What does a robot model actually do between a camera and a motor?

A robot model turns what the sensors show into what the joints should do next, and every argument about approaches is an argument about how that mapping gets learned. The model does not drive motors. It proposes a short sequence of motions, at its own pace, and something underneath turns proposals into motion that respects joint limits, stops on unexpected contact, and behaves sensibly when no new proposal arrives. That underneath layer is not optional and not interesting to write papers about, which is why it is missing from most of what you have read. The two bets differ in what the network is asked to learn. A vision-language-action model learns the mapping itself, copying the association between what a scene looked like and what a person did. A world model learns the dynamics — what follows what — and leaves the choosing to a planner that scores imagined outcomes. What a robot foundation model actually is covers the family both belong to, including the parts of the pipeline neither one replaces.

What are your options if you want a robot that decides for itself?

Six approaches are actually in use, and the layer they run on is a separate decision. You can fine-tune an open vision-language-action checkpoint on demonstrations you collect. You can train or adapt a world model and put a planner on top. You can skip learned policies and use classical perception with a motion planner, which is what most working robots still do. You can build a hybrid where a learned model chooses what to do and written code executes it. You can use a vision model for recognition only and script the behaviour. Or you can keep a person in the loop and let software assist. Underneath any of these sits the layer moving data between programs, which is usually ROS 2 for the drivers, simulation and tooling, or HORUS, an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages are not serialised between processes on one machine, which suits the arrangement where a Python model and a controller written in something faster sit on the same board. Both are Apache-2.0, and neither one trains a model or runs inference for you, so the choice of layer is genuinely separate from the choice of bet.

How do the two bets compare side by side?

The table mixes approaches with the layer they run on, because teams routinely confuse the two and then argue past each other. The first five rows are decisions about what your robot learns. The last two are decisions about what carries data between the programs on it.

What if you train models but have never shipped hardware?

Start with the robot rather than the model, because the gap in your knowledge is not where you expect it to be. The modelling half is the half you already have. The half that will consume your next two months is physical: mounting a camera so it does not move relative to the arm, calibrating so the model's idea of a position matches the arm's, knowing which joint states belong with which image, and resetting a scene between attempts without introducing a pattern the model will happily learn. Time alignment deserves special fear. If an image is paired with joint positions from a slightly different moment, a policy learns the offset as though it were part of the task, and the resulting robot reaches for where things used to be. Nothing in the training logs will show you this. The discipline worth importing on day one is to run the whole task once with written code before any learning is involved, because a team that cannot do the task deliberately cannot collect data teaching a model to do it.

What computer will the model run on?

The bet you can afford is the one that fits on the machine you are shipping. A large vision-language-action model wants a serious accelerator, and a world model with a search loop on top wants more than that, because planning means running the predictor many times before committing to anything. Three ways out exist and each has a cost. Put the model off-board and accept a network between thinking and acting, which forces you to define what the robot does while waiting. Distil to something smaller and accept that it handles fewer situations. Or split the work, letting the model advise at its own pace while a written controller drives at a steady one. The third is what most working robots do, and it changes the plumbing question: once perception and control are separate programs on one board, large camera frames start crossing between them constantly. What to run on a robot with a model on board works through that arrangement in detail.

How soon does the robot have to do something useful?

If a customer is expecting something this quarter, neither bet ships, and classical perception with a planner does. The uncomfortable truth about both approaches is that the demo is the easy part and the last stretch is where projects die, because a policy that works in your workspace meets a different table height, a different light, a shinier object, and degrades in ways you cannot patch by editing a file. Deadlines change what is rational. With months, fine-tune an existing checkpoint on one narrow task, constrain the workspace so the variation the policy must handle is variation you actually recorded, and keep a written fallback for when the policy declines to converge on anything. With a year or more and a research mandate, a world model is a defensible bet. With weeks, use the stacks that exist and put the learned part somewhere the robot can survive being wrong, which usually means recognition rather than motion. The teams that regret their choice are almost always the ones who picked the approach they wanted to be true and then discovered the calendar had a different opinion.

What do you need to know before either bet pays off?

You need to be able to run, reset and evaluate the robot reliably, which is a set of skills nobody teaches in a modelling course. Running means a rig where an operator can drive the robot through the task without fighting the software. Resetting means putting the scene back without accidentally teaching the model that objects always start in the same place. Evaluating is the one teams underestimate most: you need a way of deciding whether a change made things better that does not amount to watching one video and feeling encouraged. Trials have to be defined in advance, scenes have to vary deliberately, and somebody has to write down what counts as success before seeing the result. On top of that sit the ordinary robotics skills — frames, calibration, safety envelopes, and what the machine does when a program stops answering. Why simulation results keep failing to transfer covers the version of this problem that bites teams training in simulation first.

What does it look like when the model is right and the robot still fails?

The model chooses correctly and the robot misses anyway, which is the most common failure on a real machine and the one papers never show. The pattern is recognisable once you have seen it. The policy proposes a good grasp, and the gripper closes a moment after the object has been nudged aside. The arm reaches confidently for a position the mug occupied when the image was captured rather than now. The robot succeeds many times in the morning and starts failing after lunch, when the light through the window changed. A joint saturates, the controller quietly clips the command, and the motion that reaches the world is not the motion the policy asked for. None of these are modelling failures, and retraining will not touch them. They are timing, calibration and control problems wearing a model-shaped mask, and the way to tell is to replay the same episode with the policy replaced by a recorded sequence of motions. If the robot still fails, the model was never the problem.

What do you give up by committing to one of these bets?

You give up the ability to fix behaviour by editing code, which is a larger loss than it sounds. When a learned policy does the wrong thing, the repair is data: collect more, collect differently, retrain, evaluate, hope. That loop takes days where a written behaviour takes an afternoon, and it is not guaranteed to converge on the fix you wanted rather than on a new quirk. You also give up predictable motion, because a policy will occasionally propose something no engineer would have written, which matters when a person stands nearby. Choosing a world model gives up the near term entirely: the payoff is a research result, and the risk is that a model predicting beautiful video still plans badly, since predicting and choosing are different skills. Both bets commit you to data infrastructure, a fleet of robots that need to keep working, and people whose job is collection. That last cost is the one that surprises teams, because it never appears in a paper.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the robot's job is known and the difficulty is getting it done rather than learning it, which describes most robots that make money today. If you need navigation, manipulation planning, a driver for a specific arm or camera, or a simulation setup that matches your deployment, those exist in the ROS 2 ecosystem and reproducing them is work with no reward. If you are hiring, or handing a project to a team that maintains it afterwards, ROS 2 is the shared vocabulary. If your programs are spread across several computers, ROS 2 with a transport suited to that spread is the right structure. And ROS 2 sits comfortably underneath a learned policy, so choosing it is not a vote against either bet. HORUS is not the answer for a team that needs a navigation stack off the shelf, drivers for unusual hardware, or a stack the next three hires already know. The narrow case is one computer, large frames, and a Python model feeding a controller written in something faster.

Do these models remove the need for a control stack underneath?

No, and here is why: neither a vision-language-action model nor a world model commands a motor. Both produce proposals — a short sequence of desired motions, or a plan through an imagined future — and something has to convert a proposal into commands at the steady cadence a machine requires. That something enforces joint limits, smooths between proposals so motion is not a series of jerks, stops when a force sensor says stop, and decides what to do during the gaps when the model has not produced anything new. Those gaps are guaranteed: a large model does not answer on the same rhythm the arm needs, and the controller has to keep running regardless. Teams that skip this layer discover it the same way every time, when the robot freezes mid-motion waiting for a result and then lurches when one finally arrives. What sits between the model and the motor walks the whole chain, and it is a shorter chain than most people building models expect.

Are world models just simulators with a new name?

Partly, but not the way you think. Both a simulator and a world model answer the question of what happens if the robot does this, so the family resemblance is genuine. The difference is where the answer comes from. A simulator contains rules people wrote, which means it is inspectable, repeatable and wrong in ways you can name — contact is approximated, cloth is a nightmare, granular material is politely ignored. A world model learns from watching, which means it can represent things nobody wrote rules for, and can also produce a future that looks plausible frame by frame while being physically impossible, with no warning that it has left the territory it learned. That is why prediction quality and planning usefulness come apart, and why a model producing convincing video is not yet a model you can plan through. What each term is actually claiming untangles the marketing around this.

How do you decide which bet to make?

Answer three questions in order and the decision usually makes itself. Can a person demonstrate the task through a teleoperation rig, repeatedly, without hurting anyone? If yes, a vision-language-action model is the shorter path, and the work becomes collection discipline rather than architecture. If no, ask whether abundant observation of the task exists even without demonstrations, because that is the condition where learning dynamics and planning through them earns its cost. Then ask what happens if the model is wrong at the worst moment, and whether your machine can survive that answer. A robot whose worst case is a dropped object can carry a lot of research. A robot whose worst case involves a person cannot, and belongs in the hybrid arrangement where written code owns the motion. If none of the three questions has an answer yet, you do not have a modelling decision to make — you have a robot nobody has run end to end, and the next month belongs to the hardware.

Decide by situation rather than by allegiance:

• If a person can teleoperate the task and you have robots to collect on -> a vision-language-action model, because demonstrations are exactly the input it needs.
• If nobody can demonstrate the task but recordings of it are plentiful -> a world model with a planner, because predicting consequences is the only route left.
• If the task is defined and the deadline is real -> classical perception with a planner, because it ships and it can be debugged by reading code.
• If you need novel decisions but explainable motion -> a hybrid, with the learned part choosing and written code executing.
• If you have never run the robot end to end -> spend a month on the machine, because neither bet survives badly timed data.

When two options stay close, weigh them on the five axes of the HORUS Fit Framework: ecosystem size, setup effort, team size fit, deployment target, and licence. Take whichever loses on fewest — five plain questions about your team and your machine rather than about the software, with no scores attached. And if your robot is heading towards a model and a controller sharing one computer, star HORUS on GitHub so it is in your list when you start building.