What Is a Robot Foundation Model?

A robot foundation model is one large pretrained model you adapt to a task, instead of training a policy per job or hand-writing the motion. A foundation model decides what to do next and nothing else; reading sensors on a rhythm and stopping the machine belong to a stack such as ROS 2 or HORUS underneath. The verdict flips when the task is one repeatable motion in a fixed cell. The rest of this post is for people who work with models and are now deciding what has to exist beneath one on real hardware.

You saw the demonstration everybody saw. One model, several different robots, somebody asking for a coffee in ordinary English, an arm picking up a mug it had apparently never met. So you pulled a checkpoint, recorded an afternoon of teleoperated demonstrations on your own arm, fine-tuned, and watched it work. Sometimes.

It lifts the cup. Then it reaches past the cup. Then it closes the gripper on air and carries on as though something were in there. You move a lamp and the whole thing gets worse, and you cannot tell whether the model is confused, the camera changed its own exposure, or a command arrived at the arm late. Nothing throws. The log says the policy ran, every cycle, exactly as asked.

Then the questions arrive that you have been quietly postponing. What happens when a frame does not turn up? What stops the arm when somebody leans into the workspace? Does the machine do the same thing tomorrow afternoon, in different light, with nobody standing beside it ready to restart the script?

Should you build your robot on a foundation model?

Use a foundation model for the part of the job you cannot write down, and never let it be the only thing standing between the machine and a person. That is the entire decision, and it is smaller than the discussion around it suggests. A robot foundation model is a source of behaviour. It answers what to do next when the situation is unfamiliar, and it answers that far better than the state machine you would have written by hand. What it does not do is read an encoder on a beat, notice that camera frames stopped arriving, hold a joint steady while something heavier than expected pulls on it, or halt when the cell door opens. Those jobs do not vanish as models improve. They get harder, because a machine trusted to improvise ends up in rooms a scripted machine was never allowed near. So the decision splits cleanly in two: choose where behaviour comes from, and separately choose what the machine runs on. Teams that treat those as one question lose months to the confusion.

What is a robot foundation model in plain terms?

A robot foundation model is one large model, pretrained across many robots and many tasks, that you adapt to your machine instead of training something new for every job. The shape is familiar if you have worked with language models. Pretraining happens on somebody else's enormous pile of data, made of recorded robot episodes from many labs and many arm types, usually mixed with images and text from the web. Out of that comes a model that already knows what a mug is, what a drawer does, and roughly how a gripper should approach a handle. You then adapt it, with a modest set of demonstrations recorded on your own hardware, or sometimes with nothing but a written instruction. The input is typically camera frames plus a sentence. The output is actions, emitted again and again as the scene changes: joint targets, gripper poses, small corrections. That output is what separates a robot foundation model from a chat model with a webcam attached, and the acronyms in this area are worth untangling once.

What are your actual options for getting a robot to do a task?

You have seven realistic ways to make a machine do useful work, and they differ mostly in how much behaviour you write yourself versus inherit. You can write the motion by hand, as rules and waypoints. You can train a small policy on your own demonstrations for that one task. You can adapt an open robot foundation model checkpoint. You can call a hosted model over a network and let it drive a vendor arm. You can buy a robot whose behaviour ships with it. You can stay in simulation. Underneath any of those, the machine is either one program or several that must agree, and that is where middleware lives: ROS 2 brings the largest body of existing robotics code anywhere, from navigation to sensor drivers, while HORUS is an open-source real-time robotics middleware for Rust, Python and C++ in which the three languages share the same shared-memory ring buffers, so messages between processes on one machine are not serialised. Neither one runs your model. Read the table as descriptions of situations rather than a ranking.

What should you pick if you come from machine learning?

Buy an arm you did not build, adapt an open checkpoint on it, and give the layer underneath its own place in the calendar rather than treating it as glue. Somebody arriving from machine learning has one genuine advantage and one predictable blind spot. The advantage is real and undervalued: data collection, evaluation discipline and knowing when a result is a fluke are exactly the habits a physical system will demand. The blind spot is assuming that everything below the policy is a driver call away. It is not. It is scheduling, freshness checks, a stop that works while inference is running, and the plumbing that gets a camera frame from one process into another without either waiting on the other. The failure has a signature: the machine works while you watch it, because you restart the script, you reseat the cable, you rerun after a dropped frame. What you have not built is a machine that behaves identically when nobody is in the room. Fix the boundary between the process that thinks and the process that moves early, and you can swap models endlessly without touching the robot.

What hardware does a foundation-model robot actually need?

A foundation-model robot needs an accelerator that can hold the model, a machine that fails limp rather than stiff, and cameras that do not change their own settings behind your back. Take those in order. The accelerator sets your whole architecture: if the model fits onboard, you get one computer and a scheduling problem; if it does not, you get two computers and a network problem, and you must decide what the arm does when the link stalls. The machine itself matters more than people expect. You want something that halts or goes slack on command, has a physical cut that no software can override, and lacks the reach or force to hurt anyone during the long months when your policy will be wrong. That single requirement rules out a lot of impressive hardware. Then the cameras. Automatic exposure and automatic white balance will teach your model about the afternoon sun rather than about the task, and the symptom is a policy that works before lunch and not after. Fix the optics before blaming the checkpoint.

What if you need something working next month?

Skip the foundation model entirely and script the motion, because a month buys a working behaviour and does not buy a learned one. This is unpopular advice and it is nearly always right. Adapting a checkpoint means a teleoperation rig, a demonstration session, a training run, an evaluation protocol and at least one round of discovering that your data was collected in a way that does not generalise. Each of those is a week when it goes well. The version that ships on time is a vendor arm with a supported interface, motion written as waypoints, a simple perception step for wherever the object actually is, and a hard-wired stop within reach of whoever is standing there. If the demonstration must look adaptive, put the model somewhere it can be slow and reversible, such as choosing which of three scripted routines to run. That gets you variety without putting a learned policy directly on the motors. Then, after the deadline, take the time you did not have and do the learned version properly.

What if you have never written code that has to keep a rhythm?

Write a small loop that runs on a beat before your policy needs one, because the idea takes an afternoon to learn and takes months to learn the hard way. A control loop reads where things are, works out the difference from where they should be, sends a correction, waits, and repeats, and it has to hold that beat while everything else on the machine is busy. That last clause is the whole difficulty. A loop cannot wait on a log line being flushed, a file being written, or a model finishing a thought. The pattern that works with learned behaviour is a division of labour: the policy publishes a target whenever it finishes thinking, and a separate loop underneath drives smoothly towards the most recent target it has seen. A robot built that way moves like a machine. A robot whose motors receive commands only when inference happens to finish moves in lurches, and no amount of retraining fixes that, because the problem is not in the model at all.

What does it look like when a foundation model goes wrong on real hardware?

It does not crash, it commits confidently to the wrong action, and that is what makes learned behaviour hard to debug on a machine. A conventional bug announces itself: an exception, a stack trace, a value out of range. A policy failure looks exactly like a policy success from inside the software. The arm moves smoothly to a place nothing is. The gripper closes on nothing and the next step proceeds as though the object were held. A drawer gets pushed instead of pulled with the same easy confidence as a correct grasp. Worse, the same failure can come from three unrelated places, and they look identical from the outside: the model genuinely misread the scene, the camera handed over a stale frame, or the command reached the joint after the moment it was meant for. Teams burn weeks retraining a policy whose real problem was that frames were arriving from a queue that had quietly backed up. Build the plumbing so those three causes look different, or you will diagnose all of them as the model.

What do people try first, and why does it stop working?

Almost everyone starts with one Python script that reads the camera, runs the model and sends joint commands in a single loop, and that works until the model is big enough to matter. The break happens in a specific order, and you can predict it. First the loop rate becomes whatever inference takes, so the arm moves in steps rather than a motion. Then you add threads, and now the camera thread and the inference thread are copying large frames between them, and the copying costs more than anybody budgeted. Then somebody needs a compiled language for the part that talks to the motors, and suddenly one process must become two, and the two must share images without one blocking the other. Then a safety review asks what stops the machine while Python is in the middle of a forward pass, and the honest answer is nothing does. That progression is why projects outgrow their first framework at almost exactly the moment the demo starts working.

What do you give up by building on a foundation model?

You give up the ability to explain one specific bad afternoon, and everything else follows from that. When hand-written motion drives into a table, someone reads the code and finds the wrong sign, and the fix takes an hour. When a policy drives into a table, the fix is more data, a different prompt or another training run, and the gap between noticing and correcting stretches from minutes into days. You give up repeatability in the small: the same machine in the same room with a slightly different starting pose can make a different attempt. You give up cheap safety arguments, because pointing at a rule persuades a reviewer and saying you tested a great deal does not. You take on evaluation as genuine infrastructure, since somebody has to answer whether the robot is better this week than last. And you take a dependency on a checkpoint you did not train, whose licence, size and future you do not control. None of this argues against learned behaviour. It argues for keeping the parts that must never surprise anyone outside the learned part.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the robot has to move through a building, and whenever most of the machine's value is code that already exists. Mapping and navigation are the clearest case by a distance: a mobile base that must build a map, localise in it and drive to a goal is a ROS 2 project, because those packages represent years of work you will not reproduce alongside your model work. ROS 2 also wins when your sensor ships a ROS 2 driver and nothing else, when the system spans more than one computer, when a safety or integration partner expects the ecosystem, and when the lab around you already speaks that vocabulary. HORUS is not the answer for those projects, and choosing it there means rebuilding plumbing you could have inherited for nothing. There is a second case worth naming plainly: if the learned policy is one step inside an otherwise conventional pipeline, ROS 2 already has a natural place to put that step, and you should use it.

Will a robot foundation model remove the need for the software under it?

No, and here is why: a better model removes hand-written behaviour from the middle of the stack, and adds nothing at all to the bottom of it. The trend is real and worth taking seriously. Perception pipelines that occupied a team for a year now fall out of a checkpoint. Task sequencing that filled a large state machine collapses into an instruction and a handful of skills. That will continue, and a great deal of tedious robotics work is genuinely going away. What does not shrink is everything beneath the decision. A stronger model does not read an encoder, does not decide what happens when frames stop arriving, does not hold a joint steady against an unexpected load, does not stop the arm when someone opens the door, and does not make a message arrive before the next cycle needs it. If anything the demands underneath rise, because a machine trusted to improvise gets deployed where a scripted machine never was, and the cost of a missed reading rises with it. The middle is being eaten. The floor is not.

Is a robot foundation model the same thing as a vision-language-action model?

Partly, but not the way you think. Most robot foundation models you can actually download today are vision-language-action models, so in practice the terms often point at the same file. The words mean different things though, and the difference matters when you are reading claims. Vision-language-action describes an architecture: images and language go in, actions come out. Foundation model describes a role: pretrained broadly, meant to be adapted, expected to serve many downstream jobs rather than one. A model can be one without the other. A small policy that takes images and text and emits actions is a vision-language-action model and is nobody's foundation. A large world model that predicts video and is adapted for planning is playing the foundation role without ever emitting an action directly. The useful question when someone shows you a model is not which label applies, but what was pretrained, on whose data, and how much of your own hardware it has ever seen. What survives the move from simulation to a real room is the same question in different clothes.

How do you decide whether your problem needs a foundation model?

Write down the behaviours you cannot describe as rules, and the length of that list is your answer. The exercise takes ten minutes and ends most arguments. If the list is short, and mostly reads like handling variation in where an object sits, you have an automation problem wearing a model as a hat, and waypoints plus a decent perception step will serve you better and more predictably. If the list is long, full of things like recognising a container it has never seen or working out that a lid is on backwards, then you have a genuine learning problem, and your bottlenecks will be data collection and evaluation rather than architecture. Now run the exercise the other way. Imagine a perfect policy that always emits exactly the right action. Does the machine still need to read sensors on a beat? Yes. Still need to stop when a person leans in? Yes. Still need two programs on one board to share a camera frame without either waiting? Almost certainly. Nothing on that list improves when your model does, and that list is your robotics project.

Decide by situation rather than by ambition:

• If your task is the same motion in the same place every time -> hand-written motion, because rules you can read beat a policy you cannot.
• If one narrow job resists description but you can demonstrate it -> a small task-specific policy, because your own data will beat borrowed breadth on narrow work.
• If instructions and objects vary constantly -> an adapted foundation model checkpoint, because that variety is exactly what pretraining bought you.
• If the robot must map a building and navigate to a goal -> ROS 2, because navigation is inherited code and rewriting it is not your project.
• If a Python policy and a compiled control loop live on one board -> a shared-memory middleware, because moving camera frames between them is what costs you the motion.
• If a date is a month away -> script it, because a month buys a behaviour and not a robot.

When two options stay close, weigh them on the five axes of the HORUS Fit Framework: ecosystem size, setup effort, team size fit, deployment target, and licence. Take the one that loses on the fewest: five honest questions about your situation rather than scores about the software. If your project keeps landing on one machine with a policy in Python and a loop that must not wait for it, star HORUS on GitHub so it is in your list when you start building.