Are Vision-Language-Action Models Replacing Classic Robot Software?

No. Vision-language-action models are replacing the hand-written task and grasp logic, not the drivers, safety limits, control loops and messaging underneath them. A policy decides what motion to make; something else still gets camera frames to it on time and stops the arm before it hits the table. That second job stays with ROS 2, a vendor kit or HORUS, and it collapses into the model only on a bench rig running one script. The rest of this post is for someone who trains policies and is deciding how much classic robot software they still need to learn.

You have a checkpoint that works. In the evaluation harness it picks up the block, and it does it often enough that you started telling people about it. Then you put it on the arm on your desk and the same policy behaves like a different piece of software. It hesitates. It commits to a motion a beat after the moment it should have. The gripper closes on air, because the scene it was reasoning about has already moved on.

You go looking for advice, and the answers split into two camps that do not talk to each other. One camp says the entire robotics software tradition is about to be obsolete, that the model is the controller now, and that planners and state machines are legacy furniture. The other camp says nothing has changed, go and learn transforms and lifecycle nodes like everyone before you.

Neither camp describes your afternoon. Your afternoon is a policy that seems fine and a robot that is not, and no obvious way to work out which of those two stories applies to you.

Are vision-language-action models replacing classic robot software?

They are replacing one layer of it and leaving the rest standing. The layer they replace is the part people were always least happy writing: the hand-coded sequence that says approach, align, close the gripper, lift, plus the pile of special cases underneath it for when the object sits at an angle nobody planned for. A policy trained on demonstrations does that better than a state machine, and without anyone enumerating the cases.

Everything else survives. Somebody still has to talk to the motors, read the encoders, and notice the emergency stop before the next command goes out. Somebody still has to decide which program runs where, move camera frames from the driver to the model, and get the model's output to the joints while it still matches the world.

None of that is a research problem, and all of it is work. So the honest version of the headline is that one job in the stack has changed hands. People announcing the end of robot software are usually describing a bench where the rest of the stack is one script long.

What is a vision-language-action model in plain terms?

A vision-language-action model takes camera images and a written instruction and produces robot motion directly, rather than producing a description of the motion. That last part is the whole difference. A language model asked to tidy a table will write you a tidy plan. A vision-language-action model asked the same thing outputs the next set of joint positions, then the next, over and over, while watching what its own motion did to the scene.

The training data is human demonstrations: somebody teleoperating the robot through the task, many times, in many arrangements. The model learns the mapping from what the camera sees to what the human did next. It is imitation at scale, with language attached so you can say which learned behaviour you want.

That explains both the strengths and the limits. These models are good at contact and at recovering from small mistakes, because the demonstrators were. They are poor at anything nobody demonstrated, and they inherit the pace of the person they copied. The surrounding acronyms get muddled constantly, and the untangled version is here.

What are the actual options for the software around the model?

There are about seven, and most teams use two at once without having decided to. You can wrap the policy in a single Python script that owns the camera, the model and the arm, which is how nearly everyone starts. You can adopt a policy runtime and dataset toolkit, the kind that ships with recording tools, training scripts and a handful of supported arms. You can build on ROS 2, which brings drivers, coordinate frames, visualisation and a very large number of people who have already hit your problem. You can use the arm vendor's own SDK and live inside its boundaries. You can write the control layer yourself in C++. You can stay in simulation. Or you can put a middleware underneath whichever of those you picked.

That last option is what HORUS is: an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages between programs on one machine are not serialised. Apache-2.0, validated in simulation. It trains nothing and simulates nothing. It is wiring, not brain, and answers a narrower question than ROS 2 does.

How do the options compare side by side?

Read the last column first. Almost nobody is argued into a choice here; they are eliminated into one, and the quickest route is finding the rows already impossible for you.

Should you learn classic robot software if you only want to train policies?

Yes, and less of it than you fear. You do not need inverse kinematics from first principles or a derivation of the Kalman filter, and you can go a long way without writing a motion planner. Those are specialities, and the field has libraries for them.

What you do need is the part nobody puts in a course. How programs on one machine hand data to each other, and what happens when one of them is late. What a control loop is, and why it must never wait for anything slow. Why a camera frame has an age, and why that age matters more than its resolution. What an emergency stop does to software that is halfway through a motion.

That is a week of reading and a month of being surprised, not a degree. The payoff is direct: it is the difference between knowing your policy is fine and guessing. Teams that skip it spend months retraining a model to fix something that was never in the model. What sits between the model and the motor walks through that middle ground.

What hardware does a robot running a policy actually need?

More than a laptop and less than people assume. The model needs an accelerator on board, because sending frames to a data centre and waiting for joint commands back does not hold up for anything that touches the world. In practice that means one of the small compute modules built for this job, or a mini PC with a discrete card, plus a power budget you have to plan for if the machine moves.

The rest of the machine matters more than the accelerator does. Cameras that deliver frames at a steady rate rather than in bursts. An arm whose controller accepts commands at the rate the policy produces them. A wrist camera, because most published policies were trained with one and the view they learned from is the wrist's view. And a physical stop that cuts power without asking software for permission.

The part that surprises people is that a busy accelerator is itself a problem. When the model saturates the machine, the camera driver and the loop watching the joints compete with it for the same computer, and that contest stays invisible until the arm starts arriving late.

How long before a policy is driving real hardware?

Faster than you expect to a first success, and much slower than you expect to something you would leave running. Getting a published checkpoint to move an arm the toolkit already supports is a weekend, sometimes an evening. That first success is real, and it is also the least informative moment of the entire project.

The long part is everything after it. Collecting your own demonstrations, because a policy trained on somebody else's setup rarely transfers to your table and your objects. Then fine-tuning. Then discovering that the failures cluster: the same object, the same corner of the workspace, the same hour when the sun comes through the window.

Then the part nobody schedules, which is turning a demonstration into something that runs unattended. What happens when the gripper is holding something and the model produces nonsense, when a person walks into the workspace, when the camera driver drops out for a moment and comes back.

Budget a weekend for the demonstration and a season for the behaviour. Teams that plan for the first and not the second are the ones whose progress updates go quiet.

What if you know machine learning and almost nothing about robots?

You are in a better position than the reverse, and you will be surprised in specific ways. Everything you know about data, training and evaluation transfers directly, and it is the scarcer half of this skill set. What does not transfer is the assumption that the world holds still while you think.

The first surprise is that there is no epoch. The robot is in a state your last output caused, and a decision made a moment too late is not a slightly worse decision, it is a wrong one about a scene that no longer exists.

The second is that failure is not a metric going down. It is a physical event: something falls, something bends, somebody resets the table before the next attempt.

The third is that the evaluation you trust is the one you built, and it flatters you. Success in a harness with reset scripts and fixed object placement predicts little about the same policy on a bench somebody has worked at all day.

None of that is hard to learn. It is simply not in the papers.

Why does the first setup people build stop working?

Because the single script that made the first success possible is exactly the thing that fails second. The starting shape is always the same: one Python program opens the camera, runs the model, and writes joint commands, all in one loop, and it works because there is only one of everything.

Then you add a second camera. Then a safety check that has to run whether or not the model is thinking. Then a logger, because you need to know what happened. Then a small web page so a colleague can watch. Each addition is reasonable, each lands in the same loop, and the loop now waits for whatever is slowest in it.

The symptom is not a crash. It is the arm arriving late, intermittently, in a way that looks exactly like the model getting worse. So people retrain, the retrained model behaves the same, and two months have gone.

The tell is simple. If the policy scores the same offline as it did before and the robot behaves worse, the fault is between your programs, not inside the network.

What do you give up by building the robot around a policy?

You give up being able to explain why it did that. A planner that fails leaves a trail: the goal was here, the path was blocked, the check refused. A policy that fails gives you a video and a shrug. On a research bench that is tolerable. On a machine that works near people it is a real cost, and it surfaces in safety review rather than in development.

You also give up the ability to fix one specific behaviour. You cannot patch a bug into a network; you collect more demonstrations of the thing it got wrong, retrain, and hope. That loop takes days, against the minutes it takes to change a line in a state machine.

And you inherit a data obligation that does not end. Every new object, gripper, lighting condition and camera position is potentially a new collection campaign. The model is not software you finish; it is something you keep feeding.

Those are genuine trades, and most teams accept them anyway, because the alternative is enumerating cases that nobody can enumerate.

When is ROS 2 the better choice?

Often, and for most robots that do more than one thing. If your machine has to navigate, if it carries a lidar, if it needs coordinate frames between a base and an arm and a camera, if you want visualisation you did not write, if you need drivers for hardware you did not build, then ROS 2 is the answer and the argument is not close. Decades of other people's work is sitting there for free.

ROS 2 is also right when the team will grow, when you intend to hire, and wherever being inside an ecosystem beats owning the last detail of how messages move.

HORUS is not the answer to any of that. There are no drivers, no navigation, no planners, and no community the size of a conference behind it, and it does not try to replace ROS 2 for a robot that mostly needs what ROS 2 already has. The narrower project earns a look only once you have a specific, repeatable problem with how your own programs pass data on one machine.

Will a bigger model eventually absorb the rest of the stack?

No, and here is why. The jobs left over are not jobs a model is bad at; they are jobs that cannot be done by anything that thinks in variable time. A safety limit has to fire on the cycle it is needed, every cycle, including the cycle where the model happens to be busy. A motor controller has to produce a command on a fixed heartbeat whether or not anything upstream is ready. Those are guarantees, and something that occasionally takes longer than usual cannot make one.

There is a second reason, which is that hardware does not go away. Something has to speak the protocol the arm actually speaks, cope with the encoder that reports nonsense once an hour, and reconnect the camera when a cable gets nudged. Nobody is training a model to do that.

What will happen, and is already happening, is that the layer above the model gets thinner. Task planners, behaviour trees and hand-written sequences are being eaten from above. The layer below is not moving, and why demonstrations flatter these systems is the same story from the other end.

Does a policy that outputs joint commands remove the need for a control loop?

Partly, but not the way you think. It is true that a policy producing joint targets has absorbed a job that used to belong to a planner and a trajectory generator, and that is a genuine simplification: fewer components, fewer interfaces, less tuning.

What has not moved is the loop below that. Something is still closing the loop around each joint, comparing where the joint is against where it was told to be, and doing that whether or not a new target has arrived. That loop lives in the arm's controller or in your own code, and it runs far more often than the policy produces anything.

The dangerous version of the misunderstanding is believing the policy's output rate is the robot's control rate. It is not. The policy sets goals; the loop chases them. When the policy is late, the loop holds the last goal, which is why a late policy reads as hesitation rather than a crash.

Understanding that split is most of what separates people who debug this in an afternoon from people who retrain for a month.

How should you decide what to build on?

Decide by naming the thing blocking you right now, then picking the smallest option that removes it. Most wrong choices here come from picking for a problem the project does not have yet.

• If you have a checkpoint and an arm on a desk -> one Python script, because anything else is ceremony before you know the policy moves metal.
• If your bottleneck is demonstrations -> a policy runtime and dataset toolkit, because collection and training are the actual work.
• If the robot must navigate, sense and manipulate -> ROS 2, because the drivers, frames and planners already exist.
• If you are selling the machine -> check the licence of every layer early, because that conversation gets expensive late.
• If the policy scores the same and the robot got worse -> look at how your programs pass data, because the fault is between them.

The HORUS Fit Framework reduces the stack question to five things you can judge without a benchmark: ecosystem size, setup effort, team size fit, deployment target and licence. For policy-driven robots, deployment target and team size fit usually settle it. What the word autonomous is doing in all this is covered in what autonomy actually means in marketing.

The week your model is fine and the arm still hesitates is a bad week to start reading about the layer underneath. Put HORUS on the shelf before then: star it so it is in your list when you start building.