VLA vs VLM vs LLM: The Robot AI Acronyms, Untangled

A large language model plans in text, a vision-language model answers questions about what a camera sees, and a vision-language-action model produces motion directly. The useful arrangement for nearly every robot is a language or vision-language model above control code you wrote, with ROS 2 or HORUS moving data between the parts. That flips only if you hold demonstration recordings and an arm matching a published action model. The rest of this post is for someone fluent in models who now has hardware on the bench and one month to spend well.

You can read a model card over a coffee. You know what a tokeniser is, you have fine-tuned something that mattered, and you can tell a real result from a cherry-picked one three pages into a paper. Then a robot arm arrives on your desk and the vocabulary stops behaving. One thread says the future is VLAs and everything else is a toy. Another says a VLM is plenty. A third says you only need an LLM with decent function calling, and all three are written as though the other two are obviously wrong.

The videos make it worse. Somebody says tidy the table, an arm does it, and the caption names an acronym without saying what else was running underneath, who wrote it, or how many takes there were. So you go looking for the checkpoint, find one, and discover it expects a particular arm with a particular gripper and a camera at a particular place on the wrist.

The arm on your bench has none of that. Your camera is on a tripod. And you have started to suspect that the acronym is not the part you are missing.

Which of these three belongs on your robot?

A language model or a vision-language model above actions you wrote, unless you already hold demonstration recordings of your own machine, in which case an action model can replace some of those actions. The acronyms sort themselves once you separate the two jobs inside any robot. One job decides what should happen next: the cup goes in the sink, so grasp it, lift it, carry it, release it. The other job produces motion and cannot pause to think, because it reads where the arm is, compares that with where it should be, nudges it closer, and refuses anything that would drive it through the table.

A language model is good at the first job, including the tedious parts, such as working out that the blue one means the mug or that a drawer opens before anything goes into it. A vision-language model does the same job with eyes. An action model is the only one of the three that does the second job, and it does it by having watched an enormous number of recordings of that job on hardware close to yours. When those recordings do not exist for your machine, the choice is already made for you.

What do the three acronyms actually stand for?

Large language model, vision-language model and vision-language-action model, and the difference that matters is what comes out rather than what goes in. Inputs are the part everyone quotes: text for the first, images and text for the second, images and an instruction and the robot's own joint positions for the third. Outputs are the part that decides your architecture.

Text coming out means something reads it before anything moves. You can log the plan, print it, check it against a rule, reject it, or hand it to a person for approval, and none of that costs you the robot. Motion coming out goes to the joints, and nobody reads it in time to object.

The family tree is real even where the marketing is loose. Many action models are a vision-language model with an action head attached and training on robot recordings layered on top, which is why capabilities like reading a label or understanding a preposition carry over. Phrases such as robotics foundation model almost always mean the third category. What none of the three means is a robot, a controller, or a driver.

What are your actual options for putting a model on a robot?

Six shapes cover nearly everything people build. A hosted chat model above named actions you wrote is the first and by far the most common. An open-weight vision-language model served on the robot is the second, chosen when the machine must work with no network. A published action checkpoint fine-tuned on your hardware is the third. A policy trained from your own demonstrations is the fourth, and it is a research programme rather than a weekend. Classical perception with no model in the loop is the fifth, and for a fixed repeated task it beats all of the above. A robot maker's own behaviour tool is the sixth, quickest to a first motion and hardest to leave.

None of those six is the layer that carries data between the camera process, the control loop and your Python. That layer is ROS 2, a vendor stack, or a middleware such as HORUS, an open-source Apache-2.0 project for Rust, Python and C++ in which all three languages share the same shared-memory ring buffers, so a camera frame is not serialised between processes on one machine. That project is validated in simulation, hands you no drivers or planners, and runs no models itself.

How do the options compare side by side?

Read the last two columns first, because most teams are eliminated into an option rather than argued into one. The hardware you own, the data you do not have, and the deadline someone already promised will cross out most of this table before taste gets a vote. The rows are not exclusive either: a hosted model choosing between actions, one of which is a learned skill, on top of a stack that came with your arm, is an ordinary and sensible system.

Are you doing research, shipping a product, or learning?

Researchers should follow the arrangement their field already argues about, product teams should choose what they can explain after a failure, and learners should choose whatever shortens the loop between a change and its effect. In research the acronym is part of the claim. If the papers you are answering evaluate action models on a standard arm with a standard task suite, using anything else means your numbers cannot be compared and your code cannot be run by a reviewer.

A product team has a different constraint, and it arrives the first time a robot knocks a customer's glass off a counter. Somebody asks why it did that. A plan written in text answers the question, because you kept the plan, the action it chose, and the reason the action refused or did not. A policy answers with a recording and a shrug. That pressure pushes most product teams towards a language layer over named actions, with any learned skill boxed inside one action that has its own limits around it.

A learner should optimise for staying interested, which usually means a small arm, a short action list, and one model above it.

What computer will the model actually run on?

The machine you own decides this before your preference does. A hosted model needs no graphics card at all and needs a network, so the robot waits for an answer that arrives over the internet, and nothing time-critical can be allowed to depend on that wait. An open-weight vision-language model needs a capable card with enough memory to hold the weights, and that card is often the same one your control loop shares a computer with.

An action model is the most demanding, because motion cannot wait for a data centre, so the weights have to sit on the robot with an accelerator to run them. That single fact removes the option for a large share of small builders, and no configuration flag argues with it.

Two practical notes. Keep the heavy model off the computer that runs the control loop when you possibly can, because a busy machine makes a loop late, and a late loop is how an arm overshoots. And a workstation on the same bench with a cable to the robot is a perfectly legitimate arrangement that people talk themselves out of for no reason.

How soon does the robot have to do something useful?

If the deadline is weeks away, put a hosted model above a handful of hand-written actions, and if it is quarters away, learned skills start to be worth their data. The reason is that the cost of an action model is almost never the training run. The cost is collecting demonstrations, which means a person teleoperating the arm through the same task hundreds of times, a rig to record it, and the discovery that a good share of the recordings are unusable because a hand blocked the camera or the object started somewhere the model will never see it again.

The short path is unglamorous and works. Write four or five actions, test each with no model anywhere near them, then let a model choose between them and put the chosen sequence on screen before it runs. You will have something demonstrable in days, and every action you wrote survives the day you swap the model for a better one.

The long path buys you skills nobody can hand-code, like reliably picking up something soft. It buys nothing else.

What do you need to know before any of this works?

Enough to write and test the actions the model will call, which is ordinary software and mechanical thinking rather than machine learning. Coordinate frames come first, because most early failures are a sign flipped or two frames confused, and the arm confidently reaching for a point behind itself. Units come second. Then the failure paths: what a grasp does when nothing is between the fingers, what happens when the object moves halfway through, and how the whole machine stops when a person says stop.

The awkward truth for someone arriving from models is that the unfamiliar part is not the model. It is everything underneath: the driver, the loop, the clock, the process that has to be running before another process makes sense. That layer is where the month goes, and it is worth reading a roadmap for connecting a model to a real machine before choosing an acronym.

None of that requires a background in control theory. It requires accepting that the interesting part is not the interesting part.

What do people try first, and why does it stop working?

Nearly everyone first hands a chat model direct control of joint positions inside a loop, and it stops working the moment the world moves. The failure is structural rather than a prompt problem. A model answers when it answers, and the arm keeps travelling during the wait, so every command is written for a world that has already changed. The model cannot feel the gripper slip, because nothing told it, and it will keep narrating a successful grasp of an object now lying on the floor.

The second attempt is a larger prompt: the whole scene, every object, the joint angles, a list of rules. That produces longer, more confident nonsense, and eventually a state machine appears anyway, badly, written in English inside a system prompt where nobody can test it.

The third attempt is the one that works, and it is the boring one. Write the actions in code, give each a precondition and a stopping rule, and let the model pick and order them. The reasons this arrangement works are the same ones that explain why a model that writes software cannot pick up a cup.

What do you give up by choosing a vision-language-action model?

You give up explanation, and you take on a data pipeline that never really ends. When a policy does the wrong thing, there is no sentence to read. There is a recording, a hunch, and the standard remedy of collecting more demonstrations of the situation that went wrong, which is slow and cannot be scheduled with any confidence.

You also give up portability, more completely than people expect. A policy is bound to the body it learned on: link lengths, gripper shape, where the wrist camera points, even the lighting in the room where the recordings were made. Move the camera a few degrees and behaviour changes in ways nobody can predict from the weights.

And you give up cheap rules. With named actions, a new constraint such as never reach across the laptop is an afternoon of work and a test that proves it. With a policy, the same constraint is a data collection campaign, or a supervisor written in ordinary code that sits above the policy and overrides it, at which point you have quietly rebuilt the thing the policy was supposed to replace.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the robot has to move through a building, perceive its surroundings, and be worked on by more than one person, and HORUS is not the answer for a team whose real gap is that there are no drivers, no map and no arm planner yet. If you are choosing between writing a motion planner and adopting one, adopt. That code carries years of work and has run on far more robots than yours will.

ROS 2 also wins in this particular corner of robotics for a reason people underestimate. Almost all of the practical machinery around action models assumes it: the arm drivers, the teleoperation rigs used to collect demonstrations, the dataset formats, the evaluation harnesses in the papers. Choosing differently means porting that machinery yourself before the interesting work starts.

And it wins on recording and replay. Capturing a run and playing it back at a desk afterwards changes how quickly a team understands a robot, and when the thing you are debugging is a policy, a recorded run is often the only evidence you have.

Does a vision-language-action model remove the need for control code underneath?

No, and here is why: a policy emits a stream of targets, and something still has to enforce joint limits, hold a stop when a person presses the button, and refuse the confident command that would put the elbow through a monitor. A policy trained by imitation has no concept of a limit it never saw violated in the recordings, and confidence is not caution.

There is a timing half to this as well. A policy expects its inputs at the rhythm it was trained on, and a stale camera frame does not make the policy a little worse, it makes the policy behave like a different policy, one that is acting on a scene that has moved on. Keeping frames fresh across processes is plumbing work, and it belongs to you regardless of which model you chose.

Then there is everything the policy was never asked about: battery state, temperature, a motor that has started drawing more current than usual, logging that lets you reconstruct what happened, and the clean shutdown that leaves the arm somewhere safe.

Will a larger model fix a robot that keeps grabbing the wrong object?

Partly, but not the way you think: a larger model fixes reasoning mistakes, and most wrong-object grabs are not reasoning mistakes. They are a correct decision applied to a picture of where things used to be, or to a scene description that was assembled from three frames taken at different moments and quietly stitched together as though they agreed.

The two causes have different fingerprints. Freeze the scene, run the same instruction repeatedly, and see what happens. If the robot reaches for the same wrong object every time, the problem is grounding, and a better model or a better prompt genuinely helps. If it succeeds most times and fails when the machine is busy, when another program starts, or when someone walks past, the model was never the problem, and the reason a robot behaves differently on every run is the piece worth reading next.

That test costs an afternoon and routinely saves a quarter of misdirected work, because the second failure pattern will follow you to every model you upgrade to.

How do you decide which one you need?

Write down the one sentence describing what the robot must do next month, then pick the smallest arrangement that could produce it. Not the most advanced one. Almost every regretted choice in this area comes from picking a category for a problem the project did not have yet, and paying the data cost to find out.

• If you want judgement between skills you can already perform -> a language model above named actions, because the model arranges capabilities and cannot invent them.
• If the robot must describe or find things without a network -> a vision-language model on the machine, because eyes offline are the whole requirement.
• If contact is the hard part and you can record demonstrations for months -> an action model, because no prompt teaches a gripper what soft feels like.
• If the task repeats with barely any variation -> classical perception, because a model is an expensive way to solve a solved problem.
• If the robot works alone but the programs stop agreeing with each other -> fix the layer underneath first, because no acronym above it will hold.

The HORUS Fit Framework reduces the layer beneath the model to five axes you can judge without running a benchmark: ecosystem size, setup effort, team size fit, deployment target and licence. For model-driven robots, deployment target and setup effort usually decide it, and licence matters more than expected the day a client asks whether the work can be handed over.

The week your plan is right and the arm still misses is a bad week to start reading about middleware. Put HORUS on the shelf before then: star it so it is in your list when you start building.