How Do You Connect an LLM to a Real Robot? A Beginner's Roadmap

Wire the language model in as a slow planner above a control layer that already moves the robot safely, with ROS 2 or HORUS underneath. The model chooses which tested behaviour to run next, while the code keeping the wheels straight never waits for a sentence. That flips only when the robot's whole job is conversation and gesture, where one script calling a hosted model is the entire project. The rest of this post is for someone who has shipped model-driven software and is now holding a robot arm, wondering what goes between them.

The demo works on your laptop. You describe a task in a sentence, the model produces a plan, the plan calls your functions, and the printed result reads like something a competent assistant would say. Then you point the same code at an arm on a desk, and everything you were sure about turns back into a question.

The plan mentions a mug. Which mug, where is it now, and is it still there after the model finished thinking? Your library has a pick-up function and a move-to function, and neither knows what to do when the gripper is already holding something. The arm runs the first step, something slips, and the model carries on with step two as though the world still matched the description you handed it a moment ago.

Then the failures blur together. The robot does the wrong thing and you cannot say whether the model chose badly, the driver answered late, the calibration drifted, or a cable is loose. Every demo video online cuts before this part. Nobody films the layer underneath, and the layer underneath is the job.

How do you actually connect a language model to a robot?

You connect a language model through a layer of actions the robot already performs on its own, and the model only chooses among them. Three pieces sit between the sentence and the motors. The first is a small set of behaviours that each work when nobody is watching — go to the charger, look at the table, close the gripper, stop — every one tested alone and able to report whether it finished or failed. The second is a description of the world in words the model can read, written by your perception code rather than by the model. The third is a supervisor beneath the model that refuses anything outside the limits, whatever the sentence said. The model's job is to pick the next behaviour and fill in a blank or two. The job it must never hold is producing motor commands, because its answer arrives at unpredictable moments and is occasionally wrong in a way that sounds right. A control loop tolerates neither property, and what a control loop is doing between decisions explains why.

What does connecting a language model to a robot actually mean?

Connecting a language model to a robot means giving the model a vocabulary of things the robot can already do, a description of what the robot can see, and permission to pick one. Nothing in that resembles wiring a chip to a board. The model never touches hardware, never sees a voltage, never runs on a schedule. Picture a dispatcher and a driver. The dispatcher knows the goal, reads the situation report and says which stop comes next; the driver knows the road and the brakes and would refuse an instruction that put the van through a wall. Your robot needs the driver to exist before the dispatcher is worth hiring, and most disappointment in this field comes from teams who hired an articulate dispatcher for a vehicle with nobody at the wheel. In practice the vocabulary is a handful of named actions with a few parameters each, the situation report is a short paragraph your own code writes from sensor data, and the permission is a checked list of what the model is allowed to ask for.

What are the real options for putting a model on a robot?

There are about eight arrangements people actually ship, and they differ mainly in where the model runs and how much the model is allowed to decide. At the simple end, a hosted model talks over the network to a small command server on the robot exposing three or four actions. In the middle sit the arrangements that borrow an existing stack: ROS 2 already provides drivers, navigation and motion planning, with actions that report progress and can be cancelled, which is the exact shape a planner above them needs, while HORUS, an open-source real-time middleware for Rust, Python and C++, suits the case where the model process, the perception code and the control loop all live on one machine and should share the same ring buffers rather than copy data across a language boundary. At the far end, an end-to-end policy learns the motion itself. Read the table as a set of situations rather than a ranking, and notice how several rows describe the same person in different months of one project.

What should you do if you are an ML engineer who has never built a robot?

Buy or borrow a machine that already works without you, then spend a fortnight making it do one dull task the same way every time from a plain script. Your existing skills transfer better than roboticists admit: designing evaluations, sorting failures into categories, spotting a model that is confidently wrong, and distrusting a metric that only looks good on examples you chose yourself. What does not transfer is physical. Objects move while you are thinking. Grippers slip on the same object they held ten minutes earlier. Calibration drifts over an afternoon, so a plan that was right is now aimed slightly beside the target. A retry in software is free; a retry on hardware knocks something over. So do not begin by building an arm, and do not begin with the planner. Begin with the smallest embodiment you can get running, such as a desktop arm or a small mobile base with a supported driver, and keep a log of every way the dull task failed. That log is the specification for everything the model layer will later have to survive, and if you are still choosing what to write it in, the first-language question has its own answer.

What hardware does this need, and what should run where?

You need two computers' worth of work even when both halves live on one board: something that keeps the robot steady and is never surprised, and something that thinks. The steady half wants a modest computer running the loop, the driver and the limits, carrying nothing that can pause unexpectedly. The thinking half wants either a network connection to a hosted model or an accelerator holding a smaller local one. Where things run follows from one rule: anything that must happen whether or not the model answers stays on the robot. Motor control, limits, the stop path and the code noticing an empty gripper all live below, always. Perception can sit on either side, though shipping every frame across a network becomes expensive and fragile quickly. Planning can be remote. The test that settles arguments is to unplug the network mid-task. A robot split properly finishes the current action, notices nothing is coming, and stops in a state you would happily walk up to. A robot split badly carries on executing a plan for a world that has gone.

How long does this take if you want something working by next quarter?

A convincing demonstration takes a few weekends; a robot you would leave running with someone else in the room is a different order of work. The demonstration is quick because the pieces exist — a hosted model, a couple of actions, a microphone if you want one — and the first time a spoken sentence turns into motion, the thing feels finished. It is not, and the gap is not in the model layer. The long work is making each action succeed or fail cleanly and say which, because a planner handed the word error and nothing else will retry from a position it does not understand. Budget the failure handling, not the demo. A rough order that holds up: one action driven from a script this week, a second action and a refusal path next, the model choosing between two actions in simulation after that, then hardware with a person on the stop button. Teams who reverse the order get a video in a fortnight and spend two quarters discovering what the video hid.

What do you need to understand before you start?

Enough Python to write a small service, plus one honest mental model of what the control loop is doing while the model thinks. You do not need to write drivers, derive kinematics, or learn C++ in the first month. You do need four things. First, how an action is invoked and how you learn it finished, because everything above rests on that answer being trustworthy. Second, what happens when an action stops halfway, since a half-open gripper is a state your planner will meet. Third, enough about coordinate frames to know what the word left means to the robot as opposed to the camera. Fourth, the habit of treating model output as untrusted input. The instinct that transfers worst from machine-learning work is that a retry costs nothing; on hardware a retry moves mass. If you are wondering whether the layer above the loop should be Python at all, whether Python is quick enough for the parts that must keep time is the more useful question to settle early.

What do people try first, and why does it stop working?

Almost everyone starts by asking the model to produce motor commands or a block of code, which works on a tidy desk and comes apart the moment anything moves. The failure has one cause wearing several faces. A model writes its answer for the world you described a moment ago, with no clock, no feedback and no way to notice the mug moved while the sentence was being composed, so the plan is aimed at a memory. The second common attempt stuffs the whole robot state into one enormous prompt, which grows every week until the model answers confidently about the wrong object. The third asks the model on every cycle, which makes the robot pause to think between corrections and turns smooth motion into a stutter. Every fix has the same shape: fewer decisions for the model, more structure around them, and a layer beneath that can say no. The model picks a tested behaviour, the behaviour re-checks the world as it starts, and the loop runs underneath whether or not anything upstairs has replied.

What does it look like when this goes wrong on real hardware?

It looks like a machine doing something indefensible while explaining itself beautifully. The arm reaches for the mug a person moved while the model was thinking, closes on the edge of the table, and the log calmly says it is retrieving the mug for you. A base drives towards a room whose door has since shut and keeps nudging the doorway. An action half-completes, reports a failure with no detail, and the planner starts its next step from a pose nobody recorded. The trap is that the log reads like reasoning, so a team spends a week debugging sentences when the real bug is that nothing re-checked the world at the start of the action. The other classic shape is the robot that works for a fortnight and then does not, because somebody moved a table, which means the whole thing was leaning on a coincidence. If your robot already behaves differently from run to run before any model is involved, that is a separate diagnosis and worth fixing first.

What do you give up by letting a model decide what the robot does?

You give up the ability to state in advance exactly what the machine will do. A state machine is boring and inspectable: you can print every path and hand the list to a colleague, a customer or a safety reviewer. A model-driven robot cannot be enumerated, which turns testing from proof into sampling. You fix a bad behaviour by changing a prompt or a description, and nothing tells you which of yesterday's good behaviours you broke, so you accumulate scenarios and re-run them, which is evaluation work rather than engineering work. With a hosted model you also take on a dependency you do not control, and a provider updating that model changes your robot's behaviour on a day when your code did not change at all. You give up the crisp answer to why did it do that. All of this is worth paying when the task space is genuinely open and nobody could write the branches in advance. It is a poor trade when the robot does five known things and somebody simply wanted it to accept spoken instructions.

When is ROS 2 the better choice?

ROS 2 is the better choice the moment the hard part of your robot is perception, navigation or manipulation rather than the language layer on top. A machine that must map a building and drive to a named room is a ROS 2 project, because that navigation work represents years you will not reproduce alongside your model work. An arm that must plan around obstacles qualifies for the same reason. If the lidar you bought has one driver anyone has tested and that driver is a ROS 2 package, the decision was made in the shop. Most published work joining language models to robots also sits on ROS 2, so the examples, the bridges and the simulator assets you will want to copy assume it. HORUS is not the answer for those projects: choosing it there means rebuilding plumbing that already exists in order to lose the packages that were the point of the robot. And if you are joining a lab or a company that already speaks ROS 2, shared vocabulary beats any comparison you could run.

Does a language model replace the control code you would otherwise write?

No, and here is why: a language model is good at deciding what should happen and has no means of making anything happen on time. Control code exists to close the gap between what was asked for and what the machine actually did, checked over and over, and that checking has to continue while everything above is idle or thinking. A model has no clock, no feedback and no memory of the previous cycle unless you hand one over, and consulting a model every cycle would leave the robot hesitating between corrections. So adding a model adds a layer above the code you were always going to write, and removes none of it. There is one place where work genuinely disappears, and it deserves to be named precisely: the branching mission logic. That enormous tree of conditions describing what to do when the door is shut, the shelf is empty, or the person asked for something odd is the part a model can absorb, and on many robots it is the part nobody enjoyed writing.

Do vision-language-action models make this roadmap obsolete?

Partly, but not the way you think. End-to-end policies genuinely replace some hand-written skills, and they replace none of the structure around those skills. A vision-language-action model takes camera images and an instruction and produces motion directly, which is a real advance for tasks that resist hand-coding: folding a cloth, grasping something unfamiliar, wiping a surface where every attempt differs. What such a model does not provide is the stop path, the way a skill is invoked, the check on whether the skill finished, the record of what happened, the retry policy or the calibration, and those are most of what this roadmap covers. The work moves rather than vanishes, too: instead of writing a skill you are collecting demonstrations, and a policy tends to work on hardware resembling what it was trained on. Read a policy as a better skill, not a better architecture. Teams shipping them still build the action layer, the supervisor and the state tracking, because a policy that misreads a scene needs a floor beneath it exactly as a hand-written skill does.

How do you decide what to build first?

Build the boring layer first and let the model arrive last. In practice that means one action driven from a script, with an unambiguous signal for finished and for failed, then a second action, then the refusal path that stops both when a limit is crossed, and only then a model choosing between the two. The order feels backwards because the model is the interesting part, and it is right because every weakness underneath is inherited by everything above. Two questions settle most of the remaining arguments. First: if the model were deleted tomorrow, would the robot still do anything useful? A no means the machine is not yet a machine, and the model is propping up a robot that does not exist. Second: which failure would embarrass you in front of the person paying for this? Name that one, then make it impossible in the layer beneath the model rather than discouraged in the layer above, because a sentence is a request and a limit is a fact.

Decide by situation rather than by ambition:

• If you have a working agent and no robot -> buy a machine with a supported driver, because the physical half is the part you have not done.
• If the robot must map a building or plan around obstacles -> ROS 2, because those packages are the project, not the garnish on top.
• If the robot only has to listen, answer and gesture -> one script calling a hosted model, because there is no loop worth protecting.
• If a Python model process and a C++ control loop are fighting over frames on one machine -> a middleware built for that boundary, because the boundary is where your time goes.
• If you cannot yet make the robot do one task ten times from a script -> stop and do that first, because the model layer inherits those failures.

When the foundation question arrives, weigh the candidates on the five axes of the HORUS Fit Framework — ecosystem size, setup effort, team size fit, deployment target and licence — and take the option that loses on the fewest, with no scoring involved. If your robot is becoming one machine running a Python model process beside C++ or Rust control, star HORUS on GitHub so it is in your list when you start building.