LLM Robot Control: How Language Models Move Real Hardware

A language model never drives motors directly; it picks among behaviours your robot already performs, with ROS 2 or HORUS carrying the messages underneath. Motion arrives on a fixed schedule and a model answers whenever it finishes thinking, so the two cannot share a job. That flips only when the machine's whole purpose is talk and gesture, where nothing underneath needs protecting. The rest of this post traces the whole chain from a typed sentence to a moving joint, for people who ship model-driven software and now own hardware.

You have shipped things that reason. Tools, retries, evaluation harnesses, a model that plans a task and calls your functions and mostly gets it right. Then somebody puts a small arm on the desk next to your laptop, and the question changes from what should happen to what physically happens, and you notice how little of your intuition survives the move.

The first thing you notice is the silence. In software, a call returns or throws. Here the arm moves, or it moves partly, or it moves and then something rolls off the table, and nothing anywhere raises an exception. The second thing is time. The model takes as long as it takes, and the arm does not pause politely while it thinks — it holds a position, and holding a position is itself a job that never stops.

The third thing is that nobody's demo video shows any of this. They show a sentence and a completed task, cut together, and you are left guessing what was under the cut. That layer is the actual work, and this is what it contains.

Should a language model be the thing that moves your robot?

No language model should hold the wheel, and every arrangement that survives contact with hardware puts the model above the wheel instead. The reason is a mismatch of clocks. A joint holding a position needs a fresh command on a rhythm it can count on, computed from what the encoder said a moment ago. A language model produces its answer whenever inference happens to finish, from a description of the world written before it started, and occasionally produces something wrong that reads exactly like something right. Neither property is survivable inside a loop that keeps an arm from falling. So the working pattern has three layers. At the bottom, code running on its own schedule that holds the machine steady. Above that, a set of named behaviours — go to the charger, close the gripper, look at the table — each tested alone and each able to report whether it finished or failed. On top, the model, choosing which behaviour runs next and filling in a blank or two. What joins those layers is the actual engineering, and it has more parts than the diagrams suggest.

What does LLM robot control actually mean in plain terms?

LLM robot control means a language model decides what the robot should do next, in words, while entirely separate code turns that decision into motion. Nothing in the phrase implies the model touches hardware. The model does not see a voltage, does not run on a schedule, and does not know whether the last command landed. Think of a film set. The director calls the next shot; the stunt driver handles the car, feels the road, and would refuse a stunt that ends in a wall no matter how well it was described. The director's authority is real and it stops at the point where physics starts. Your robot needs a competent driver before a director is worth hiring, and most of the disappointment in this field comes from teams who hired an articulate director for a car with nobody at the wheel. In practice the director's vocabulary is a handful of named actions with a parameter or two each, and the script the director reads is a short paragraph your own perception code writes from sensor data.

What are the real ways to put a language model in charge of hardware?

There are roughly eight arrangements people actually ship, and they differ in how much the model is allowed to decide and where that decision is made. At the loose end, the model writes code that the robot then runs, which is quick to demonstrate and hard to bound. In the middle sit the arrangements that borrow an existing stack: ROS 2 already supplies drivers, navigation and motion planning behind goals that report progress and can be cancelled, which is precisely the shape a planner above them needs, while HORUS, an open-source Apache-2.0 middleware for Rust, Python and C++ whose three languages share the same shared-memory ring buffers, suits the case where the model process, the perception code and the control loop all live on one computer and the camera frames should not be copied across a language boundary on the way through. At the far end, a trained policy produces motion from images and an instruction with no named behaviour in between. Read the table as situations rather than a ranking; several rows describe the same project in different months.

What should you do if you build model-driven software and have never shipped hardware?

Buy the smallest machine that already works without you, and spend a fortnight making it do one dull thing the same way every time from a plain script, with no model anywhere near it. Your existing habits transfer better than roboticists tend to admit: designing evaluations, sorting failures into categories, noticing a model that is confidently wrong, and distrusting a metric computed on examples you chose yourself. What does not transfer is physical. A retry in software costs nothing; a retry on hardware moves mass and can knock something over. Objects change position while you are thinking about them. Calibration drifts across an afternoon, so a plan that was correct this morning now aims slightly beside the target. Grippers slip on the object they held earlier the same day. So do not start by building an arm, and do not start with the planner. Start with the smallest embodiment you can get running, and keep a written log of every way the dull task failed. That log is the specification for everything the model layer will later have to survive.

What hardware does a language-model-driven robot actually need?

You need two computers' worth of work, even when both halves run on one board: something steady that is never surprised, and something that thinks. The steady half runs the control loop, the driver and the limits, and carries nothing that can pause unexpectedly — no large model, no garbage-collected service holding the joint command. The thinking half needs either a network path to a hosted model or an accelerator big enough to hold a small local one in memory. Which side a job lands on follows one rule: anything that must still happen when the model does not answer stays on the robot. Motor control, limits, the stop path and the code that notices an empty gripper all live below, always. Perception can sit on either side, though shipping every frame over a network gets expensive and brittle quickly. The test that settles arguments in a team is unplugging the network mid-task. Split properly, the robot finishes the behaviour it started, notices that nothing is coming, and stops somewhere a person would happily walk up to.

How long before the robot can be left running unattended?

A demonstration takes days; a robot you would leave alone with the model in charge takes quarters, and the gap is almost entirely failure handling. The first stage is quick and flattering. You wire a hosted model to two behaviours, speak a sentence, the arm moves, and the whole thing feels solved. The second stage is where the calendar actually goes: making every behaviour report finished or failed with enough detail that the layer above can choose differently, adding the check that runs at the instant a behaviour starts, and writing down what happened in a form you can replay afterwards. The third stage is longer still and rarely planned for — collecting the scenarios that broke, turning each into something you can re-run on demand, and re-running all of them every time you change a description or a prompt. That third stage never ends, because a hosted model can change underneath you on a morning when your own code did not change at all. Plan for the first stage to flatter you and the second to be the real project.

What do you need to understand before wiring a model to a motor?

Enough Python to write a small service, plus one honest picture of what the control loop is doing while the model is thinking. You do not need to write drivers, derive kinematics or learn C++ in the first month. You do need four things. First, how a behaviour is started and how you learn it finished, because everything above rests on that answer being trustworthy. Second, what state the machine is in when a behaviour stops halfway, since a half-open gripper is a situation your planner will meet. Third, enough about coordinate frames to know what the word left means to the robot as opposed to the camera that saw the object. Fourth, the habit of treating model output as untrusted input, the way a web service treats a form submitted by a stranger. The instinct that transfers worst from machine-learning work is that trying again is free. If you are still deciding what to write the layer above in, whether Python keeps time well enough is worth settling early.

What does language-model robot control look like when it goes wrong?

It looks like a machine doing something nobody would defend while producing a paragraph anyone would sign off. Three failures dominate. The first is the stale world: the model reasons over a description written before it began thinking, so the gripper closes where the mug used to be and the record says the mug was collected. The second is the silent half-finish: a behaviour stops partway, reports failure with no detail, and the next step starts from a pose nobody wrote down, which is how a robot ends up dragging an object it believes it is not holding. The third is the confident detour: the instruction was ambiguous, the model picked a reading, and nothing underneath was permitted to refuse. What makes all three expensive is that the log reads like reasoning, so teams debug sentences for a week when the real bug is that nothing re-checked the world at the moment the behaviour began. If your machine already varies from run to run before any model is involved, that is a separate diagnosis.

What do people try first, and why does it stop working?

Most people start by asking the model for the whole plan up front and then running that plan to the end, which works once on a tidy desk and stops working the moment step one changes anything. The plan was written for a world that has since moved, and nothing in the sequence re-reads it. The second attempt is a long tool list — thirty functions, each individually sensible — which fails differently: the model composes pairs nobody ever tested together, and the failure looks like a bug in a function that behaves perfectly alone. The third attempt is repairing behaviour with prompt text, adding a line telling the model not to close the gripper on the table edge. That holds until the day it does not, because a sentence is a request and the machine needs a fact. Every repair converges on the same shape: fewer choices offered, each behaviour re-checking the world as it begins, and a layer underneath that refuses an out-of-bounds request without consulting anything above it.

What do you give up by putting a language model in the loop?

You give up the ability to state in advance exactly what the machine will do, and that loss is larger than it sounds. A hand-written state machine is dull and inspectable: you can print every path it can take and hand that list to a colleague, a customer or a safety reviewer. A model-driven robot has no such list, so testing stops being proof and becomes sampling, and you meet the paths you never imagined by watching them happen in a room with furniture in it. You give up the crisp answer to why did it do that, because the cause is a sentence rather than a branch. You give up cheap fixes: editing a prompt to repair one behaviour tells you nothing about which of yesterday's good behaviours you just changed, so you accumulate scenarios and re-run them, which is evaluation work rather than engineering. And with a hosted model you take on a supplier who can alter your robot's behaviour on a morning when your own repository did not change at all.

When is ROS 2 the better choice?

ROS 2 is the better choice the moment the hard part of your robot is perception, navigation or manipulation rather than the language layer above them. A machine that must map a building and drive to a named room is a ROS 2 project, because that navigation work represents years you are not going to reproduce alongside your model work. An arm that must plan around obstacles qualifies for the same reason. If the depth camera you bought has exactly one driver anybody has tested and that driver ships as a ROS 2 package, the decision was made in the shop. Most published work joining language models to robots also assumes ROS 2, so the examples, the bridges and the simulator assets you will want to copy land there first. HORUS is not the answer for those projects — choosing it there means rebuilding plumbing that already exists in order to give up the packages that were the reason for the robot. And if you are joining a lab or a company that already speaks ROS 2, shared vocabulary beats any comparison you could run yourself.

Can you make a language model safe by writing a careful enough prompt?

No, and here is why: a prompt is a request, and a machine near people needs a fact. Everything in a system prompt is advice the model may follow, and the property that makes a language model useful is that it generalises past its instructions, which is the same property that lets it generalise past your warnings. A limit that matters has to live where no sentence can reach it — in the code that clamps a joint command before it is sent, in the check that refuses a goal outside the workspace, in the hardware stop that cuts power regardless of what any software believes. Prompt text is still worth writing, because it shapes ordinary behaviour and cuts down the number of odd requests you have to refuse, but it belongs in the same category as a well-worded sign rather than a lock. The test is short and unkind. If a stranger typed the worst instruction they could think of, which line of your code stops the arm? If the answer is a sentence, there is no answer.

Is language-model robot control just teleoperation with extra steps?

Partly, but not the way you think. A robot taking instructions from a model sits closer to supervised operation than the demonstrations suggest, and pretending otherwise is how teams get surprised in front of customers. Someone still decides what the machine should attempt, and in early deployments that someone approves each step, which is teleoperation with a slower and more articulate operator. The difference that matters is not autonomy in the abstract; it is which decisions were made in advance and written down. In teleoperation a person reads the scene and picks the motion in the moment. With a model above named behaviours, the motions were chosen, tested and bounded weeks earlier, and only the ordering is decided live. That is a genuine change, because the machine keeps working when nobody is watching a video feed, and the set of things it can attempt is a list you wrote yourself. Demonstration videos rarely tell you which of the two you are watching, which is why reading robot demos carefully is a skill of its own.

How do you decide where a language model belongs in your robot's stack?

Decide by asking what would still work if the model were deleted tomorrow. If the answer is nothing, you do not yet have a robot; you have a demonstration with a model propping it up, and the fix lives beneath the model rather than inside it. Build the dull layer first: one behaviour driven from a script with an unambiguous signal for finished and for failed, then a second, then the refusal path that halts both when a limit is crossed, and only then a model choosing between the two. The order feels backwards because the model is the interesting part, and it is right because every weakness underneath is inherited by everything above. Then ask the second question: which failure would embarrass you in front of the person paying for this? Name that one, and make it impossible in the layer beneath the model rather than discouraged in the layer above. Those two questions settle more architecture arguments than any diagram, and a safety reviewer will ask you both of them later anyway.

Decide by situation rather than by ambition:

• If you can run the task from a script and now want spoken instructions -> a hosted model above two or three named behaviours, because the hard part is done.
• If the difficulty is mapping, navigation or motion planning -> ROS 2, because those packages are the project rather than a garnish.
• If the machine only listens, answers and gestures -> one script calling a hosted model, because there is no loop worth protecting.
• If a Python model process and C++ or Rust control share camera frames on one computer -> a middleware built for that boundary, because that is where your time goes.
• If you cannot yet run one task ten times from a script -> do that first, because the model layer inherits every failure beneath it.

When the foundation question arrives, weigh candidates on the five axes of the HORUS Fit Framework — ecosystem size, setup effort, team size fit, deployment target and licence — and take the option that loses on the fewest. If your robot is becoming one computer running a Python model process beside C++ or Rust control, star HORUS on GitHub so it is in your list when you start building.