Why LLM-Controlled Robots Look Great in Demos and Fail at Home

Because a demo hides the layer underneath: an LLM can choose the next step, but only a control loop beneath it keeps the robot steady. A home changes between attempts, and the hardware still needs commands on a rhythm no language model can hold. That flips when the robot is slow and forgiving, where the model can drive it directly and neither ROS 2 nor HORUS is needed. This post is for someone who has an agent working in software and wants to know what actually stands between that and a robot in a room.

You watched the video. A person asks for a cup of tea, the robot pauses for a beat, and then it moves, and the movement looks like understanding. You have built agents. You know how to give a model tools, how to make it plan, how to make it call functions and read the results. The gap between that and the video looks like an integration problem, and integration problems are the kind you know how to solve.

So you buy an arm. You wire up the model, give it a set of functions, and ask it to pick up a mug. Sometimes it works. Often it reaches slightly past the mug and closes on air. Once it pushed the mug over. Once it moved smoothly to the right place, held for a moment, and then everything went still while the model thought about what to do next, and the arm sagged.

Nothing in that is an error you can read. There is no traceback, no failed tool call, no bad response to inspect. The model's reasoning looks fine when you print it. The machine simply does not do what the reasoning says, and the failures do not repeat the same way twice.

Why do LLM-controlled robots work in demos and fail in a real home?

A demo is a scene somebody chose, and a home is a scene nobody chose. That single difference accounts for most of the gap. In a demo the lighting is settled, the mug is where it was during rehearsal, the table height is known, the floor is clear, and the person who built the system is standing just off camera ready to reset anything that drifts. Every one of those is a variable that has been quietly removed. In a kitchen, none of them are removed. The mug is behind a cereal box, the light is coming through a window that was dark yesterday, a chair is pulled out, and there is a cat. A model that chooses well when the scene matches its expectations is not the same as a system that recovers when the scene does not. The second thing a demo hides is the number of attempts. Demo footage is one successful run. The interesting question about a home robot is what its tenth attempt looks like on a day nobody prepared, and that footage does not get published.

What does it actually mean to put a language model in control of a robot?

It means the model decides what happens next, and something else entirely makes it happen. That split is the whole subject, and blurring it is where projects go wrong. Deciding is intermittent: it happens when a goal changes, when a step finishes, when something unexpected appears. Making it happen is continuous: while an arm holds a mug, something is reading positions and issuing corrections without pause, and the moment those corrections stop arriving the arm stops behaving like an arm. A language model is built for the first job and structurally unsuited to the second, not because of the model's quality but because thinking takes a variable amount of time and holding a grip does not tolerate variability. The productive framing is that the model is a manager, not a hand. Managers choose the work. Something with steady hands does it. Systems that fail in kitchens are almost always systems where the manager was handed the tools directly, and the gap between writing an app and picking up a cup is exactly this one.

What are your options for connecting a model to real hardware?

There are seven arrangements worth knowing, and they differ mainly in how much sits between the model and the motors. The model can write and run code against a vendor SDK, which is the fastest thing to try and the least predictable. The model can select from a library of skills you wrote and tested, which is the arrangement most working systems converge on. You can run ROS 2 with the model as one participant among many, inheriting navigation, mapping, simulation and drivers along with a substantial amount of setup. You can put a middleware between the model process and the control loop: HORUS is an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages between processes on one machine are not serialised, which fits a Python process holding the model beside a C++ or Rust loop holding the hardware. You can run a vision-language-action policy on board instead of a chat model. You can keep a person in the loop and let the model assist. Or you can stay in simulation, which is the cheapest way to learn what your plan actually assumes.

What should you build on if you are an AI engineer with no robotics background?

Start with a library of skills you wrote yourself, and let the model choose between them rather than compose motion from scratch. This is the opposite of the instinct, which is to give the model the lowest-level control available and let capability emerge from prompting. It is also the arrangement that survives contact with a real room. A skill is a small, tested behaviour with a clear end: move to a named position, close the gripper until it grips, lift by a hand's width, stop if resistance is felt. You verify each one on hardware, alone, until it is dull. Then the model's job becomes choosing skills and reacting to their outcomes, which is precisely what a model is good at, and the parts that must never surprise you are the parts you tested. The failure mode you avoid is the one that ruins demos: the model producing a coordinate that is subtly wrong, with nothing underneath it that knows the difference between a mug and a table edge.

What should you use if the robot carries a small computer rather than a workstation?

Split the work by where it has to run, not by what is convenient. A small board on the robot can hold the control loop and the safety limits comfortably, because those are small pieces of arithmetic repeated forever. It usually cannot hold a large model at all, which means the model runs somewhere else and its decisions arrive over a network that will, at some point, stall. Design for that stall before it happens. The control loop keeps its rhythm whether or not a decision arrives; a missing decision means continue what you are doing or stop safely, never freeze mid-motion. This is also where the shape of your message passing starts to matter: on one board, keeping messages in memory both sides already share avoids packing and unpacking data on a path where the timing matters. A tour of what sits between the model and the motor is worth reading before you commit to a layout, because moving that boundary later is expensive.

What should you do if you need a working demo in three weeks?

Constrain the scene and script the motions, and be honest with yourself about what the demo proves. This is not cheating; it is what almost every impressive robot video is. Fix the table height, mark the object positions, control the lighting, and use the model for the part that genuinely benefits from language: understanding the request, choosing the sequence, explaining what it is doing. Do not spend the three weeks trying to make perception general, because general perception is not a three-week problem and a half-finished version of it is worse than a marked table. What matters is that you write down which variables you removed. That list is the actual roadmap to a product, and teams that never write it down end up believing their demo was closer than it was. The distinction between what a demo shows and what a machine can do unattended is covered in this look at demos versus autonomy, and it is the most useful lens you can bring to your own footage.

What should you choose if Python and prompts are your whole toolkit?

Stay in Python for everything except the loop that keeps the hardware steady, and get that loop from somewhere else. Python is entirely capable of running your model, choosing skills, talking to a vendor SDK and logging what happened, and there is no reason to leave it for that work. What Python is poor at is holding a rhythm while also doing something heavy in the same program, because the heavy work and the rhythm compete. In practice you have three ways out. Use hardware whose own controller already holds the loop, which most commercial arms do, and command it at the level of finished motions. Use a framework that ships a control layer written by somebody else. Or run a second process in a language that keeps time and pass decisions to it. All three keep you writing Python where the interesting work is. The layer question underneath all of this is the subject of framework, middleware or SDK, and it is worth settling early.

What do teams try first, and why does it stop working?

Nearly everyone starts by giving the model direct control and a rich set of low-level tools, and it stops working the first time the room disagrees with the model's picture of it. The early results are encouraging, which is the trap. The arm reaches the mug, the model narrates its plan convincingly, and it feels like the remaining work is more prompting. Then a run happens where the grasp misses by a small margin and nothing notices, because nothing in the system was ever told what success feels like. The second attempt is a retry loop, which produces a robot that fails repeatedly and confidently. The third is more context: photographs, measurements, longer prompts. That helps in the demo room and not elsewhere, because the problem was never the model's understanding of the scene. It was that the system had no closed loop between what it did and what happened. Every one of those steps is a reasonable engineering response. Together they postpone the one change that fixes it, which is putting something underneath the model that watches the world and reacts without asking permission.

What do you give up by putting a language model in charge?

You give up repeatability, and repeatability is the property that turns a robot into a product. The same request on the same day in the same room will not produce byte-identical behaviour, which means testing changes character: you can no longer verify a fix by running it once. You also give up a clean account of why something happened. When a scripted robot misbehaves, the cause is in the script. When a model-led robot misbehaves, the cause is somewhere in a chain of decisions that will not reproduce, and the log tells you what was decided rather than why. And you give up a certain kind of guarantee. If the model can choose freely, then the set of things the robot might do is not enumerable, so safety has to come from limits underneath that hold regardless of what was chosen. That is not an argument against models on robots. It is the reason the tested-skills arrangement keeps winning: it puts the unpredictable part where unpredictability is welcome.

When is ROS 2 the better choice?

ROS 2 is the better choice as soon as your robot has to move around a building rather than stand at a bench. Mapping a space, planning a route through it, avoiding a person who walks into the corridor, knowing where the robot is after a wheel slips: these are solved problems with years of work behind them, and no team reproduces them alongside their model work. If your sensor ships only a ROS 2 driver, that decides it too, as does wanting a simulator that speaks to your code without custom glue, or wanting to record a session and replay it to understand a failure. HORUS is not the answer for any of that, and choosing it for a mobile robot means writing navigation you could have inherited. The cost of ROS 2 is a genuine stretch of learning before anything new happens, and a strong preference for Linux. For an AI engineer, that cost is usually worth paying once the robot has wheels, and rarely worth paying while it does not.

Is the problem just that the model is not good enough yet?

No, and here is why: the failures are in a part of the system the model was never doing. A better model chooses better steps, describes its intentions more accurately and recovers from ambiguity more gracefully, and none of those things hold an arm steady between decisions. The gap in a kitchen is rarely a bad plan. It is a good plan executed against a world that moved, with nothing in the loop that noticed the movement. Watch enough failed attempts and the pattern is consistent: the reasoning is sound, the motion command matches the reasoning, and the outcome does not match the command, because the object was a little further away or the surface was a little lower or the gripper closed a moment early. No amount of language capability addresses that, since none of it is expressed in language. What addresses it is feedback: something measuring the difference between what was asked and what occurred, quickly enough to correct while the motion is still happening.

Will a bigger model or a robot foundation model fix this?

Partly, but not the way you think. Models trained directly on robot data do change the picture, because they close a loop that language models cannot: they map what a camera sees onto what the joints should do, continuously, without a translation step where meaning gets lost. That genuinely removes one of the failure modes above. What it does not remove is the need for a layer underneath that keeps time and enforces limits, because a learned policy still emits commands that something has to execute, and still has no concept of the table it is about to strike unless something separate is watching for it. So the honest expectation is a shift rather than a solution. The decision-making becomes far better at the parts that resist being written down. The engineering underneath stays exactly where it was: a loop that runs on a rhythm, limits that stop motion when a boundary is crossed, and a way of carrying data between the thinking part and the acting part without either waiting on the other.

How do you decide what to build on?

Answer one question first: does the model sit above your control path, or inside it? If the answer is inside, that is the thing to change, and no other decision matters until it does. Once the model is above the control path, the remaining choices are ordinary. Ask whether your robot moves around a space or stays in one place, because that decides whether you need borrowed navigation. Ask whether your tasks can be written as a finite list of skills you could test one by one, because a yes means the tested-skills arrangement and a no means you are heading toward learned policies and the data collection that implies. Ask where the model runs, because a model on another machine makes network stalls a design input rather than a surprise. Ask whether anyone on the team has held a control loop before, because if nobody has, buying hardware that already holds its own is the least painful path by a wide margin.

Decide by situation rather than by what the videos suggest:

• If you are testing an idea this week -> the vendor SDK with a scripted scene, because nothing else gets you feedback faster.
• If the robot drives around a building -> ROS 2, because navigation and mapping are the project and rebuilding them is not.
• If your tasks fit a list of testable skills -> a skill library the model selects from, because the unpredictable part belongs where surprises are cheap.
• If the model is Python and the loop must keep time -> a shared-memory middleware between two processes, because that boundary is where the stutter appears.
• If the task resists being written down at all -> a learned policy and the data pipeline it needs, because prompting will not get there.
• If a deadline is close -> constrain the scene and write down what you removed, because that list is your roadmap.

When two options stay close, weigh them on the five axes of the HORUS Fit Framework — ecosystem size, setup effort, team size fit, deployment target, and licence — and take the one that loses on the fewest. No scores and no numbers, just five honest questions about your situation rather than about the software. And when your model finally needs a loop underneath it that keeps time, star HORUS on GitHub so it is in your list when you start building.