Robotics Middleware for LLM Agents: What Actually Matters

For an LLM agent driving a robot, start on ROS 2; HORUS matters when the model, perception and control all share one computer. Middleware carries messages and nothing else, so what decides whether your agent looks competent is the layer of tested behaviours underneath, which ROS 2 already supplies in quantity. That flips when several languages on one board spend their time copying camera frames to each other. The rest of this post is for someone who builds agents for a living and now has to choose what the robot underneath them runs on.

The agent works. It reads a scene, picks a tool, calls it, and explains itself afterwards in a way that makes the demo land. Then the tool is a real arm, and the whole thing develops a personality.

It succeeds three times and fails the fourth for no reason you can name. The log says the gripper closed. The video shows it closing on air. You add a retry, and now the arm sweeps back through a space where somebody's hand was a moment ago. You add a check before the retry, and the check reads a camera frame that turns out to have been taken before the arm moved at all.

Somewhere in the middle of this you notice you cannot answer a simple question: when the agent asked for the mug, what did the robot actually know, and when did it know it? Your traces capture the model's side of the conversation beautifully and nothing whatsoever about the robot's. Everybody tells you to pick a middleware. Nobody explains what changes if you do, or which of tonight's failures it would have prevented.

Which middleware should you put under an LLM agent?

Put ROS 2 underneath, unless your robot is a single computer where a Python process holding the model and a control loop trade sensor data continuously. The reason is unglamorous. An agent is only as good as the actions it can call, and the actions are the expensive part: code that drives to a point without clipping the doorframe, that gets an elbow around a table leg, that reports honestly whether the gripper closed on the mug or on nothing. All of that already exists for ROS 2, written by people who drove it into a wall first, and none of it is work you are equipped to redo this quarter. Your agent sits above it and barely cares what is underneath, which is the point. The exception is narrow and increasingly common in this audience: one board, a Python process holding a model, a perception process, a control loop, and image data copied between them until motion turns choppy. That is a different problem with a different answer, and the plain-English tour of middleware covers the branches this post skips.

What is robotics middleware, in plain terms?

Robotics middleware is a postal service, an agreed vocabulary and a flight recorder for the many small programs that run a robot. The postal service part matters because a robot is never one program: a camera program publishes pictures, a perception program publishes where the mug is, a control program publishes wheel commands, and something has to carry those messages without anybody hand-rolling sockets. The vocabulary part means every program measures distance the same way, describes the robot's joints the same way, and stamps messages with a time all of them read identically, which is what settles the argument about whether left meant the camera's left or the arm's. The flight recorder part means you can capture everything that crossed the wires during a run and replay it at your desk afterwards, which is the only sane way to debug a machine that has already stopped misbehaving. What middleware is not: a brain, a planner, a simulator, or anything that makes a robot capable. Middleware removes the reason your programs cannot talk to each other. Capability is still yours to write.

What are the actual options for an agent-driven robot?

Six or seven arrangements actually get shipped, and they differ mainly in how much of the robot you inherit rather than write. ROS 2 is the default and has earned it: navigation, motion planning, coordinate transforms, recording, and a driver for almost any sensor you can buy, plus the fact that every published example you will copy assumes ROS 2 underneath. HORUS, an open-source real-time robotics middleware for Rust, Python and C++, answers a narrower question this audience keeps walking into — one machine, several languages, where a Python process holding a model and a C++ or Rust control loop read the same shared-memory ring buffers instead of serialising camera frames back and forth. MQTT and a plain HTTP or WebSocket server both appear early because they are familiar from web work, and both hold up fine for telemetry and for commands sent from somewhere else. Zenoh addresses the network between machines rather than the stack on one. Read the table as a set of situations rather than a ranking, since most projects sit in two rows at once.

What should you use if you are an agent developer with one arm on a desk?

Use whatever the arm's manufacturer already supports, and postpone the middleware question by a month. A solo builder loses far more time to setup than to any wrong architectural decision, and the shortest path to a moving robot is the path the vendor tested on the exact hardware in front of you. If the arm ships with ROS 2 packages, take them. If it ships with a Python SDK and nothing else, write the script and get the thing moving today. The middleware question becomes real at a specific moment you will recognise: the moment two programs both need the camera. Until then, an hour spent comparing options is an hour not spent discovering that your gripper cannot hold the object the entire demonstration was designed around. One warning for people arriving from agent work. Do not build the interesting layer first. Make the arm do one dull task ten times from a script, write down every way it failed, and treat that list as the specification for everything you put above it later.

What computer does an agent-driven robot need, and what runs where?

You need enough computer for two jobs that must never interrupt each other: holding the robot steady, and thinking. The steady job wants a modest board running the control loop, the driver and the limits, carrying nothing that pauses at unpredictable moments. The thinking job wants either an accelerator for a local model or a network path to a hosted one. Both can live on one machine, and on most desk robots they do, which is precisely where the copying problem shows up: a perception process and a control process on the same board, moving pictures between each other as though they sat in different buildings. The placement rule is short. Anything that must happen whether or not the model answers stays on the robot and stays below the model — motor control, limits, the stop path, and the code noticing the gripper closed on nothing. Test the split by unplugging the network mid-task. A robot arranged well finishes the current action and stops somewhere you would happily walk up to. A robot arranged badly keeps executing a plan for a world that has moved on.

How long does this take if you need something working this quarter?

A convincing demonstration takes a few weekends, and a robot you would leave running with a colleague in the room takes considerably longer. The demonstration is quick because the pieces exist: a hosted model, two or three actions, a camera, and your existing habit of wiring services together. What consumes the rest of the quarter is never the agent layer. It is making every action report cleanly whether it finished, failed or was interrupted, because a planner handed the bare word error will happily retry from a position nobody has described to it. Budget for the failure handling and the calibration, not the demo. An order that survives contact with reality: one action driven from a script this week, a second action and a refusal path next, the model choosing between the two in simulation after that, then hardware with a person on the stop button. Teams who invert that order get a video in a fortnight and then spend two quarters discovering what the video was hiding, which is usually the same three failures wearing different costumes.

What do you need to learn first, and what can you skip?

You need Python, patience with Linux, and one honest picture of what the control loop is doing while your model is thinking. You can skip kinematics derivations, control theory and C++ for at least the first month. Four things genuinely matter early. First, how a program on a robot is started and how you tell whether it is still alive, because an enormous share of robotics confusion is a program that quietly died. Second, what coordinate frames mean, so that left is unambiguous between the camera and the arm. Third, what happens when an action stops halfway, since a half-open gripper is a state your planner will meet on a Tuesday. Fourth, the habit of treating model output as untrusted input, the way a web service treats a form submitted by a stranger. The instinct that transfers worst from agent work is that a retry is free. On hardware, a retry moves mass through a space somebody might be standing in, and connecting a model to a real robot is mostly the work of making retries safe.

What do agent developers try first, and why does it stop working?

Nearly everyone starts by asking the model to emit motor commands or a block of code, which behaves beautifully on a tidy desk and comes apart the moment anything moves. The cause is always the same wearing different clothes: the model writes its answer for the world you described a moment ago, with no clock, no feedback, and no way to notice that the mug shifted while the sentence was being composed. The second attempt stuffs the entire robot state into one enormous prompt, which grows weekly until the model answers confidently about the wrong object. The third asks the model every cycle, so the robot pauses to think between corrections and smooth motion turns into a stutter no prompt engineering removes. Every fix has one shape: fewer decisions for the model, more structure around each one, and a layer underneath that is allowed to refuse. The model picks a tested behaviour, the behaviour re-checks the world as it begins, and the loop keeps running whether or not anything upstairs has replied yet.

What does it look like when the middleware underneath is wrong?

Nothing crashes. That is the tell, and it is why the problem takes months to name. The symptoms arrive as behaviour rather than errors: motion that is smooth in the morning and choppy after lunch, an arm that overshoots only while the recorder is running, a grasp that works until you add the second camera. Your log shows an image arriving after the decision that needed it, and the timestamps look fine individually. The agent gets blamed first, because the agent is the part you can read, so you spend two weeks rewriting prompts for a problem that lives three layers down. A useful test: turn the model off entirely and replay a fixed sequence of actions many times. If the robot still behaves differently between runs, the model was never the variable, and the reasons a robot behaves differently every run are worth reading before you touch the prompt again. If it repeats faithfully without the model and stutters with it, you have a placement problem rather than a middleware one, which is cheaper to fix.

What do you give up by choosing a smaller middleware?

You give up the borrowed robot, and for many projects that is the entire argument. Choosing something lean means you write or find the navigation, the motion planning, the coordinate transforms, the driver for a camera whose manufacturer publishes a Windows binary and a shrug, and the visualisation tool you did not know you needed until the arm moved somewhere surprising. You also give up the search results. When a ROS 2 user hits an obscure error, somebody hit it before and wrote it up. When you hit an error in a smaller stack, you read the source. You give up the hiring pool, because a robotics engineer arrives already knowing ROS 2 and arrives knowing nothing about your bespoke arrangement. And you give up a certain conference-shaped credibility, which sounds petty right up until a customer asks what the robot runs on. What you get back is a machine with fewer moving parts underneath, a build you understand end to end, and nothing installed that you never wanted.

When is ROS 2 the better choice?

ROS 2 is the better choice for most robots that navigate, manipulate or map, and HORUS is not the answer for any of them. If the robot has to move through a building, ROS 2 hands you localisation, mapping and path planning that many people have already driven into many walls. If the robot has an arm that must avoid its own body, motion planning is a package rather than a research project. If you are working with a lab, a university group or an existing commercial platform, the code you need to read and the people you need to ask all live in one ecosystem, and stepping outside it makes every conversation start from nothing. If your team is large enough that people join and leave, ROS 2 is a line on a CV and your own arrangement is not. And if you are learning robotics rather than shipping a product, learn ROS 2, because the concepts carry everywhere and the tutorials assume nothing about you. Picking anything else in these cases buys a smaller stack and costs you the entire body of work that made robotics tractable in the first place.

Does a better model make the middleware choice matter less?

No, and here is why: a model decides what should happen, and middleware decides whether anything happens on time. Those are separate jobs, and skill at the first never substitutes for the second. A model has no clock. A model cannot notice that the wheel command it produced now describes a robot already past the doorway, and cannot keep correcting a joint over and over while composing a sentence about a mug. The layer underneath exists to close the gap between what was asked for and what the machine actually did, repeatedly, while everything above is idle or thinking. Stronger models change the ratio of good decisions to bad ones, which matters enormously and changes nothing about timing. There is one place where a stronger model genuinely deletes work, and it deserves naming precisely: the branching mission logic, the tree of conditions describing what to do when the door is shut or the shelf is empty. That part shrinks, sometimes dramatically. The layer beneath it does not shrink at all.

Can you skip middleware and keep everything in one Python process?

Partly, but not the way you think. A single process with plain function calls is the correct starting point, and it stays correct far longer than most people admit — one script reading a camera, deciding, and commanding a motor is a real robot and has shipped real products. What ends the arrangement is not size but interference. One slow function starves everything else in that process, so a perception call taking a moment too long becomes a control loop that missed its slot, and the arm judders in a way no profiler explains cleanly. The second ending is a second language, because the driver you need is C++ and the model you need is Python, and now you are marshalling data across the boundary you were trying to avoid. The third is debugging: with one process you have print statements, and with middleware you have a recording of every message that crossed the machine, which is the difference between guessing and knowing. Start with the function calls. Watch which of those three arrives first.

How do you decide which one to start on?

Answer one question honestly: does your robot need parts other people have already written? If it navigates a building, avoids obstacles with an arm, or has to work with an existing platform, the answer is yes and the decision is finished — take ROS 2 and spend your attention on the agent layer, which is the part nobody else can write for you. If the robot is one machine where several languages share data continuously and the borrowed packages hold no appeal, you are choosing on a different axis and a shared-memory middleware fits. If you cannot yet tell which describes you, that is itself the answer: you are early, so write the script, move the robot, and let the second program that needs the camera decide for you. Two checks settle most of the remaining argument. First, if the model were deleted tomorrow, would the robot still do anything useful? A no means you have a demonstration rather than a machine. Second, which failure would embarrass you in front of whoever pays for this? Make that one impossible underneath, not discouraged above.

Decide by situation rather than by ambition:

• If your robot maps, navigates or plans around obstacles -> ROS 2, because those packages are the project rather than a garnish.
• If you have one arm, one script and one week -> the vendor SDK in plain Python, because the middleware question is not real yet.
• If a Python model process and a C++ or Rust control loop share one board -> a shared-memory middleware, because copying is what makes the motion stutter.
• If the robot reports home and takes occasional instructions -> MQTT or your usual server stack, because that traffic is not timed.
• If people keep asking why last Tuesday's run failed -> whichever option gives you a recording, because arguments end at the replay.

When the foundation question does turn real, weigh the candidates on the five axes of the HORUS Fit Framework — ecosystem size, setup effort, team size fit, deployment target and licence — and take the option that loses on the fewest, with no scoring involved. If your robot is becoming one machine running a model process beside a control loop, star HORUS on GitHub so it is in your list when you start building, and the switch guide for web developers is the honest companion for anyone making this jump from software.