Best Middleware for Robots That Run an AI Model On-Board

For robots running a model on-board, ROS 2 is the safe default, while HORUS fits when the model and the control loop share one computer. The reason is ecosystem: an on-board model still needs cameras, drivers and a navigation stack, and ROS 2 has them. The verdict flips when large frames move between programs on one machine and the copying starts eating the time your controller needed. The rest of this post is for people who already train or fine-tune models and now have to put one on a machine that moves.

Your model works. In a notebook, on your desk, against recorded footage, it identifies the mug and returns a grasp pose and you have watched it do this a hundred times. Then you put it on the robot and the robot behaves like something with a bad connection. The arm moves in small jerks. It reaches for where the mug was rather than where the mug is. Occasionally it freezes mid-motion for no reason you can name, then continues as if nothing happened.

So you go looking for the cause and the internet offers a shape of answer you did not expect. Everyone wants to talk about the plumbing between programs. Threads about serialisation. Threads about which transport layer is underneath. Threads where someone insists the whole framework is wrong for this and someone else insists that person is a hobbyist.

You did not sign up to have opinions about transports. You wanted the arm to reach the mug. But the jerking is real, the demo is in three weeks, and you have started to suspect that the thing carrying data between your programs is not a detail you can keep ignoring.

Which middleware should you pick for a robot with a model on board?

Pick ROS 2 unless the model and the control loop live on the same computer and the data between them is large, in which case a shared-memory middleware fits better. That single split covers most decisions. A robot where the model advises rather than steers — recognising objects, reading signs, choosing a destination — has time to spare, and the ecosystem around ROS 2 is worth more than any saving in how data moves. A robot where the model sits inside the loop that keeps it upright or on target has a different problem, because every copy of a camera frame between programs is time subtracted from the response. The mistake to avoid is deciding this from vibes about which project sounds more modern. Decide it from where your model sits: advisory or in the loop, one computer or several, small messages or large frames. Those three answers determine the outcome far more than any framework's marketing does, and they are questions about your robot rather than about software.

What does middleware actually do on a robot that runs a model?

Middleware moves data between the separate programs on a robot, which on a machine with an on-board model means at least four: the camera driver, the model, whatever plans the next action, and the thing commanding joints. Each runs at its own pace and none should be able to stall the others. Middleware handles the introductions, so the model does not need to know where the camera driver lives, and carries the data across, so nobody writes socket code. It also settles what happens when producers and consumers disagree about pace, which is the part that matters most here, because a camera producing frames faster than a model consumes them is the normal condition rather than an error. The important consequence is architectural. Middleware is what lets you separate the slow thinking part from the fast reacting part, and that separation is why a well-built robot keeps moving smoothly while a model takes its time. What sits between the model and the motor walks the whole chain from a camera frame to a joint command.

What are the actual middleware options for an on-board model?

You have about seven realistic options, and they sort by how much ecosystem you inherit against how directly your programs share data. ROS 2 with its default transport is the baseline: every driver, every visualisation tool, every tutorial, and a message system that works well until large frames start crossing between programs on one box. ROS 2 configured to use Zenoh underneath keeps the ecosystem and behaves better across machines and unreliable networks. HORUS is an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages are not serialised between processes on one machine — which suits the case where your Python model and your C++ or Rust controller both need the same camera frames without copies passing between them, and like ROS 2 it is Apache-2.0. You can also run no middleware at all and keep everything in one Python process, use a broker such as MQTT for a robot that mostly reports and receives commands, accept the robot maker's own SDK, or build a small in-house layer.

What should you use if you are an ML engineer new to robots?

Start with ROS 2, because your scarce resource is robotics knowledge rather than data movement, and ROS 2 is where that knowledge is written down. Someone arriving from machine learning usually has the model half solved and the robot half not started, and the gaps are not where they expect. Coordinate frames will cost you a week. Time synchronisation between a camera frame and the joint positions at the moment it was captured will cost you another, and getting it wrong shows up as a model that seems to have aged badly. Knowing which behaviour is safe when perception stops arriving is a design decision nobody in machine learning had to make before. All of that has established answers in ROS 2 and no answers at all if you start by writing your own layer. The one habit worth importing on day one is separation: never put a model call inside the loop that commands the machine. Build it as two programs from the beginning, even when one process would run today, because splitting them later means rewriting the part you understand least.

What should you use on a Jetson or one small on-board computer?

On a single small computer, the shape that matters is how many copies of each camera frame get made before the model sees it, and that is where a shared-memory middleware earns its place. A small board has limited memory bandwidth and a fixed power budget, and both are consumed by the same work: passing large frames between programs. ROS 2 runs perfectly well on this class of hardware for a robot whose model is advisory, and the ecosystem remains worth more than anything you would save by hand-rolling. The pressure appears when a camera feeds a model, the model feeds a planner, and the planner feeds a controller, all on one board, with the frames copied at every hop. You feel it as a machine that gets hot, drains its battery faster than expected, and responds late under load rather than one that fails outright. Middleware choices for Raspberry Pi and Jetson robots goes further into what small boards actually tolerate, including the option of splitting perception onto a second computer.

What should you use if you need a working demo this month?

Use whatever your team already knows, and for most teams that is ROS 2 or a single Python process. A month is not enough time to learn a framework and a robot and a deployment target at once, and the demo failure mode is never that the plumbing was suboptimal. It is that the model was never wired to the arm at all, or that the arm reached for a stale pose and knocked the mug over in front of the room. Two decisions protect a short timeline. First, split the model and the controller into separate programs immediately, so a slow inference call cannot freeze the machine mid-motion. Second, define what the robot does when a result does not arrive: hold position, continue the current motion, or stop. Demos survive on that definition, because the failure that ruins a demo is a robot doing something surprising rather than a robot doing something slow. Change the middleware after the demo, when you know which part of the system actually hurt, not before, when you are guessing.

What if your team writes Python and nothing else?

Stay in Python for perception and accept that the loop commanding joints will eventually want a different language. Python is entirely reasonable for the model side, because the heavy work happens inside the model runtime rather than in the interpreter, and the interpreter is mostly waiting. Where Python becomes visible on the machine is the loop that must keep its rhythm, because a garbage collection pause or a moment of lock contention turns into motion you can watch. Teams meet this in a predictable order. Everything in Python works, then the arm develops a small hitch under load, then someone discovers the hitch is not in the model at all. The good news is that the fix is local: move the innermost loop, keep everything else. That is much easier when the two halves were already separate programs, and much harder when they share a process. A middleware where Python and a faster language read the same data without copies makes that migration a rewrite of one program rather than a rewrite of the robot.

What goes wrong when a model and a control loop share one machine?

They interfere, and the interference shows up as motion rather than as an error message, which is why it takes teams so long to diagnose. The characteristic symptoms are worth memorising. The arm moves in small jerks instead of one smooth reach. The robot acts on where the object used to be, so it grasps just behind a moving target. It freezes mid-motion for a moment, then resumes. Under sustained load the board runs hot and everything degrades together, so it looks like a hardware fault. Underneath, three distinct causes produce these same symptoms. The controller is blocked waiting for an inference result. Frames are being copied so many times that each one arrives after the moment it described has passed. Or the model and the controller are competing for the same cores and neither gets a clean run. The diagnosis matters because the fixes differ. Blocking is fixed by separating the loops. Copying is fixed by how data moves between programs. Contention is fixed by pinning work to cores or moving perception to another computer.

What do teams try first, and why does it stop working?

Almost everyone starts by putting the model call directly in the control loop, and it stops working the first time the robot needs to move while thinking. It is the obvious construction: get a frame, run the model, decide, command the motor, repeat. It works in a notebook, it works in a slow demo, and it fails the moment the machine has momentum, because the world does not pause while the model runs. The second attempt is usually threads inside one Python program, which helps a little and introduces a new class of bug that only appears under load. The third attempt is separate programs with a queue between them, which is the right shape and where teams discover that queues have a policy question hiding in them: when the model falls behind, do you process every frame late or skip to the newest one? For a robot that must act on the present, the answer is almost always the newest frame, and a queue that silently prefers the oldest is a common cause of a robot that always seems to be reacting to the recent past.

What do you give up by leaving the ROS 2 ecosystem?

You give up the largest collection of ready-made robot software that exists, and for most teams that is a heavier loss than any gain in how data moves. The sensor drivers are the immediate cost: somebody has already made your specific camera, your specific lidar and your specific arm work with ROS 2, and reproducing that is unglamorous work with no upside. The navigation and mapping stacks are years of accumulated fixes for situations you have not encountered yet. The visualisation and recording tools are how you debug a robot that misbehaved twenty minutes ago in a room you were not in. Hiring changes too, because ROS 2 experience is a line on a résumé and any smaller ecosystem is something you must teach. Community answers matter more than people admit: when you are stuck at eleven at night, the question is whether somebody else has already been stuck the same way in public. Leaving is a defensible decision when the machine demands it, and it is expensive in exactly these ways rather than in the ways a benchmark would show.

When is ROS 2 the better choice?

ROS 2 is the better choice for most robots that run a model on board, and that is not a diplomatic hedge. If your model is advisory rather than inside the control loop — classifying, detecting, choosing a destination while a conventional controller handles motion — the ecosystem outweighs everything else. If your robot needs mapping or navigation, those stacks exist there and nowhere else in comparable shape. If your programs are spread across more than one computer, ROS 2 with a transport suited to that spread is the right structure and shared memory helps you not at all across a network. If you are hiring, or handing the project to a team that will maintain it after you, ROS 2 is the vocabulary they already have. HORUS is not the answer for a team that needs navigation off the shelf, drivers for unusual hardware, or a stack their next three hires will already recognise. The case for shared memory is narrower and specific: one computer, large frames, a model and a controller that must see the same data without copies between them.

Does the middleware make your model run faster?

No, and here is why: middleware never touches inference. Your model's speed is decided by the model itself, the runtime executing it, the precision you quantised to, and the chip doing the arithmetic. Swapping how programs talk to each other changes none of those. A middleware choice can only affect what happens on either side of inference — how a frame reaches the model, and how a result reaches the thing that acts on it. That sounds minor and often is not, because a result arriving after the moment it describes has passed is useless regardless of how quickly it was computed. This is why teams chasing smoother motion should look at both places before choosing. If your model is genuinely slow, the fix is a smaller model, a better runtime, quantisation, or different hardware, and no messaging layer will rescue you. If your model is fine but the robot is still jerky, the problem lives in the plumbing or in how the controller behaves while waiting, and that is where a middleware decision actually pays.

Is zero-copy messaging the thing that decides this?

Partly, but not the way you think. Avoiding copies matters when messages are large and cross between programs often, which describes camera frames and describes almost nothing else on a robot. Joint commands, poses, states and status messages are small, and how they travel is not what makes a machine jerky. So the question is not whether a middleware avoids copies but whether your robot has the traffic that makes copies matter. A robot with one camera feeding one advisory model does not. A robot with several cameras feeding a model that then feeds a controller, all on one small board, absolutely does. The second half people miss is that removing copies does not fix a controller that blocks while waiting for a result. Both problems must be solved, and only one is a middleware decision. What zero-copy messaging is and why roboticists care covers the mechanism, which is worth understanding before you let it decide a stack for you.

How do you decide what to build on?

Answer three questions about your robot and the choice mostly makes itself. Does the model sit inside the loop that keeps the machine steady, or does it advise a controller that runs regardless? Does everything run on one computer, or are programs spread across several? Is the traffic between programs mostly large frames, or mostly small messages? An advisory model, several machines, or small messages all point at ROS 2 and its ecosystem, and you should take it without agonising. A model inside the loop, one computer, and large frames point at shared memory. If you cannot answer the questions yet, you do not have a middleware problem — you have an architecture you have not measured, and the honest next step is instrumenting the robot until you can say which of the three causes is producing your symptom. Teams that skip that step tend to change frameworks and find the same jerk waiting for them on the other side, having spent a month on the move.

Decide by situation rather than by preference:

• If your model advises and a conventional controller drives -> ROS 2, because the ecosystem is worth more than anything else on offer.
• If your programs are spread across several computers -> ROS 2 with a transport built for that, because shared memory does nothing across a network.
• If a Python model and a faster controller share one board and large frames -> a shared-memory middleware, because copies are what you are losing.
• If your demo is in three weeks -> whatever your team already knows, with the model and the controller as separate programs from the start.
• If the robot is jerky and you cannot say why -> instrument before switching, because the cause may not be the plumbing at all.

When two options stay close, weigh them on the five axes of the HORUS Fit Framework: ecosystem size, setup effort, team size fit, deployment target, and licence. Take whichever loses on fewest — five plain questions about your situation rather than about the software, with no scores attached. And if your robot is heading towards a model and a controller sharing one on-board computer, star HORUS on GitHub so it is in your list when you start building.