What Changes When an AI Model Is Driving the Robot?

When a model drives the robot, timing replaces accuracy as the hard part, and your stack choice narrows to ROS 2 or a shared-memory middleware. The model produces answers at its own pace while the machine keeps moving, so what the robot does between answers decides whether it looks controlled. ROS 2 remains right for most builds; HORUS fits when the model and the controller share one computer. The rest of this post is for people who train models and now have to put one on a machine that moves.

Your policy works. It works on the recorded episodes, it works in the simulator, and it worked the three times you filmed it. Then it runs the actual machine for an hour and you have a different kind of problem, one that does not resemble anything in machine learning.

The robot hesitates in the middle of a reach. It closes the gripper a moment after the moment that would have worked. It performs the same task twice from the same setup and succeeds once, and no metric you have explains the difference. There is no validation split for this. You cannot re-run the failure, because the failure involved a real cup that has since been moved by a real person.

The advice you find is unhelpfully split down the middle. One half discusses architectures and data, which you already understand. The other half discusses transports, threads and process boundaries in a tone implying you should have understood that too. Nobody joins the halves, and the join is exactly where your problem lives: a model that is fine, and a machine that behaves as though it is not.

What actually changes when an AI model is driving the robot?

The hard problem stops being whether the answer is right and becomes whether the answer is still true when it arrives. A model on a dataset is judged against a fixed picture that waits patiently for a verdict. A model on a robot is judged against a world that kept moving while inference ran, so a perfect grasp pose for the scene as it was can be a collision with the scene as it is. Three things change together. Your output has a shelf life, and acting on an expired one produces the small errors that look like a bad model. Your model is now one of several programs sharing a machine, competing for memory bandwidth and processor time with the loop that keeps the robot steady. And failure stops being a number on a chart and becomes a physical event with consequences, so the question of what the machine does when nothing arrives becomes as important as what it does when something does. What sits between the model and the motor traces the whole chain from a camera frame to a joint command.

What is the software between a model and a motor?

Between a model and a motor sits middleware, which is the layer that lets separate programs on a robot exchange data without knowing about each other. A robot running a model is never one program. The camera driver is one, inference is another, something decides the next action, and a controller commands joints, each running at its own pace and each obliged to keep working when the others are slow. Middleware handles the introductions and the delivery, so the model receives frames without knowing which process produced them, and the controller receives decisions without waiting for the process that made them. The architectural consequence is the whole point: middleware is what lets the slow thinking part and the fast reacting part run at different rhythms without one dragging the other down. Every middleware also brings conventions — message shapes, a launch system, a way of naming coordinate frames — and those conventions are what the surrounding ecosystem of drivers and tools is built on. When someone says a robot runs ROS 2, most of what they mean is the conventions.

What can you run a model-driven robot on?

There are about seven realistic foundations, and they differ mainly in how much existing robot software you inherit against how directly your programs share data. ROS 2 with its default transport is the baseline and the right answer for most machines, because the drivers, the navigation stack, the recording tools and the people all assume it. ROS 2 with Zenoh underneath keeps that inheritance and handles several computers or an unreliable network better. HORUS is an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages are not serialised between processes on one machine, which fits the case where a Python model and a Rust or C++ controller sit on one board and must see the same camera data without copies between them; like ROS 2 it is Apache-2.0. Then come the pragmatic options: everything in one process, a robot learning framework's own tooling, the vendor's SDK, or a small layer you write and own.

What should you use if you are a machine learning engineer new to robots?

Use ROS 2, because the half of the problem you have not met yet is the half ROS 2 has already written down. The gaps are rarely where people from machine learning expect. Coordinate frames cost a week and feel like an insult to your intelligence. Matching a camera frame to the joint positions at the instant of capture costs another week, and getting it wrong presents as a policy that has mysteriously degraded since training. Choosing what the machine does when perception stops arriving is a design decision that has no equivalent in offline work. All of that has settled answers in ROS 2 and no answers in a layer you wrote yourself last month. Import one habit on day one regardless of what you choose: never put the model call inside the loop that commands the machine, even when a single process would work today. Splitting them later means rewriting the part of the system you understand least, usually under deadline, usually while the robot is the only copy of the hardware.

What should you use if the model and the motors share one computer?

On a single computer, the deciding factor becomes how many copies of your largest messages are made before they reach the thing acting on them, and that is where a shared-memory middleware becomes worth the trade. A small board has a fixed amount of memory bandwidth and a fixed power budget, and moving camera frames between programs consumes both without producing anything. ROS 2 handles this class of hardware perfectly well when the model advises rather than drives, and the ecosystem stays worth more than any saving. The strain appears when a camera feeds a model, the model feeds a planner, and the planner feeds a controller, all on one board, with copies at every hop. You experience it as a machine that gets hot, drains its battery earlier than expected, and responds late under load rather than one that fails cleanly. Middleware for Raspberry Pi and Jetson robots covers what these boards actually tolerate, including moving perception onto a second computer.

What should you use if the robot has to work in a month?

Use whatever your team already knows, which usually means ROS 2 or a single Python program. A month cannot absorb a new framework, a new robot and a new deployment target at once, and the thing that ruins a deadline is never that the plumbing was suboptimal. It is that the policy was never wired to the arm at all, or that the arm reached for a position the object had already left and swept it onto the floor in front of an audience. Two decisions protect a short schedule whatever you build on. Separate the model and the controller into different programs immediately, so a long inference call cannot freeze the machine mid-motion. Then decide what the robot does when a result does not arrive: hold, finish the current motion, or stop along a safe path. That definition is what a demo survives on, because an audience forgives a robot that is slow and remembers a robot that lunges. Revisit the foundation afterwards, once you know which part actually hurt.

What if your team only writes Python?

Stay in Python for the model and accept that the loop commanding joints will eventually want a different language. Python is entirely reasonable on the perception side, because the heavy arithmetic happens inside the model runtime while the interpreter mostly waits. Where Python becomes visible on a machine is the loop that must hold its rhythm, since a garbage collection pause or a moment of lock contention turns into motion you can watch from across the room. Teams meet this in a predictable sequence. Everything in Python works, then the arm develops a small hitch under load, then somebody discovers the hitch has nothing to do with the model. The encouraging part is that the fix is local: move the innermost loop, keep everything else where it is. That is a small job when the two halves were already separate programs and a large one when they share a process, which is the practical reason to split them before you have any evidence you need to.

What does it look like when a model-driven robot goes wrong?

It looks like a machine with a bad connection rather than a machine with a bug, which is why the diagnosis takes teams so long. Learn the signatures. The arm moves in small jerks instead of one continuous reach. The robot acts on where the object used to be, so it grasps just behind a moving target. It freezes mid-motion for a beat and then resumes as though nothing happened. Under sustained load everything degrades at once and it reads as a hardware fault. Three separate causes produce that same set of symptoms. The controller is blocked waiting for a result. Sensor data is copied so many times on the way in that every decision describes a moment that has already passed. Or the model and the controller are fighting for the same processor cores and neither gets a clean run. The distinction matters because the fixes are unrelated: blocking is fixed by separating loops, copying is fixed by how data moves, contention is fixed by pinning work to cores or moving perception elsewhere.

What do teams try first, and why does it stop working?

Almost everyone starts by calling the model inside the control loop, and it stops working the first time the robot must move and think simultaneously. The construction is the obvious one: read the sensor, run the model, decide, command the motor, repeat. It works on a bench, it works in a slow demo, and it fails as soon as the machine carries momentum, because the world does not politely wait during inference. The second attempt is threads inside one Python program, which helps slightly and introduces a category of bug that only appears under load. The third attempt is separate programs with a queue between them, which is the correct shape and where a hidden policy question surfaces: when the model falls behind, do you work through every stale frame in order or jump to the newest? For a robot acting on the present, the newest is nearly always right, and a queue that quietly prefers the oldest produces a machine that permanently appears to be reacting to the recent past.

What do you give up by leaving the ROS 2 ecosystem?

You give up the largest collection of ready-made robot software that exists, and that loss is usually heavier than any gain in how data moves. Sensor drivers go first: somebody has already made your camera, your lidar and your arm work with ROS 2, and redoing that work produces nothing your users will ever see. Navigation and mapping represent years of accumulated fixes for situations you have not encountered yet. The recording and visualisation tools are how you understand a failure that happened while nobody was watching, which is most failures on a machine driven by a model. Hiring changes as well, because ROS 2 on a résumé is a filter you can use and anything smaller is something you teach at your own cost. Community answers count for more than teams admit, since at eleven at night the useful question is whether somebody has been stuck this way in public before. Leaving is defensible when the machine demands it, and it costs you in these ways rather than in any way a benchmark would show.

When is ROS 2 the better choice?

ROS 2 is the better choice for most robots with a model on board, and that is not a polite hedge. If your model advises rather than drives — labelling objects, reading signs, picking a destination while a conventional controller handles motion — the ecosystem outweighs everything else on the table. If the robot needs to map a space or navigate around obstacles, those stacks exist there and nowhere else in comparable condition. If your programs are spread across several computers, ROS 2 with a transport built for that spread is the correct structure, and shared memory does nothing for you across a network. If somebody else will maintain this after you, ROS 2 is the vocabulary they already have. HORUS is not the answer for a team that needs navigation off the shelf, drivers for unusual sensors, or a stack their next few hires will recognise immediately. The narrow case is specific: one computer, large sensor data, and a model and controller that must see the same data without copies passing between them.

Does better plumbing make the model itself better?

No, and here is why: middleware never touches inference. What your model produces, and how quickly, is decided by the model, the runtime executing it, the precision you quantised to and the chip doing the arithmetic. Changing how programs talk to each other alters none of those. A middleware choice can only affect what happens on either side of inference: how sensor data reaches the model, and how the result reaches whatever acts on it. That sounds minor and frequently is not, because a correct answer that arrives after the moment it describes is worth nothing regardless of how it was computed. The practical consequence is that you should check both places before changing anything. If the model itself is the bottleneck, the fix is a smaller model, a better runtime, quantisation or different hardware, and no messaging layer will save you. If the model is fine and the machine still moves badly, the problem is in the plumbing or in how the controller behaves while waiting, and only then is a middleware decision the answer.

Is the model really in control of the robot?

Partly, but not the way you think. On almost every working robot, the learned part chooses what to do and a conventional controller decides how to move, and the controller runs continuously whether or not a new decision has arrived. That division is what makes a machine look composed instead of twitchy: the joints are being commanded on a steady rhythm from something simple and predictable, while the model contributes goals at whatever pace it manages. Demos that put a model directly in charge of joint commands exist, and they tend to work in one room with one lighting condition. The consequence for your architecture is that "the model is driving" is a description of intent, not of wiring. Somewhere below your policy there is a loop you did not train, which is holding the arm steady, refusing commands that would exceed a limit, and deciding what happens when nothing new arrives. Why language-model robots look great in demos and fail at home is a longer tour of that gap.

How do you tell which problem you actually have?

Record a run and look at the timestamps, because the three common causes leave different fingerprints and guessing between them wastes weeks. If your controller's commands stop appearing whenever the model is busy, the controller is blocked and the fix is separating the loops rather than changing anything about the model. If commands keep flowing but consistently describe an older state of the world than the sensors captured, data is being copied or queued on the way in and the fix is in how programs share it. If everything degrades together as load rises and the board runs hot, the two are competing for the same cores and the fix is pinning work or moving perception onto other hardware. This is also why recording matters more than any framework choice: without a log holding sensor data, model outputs, commands and times, you are debugging a machine by watching it, which does not work for a failure that happens once an hour. What happens when a robot program crashes mid-motion covers the other half of this, which is what the machine does while you are not looking.

Decide by situation rather than by preference:

• If your model advises and a conventional controller drives -> ROS 2, because the ecosystem is worth more than anything else available.
• If perception and control live on different computers -> ROS 2 with a transport built for that, because shared memory stops at the edge of one machine.
• If a Python model and a faster controller share one board and large sensor data -> a shared-memory middleware, because copies are what you are losing.
• If the demo is in three weeks -> whatever you already know, with the model and the controller as separate programs from the start.
• If the robot behaves badly and you cannot say why -> record a run before switching anything, because the cause is often not the plumbing.

When two options stay close, weigh them on the five axes of the HORUS Fit Framework: ecosystem size, setup effort, team size fit, deployment target, and licence. Take whichever loses on fewest — five plain questions about your situation rather than about the software, with no scores attached. If you are choosing a stack for a young company rather than a single robot, the startup version of this decision covers what changes. And if your machine is heading towards a model and a control loop on one on-board computer, star HORUS on GitHub so it is in your list when you start building.