What Sits Between the Model and the Motor?

Between a model and a motor sits a timing layer you either write yourself or inherit from middleware such as ROS 2 or HORUS. That layer reads sensors on a rhythm, carries frames between the program that thinks and the program that drives, and halts the machine when readings stop arriving. You can skip it only while a person's hand stays on a physical stop. The rest of this post is for people who train or prompt models and are now deciding what to put underneath one that is bolted to hardware.

The policy works. In a notebook it takes a frame, returns a pose, and the numbers look reasonable every time you run it. Then somebody bolts it to an arm.

The arm does not move the way the numbers moved. It reaches, stops, jerks, reaches again. You add a pause and it becomes slow and still jerky. You take the pause out and it overshoots and has to come back.

Then the quiet failures start. The camera drops out for a moment and nothing complains, because the last frame is still sitting in a variable and looks exactly like a frame. The gripper reports closed while holding air. The arm goes for a spot the mug left two seconds ago, and no line of your code considers any of that an error, because there is no error to catch.

So you go reading, and every answer names a different missing piece. One says you need a real-time kernel. One says ROS 2. One says middleware. One says a faster loop. None of them tells you what the missing piece does, only what it is called, and you cannot tell which of them describes the hole in your own project.

Do you need anything between your model and the motors at all?

Yes, and the only genuine question is whether you write that layer or inherit it. A model produces a decision. A motor accepts a command, holds it until the next one arrives, and does not care whether the sender is still alive. Something has to close the distance between those two facts: turn a decision into a command, repeat it on the beat the motor expects, notice when decisions stop coming, and bring the machine to rest. You can write all of that in a few hundred lines, and plenty of working robots do exactly that. What you cannot do is leave it out, because leaving it out means the motor holds its last instruction while the model thinks, and holding the last instruction is precisely what a runaway looks like from across the room. The choice in front of you is not whether this layer exists. It is how much of it you want to be responsible for on the morning it misbehaves in front of somebody who matters.

What is that layer made of, in plain terms?

Three things sit there: a loop that keeps time, a path for data between programs on the same computer, and a stop that still works when everything else is busy. Follow a single decision from end to end. The model looks at a picture and says the hand should be somewhere it currently is not. A trajectory step turns that into a sequence of small moves, because motors do not teleport. A controller runs on a beat, comparing where each joint is against where the current small move says it should be, and issues a correction. A driver turns that correction into whatever the motor controller speaks. Encoder readings travel back up the same chain, and something checks that they are recent rather than merely present. Alongside all of it runs a path that can cut motion without asking the model's permission. Middleware is the second of those three: the part that moves the picture, the goal and the joint states between the programs that produce and consume them, so nobody has to hand-write sockets or files.

What are your actual options for building that layer?

You have seven realistic options, and they differ mainly in how much of the fast half you write and where that half runs. You can keep everything in one Python process, which is where most first attempts live. You can buy an arm whose vendor already ships the fast half and speak to it through their interface. You can put the timing-critical part on a microcontroller and leave the model on the host, which is how a large share of shipped machines are actually built. You can stay in simulation while the policy is the thing being studied. You can buy a finished platform and accept the hooks it exposes. Or you can treat the machine as several programs that must agree on one computer, which is where middleware arrives: ROS 2 brings the largest body of existing robotics code anywhere, while HORUS is an open-source real-time robotics middleware for Rust, Python and C++ in which the three languages share the same shared-memory ring buffers, so messages between processes on one machine are not serialised. Neither one runs your model.

Read those rows as descriptions of teams rather than as a ranking.

What should you pick if your background is models rather than machines?

Buy an arm you did not build, leave your model where it is, and be deliberate about exactly one boundary: the line between the program that thinks and the program that moves. Somebody who trains policies arrives with a real advantage and a predictable blind spot. The advantage is that evaluation, data collection and loop design are already habits. The blind spot is treating everything below the decision as plumbing that somebody will write in an afternoon. It is not plumbing, it is most of the calendar. The specific way it hurts is that the machine works while you are watching, because you restart the script, you nudge the arm back, you rerun after a dropped frame. What you have not built is a machine that behaves identically when nobody is in the room. Get the boundary right early and you can swap models fifty times without touching the machine underneath. Get it wrong and every model change becomes a hardware debugging session, which is the slowest loop available to you.

What hardware decides how much of this layer you have to write?

Whether your motor controller already closes a loop of its own decides more than the processor does. An industrial servo drive or a vendor arm accepts positions and handles current and velocity internally, so you write goals and supervision. A bare brushless motor with an encoder and a cheap driver leaves the entire chain to you, and that is a different project with a different timeline. Check that before anything else. The second question is what the machine does when told to stop: something that goes limp or holds still on command, with a power cut no software can veto, buys you months of forgiveness while your model is wrong. The third is the computer. A model usually wants an accelerator and a control loop usually wants to be left alone, and putting both on one small board is normal and is also where teams discover that thinking and timing compete for the same machine. The tell is an arm that moves smoothly until the model is asked a question, then hesitates mid-reach.

What if somebody wants to see the robot move within a month?

Buy a machine that already moves, drive it from the model you already have, and write no middleware at all. A month is enough to demonstrate a behaviour and nowhere near enough to build a robot, and confusing the two is how demonstration deadlines die. The version that works: a vendor arm or a small mobile base with a supported interface, one program that grabs a frame, asks the model and sends a goal, and a hard-wired stop within reach of whoever is standing there. The version that fails: choosing a framework in week one, fighting its build system in week two, discovering in week three that the driver for your camera is a community fork somebody abandoned, and arriving at the demonstration with a system that starts cleanly and has never picked anything up. If the date matters, spend the first week on hardware you can physically get hold of and the second on making the machine move properly with the model switched off entirely.

What if nobody on the team has written code that must finish on time?

Have one person write a loop that holds a beat before the model is involved, because the idea takes an afternoon to learn and discovering it under deadline takes months. A control loop repeats on a rhythm: read where things are, work out the difference from where they should be, send a correction, wait, repeat. The loop has to hold that rhythm while the rest of the machine is busy, which means it cannot wait on a log line being flushed, a file being written or a model finishing a thought. The arrangement that works is a division of labour by rhythm: the model emits a goal now and then, and a loop underneath drives smoothly towards whatever the latest goal is. A machine built that way moves like a machine. A machine whose motors hear from the model directly moves like a model, which is to say in fits. The timing side of a control loop is worth an afternoon before you need it.

What do people put between the model and the motor first, and why does it stop working?

Almost everybody starts with one loop that captures a frame, runs inference and sends a command in sequence, and that arrangement fails the first time the machine has to do two things at once. Starting there is correct: one file, no infrastructure, quick to change. It breaks in a predictable order. Capture, inference, motion and safety all share one sequence, so the slowest of them sets the pace for everything, and that pace is never steady. Threads come next, which converts a slow machine into an unpredictable one. Then queues, then shared state, then somebody notices the arm has been acting on a picture from three moves ago. The arrangement that survives splits work by rhythm rather than by feature: fast things that must never wait, slow things allowed to think. Once split, the only remaining question is how the two halves share a camera frame on one computer without either one waiting. That is the same wall behind why a model that writes code cannot pick up a cup.

What changes as the robot is asked to do more?

The model barely changes and the layer beneath it multiplies. A supervised arm on a bench needs one camera, one loop and a person nearby, and a single script genuinely covers it. Add wheels and you inherit localisation, obstacles and usually a second computer. Add unattended running and you inherit recovery behaviour, freshness checks and logs somebody can read the following morning. Add a second sensor and you inherit the question of which reading belongs with which frame. Through every one of those steps the model is the same weights behind the same interface. There is a second effect that catches teams by surprise. A supervised machine has a person as its last line of defence, and an unattended one does not, so the cost of one missed reading rises at exactly the moment the human leaves. That is why a team can live on one script happily for six months and then hit a wall inside a week. The machine did not get worse. The standard it is being held to got higher.

What do you give up by keeping the model out of the fast loop?

You give up the single-file simplicity where one edit fixes everything, and that loss is real rather than rhetorical. Two programs mean two things to start in the right order, two places to look when something is wrong, and a message shape that both sides have to agree on and keep agreeing on as the project changes. You give up the ability to step through one decision in one debugger from picture to motion. You take on deployment: something has to launch both halves on boot and notice when one dies. You also give up a genuine research direction, because some policies are meant to run the fast loop themselves, reacting to contact within their own inference rather than handing goals to a controller. If that is your project, the split is the wrong architecture and you should say so early. For everybody else, the trade is a boundary you design once against a class of failure that otherwise never fully goes away.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the robot has to move around a building, and whenever most of its value is code somebody else already wrote. Mapping and navigation are the clearest case: a mobile base that must build a map and drive to a goal is a ROS 2 project, because those packages represent years of work nobody should reproduce alongside their model work. ROS 2 also wins when your depth camera or lidar ships a ROS 2 driver and nothing else, when the system spans more than one computer, when you want a recording of every message to replay after a bad run, and when the lab or company around you already speaks that vocabulary. HORUS is not the answer in those situations, and choosing it there means rebuilding plumbing you could have inherited for free. There is one more honest case: if the model is a single step inside an otherwise conventional pipeline, ROS 2 already has a well-worn place to put that step, and connecting a model to a real robot gets easier rather than harder.

Is the layer under the model just a driver you can download?

No, and here is why: a driver moves bytes to a motor controller and stops there, while the part you are missing decides when those bytes go out, what happens when they are late, and who is allowed to interrupt them. Drivers are real and you should absolutely download them rather than write them. But a driver has no opinion about rhythm. It will happily send a command at an uneven pace, send one built from a stale reading, or send nothing at all for a while and report no problem, because none of those things is its job. The questions a driver does not answer are the ones that decide whether your machine is trustworthy: how old is too old for this reading, what does the arm do when the model has said nothing for a while, what happens on the first command after a restart, and which code path can stop motion when the main program is stuck. Every one of those is yours, whether you write it or adopt something that has answered it already.

Will end-to-end models remove this layer entirely?

Partly, but not the way you think. End-to-end policies genuinely do remove work, and they remove a specific kind: the hand-written behaviour in the middle. Perception pipelines that once took a team a year now come out of a checkpoint. Task sequencing that filled a state machine collapses into a policy and a handful of skills. Some policies really do emit joint targets on a steady beat, which absorbs part of the controller too. That trend is real and it will continue. What does not shrink is everything beneath the decision. A better policy still does not decide what happens when frames stop arriving, still does not stop the arm when the cell door opens, still does not get its own input delivered on time, and still does not make a command reach a motor before the next cycle needs it. If anything, a stronger policy raises the demands on that layer, because a machine trusted to act alone gets deployed where a supervised machine never went. The middle is being eaten. The bottom is not.

How do you work out which part of this you actually need?

Ask what your machine would still need if the model were always right, and whatever remains on the list is the layer you have to build. The exercise takes five minutes and settles most arguments in a team. Imagine an oracle that emits exactly the correct next action forever. Does the machine still need a reading on a rhythm? Yes. Does it still need to stop when somebody reaches into the workspace? Yes. Does it still need two programs on one board to share a camera frame without one waiting on the other? Almost certainly, if inference lives in Python and motion does not. Nothing on that list improves when your model improves. Now run it backwards: list the behaviours you could not write down as rules if you tried. A short second list means you have an automation problem wearing a model as a hat, and rules will serve you better and fail more legibly. A long one means the model is the point, and your bottleneck is data and hardware rather than architecture.

Decide by situation rather than by ambition:

• If you have a policy, a bench arm and a person watching -> one process, because nothing you add before that point pays for itself.
• If inference is in Python and the loop must not wait for it -> two programs on one board with a shared-memory transport, because copying frames between them is where the motion goes.
• If the robot must map a building and drive to a goal -> ROS 2, because navigation is borrowed code and rewriting it is not your project.
• If the fast half is small and must never pause -> a microcontroller, because the host will always be busy at the wrong moment.
• If somebody wants a demonstration next month -> a finished platform and one script, because a month buys a behaviour and not a robot.

When two options stay close, weigh them on the five axes of the HORUS Fit Framework: ecosystem size, setup effort, team size fit, deployment target, and licence. Take the one that loses on the fewest, and notice that all five are questions about your situation rather than about the software. If your project keeps landing on one board with a model in Python and a loop that must not wait for it, star HORUS on GitHub so it is in your list when you start building.