What an AI Model Cannot Decide For Your Robot

A model cannot decide a robot's rhythm, its limits, or what it does when the model is late; ROS 2 and HORUS hold that layer. Those choices exist whether or not anyone makes them deliberately, and a robot that has not made them has made them by accident, usually badly. The verdict flips only if the machine cannot hurt anything and a person's hand is on the stop. The rest of this post is for people who train or prompt models and are now writing the code that stands between a model and a motor.

The demo went well. The model looked at the table, said something sensible about the mug, and the arm went and got the mug. Everybody in the room understood what had happened, which almost never occurs in this field, and for about a day it felt like the hard part was over.

Then the questions started arriving from people who build machines for a living, and they were not the questions you were braced for. Nobody asked how the model works. They asked what the arm does between predictions. They asked what happens if the model asks for the gripper to close when it is already closed on a finger. They asked what happens when the laptop running the model goes to sleep, or the network hiccups, or somebody unplugs the camera while the arm is extended over the bench.

You do not have answers, and worse, you are not sure the questions are yours to answer. Surely something in the stack handles that. So you go looking for the thing that handles it, and slowly it becomes clear that no such thing exists, that these are decisions, that somebody has to make them, and that the somebody is you.

What can an AI model not decide for your robot?

A model cannot decide anything that has to be true at a specific moment whether or not the model has produced an answer. That is the whole boundary, and it is worth memorising in that form, because it sorts every borderline case correctly. Concretely, a model cannot decide the rhythm the joints are commanded on, because the rhythm has to hold even while the model is thinking. It cannot decide the limits on speed, force and joint range, because a limit that can be argued with is not a limit. It cannot decide what the machine does when the model itself produces nothing, since by definition it is absent from that decision. It cannot decide what happens when two things ask for conflicting motion at once. And it cannot decide what the robot does when its own inputs go stale, because the model has no way to know that the picture it is looking at is old. Every one of those has to be written down by a person, in code, in advance.

What kind of decisions are we actually talking about?

They are the decisions a machine makes constantly and silently, and there are roughly six of them. First, timing: how often the joints get a new command, and what the machine does on the cycles where nothing new has arrived. Second, limits: the ceilings on speed, force and joint range that hold no matter what any higher layer requests. Third, arbitration: what happens when a policy, a teleoperator and a safety behaviour all want the arm somewhere different in the same instant. Fourth, staleness: how old an observation is allowed to be before acting on it is worse than not acting. Fifth, failure behaviour: what the machine does when a program dies, a cable is pulled, or a message stops arriving. Sixth, recording: what gets written down so that the argument afterwards about what happened can be settled with data rather than recollection. None of the six is glamorous. All six are load-bearing, and all six exist in your robot right now whether or not anyone chose them.

What does a robot look like when nobody made these decisions on purpose?

It looks fine for weeks and then fails in a way nobody can reproduce. The classic version goes like this. The model is a little slow, so the arm is commanded from whatever the last prediction said, and since nobody decided how long a stale command should be honoured, it is honoured forever. Most of the time a prediction arrives before that matters. Then one afternoon the process running the model is paused by the operating system for a moment longer than usual, and the arm continues confidently toward a position that made sense a second ago and does not any more. Nothing errored. No log line says anything unusual. The only record is a dent in the bench and four people who disagree about what they saw. The second common version is quieter: the arm has been slightly wobbly for months, everyone assumes the hardware is cheap, and the real cause is that inference and the control loop share a process, so every prediction steals the cycle that would have kept the arm steady. What happens when the model is slower than the robot is that failure in detail.

What are your options for making these decisions explicit?

You have about seven options, and they differ mostly in how many of the six decisions come pre-made versus how many you write yourself. ROS 2 with its usual setup gives you a shape for all of them: a control framework with a defined rhythm, lifecycle states that say what a component does when it is not running, recording tools, and a large community that has already argued about the sensible defaults. A vendor SDK with your own Python loop leaves every decision to you but keeps the surface small enough to hold in your head. HORUS is an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages are not serialised between processes on one machine, which suits the case where a Python model and a faster controller live on one board and you want the boundary between them to be explicit rather than accidental; like ROS 2 it is Apache-2.0 and it is validated in simulation. Below all of these sits the option most teams reach for eventually, which is putting limits and the stop into firmware where no software mistake can reach them.

What should you do if you are the model person and there is no robotics person on the team?

Write the six decisions down as a list, with your current answer beside each one, even when the honest answer is "nobody chose". That document takes an hour and is the single most useful artefact a model-first team can produce, because it converts invisible defaults into visible choices that can be argued about. Then attack them in order of how badly the machine behaves when each one is wrong, which almost always puts failure behaviour first and recording second. Do not try to become a controls engineer in a fortnight; instead learn to read what your existing controller already does, since most of these decisions have a default hiding somewhere in a vendor library that you can find and either accept deliberately or change. And get a hardware stop wired before you get anything else right, because it is the one answer that works even when every assumption in your list turns out to be wrong.

What changes if the model runs on the robot instead of a server?

The staleness decision stops being theoretical and becomes the one that governs the machine. When inference runs on a server, everyone can see the network in the diagram, so somebody thinks about what happens when the network is slow, and often handles it. When inference runs on the same board as the control loop, the connection is invisible, so nobody thinks about it, and the two processes quietly compete for the same cores. That competition is worse than a network, because a network at least tells you when it fails. On board, a busy model simply makes the controller late, and lateness in a control loop is felt in the machine rather than reported in a log. This is where the way messages move between the two processes becomes a real design decision rather than plumbing, and where large observations crossing a process boundary deserve deliberate thought. What sits between the model and the motor traces that path from a prediction down to a moving joint.

What should you do if someone is watching it run next week?

Pick the two decisions that turn a bad demo into a broken machine, and make only those. Those two are failure behaviour and limits. Failure behaviour means deciding, and then testing, what the arm does when the process holding the model disappears: kill it deliberately, with the arm extended, and watch. If the answer is "keeps going", you have found the thing worth your week. Limits mean ceilings on speed and joint range enforced below the model, so the worst prediction anyone can imagine still cannot produce a motion that damages anything. Everything else on the list can wait. Do not restructure the stack, do not migrate anything, and do not add a component you have never run before, because a demo is exactly the wrong moment to meet a new failure mode. Also record the session, because if it does go wrong you want to be able to answer the question that will be asked afterwards.

What if you have never written code that runs on a fixed rhythm?

Then use a loop somebody else has already written, and spend your effort learning to read what it does. Code that runs on a rhythm is a real specialism and it is unforgiving in a way ordinary application code is not: a mistake does not throw, it just makes the machine feel wrong in a way that is hard to describe and harder to bisect. Vendor controllers, the ROS 2 control framework and motor-controller firmware all contain a loop that has already survived contact with real hardware. What you should learn, and it is a week rather than a season, is the three questions to ask any such loop. What rhythm does it run on, what does it do when its input is stale, and what does it do when two commands conflict in the same cycle. If you can answer those three about your existing loop, you understand your machine better than most teams understand theirs.

What do you give up by writing these decisions down yourself?

You give up speed early and a certain amount of dignity in meetings. Writing the decisions down forces you to admit, in front of colleagues, that the impressive demo was resting on defaults nobody chose, and that is an uncomfortable half hour. It also slows the first month, because each decision invites an argument and some of those arguments are genuinely hard, particularly arbitration between a policy and a human operator. You will also produce a document that ages, and an out-of-date decision list is arguably worse than none, since people trust it. The costs are real. What you get back is that failures become explicable: when the arm does something strange, you can point at a line in the list and say either "that decision was wrong" or "that decision was never made". Teams without the list argue about what happened instead, and that argument never converges.

When is ROS 2 the better choice?

ROS 2 is the better choice for most teams building a machine that other people will eventually operate. It has already made a defensible version of most of these decisions and written down why. Lifecycle states give you a vocabulary for what a component does when it is not running. The control framework gives you a rhythm and a place to enforce limits. The recording and visualisation tools mean the argument about what happened can be settled by replaying it. If the robot moves around, ROS 2 hands you mapping and navigation that would otherwise be years of work. If the robot spans several computers, ROS 2's networking is built for that. If you will hire, this is the vocabulary candidates already speak, and if you collaborate with a university or a supplier, it is the shared language. HORUS is not the answer when what you need is a driver, a planner, a mapper or an established convention that ROS 2 already ships, because choosing it there means writing all of that from nothing.

Can you just put the safety rules in the prompt or the reward?

No, and here is why: a rule expressed to a model is a preference, and a rule enforced below the model is a constraint, and only one of those holds when the model is absent. Put "never move faster than a walking pace" in a prompt and the model will usually respect it, which is the problem, because usually is a word that has no place near a moving machine. The constraint has to survive three situations the prompt cannot. It has to hold when the model produces something malformed. It has to hold when the model produces nothing at all, which is when the machine is at its most dangerous, since a limit written into a prediction cannot exist inside a prediction that never arrived. And it has to hold when the process is killed mid-motion. Reward shaping has the same shape of failure: it changes what the policy tends to do, not what the machine is capable of doing. Can an AI agent run a robot unsupervised covers the rest of that argument.

Does a better model shorten this list?

Partly, but not the way you think: better models remove work from the middle of the stack and add none to the bottom. A stronger model genuinely does replace things that used to be hand-written, and anyone who denies it has not watched a good one work: the tuned state machine, the scripted approach sequence, the hand-built perception pipeline. That middle layer really is shrinking, and it will keep shrinking. What a better model does not touch is the six decisions, because they are not about competence. They exist because a robot is a physical object with momentum that keeps moving while software is thinking, and no amount of model quality changes the fact that a process can be paused, a cable can be pulled, or a message can fail to arrive. If anything, a stronger model raises the stakes: it will attempt more ambitious motions, closer to people, with more confidence, which makes the layer underneath more important rather than less. Can you fine-tune a robot the way you fine-tune a model makes the same point from the training side.

How do you find the decisions your robot is already making by accident?

Break things on purpose, one at a time, and write down what the machine did. This takes an afternoon and it is the highest-value afternoon available to a model-first team. Kill the process holding the model while the arm is mid-motion, and watch. Unplug the camera and watch. Send a command that asks for a position outside the joint range and see what stops it, if anything does. Pause the model process for a few seconds using an ordinary operating-system signal, and see whether the arm holds, drifts or continues. Send two conflicting commands in one cycle and see which wins. Each of those experiments turns one invisible default into a written answer, and about half of the answers will surprise somebody on the team. Do all of this with the arm over an empty bench, slowly, with a hand near the stop, and do it before anyone external ever sees the machine run.

How should you decide what to write down first?

Order the six decisions by what the machine does when each one is wrong, and start where the answer is physical damage. Failure behaviour is almost always first, because the absence of a command is the most dangerous input a robot receives and the one nobody tests. Limits are second, because they are the only answer that holds when everything above them is wrong. Recording is third, and it feels administrative right up to the first incident, after which it feels like the most sensible thing anyone did all quarter. Timing, staleness and arbitration come after that, and they are the ones where the choice of stack matters most, since each option in the table above makes them either easy or tedious. Do not attempt all six in one sprint. One decision, written down, tested by deliberately breaking something, and then left alone is worth more than six that exist only as an intention in a document.

Reduced to a decision list:

• If you are running a model on a machine that can injure someone -> limits in firmware plus a hardware stop, because software you can edit is software that can be wrong.
• If you are building something others will operate -> ROS 2, because the decisions come with a shape and a vocabulary already agreed.
• If you are one week from a demo -> failure behaviour and limits only, because those two are what separate an awkward demo from a broken machine.
• If you are running inference beside a controller on one board -> the shared-memory option in the table above, because the boundary between them should be explicit.
• If you are on a desk rig that cannot hurt anything -> one Python process with a watchdog, because the ceremony would cost more than the risk.
• If you cannot say what the arm does when your model process dies -> stop reading and go find out, because everything else depends on that answer.

Weighing those options is what the HORUS Fit Framework is for, and its five axes are deliberately not numbers: ecosystem size, setup effort, team size fit, deployment target, and licence. You can answer all five honestly about your own team in an afternoon, which is more than can be said for most comparisons in this field.

If the one-board case is the robot you are building, star it so it is in your list when you start building: HORUS is Apache-2.0 at github.com/softmata/horus.