What AI Engineers Misunderstand About Robot Timing

AI engineers treat robot timing as average speed, when what matters is arriving on time every cycle — the gap ROS 2 and HORUS both address. A model is judged on how it does on average; a machine that moves is judged by its worst cycle, because a single late command becomes a visible jolt. That stops mattering only for a light bench arm that somebody is always watching. The rest of this post is for engineers who train models and now have to make one drive hardware that does not wait.

Your robot is fine when it is idle and strange when it is working. The arm reaches smoothly in an empty scene, then develops a small shudder the moment the camera pipeline is running. Tracking a moving object, it stays perfectly on target in a straight line and drifts wide on every turn. The motors make a faint noise that was not there last week. Nothing in the logs looks wrong, no exception is raised, and every unit test passes.

So you do what has always worked: you profile. You find the slow function, you make it faster, and the shudder is still there. You try a smaller model and the shudder is smaller but has not gone. Somebody suggests the problem is the scheduler and somebody else says it is garbage collection and a third person says you need a different kernel, and none of them can tell you how to check which. The frustrating part is that your instinct — make the slow thing faster — is the correct instinct for every system you have ever optimised, and here it keeps producing improvements that do not fix the symptom.

Does your robot actually have a timing problem?

You have a timing problem if the motion degrades under load and cleans up when the machine is idle, and you do not have one if the robot misbehaves the same way on every single run. That test separates two families of fault that feel identical from the outside. A timing fault is load-dependent and intermittent: the arm is smooth in an empty room and shudders once perception starts, or the robot is fine for a few minutes and gets sloppy as the board warms up. A logic fault is consistent: the same wrong reach, the same offset, the same overshoot, every attempt, whether the machine is busy or not. Run the robot with everything else switched off, then run it again with the whole system loaded, and compare the motion rather than the numbers. Most teams who suspect timing find on that test that they have a coordinate frame error or a stale calibration, both of which are much cheaper to fix and much easier to mistake for something exotic.

What does timing mean on a robot, in plain terms?

Timing on a robot means three separate things, and confusing them is the whole misunderstanding. The first is how long a single step takes from input to output — the thing you already profile. The second is how much that duration varies from one step to the next, which barely matters in a web service and matters enormously to a joint, because a controller is designed around an even rhythm and a wobbly rhythm makes its corrections wrong rather than merely late. The third is whether a step ever finishes after the moment its answer was needed, which is a different question entirely: work that is on average quick can still be too late occasionally, and occasionally is what you feel in the hardware. A robot lives or dies on the second and third of these, while model engineering lives almost entirely on the first. Latency, jitter and determinism explained simply unpacks the vocabulary properly, and it is worth reading before you argue with anyone about it.

What are your actual options for keeping a robot on time?

You have about seven realistic options, and they sort by how much of the machine's rhythm you take responsibility for. ROS 2 with its usual setup is the baseline, and it is enough for a great many robots: the control framework already exists, and the message settings needed to stop a slow consumer from blocking a fast producer are documented. ROS 2 on a preemption-focused kernel, with priorities and reserved cores, is the same thing tuned, and is what most serious ROS 2 machines actually run. HORUS is an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages are not serialised between processes on one machine, which removes the packing and copying of large messages as a source of variation when a Python program and a faster one share a computer; like ROS 2 it is Apache-2.0. You can also collapse everything into one process, push the inner loop onto a microcontroller, take the vendor's own control layer, or run a real-time operating system underneath the whole thing.

What changes if your background is training models?

Three habits from model work stop being true, and unlearning them is most of the transition. First, the average stops being the thing you optimise. A training loop is judged by throughput over hours, so a stall matters only in aggregate; a robot is judged by the one cycle that arrived late, because that cycle is a jolt somebody watched. Second, you lose the retry. When a batch fails you rerun it, and when a request is slow the user waits; a joint that missed its moment cannot be given the moment back, and the machine has already moved. Third, batching turns from a friend into a trap. Waiting to accumulate work is how you make a model economical and how you make a controller unsteady, because the wait is invisible in your metrics and visible in the arm. The mental shift is from optimising the middle of a distribution to defending its worst end, and it changes which fixes are even worth trying.

What should you expect on one small on-board computer?

On one small board, expect the timing to be decided by contention rather than by any single slow function. Everything on that computer shares the same cores, the same memory bandwidth and the same thermal budget: your model, your controller, the camera driver, the logger, and whatever the operating system decided to do this second. Work that ran cleanly in isolation now waits behind something else, and the waiting is not evenly distributed. Two effects surprise people. The board gets hot and quietly slows itself down, so a robot that was steady in the first minutes gets sloppier as a session goes on. And accelerator work does not run for free — moving a large observation to and from the accelerator occupies the same bus everything else is using. The practical response is to reserve cores for the loop that must not wait, keep large data out of the path between programs where you can, and treat the robot warming up as a test condition rather than an edge case.

What can you actually fix before a demo next month?

Before a demo, fix the structure and leave the kernel alone, because structural fixes are quick and reversible and kernel adventures are neither. Four changes carry most of the benefit. Split the model and the controller into separate programs, so a slow prediction cannot freeze the machine mid-motion. Make the controller act on the newest available result and throw away everything older, rather than working through a queue that grows all evening. Write down what the robot does when results stop arriving entirely — hold, coast, or relax — and put that behaviour in code that runs even when the producing program has died. Get logging, disk writes and anything that talks to a network out of the loop that commands joints. None of these require a new middleware, a new language or a rebuilt operating system, and together they remove the failures that actually ruin demos. Tuning priorities and reserving cores is real work with real gains, and it belongs to the month after the demo.

What should you do if you have never written a control loop?

Use somebody else's control loop and spend your attention on what you put inside it, because that is where beginners cause damage. ROS 2 ships a control framework, most vendor SDKs include one for their own arm, and both have already solved the rhythm, the hardware interfaces and the mode switching. Take one. Then learn the short list of things that must never happen inside the loop that commands joints: calling the model, writing to disk, waiting on a network, taking a lock that a slow program also takes, and allocating memory in a language where allocation can pause everything. Every one of those turns an occasional hiccup elsewhere into visible motion. The second thing to learn is what the loop should do when its inputs are stale, which is a decision rather than a technique, and one you should make deliberately on paper before you make it accidentally in code. What a control loop is and why its timing matters is the ground-level version.

What does bad timing look like on the actual machine?

It looks like a machine that seems mechanically faulty, which is why teams spend weeks checking hardware that was fine all along. The classic signs are worth memorising. The arm shudders on a reach that should be one smooth motion. The motors emit a faint whine or buzz that appears only under load. Tracking is accurate in a straight line and drifts wide on every turn, because the correction arrives after the direction has already changed. A mobile base weaves slightly instead of driving straight. The robot behaves for the first minutes of a session and gets sloppier as the board heats. Everything is perfect on the bench and unsteady with the covers on. Underneath, these share a small set of causes: the controller is blocked waiting on something slow, the rhythm is uneven because the loop competes with other work, or commands are being computed from data that already went stale. All three are invisible to a profiler that reports averages, and all three are obvious the moment you record the interval between commands and look at the worst ones.

What do teams try first, and why does it stop working?

They add threads, and it stops working because threads change who waits rather than removing the waiting. The sequence is remarkably consistent. First, threads: perception moves to its own thread and the shudder improves slightly, because the loop is no longer blocked every cycle, but now the command sometimes comes from the previous observation in a pattern nobody can reproduce. Second, a queue between them, which fixes the tearing and introduces a new problem, because the queue fills and the robot starts acting on beliefs from a while ago. Third, tuning the sleeps, which works on the bench and stops working when the load changes. Fourth, faster hardware, which raises the average and leaves the worst cycles roughly where they were. Fifth, a rewrite in a faster language, which helps genuinely but costs a month and does not by itself fix a controller that still waits on a prediction. The step that actually resolves it is usually structural: separate the loops, drop stale data instead of queuing it, and never let the fast loop wait on the slow one.

What do you give up by chasing timing too early?

You give up development speed, and for an early project that is the wrong trade almost every time. A loop built to never pause is a loop with rules: no allocation on the hot path, no convenient library that might block, no logging where logging would be useful, and often a language your team is slower in. Those rules are correct for a machine that has to hold its rhythm and expensive for a robot whose behaviour is still changing weekly. You also give up ecosystem reach if you leave the biggest one to get timing behaviour you could have configured where you already were. And you give up the option of being wrong cheaply, because a stack chosen for timing is harder to walk back than a stack chosen for convenience. The honest sequence is: get the robot doing the task at all, find out whether the motion is actually load-dependent, and only then pay for rhythm. Teams that invert this often ship a very punctual robot that does the wrong thing.

When is ROS 2 the better choice?

ROS 2 is the better choice for most robots with a timing complaint, and that is not a hedge. A great many teams who believe they need something else need message settings that stop a slow subscriber from holding up a fast publisher, a control loop that is not sharing a process with inference, and a kernel configured the way ROS 2 documentation has described for years. If your robot has to navigate or map, those stacks exist there and nowhere else in comparable shape. If your programs sit on more than one computer, shared memory does nothing for you across a network, and the transport question is a different question. If you are hiring, ROS 2 is what candidates already know. HORUS is not the answer for a team that needs mapping off the shelf, drivers for unusual hardware, or programs spread across several machines, and the project's validation so far is in simulation rather than across a fleet of shipped robots. The narrow case is one computer, large messages, and Python beside a faster language.

Is the fix just making the model produce results sooner?

No, and here is why: a controller that waits for a result is unsteady no matter how quickly the result comes. If your joint commands are produced only when a prediction lands, then the rhythm of your machine is the rhythm of your model, and models are irregular by nature — a slightly harder scene, a cache miss, another process touching the accelerator, and this cycle takes longer than the last. Making the model quicker narrows the variation without removing it, which is exactly why teams report that a smaller model helped and did not solve it. The structural fix is to stop coupling the two. Let the controller run on its own rhythm and use the most recent result it has, whether that result is new or a few cycles old. Then a slow prediction produces a slightly stale target instead of a missed command, and stale targets show up as a gentle lag rather than a shudder. Genuinely slow inference is still worth fixing, but as a separate problem with separate tools.

Is Python the reason the robot stutters?

Partly, but not the way you think. Python is a real source of timing variation in the loop that commands joints, mostly through garbage collection pauses and lock contention, and that is why control loops so often end up in C++ or Rust. But Python is rarely the reason a robot stutters in the first month, and swapping languages is an expensive way to discover that. Far more common causes come first: the controller is waiting on inference, a queue has filled with stale observations, logging sits inside the hot path, or the loop is competing with perception for the same cores. Fix those and many robots become steady while still running Python everywhere. The place Python genuinely has to go is the innermost loop of a machine that is fast, heavy or both — and the migration is far cheaper if that loop was already a separate program, which is the argument for splitting them early even when you have no intention of rewriting anything. Do AI robots still need traditional robotics software? covers what that split looks like.

How do you tell which timing problem you have?

Record the interval between joint commands and look at the worst intervals, not the average, and the answer usually falls out in an afternoon. If the intervals are even when the system is idle and ragged when perception runs, the loops are coupled and the fix is structural. If the intervals are even but the robot acts on stale beliefs, you have a queue that should be a latest-value slot. If the intervals are ragged even with nothing else running, the loop is losing the processor to something and you are in scheduling and priority territory. If the intervals are perfectly even and the motion is still wrong, you do not have a timing problem at all and you should go and check your frames and your calibration. This ordering matters because the four causes have entirely different fixes, and the popular advice — rewrite in a faster language, change middleware, buy a better board — is aimed at the least common of them. Measure the intervals before you spend a month on any of it.

Decide by situation rather than by instinct:

• If the motion is smooth idle and ragged under load -> separate the loops first, because coupling is the most common cause and the cheapest to fix.
• If the robot acts on old beliefs -> replace queues with a latest-value handoff, because a growing queue is a robot living in the past.
• If the machine is fast or heavy enough to hurt something -> put the inner loop on a microcontroller, because a rhythm that survives a busy computer has to live outside it.
• If large messages cross between programs on one board -> a shared-memory middleware, because packing and copying is variation you can simply delete.
• If your programs sit on several computers -> ROS 2 with a transport built for that, because shared memory does nothing across a network.

When two options stay close, weigh them on the five axes of the HORUS Fit Framework: ecosystem size, setup effort, team size fit, deployment target, and licence. Take whichever loses on fewest — five plain questions about your situation rather than about the software, with no scores attached. And if your machine is heading towards Python and a fast control loop on one on-board computer, star HORUS on GitHub so it is in your list when you start building.