Why Does My Robot Behave Differently Every Run?

Your robot behaves differently every run because message order and timing change between runs, and neither ROS 2 nor HORUS removes that alone. The same code can see a laser scan before an odometry update on one run and after it on the next, and each ordering produces a different decision. That verdict flips when the variation comes from the physical world, which no middleware can make repeat. The rest of this post is for anyone whose robot passes a test on Monday, fails the same test on Tuesday, and changes nothing in between.

It always gets described the same way. The robot ran the course clean four times, then clipped the doorframe on the fifth, with the same map, the same battery, and the same starting pose. Nobody touched the code between runs. You try again and it is fine, so you half convince yourself you imagined the whole thing, and then it happens twice in one afternoon in front of a customer. Sometimes the shape is milder: the arm reaches the right spot most mornings and overshoots by a hand's width once a week. Sometimes the robot works when you start the nodes by hand in a particular order and stalls when a launch file starts them all at once. You add a print statement to find out why, and the problem goes away. You take the print statement out, and the problem comes back somewhere else. Then somebody says the words race condition, everybody nods, and nobody can say which race, or where, or how to watch it happen. The maddening part is not the failure. The maddening part is that you cannot cause it on purpose, so you cannot tell whether a fix worked.

Should I hunt down every source of run-to-run variation, or design around it?

Design around it, and hunt only the variation that reaches the robot's behaviour. Every robot varies between runs, because the computer underneath is sharing time between programs, the sensors are reporting a world that never repeats, and every link — even a cable between two boards — delivers messages when it is ready rather than when you asked. Chasing all of that is an infinite job. The question worth answering is narrower: does the variation change what the robot does? A planner that considers obstacles in a different order and produces the same path is varying harmlessly. A safety check that sometimes reads the newest distance measurement and sometimes the one before it is not, because on the run where it reads the older one the arm keeps moving. So the work splits in two. Reduce the variation in the short chain between sensing and acting, where a different order means a different decision, and leave the rest alone. Most teams do the opposite. They try to stabilise the whole system, get nowhere, and never look closely at the four or five steps that actually decide anything.

What does run-to-run variation actually mean in plain terms?

Run-to-run variation means the same program, given the same job, takes a different path through its own code each time it starts. Not a different program, and usually not different data: the same instructions, arriving at slightly different moments, in a different order. A robot is several programs at once — one reading a camera, one reading wheel encoders, one deciding, one driving motors — and none of them controls when the others get to run. The operating system decides that, based on what else the machine is doing. So on one run the decision code wakes up with three fresh readings waiting; on the next it wakes up with two, acts on those, and finds the third a moment later. Both runs are correct in the sense that no rule was broken. They simply are not the same run. Put a network between the programs and the possible orderings multiply, because delivery time now varies as well as wake-up time. The word people reach for is nondeterminism, which sounds like randomness and is not. Nothing is rolling dice. The order is just not being specified by anyone.

What are my actual options for making a run repeat?

Six approaches work, and most projects end up using two or three together. You can record every input during a run and replay the recording, so at least the failure can be examined on a desk. You can force the steps that must agree into a fixed order on one thread, which ROS 2 supports through its executor and callback group settings. You can run the whole thing in simulation with a fixed time step and a fixed seed, so a run repeats exactly and a code change can be judged honestly. You can move the sensing-to-acting chain onto a single computer where processes hand messages to each other through shared memory instead of a network stack, which is what HORUS is built for and what ROS 2 approximates by composing nodes into one process, removing the network as a source of ordering surprises. You can give the timing-critical work a fixed slot on the machine using a real-time kernel and priorities you choose yourself. Or you can leave the ordering alone and write behaviour that reaches the same outcome no matter which message landed first.

Which approach fits which kind of project?

The approach that fits depends on where your variation is entering, not on how serious the symptom looks.

The last row is the cheapest and gets skipped most often, which is why teams argue about the other six without evidence.

What should I try first if I am one person with one robot?

Record one bad run end to end and stop trying to catch the failure live. One person cannot sit and watch for an intermittent fault; the odds are bad and the attention runs out before the robot misbehaves. Capture everything the robot took in during a run — every sensor message, every command, with the time each one was handled — and keep the good runs too, because the comparison is where the answer lives. Then run the robot until you get a bad one. With a good run and a bad run side by side, find the moment the behaviour diverges and look at the last few messages before it. Most of the time the difference is visible right there: two readings arrived in the opposite order, or one never arrived and the code carried on with a stale value. That is a short investigation once the recordings exist and an unbounded amount of guessing without them. Set up the recording before you touch a line of code, because every change you make without it destroys the evidence you were about to need.

What if my robot is a laptop talking to a microcontroller?

Suspect the link between the two before you suspect either side. A laptop and a microcontroller disagree about time in a way that produces exactly this symptom: the microcontroller runs a tight loop with nothing else competing for it, while the laptop runs your decision code alongside a browser, an editor, and whatever the operating system feels like doing. Messages leave the microcontroller evenly and arrive at the laptop in clumps, so a command computed from a reading can rest on something older than you think, and how much older changes with what else is open. Serial and USB links make this worse by buffering, delivering several messages together so your code handles them in a burst, which looks orderly in a log and is not. Two moves help more than anything else here. Timestamp each message where it is produced, on the microcontroller, rather than where it is received. And move anything that must react quickly onto the microcontroller itself, leaving the laptop with work that can afford to wait: mapping, logging, and the interface a person looks at.

What if I have a demo in two weeks and cannot rewrite anything?

Make the robot stop safely when its inputs disagree, and leave the variation where it is. Two weeks is not enough to change how a system is put together, and a half-finished restructure is worse than the original problem. What does fit in two weeks is a guard: before the robot acts, check that the readings behind the action are recent and consistent with each other, and when they are not, slow or stop rather than continue. A robot that pauses and resumes looks careful. A robot that occasionally drives into a table does not, and that is the only part an audience remembers. While you are there, record every demo run, because a failure in front of an audience is the reproduction case you have been unable to produce on the bench and it would be a waste to lose it. Pin down the start-up sequence too if launch order is implicated — starting nodes by hand is not a solution, but removing one variable costs nothing. Restructure after the demo, with recordings in hand.

What if I have never debugged a timing problem before?

Write down the order you believe things happen in, then make the robot tell you the order it actually used. Most people have never done this, and it is the entire skill. The belief is usually something like: the camera reads, the detector runs, the planner decides, the motors move, and the cycle repeats. The reality on a bad run is that the planner decided using the detection from the previous cycle, because the detector had not finished yet. You cannot see that by reading code, because the code says nothing about when it runs. You see it by stamping each message where it is created, then printing, for a single decision, which stamps went into it. When the stamps behind one decision are older than expected, or two of them come from different cycles, the mystery collapses into an ordinary problem with an ordinary fix. Learning to read that is worth more than any framework choice you will make this year, and what middleware actually does in a robot is a sensible next step once the ordering makes sense.

What do most teams try first, and why does it stop working?

Most teams add a sleep, and it works until it does not. The sequence is nearly always the same. First more logging, which changes the timing and moves the fault somewhere else. Then a short wait before the step that seems to run too early, which makes the failure rare enough to ignore for a fortnight. Then a retry around the thing that intermittently fails, which turns a visible fault into a slow one. Then queue sizes get raised, because messages are clearly being lost, and now the robot acts on older data more often rather than less. Each move is locally reasonable, and together they build a system whose timing nobody understands, where every new symptom has four plausible causes. The reason the sequence stops working is that none of those steps changed the ordering. They changed how often one particular ordering shows up. Make the machine busier — a warmer room, a bigger map, a second camera — and the odds shift back. Fixes that depend on odds get undone by the next feature you add.

What do I give up by making my robot repeat itself?

Headroom, some flexibility, and depending on the route, part of an ecosystem. Forcing a chain of steps into a fixed order on one thread means the slowest step sets the pace for all of them, so a heavy vision stage that used to overlap with planning now blocks it. Keeping the critical path on one computer means that computer has to be big enough for the whole path, and you lose the easy escape of moving a hungry node to a spare machine. Pinning work to fixed priorities means deciding in advance what is allowed to be late, which is uncomfortable and occasionally wrong. Stepping outside the ROS package set for part of the system means writing or wrapping drivers that already exist there, and explaining that choice to everyone who joins afterwards. None of these costs is hidden, and all of them are cheaper than an intermittent fault nobody can reproduce. They are still real, and a team that pretends otherwise reverts the change six months later when the vision stage grows.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the parts that disagree with each other do not fit on one computer, or whenever the tooling you need already lives there. Some concrete cases. Your sensors ship with ROS drivers and nobody wants to write a driver, so ROS 2 wins outright. You need navigation or motion planning that thousands of people have already argued over, and rewriting either of those to chase an ordering bug is a bad trade. Your graph genuinely spans machines, with a robot and an offboard computer that both hold state — that is a distribution problem, and HORUS is not the answer, because a shared-memory path between processes stops at the edge of one machine. You work in a lab where a recorded run must be replayable years later by somebody else, and the recording format everyone reads is the ROS one. Or your team already knows ROS 2 well and the variation sits inside one badly written callback rather than in the plumbing. In every one of those, the cheaper fix is inside ROS 2 rather than beside it.

Is my robot varying because my own code has a bug?

Partly, but not the way you think. There is almost certainly a defect in your code, in the sense that something assumes an order nobody guaranteed. But calling that a bug suggests a line you can find and correct, when what you have is an assumption spread across several files: a callback reading a variable another callback writes, a decision trusting the newest reading without checking how new it is, a state machine that only advances when two events arrive the right way round. Fixing the line where the symptom appears makes the symptom move. What removes it is making the assumption explicit — this step needs a reading no older than the current cycle, and here is what happens when there is not one. That is a design change, small but genuine, and it is why "just fix the bug" is unhelpful advice here. The code is not wrong on any single line. The code is right in a world where things arrive in the order they were written down, and no running robot lives in that world.

Will putting everything on one computer make my robot deterministic?

No, and here is why: a single computer removes one cause of variation and leaves the others standing. Deleting the network deletes delivery jitter, dropped packets and the discovery dance between processes, which is a real win and the reason so many teams consolidate eventually. What remains is the operating system choosing which of your programs runs next, memory being allocated at awkward moments, a driver taking longer on some frames than others, and sensors reporting a world that never repeats. A shared-memory path makes the handoff between processes cheap and consistent, so messages are there before the next control cycle needs them and the robot does not stutter. Consistent handoff is not the same as identical runs. If your goal is two runs producing matching logs line for line, you need recorded inputs and a fixed clock, which means replay or simulation rather than a middleware choice. If your goal is a robot that behaves the same way, one machine plus a fixed order through the critical chain gets you most of the distance.

How do I tell which kind of variation I am dealing with?

Run three checks in order and let them narrow it down. First, push the same recorded inputs through the same code twice with no hardware attached: if the two runs differ, the variation lives inside your software, and better sensors or a quieter network will not help. Second, run on hardware but hold the robot still against an unchanging scene, lights on, nothing moving: if runs differ now but agreed in replay, the variation enters through timing, and ordering and scheduling are the fix. Third, let the robot move and watch whether the differences grow: if they do, the world is the source, and the answer is behaviour that tolerates it rather than plumbing that suppresses it. Those three checks map to three different jobs. Software-internal variation is a design fix in your own code. Timing variation is a middleware and scheduling question, which is also where the real-time claim gets confusing. World variation is not a defect at all, and treating it as one wastes months.

A short version, by situation:

• If you are one person who cannot reproduce the failure -> record every run and compare a good one against a bad one, because the divergence is visible in the last few messages before it.
• If you have a small chain between sensing and acting -> force that chain into a fixed order, because that is where a different order becomes a different decision.
• If everything with a deadline already fits on one computer -> keep it there and let the processes share memory, because a network you deleted cannot reorder anything.
• If the variation comes from the world rather than the software -> stop chasing it and make the behaviour tolerate it, because friction and lighting will not repeat for anyone.
• If you are two weeks from a demo -> make the robot stop safely when its inputs disagree, because a safe pause reads better than a lucky run.

When you are weighing middleware rather than chasing one fault, the HORUS Fit Framework compares the options on five things that are not numbers: ecosystem size, setup effort, team size fit, deployment target, and licence. If the word keeps coming up in your team's arguments, what determinism actually means is worth settling before you choose anything.

HORUS is open source under Apache-2.0 and the repository is linked below. Star it so it is in your list when you start building.