What to Do When Simulation and Hardware Disagree

When simulation and hardware disagree, record both runs and find the first moment they diverge, rather than tuning physics or adding noise to the model. Most disagreements live in the seam between two code paths — timing, sensor reads, a queue — which is why the message layer, ROS 2 or a single-machine runtime like HORUS, belongs in the argument. That flips when the task is contact: grasping and footfalls are physics. The rest of this post is for a team that has a robot on the bench, a simulation that says otherwise, and no agreement about which one is lying.

It works on screen. It has worked on screen twenty times in a row, which is why everybody signed off on it. On the bench the robot does the first part of the task and then does something nobody has seen before: reaches past the object, clips the table edge, oscillates where it was smooth, or simply stops. Someone changes a gain and it fails differently. Someone says the simulation is not realistic enough, so a week disappears into friction values and mesh quality and the robot fails in exactly the same place. Someone else says add noise to everything, so noise gets added, and now the simulated robot is worse as well. In standup the argument has settled into two camps who are not going to convince each other: the simulation is wrong, or the hardware is broken. Both camps are usually wrong, and the reason is the thing nobody has said out loud yet — the code that ran on the bench is not the code that ran on screen, and nobody can point to where the two versions parted company.

What is the first thing to do when the simulation and the robot disagree?

Capture a recording from the real robot and replay it through the simulated version of your code. Not a video — the actual inputs the robot acted on, every sensor reading and every command, stamped with when each one arrived or left. Then push that exact history through the code as it runs in simulation and watch what the code decides. This single test splits the problem in half in an afternoon and settles most standup arguments permanently. If the replayed behaviour matches what the hardware did, then the two sides are running the same code and the difference lives in the physical world: sensing, actuation, contact, something the model does not capture. If the replayed behaviour does not match, then simulation and deployment are not the same code, and no improvement to the physics will rescue that. Teams that run this test stop tuning friction the same week. Teams that skip it can spend a month on parameters chasing something that lives in a queue. Everything else in this post is downstream of which half of that split you land in.

What does it actually mean for a simulation and a robot to disagree?

A disagreement is one of four things, and naming which one you have is most of the work. The first is different code: the simulated path and the deployed path went through different wrappers, different loops, a different way of reading the camera, and the versions drifted apart months ago inside something somebody wrote in an afternoon. The second is different inputs: the same code receiving a sensor value in another frame, another unit, another order, or a frame captured while the arm was somewhere else. The third is different timing: the simulated loop runs on a clean cadence and the real one waits on a driver that occasionally takes longer, so the robot acts on a picture of a moment that has already passed. The fourth is a different world: friction, compliance, backlash, a surface that slips. Only the fourth is what people mean when they say the simulation is not realistic enough, and it is the least common of the four on any task that does not centre on contact.

What are the real options for finding where the disagreement starts?

There are seven approaches in common use, and a healthy team keeps three of them running. Recording and replay is the strongest, because a replay answers what the code would have decided given what actually happened. Bisecting the stack — running one component against recorded inputs while the rest is stubbed — narrows a divergence to a single module. A hardware-in-the-loop bench puts real motors and sensors on the desk with nothing that can fall over. Slowing the robot down turns fast failures into observable ones. Tuning the physics model is the right tool when contact is the subject and the wrong tool otherwise. Then there is the choice underneath all of them: what the two sides share. ROS 2 lets a simulated robot and a physical robot present the same interface, so one stack can be pointed at either without an adapter in between. HORUS is the smaller option — an open-source real-time middleware under Apache-2.0 where Rust, Python and C++ share the same shared-memory ring buffers, so processes on one machine exchange messages without serialising them and a control loop written once runs unmodified on both sides.

Which approach fits which kind of disagreement?

The approaches are not interchangeable, and choosing the wrong one is how a week disappears.

Most teams end up with recording and replay plus one shared layer, and reach for the rest only when the first two run out.

What do teams try first, and why does it stop working?

Teams try three things first, in a predictable order, and all three stop working for the same reason. First comes tuning the physics: friction values, masses, damping, mesh quality. That fails because it is an attempt to change the world to match a robot whose problem is not the world. Second comes adding noise everywhere, on the theory that a simulation that has seen chaos will survive reality. That fails because a difference in kind is not a point inside your noise distribution, and it has a side effect nobody wants: the simulated robot now also behaves worse, so you have lost your reference. Third comes rewriting the controller, usually the part of the code the team understands best and can therefore blame most confidently. That fails because the controller was receiving bad information on time or good information late, and neither is fixed by changing what the controller does with it. What all three share is that they change something before knowing where the divergence starts. The order that works is the reverse: locate first, change second.

What should a two-person team do about it?

Spend a day building the recording path before spending another hour on the robot, because a small team's scarcest resource is the ability to reproduce a failure without being present when it happens. On a two-person team both people are usually in the room during a bench session, both remember the failure differently, and neither can reconstruct what the sensors were reporting. A recording removes the argument entirely and lets whoever is free work on the failure at a desk while the other person keeps the hardware moving. The second habit worth the day is a single switch that points the whole stack at either the simulator or the robot, so nobody has to remember which branch is the real one. Two people cannot afford two code paths — there is nobody to notice when they drift. What a small team should not do is buy more simulation. Fidelity costs time you do not have and answers a question you have not yet shown you are asking, which is the same trap described in why a robot that works in simulation fails in a real room.

What should you do when the robot is one small computer with a camera and some motors?

Suspect timing first on a machine like that, because a single board is where timing problems are most common and least visible. Everything shares one processor: the camera driver, your control loop, the model if you are running one, the logging. In simulation those take turns politely because the simulator waits. On the board they compete, and the loop that ran on a steady cadence at a desk now occasionally runs late, so the robot acts on a picture of a moment that has passed. The visible symptom is an arm reaching for where the object used to be, or a base that oversteers every turn, and both look exactly like a control bug. There is also a structural gain available here that larger robots cannot use. When everything is on one computer, messages between the parts never leave that computer, so the simulated version and the deployed version can be the same processes with the same plumbing and no bridge in the middle. Removing the bridge removes a whole class of difference before you go looking for it.

What should you do when there is a demo in two weeks?

Stop trying to fix the disagreement and start narrowing the conditions until the robot is reliable inside them. Two weeks is not enough to find and fix a divergence you have not yet located, and pretending otherwise is how teams arrive at a demo with a robot that is worse than the one they had a fortnight earlier. The move is to establish what the robot does do dependably — one route, one object, one lighting condition, one floor — and then defend that envelope: rehearse in the actual room, at the actual time of day, with the actual battery charge. Meanwhile build the recording path anyway, because every demo rehearsal is a free experiment and the failures you capture this fortnight are the ones you will debug next month. What not to do in two weeks: change the physics model, change the middleware, or rewrite the controller. Each of those trades a failure you understand for one you do not, and there is no time left to learn the new one.

What should you do if nobody on the team has debugged hardware before?

Slow the robot down, bring up one component at a time, and record everything from the first day. Those three habits do more for a team new to hardware than any tool, and all three are unglamorous enough that inexperienced teams skip them. Slowing down works because nearly every early failure is survivable at low speed and expensive at working speed, and because a failure you can watch is a failure you can reason about. Component-by-component bring-up works because a robot with one joint moving has exactly one thing that can be wrong. Recording works because the failures that teach you most happen once and are never seen again, and a team without recordings is left with three conflicting memories. Add one more: give the machine a physical stop that does not pass through your code at all. Beyond that, expect the first fortnight of hardware to produce behaviour that looks impossible, and expect most of it to turn out to be a unit, a frame, a sign or a cable. That pattern is what run-to-run variation on real robots usually resolves into.

What do you give up by forcing simulation and hardware onto one code path?

You give up convenience early and buy it back later, which is the right trade for anything you intend to ship and the wrong one for a two-week experiment. The costs are three and they are specific. You have to write an interface over sensors and actuators before you fully understand what either side needs, and some of that will be wrong and get redone. You lose the simulator's conveniences — the exact pose of every object, a reset button, a pause — or you hide them behind something the hardware side must also satisfy, which is more work than it sounds. And you move more slowly at the start, exactly when moving quickly feels like progress. What you buy is the one property that makes a disagreement diagnosable: when the robot behaves differently from the simulation, the difference is in the world rather than in your code, so there is one place to look instead of two. Teams that keep two paths spend their hardware weeks discovering that the two versions parted company long ago, and that discovery always arrives at the worst possible moment.

When is ROS 2 the better choice?

ROS 2 is the better choice for most teams facing this problem, and the recording tools are the reason. ROS 2 ships a message format, a recorder, a viewer and a replay tool that work together, which means the single most useful thing in this post — capturing a real run and pushing it back through your code — is available on day one rather than being a project. That advantage is large and nothing else in robotics currently matches it. ROS 2 is also the answer when the simulator you want already speaks its message types, which covers Gazebo and most vendor rigs, and when the setup spans machines: a workstation running the simulator while the robot runs the controller is a standing arrangement, and HORUS is single-machine middleware validated in simulation, so shared memory stops at the edge of one computer and the multi-machine case is not the case it serves. And ROS 2 is the answer whenever students or outside collaborators are involved, since a disagreement between simulation and hardware already carries enough unknowns without the message layer being one of them.

Is the simulation just not realistic enough?

No, and here is why: most disagreements are not caused by the things a more detailed model improves. The failure usually lives in the seam — a wrapper, a loop, a sensor read, a queue — and that seam is in your code, not in the physics engine. Adding fidelity there is like sharpening a photograph of the wrong room. There is one genuine exception and it is worth stating clearly, because it is where the realism camp is right: contact. Grasping, footfalls, pushing, sliding, anything where surfaces meet and slip is genuinely hard to model and genuinely responsible for transfers that fail. If your task is nothing but contact, look at the model early. If your task barely touches anything — a mobile base navigating a corridor, an arm moving through free space to a taught pose, a perception pipeline — then physics is close to the last thing to suspect, and the replay test will show you that in an afternoon rather than a month, as what simulators get wrong about the real world sets out in more detail.

Does this mean the time spent in simulation was wasted?

Partly, but not the way you think. What was wasted is narrow: some tuning against a model that turned out not to match, and any confidence built on runs that were never comparable to a real one. What was not wasted is nearly everything else — the behaviour logic, the perception pipeline, the state machine, the failure handling, every bug caught before it cost a gearbox. The real loss is subtler and worth naming, because teams keep repeating it: the simulated success taught you the robot worked, and that belief is what delayed the hardware. Simulation's honest job is to tell you when something is definitely broken, and it does that job very well. Simulation cannot tell you something definitely works, and every team learns this the same expensive way. The correction is not to simulate less. It is to change what simulation is for once hardware exists: a place to replay real recordings, rehearse dangerous manoeuvres and check a change before anyone risks the machine. Judged that way, the simulator gets more valuable after the disagreement, not less.

How do you decide which disagreement you actually have?

Answer three questions in order and the diagnosis usually falls out. First: does the replayed run match the real run? If yes, your code paths agree and the problem is in the physical world, so look at sensing, actuation and contact. If no, stop everything else — your two versions are not the same code and that is the whole bug. Second: does the task involve surfaces meeting? Grasping, walking, pushing and sliding put physics genuinely in the frame; free-space motion and navigation almost never do, whatever anybody in standup believes. Third: does the failure repeat? A failure that repeats identically is logic or a model. A failure that varies run to run is timing, and timing is the one that hides best in a simulator because a simulator waits for your code when the world does not. When the answers conflict, follow the first, because a divergence between two code paths makes every other measurement meaningless. If you have not yet chosen the machine this will happen on, the choice of body affects how often it happens, which is the subject of quadrupeds versus humanoids for a first project.

A short version, by situation:

• If you have a failure you cannot reproduce -> record a real run and replay it through simulated code, because that splits the problem in half in an afternoon.
• If the replay matches the hardware -> look at sensing, actuation and contact, because your code paths already agree and the difference is in the world.
• If the replay does not match -> stop tuning anything, because you have two code paths and only one of them was validated.
• If the failure varies from run to run -> suspect timing, because a simulator waits for your code and the world does not.
• If a demo is two weeks away -> narrow the conditions and defend them, because a divergence you have not located will not be fixed in a fortnight.

When you want to compare the layers rather than the symptoms, the HORUS Fit Framework lines them up on five things that are not numbers: ecosystem size, setup effort, team size fit, deployment target, and licence. For this problem, ecosystem size and deployment target carry the decision, because recording tools and the number of computers on the robot settle it before anything else does.

HORUS is open source under Apache-2.0 and the repository is linked below. Star it so it is in your list when you start building.