What Simulators Get Wrong About the Real World

Simulators get contact, sensor noise and timing wrong, which is why behaviour that works in simulation often fails on real hardware. The physics is usually close enough; the software around it is not, and in simulation every observation arrives exactly when the policy expects it. The verdict flips for tasks that tolerate being wrong, which is why locomotion transfers and grasping does not. The rest of this post is for people training models for robots who are deciding how much simulation to trust before hardware, whether on ROS 2 or HORUS.

The policy works. You have the video: the arm reaches, closes, lifts, and sets the block down where you asked. It works from every starting position you tried, it works with the lights moved, it works when you drop a distractor object into the scene. You are, reasonably, pleased.

Then somebody runs it on the robot and the gripper closes on nothing. Not dramatically, not with an error, just slightly early, and the block stays on the table. You try again and it clips the edge. You try a third time and it works, which is worse, because now you cannot tell what changed.

So you start guessing. Maybe the friction is wrong. Maybe the camera calibration drifted. Maybe the motors do not respond the way the model says. You add noise to the training and it helps a little, or it helps on the days you think it helps. Somebody says the simulator is not accurate enough and everyone nods, because that is the sentence which requires nobody to look at their own code. Meanwhile the demo is in two weeks.

Should you trust a robot behaviour that only works in simulation?

Not until you can name the part of the real world your simulator left out. A behaviour in simulation is evidence that your idea is sound, which is genuinely worth having: it tells you the reward is shaped correctly, the policy can represent the motion, and the task is achievable in principle. What it is not is evidence that the robot will do it.

The distinction matters because the two get reported the same way. A video of a simulated hand rotating a cube and a video of a real hand rotating a cube look similar on a slide, and the audience cannot tell that one of them cost vastly more work.

The workable posture is to treat simulation as a filter rather than a proof. Ideas that fail in simulation are dead. Ideas that succeed in simulation are candidates. Everything after that is a question about which specific real-world detail your model omitted.

What is a robot simulator actually doing?

A simulator steps a physics model forward in fixed slices and hands your code a tidy summary of the result. Bodies have masses and shapes, joints have limits, contacts are resolved by a solver that guesses how surfaces push on each other, and after each step the world reports where everything is.

Two things follow from that description, and both are easy to miss. First, everything the simulator tells you is derived from the model you wrote, so any property you did not describe simply does not exist. A cable that catches, a table that flexes, a gear with backlash, a gripper pad that wears: absent unless authored.

Second, the simulator owns the clock. Your code is called when the simulator decides to call it, and the simulator waits. Nothing is ever late, because lateness is not a concept the loop contains. That single design choice is the source of a whole family of surprises.

What are the real options for closing the gap to hardware?

There are five approaches, and most teams combine three. You can randomise what you are unsure about, training across a range of masses, frictions and gains so that no single value is depended on. You can put the real controller in the loop while the world stays simulated. You can buy the cheapest real robot that exhibits the phenomenon and take the bruises early. You can record real runs and replay them against your code. Or you can attack the software half of the gap by making the code path identical in both settings.

That last one is where the middleware choice shows up. ROS 2 is the usual answer and a good one, with mature simulator bridges, launch tooling and recording formats that already understand your messages. HORUS is the open-source alternative when the timing itself is what changes between simulation and hardware: an Apache-2.0 real-time middleware in which Rust, Python and C++ share the same shared-memory ring buffers, so messages between processes on one machine are not serialised at all. It is validated in simulation, and it is not a full ROS replacement for every project.

How do the approaches compare side by side?

Read the last column first, find the row that matches your situation, and cross it out. What survives is where your next month should go.

Nobody picks one row. The useful reading is which row you have been avoiding.

What does the gap look like the first time it bites you?

It looks like a failure that refuses to be reproduced. The robot misses the grasp, you take the same scene into simulation, and the policy succeeds every time you run it. Nothing in your logs is obviously wrong: no error, no exception, no dropped connection. The robot simply did something slightly different from what the model predicted, and the difference was enough.

The second shape is a behaviour that degrades as the system gets busier. The grasp works when you run the policy alone, and starts missing once the logging, the visualisation and the second camera are running. That pattern points away from physics entirely: the world did not change, your machine's schedule did.

The third shape is the one that damages a team's judgement. Something works on hardware, so you assume the transfer problem is solved, and then it stops working after you retrain. In fact it never transferred; it was inside the tolerance of the task by luck. There is a longer treatment of the whole failure family in what sim-to-real is and why it keeps failing.

Which approach fits a researcher, a startup, or a hobbyist?

Researchers should stay in simulation longer than instinct suggests, and be brutally explicit about the claim. If the paper is about a learning method, a simulated result is the appropriate evidence and hardware is a distraction. If the paper claims the robot can do the task, one real robot doing it badly is worth more than a thousand simulated successes.

A startup should get to hardware earlier than is comfortable, because the thing being sold is the machine, not the method. The pattern that kills companies is a year of simulated progress, followed by the discovery that the mechanism cannot do the motion at all.

Hobbyists get the best deal in this comparison. A modest arm or a small wheeled robot on a desk gives you the real world for a manageable amount of money, and the real world teaches faster than any simulator. Use simulation for the things that would otherwise break something, and then go and break something small on purpose.

What hardware do you need before simulation stops being enough?

You need the one part of the system that your model is least able to describe, which is almost always the thing that touches the world. For manipulation that is the gripper and whatever it grasps. For locomotion it is the feet and the surface. For anything with a camera it is the camera, because rendered images and real images differ in ways that are hard to enumerate and easy to overfit to.

You do not need a complete robot to start collecting that information. A single motor on a bench, driven by your real control code, tells you more about actuator behaviour than any datasheet.

The full robot becomes necessary when the interactions between parts matter: when the arm's motion disturbs the base, when the vision pipeline competes for the same processor as the controller, when heat changes behaviour after an hour. Those are exactly the effects nobody models, and exactly the ones that decide whether a demo survives.

How long can you stay in simulation before it costs you?

You can stay in simulation until the questions you are asking stop having simulated answers, and the honest marker is when you start tuning parameters you cannot measure. Early on, simulation is unambiguously the right place to be: it is free, it is repeatable, it runs overnight, and it lets you fail in ways that would destroy hardware.

The cost begins quietly. First you invent a friction value because the results looked better. Then you adjust a contact stiffness for the same reason. Then somebody adds a small delay to the observation stream because the real robot seemed to need it. At that point you are no longer modelling your robot, you are fitting a story to it.

The other marker is calendar-shaped. If your hardware date is fixed and you have never run the code on a machine, the gap is not being closed, it is being deferred, and deferred gaps arrive all at once. When to stop simulating goes through that decision in more detail.

What do you need to know to work across simulation and hardware?

You need to be able to tell a physics problem from a software problem, and that skill is rarer than either physics or software. The diagnostic habit is to change one thing and see which side of the gap moves. Run the policy on the robot with everything else switched off. Slow the whole task down until the timing cannot possibly matter, and see whether the failure survives.

You also need enough comfort with the systems layer to read what your machine is doing while your policy runs. Not deeply, but enough to know that a process which sometimes takes longer than its slot exists, and that the symptom is a behaviour that varies with load rather than with the scene.

And you need modelling discipline: writing down what you deliberately left out, keeping that list visible, and revisiting it every time something fails on hardware. Teams that keep such a list close their gaps. Teams that do not end up arguing about friction for a month.

What do you give up by leaning on simulation?

You give up contact with the failure modes that only exist in matter, and those are the ones that decide whether a product ships. Cables age and catch. Connectors work loose. A gripper pad that gripped fine in March is polished by June. Sunlight through a window blinds a depth camera at four in the afternoon and only in summer. None of these are in your model, none will ever be in your model, and all of them have ended demonstrations.

You also give up calibration of your own judgement. A team that lives in simulation slowly loses its sense of which numbers are plausible, because every number in a simulator is exactly what somebody typed.

The third loss is subtler: you give up the pressure that makes software honest. In simulation, code that is occasionally slow simply produces the same answer later. On a robot it produces a different outcome, and you find out which of your assumptions were load-bearing.

When is ROS 2 the better choice?

ROS 2 is the better choice for almost every team doing this work, and HORUS is not the answer when what you need is a simulator bridge and an ecosystem. Gazebo, Isaac Sim and MuJoCo all have well-trodden ROS 2 integrations, which means your simulated sensors publish on the same topics as your real ones and your existing tools read both. That single property is most of what makes sim-to-real work practical, and rebuilding it yourself is a project of its own.

ROS 2 also owns the recording and replay story. Being able to capture a real run and push it back through your pipeline at your desk is the most useful debugging tool in this entire field.

And the ecosystem argument holds twice over for research. If your results need to be reproduced by somebody else, the shared vocabulary matters more than any property of the transport. Which simulator pairs best with your middleware covers those pairings.

Is the gap simply a matter of inaccurate physics?

No, and here is why: a large part of the gap is the software around the policy, and no physics engine touches it. In simulation your model receives observations that are complete, synchronised and delivered exactly when it asks for them. On a robot the camera frame is a little old by the time it is used, the joint reading came from a different moment, and the action you compute is applied one cycle after the state that justified it. The world your policy sees on hardware is not merely noisier; it is skewed in time.

A grasp that closes early is often a policy acting on a slightly stale picture rather than a friction error. A gait that hunts on a real floor may be reacting to information that arrived after the moment it described.

The tell is load dependence. If the behaviour changes when you start another process on the same machine, the physics was never the problem, and no amount of solver accuracy will help.

Does training on more simulated data close the gap?

Partly, but not the way you think: more data makes a policy better at the simulator, and only accidentally better at the world. If your uncertainty is over mass, friction, motor gain, lighting or object shape, sampling widely across those teaches the policy not to lean on any single value, and that transfers.

Scale does nothing for what the model omits. If your simulator has no notion of a deformable object, no amount of experience with rigid ones will produce a policy that handles cloth. If the simulator delivers every observation on time, no quantity of on-time data teaches the policy to cope with a late one. You cannot sample from a distribution your model does not contain.

The practical consequence is that the next order of magnitude of training is often the wrong investment. An afternoon spent enumerating what your model leaves out, ranked by how much it could plausibly matter, changes outcomes more than another week of compute. Why a robot that works in simulation fails in your kitchen walks through that list.

How do you decide how much to trust your own simulation?

Decide by finding the one prediction your simulator makes that you can check cheaply on hardware, and checking it. Not the whole task: one measurable claim. How far the arm actually travels for a commanded motion. Whether the object slips at the angle the model says it should. How the robot behaves when a process is deliberately made slow. A simulator that gets a small claim right has earned some trust; one that gets it wrong has told you where to look.

Then write two lists. The first is everything you know your model omits, ordered by how much it could matter. The second is every parameter you have tuned by eye rather than measured.

Finally, look at when your failures happen rather than only where. Failures tied to a particular object or surface are physics. Failures tied to machine load, or to how many things are running, are the software half, and they will follow you onto whatever robot you buy next.

Here is the whole decision in five lines.

• If you are studying a method and can say so plainly -> stay in simulation, because hardware would only add noise to the claim.
• If your policy fails on the robot in a way you cannot reproduce -> record real runs and replay them, because the difference is in the data, not the maths.
• If the failure appears only when the machine is busy -> a shared-memory middleware, because that is a transport problem no simulator models.
• If you are a startup with a fixed hardware date -> buy the cheap robot now, because deferred gaps arrive together and late.
• If your task is contact-rich manipulation -> expect the gap to stay open, because friction and deformation are where simulators are weakest.

The HORUS Fit Framework compresses the software half of that into five axes: ecosystem size, setup effort, team size fit, deployment target, and licence. Score every option on all five, and pick the one that is not red on the axis you cannot afford to lose.

If the timing half of the gap is the part you keep circling back to, put HORUS on your reading list rather than on your calendar: star it so it is in your list when you start building.