What Is Sim-to-Real, and Why Does It Keep Failing?

Sim-to-real is moving robot behaviour from a simulator onto hardware, and it keeps failing on contact, timing and sensing rather than on visual realism. The gap is physical, so it closes by measuring the real machine and running the same code in both places, which is what a stack such as ROS 2 or HORUS is for. That flips when your robot barely touches anything. The rest of this post is for someone whose policy works in simulation, fails on the bench, and cannot yet say which part of the gap is to blame.

The policy is beautiful in the simulator. It reaches, it grasps, it places, and it does that over and over without dropping anything. You have watched it enough that you can predict the trajectory before it happens. Then you load the same weights onto the machine on your bench, and the arm reaches past the object, closes on air, and carries on as though the table were not there.

So you go back to the simulator, and everything still works. That is the part that costs people a week. There is no bug to find, no crash, nothing in a log. The two worlds simply disagree, and neither of them will tell you where.

You try the obvious things. Better textures. More lighting variation. More episodes. A camera setting that matches the render more closely. Each one helps a little on some objects and not at all on others, and none of them produce the moment where the thing suddenly works. Meanwhile the hardware sits on the bench being expensive, and you have started to wonder whether the months in simulation bought you anything at all.

What is sim-to-real, and why does it keep failing?

Sim-to-real is developing robot behaviour inside a simulator and then running that behaviour on the physical machine. It keeps failing because a simulator contains only the parts of the world someone chose to put in it, and the parts nobody put in are usually the parts that decide whether a real robot succeeds. Shapes and distances transfer well. Friction, backlash, cable drag, a gripper pad worn smooth by a summer of testing, a motor that warms up over an afternoon and delivers less than it did at nine in the morning: none of those exist in the model unless somebody modelled them, and nobody models all of them.

The second reason is timing. In simulation, the world waits for your program. A frame is produced, a decision is made, the world advances one step. On hardware nothing waits. The camera keeps producing frames while you think, the arm keeps moving while the decision is in flight, and a command computed for where the robot was gets applied to a robot that has since moved. The behaviour did not get worse when you deployed it. The conditions it quietly assumed stopped being true.

What is the reality gap in plain terms?

The reality gap is the collection of small disagreements between a simulated robot and the metal one, each too small to worry about alone and large enough together to change the outcome. It helps to break it into four piles. There is the physics pile: friction, mass distribution, joint stiffness, how two surfaces behave in the instant they touch. There is the sensing pile: real cameras blur, real depth sensors lose dark and shiny surfaces, real encoders have a little play in them.

There is the timing pile, which is the one people forget: on hardware, sensing, computing and acting all take time, and that time varies from cycle to cycle. And there is the world pile, which is everything the scene contains that your simulation never included, from a cable across the workspace to sunlight moving across the bench over the course of a day.

Naming the pile matters because each has a different remedy. Physics gaps close by measuring the robot. Sensing gaps close by degrading the simulated sensor rather than improving it. Timing gaps close in software architecture. World gaps close by testing outside the tidy corner of the lab.

What does a sim-to-real failure look like on the real robot?

It looks like a robot that is confidently wrong rather than a robot that is broken. The arm moves smoothly, at the speed it moved in simulation, along a path that would have been correct if the object had been where the model believed. Nothing stumbles, nothing errors, no log line appears. The gripper closes just short of the mug, lifts nothing, and the policy proceeds to the placement step as though it were carrying something.

The second signature is inconsistency that tracks physical conditions rather than software ones. It works on the matte block and not the glossy one. It works first thing and not after the arm has been cycling for an hour and the joints are warm. It works near the middle of the workspace and fails at the edges, where the view the camera gets is least like the views the model was shown.

The third is the one people miss: it worked yesterday, on this hardware, with these weights, and today it does not. That points at the machine and its timing rather than at the model. Why a robot behaves differently every run walks through those causes.

What are your actual options for closing the gap?

There are eight honest approaches, and most teams need two or three rather than one perfect choice. You can randomise the simulator, varying friction, mass, lighting and camera placement until the policy stops depending on any single value. You can do the opposite and measure the real machine, then bend the simulator until it matches that particular robot. You can train in simulation and fine-tune with a modest amount of real experience. You can collect human demonstrations on the hardware and learn from those. You can put a hand-written controller underneath the learned policy so a bad command is refused instead of executed. You can wire real sensors and real actuators into the simulator so part of the loop is genuinely physical.

And you can attend to the software the policy runs inside, which is where ROS 2 and HORUS sit as two different answers. The first keeps one node graph across the simulator and the robot through its supported bridges. The second is an open-source real-time robotics middleware for Rust, Python and C++ that share the same shared-memory ring buffers, so a Python policy and a C++ controller on one board exchange messages without serialising them, published under Apache-2.0 and validated in simulation.

How do these approaches compare side by side?

Read the last column first. Most people can strike four rows immediately, and the real argument is between the two that remain.

Are you a researcher, a startup, or someone learning at home?

If you are a researcher, the gap is often not the thing you are being paid to close, and it is worth saying that plainly. Publishing a result in simulation is a legitimate contribution, and bolting a real arm onto the work at the end costs more than the paper is likely to gain. Decide early whether hardware is part of the claim, because a half-hearted hardware section is worse than none.

If you are a startup, the gap is the entire product risk and it should be attacked first, not last. The failure mode is a team that spends most of a year making the simulated version better and then discovers on the bench that the gripper was never going to hold the part. Get one real machine touching one real object in the first month, even if the behaviour is hand-written.

If you are learning at home, simulation is genuinely the right place to start, and the honest advice is to buy something small and cheap that touches the world sooner than feels comfortable. When to stop simulating and build the real thing covers where that line falls.

What hardware are you deploying onto?

The gap is much wider for robots that make contact than for robots that do not. A wheeled robot driving across a flat floor lives in the part of physics simulators model well, and policies trained in simulation often transfer with modest fuss. An arm that grasps, a hand that manipulates, or a legged machine that pushes off a surface lives entirely in contact, which is where every simulator is at its weakest.

The compute on the robot matters as much as the mechanics. A policy trained on a workstation and deployed to a small board runs inside a different timing regime, and a decision that used to arrive in time now arrives after the moment it was for. That shows up as behaviour that degrades rather than software that fails, which is why it is so often misdiagnosed as a model problem.

Then there is the question of how many machines are involved at all. A policy on a laptop driving an arm over a network is a different system from a policy on the robot's own board. Whether your robot should run one computer or several is worth settling before you blame the model.

How long until this has to work on the real robot?

With a demo in two weeks, do not touch the training pipeline. Constrain the world instead. Fix the lighting, use the objects that already work, mark the positions on the bench, and put a hand-written check underneath the policy so nothing embarrassing happens on stage. This is not cheating; it is choosing which variables to fight, and every demo you have ever seen did some version of it.

With a couple of months, spend the first week measuring rather than training. Command known motions and compare what the joints did against what the model said they would do. Record real camera frames and feed them to the policy offline. Most teams find something concrete in that week, and it is usually a timing or calibration issue rather than anything to do with learning.

With a year, change the order of the work. Build the hardware loop first, keep it running continuously, and treat the simulator as a way to generate candidates that the real robot then judges. Teams that get this right end up with a simulator that improves month by month, because reality keeps correcting it.

What does your team already know how to do?

If your team is strong in machine learning and thin on hardware, expect the gap to look like a modelling problem for longer than it should. That is a predictable bias, not a criticism: the tools you know are the explanations you reach for. The cure is one person who owns the physical machine, who knows what the joints sound like when they are struggling, and whose first instinct on a failure is to check a cable rather than a checkpoint.

If your team is strong in controls and thin on learning, you will close the physics gap quickly and then stall on perception, because a camera that mostly works is much harder to characterise than a motor that mostly works.

Mixed-language teams have a specific version of this problem. The policy is in Python, the controller is in C++, and the two meet at a boundary that behaves one way on a workstation and another way on the robot. If that boundary keeps changing character between simulation and hardware, the timing gap is partly yours to fix in architecture rather than in training.

What do you give up by developing in simulation first?

You give up calibration of your own judgement. Months inside a simulator teach you what is hard in the simulator, and that is not the same as what is hard in the world. Teams that stay in simulation too long become extremely good at problems the physical robot does not have, and strangely helpless in front of a loose bolt or a warm motor.

You give up the failures that would have taught you the most. A real gripper failing on a real mug tells you something specific about that pad, that surface and that approach angle. A simulated failure tells you about your own assumptions, which is worth less than it feels at the time.

And you give up a certain amount of time, permanently. Some of the work done in simulation does not transfer, and the honest position is that you cannot always tell in advance which part. What simulation buys you in exchange is real: you can try approaches that would break hardware, you can run overnight, and you can put a new person on the problem without letting them near an expensive machine. Why a robot that works in simulation fails in your kitchen is the longer version of that trade.

When is ROS 2 the better choice?

ROS 2 is the better choice for most robots crossing from simulation to hardware today, and HORUS is not the answer for a project whose difficulty lives anywhere except same-machine message traffic. The simulator bridges, the sensor drivers, the transform machinery that tells you where the camera is relative to the gripper, the tools that record a run and play it back: those exist, they are mature, and rebuilding them to save copies between processes is a bad trade.

Choose ROS 2 when your robot navigates, maps, or plans arm motions, because those packages represent decades of work. Choose ROS 2 when your sensors ship with drivers written for it, when your robot spans more than one machine, and when you want to hire people who already speak the vocabulary. Choose ROS 2 when your gap is contact physics or perception, which is most of the time, because no messaging layer has any effect on either.

The narrower case for a shared-memory middleware appears only when several programs on one board pass large data every cycle and your loop is late because of it.

Is a better simulator the answer?

No, and here is why: the simulators available now are already better than the models people put into them. A modern simulator can represent contact, deformation and sensor noise far more faithfully than the default settings you are running, and the reason your robot fails is usually that nobody supplied the friction of your actual gripper pad, the mass of your actual payload, or the delay in your actual control path. Upgrading the engine changes none of that.

There is a second reason. A more faithful simulator costs more to run, and that cost is paid in episodes you no longer have time for, which leaves a policy that has seen less variation. Teams routinely trade away breadth for fidelity and end up transferring worse than before, having spent a month on the switch.

The exception worth naming is when your task depends on something the engine genuinely cannot express, such as cloth, granular material, or fluid. Then the engine is the constraint. Choosing between the major simulators is the right read if that is your case.

Does more training data close the gap?

Partly, but not the way you think: more data makes a policy better at the distribution it was trained on, and the sim-to-real gap is precisely the part of the world that is outside that distribution. A far larger pile of episodes from the same simulator with the same parameters produces a policy that is more confident about a world that does not exist. Confidence is not the property you wanted.

What does help is variation rather than volume. Episodes that differ in friction, mass, lighting, camera placement and object shape teach the policy that those things are not to be relied upon, and a robot that has been trained not to rely on them handles the real value it eventually meets. That is a different axis from how much data you collect.

The other thing that helps is real data, even in small quantities. A modest set of episodes from the physical robot, used to correct rather than to train from scratch, routinely does more than a much larger simulated set. Real data carries the information nobody thought to model.

How do you work out which part of the gap you have?

Run three tests before changing anything, in this order. First, replay real sensor data through the policy offline and watch what it decides. If the decisions are already wrong on real frames, your gap is perception, and nothing about physics or timing will save you. Second, hand the policy perfect state information on the real robot, bypassing perception entirely. If the behaviour is now correct, perception was your problem and you have confirmed it twice.

Third, command a known motion on the hardware and compare what the joints actually did against what the simulator says they should have done. Disagreement there is physics or actuation, and it is fixable by measurement rather than by training. If all three tests look fine and the robot still fails intermittently, you are looking at timing, and timing lives in how your programs are arranged and scheduled rather than in the model.

Most teams skip these because each takes a day and training feels more productive. The three days are cheaper than the month.

Here is the whole decision in five lines.

• If your robot barely makes contact with anything -> transfer directly, because the modelled part is the part you rely on.
• If your policy fails on real camera frames offline -> fix perception first, because no physics change touches that.
• If the same weights behave differently hour to hour -> suspect timing and heat, because the model did not change.
• If you will ship many units -> randomise rather than calibrate, because a fit to one machine does not transfer to the next.
• If programs on one board pass images every cycle and the loop is late -> the messaging layer, because that is the one case where it is the culprit.

The HORUS Fit Framework compresses the software half of this onto five axes you can score any option against: ecosystem size, setup effort, team size fit, deployment target, and licence. For sim-to-real work, ecosystem size carries the most weight, since simulator bridges and sensor drivers live in that column.

When the gap turns out to be traffic between programs on the robot's own board, the useful thing is having the option on your shelf rather than starting a search under deadline. Put HORUS on that shelf now: star it so it is in your list when you start building.