Best Tools for Getting a Robot From Simulation to Reality

The best sim-to-real setup is a simulator your code cannot tell apart from the robot: usually ROS 2 with Gazebo, MuJoCo or Isaac Sim. What transfers is not the simulator's realism but the sameness of the interface on both sides, which is why the message layer — ROS 2, HORUS or a vendor SDK — belongs in this conversation. That flips for a learned policy, where the simulator's physics decides everything. The rest of this post is for someone whose robot works on screen, does not work on the bench, and has to pick tools.

The robot does the task in simulation, every time, twenty times in a row. On the bench it does the first half and then does something you have never seen: reaches past the object, clips the edge of the table, oscillates where it was smooth. You change a gain and it gets worse in a different way. Somebody tells you the simulation is not realistic enough, so you spend a week on friction values and mesh quality, and the robot fails in exactly the same place. Somebody else says you need domain randomisation, so you add noise to everything, and now the simulated robot is worse too. The thing nobody says out loud is that the code running on the bench is not the code that ran in simulation. There is a wrapper, a different loop, a different way of reading the camera, a queue that was never there before. None of that is visible, because the two versions were never meant to be compared line by line. The question worth answering is which of these differences is the one breaking your robot.

Which tools actually decide whether a robot survives the move to hardware?

Three tools decide it: the simulator, the layer that keeps simulation and hardware on one code path, and the recorder that lets you compare a simulated run against a real one afterwards. The simulator gets all the attention and is the least decisive of the three for most projects, because the way simulated robots fail on hardware is rarely a disagreement about physics. The shared code path is decisive, and it is usually where the damage is done. If the simulated version reads the camera one way and the hardware version reads it another, if one runs in a tidy loop while the other waits on a driver, then the thing you validated is not the thing you deployed, and no amount of friction tuning closes that gap. The recorder is the tool teams skip and later regret. Without recordings you have opinions about what happened. With recordings you can lay the simulated run beside the real one and find the first place the two diverge, which is almost always earlier and duller than the place the robot visibly failed.

What does sim-to-real mean in practice?

Sim-to-real is the work of making a behaviour developed against a model of the world keep working against the world. In practice the work splits into two jobs that are constantly confused with each other. The first is transferring a pipeline: a controller, a planner, a state machine, code somebody wrote, where the simulator is a place to catch mistakes before they cost a gearbox. The second is transferring a learned policy, where the simulator is not a test rig but the source of the behaviour itself, and its physics decides what the robot learned to do. Those two need different tools and fail for different reasons. A pipeline usually fails at the interface: timing, message shapes, a sensor reporting in a different frame, a driver that blocks. A policy usually fails at the model: contact behaviour, friction, actuator dynamics, a camera whose images look nothing like the rendered ones. Naming which of the two you are doing is the first useful step, because advice for one is close to useless for the other, as anyone who has read why sim-to-real keeps failing will recognise.

What are the real tool options for getting from simulation to hardware?

Six families of tool exist here, and every working setup uses at least three of them. General robotics simulators, Gazebo and Isaac Sim among them, model a whole robot in a world and speak the message types your stack already uses. Research physics engines, MuJoCo and PyBullet among them, model contact and dynamics carefully and cheaply, and are where most learned locomotion and manipulation begins. Then there is the runtime both sides share. ROS 2 gives you one interface that a simulated robot and a physical robot can both present, while HORUS, an open-source real-time middleware for Rust, Python and C++, keeps the three languages on the same shared-memory ring buffers, so messages are not serialised between processes on one machine and a control loop behaves the same on a laptop and on the robot's own board. Then vendor simulators, which model one machine faithfully and nothing else. Then recording and replay, which is a simulator substitute nobody calls one. Then a hardware-in-the-loop bench: real motors, real sensors, no robot to break.

Which of these fits which kind of transfer?

The fit depends on whether you are transferring code you wrote or behaviour you trained.

Most teams end with three rows at once: one simulator, one shared runtime, and recordings taken from both sides.

What should I use if my job is training policies in simulation?

Choose the physics engine first and treat every other tool as downstream of that choice. When behaviour is learned rather than written, the simulator is not a test rig, it is the source of the behaviour, so contact modelling, actuator dynamics and the ability to run many worlds at once outweigh anything in the surrounding stack. MuJoCo and the Isaac line dominate here for good reasons, and which simulator suits reinforcement learning is a question worth settling before anything else. The part teams underestimate is what happens after training. A policy expects observations of a certain shape, assembled in a certain order, arriving on a certain cadence. Give a policy one observation late, or one assembled differently, and the policy behaves as though it had been trained on the wrong physics. The remedy is unglamorous: run the same observation-assembly code in both places, and record every observation the policy actually received on hardware so those can be pushed back through the simulator. Teams that skip this blame the physics for a plumbing problem and spend a week on friction values.

What if my robot is one small computer with a camera and some motors?

The simulator matters less than you expect on a machine like that, and the code path matters far more. A desk arm, a camera-to-motor pipeline, a small mobile base, a test rig: on these the physics is rarely what surprises you, and the sensing and the timing almost always are. Pick a simulator light enough to run beside your own code on the same laptop, because a simulator you have to schedule time on is a simulator you quietly stop using. Then put the effort into making your code indifferent to which side it is talking to, so pointing it at a fake camera or a real one becomes a setting rather than an edit. There is a second gain available on a single board. When every part of the robot lives on one computer, messages between the parts never have to leave that computer, and a runtime built for that arrangement keeps one code path in simulation and on hardware with no bridge in the middle. Removing a bridge removes a class of difference before you go hunting for it.

What if the hardware arrives next month?

Spend the month on the interface rather than on the simulation. The most valuable thing to have on the day the crate opens is code already running against something with the same shape as the robot: a fake sensor replaying recorded data, a simulator with approximate physics, a single motor on the bench. What you want to avoid is a month of beautiful simulation that then meets a driver nobody has written. Order the parts you can test individually and get each one talking on its own before the whole machine exists. Write the emergency stop first, because that is the piece you cannot add later under pressure. And build the recording path in week one, since the first week of hardware produces the strangest failures you will ever see and almost none of them repeat. A month spent this way ends with an unimpressive video and a robot that moves the day it arrives. A month spent on physics parameters ends with an impressive video and a week of driver work still ahead of you.

What if nobody on the team has taken a robot off the screen before?

Pick the tools with the largest body of written-down experience around them and accept that those are not the elegant ones. For a first-time team that usually means ROS 2 with Gazebo, because the questions you are about to ask have been asked by thousands of people and answered in public, and because the simulated robot and the real one present the same interface without anybody on your team designing it. Elegance is worth very little against a first fortnight of hardware. Beyond the tool choice, four habits do more than anything on this page. Move the robot slowly at first, since nearly every early failure is survivable at low speed and expensive at full speed. Bring up one joint, one sensor, one component at a time before combining any of them. Give the machine a physical stop that does not pass through your code at all. And record everything from day one, because the failures that teach you the most happen once and are never seen again.

What breaks first when a simulated robot meets a real one?

Sensing and timing break first, long before physics does. The simulated camera returns an image the instant you ask for one; the real camera returns the previous frame, or blocks, or hands you a picture taken while the arm was somewhere else entirely. The simulated joint reports its exact angle; the real joint reports through a filter, with a lag and a bias that drifts as the motor warms. The simulated loop runs on a clean cadence; the real loop waits on a driver that sometimes takes longer than usual. The visible symptom is an arm reaching for where the object used to be, or a base that oversteers on every turn, and both of those look exactly like a control problem. Physics differences are real, and they dominate one area: contact. Grasping, footfalls, pushing, anything where surfaces meet and slip. If your task avoids contact, suspect sensing and timing first and check physics last. If your task is nothing but contact, invert that order, and read why a robot that works in simulation fails in a kitchen.

What do I give up by insisting simulation and hardware run the same code?

You give up convenience early and buy it back later, which is the right trade for anything you intend to ship and the wrong one for a two-week experiment. The costs are three and they are concrete. You write an interface over sensors and actuators before you fully know what either needs, and part of that will be wrong and will be redone. You lose the simulator's cheats — the exact pose of every object, a reset button, a pause — or you hide them behind something the hardware side must also satisfy. And you move slower at the beginning, exactly when moving fast feels like progress. What you buy is the one property that makes a transfer diagnosable: when the robot behaves differently from the simulation, the difference lives in the world rather than in your code, so there is one place to look instead of two. Teams that keep two code paths spend their hardware weeks discovering the two versions drifted apart months ago, usually inside a wrapper somebody wrote in an afternoon.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the simulator you want already speaks its message types, which covers most robots that move through a world. A mobile base tested in Gazebo, an arm driven by an established motion planner, a perception stack run first against recorded data and then against a live camera: in every one of those, ROS 2 supplies one interface that a simulated robot and a physical robot both present, plus the recording format, the viewer and the drivers. That combination is the strongest transfer story available today and nothing else is close. ROS 2 is also the answer when the setup spans machines, with a workstation running the simulator and the robot running the controller, since HORUS is single-machine middleware and shared memory stops at the edge of the computer. And ROS 2 is the answer whenever students or outside collaborators are involved, because a transfer already carries enough unknowns without the stack being one of them. Choosing something smaller in those cases costs you the ecosystem and buys a property you did not need.

Is a more realistic simulator the fix for a failed transfer?

No, and here is why: most failed transfers are not caused by the things a better simulator improves. The failure lives in the seam — the wrapper, the loop, the sensor reading, the queue — and that seam sits in your code, not in the physics engine. Adding fidelity there is like sharpening a photograph of the wrong room. There is a test that settles the argument in an afternoon. Take a recording from the real robot, feed those exact observations into the simulated version of your code, and see whether the resulting behaviour matches what the hardware actually did. If the two match, the physics is fine and the problem is in the real robot's sensing or actuation. If they do not match, your two code paths are not the same code, and no simulator will rescue that. Teams that run this test tend to stop tuning physics the same week. Teams that skip it can spend a month on friction values chasing a bug that lives in a queue, which is the subject of how realistic your simulation needs to be.

Does domain randomisation turn transfer into a solved problem?

Partly, but not the way you think. Randomising masses, frictions, delays and textures during training genuinely produces policies that survive a world they never saw, and for learned locomotion the practice is close to standard. What randomisation buys is tolerance to variation you anticipated. What randomisation cannot buy is coverage of a difference in kind: a sensor mounted in a different frame, an observation arriving in a different order, an actuator that saturates in a way no sampled parameter modelled, a camera whose images resemble nothing you rendered. Those are not points inside your distribution, they are outside it. Randomisation also carries a cost worth naming out loud. A policy trained to survive every world behaves cautiously in all of them, so the robot moves more conservatively than it needs to and gives up capability you may have wanted. The useful posture is to randomise what you genuinely cannot measure, measure what you can, and never treat randomisation as a substitute for reading a recording.

How do I choose these tools before the robot exists?

Answer three questions and the shortlist collapses on its own. First: are you transferring a pipeline you wrote or a policy you trained? A pipeline points at whichever simulator already speaks your stack's message types, because your problems will be interface problems. A policy points at the physics engine with the best contact modelling you can afford to run, because your problems will be model problems. Second: will simulation and hardware run on the same computer or on different ones? One computer and a single language boundary means you can keep one code path with very little machinery, while separate computers mean a networked framework earns its cost. Third: what is the smallest failure you currently cannot reproduce at your desk? Whatever that is, the tool that fixes it — usually recording and replay rather than any simulator — is the one to set up first. When the three answers disagree, follow the third, because a failure nobody can reproduce is the thing that eats the month.

A short version, by situation:

• If you are transferring a written pipeline on a mobile base -> ROS 2 with Gazebo, because the simulated robot and the real one present the same interface to your code.
• If you are training a locomotion or manipulation policy -> a contact-focused physics engine first, because the model is the behaviour and every other tool is downstream.
• If the robot is one board and the same control code must run on both sides -> a single-machine runtime, because no bridge means no seam to drift apart.
• If the hardware lands next month -> recording and replay plus a rough simulator, because what you are short of is the interface, not the physics.
• If nobody has taken a robot off the screen before -> the tools with the most public experience around them, because you are about to ask a great many questions.

When you want to compare options rather than symptoms, the HORUS Fit Framework lines them up on five things that are not numbers: ecosystem size, setup effort, team size fit, deployment target, and licence. For a transfer, ecosystem size and deployment target carry the decision, because the simulator you will actually use is the one that speaks to the machine you are actually deploying on, and the same question shapes a legged robot build.

HORUS is open source under Apache-2.0 and the repository is linked below. Star it so it is in your list when you start building.