How Realistic Does Your Simulation Actually Need to Be?

Most robots need contact and timing right and tolerate crude visuals, so an approximate simulator beats a photorealistic one unless vision is what you train. Realism costs iteration speed, and a policy that has seen many rough varied worlds transfers better than one that has seen few beautiful ones. That flips when what crosses to the real robot is a camera image, because then appearance is the physics. This post is for someone training a policy in simulation and deciding what runs on the robot afterwards: ROS 2, a vendor stack, or HORUS.

You have a policy that works in simulation and a growing suspicion that this means nothing. The arm picks the block up every single time in the viewer, and you have watched it do so often enough to distrust it, because the surface it lifts from is a grey box and the block has no weight anybody can feel.

So you go looking for advice and get two answers. One camp says the fidelity gap is the whole problem and you should be rendering properly, modelling the compliance of the gripper pads, and simulating what the camera does to a moving scene. The other camp says none of that matters, that you should vary everything crudely and train against the variety instead.

Both camps have working robots, which is the annoying part. And underneath the argument sits a purchase you may be about to make: whether to spend on a machine that can run the heavy simulator, and whether the weeks spent making the model prettier are weeks you get back later on real hardware, or weeks you simply spend.

How realistic does your simulation actually need to be?

Realistic in the dimensions where your task's failures live, and deliberately crude everywhere else. That sounds evasive until you write down what your robot can actually fail at.

A wheeled robot navigating a corridor fails at knowing where it is and at reacting to an obstacle in time, so geometry and sensor noise matter and the lighting does not. A gripper picking up something soft fails at contact, so friction, deformation and the moment of slip matter and the wall texture does not. A policy that consumes camera frames and outputs motion fails at appearance, so lighting, clutter and camera placement matter enormously and friction rarely decides anything.

Fidelity is not one dial. It is at least six: geometry, mass and inertia, contact and friction, actuation, sensing, and timing. Turning all six up costs you the property that makes simulation worth having, which is being able to run the experiment again after changing your mind. Pick the two dimensions your failure lives in, make those honest, and let the rest be cartoonish.

What does simulation fidelity actually mean?

Fidelity means how closely each separate aspect of a model matches the world, and those aspects come apart. Geometry is whether the shapes are the right shapes. Mass and inertia are whether parts weigh the right amounts and resist turning the way real ones do.

Contact is what happens when two surfaces meet: whether they stick, slide, bounce or deform. It is the hardest part of the whole business and the part every simulator approximates somewhere. Actuation is whether a commanded motion produces the motion a real motor would produce, including the truth that real motors have limits and get warm. Sensing is whether the camera, encoder or inertial unit reports what a real one would, noise and delay included. Timing is whether things happen in the same order and rhythm they will on the machine.

People say realistic and almost always mean the picture, which is the aspect that matters least for most robots and most for the ones learning from pixels. What sim-to-real is and why it keeps failing works through the same split.

What are your actual options for simulating a robot?

You have five practical options, and the choice is mostly about which fidelity dimension you are buying. MuJoCo is the contact and dynamics choice, favoured for manipulation and locomotion research because the physics is trustworthy where things touch. Isaac Sim and Isaac Lab are the appearance and scale choice, giving rendered scenes and many simulated worlds at once on a suitable graphics card. Gazebo is the ecosystem choice, wired into ROS 2 so the same nodes, topics and tools that drive your real robot drive the simulated one — and it is worth saying that a middleware is not a simulator, which is equally true of HORUS, an Apache-2.0 layer where Rust, Python and C++ programs share shared-memory ring buffers on one machine, itself validated in simulation, and relevant here only as the thing your code runs on in both worlds.

PyBullet is the low-friction choice for prototyping and coursework. The fifth option is a hand-written model of your specific system, which sounds primitive and is occasionally correct when the dynamics are simple enough to write down.

How do the simulator options compare?

The rows are ordered by what each tool is honest about, not by popularity.

Read the last two columns first. Most teams choose on the first two and meet the disqualifier after the hardware invoice arrives.

Which simulator fits if you have trained models but never built a robot?

Start with MuJoCo and resist Isaac Sim until you have a specific reason for it. Someone arriving from machine learning has one predictable blind spot: the interesting difficulty in robotics is not the model, it is everything the model quietly assumes about the world.

MuJoCo punishes that assumption quickly and cheaply. The gripper closes on the object and the object squirts sideways out of the fingers, and you learn more about contact in one afternoon than in a month of reading papers. Isaac Sim is the better tool eventually, and it arrives with a toolchain, an asset pipeline and a vocabulary that absorb your first fortnight before a single policy trains. Spend that fortnight later, once you know which fidelity dimension your task actually needs.

The other habit worth breaking early is treating simulation purely as a dataset generator. Watch the robot fail in the viewer, repeatedly, slowed down. Most of what separates people whose policies transfer is that they have watched a lot of failure with their own eyes. Whether you can learn robotics entirely in simulation covers what that route teaches.

Does your graphics card decide how realistic you can be?

Yes for appearance, no for physics, and that split is the useful thing to know before spending money. Rendering many photorealistic worlds at once is exactly what a capable graphics card buys, and without one the modern vision-policy workflow is genuinely out of reach. You will spend the day waiting on your own experiments and run fewer of them, which is the real cost.

Contact physics is a different story. The simulators that are trustworthy where surfaces meet run happily on an ordinary laptop, and a laptop is enough to learn whether your gripper design works, whether the object slips out under load, and whether your controller recovers when something nudges it.

So the honest sequence without hardware money is to do the physics work first on the machine you own, establish the task is feasible, and only then decide whether the vision half justifies the purchase. Renting a capable machine by the hour for a training run usually beats buying one for a project you might abandon. Whether you need a powerful graphics card to simulate robots goes through the cases.

What should you use if you have a few weeks before a deadline?

Use the simulator you already know, and spend the weeks on the task rather than on the model. Deadlines are lost to toolchain installation far more often than to insufficient physics.

If you have never run any of them, PyBullet or MuJoCo will have something moving within a day. Isaac Sim will not, and its setup will eat the first week whether or not you needed it. A beautiful simulator you are still configuring on the last Friday is worth less than a crude one you have been iterating in since Monday.

Then narrow the task until it fits. The most common deadline failure in simulation work is scope: a demonstration of one grasp, on one object, under one lighting condition is achievable, and a general manipulation policy is not, and teams routinely aim at the second and arrive with neither. Decide now which single behaviour must work, build the smallest world containing it, and add realism only where that behaviour actually breaks. If the deadline includes running on real hardware, cut the scope again. When to stop simulating and build the real thing is the companion decision.

What skill level does each simulator assume?

They assume very different things, and the gap between them is wider than the marketing suggests. PyBullet assumes a little Python and nothing else, which is why courses start there. MuJoCo assumes you will learn a description format and think in terms of bodies, joints and contacts, which is an afternoon of confusion followed by a fairly pleasant tool.

Gazebo assumes fluency in ROS 2, and this is the one people underestimate. If you do not already have that fluency, Gazebo is not a simulator you are learning, it is an ecosystem you are joining. Isaac Sim assumes comfort with a heavy application, an asset pipeline, and a machine learning workflow running underneath it, and it rewards teams more than individuals.

The skill nobody lists is the one that matters most, which is reading a simulation critically. Noticing that your object never slips because a friction value is absurd, or that your policy is quietly exploiting a bug in how contacts resolve, is what separates useful simulation from a convincing waste of a month.

What does it look like when your simulation was realistic in the wrong dimension?

It looks like a policy that works perfectly in a beautiful scene and fails on the first real attempt in a way that surprises everybody. The shapes are recognisable once you know them.

The rendered version is stunning and the robot still misses, because the camera on the machine sits slightly higher than the one you placed in the model and the policy learned the view rather than the task. Or the physics is meticulous and the robot still fails, because the real gripper's rubber deforms under load and your model's fingers are rigid. Or everything works until the robot is asked to move more briskly, and then nothing works, because your simulation stepped through a tidy sequence while the real machine has several programs competing for one computer.

That last one is the sneaky one, and it is not a physics problem at all. It is what happens when the timing your model quietly assumed does not survive contact with a real software stack. Why a robot that works in simulation fails in your kitchen collects the rest of that family.

What do you give up by choosing the most realistic simulator you can run?

You give up attempts, and attempts are usually what actually produce a working policy. A heavier simulator means fewer experiments in a week, longer waits between having an idea and finding out whether it was any good, and a stronger temptation to defend a run you already paid for rather than throw it away.

You also give up portability. The heaviest tools tie you to particular hardware, particular drivers and a particular version, and keeping that combination working becomes somebody's actual job rather than a footnote.

And you give up something subtler, which is the discipline that crude simulation forces on you. When the model is obviously wrong, you design for tolerance to being wrong: you vary parameters, add margin, and refuse to trust any single number the simulator gives you. When the model is gorgeous, you start believing it, and belief in a model is precisely the failure mode that sim-to-real is famous for. The cost is not that realism is bad. It is that realism is expensive in the currency you are short of.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the simulator has to drive the same software that will drive the machine, which covers most robotics work outside pure policy learning. Gazebo and ROS 2 grew up together, so a simulated robot publishes the same topics as a real one, your navigation stack cannot tell which it is talking to, and a simulated run can be recorded and compared against logs from hardware.

Add to that everything you would otherwise write yourself: coordinate frame handling, controller interfaces, visualisation, and a description format shared by half the tools you will ever touch. ROS 2 also wins for a team, because it is the shared vocabulary a new colleague already speaks.

HORUS is not the answer to any of that. A shared-memory middleware for one machine brings no simulator integration, no planner and no visualiser, so adopting it because the messaging story sounded appealing would cost months and buy nothing. The narrow case where a single-machine layer earns its place comes later, when mixed-language code on one onboard computer must exchange data on every cycle.

Does better physics make sim-to-real work?

No, and here is why. Sim-to-real does not usually fail because a model is imprecise. It fails because the model is wrong in some particular way and the policy learned to depend on that particular wrongness.

A policy trained in one very accurate world learns that world, including its exact friction, its exact masses, and any small quirk in how its contacts resolve. On the real machine one of those is different, and the policy has no experience of anything else. The strategies that actually transfer work in the opposite direction: vary everything you are unsure about across a wide range, train against that variety, and accept a policy that scores worse in simulation because it survives contact with reality. This is why teams with cruder simulators regularly beat teams with prettier ones.

The exception is real. When a specific physical effect dominates your task and your model omits it entirely, no amount of variation helps, because you are varying around a world that does not contain the phenomenon at all. Adding that one effect beats raising fidelity everywhere.

Is photorealistic rendering a waste of time?

Partly, but not the way you think. Rendering is largely wasted on the majority of robot tasks, where the policy never sees a picture and visual quality only affects whether the video looks good in a talk. But for anything consuming camera frames, appearance is not decoration, it is the input, and a model trained on flat grey surfaces under uniform light has learned a world that does not exist.

What is genuinely wasted is realism spent in the wrong place even within vision. Perfect materials on objects your robot never looks at buy nothing. Matching where the camera sits, how wide it sees, how it handles bright windows and how it smears a moving scene buys a great deal, and those are cheaper to model than they sound.

The other underrated use of rendering is for humans rather than models. A scene you can actually read makes it possible to see why the robot failed, and a surprising share of debugging is somebody watching a replay and saying that the arm went behind the object. Spend on realism your model consumes, and on clarity you consume.

How do you decide how much realism to buy?

Write down the single most plausible way your task will fail, then buy realism only in that dimension. The exercise takes an afternoon and saves months.

If your honest answer is that the gripper will not hold the object, you need contact and you can render everything in flat colours. If it is that the robot will not recognise the object in a real kitchen, you need appearance and camera modelling and you can accept approximate physics underneath. If it is that the robot will do the right thing at the wrong moment, no simulator setting helps at all, because the problem lives in the software layer running on the machine.

Then run the cheapest possible version of the entire loop before improving any single part of it, because the gap you find is almost never the gap you predicted. A crude end-to-end attempt that reaches real hardware in a week teaches more than a beautiful model that reaches it in a quarter. Why humanoid robots still move slowly is the same argument seen from the hardware end.

If you are training a manipulation policy -> contact fidelity first, because the failure is at the fingertips.

If you are training from camera frames -> appearance and camera modelling, because for a vision policy the picture is the physics.

If you are building a navigating robot with a team -> Gazebo with ROS 2, because a simulator driving the same code beats better contact.

If you already have hardware on the desk -> the crudest simulation that runs, because a measured gap beats a modelled one.

If your failures track the clock rather than the geometry -> stop tuning the model, because the problem is on the machine rather than in the world.

When you weigh the layer underneath any of this, the HORUS Fit Framework asks five questions with no numbers in them: ecosystem size, setup effort, team size fit, deployment target, and licence. Answered honestly, that shortlist usually collapses to one option.

If mixed-language code on a single onboard computer is where your project is heading, HORUS lives at github.com/softmata/horus under Apache-2.0 — star it so it is in your list when you start building.