Best Simulators for Reinforcement Learning in Robotics

Train robot policies in Isaac Lab if you own NVIDIA graphics hardware, in MuJoCo if you do not, and in Gazebo essentially never. Reinforcement learning needs many copies of the same scene stepping at once under one training loop, which general-purpose simulators were never shaped to give. That flips when the hard part is the machine rather than the policy, which is a ROS 2 or HORUS question instead. The rest of this post is for someone with a machine learning background who has to pick a simulator this month and live with the choice for a year.

You have trained models before. You have never trained one whose output ends up as a torque on a joint, and that difference is turning out to be the whole problem. The tutorial you followed had a simulated creature learning to walk by dinner time, and it felt like everything after that would follow. Then you swapped in something shaped like a real robot and the reward curve went flat for a week.

Somebody in the lab told you to use MuJoCo. Somebody else said anyone serious is on Isaac Lab now. A third person sent a video of a simulator that renders like a film and asked why you would use anything plainer. None of them mentioned which of these runs on the laptop you were actually issued.

Meanwhile the good machine has one graphics card and three people who want it. Your last run took the whole night and died at dawn on a memory error you cannot parse. And underneath it sits the worry you have not said out loud: that the policy will learn beautifully in whichever simulator you pick, and then do nothing recognisable when it reaches an arm made of metal.

Which simulator should you use to train a robot policy?

Use Isaac Lab when your team has NVIDIA graphics hardware and the robot walks or reaches, and use MuJoCo when it does not, or when contact is the crux of the problem. That is the whole decision for most projects, and it is settled by the hardware already in the building and the shape of the task, not by which simulator is more advanced.

Isaac Lab exists to run enormous numbers of copies of one scene on a graphics card and feed a learning algorithm from all of them at once. If you are teaching a four-legged robot to cross gravel, or a humanoid to stay upright when somebody shoves it, that is the machinery you want, and building it yourself would cost you a season.

MuJoCo exists to model contact honestly and to stay small enough that a person can open the model file and understand what is being simulated. If the problem is a hand closing around an object, a foot rolling over an edge, or anything where the answer depends on how two surfaces meet, that work has lived in MuJoCo for years and the people who solved it before you left their code there.

The common mistake is choosing for the paper you hope to write next year rather than the experiment that is blocked this week.

What does a reinforcement learning simulator actually have to do?

A training simulator has one job that other simulators do not: run huge numbers of the same situation together and hand results back before the learning loop goes idle. Everything else follows from that, and picture quality does not make the list.

Three things matter. The first is stepping many worlds at once, because a policy learns from experience it has not had yet, and the only way to get that experience is volume. The second is resetting cheaply, since a training run spends much of its life throwing away a failed episode and starting over, and a simulator that pauses to think on every reset will spend the night thinking. The third is a clean seam where your code hands over an action and receives an observation and a reward, with nothing surprising in between.

There is a fourth that people find out about late, which is repeatability. If the same seed and the same actions give a different rollout every time, you cannot tell an improvement from noise, and a fortnight disappears into arguing with a graph. General-purpose simulators are tuned to look correct to a person. A training simulator has to be the same twice.

What are the real simulator options for reinforcement learning?

There are about six that matter, and most groups end up running two of them for different jobs. Isaac Lab, built on NVIDIA's Isaac Sim, is where legged locomotion and large-scale manipulation training happen. MuJoCo, open source under Apache-2.0, is the home of contact-heavy control, and its JAX version puts the same physics on an accelerator. Brax suits people who want the whole training loop inside JAX with nothing crossing a language boundary. PyBullet is still the quickest way to test an idea in Python on a laptop nobody else wants. Gazebo appears here mainly to be ruled out for training, because Gazebo is built to test a whole robot with its sensors, not to rehearse one motion until it is learned. And if you bought a robot, its maker may ship a tuned model of it, which beats any argument about physics engines.

One distinction saves a quarter of wasted work: the simulator is not the layer your robot's programs talk through. That layer is ROS 2, a vendor stack, or a middleware such as HORUS, an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages between programs on one machine are not serialised. That project is Apache-2.0 and validated in simulation, and it trains nothing, simulates no physics and replaces none of the tools ranked here.

How do the reinforcement learning simulators compare side by side?

Read the last column first, because most teams are eliminated into a simulator rather than argued into one. The graphics card already in the building, the language your group writes, and the week you can genuinely spare will cross out most of this table before anyone's preference gets a vote.

No column ranks physical accuracy, because for a training project accuracy is almost never the constraint that decides anything.

Which simulator fits the kind of work you are actually doing?

Match the simulator to the sentence you would use to describe your project to a stranger, because that sentence usually contains the answer. Four descriptions cover nearly everyone.

If you are a research student who has to publish, pick whatever the papers you are competing with used, because reviewers compare against numbers produced on that setup and reproducing a baseline in a different simulator is a term of work with no reward at the end.

If you are a startup with a manipulation product, MuJoCo first and a vendor model second. Your bottleneck is not throughput of episodes but knowing whether a grip holds, and you will iterate on the model of the object far more than on the learning algorithm.

If you are one person with a laptop and curiosity, use PyBullet or MuJoCo for a fortnight before you consider anything heavier. Momentum matters more than correctness while you are still learning what a reward function does to behaviour.

If you bought a legged robot and want it to walk somewhere new, Isaac Lab, because that is the exact shape of problem it was built around and the community you will ask for help lives there.

What hardware do you need before any of these will run?

Isaac Lab needs a recent NVIDIA graphics card with plenty of video memory, current drivers and a great deal of disk space, and on a laptop with integrated graphics it does not run slowly, it does not run. That single fact resolves the choice for more teams than any argument about physics fidelity.

MuJoCo runs on the processor in an ordinary laptop, which is why it survives in groups where the shared machine is booked out for a fortnight. The JAX version wants an accelerator, but it will happily use a rented one for a night, which turns a hardware purchase into an expense line.

The failure mode nobody warns you about is queueing. One good machine and three researchers produces a rota, and a rota produces experiments designed around when the machine is free rather than around what you want to learn. Two people running modest experiments continuously will out-learn three people sharing one large machine, almost every time.

Before you spend on hardware, work out whether the graphics card is genuinely your limit or whether it just feels like the limit, which is the same question people ask about simulation and graphics hardware in general.

How long will each option take before you see a policy learn?

MuJoCo and PyBullet can show a policy improving on the same day you install them, and Isaac Lab typically takes the better part of a week before the first custom scene trains. Those are the honest starting figures, and they are not the ones that matter.

What matters is the second timeline, which nobody puts in the marketing: how long until your own robot, your own task and your own reward function work. That is measured in weeks regardless of which simulator you chose, because most of it is spent discovering that your reward rewards the wrong thing. The classic version is a policy that learns to fall over in a way the scoring function likes, or a gripper that learns to knock the object off the table because the episode ends and the penalty never lands.

So the install time is not the cost. The cost is model authoring and reward design, and both are simulator-independent. Choose for the thing you will spend months on, which is iterating, not for the thing you spend an afternoon on, which is installing.

What skills does your team need to get value from each option?

Every option on this list assumes fluent Python, and the differences begin above that line. Isaac Lab additionally assumes somebody who is not frightened of graphics drivers, container images and multi-gigabyte installs, and who will keep that setup working when a driver update breaks it a month from now.

MuJoCo assumes somebody who can read a physics model and reason about contact, which is a mechanical engineering habit more than a software one. It is a small tool with a steep interior: getting a scene running is quick, getting a scene that behaves like the real object is a skill.

Brax and the JAX-based options assume a particular way of writing code, where everything is arrays and nothing is a loop with side effects inside it. A team that has not written in that style before will spend its first weeks fighting the framework rather than the robot.

The skill nobody lists is scepticism. Someone has to be willing to say that the graph is going up for a reason nobody wants, and to look at a video of the policy rather than the reward curve.

What do you give up by training a policy in simulation?

You give up the ability to explain, in a sentence, why the robot did what it did. A learned policy is a large table of numbers that produces motion, and when it does something strange on a Tuesday there is no line of code to point at, no threshold to adjust and nothing to show a customer who is asking what went wrong.

You also give up a certain kind of testing. A written controller can be checked case by case, and you can argue that the arm stops before it hits the table because a rule says so. With a policy you can only observe that it stopped every time you tried, which is a weaker claim and a harder one to put in a safety document.

And you inherit a dependency. The policy behaves as it does because of the exact simulated world it grew up in, so changing the robot's mass, swapping a motor or replacing a gripper can quietly invalidate months of training in a way that editing a controller would not. That is worth knowing before you make learning the centre of a product rather than a component in one.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the hard part is the robot rather than the policy, and that is the majority of projects that describe themselves as learning projects. Nothing on this list gives you a driver for a laser scanner, a coordinate frame tree, a map of a building, a navigation stack that avoids a chair, or a motion planner that keeps an arm out of its own body. ROS 2 gives you all of it, along with people who have already debugged what you are about to debug.

If your robot needs to move around a real building, ROS 2 is the answer and a training simulator is a component you plug into it. If you are hiring, ROS 2 is the answer because you can hire for it. If you are handing the system to somebody else in a year, ROS 2 is the answer because they will recognise it.

HORUS is not the answer to any of that. A middleware moves messages between programs on one machine, so it is the wrong tool for a project whose real need is a decade of drivers, planners and community answers, and pretending otherwise would cost a reader a quarter they cannot get back.

Is a photoreal simulator required for a policy that transfers?

No, and here is why: photorealism only helps the part of the robot that looks at pictures, and most policies that reach real hardware are not looking at pictures at all. A walking policy reads joint angles, forces and an orientation estimate. None of those get better because the grass was rendered convincingly.

Where appearance genuinely matters is when a camera image is the input to the network you are training, and that is a real and narrow case. If a policy has to recognise a red mug on a wooden counter, images that look drawn will teach it to recognise drawings, and it will fail on a photograph of your kitchen.

Even there, appearance is not the whole story. Teams get further by randomising the simulated world aggressively, so lighting, textures, masses and delays are never the same twice, than by making one beautiful world. A policy trained on many wrong worlds is more likely to survive the real one than a policy trained on one convincing world, and that is the practical shape of why simulated success keeps failing to transfer.

Will more training time fix a policy that fails on the real robot?

Partly, but not the way you think. More training makes a policy better at the world it was trained in, which helps only if that world resembles the one where it is failing, and if the failure is timing rather than skill then training longer makes the graph prettier and the robot no better.

There are three separate failures wearing the same costume. The first is a physics gap: the robot is heavier, the joints have play, the friction was guessed. That one repeats — the same mistake in the same place every run — and more randomisation during training genuinely helps.

The second is a perception gap, where the real camera sees something the simulated one never did. That repeats too, and it is fixed with real data rather than more episodes.

The third is a timing gap, and it does not repeat. The robot behaves differently on different days, worse when everything is running, better when you switch half the programs off. No amount of extra training touches it, because the policy is fine and the observations are arriving late. Run the robot with the heavy programs disabled before you buy another week of compute.

How should you pick a simulator this month?

Write down the one thing you need to learn before your next deadline, then pick whichever simulator answers that with the least model authoring. Not the better simulator. The one that answers your question. Nearly every regretted choice here comes from picking for a problem the project did not have yet, and the deeper split between the general-purpose simulators is covered in how the main simulators divide the work.

• If you are training a legged robot to walk somewhere new -> Isaac Lab, because many copies of one scene on a graphics card is the entire job.
• If the outcome depends on fingers, feet or friction -> MuJoCo, because contact is what MuJoCo was built to get right.
• If nobody in the building owns a capable NVIDIA card -> MuJoCo on the processor you already have, because a simulator one person can run is a simulator nobody runs.
• If you want the whole loop in one language with no boundary -> Brax, because moving data between frameworks is where runs quietly stall.
• If you are integrating a whole robot with drivers and sensors -> Gazebo, because that is a different question from training.
• If the policy is fine and the real machine stutters -> stop tuning rewards, because the fault is in the software around the policy.

The HORUS Fit Framework reduces the wider stack choice to five things you can judge without running a benchmark: ecosystem size, setup effort, team size fit, deployment target and licence. For simulators, setup effort and team size fit decide it most often, and licence matters more than people expect on the day a client asks whether the whole training setup can be handed over.

The week your trained policy behaves in simulation and hesitates on the bench is a bad week to start reading about the layer underneath it. Put HORUS on the shelf before then: star it so it is in your list when you start building.