Best Robotics Stacks for Reinforcement Learning Projects

Train in Isaac Lab or MuJoCo, run the robot on ROS 2, and consider HORUS only when programs on the machine interfere. A reinforcement learning project is two stacks wearing one name: a training stack that needs volume and repeatability, and a robot stack that needs drivers and predictable timing. The verdict flips if your policy must run on a small onboard computer beside hungry cameras. The rest of this post is for someone who trains models and now has to choose the software a physical robot will run for the next year.

You have a policy that works. It walks in the viewer, it reaches for the block, and you have shown the video twice. Now somebody has asked when it goes on the robot, and you have realised you do not know what "on the robot" means as a piece of software.

Nobody in your reading list covered this part. The tutorials end at a trained checkpoint. The papers end at a table. The one repository that goes further has a launch file, three configuration formats and a build system you have never used, and its README assumes you know what a node is.

So you asked around and got four answers. Just use ROS 2. Never use ROS 2. Use whatever the robot came with. Write a Python script, it is only a control loop. Each sounded certain, and none asked what your robot was or how many computers it has.

Underneath it sits the thing you have not said out loud. The policy is the part you understand. Everything wrapped around it will decide whether this ends up as a product or another video.

Which stack should you build a reinforcement learning robot on?

Train in Isaac Lab when your group owns NVIDIA graphics hardware and in MuJoCo when it does not, then run the trained policy on ROS 2 on the robot itself. That settles it for most projects, decided by the hardware already in the building and the shape of the robot, not by which stack sounds more modern.

The two halves pull in opposite directions, which is why no single tool covers both well. Training wants volume, repeatability and a machine you can leave alone overnight. The robot wants drivers for its motors, a clean way to read its sensors, and programs that take their turn without treading on one another. Tools that try to be both end up mediocre at each.

The expensive mistake is letting the robot half happen by accident. Somebody clones the training repository, wires a Python script straight to the motor controller because that is the quickest thing to do on a Thursday, and a year later that script is the product. It works while the robot does one job on a bench. It stops working the week a second camera arrives.

What is a robotics stack, as opposed to a simulator or a training library?

A robotics stack is everything between your trained policy and the metal: the code that speaks to motors and sensors, the layer that carries messages between programs, the thing that starts it all in the right order, and the tools you stare at when it misbehaves. The simulator is not part of it. Neither is the learning library.

Naming the layers helps, because job adverts and README files mix them up. Drivers turn a manufacturer's protocol into numbers you can read. Messaging carries those numbers between the program watching a camera, the program running your policy and the program commanding joints. Orchestration starts those programs, restarts them when one dies, and tells you which. Tooling records what happened so you can watch it back slowly, which is how most real faults are found.

Your policy occupies one small slot in the middle of that. It is the part everyone discusses and the part with the least code. The layers around it are covered in more detail in what sits between the model and the motor, and they decide whether the robot behaves on a Tuesday when nobody is watching.

What are the actual stack options for a reinforcement learning project?

About seven combinations come up repeatedly, and most teams land in one of the first three. The common one is ROS 2 on the robot with Isaac Lab or MuJoCo for training, which brings drivers, coordinate frames, recording tools and people who have already hit whatever you are about to hit. The second is a vendor stack, where the robot arrived with its own software and your policy lives inside the maker's boundaries. The third is a plain Python runtime: your policy, a driver library and a loop, which is more honest than purists admit and survives longer on a small robot.

The fourth is a learning-first framework in the LeRobot style, shaped around demonstrations and datasets rather than around a robot's plumbing. The fifth is a middleware chosen for a single machine, such as HORUS, an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages between programs on one computer are not serialised. That project is Apache-2.0 and validated in simulation, and it trains nothing and replaces neither ROS 2 nor your simulator.

How do the reinforcement learning stacks compare side by side?

Read the last column first, because teams are usually eliminated into a stack rather than argued into one. The robot you bought, the number of computers on it and the people you can hire will cross out most of this table before anybody's preference gets a vote.

No column ranks raw speed, because speed is almost never the thing that decides a stack for a learning project.

Which stack fits the kind of team you are actually on?

Match the stack to the sentence you would use to describe your project to a stranger, because that sentence usually contains the answer already. Four descriptions cover nearly everyone who reaches this question.

If you are a research group publishing results, use ROS 2 on the robot and whatever simulator your field compares against. Reviewers and collaborators recognise it, students arrive already knowing it, and the cost of being unusual is paid every time somebody new joins.

If you are a startup with one robot and four engineers, start with a plain Python runtime and move deliberately. Your risk is not architecture, it is running out of runway before the robot does anything worth showing. Add structure at the first sign of programs interfering, not before.

If you inherited a bought platform, use the vendor stack until it stops you. Fighting a maker's software to gain purity costs a quarter and gains nothing a customer can see.

If you are one person learning, choose whatever gets a joint moving this week. Momentum is the scarce resource, and a stack nobody runs teaches nobody anything.

What hardware decides which of these stacks you can run?

Two hardware facts decide almost everything: whether your group has a capable NVIDIA graphics card for training, and how many computers your robot actually has. Everything else is preference.

On the training side, Isaac Lab needs a recent NVIDIA card with generous video memory, current drivers and a lot of disk. On a laptop with integrated graphics it does not run slowly, it does not run at all. MuJoCo runs on an ordinary processor, which is why it survives in groups where the good machine is booked out for a fortnight, and the fuller version of that trade-off is in which simulator to train a robot policy in.

On the robot side, the number that matters is the count of computers, not their strength. One computer means every message can stay inside one machine, and a middleware built for that case has a real advantage. Two or more computers, or a robot that talks to a laptop over the network, means you inherit network problems and want a stack that already handles them.

The trap is buying a bigger onboard board before establishing which fact actually limits you.

How long does each stack take before a policy moves a real joint?

A plain Python runtime can move a joint the same afternoon, a vendor SDK usually within a week, and ROS 2 typically takes a few weeks before a newcomer has a policy driving hardware without help. Those are the honest starting costs, and they are not the ones that matter.

The second timeline is the one nobody advertises: how long until the robot does its task reliably enough that you would leave the room. That is measured in months on every stack, because most of it is spent discovering that your policy assumed something the world does not provide. The gripper closes before the object is in it. The camera image is one step stale and the arm reaches where the block used to be. None of that is stack-specific.

So the install cost is not the decision. The decision is which stack you want to be inside during the months of debugging that follow, because that is where the hours go. Pick the one whose tools show you what happened rather than the one that started fastest.

What skills does each stack assume your team already has?

Every option here assumes fluent Python, and the differences start above that line. ROS 2 additionally assumes Linux comfort, patience with build systems, and somebody willing to learn how coordinate frames and message timing work rather than guessing.

A vendor stack assumes a tolerance for opacity. When something misbehaves inside the maker's code you cannot read it, and progress depends on a support channel rather than on your own debugging.

A single-machine middleware assumes somebody who thinks in processes: which program owns which piece of work, what happens when one of them stalls, and how data gets from one to another without a copy nobody budgeted for. That is a systems habit, and it is learnable in weeks by someone who wants it.

The skill nobody lists is knowing which layer a fault lives in. A machine learning engineer with no robotics background will blame the policy for a timing fault every time. How much of that background you actually need is worked through in how much robotics you need for embodied AI work.

What does it look like when the stack, rather than the policy, is the problem?

The signature of a stack problem is inconsistency: the same policy, the same starting position, and a different outcome each run. A policy fault repeats. A stack fault wanders.

Watch for four symptoms. The robot behaves better when you switch the logger off, which means programs are competing for something. The arm overshoots only when the camera is running, which means the policy is acting on a stale observation. Behaviour degrades over minutes rather than failing outright, which usually means a queue somewhere is filling. And the robot is fine on the bench and hesitant on the floor, which often means the wireless link joined the control path when nobody intended it to.

The tell that separates the two cases is cheap to run. Disable everything except the policy and the motor driver, and try again. If the behaviour becomes clean, the policy was never the problem and no amount of retraining will help. If it stays broken in the same way each time, you have a policy or a physics gap, and the training loop is where to go next.

What do you give up by building a robot around a learned policy?

You give up the ability to explain, in one sentence, why the robot did what it did. A learned policy is a large table of numbers that produces motion, and when it does something odd on a Tuesday there is no line of code to point at, no threshold to nudge, and nothing to show a customer asking what went wrong.

You also give up a kind of testing. A written controller can be checked case by case, and you can claim the arm stops before it hits the table because a rule says so. With a policy you can only report that it stopped every time you tried, which is a weaker claim and much harder to put in a safety document.

And you inherit a hidden dependency on the world the policy grew up in. Change a motor, add weight to the gripper, or swap a camera, and months of training can quietly stop applying in a way that editing a controller never would. That is worth knowing before you make learning the centre of a product rather than one component inside it.

When is ROS 2 the better choice?

ROS 2 is the better choice for most robots that leave a bench, and that includes most projects that describe themselves as learning projects. If the robot must build a map, avoid a chair, plan an arm motion around its own body, accept a new laser scanner next quarter, or be maintained by somebody you have not hired yet, ROS 2 is the answer and nothing else on this list is close.

The reason is not technical elegance. It is that a decade of drivers, planners, visualisers and answered questions exists, and none of it exists for the alternatives. When your gripper's protocol turns out to be undocumented, somebody has already written that driver. When you hire, you can hire for ROS 2 by name.

HORUS is not the answer to any of that. A middleware moves messages between programs on one machine, so it is the wrong tool for a project whose real need is drivers, navigation and a community, and recommending it there would cost a reader a season they cannot get back.

Do you need special middleware to run a trained policy?

No, and here is why: a trained policy is a function that turns numbers into numbers, and running it needs nothing more exotic than the code that already ran it during evaluation. Loading a checkpoint and calling it in a loop is genuinely all the policy requires.

What needs help is everything around the policy. Middleware exists because a robot is many programs at once, and those programs have to hand each other data without blocking, dying together, or arriving late. A single-process script that reads an encoder, calls a policy and writes a command needs none of that, and pretending otherwise sells complexity to somebody who does not need it yet.

The honest rule is that middleware earns its place at the second program, not the first. One program is a script. Two programs are a conversation, and conversations need rules about who speaks when. If your robot is still one program and it does its job, you have not yet met the problem middleware solves, and adopting one early buys you concepts to learn rather than faults to fix.

Will a bigger onboard computer fix a policy that hesitates on the robot?

Partly, but not the way you think. A larger board helps when the machine is genuinely saturated, and that is a minority of hesitating robots. Far more often the computer has capacity to spare and the problem is arrangement: one program is holding a resource another needs, or an image is copied more times than anybody counted, or the policy is waiting on data that arrives just after it needed it.

Three separate faults wear the same costume. Saturation, where everything slows together and a bigger board really does help. Contention, where the robot behaves well until a second program starts, and no amount of hardware fixes it because the two will still take turns badly. And staleness, where the numbers arrive intact but describe a moment that has passed, which produces an arm that keeps correcting for where the block used to be.

Tell them apart before you buy. Run the policy alone, then add one program at a time and watch which addition changes the behaviour. That afternoon is cheaper than a new board and answers a question the board would not.

How should you decide which stack to commit to?

Write down the one thing your robot must do before your next deadline, then pick the stack that gets you there with the least software you did not intend to learn. Not the best stack. The one that answers your question. Almost every regretted choice in this area comes from adopting a stack for a problem the project did not have yet.

• If the robot must navigate, sense and be handed over to other people -> ROS 2, because the drivers and the hiring pool already exist.
• If you have one small robot and one person -> a plain Python runtime, because structure you do not need yet is just homework.
• If the robot came with the maker's software -> the vendor stack, because fighting it costs a quarter and gains nothing visible.
• If your bottleneck is collecting demonstrations -> a learning-first framework, because the data is the project and the plumbing is not.
• If everything runs on one computer and the complaint is timing -> a shared-memory middleware, because the network was never your problem.
• If your requirements are genuinely strange -> your own layer, but only with somebody who will maintain it for years.

The HORUS Fit Framework reduces the wider choice to five things you can judge without running a benchmark: ecosystem size, setup effort, team size fit, deployment target and licence. For learning projects, deployment target and team size fit decide it most often, and licence matters more than expected on the day a client asks whether the whole system can be handed over.

The week your policy behaves in simulation and hesitates on the bench is a bad week to start reading about the layer underneath it. Put HORUS on the shelf before then: star it so it is in your list when you start building.