From Hugging Face Model to Working Robot: The Steps Nobody Explains

The shortest path from a Hugging Face checkpoint to a moving robot runs through LeRobot on a supported arm, not through ROS 2 or HORUS. The model is the small part; what takes the months is data, calibration, a safety stop and the plumbing that gets images in and joint commands out on time. That flips once the robot has several programs of yours competing on one computer. The rest of this post is for someone comfortable with Python and models who has never made a physical machine move.

You have downloaded a policy. The model card shows a robot arm picking up a cube and placing it in a bin, and the code to load the weights is four lines you have written a hundred times before.

Then the page ends. There is a section called training, a section called evaluation, and nothing at all called what to plug this into. The example notebook takes an image and returns an array of numbers. Nobody says what those numbers are, what units they are in, or what happens to the arm when you send the first one.

You start searching and the results are worse than useless. Half of them assume you already own a specific arm. Half assume ROS 2 and a background you do not have. A forum thread says just publish to the joint controller, as though that sentence contained an instruction. A video shows the whole thing working end to end, filmed after months of work that nobody filmed.

And there is the quiet worry underneath: whether the machine will hurt itself, or you, while you find out.

What is the shortest path from a downloaded model to a robot that moves?

Buy an arm that LeRobot already supports, record your own demonstrations of one narrow task, fine-tune a pretrained policy on them, and run it on that arm. That is the whole path, and every part of it is deliberately unambitious.

It works because it removes the three things that stop people. The arm is supported, so drivers and calibration exist and somebody else debugged them. The data is yours, so the policy learns your camera angle and your table rather than a laboratory's. And the policy is pretrained, so you are adapting something that already understands what a hand looks like instead of teaching that from nothing.

What people try instead is picking the most impressive checkpoint on the hub and pointing it at whatever hardware they have. That fails at the interface, not at the intelligence: the model emits numbers the arm does not interpret the way the model meant them, and the arm moves somewhere alarming.

Start narrow. One task, one camera position, one arm. Breadth is a later problem and a much easier one. What LeRobot covers is set out in LeRobot explained.

What has to exist between a checkpoint and a moving joint?

A loop, running continuously, that nobody writes about because everybody assumes you already have one.

The loop does six things in order. It grabs a camera frame and the current joint positions. It shapes them into whatever the model was trained to receive, at the same size and in the same order. It runs the model. It converts the output into commands the arm understands, in the arm's own units and conventions. It checks those commands against limits, so a nonsense output does not become a nonsense motion. And it sends them, then does it all again before the arm has finished the previous move.

Every one of those steps is somewhere a project dies. The frame is a different size than training. The joint order differs by one. The model returns positions where the arm wants offsets. The safety check does not exist, because in a notebook nothing physical was at risk.

None of this is model work, and all of it is the job. The layer this loop lives in is described in what sits between the model and the motor.

What goes wrong the first time you point a model at real hardware?

The arm jumps. That is the single most common first experience, and it is worth expecting rather than discovering.

It jumps because the policy's first output describes a pose far from where the arm is standing, and a naive command says go there now. On a small arm that is a startling noise. On anything larger it is a broken fixture or a broken hand. The fix is unglamorous: move toward the first prediction slowly, and refuse commands that ask for more motion in one step than you have decided to allow.

The second thing that goes wrong is subtler. The robot performs the task almost correctly and then drifts, because the camera sits differently than during recording, or the table has a different sheen under afternoon light, or the object starts a hand's width left of where every demonstration began.

The third is that it works beautifully three times and fails the fourth with no visible difference. That is usually the loop running late rather than the model being wrong, and it is the hardest of the three to diagnose because nothing in the model changed.

What are the actual options for wiring a model to a robot?

There are five, and the right one depends on how many programs your robot runs. LeRobot is the first and the one most people should take: a supported arm, a recording workflow, a training script and a run script that already agree with each other. A vendor's own software development kit plus a single Python script is the second, and it is honest work when the arm is not on LeRobot's list. ROS 2 is the third, and it becomes correct the moment you want navigation, a second machine or anything a community has already built.

The fourth is a shared-memory middleware such as HORUS, an open-source real-time robotics middleware for Rust, Python and C++ in which the three languages share the same shared-memory ring buffers, so messages between processes on one machine are not serialised. That project is Apache-2.0, validated in simulation, and it is important to be clear that it is not an inference engine, trains nothing and will not make a policy any better. The fifth is a simulator with no hardware at all, which is a legitimate answer for months of the work.

How do the options for running a policy on a robot compare?

Read the first and last columns, because the honest constraint is almost always what you already own and how many people you are.

What can one person with a laptop and one arm actually achieve?

A single narrow task done reliably, which is far more than most people manage and far less than the videos suggest.

Concretely: an arm that picks one kind of object from roughly one region of a table and puts it somewhere, with a camera that has not moved since you recorded. That is a genuine result. It teaches you the whole pipeline, and every later ambition is a variation on it.

What one person does not achieve alone in a first project is generalisation. A policy that handles new objects, new lighting, a moved camera and an unfamiliar room is a different scale of undertaking, because the data behind that is not something an afternoon of teleoperation produces.

The useful move is to shrink the task until success is boring, then widen one dimension at a time. Add a second object. Move the camera. Change the lighting. Each widening tells you something specific about what your data lacked, which is the actual skill being learned. A broad first attempt teaches nothing except that it failed. If robotics is new territory entirely, the route in from Python and language models is worth reading first.

What hardware do you actually need before any of this works?

Four things, and one of them is the one people skip.

An arm with position feedback, so the policy knows where the joints are rather than guessing. A camera mounted so it cannot move, on a stand or a clamp rather than balanced on a book, because a camera that shifts between recording and running has silently changed the task. A computer that can run the model while also running the loop, which for vision-based policies generally means a graphics processor and for a small arm often means a development board rather than a desktop.

The fourth is a way to stop everything instantly that does not involve software. A physical switch, or a power strip your foot can reach. Every person who has worked near a machine has a story about the moment they needed one, and the ones who had it tell the story cheerfully.

What you do not need is an expensive arm. Low-cost arms in the LeRobot ecosystem exist precisely so the first project costs less than a phone, and a cheap arm teaches the same lessons a costly one does.

How long does the whole thing take from download to working demo?

Weeks rather than days, and the split is not where people expect.

Getting the model to produce numbers takes an afternoon. Getting the arm to move under your control takes another one, plus a day lost to a driver or a cable. Recording demonstrations takes longer than it sounds, because a session of careful teleoperation is tiring and your first batch is usually not good enough to keep. Fine-tuning is mostly waiting. And then the long part starts, which is the fortnight of small failures: the drift, the misgrasp, the run that worked yesterday.

Teams that plan this well book the calendar in that shape, with the hardware and the data on the critical path and the model treated as the easy part. Teams that plan it badly assume the model is the project, allocate a week, and spend the following month discovering that the interesting difficulties are all physical.

If you have a fixed deadline, cut the task rather than the loop. A smaller task delivered is worth more than an ambitious one that jumps.

What do you need to know beyond Python and models?

Four unglamorous things, none of which appear on a model card.

Units and conventions. Whether an angle is degrees or radians, which direction counts as positive, and where zero is on each joint. Most first-day disasters are conventions, not intelligence.

Calibration. What the camera actually sees relative to where the arm actually is. If the two disagree, the policy learns your error and hides it, then reveals it the moment anything moves.

Timing. That the arm expects a new command on a schedule and something has to keep that schedule while the model runs. This is the concept that separates a script from a control loop.

Safety. Limits on speed and reach, an idea of what happens when the model outputs nonsense, and a habit of keeping your hands out of the working area even when you are confident.

None of these require an engineering degree, and all of them require accepting that a physical machine does not have an undo. That shift, more than any library, is what makes the difference in the first month.

What do you give up by starting with LeRobot?

You give up generality, and for a first project that is a good trade.

LeRobot is built around a particular way of working: certain arms, certain policy families, a dataset format, a teleoperation workflow. Stay inside that and everything fits together. Step outside it, with an unsupported arm or a sensor nobody anticipated, and you are writing adapters against a moving library.

You also give up the wider ecosystem. LeRobot is not where you find a navigation stack, a mapping library, drivers for arbitrary hardware or motion planning around obstacles. If your robot eventually drives around a building, that part is ROS 2 work regardless of what trained the grasping.

And you give up some control over the loop. The convenience comes from decisions already made about how data flows from camera to policy to arm, which is exactly what you want on day one and exactly what you start to fight in month six, when your own perception program and your own policy program are both running and interfering with each other.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the robot is more than an arm and a camera, and HORUS is not the answer to any of the situations in this section.

It is better when the machine moves around a space, because navigation, mapping and localisation exist there and nowhere else in a form you can adopt in an afternoon. It is better when you have hardware nobody in the learning ecosystem supports, because a driver probably exists. It is better when several people are working on the robot, because separate programs with clear boundaries beat one large script every time. It is better in a university or a lab, where your collaborators and the papers you are reproducing already assume it. And it is better whenever the robot has to do a great many things that are not learned at all: charging, docking, reporting, waiting, recovering.

The pattern that works for many teams is both, with the policy loop kept apart from the rest. Middleware for a robot that runs a model on board covers where that boundary tends to fall.

Does a bigger model make the robot work better?

No, and here is why: almost every failure in a first robot project is physical, and physical failures do not respond to parameter count.

The arm jumped because nothing limited the step size. The grasp missed because the calibration was off by the width of a finger. The policy drifted because the camera moved. The run failed on the fourth attempt because the loop was late. A larger model changes none of that, and a larger model that answers more slowly makes the last one worse.

The place a bigger model genuinely helps is generalisation: handling objects and scenes you never demonstrated. That is a real benefit and it arrives after the physical pipeline is working, not instead of it.

The diagnostic question is simple. Does the robot fail in the same way every time, or differently each time. The same way every time is usually your data or your setup. Differently each time, with the model unchanged, is usually timing or hardware. Scaling the model addresses neither.

Can you skip the hardware and do everything in simulation?

Partly, but not the way you think: simulation is where you should write and debug the entire loop, and it is not where you find out whether the task works.

Everything structural transfers. The order of operations, the shape of the data, the units, the safety limits, the handling of a late frame, the recovery when the model produces nonsense. Writing those against a simulated arm costs nothing and breaks nothing, and the same code then meets the hardware already debugged.

What does not transfer is contact. How the gripper deforms the object, how much friction the table actually has, what a real camera does with a shiny surface, how a cable tugs at the wrist near the edge of the workspace. Manipulation lives in exactly those details, which is why a policy that succeeds in a simulator can fail on the bench in an entirely uninteresting way.

The workable sequence is to build the loop in simulation, collect demonstrations on hardware, and expect a gap. Why sim-to-real keeps failing is the honest version of that gap.

How do you decide what to do this week?

Decide by naming the one task you want to see, then buying or borrowing the arm that already supports it.

The mistake that costs the most is the survey. People spend a month comparing policies, frameworks and arms without touching hardware, and end the month knowing more and having moved nothing. The first physical result reorders every opinion you formed while reading, usually within an hour of the arm powering up.

So this week: pick one task a person could do without thinking. Get the arm moving under your own commands before any model is involved. Record demonstrations only after that works. Then fine-tune, then run, then start the fortnight of small failures that is where the actual learning happens.

Keep the middleware question closed until a second program exists. One script talking to one arm needs nothing between them, and reaching for a framework before that is how a first project becomes a research project.

• If you have no hardware yet -> LeRobot plus a supported low-cost arm, because the supported path removes the failures that stop beginners.
• If you already own an arm nobody supports -> the vendor SDK and one careful Python loop, because adapters cost less than a new robot.
• If your robot also has to drive around -> ROS 2 alongside the policy, because navigation and mapping are not things you write yourself.
• If perception, policy and control share one computer and the arm stutters -> a shared-memory layer under those programs, because copying is what is costing you.
• If the robot is a humanoid rather than an arm -> start with middleware for humanoid projects, because the timing question arrives much earlier there.
• If you are waiting on hardware -> build the whole loop in simulation, because that work is not wasted.

The HORUS Fit Framework reduces the middleware half of this to five things you can judge without a benchmark: ecosystem size, setup effort, team size fit, deployment target and licence. For model-driven robots, deployment target and setup effort usually decide it, and licence matters the day a demo turns into a product.

The week your robot starts hesitating between programs is a bad week to start reading about middleware. Put HORUS on the shelf now: star it so it is in your list when you start building.