Why a Robot That Works in Simulation Fails in Your Kitchen

Your robot fails in the kitchen because simulation hides timing, sensing and contact, and neither ROS 2 nor HORUS closes that gap alone. The simulator advanced its clock only when your code was ready; a kitchen has its own clock, reflective floors and a sagging battery. The one exception: if the robot misses its deadlines on real hardware, your gap is timing, not physics. This is for someone who trained a policy that behaves beautifully in a simulator and then watched it grope, stall and knock over a mug.

In the simulator the arm reaches, the fingers close on the mug handle, the mug goes on the shelf, and it works every single time you run it. On the counter in front of you, the same policy reaches, closes on nothing, and lifts an empty hand to the shelf with total confidence.

So you retrain. The next version is more careful, and now it stops halfway to the mug and sits there while the camera keeps streaming. You add noise to the training images. Better. Then the sun moves, the tiles by the window turn into a mirror, and the robot decides there is a hole in the floor.

The confusing part is that nothing crashes. There is no error message, no failed test, no stack trace to paste into a search box. Every process is running, every stream has data on it, and the robot is calmly doing the wrong thing. You cannot tell whether the model is wrong, the camera is wrong, or the whole stack is politely answering a question about a moment that has already passed. And every fix you reach for is another training run, because training runs are the tool you own.

Should you fix the simulator or fix the robot?

Fix the robot first, because a simulator will agree with itself forever. Every hour spent making the simulated kitchen more like your kitchen is an hour spent on a guess about which difference mattered, and the only place that guess can be tested is the real counter. The shortest path is the unglamorous one: put the policy on the machine, watch it fail in the room, and only then decide what the simulator was lying about.

That is uncomfortable advice for people who arrived from machine learning, where the loop is cheap and lives entirely on hardware you control. Robotics inverts the economics. The scarce resource is not compute, it is the number of times a physical object can try, and almost all the information sits in the first handful of real failures.

There is a second reason to start with the robot. Some of what breaks in a kitchen is not learnable at all. An image that reaches the policy after the arm has already committed is not a modelling error, and no amount of retraining teaches a network to repair a message that showed up too late to matter.

What does the sim-to-real gap actually mean?

The sim-to-real gap is not one gap but three, and treating the three as one is why fixes keep half-working. The first is contact. The simulated gripper and the real gripper disagree about friction, the mug is heavier on the handle side, the table has a lean you never noticed, and the wheels slip on grout lines.

The second is sensing. A simulated camera reports the world; a real camera reports light, and light does things the simulator was never asked to model. Glare off a kettle. Motion blur whenever the base turns. A depth sensor that returns nothing whatsoever where a glass of water is standing.

The third is timing, and it is the one that gets skipped. In a simulator the world waits for the policy: nothing moves until your code produces an action, so every observation is fresh and every action lands exactly where it was aimed. In a kitchen nothing waits. The mug keeps tipping while the model is still thinking, and if the answer arrives after the instant it described, the robot acts confidently on a world that has already moved.

What are your actual options for closing the gap?

There are several honest options, and most teams end up using three of them together rather than picking one. You can improve the simulator, randomising textures, mass, friction and lighting until the policy stops leaning on any single detail. You can move onto hardware early, testing with a deliberately unfinished policy in the first week. You can give the parts of the robot that must not hesitate their own microcontroller, so they never queue behind a general-purpose computer. You can record everything the robot saw and decided, so a confusing failure becomes something you can study repeatedly at your desk.

And you can choose what the robot's programs talk through. ROS 2 is the standard robotics middleware, with drivers, mapping, visualisation and simulator bridges written by other people. HORUS is an open-source real-time middleware for Rust, Python and C++ in which all three languages share the same shared-memory ring buffers, so messages between processes on one machine are never serialised; the project is Apache-2.0 and validated in simulation. Those two answer different questions, which the table below makes clearer than any paragraph can.

How do those options compare side by side?

Read the last column first and cross out every row that describes you, because for most people three of the seven disappear immediately.

What does the failure actually look like in a kitchen?

Kitchen failures are quiet, specific and physical, and they rarely resemble the failure modes you learned to debug in training. The gripper closes a hand's width behind the mug, because the depth sensor saw the reflection in the tiles rather than the tiles. The arm reaches perfectly and then stops, holding position, because a message it was waiting for never came and nothing in the stack thought that was an error.

The base rolls forward, catches a rug edge, and now the map and the room disagree by a small amount that grows all afternoon. A cupboard door that was open during every training run is closed today, and the policy has never in its life seen a closed cupboard.

The most instructive failure is the intermittent one: the same reach works, works, works, then misses, with nothing in the room having changed. That pattern almost always means the robot is not doing the same thing each time internally, which is a different investigation from any perception question. If that is your situation, why a robot behaves differently on identical runs is the more useful thread to pull.

Are you a machine learning engineer, a roboticist, or a product team?

Your background decides which of the three gaps you will find first, and which one you will miss for months. If you came from machine learning, you will instinctively treat every failure as a data problem, and you will be right often enough that the habit survives past its usefulness. The failure that finally breaks the habit is the one where the recorded image is perfect and the robot still did the wrong thing.

If you came from robotics, you have the opposite blind spot. You will suspect the clock, the driver and the wiring before you suspect the policy, and you will spend a week on message plumbing when the actual problem is that the model never saw a closed cupboard.

If you are a product team, the deciding factor is neither. It is that a demo in a customer's kitchen has different lighting, different floors and different clutter from your lab, and the gap that matters is between your lab and their home rather than between simulation and anything. Product teams should be spending their scarce real-world runs in rooms they do not control.

What hardware are you actually running on?

Whatever the robot carries on board is the machine your policy really runs on, and a laptop tethered by a cable is not that machine. Development commonly happens with a workstation doing the thinking and the robot acting as a set of limbs on the end of a wire. Everything about that arrangement is friendlier than the deployed one: the network is short, the compute is generous, and a person is standing right there.

Move the same stack onto the board bolted to the robot and things change character. The board is doing camera decoding, model inference, logging and motion at once, and those jobs compete. The symptom is not a crash but a hesitation, and hesitation in a moving machine looks like clumsiness.

Then there is the split most working robots settle into: a small board handling balance, joints or wheels with nothing else to distract it, and a bigger computer doing perception and planning above it. That split exists because some parts of a robot cannot be asked to wait. Understanding what a control loop is and why its timing matters explains why the split works better than any wiring diagram.

How much time do you have before the demo?

With a week, do not touch the simulator; spend the entire week with the robot, a stop button and a notebook. Run the policy in the actual room, in the actual lighting, and write down every failure by hand. A week of that produces a list of five or six specific things, and specific things can be fixed. A week of retraining produces a slightly different policy and no list.

With a month, you can afford one structural change alongside the hardware work. Pick the change your failure list points at, not the one that sounds most impressive.

With a season, rebuild in the order the robot taught you. Almost every team that has been through this twice ends up recording real runs from the first day, testing on hardware weekly rather than at milestones, and treating the simulator as a place to check that a change did not break the obvious cases rather than as the place where the robot is judged ready.

The pattern to avoid is spending the whole runway on simulation and meeting the room for the first time in the last fortnight.

How much embedded and systems experience does your team have?

If nobody on the team has debugged a program that misses its deadlines, the timing gap will cost you months, because you will keep explaining timing failures as accuracy failures. This is the single most common shape of a stalled embodied-AI project: strong modelling people, no one comfortable below the framework, and a robot that behaves unaccountably.

The fix is not hiring a systems person immediately. It is learning to ask one question of every failure: did the robot know the right thing and act late, or did the robot act promptly on the wrong belief? Answering that requires only that the robot writes down what it saw and when it decided, which any team can build.

If you do have systems people, the risk inverts. A team fluent in threads and buffers will happily rearchitect the message layer while the real problem is that the training data contains one kind of lighting. Both mistakes are the same mistake: reaching for the tool you are best at rather than the one the failure points to.

What do you give up by moving your work onto real hardware?

You give up cheap repetition, which is a genuinely large loss and worth naming honestly. In simulation you can run many attempts overnight, reset the scene instantly, try a variant while you sleep, and never worry about a servo burning out or a mug hitting the floor. On real hardware every attempt costs setup, supervision and wear, and some attempts cost a broken part.

You give up reproducibility. A simulated run can be repeated exactly; a real run cannot, because the battery is lower, the light has moved and the mug is a finger to the left. That makes bug hunting harder and makes recording essential rather than optional.

You give up safety margin, too. A policy that fails gracefully in a simulator can shove a real gripper into a real countertop, so you now need limits, a stop button and someone watching.

What you buy in exchange is the only currency that counts: knowledge of what your robot actually does in the room where it will live. Everything else is a proxy for that.

When is ROS 2 the better choice?

ROS 2 is the better choice for most teams facing a sim-to-real problem, and HORUS is not the answer when the gap is physics, perception or data. If your robot needs a lidar driver, a mapping stack, coordinate frame bookkeeping, a visualiser that shows what the robot believes, or a bridge to the simulator you already use, ROS 2 has all of it and the alternatives do not.

Use ROS 2 when your hardware ships with ROS 2 drivers, which most research and commercial platforms do. Use it when you want recorded runs in a format your colleagues already know how to open. Use it when the team is more than a couple of people, or when new members must be productive without a week of onboarding into a house-built system.

Use it, above all, when your failures are about what the robot believes rather than about when the robot knows things. Most kitchen failures are belief failures. Swapping the message layer to chase a timing problem you have not confirmed is the expensive version of guessing, and the guess is usually wrong.

Is the answer simply a better simulator?

No, and here is why: a better simulator narrows the gap it already models and does nothing at all about the gap it does not. Contact modelling, lighting variation and sensor noise are all things a good simulator can be pushed to represent, and pushing it is worthwhile. The category it cannot represent is the one where your own software is the environment: the camera process falling behind, the log writer stalling on a full disk, the model taking longer on a cluttered scene than on an empty one.

There is a subtler trap in simulator investment. The more faithfully a simulator reproduces your lab, the more confidently a policy learns things that are true only in your lab. Fidelity and generality pull against each other, and a team optimising for fidelity often gets a robot that is superb in one room.

Choosing a simulator still matters, because they differ in what they model well. How the main simulators compare for real projects is worth reading before committing months to one.

Is the kitchen gap really just a physics problem?

Partly, but not the way you think: physics explains why the grasp slips, and it never explains why the same grasp works four times and then fails. Contact modelling is genuinely hard, and a mug that rotates in the fingers is a real physical mismatch that better modelling and randomisation can improve.

But the failures that consume whole weeks are almost never the physics ones, because physics failures are repeatable. You can stand there and watch the mug rotate the same way ten times, and repeatable problems get fixed. The ones that burn the calendar are the failures that come and go, and those come from ordering rather than from mechanics: which piece of information reached which program, in what order, and whether the number the arm used described the world at the moment the arm moved.

The practical test is cheap. Do the same failure at the same moment in the routine and it is probably physics. Same room, same setup, random timing, and it is probably not.

How do you tell which problem you actually have?

Make the robot record what it saw and what it decided, then look at those two together after a failure. That single habit separates the three gaps faster than any other technique, and it costs an afternoon to build. If the stored image does not contain the mug, the problem is sensing. If the image contains the mug and the decision is wrong, the problem is the model or its training data. If the image contains the mug, the decision is right, and the robot still missed, the problem is when things happened.

The second technique is to remove things. Run the policy with the logger off, then with a lower camera resolution, then with the display closed. If the behaviour improves as you remove load, you are looking at a robot whose parts are getting in each other's way, and no amount of retraining will help.

The third is to change one thing in the room. Move the light, then move the mug, then change nothing and simply run again. Which change breaks the robot tells you which gap you are in.

Here is the whole decision in five lines.

• If your robot has never left the simulator -> hardware in the loop this week, because a list of real failures is the only reliable plan.
• If the same failure repeats identically -> contact and perception work, because repeatable failures are modelling failures.
• If failures come and go with nothing changing -> look at ordering and timing, because the room is not the variable here.
• If you need drivers, mapping and a visualiser -> ROS 2, because that work is already done and freely available.
• If your loop slips only once cameras and logging run together -> a shared-memory middleware, because the trouble is between your own programs.

The HORUS Fit Framework reduces the choice to five axes: ecosystem size, setup effort, team size fit, deployment target, and licence. For a team crossing from simulation into a kitchen, ecosystem size decides the first year and deployment target decides the second.

When the robot leaves the tether and the loop starts slipping, you will want the shared-memory option already on your shelf rather than found in a panic. Put HORUS on that shelf now: star it so it is in your list when you start building.