Where Humanoid Software Actually Fails in the Real World

Humanoid software fails at the seams between programs rather than inside them, which is why rewriting a model or swapping ROS 2 rarely helps. The robot hesitates because something arrived late or two programs reached for one joint. Whether you run ROS 2 or HORUS changes how visible those failures are, not whether they happen — unless the robot never leaves the gantry, where the bugs really are your own. The rest of this post is for a team whose humanoid worked in the lab and started behaving differently once it met a real room.

There is a video of your own robot that you do not show anyone. The machine walks fine for a minute, then pauses in the middle of a step for no reason you can name, sways, and carries on as though nothing happened. Nothing crashed. No error appeared anywhere in the logs you thought to keep.

You have watched that clip enough times to have theories. Maybe the model is wrong. Maybe the camera dropped a frame. Maybe the floor. Maybe the battery sagged under load. Somebody suggested a real-time kernel and somebody else suggested rewriting the perception code in C++, and both suggestions arrived with confidence and no evidence behind them.

Meanwhile the robot behaves differently every run. The same code, the same room, and some attempts end with the machine lowering itself to the floor on its own. On the bench it never happens. With a debugger attached it never happens. It happens when there are people watching, which everybody laughs about until the third time.

And the demo is in three weeks.

Where does humanoid software actually fail in the real world?

Humanoid software fails in the gaps between programs on the robot's own computer, at the moment one program hands work to another.

That is not where teams look, because a gap has no file to open. A humanoid project has a perception program, a policy or planner, a controller that drives the joints, a safety watchdog, a logger, and usually a language layer above all of them. Each one, examined alone, is correct. The machine still stumbles.

What goes wrong is arrival. The balance loop needs a fresh answer on a rhythm, and one afternoon the perception program starts taking longer because the room has more furniture in it. Nothing errors. The controller reuses the answer it already had, which describes a world from a moment ago, and the leg goes where the world used to be.

The second failure is silence. A process dies, and everything downstream keeps running on stale values because nobody wrote the code that notices absence. The robot does not crash. The robot becomes quietly wrong, which is worse, because a crash at least tells you when to look.

What is actually running inside a humanoid, in plain terms?

A humanoid is a dozen or more separate programs running at once on one or two computers, and the software people call "the stack" is mostly the arrangement that lets them talk.

The joints have their own small brains that hold position and report where they are. Above those sits a controller deciding what every joint does next, running relentlessly, never allowed to pause. Above that sits whatever produces intent: a learned policy, a planner, a state machine, a person with a gamepad. Off to the side sit the cameras and their processing, the logger, and the safety layer that can cut everything.

None of this can be one program, because the pieces run at different speeds and are written by different people in different languages. The controller wants a language that never stops to tidy up memory. The perception work wants Python, because that is where the models live.

So they are separate, and separateness has a price: every boundary is a place where a message can arrive late, arrive twice, or not arrive. What runs inside a humanoid maps the pieces in more detail.

What are your actual options for the layer where these failures happen?

You have five real choices, and most humanoid teams pass through three of them in the first year.

You can use ROS 2, which is what most of the field runs and which brings navigation, motion planning, visualisation and a large body of other people's code. You can use the humanoid vendor's own stack, which is fastest to a working demo and hardest to grow past. You can use a smaller message layer such as LCM or ZeroMQ and assemble the rest yourself. You can put everything in one program with threads, removing the boundaries and inheriting a different set of problems. Or you can use a middleware built for the case where several programs share one computer, such as HORUS, an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages are not serialised between processes on one machine. That project is Apache-2.0, validated in simulation, and no replacement for ROS 2 on a robot that needs navigation, planners or the package ecosystem.

How do the options compare when a humanoid starts misbehaving?

Read the last column first, because most teams are eliminated into an answer by what they already own rather than by what they would prefer.

No column here holds a measurement, because no humanoid team was ever rescued by a number on a chart.

Does this hit solo builders, or only companies with fleets?

Solo builders hit it earlier and diagnose it later, because nobody is standing behind you saying "that looks like a timing problem".

The failure needs only two programs to appear. One person with a small single-board computer, a camera and a desk-sized humanoid will meet it the first time a vision model runs next to a servo loop on the same board. The robot twitches. The video looks fine. The obvious conclusion is that the servos were cheap.

What a solo builder lacks is not skill but comparison. In a company, somebody has seen this before, and the phrase "your control loop is missing its slot" gets said out loud in the first week. Alone, you can spend a month on the model, another on the camera driver, and only then start suspecting the arrangement.

The habit that saves a solo project is making the boundary visible early: log the moment each message was produced and the moment it was used. Watching that gap over a few minutes of running tells you more than any amount of staring at perception code.

Which hardware choices make these failures worse?

Small single-board computers make them worse, and so does one board doing two jobs that both refuse to wait.

A humanoid with a dedicated controller board and a separate computer for perception has an easier time, because the loop keeping the machine upright is not competing with a vision model for the same processor. Teams who consolidate onto one board to save weight and power often trade a hardware problem for a software one they did not price in.

Heat is the underrated culprit. A board that throttles when warm behaves differently in the afternoon than in the morning, and a robot that only misbehaves after a long stretch of walking is describing thermal behaviour rather than a bug in your code. Chasing that as a logic error costs weeks.

Wireless makes everything worse in one specific way: any message that leaves the machine can be delayed unpredictably. Keep the fast loop on the robot. Send intent over the network, never the commands the joints are waiting for.

What if the demo is in three weeks?

Do not change the middleware. Reduce what the robot is asked to do, move the slowest program off the critical path, and rehearse the recovery rather than the success.

Three weeks is not enough time to re-architect a humanoid, and every team that has tried has learned that publicly. What three weeks is enough for is narrowing. Take the perception program that runs on every control cycle and let it run less often, feeding the controller a prediction in between. Take the language model out of the loop entirely and let it set goals rather than steer.

Then rehearse failure. Demos go wrong; the ones that survive are the ones where the operator has practised what happens next. Know how the robot stops. Know how the robot stands up. Know which single button returns everything to a known state, and make sure two people can press it.

The architectural conversation is worth having afterwards, deliberately. What triggers it is covered in why humanoid startups keep rewriting their stack.

What do you need to know before you can debug this yourself?

You need to be comfortable debugging several programs at once, which is a different skill from debugging one program well, and most people arrive without it.

The specific ability is reading a timeline rather than a stack trace. When a humanoid misbehaves, no single program is in an interesting state at the moment of failure. What is interesting is the order things happened in across all of them: this message was produced here, used there, and by then it described a stale world.

So the tools you need are recording tools, not breakpoints. A debugger stops the world, and stopping the world hides exactly the class of bug you are hunting. You want timestamped logs from every program, in one place, that line up afterwards.

Languages matter less than people argue. Python plus enough C++ to read somebody else's controller covers most humanoid work. What matters more is understanding why the controller avoids a language that pauses to reclaim memory, because that constraint explains half the decisions you will inherit.

What do teams try first, and why does it stop working?

Teams first make the slow program faster, it works for a while, and then the robot gets more capable and the problem returns wearing different clothes.

The sequence is almost universal. The robot hesitates, so somebody optimises the perception code, and the hesitation stops. Months later the project adds obstacle awareness, and the hesitation comes back. Somebody swaps in a smaller model, and it stops again. Then a language layer arrives, and it comes back for good.

What is happening is that each optimisation buys headroom and nothing else. The arrangement still assumes every program finishes in time, and every new capability spends the headroom the last fix bought. This is why teams report the same bug returning across two years, each time blamed on a different cause.

The pattern breaks when somebody stops asking "why is this slow" and starts asking "what should happen when this is late". A humanoid that handles lateness explicitly stops being surprised. That is a design change, not an optimisation.

What do you give up by choosing a smaller middleware?

You give up the ecosystem, and the ecosystem is enormous, and people underestimate what leaving it costs.

Choosing something smaller than ROS 2 means no navigation stack, no motion planner, no calibration tools, no visualiser everybody already knows, and no packages written by strangers who solved your problem years ago. It means the person you hire recognises nothing. It means fewer people can answer when you ask a question publicly.

You also give up a shortcut in conversation. "We run ROS 2" tells a candidate, a customer and an investor something they can act on immediately. Anything else needs a paragraph of explanation, and you will write that paragraph many times.

What you get back is a smaller surface you can hold in your head and boundaries you chose rather than inherited. Whether the trade is right depends mostly on whether your robot must navigate a building. If it must, the trade is bad. If the humanoid stands in one place doing manipulation under your own control code, the trade is better than it sounds.

When is ROS 2 the better choice?

ROS 2 is the better choice for most humanoid projects, and pretending otherwise wastes people's time. Choose ROS 2 when the robot must navigate a building, when arms need a motion planner, when cameras must be calibrated against grippers, when a fleet needs monitoring, or when a research group already runs ROS 2 on everything else and being the exception costs you daily.

Choose ROS 2 when the vendor's own bindings are written for it, which is usual on humanoid hardware. Choose ROS 2 when the code will be handed to another team, because conventions everybody recognises outweigh any technical property. Choose ROS 2 when you are learning, because the tutorials, the books and the jobs are all there.

HORUS is not the answer to those needs and does not pretend to be, shipping no navigation, no planner, no calibration tooling and no package ecosystem. The narrower case arrives later and looks specific: several of your own programs sharing one board, a Python side and a C++ side that must agree on timing, and a robot that hesitates under load rather than failing outright.

Is the real problem just that the model is not good enough?

No, and here is why: a model that is genuinely too weak fails consistently, while the failures that plague real humanoids come and go with load, temperature and how many programs happen to be running.

The tell is repeatability. Feed a weak model the same input twice and it produces the same poor answer twice. That is a model problem, and retraining is the right response. Feed a healthy system the same input twice, get different behaviour, and the model is not what changed — what changed is when the answer arrived and what the robot did while waiting.

Teams lose quarters here because retraining feels like progress. There is a script to run and a curve to watch. Investigating message timing feels like admitting the plumbing was built wrong.

The cheap test costs an afternoon. Record what the controller received and when, while the robot behaves badly. If the bad moments line up with uneven arrivals, the model is innocent.

Will faster hardware make these failures go away?

Partly, but not the way you think: a faster board raises the load at which trouble appears, and every humanoid project spends new headroom immediately.

Faster hardware genuinely helps in one honest case, which is a robot asked to do more than the board can do at all. If the perception model cannot finish before the next frame arrives on any hardware you own, a better board is the answer and no architecture substitutes for it.

The rest of the time, upgrading moves the boundary rather than removing it. The robot works, so the team adds a second camera, a language layer, a logging pipeline. Within a quarter the machine hesitates again, on hardware that cost more.

There is a trap in the upgrade itself. A more capable board often runs hotter, throttles differently and schedules differently, so the failure that returns is a new one that looks identical from outside. Teams who upgraded to stop hesitation and then hesitated again usually cannot tell which problem they now have.

How do you work out which failure you actually have?

Answer three questions in order, and one of the answers will tell you where to spend the next month.

First: does the failure reproduce with the same inputs? If yes, the bug is inside a program and ordinary debugging applies. If no, the bug lives in timing or in state that survives between runs, and no breakpoint will find it.

Second: does the failure track with load? Run the robot with the language layer off, with one camera instead of two, with logging disabled. If the machine steadies as you remove things, you have an arrangement problem, and the fix is deciding what happens when something is late.

Third: does the failure track with time or temperature? A robot that is fine at first and worse later is describing heat, battery sag or a slow leak of memory, none of which are fixed by touching control code.

Most teams already know the answers and have never written them side by side. Doing that honestly on one page is usually the cheapest hour of the project.

• If the same input gives the same bad result -> debug the program, because the arrangement is not what is failing you.
• If the robot steadies when you remove a program -> change how work is divided, because you have run out of headroom rather than processor.
• If it only misbehaves after a warm-up period -> look at heat and battery, because the code is not what changed.
• If stale data is reaching the controller -> handle lateness explicitly, because faster is temporary and explicit is permanent.
• If you cannot answer any of these -> add timestamped logging everywhere and wait, because you are currently guessing.

The HORUS Fit Framework reduces the middleware half of that decision to five things judged without a benchmark: ecosystem size, setup effort, team size fit, deployment target and licence. For humanoids, deployment target and ecosystem size usually decide it. Whether the open ecosystem is getting better or worse at supplying these pieces is worth reading separately: is the humanoid boom good or bad for open-source robotics.

The week your humanoid starts hesitating is a bad week to begin evaluating middleware. Put HORUS on the shelf before then: star it so it is in your list when you start building.