What Software Runs Inside a Humanoid Robot?

A humanoid runs six layers of software, from joint firmware to behaviour, and the middle layer is ROS 2, a vendor stack or your own. Balance and safety live close to the motors and never move; perception and behaviour sit above them and change weekly. The verdict on the middle layer flips once several programs on one onboard computer start interfering, which is where HORUS belongs. The rest of this post is for someone planning a humanoid, or unpacking a bought one, deciding which layers to write and which to adopt.

You have watched the videos. A machine folds a towel, catches itself on a kerb, carries a crate across a warehouse, and the comments underneath argue about whether any of it is real. What no video tells you is what is actually running on the robot while it does those things.

The explanations you find are contradictory in a specific way. One says humanoids are end-to-end now, a single trained model taking in camera frames and putting out joint commands, no traditional software involved. Another describes a stack of controllers, estimators and planners that sounds like an aircraft. Both are written confidently. Both cannot be describing the same machine.

Meanwhile something concrete is coming. A small humanoid is on order, or a bought platform sits in a crate with an SDK you cannot see inside, or your team has been asked whether a legged machine could do a job that a wheeled one cannot. You are not looking for theory. You want to know which parts you will have to write, which parts you can take from somebody else, and which part catches people out.

What are the layers of software inside a humanoid robot?

Six layers, sorted by how quickly each one has to react. At the bottom is firmware in the joints: small boards next to each motor that hold a commanded torque or position and answer the body over a wired bus. Above that is the balance and whole-body layer, which reads the inertial sensor, the joint positions and the forces at the feet, works out where the body actually is, and commands every joint again, continuously, without pausing.

Beside those two sits a safety layer, part software and part wiring: limits, temperature checks, a watchdog that notices when something has stopped answering, and the behaviour when power is cut, which for a legged machine is never simply going limp.

Above them is the messaging layer that moves data between programs, then perception, which turns camera and depth images into an understanding of the room and where the robot is in it. On top sits behaviour: choosing what to do next, sequencing a task, deciding to give up and ask a human.

The bottom three change rarely. The top two change every week. That difference in pace explains most of how humanoid teams organise themselves.

What does middleware do inside a humanoid?

Middleware is the part that carries data between the programs on the robot so they do not have to know about each other. Perception writes what it sees, the behaviour layer reads it, the controller publishes what the body is doing, a logger records everything, and none of those programs contains the address or the schedule of any other.

That sounds like a small convenience and is not. A humanoid has many producers and many consumers of the same data, and the moment there are more than about three programs, connecting them directly becomes a job nobody wants. Middleware also provides the things you only miss at three in the morning: a way to see what a program is publishing without stopping it, a recording of a run you can replay at a desk, and a single place where a message format is defined.

What middleware does not do is any of the robot's actual work. Nothing in it balances, plans, or recognises a mug. Choosing it well makes the work possible; choosing it badly makes the work fight itself, and what middleware actually does in a robot covers the general case in more detail.

What are the real options for the software that connects everything?

There are six arrangements humanoid teams actually use, and most robots contain at least two of them. ROS 2 is the common answer above the control layer, bringing perception packages, tooling, recording and a hiring pool. A bought humanoid usually arrives with the maker's own stack and an SDK, which is the fastest route to a moving machine and the narrowest one. Larger companies write a custom layer, usually in C++, tuned to their exact hardware. Below all of them, the joints speak a wired bus with firmware at the far end, and that layer is rarely a choice you make.

The remaining option is a shared-memory middleware such as HORUS, an open-source real-time robotics middleware for Rust, Python and C++ where the three languages share the same shared-memory ring buffers, so messages between programs on one onboard computer are not serialised. That project is Apache-2.0 and validated in simulation. It addresses programs on one machine getting in each other's way, and it does not supply balance control, perception or planning, which on a humanoid is most of the difficulty.

How do the options compare for a humanoid?

Read the last column first, because a humanoid project is usually eliminated into an answer rather than argued into one. Whether the hardware was bought or built, and how many engineers exist, remove most of this table before anyone gets to express a preference.

Are you a research lab, a startup, or one person with a small humanoid?

Labs should take ROS 2, startups should take whatever their hardware came with until it hurts, and an individual should take the SDK in the box. In a lab the currency is comparability: a controller nobody else can run is a controller nobody cites, and the packages, message types and recording formats other groups use are worth more than any private arrangement.

A startup faces a different clock. The stack is not the product in year one, the demonstration is, and teams that rewrite the vendor layer before knowing what the robot must do usually discover their assumptions were wrong after the rewrite. The right moment to take control of a layer is when that layer has visibly become the thing stopping you.

One person with a small humanoid, one of the desktop machines that now cost less than a laptop, should write behaviour and leave everything below it alone. The walking already works. Making the robot do something interesting with its arms and its eyes is a full project, and it is the part that teaches you what the rest of the stack is for.

What computers are actually inside a humanoid?

At least three kinds, and knowing which is which prevents most architectural mistakes. There are microcontrollers in the joints, one per actuator or per group, running firmware whose only job is to keep a motor honest and answer the bus on time. They have no operating system worth the name and cannot be asked to do anything else.

There is a main computer running Linux, usually with a real-time kernel patch, carrying the balance controller and the state estimator. That machine is deliberately boring, and one of the strongest habits in humanoid engineering is refusing to put anything else on it.

Then there is a computer with a graphics processor for perception and learned models, which is warm, hungry and shared by every program that wants a picture understood. This is where crowding actually happens: a detector, a mapper, a language model and a logger all want the same accelerator, and each one slowing the others is a common and confusing failure. A robot's onboard computers are also busy with things you did not write, which what runs on a robot besides your own code sets out.

How soon does the humanoid have to do something real?

If the demonstration is weeks away, use the stack the robot came with, and if it is a year away, decide the layers deliberately. Humanoid projects lose their time in unexpected places: a calibration that drifts after every fall, a hand that needs rebuilding, a battery that sags under load, a fortnight spent discovering that the floor in the demonstration room is more slippery than the floor in the lab.

Software layering is not the hard part of that schedule, but it competes for the same weeks, and a team that spends its first quarter designing message infrastructure arrives at the demonstration with beautiful plumbing and a robot that has never picked anything up.

The order most experienced teams use is to build the top down and the bottom last. Get the robot doing the task with whatever is in the box, find out where the box's limits actually are, then replace the layer that turned out to matter. Doing it the other way, choosing the messaging layer first, means committing before you know which of your programs will be the heavy ones.

What does your team already know how to do?

The skills already in the room decide more than the architecture diagram does, because a humanoid punishes learning on the job in a way a wheeled robot does not. A team with control engineers is comfortable in the balance layer, thinks in terms of estimation and feedback, and will usually want its own controller sooner than is wise. A team from machine learning will want to train the behaviour and will underestimate how much written logic surrounds any policy that runs on real hardware.

A team of software engineers with no robotics background tends to make one specific mistake, which is treating the robot as a distributed system and putting a network between things that should never have been separated.

The language mix is unavoidable here. Control code is C++, learning code is Python, firmware is C, and new stacks increasingly use Rust, and those languages have to share data on one machine every cycle. That boundary is where humanoid software gets untidy, and it is worth deciding on purpose rather than discovering later.

What do you give up by writing the stack yourself?

You give up years, and that is not a figure of speech. The pieces that look simple in a diagram, a state estimator that stays honest after a fall, a controller that handles a foot landing on a cable, a safety layer somebody will sign off on, are each months of work by people who have done it before, and they are boring months without a demonstration at the end.

You give up tooling too, which is the cost people forget. Recording a run and replaying it at a desk, watching a value change live, a simulator connection that works, a way to introspect what a program believes: all of that arrives free with an established stack and has to be built, badly and twice, in a private one.

And you give up hiring. Somebody who knows ROS 2 is productive on your robot in a fortnight. Somebody joining a private stack is productive when the one person who wrote it has time to explain it, which is never during the month you need them.

When is ROS 2 the better choice?

ROS 2 is the better choice for most humanoid work above the control layer, and HORUS is not the answer for a team whose real gap is that the robot cannot see, map or plan yet. Perception, manipulation planning, coordinate frame handling, recording, visualisation and simulation connections all exist there, written and debugged by people who spent years on them, and rewriting that to gain anything is a bad trade.

ROS 2 also wins in research outright. A humanoid paper with code that runs is worth several without, students arrive already knowing the tools, and a collaborator can run your controller on their machine without a week of setup.

And it wins when a robot must work with other machines. Once a humanoid shares a building with wheeled robots, lifts, doors and a fleet manager, common interfaces stop being a preference. The honest limitation is at the bottom: almost nobody puts the balance loop under general middleware, because that loop belongs close to the motors, which is what a control loop is and why its timing matters.

Is a humanoid just a wheeled robot with legs?

No, and here is why: a wheeled robot that stops computing sits still, and a humanoid that stops computing falls over. That single difference reorganises everything. On a wheeled machine, a program that is late produces a jerky path or a delayed stop. On a legged machine, a program that is late during a step produces a two-metre object hitting the floor, and every design decision downstream is shaped by preventing that.

It changes what the safety layer must be. Cutting power to a wheeled robot is safe; cutting power to a standing humanoid is a decision about how it falls, which is why these machines have deliberate collapse behaviours and sometimes mechanical brakes.

It changes the estimator. A wheeled robot usually knows roughly where it is from its wheels. A humanoid has no such comfort: position comes from combining an inertial sensor with the geometry of whichever foot is on the ground, and the estimate drifts every time a foot slips.

And it changes the arms, which touch people and are heavy enough to matter.

Can one learned model replace all of this software?

Partly, but not the way you think: learned models have genuinely replaced large parts of the walking and manipulation layers, and they have not replaced the software around them, which is most of the code on the robot. A locomotion policy trained in simulation is now an ordinary way to make a humanoid walk over rough ground, and it beats hand-written controllers at exactly that. Those policies are trained in tools like the ones covered in choosing between Isaac Sim, Gazebo and MuJoCo.

What surrounds the policy is still written by hand: the joint limits, the checks on what the policy is asking for, the watchdog, the fall behaviour, the state estimator feeding it, the code that decides when the policy is out of its depth and something simpler should take over.

The reason is not scepticism about learning. It is that a trained model cannot promise what it will do in a situation nobody trained it on, and a machine that weighs as much as a person needs somebody able to promise what happens next. So the model handles the skill and written logic handles the consequences.

What does it look like when the software is the problem?

It looks like a robot that works perfectly in a quiet lab and behaves differently the moment everything is running at once. The classic version: the humanoid stands rock steady, then somebody starts the perception program, and the machine begins to sway. Nothing changed in the controller. The controller simply stopped receiving what it needed in time, because two programs are now sharing one computer and neither was told who has priority.

The other signature is the demonstration that degrades. The first ten minutes are clean, then the machine gets warm, the processor slows itself down to survive, and the walk becomes uneven in a way nobody can reproduce on a cool morning.

A third is the arm that stops before it hits the table on most days and does not on one of them, because the picture it was acting on described the table where it used to be.

These are timing problems, not logic problems, and no amount of reading the code finds them. The way to tell is to run the same motion with the heavy programs switched off. If steadiness returns, the code is fine and the arrangement is not.

How do you decide what to build and what to adopt?

Decide by asking which layer is stopping you today, and refusing to touch any other one. Every humanoid team has exactly one layer that is currently the constraint. Working on any other is displacement activity, and the reason it happens is that the constraining layer is usually the least pleasant one.

Here is the decision in five lines.

• If you bought a humanoid and need a result this quarter -> the maker's stack, because the fastest route is the one already running.
• If you are in a lab and want your work to be read -> ROS 2, because comparable work is worth more than private work.
• If the robot cannot yet see, map or plan -> take existing packages, because writing perception is a project rather than a task.
• If the robot is steady alone and unsteady with everything running -> a shared-memory layer, because the trouble is between programs on one machine.
• If the balance loop itself is late -> move it closer to the motors, because no messaging choice fixes a loop on a crowded computer.

The HORUS Fit Framework reduces the choice to five axes you can judge without a benchmark: ecosystem size, setup effort, team size fit, deployment target and licence. On a humanoid, deployment target usually decides, because everything must run on the machine itself, and licence matters the day a customer asks what is inside the robot standing in their building.

The week the controller starts missing what perception sends is a bad week to begin evaluating middleware. Put HORUS on the shelf before then: star it so it is in your list when you start building.