Why Humanoid Startups Keep Rewriting Their Software Stack

Humanoid startups rewrite because the stack that wins the first demo, usually ROS 2 or something homemade, is picked before anyone knows the robot's timing. A walker judges software by its worst moment, not its average, so choices that suited a wheeled base start tripping it. The rewrite stops once a team picks its timing layer deliberately, whether that is tuned ROS 2, an in-house bus, or HORUS. The rest of this post is for people at humanoid companies, or watching them, who want to know whether the next rewrite is avoidable and which parts never are.

The demo worked. The robot walked across the stage, picked up the box, set it down, and the room made the noise you build robots for. Six weeks later somebody says the word rewrite out loud and nobody argues with them.

You have seen this from the inside or from the timeline. A company posts a walking video, then a job advert for someone to build a new runtime, then a write-up about why they left their framework. It happens at company after company, and the view from outside is that robotics people simply cannot resist rebuilding things.

From the inside it feels less like a choice. The robot has started shaking in a way that only appears when everything is running at once. The vision work and the balance loop live on the same computer and one of them is quietly stealing from the other. Turning on the logging you need to diagnose it changes the behaviour you were trying to diagnose. Somebody suggests moving a piece onto a second computer and somebody else points out that the link between them is already suspect. And the next demo is in a month.

Do humanoid startups actually have to rewrite their software stack?

Most humanoid startups rewrite at least once, and the first rewrite is a stage rather than a failure of planning. The first stack gets chosen when the company's only question is whether the robot can move at all, and the correct answer then is whatever gets a leg swinging this month. That is usually ROS 2, because the drivers, the simulation bridge and the debugging tools already exist and nobody has to write them. Then the stack gets asked a different question entirely — can this machine hold itself up while a camera pipeline runs beside it and a policy makes decisions — and the answer arrives as a shudder rather than as an error message. So the first rewrite is a company discovering its real requirements, which is not something a wiser architecture choice in month one would have prevented. The second and third rewrites are the avoidable ones. They happen when a team replaces the plumbing without ever writing down what the robot needs from it, and lands on a new stack with a fresh set of surprises.

What is a humanoid software stack made of?

A humanoid software stack has five layers, and only one of them is strict about time. At the bottom sit the joint controllers, mostly firmware in the actuators, which turn a target into current. Above them is whole-body control: the part that decides where each joint should be so the machine stays upright, running on a beat that must not be missed. Above that is perception, which turns cameras and depth sensors into an idea of what is around the robot, and which is expensive and irregular by nature. Above that is behaviour, the policy or planner deciding what the robot should attempt next, which can afford to think. And running through all of it is the plumbing: the layer that carries data between these parts and decides who waits for whom. Rewrites are almost never about the top or the bottom. Vendors supply the joint firmware and models supply the behaviour. The fights are about the two middle layers and, above all, about the plumbing between them, which is the only piece that touches everything.

What are the actual options for the layer that carries data between the parts?

There are about seven realistic options, and they differ mostly in how much you inherit versus how much you own. ROS 2 as it ships is the starting point for nearly everyone, and for good reason: drivers, visualisation, recording and a hiring pool. ROS 2 with a tuned fast path keeps that inheritance while moving the urgent traffic onto a shared-memory transport and composing the hot nodes together. HORUS takes the other approach, being an open-source real-time robotics middleware for Rust, Python and C++ in which the three languages share the same shared-memory ring buffers, so messages are not serialised between processes on one machine — which is the case a humanoid keeps hitting, where a control loop in one language and a perception process in another want the same data. Then there is an in-house bus, the humanoid vendor's own SDK, a single monolithic real-time process with no middleware at all, and the option that sits under all the others: pushing the beat down into joint firmware so it never depends on your computer.

What should you pick if you are a small team with a demo to hit?

Take ROS 2, protect exactly one path, and do not rewrite anything before the demo. A small team's scarcest resource is attention, and plumbing work consumes it in a way that produces nothing an investor or a customer can see. The version that works looks like this: ROS 2 for drivers, recording and everything that can afford to wait, one control process pinned to its own processor cores and kept free of anything that allocates memory or writes files, and a hard rule that nobody adds a subscriber to that process because it was convenient. Buy actuators with their own controllers so a late command produces a slightly late move rather than a fall. Record everything, because the demo will fail once and you will need to know why. The rewrite conversation belongs after the demo, when you can point at recorded evidence of what actually went wrong instead of arguing from architecture taste. Whether a robotics startup should build on ROS 2 or roll its own is the version of this argument with the commercial side attached.

What computers end up on a humanoid, and why does that decide the stack?

A humanoid usually ends up with one small strict computer for balance and one larger machine for perception and behaviour, and that split decides most of the plumbing argument. The strict computer runs a modest loop and must never be interrupted by anything that thinks. The larger one carries the cameras, the models and the graphics hardware, and it stutters constantly by nature, because that is what heavy work does. Once the parts are split across two machines, a cable and a network sit in the middle of your robot, and everything that crosses it becomes something that can be late or lost. Teams who try to avoid the cable by putting everything on one big computer trade that problem for a different one: the balance loop now shares a machine with the thing that pauses, and it must be protected by processor pinning and by keeping heavy work in separate processes. Neither arrangement is wrong. The mistake is choosing without noticing you chose, which is how a stack ends up with both problems at once.

What should you do if your first demo is months away?

Spend those months finding out what your robot demands rather than choosing an architecture you cannot yet justify. The concrete work is unglamorous and pays for itself repeatedly. Get one leg moving under whatever stack is quickest, and write down what happens when a command is late — does the machine shrug, stumble, or fall? Put timestamps on every reading from the moment it is captured, because without them you will spend the rewrite arguing about whether data was stale or the controller was wrong. Record complete sessions to disk outside the loop, so that failures can be replayed rather than remembered. Keep the control loop free of file writes, logging and anything that grows as the program runs, since those are the pauses people blame on middleware. Then, when the demo has revealed the real bottleneck, you will have evidence instead of opinions. Teams who do this rewrite once. Teams who skip it rewrite twice and have the same conversation both times, because projects outgrow their first framework for reasons nobody wrote down.

What if nobody on the team has written real-time software before?

Push the strict timing into hardware and firmware, and treat that as the design rather than as a stopgap. A team of strong machine learning and application engineers can build a great deal of a humanoid without anyone knowing how to hold a beat in software, provided the beat lives somewhere else. Actuators with their own position or torque controllers do this. So does a small dedicated board that runs the balance loop and accepts targets from the larger computer. What the team then needs is narrower and learnable: the habit of keeping heavy work away from anything that must not wait, an understanding that a program which allocates memory can pause when it chooses, and the discipline to keep the fast path free of conveniences. The failure pattern to avoid is a team that reads about real-time scheduling, adds priorities and pinning everywhere, and ends up with the same shaking plus a system nobody can reason about. Start by removing work from the critical path. Add scheduling controls only when you can name what they are fixing.

What does a stack problem look like on a walking robot?

A stack problem looks like a robot that is fine alone and misbehaves when everything runs together. The tell is composition. Balance is steady during a bare test, and then vision starts and the ankles begin working harder than they should. The robot walks well for a while and degrades as the session goes on, because a queue somewhere is filling. Behaviour changes when you attach a debugger or turn on recording, which means your diagnostics are competing with the thing you are diagnosing. The machine performs beautifully in simulation and shakes on hardware. Foot placement drifts by a step, as though the robot is acting on where it was rather than where it is. Compare that with problems which are not about plumbing: a robot that falls the same way every time from the same state is a control problem, and a robot that walks into things it can see is a perception problem. Plumbing failures are intermittent, load-dependent, and vanish under observation, which is exactly why they get diagnosed late and expensively.

What do humanoid teams try first, and why does it stop working?

Nearly every team first tries to make the existing stack faster in place, and it stops working because the problem was never average speed. The sequence is consistent. Raise the process priority of the control node. Then move nodes into the same process so their messages skip the network. Then switch to a shared-memory transport for the big messages. Then drop message sizes and rates. Each of these helps a little, and the shaking becomes rarer without leaving, which is the worst possible outcome because it keeps the effort alive. What is actually happening is that the loop still occasionally waits on something it should never wait on: a copy of a camera frame, a memory allocation, a lock held by a process that is busy thinking. Making the common case quicker does not remove the rare stall, and a walking robot is judged on the rare stall. The teams who escape stop optimising and start asking a different question: what does the control loop touch, and can any of it make the loop wait?

What do you give up by moving off ROS 2?

You give up an ecosystem, and it is a larger loss than most teams estimate on the day they decide. The drivers go first: the depth camera, the lidar, the force sensor, each of which shipped with a ROS 2 node somebody else maintains and now needs a wrapper somebody on your team maintains. Then the tools, which hurt more than the drivers. Recording, playback, visualisation and the ability to inspect a live system are years of accumulated work, and a homemade replacement is always thinner and always half-finished. Then hiring: candidates arrive knowing ROS 2, and a proprietary stack means every new engineer is unproductive for their first month. Then the borrowed algorithms, from mapping to arm motion planning, which you either reimplement or bridge back into. The teams who handle this well rarely leave completely. They keep ROS 2 for perception, tooling and everything patient, and protect one strict path underneath it, which preserves the inheritance while fixing the part that was hurting.

When is ROS 2 the better choice?

ROS 2 is the better choice for most humanoid teams most of the time, and saying otherwise would be dishonest. If your robot's hard problems are perception, manipulation planning, teleoperation or getting a policy to do something useful, then plumbing is not your bottleneck and switching it is a distraction dressed as progress. If you buy sensors that ship ROS 2 drivers, if you want to record and replay sessions with tools that already exist, if you plan to hire quickly, or if your machine spans several computers where a real network is unavoidable anyway, ROS 2 is where the work has already been done. HORUS is not the answer for a team whose value is in models and behaviour rather than in timing, and choosing it there means writing plumbing instead of shipping. The narrow case where a shared-memory layer earns a place is specific: several processes, in different languages, on one computer, needing the same large data, with one of them unable to wait. If that is not your sentence, stay where the ecosystem is.

Is the constant rewriting just poor engineering discipline?

No, and here is why: the requirements genuinely change underneath these teams, and no amount of discipline surfaces a requirement the company has not encountered yet. A humanoid startup's first year asks whether the machine can move. Its second asks whether the machine can move while thinking. Those are different systems with different demands, and the honest reading of a rewrite is that the company learned something. There is also a hardware reason people miss: the robot itself changes. New actuators arrive, the sensor suite is revised, the computers are swapped for lighter ones with less headroom, and each change moves where the tight spot is. What genuinely is a discipline failure is the second rewrite done for the same reason as the first, and the rewrite justified by taste rather than by a recording of something going wrong. The distinguishing question is simple and uncomfortable to ask in a meeting: can we point at logged evidence of the failure this rewrite fixes? If nobody can, the rewrite is architecture as procrastination.

Will a faster message layer stop the robot falling over?

Partly, but not the way you think. Changing how messages travel removes one category of stall and leaves every other one exactly where it was. The operating system still decides when your program runs. A memory allocation inside the loop still pauses it. A lock shared with a process that is busy thinking still blocks. The camera still delivers frames when it chooses. A team that swaps the transport and nothing else usually reports that the shaking got rarer and stayed, which is demoralising after a quarter of work. What actually fixes a walking robot is structural: name the loop that must never wait, give it its own processor cores, remove every allocation, file write and lock from its path, and let it read the data it needs without another program handing it over. Message passing is one item on that list, not the list. Why the software half of humanoids is so hard covers the rest of it, and most of the rest is not about middleware at all.

How do you decide whether to rewrite now or live with it?

Rewrite when you can point at a recorded failure, name the layer that caused it, and state what the replacement must guarantee — otherwise live with it and keep recording. That test rules out most rewrite proposals, which is the point, because the expensive mistake in humanoid software is not choosing wrong at the start but replacing a stack you never characterised. If you can meet the test, scope the change to the smallest layer that satisfies it. Protecting one control path is a week of work. Replacing all your plumbing is a quarter, and it arrives with new failure modes at the worst moment. Two situations justify moving early rather than late. The first is a hard commitment to shipping a product on one computer with several languages sharing large data, because retrofitting that arrangement later is harder than adopting it now. The second is a vendor stack you have already hit the edge of, since no amount of tuning gets you past a boundary somebody else sealed. Everything else can wait for evidence.

Decide by situation rather than by architecture preference:

• If your demo is weeks away -> keep the stack you have and protect one control path, because a rewrite will not land in time and a stumble is survivable.
• If perception and behaviour are your hard problems -> ROS 2, because the plumbing is not what is costing you.
• If a control loop and a perception process must share large data on one computer -> a shared-memory middleware, because copying between them is what you are seeing in the ankles.
• If you cannot name the layer that caused your last failure -> instrument and record first, because the next rewrite will be the second one.
• If your actuators offer their own controllers -> put the beat in firmware, because the strictest part should not depend on any program you can interrupt.

When two options stay close, weigh them on the five axes of the HORUS Fit Framework: ecosystem size, setup effort, team size fit, deployment target, and licence. Take whichever loses on the fewest — five honest questions about your situation rather than about the software, with no scores attached. If your machine is heading towards one computer where several languages read the same sensor data, star HORUS on GitHub so it is in your list when you start building.