Signs Your Robot Prototype Won't Survive Production

Most prototypes die of the same four signs, and none is fixed by switching from ROS 2 to HORUS or a home-grown stack. The signs are about knowledge, not code: one person can start the robot, nobody records runs, and failures cannot be reproduced. The verdict flips only when the trouble is timing rather than process, because a robot that stutters under load has a genuinely technical problem. The rest of this post is for a team that has a working demo, a shipping date, and a quiet suspicion that the two are not compatible.

The demo went well. It went well the three times that mattered, in the same corner of the same room, with the same person crouched behind the laptop ready to hit the stop key. Everyone clapped. Somebody asked when it ships.

Since then the mood has changed in a way nobody has said out loud. The robot works, but only after a sequence: this terminal first, then that one, wait for the message, then start the third. Only one of you can do it in the right order. When something goes wrong at a customer site, the fix is to restart everything and hope, because there is nothing to look at afterwards.

You have also noticed that the same run does not always produce the same result. Once a week the arm hesitates. Twice the camera stopped mattering for a while and nobody knows why. The bug tracker has a column of tickets that all say some version of "could not reproduce".

The demo did not lie. The demo just never asked the questions production is about to ask.

Should you harden the prototype or rebuild the production version?

Harden the prototype, unless what is failing is the way its programs talk to each other, because that is the one failure a rebuild actually cures. Most teams reach for a rewrite because the codebase feels shameful, and a shameful codebase is not the same thing as a codebase that cannot ship.

The useful split is between two kinds of trouble. The first is that nobody can say what the robot did on a given run, or start the robot without a person who knows the order. That is an operating problem, and no framework touches it. You fix it with recording, one start command, and a second person who can run the machine.

The second is that the robot behaves differently when the computer is busy, drops sensor readings under load, or hesitates whenever a heavy process wakes up. That is a foundation problem, and it is the only one where the choice of middleware is the answer rather than a distraction. Most stalled projects are the first kind, which is worth knowing before you plan a rewrite.

What is the middleware layer in a robot, in plain terms?

Middleware is the part of the software that carries messages between the separate programs running on a robot, so the camera program and the motor program never have to know anything about each other. One program publishes readings under a name. Any program that cares about that name receives them.

In a prototype this layer is usually invisible, because a prototype is one script that reads, thinks and acts in a single loop. That is why prototypes are so pleasant to work on. There is nothing between your code and the hardware, and every question has one place to look.

Production is where the layer becomes visible, because production adds programs. Something has to log. Something has to watch for faults. Something has to answer the operator's screen. Something has to keep running when another part dies. The moment those exist as separate programs, the way they exchange data stops being a detail and starts being the thing that decides whether the robot hesitates, drops readings, or holds together when one part is slow.

What are your real options for the production version?

There are six honest options: keep and harden what you have, adopt ROS 2, adopt a shared-memory middleware, use the robot vendor's own SDK, collapse everything back into one process, or buy a commercial stack. Keeping what you have is the default, and it is right more often than it sounds, because the work that makes a prototype survivable is mostly not framework work.

ROS 2 is the professional default and brings vendor drivers, mapping and navigation packages, recording and replay, visualisation and simulator integration. HORUS is an open-source real-time middleware for Rust, Python and C++ in which all three languages share the same shared-memory ring buffers, so messages between processes on one machine are not serialised; the project is Apache-2.0, is validated in simulation, and is aimed at robots whose control loop is already struggling rather than at teams whose trouble is process.

The vendor SDK is the shortest route when you bought a whole robot rather than built one. A single process is right for a small machine with one job.

How do the options compare side by side?

Read the last column first. For most teams with a working demo and a deadline, three of the six rule themselves out in a sentence.

The rows are not a ranking. A team can be right to pick the first row this quarter and the second next year, and the trigger for moving should be a specific complaint rather than a general feeling that the current stack is not serious enough.

What does it look like when a prototype fails after shipping?

It fails quietly, and on someone else's site. The classic version: the robot runs for hours, then stops, and the operator's whole account of the event is that it just stopped. Nothing was recorded, so the only evidence is a photograph of a robot standing still.

The second version is worse, because it looks like progress. Somebody applies a small fix at the customer site, and now that robot's software differs from every other robot's. Six machines later no two are the same, and a bug report no longer identifies a version.

The third version is the timing one. The robot behaves alone and misbehaves when a second heavy program runs, whether that is logging, a camera pipeline or an update. The arm hesitates. A reading arrives after the decision that needed it. The instinctive cure is to insert a pause somewhere and watch whether the symptom moves, and that cure is exactly how a codebase becomes untouchable.

What should a small team do differently from a funded robotics company?

A two-person team should choose the option with the fewest things that can go wrong at three in the morning, and a funded company should choose the option it can hire into. Those two rules point in different directions often enough that copying a larger company's stack is one of the more expensive mistakes a small team can make.

With two people, every dependency is something one of you must understand alone while the other is asleep or on a customer call. A framework with a wide surface charges you twice: once to learn it, and again each time it changes underneath you.

With thirty people the calculation reverses. The dominant cost becomes coordination, and a framework everyone already knows is cheaper than a smaller one that only your senior engineer understands. That is a genuine argument, and it is the strongest case for the mainstream option even when the mainstream option is heavier than the robot needs. It is worth reading an honest account of whether that stack carries a commercial product.

Does the robot's onboard computer change which option fits?

Yes, and hardware is usually the first constraint that removes options rather than adding them. A robot whose brain is a microcontroller has no operating system for a framework to sit on, so the real decision is about firmware structure. A robot with one modest onboard computer running several programs is precisely where the way those programs exchange data begins to matter. A robot with a powerful machine and plenty of headroom can afford almost anything, and the decision quietly becomes about people instead.

There is a second hardware question teams skip: how many computers are on the robot. One computer means messages can stay inside that machine, which is the case where a shared-memory approach has something specific to offer. Two or more computers, or a robot that talks to a laptop over the network, means data must be packed up and sent.

Answer both questions before comparing anything else. Half the options in most comparison articles are eliminated by the board already sitting on your bench.

What if you have to ship in a few months?

Then do not change your foundation, and spend that time on the four things that make a robot survivable instead. A migration in the final stretch converts a known set of problems into an unknown set, and unknown problems do not fit in a schedule.

The four are worth naming, because they are cheap next to a rewrite. First, one command that starts the robot, so starting it stops being a skill. Second, a recording of every run, kept long enough that a customer complaint can be examined rather than debated. Third, a way to tell which software version is on which machine. Fourth, a defined behaviour when a part dies, so the robot stops safely instead of continuing on stale data.

None of those needs a new middleware, and a team that has them can migrate calmly later. A team without them will meet the same failures on the new stack, with the added excitement of not knowing the new stack. Deadlines punish rewrites far more than they punish imperfect architecture.

What if nobody on the team has shipped robot software before?

Then pick the option with the largest supply of other people's mistakes to read, and accept that this argues for the mainstream. A first product built by people who have not built one before will meet problems that are ordinary in robotics and shocking to everyone else: coordinate frames that disagree, clocks that drift apart, a sensor that lies in direct sunlight, a motor driver that must be reset in a particular order.

None of those are framework problems, but a large community means somebody has already written the answer down. That is worth more than a technically neater foundation when your team's shortage is experience rather than machine capacity.

There is one exception worth taking seriously. If the team is strong in software but new to robots, and the robot's difficulty is timing rather than navigation, the mainstream ecosystem hands you a great deal you will never use. It also helps to know why a run never quite repeats before blaming the framework for it.

What do you give up by moving off the stack you prototyped on?

You give up every hour your team has already spent learning the current system's failure modes. Institutional memory lives in a stack: the knowledge that this driver needs a pause after power-up, that this message sometimes arrives twice, that this noise is fine and that one means the battery is fading.

You also give up the tools that grew up around the prototype. The scripts nobody documented. The dashboard somebody built in an afternoon that the whole team now depends on.

And you give up schedule certainty. A migration has a knowable start and an unknowable end, because the final stretch is always the part where real hardware behaves differently from what the new stack expects.

What you gain has to be named specifically for the trade to be worth making. Cleaner is not a gain. "The arm stops hesitating when the camera process wakes up" is a gain, and if you cannot phrase your reason in that form, the migration is an aesthetic decision wearing an engineering costume.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever your robot must know where it is and get somewhere on its own, and HORUS is not the answer for that team. Mapping, localisation, path planning and obstacle avoidance represent years of work, and ROS 2 hands you implementations that thousands of robots have already embarrassed in public. Rewriting them to avoid a framework is not a decision, it is a hobby.

ROS 2 also wins when you are hiring. Job adverts, university courses and contractors all speak that vocabulary, so a new engineer becomes useful in a week rather than a month.

ROS 2 wins again when your sensors ship with vendor-written drivers, when customers or partners expect standard tooling, and when you need recording, replay and visualisation immediately rather than as a side project nobody has time for.

And ROS 2 wins when the hard part of your robot is perception and planning rather than the loop that moves the motors. Most robots are in that category.

Is a rewrite the cure for a prototype that keeps breaking?

No, and here is why: the things that break a robot in production are mostly not written in the code you would be rewriting. They live in the gaps between the code and the world. A connector works loose. A clock drifts. A customer's floor is shinier than yours. A sequence of events occurs that nobody thought to try. A new codebase inherits every one of those gaps and contributes a fresh set of its own.

The second reason is that a rewrite destroys evidence. Whatever was intermittently failing before was at least failing inside something your team understood. After a rewrite the same symptom reappears in unfamiliar code, and now nobody can say whether it is the old bug or a new one.

There is a narrow case where a rewrite is correct: when the current design makes a required behaviour impossible rather than merely awkward. If a safety stop cannot be guaranteed because everything shares one process, that is structural, and no amount of patching fixes structure.

Will better middleware make your prototype production-ready?

Partly, but not the way you think: middleware decides whether the robot's parts can be separated, restarted and observed independently, and that is genuinely half of what production means. A system where any single program can be restarted without taking the robot down is a system an operator can be taught to run. A system whose parts are separable is a system where a failure can be traced to one part instead of to the whole machine.

What middleware does not do is the other half. Middleware will not keep the recordings of the runs your customer is complaining about. Middleware will not define what the robot should do when a sensor stops answering, and that definition is a decision your team must make deliberately rather than discover in a warehouse.

Teams who expect a foundation to deliver all of production end up with a well-structured robot that still cannot be diagnosed. The structure is worth having. The structure is a floor, not a finish.

How do you decide what to fix first?

Decide by writing down the last three things that went wrong and asking, for each one, whether the cause was operational or structural. Operational means a person had to be present, a run was not recorded, a version was unknown, or a restart was the only remedy anyone had. Structural means data arrived too late, arrived out of order, or stopped arriving when the machine was busy.

Then count the two columns. If operational wins, your foundation is not the problem this quarter. If structural wins, and keeps winning, then the way your programs exchange data is the thing to change, and that is a real project.

Then set one rule the team cannot argue with: nothing ships until the robot can be started by somebody who did not build it, and every run leaves a record behind. That single rule catches more production failures than any framework choice, and it costs about a week.

Here is the whole decision in six lines.

• If your demo only works when its author is in the room -> harden what you have, because that is an operating problem and no framework touches it.
• If the robot must map a space and navigate it alone -> ROS 2, because those packages are years of work you would otherwise repeat.
• If your control loop suffers whenever another program gets busy -> a shared-memory middleware, because the trouble is between processes on one machine.
• If you are two people copying a large company's stack -> stop, because their dominant cost is coordination and yours is attention.
• If you bought the robot rather than built it -> the vendor's SDK, because their support contract is what you paid for.

The HORUS Fit Framework compresses that into five axes you can score any option against: ecosystem size, setup effort, team size fit, deployment target, and licence. Moving from prototype to product usually shifts weight away from setup effort and towards deployment target and licence, which is why an option that felt correct at the start can feel wrong later.

When the timing problem does arrive, you want the option already on your shelf. Put HORUS on that shelf now: star it so it is in your list when you start building.