Middleware for Research vs Middleware for Production

Research and production middleware are judged differently: ROS 2 wins research outright, while production is genuinely open between ROS 2, HORUS and a trimmed stack. Research buys breadth and forgives upkeep; production buys what a small team can maintain, restart and explain to an auditor. That flips the moment the shipped robot still depends on the ecosystem parts that made the research fast. The rest of this post is for teams whose prototype works and who must now decide whether the stack that got them here ships with the product.

The prototype works. It has worked for months, on the machine under the desk, with the terminals opened in the right order by the person who knows which one goes first. Then a date appears on a slide, and the questions change. Can this run unattended? What happens when the customer's network is not the lab's network? What happens when it falls over in the middle of the night and the only evidence is a log file somebody has to read on a phone? A new engineer joins and spends the first fortnight not writing robot code but reconstructing an environment nobody wrote down. Someone asks which open-source licences are in the shipped image, and the room goes quiet. Nothing has broken, exactly. The robot still does what it did in the demo. But every question that used to be answered with "ask the person who built it" now has to be answered with a document, and the stack was chosen in week one by someone optimising for a completely different outcome.

Should research and production run on the same middleware?

Usually yes at the start, and usually not forever. Research asks a middleware one question: how much of this robot can I avoid building? The answer decides whether the idea is testable this quarter, and every hour spent on plumbing is an hour not spent on the thing being studied. Production asks something else: what will I still be able to fix, on a machine I cannot see, long after the person who set it up has moved on? Those questions are not opposed, but they rank differently. A stack generous with parts is also generous with moving parts, and every moving part is something that can differ between the bench and the field. The teams that get hurt here are rarely the ones who chose wrong at the start. They are the ones who never noticed the question had changed, kept the research configuration all the way to the factory, and discovered at the worst possible moment that nobody could rebuild it from a blank machine. The decision worth making deliberately is not which stack is better. It is when to re-ask.

What does research middleware actually mean in practice?

Research middleware means a stack chosen for how much of the robot already exists in it. In a lab the binding constraint is calendar time against a question nobody has answered yet, so the stack that supplies the lidar driver, the mapping code, the arm planner and a simulator that speaks the same message types is the correct one, almost regardless of any other property. Upkeep is affordable there for a specific reason: someone is always present. A researcher launches the robot, watches it, and restarts it when it misbehaves. The environment lives on one workstation that one person maintains, and the consequence of a bad afternoon is a bad afternoon. Production removes the person. The robot starts itself, runs somewhere nobody is watching, and has to end up in a safe state without help when something goes wrong. Everything that a present human quietly compensated for now needs to be handled by software, written down, or designed out. That is the whole difference, and it explains why a stack can be excellent at one job and awkward at the other without anyone having made a mistake.

What do teams try first, and why does that stop working?

Almost every team ships the lab configuration first, and it works until the day it very publicly does not. The first version of production is usually the research machine in a different box: the same distribution, the same launch scripts, the same handful of manual steps compressed into a wrapper. That gets a pilot unit into a customer's building, which is genuinely the right call. The cracks appear in a predictable order. A power cut restarts everything at once, and components that a human used to launch in sequence now race each other. A network the team does not control drops packets, and parts of the robot stop hearing each other while each one individually looks healthy. Something crashes mid-motion and the arm keeps its last command instead of stopping. A field unit needs a fix, and the update mechanism turns out to be an engineer with a laptop and a plane ticket. None of these are message-passing failures in the narrow sense. All of them are consequences of a stack assembled on the assumption that a knowledgeable person is standing next to the robot.

What are the actual options once production is real?

There are five families, and only three are serious for most shipped robots. First, ROS 2 as a full distribution, carried straight through from research: the message layer, the build system, the catalogue and the tooling as one decision, with vendors who will support a deployment commercially. Second, ROS 2 trimmed hard for the product, where the team keeps the few packages that earn their place and drops the rest to shrink what has to be maintained. Third, a single-machine real-time middleware such as HORUS, an open-source Apache-2.0 project in which Rust, Python and C++ share the same shared-memory ring buffers, so messages are not serialised between processes on one computer; the fit is a product whose sensing, control and actuation all live on one on-board board. Fourth, a general message library of the ZeroMQ or LCM sort, where the team defines every message contract and supervision policy by hand. Fifth, the platform vendor's own SDK, if the robot was bought rather than built. Alongside those, a broker such as MQTT usually appears for fleet telemetry, and firmware handles anything the robot must do without asking the computer.

Which option fits research, and which fits shipping?

The right option is the one whose assumptions match the stage you are actually at, and stages are allowed to differ.

Two rows at once is normal and healthy: one stack on the robot, a broker carrying status out to people.

What should a research group that will never ship a product pick?

Take the ecosystem, take all of it, and stop thinking about this. If the output of the work is a paper, a demonstration or a grant renewal, then the value of a middleware is entirely how much of the robot you did not have to build, and one ecosystem is far ahead on that measure. Upkeep costs that would sink a product team are absorbed in a lab because someone is always in the room and the unit of failure is one afternoon. The trap for research groups is worth naming: the prototype that was never meant to ship sometimes ships anyway, because a company spins out or a partner likes the demo. When that happens the lab configuration goes into a product with none of the properties a product needs, and the team spends its first year rediscovering that. The cheap insurance is not a different stack. It is writing the environment down as a script from a blank machine while the knowledge is still fresh, and keeping the parts that touch motors separable from the parts that came from a catalogue.

What changes when the robot has to run on its own on-board computer?

Everything gets tighter, and choices that were free on a workstation start to cost. A development machine has spare cores, spare memory and a person watching. The computer that ships inside the robot has a fixed budget, a thermal ceiling, and no one nearby. Two things follow. The first is that anything the stack does in the background, and every layer of copying between components, is now competing with the control loop for the same limited machine, which is why robots that felt smooth in the lab develop a stutter on the real hardware. The second is that the shape of the robot often collapses onto a single board: the camera, the planner and the motor commands all run on one computer, and a middleware built to distribute work across a network carries machinery that this robot will never use. Neither observation makes a networked framework wrong. Both change the ranking. If the on-board picture is one computer with several languages on it, that is precisely the case where running everything on one machine simplifies the stack rather than constraining it.

What if production is a year and a half away?

Then change nothing yet, and start collecting evidence instead. A distant production date is the most dangerous moment to switch foundations, because nothing forces the decision and everyone has an opinion, so the team can burn a quarter on evaluation and land exactly where it started. What actually helps is much smaller. Write the environment down as a script that builds from a blank machine, and run that script every week so it cannot rot. Make a list of the components you did not write, and mark which ones the product truly needs — that list is the entire argument, and most teams have never written it down. Then instrument the prototype so you know which parts are near their limits when the robot is doing real work, not demo work. By the time production is close, those three artefacts decide the question in an afternoon, without a debate. The teams who do this find one of two things: the ecosystem parts are load-bearing, so they stay; or the list is short, and the stack that once earned its place no longer does.

What if the team is strong in Python and thin on C++?

Then plan for a stack that lets the two languages live together, because that is the shape your product will take whether or not you plan it. The common path is not a rewrite. It is a robot that stays mostly Python — perception glue, decision logic, the parts that change weekly — while one or two components move to a compiled language because they must run on time, every time, and a garbage-collected language will not promise that. What matters then is the seam. If crossing it means copying a message into a different representation and back, you have added work to the exact path that needed to be quick, and you maintain a translation layer forever. If the languages read the same buffers directly, the seam costs almost nothing and the team keeps writing Python everywhere it is fine to write Python. This is the practical reason to care about language interoperability before you need it, and it is treated in more depth in whether to mix languages in a robot project.

What do you actually give up by moving off the research stack?

You give up the catalogue, the shared vocabulary and the hiring pool, and none of those are small. The catalogue is the obvious loss: drivers, planners, mapping, and a simulator that already speaks the same message types. If your product depends on any of them, moving means rebuilding them, and that is a multi-year project disguised as a migration. The vocabulary loss is quieter and lasts longer. When a stack is standard, a new engineer arrives already knowing the concepts, a contractor can be dropped into the codebase, and a search for an error message returns someone who solved it. Off the standard path, every one of those becomes your team's job. There is a third cost people underestimate: introspection and recording. Being able to watch messages while the robot moves, and replay a bad run at your desk, is a tool teams only miss once it is gone. Weigh those against what you gain, which is a smaller surface, fewer things to keep configured, and a stack a small team can hold entirely in its head. Both sides of that trade are real.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the product's value comes from parts other people already built, and that covers a great many shipped robots. A machine that maps a building and navigates it should use ROS 2, because rewriting that stack is a multi-year project and it exists today. So should any robot whose graph genuinely spans several computers by design, since HORUS is a single-machine middleware and shared memory stops at the edge of one board. So should any product where hiring matters, because the pool who already know the standard stack dwarfs any alternative. So should any team that needs a commercial support contract with a named vendor, or a simulator that speaks the same message types as the real robot, or a lidar whose manufacturer ships one driver and it is a ROS 2 driver. And so should anything whose control loops run where messaging cost is simply not the problem, which is most of them. In all those cases the ecosystem decides, and choosing a smaller stack buys a property you did not need at the price of a year.

Is production middleware just research middleware with more testing?

No, and here is why: testing verifies behaviour you already have, while the gap between research and production is mostly behaviour nobody wrote. A lab stack assumes a person: someone starts the components in a sensible order, notices when one dies, restarts the robot when it wedges, and refrains from unplugging things mid-motion. None of those assumptions survive a customer site, and no amount of testing the existing code adds the missing parts. What production needs is new behaviour: components that come back up in any order after a power cut, a defined answer to what the arm does when the process commanding it disappears, an update path that does not need an engineer in the building, and a record good enough to diagnose a failure you did not witness. Testing tells you whether those work. Testing does not supply them. This is also why "we will harden it later" tends to fail — hardening is not a phase applied to finished software, it is a set of decisions about crash boundaries and start-up order, cheap early and expensive once the shape is set.

Does this mean we have to rewrite the robot before we ship?

Partly, but not the way you think. Something usually does get rewritten, and it is almost never the whole robot. The parts that change are the ones with a timing or restart requirement the current arrangement cannot meet — typically the loop that commands motors, the path from a sensor into that loop, and whatever must keep working when a neighbouring process dies. That might be a handful of components. Everything else stays, because it works and the team understands it. The failure mode is treating this as a stack decision rather than a component decision: a team decides the middleware is wrong, commits to a full migration on a deadline, and spends a year replacing known problems with unknown ones. The better move is to draw the boundary around the small set of things that must be predictable, choose deliberately for that set, and leave the rest alone. Teams that have done this describe it less as a migration and more as a hardening of one part of the robot, which is also the pattern behind how teams end up switching middleware.

How do we decide which stack ships with the product?

Answer three questions in writing, and the choice makes itself. First: which components did we not write, and does the product still need them? If that list is long and specific, the ecosystem is load-bearing and it ships with you, whatever else is true. Second: how many computers run our code in the shipped robot, and how many languages? One computer with several languages is the case where a shared-memory layer earns its place; several machines by design is the case where a networked framework does. Third: who maintains this a year after launch, and can that person rebuild it from nothing? If the answer is a script they can read, you are fine. If the answer is a machine that must never be reinstalled, fix that before choosing anything else, because it is the real risk and it is not a middleware problem. Weight the first question hardest. A catalogue takes years to replace, and no property of a transport is worth that. If you want the fleet version of this question, multi-robot projects rank the options differently.

A short version, by situation:

• If you are a lab whose output is a paper -> ROS 2, because the parts you did not write are the entire reason the work fits in a quarter.
• If you are shipping a robot that grew out of that lab -> ROS 2 trimmed to what the product uses, because you keep the load-bearing parts and stop maintaining the rest.
• If your product is one on-board computer running several languages -> a single-machine real-time layer, because the network machinery is cost with no return.
• If your product depends on mapping, navigation or a vendor's driver -> ROS 2, because rebuilding those is a year you will not get back.
• If nobody can rebuild your robot from a blank machine -> fix that first, because no middleware choice survives an environment only one laptop has.

When you want to compare options rather than symptoms, the HORUS Fit Framework lines them up on five things that are not numbers: ecosystem size, setup effort, team size fit, deployment target, and licence. Research weights ecosystem size above everything; production weights deployment target and licence far higher, and setup effort quietly decides whether a stack survives its first year in the field.

HORUS is open source under Apache-2.0 and the repository is linked below. Star it so it is in your list when you start building.