Managing Robot Software Across Many Deployed Units

Managing many deployed robots is an update problem, so the choice that matters is your delivery method, image, container or package, not your framework. ROS 2 and HORUS both ship as ordinary software you package yourself, so neither one solves fleet updates for you. The verdict flips only if your robots run a vendor's platform, because then the vendor's update service is already the answer. The rest of this post is for a team with robots at more than one customer site and a growing suspicion that nobody knows exactly what is running where.

It starts as a good problem. The robot works, somebody bought one, then somebody else bought two. Now there are machines in buildings you have never visited, and the person who commissioned each one did it slightly differently, because each site was slightly different.

Then the support calls begin, and they all have the same shape. A customer says the robot has started doing something odd. You ask which version it is running. Nobody knows. Someone half remembers logging in during a visit and changing a file so a demonstration would work, and never changing it back. The logs are on the machine, the machine sits behind a firewall on a customer's network, and reaching it means booking a trip.

Meanwhile a fix is ready and nobody wants to send it, because sending it means touching every unit at once and there is no way back if the fix is wrong. So the fix waits. The fleet drifts further apart. And the newest robot leaving your workshop is already different from the oldest one in the field, in ways nobody has written down anywhere.

Should you treat deployed robots as one fleet or as individual machines?

Treat them as one fleet: one build that every unit runs, with everything that differs between machines held as data the build reads.

The instinct runs the other way, because every site genuinely is different. One customer has a lift, another has a loading bay with a slope, a third asked for the robot to wait longer at doors. Handling each of those in the code that ships to that customer feels like service. It ends with as many codebases as customers, and a support conversation that begins by working out which one you are looking at.

The rule that holds is simple to state and uncomfortable to follow: any difference between machines must be expressible as configuration, and that configuration must be readable from your desk. If a difference cannot be reduced to data, it is a hardware variant, and hardware variants need names and an inventory rather than a quiet fork.

One exception is real. Calibration belongs to the individual machine, was measured on that machine, and must survive every update untouched. Build is fleet-wide, configuration is per site, calibration is per unit.

What does managing software across a fleet actually cover?

It covers four jobs: knowing what is running where, getting new software onto machines you cannot touch, getting information back off them, and undoing an update without a site visit.

Knowing what is running where is the inventory. Every unit stamps a build identity that traces to an exact source revision, and reports that identity whenever it has a connection. Without this, every other job is guesswork.

Getting software onto machines is delivery, and on customer networks it must be the robot fetching rather than you pushing, because nobody is going to open an inbound route into their building for you.

Getting information back is the return path: logs, health, and enough recorded detail to turn a complaint into an investigation. This is the job teams postpone and then regret, because a complaint about last Tuesday cannot be investigated with data that was never kept.

Undoing an update is rollback, and it is the one people skip entirely until the morning they cannot.

What none of this covers is whether the robot works. Fleet management makes a working robot supportable, and does nothing for one that is not.

What are your actual options for getting software onto deployed robots?

There are six delivery methods people use, and choosing between them is mostly about what happens when an update fails.

You can log in and change things by hand, which works while machines are nearby. You can push with a configuration management tool, which assumes machines are online when you push. You can replace the whole system image and keep the previous one on a second partition, the approach built for machines nobody can attend. You can pull application containers from a registry, suiting teams whose application changes far more often than the system beneath it. You can build and host distribution packages, gaining ordinary tools and partial updates at the cost of owning packaging. Or you can buy a fleet service and inherit someone else's plumbing.

The foundation is a separate decision. ROS 2 gives a parameter system for per-unit settings, and recording so a field failure can be replayed. HORUS is an open-source real-time middleware for Rust, Python and C++ where all three languages share the same shared-memory ring buffers, so messages between processes on one machine are not serialised; the project is Apache-2.0 and validated in simulation, and covers what happens inside one robot rather than how software reaches it.

How do those options compare side by side?

Read the last column first, because the delivery methods rule themselves out faster than they rule themselves in.

The last two rows are not competing with the first six. A fleet needs a delivery method whatever foundation the robot is built on.

Who on the team should own the deployed fleet?

One named person, and not the person shipping features.

The fleet owner's real job is to say no. No, that change does not go out this week. No, we are not editing that machine live to make the demonstration work. No, this build does not reach every unit until it has run on one for long enough to be boring. Somebody optimising for shipping features cannot hold that line against themselves, and asking them to is unfair rather than merely ineffective.

The deliverables are small and concrete. An inventory that says what build and what configuration each unit carries. A written rollout rule that says how a build reaches the whole fleet and who signs off at each stage. A rollback procedure that somebody other than its author has actually performed.

On a team of two, this is still one of you, wearing a different hat on a specific afternoon each week. The failure mode for small teams is not lacking the skill. It is that fleet work never wins against a feature until the week it wins against everything.

What hardware makes updating a deployed robot harder?

Storage, power behaviour, and whether the machine has a second way to start.

A robot that loses power halfway through writing a new system image and has only one place to write it is a machine somebody must drive to. Two partitions, with the previous version intact and a bootloader that falls back when the new one fails to come up, turns that trip into an inconvenient log line. Memory cards wear out and fail in ways that look like software bugs, which costs teams weeks of chasing ghosts.

The second hardware question is variants. The same software often has to run across units with different motor drivers, different cameras, or a sensor that was replaced when the original went out of production. Those differences must be detected from the machine itself, by reading an identifier from the hardware, rather than from a setting a person typed during commissioning. Anything a person types during commissioning will eventually be typed wrongly, and the wrong value will not announce itself until the robot is moving.

How long does it take to set up fleet updates properly?

A short project while you have a handful of units, and a long one once you have many and cannot take them out of service.

The cheap parts come first and stay cheap: stamping a build identity into every image, having each unit report that identity home, and shipping logs off the machine when a connection allows. A small team can do those in the time it takes to argue about them.

The expensive parts are rollback and staged rollout, and they are expensive because they touch how the machine boots and how you organise releases. They are also exactly the parts you need before the first bad update rather than after it, which is the timing trap. Nobody feels urgency about rollback until the morning they need it, and that morning is the worst possible time to build it.

The honest sequencing rule is to build the ability to fix a machine remotely before you need to fix a machine remotely, because after that point every hour of work happens with a customer waiting.

What skill level does running a fleet of robots need?

Ordinary Linux and backend skills, plus a temperament, and the temperament is the scarce half.

Nothing in fleet management is robotics-specific. Building images, hosting artefacts, having a device fetch and verify an update, collecting logs centrally: this is work that web and infrastructure people have been doing for years, and someone coming into robotics from that background will find it the most familiar part of the whole product.

What robotics adds is consequence. A web deployment that goes wrong shows an error page. A robot deployment that goes wrong leaves a machine that will not start, in a building you need permission to enter, in front of a customer who paid for it.

The temperament is the discipline to refuse hand edits on live machines, to insist on staged rollouts when everyone wants the fix out today, and to keep the inventory accurate when nothing bad has happened for months. That discipline is rarer than the skills, and it is the thing worth hiring or assigning for.

What changes as the fleet grows past what you can remember?

Failures stop being stories and become statistics, and the question changes from "why did this robot do that" to "is this machine or is this the build".

With a few units you carry the fleet in your head. You know which one has the older camera and which one sits on a floor that confuses the wheels. Past a certain point that memory fails silently, and you start answering support calls with confident recollections that belong to a different machine.

The rare bug arrives on the same schedule as before, but you now have many machines, so what used to be a curiosity every few months becomes a weekly ticket. An update that fails on a small share of units used to be one phone call and is now a queue.

Answering the machine-or-build question needs the same data shape from every unit, which only happens if the units really are the same. That is the point where drift stops being untidy and starts being the reason you cannot diagnose anything. It also matters when a program dies on a unit overnight and the fleet is the only thing that will tell you.

What do you give up by managing robots as a fleet?

Speed on any single machine, and the freedom to make one customer happy this afternoon.

Every fix now travels through the same path: build, stage, roll out. A customer who wants a small change waits for that path rather than for whoever happens to be on site with a laptop. Some of them will notice, and a few will say so, and holding the line is genuinely uncomfortable when the change is small and the customer is important.

You also take on a second product. The update system has its own bugs, its own outages, and its own maintenance, and none of that work is visible to anyone buying robots. Log collection costs storage and bandwidth that grow with every unit sold.

And you accept a fleet that is never uniform. Staged rollouts mean two versions in the field at once, deliberately, which means support must always ask which one before answering. That is the correct trade, and it is still a cost that lands on the people answering the phone.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever field failures have to be reconstructed rather than merely reported, and whenever people outside your team need to read your robot's configuration.

Recording is the strongest reason. Capturing every message from a run and replaying it into the same code on a desk is how an intermittent fault at a customer site becomes a fixable bug instead of an argument. When the whole job is supporting machines you cannot watch, that capability outweighs almost everything else, and HORUS is not the answer to that job.

The parameter system is the second reason. Per-unit and per-site settings need a mechanism with existing tooling, a documented shape, and the ability to be dumped and compared across units. Building an equivalent yourself is a month you did not plan.

ROS 2 also wins when the robot spans several computers, when the drivers come from sensor vendors, and when a customer's own engineers must be able to look inside. Those are ordinary conditions for a deployed product, not edge cases, and choosing against them costs more than it saves.

Can you just log into each robot when something goes wrong?

No, and here is why: hand fixes work until the moment they become the reason nothing can be diagnosed.

The first failure is access. Machines sit on customer networks that grant no inbound route, and the ones you most need to reach are the ones at the difficult sites. The second is state. After a hand fix, nobody can say what that machine now contains, so every later question about it starts from unknown ground.

The third is the person. Logging in requires somebody who knows the exact sequence, and that somebody becomes a single point of failure who cannot take a holiday. The fourth is safety. A hand fix on a live machine is an untested change running near people, decided under pressure, by someone who wants the call to end.

The pattern that works is to make remote access a diagnostic tool rather than a repair tool. Log in to read. Change nothing that the delivery path could deliver instead, and when you break that rule, record it in the inventory the same hour.

Does putting everything in containers solve fleet updates?

Partly, but not the way you think: containers make the application arrive the same way every time, which is real, and leave the harder half untouched.

The half they solve is genuine. Dependencies stop drifting, a build that works on the bench works on the unit, and rolling an application back means running the previous image. For teams whose trouble is that machines have accumulated different library versions over years, that alone is worth the move.

The half they leave is where robot pain lives. The kernel underneath, the device permissions, the scheduling settings, the graphics drivers: those sit outside the container and still need their own update story. A half-finished pull on a machine that lost power still needs recovering. The network to fetch an image still has to exist. Per-unit calibration still has to live somewhere that updates cannot overwrite.

Most of all, containers do not tell you what is running where. That is inventory, and inventory is a thing you build regardless of how the software is packaged.

How do you decide what to build first?

Try to answer, from your desk and without phoning anyone, what build is running on the oldest unit in the field.

Whatever you had to do to answer that is your first project, and if you could not answer at all, the first project is the inventory. Nothing else in fleet management works without a dependable answer to that question, and every team that skips it rebuilds it later under worse conditions.

After the inventory, the usual order is: get logs off the machines, because the next support call needs them; then rollback, because it is the insurance you cannot buy after the accident; then staged rollout, because it is what turns a bad build into a small incident; then configuration as data, because it is what stops the fleet drifting apart again.

One more decision belongs early, alongside those: agreeing what the team standardises now rather than later. Fleets punish teams who leave that agreement until the second customer.

Here is the whole decision in six lines.

• If you cannot say what build is on your oldest deployed unit -> the inventory first, because nothing else works without it.
• If a bad update would need a site visit to undo -> whole-image updates with two partitions, because rollback is the insurance.
• If units sit on customer networks you cannot reach -> a pull-based update the robot fetches, because inbound access is never granted.
• If explaining a field complaint means guessing -> ROS 2, because recorded runs turn arguments into investigations.
• If the machine stutters under load before updates are even the issue -> a shared-memory middleware, because that problem lives on one computer.
• If you have very few units and all are nearby -> keep working by hand, but write the inventory down today.

The HORUS Fit Framework reduces the foundation half to five axes you can score any candidate against: ecosystem size, setup effort, team size fit, deployment target, and licence. A fleet shifts weight hard towards deployment target and licence, because both become other people's problems the moment a robot leaves your building.

When the on-machine half becomes the thing slowing you down, you want the option already on your shelf rather than researched under pressure. Put HORUS on that shelf now: star it so it is in your list when you start building.