Best Ways to Update Software on a Deployed Robot

The best way to update software on a deployed robot is a whole-system image swap that can roll itself back, not a patch over SSH. Field machines fail in ways a patch cannot describe, so ship the whole system and make the fallback automatic; what runs inside, whether ROS 2 nodes or HORUS processes, changes packaging rather than the plan. That flips for a single hobby robot on your desk. The rest of this post is for a team with machines in other people's buildings, an update they are afraid to send, and no agreed way to send it.

There are six machines now and you are fairly sure two of them are running something slightly different. Somebody fixed a problem on site last month by editing a file directly, and nobody wrote it down. The build that works lives on a laptop belonging to an engineer who is between contracts. When a customer reports a fault, the first twenty minutes of the call go on establishing what is actually installed, and the answer usually surprises everyone.

Then there is the update you have been sitting on for three weeks. It is a good fix, it works on the bench, and nobody wants to be the person who sends it. The last time an update went out, one machine came up without its network service and someone had to drive there with a keyboard. Since then the process has been to update only when a person is standing next to the robot, which works only for the machines you can reach on a Tuesday. Meanwhile the fleet keeps growing and the gap between what you tested and what is actually running keeps widening.

What is the best way to update software on a robot already in the field?

Build one image of the whole system, install it beside the running one, and let the machine return to the previous image on its own if the new one does not come up working. Everything else on the list is a variation on that idea or a compromise with it. The reason is not elegance. It is that a deployed robot is a machine you cannot look at, in a building you cannot enter, on a link that drops in the middle of large downloads, and the failure that ruins a week is not a bad fix but a half-applied one. A patch that lands on four machines and half-lands on the fifth leaves you with a fleet you can no longer describe. An image swap has exactly two outcomes, old or new, and both are states you have tested. The cost is real: images are large, builds get slower, and you need room for two copies. Teams who pay that cost stop dreading updates, which changes how often they send them, and that shift is where most of the benefit actually comes from.

What does updating a deployed robot actually involve?

Updating a deployed robot means changing software on a machine that is not on your desk, without ever leaving it in a state nobody has tested. The part everyone thinks about, moving new bytes to the robot, is the easy half. The rest is deciding when it is safe to apply anything, because a robot mid-task with a load in its gripper is not a laptop and cannot simply restart. It is checking that the new software came up and is doing its job, not merely that a process exists. It is preserving the calibration, serial number and site settings that belong to that machine and to no other. It is recording what is now installed somewhere more trustworthy than a spreadsheet. And it is getting back to the last known-good state without a person present, because the machines that break are always the ones furthest away. A robot update is therefore a small distributed systems problem wearing overalls. Teams who treat it as file copying discover the difference during their first bad night, usually on the machine belonging to their most important customer.

What are the actual update methods you can choose between?

There are seven approaches in common use and most teams end up running two of them, one for software and one for settings. You can log in and change files directly, which is where every project starts. You can publish packages to your own repository and let machines install them, which is how a ROS 2 workspace usually reaches a robot. You can ship container images and swap them. You can build a whole-root image with two slots and automatic fallback, the way phones have done it for years. You can buy a hosted service that manages downloads and staged rollouts. You can separate settings from software, so tuning a threshold is never a code release. Or you can split the machine into units that move on different rhythms: perception packages on one path, and on another a small group of processes built on HORUS, an open-source real-time robotics middleware for Rust, Python and C++ in which all three languages share the same shared-memory ring buffers, so messages between processes on one computer are never serialised; the licence is Apache-2.0, the project is validated in simulation rather than proven across large fleets, and it is not a full ROS replacement, which is exactly why the split arrangement is more common than a swap.

How do the update methods compare?

Read the table by crossing out every row whose last column describes your situation, then choose among what survives by asking which one you could carry out at midnight with the person who built it unavailable.

Two surviving rows is the normal outcome rather than a failure of the table. Most teams with field machines keep the whole-root image row and the settings row together, because those two answer different questions. The row people over-rate is the hosted service, which is useful and does nothing about the states your robot can be left in. The row people under-rate is settings, because splitting configuration out is cheap, reversible, and removes a share of the updates you were nervous about sending.

What happens when an update goes wrong on a machine you cannot reach?

The machine either comes back changed in a way nobody intended or does not come back at all, and the second one costs a day. The patterns are familiar once you have seen them. A download is interrupted and half the files are new. A service that used to start now waits for something that is not there yet, so the robot boots, sits still, and never joins the network, which means you cannot even see that it booted. A configuration file is replaced by the version from the build machine, so the arm's calibration is gone and it reaches for a shelf that has moved. A library arrived that the bench never had, and the robot works for six hours and then stops in front of a customer. Every one of these is survivable if the machine can decide by itself that the new software is unhealthy and go back. None are survivable if the recovery plan is a person, a keyboard and a two hour drive. Design for the case where nobody is there, because that is the only case that actually happens.

Are you a solo builder, a small team, or a company with machines at customer sites?

The three should stop at different points, and copying a larger company's process is the usual waste. A solo builder with one robot on a bench should log in and change files, with no guilt about it, because the recovery plan is standing in the room. A small team with a handful of machines they can physically reach needs one thing above all else: an honest record of what is running where, and a build that anyone on the team can produce without the original author. That is worth more than any clever delivery mechanism, and it is usually the missing piece rather than the tooling. A company with machines in other people's buildings has crossed a line that cannot be uncrossed, because the recovery plan can no longer include a person, so automatic fallback stops being good practice and becomes the thing that keeps you solvent. The failure shared by all three groups is adopting the next group's process a year early and never finishing it, which leaves you with neither the simple thing nor the complete one.

What hardware and network are you updating over?

The hardware decides which methods are even available, and it is usually settled long before anyone thinks about updates. Two-slot images need room for two copies of the system, and a board chosen on price often does not have it, which turns a software decision into a bill of materials conversation months too late. Cheap flash storage wears out, and a method that rewrites the whole system every week ages that storage faster than one writing a few files, which matters on machines expected to run for years. Some boards cannot be told to boot from a different slot without touching the bootloader, and some bootloaders belong to the vendor rather than to you. On the network side, a robot that only sees wifi while docked has a download window shaped by when it charges, and a machine on a metered link makes image size a running cost rather than an inconvenience. Sites with no route to the internet rule out hosted services completely, and there are more of those than vendors like to admit. Check all of this before promising a mechanism, because the answers often remove half the options.

What if you need a fix on the machines this week?

Send the fix using whatever you already have, then spend the following month building the mechanism you wish you had. Urgency is the worst possible time to adopt a new delivery method, because its first outing will fail in a way nobody on the team has learned to recognise yet, and it will fail on the machines that already have an unhappy customer attached. So do the unglamorous thing: make the smallest change that solves the problem, test it on a machine matching the field build rather than your bench, write down exactly which robots received it, and put a person on the phone with each site if that is what it takes. Then treat the incident as the reason to fund the real work. Every team running a decent update process today got there after a bad week, and the ones who never had a bad week are usually the ones still updating by hand. The trap is declaring victory once the fire is out, because everything you did in a hurry becomes the drift that causes the next one.

What does each update method demand from your team's skills?

Each method moves the difficulty somewhere different, and the sensible move is putting it where your team is already strong. Logging in and editing demands nothing and produces machines nobody can describe. Packaging demands somebody who genuinely understands dependencies and versions, and that person is rarer than teams assume, because a half-understood packaging setup produces installs that report success and leave the wrong version behind. Containers demand image and registry habits many teams already have from other work, plus a clear head about which parts of a robot cannot live inside a container, which is usually anything touching hardware. Whole-root images demand a build that runs without a human and produces the same result twice, and that is the real bill, because most teams discover their build only works on one laptop. Hosted services demand less engineering and more diligence about what leaves your machines. Notice that only one of these demands anything about the robot itself. Update skill is mostly build skill, which is why what a team should standardise early starts with the build rather than the fleet.

What do you give up by moving to whole-image updates?

You give up speed on the bench, and for some teams that alone decides it. A one line change stops being an edit and a restart, and becomes a build, an image, a transfer and a reboot; if the build is slow enough, people start avoiding it, which is worse than the problem you set out to solve. You give up small downloads, which hurts on metered or slow links unless you invest in shipping only the differences. You give up per-component independence, so a fix in one corner re-tests everything, and your test suite becomes the gate on every change you make. You give up storage the hardware may not have. And you take on a discipline nobody enjoys, which is knowing exactly what is inside the image, because anything a person installed by hand on a machine disappears at the next update. Against all that you get a fleet you can describe in one sentence, recovery that needs no person, and the ability to reproduce a customer's exact software months later. Most teams find that trade obvious in hindsight and painful in the quarter they make it.

When is ROS 2 the better choice?

ROS 2 is the better choice for most robots that need updating in the field, and the cases deserve stating plainly. If the robot's value is perception, mapping or navigation, the packages are the product and no update mechanism compensates for rewriting them. If the machine is one of many reporting to a server, the ecosystem grew up in that world. If a sensor you already bought has one maintained driver and that driver is a package, the decision was made at purchase. If field support means finding a stranger who hit your exact error first, a large community is worth more than any property of the layer underneath. If you need to hire somebody next month who can debug a deployed machine, the pool that already speaks the vocabulary is much larger. HORUS is not the answer in any of those situations, and choosing a narrower foundation there leaves your team owning plumbing a community would otherwise maintain. Updating is a build and delivery problem before it is a middleware problem, and a team on ROS 2 with a two-slot image and automatic fallback is in better shape than a team on anything else without one.

Will containers solve your update problem?

No, and here is why: containers change how your application is packaged and leave every hard part untouched. What hurts is deciding when a moving machine may safely restart, keeping per-machine calibration through a change, noticing that new software came up unhealthy, and getting back without a person. A container does none of that. It also puts a boundary in exactly the wrong place for robots, because code talking to a camera, a motor controller or a serial device needs the device, the driver and often a kernel module, and the moment you start passing hardware through the boundary you have recreated the coupling you were removing plus an extra layer to debug at midnight. Containers do help with one real thing, which is stopping your application's libraries from fighting the operating system's, and that matters for a stack mixing several languages. Teams adopting them for that reason are usually pleased. Teams adopting them expecting field updates to become manageable end up with a new build system, the same bad night, and logs one level further away.

Is an over-the-air update just copying files to the robot?

Partly, but not the way you think. The transfer is genuine work, and doing it over a link that drops means resuming, verifying that what arrived is what you sent, and applying nothing until the whole thing is present and checked. But transfer is the part with existing solutions, and it is not where teams lose days. Days go into the decisions around it: whether the robot is in a state where a restart is safe, what happens when a machine is switched off mid-write, which settings survive, how the machine proves to itself that the new software works before committing to it, and how a support engineer learns what is actually installed during a call. Those are policies about your robot, not features of a delivery tool, and no product decides them for you. The useful test is writing down what your machine does if power is cut at the worst moment of an update. If the answer is a shrug, delivery is not your gap. If the answer is that it comes back on the old system, you already have the part that matters.

How do you decide which update method to commit to?

Decide by naming what you cannot do today and letting that gap pick the method, because each gap points somewhere different. If you cannot say which software is on which machine, the fix is a version string the robot reports itself, and no delivery mechanism helps until that exists. If you cannot rebuild last month's release, the fix is the build, not the update. If a partly applied change would be dangerous, you need an atomic swap. If tuning on site means a code release, split settings out and much of your urgency disappears. If recovery currently means a person driving somewhere, automatic fallback is the only item worth funding this quarter. Work in that order, because the later items are worthless without the earlier ones, and teams routinely buy a fleet service while still unable to reproduce a build. Whoever joins next inherits whatever you settle on, so it is worth making the process legible to a new engineer while you still remember why it exists.

Decide by situation rather than by tooling preference:

• If you are one person with one robot on a bench -> edit over SSH, because the recovery plan is standing in the room.
• If your team can physically reach every machine -> packages plus a reported version string, because knowing what is installed beats delivering faster.
• If machines sit in other people's buildings -> a two-slot image with automatic fallback, because recovery cannot depend on a person arriving.
• If your robots are tuned differently at every site -> ship settings apart from code, because a threshold change should never be a release.
• If motion code and perception change on different rhythms -> split them into separate units, because re-testing everything for a small fix eventually stops you shipping at all.

When two options survive, weigh them on the five axes of the HORUS Fit Framework: ecosystem size, setup effort, team size fit, deployment target, and licence. No scores, just five questions about your team rather than about the tooling, and you keep whichever loses on fewer. If your machines keep landing on one computer, more than one language, and a control loop that must hold its rhythm while an update lands beside it, star HORUS on GitHub so it is in your list when you start building. First, though, be honest about which parts of your prototype will not survive production.