Best Tools for Monitoring a Robot in the Field

For most teams the best field monitoring setup is health check-ins, shipped logs and saved event recordings, with a paid fleet platform added later. Those three answer the two questions a support call asks: is the machine alive, and what was it doing when it stopped. That flips when the failures are timing rather than logic, which is a question about ROS 2, HORUS and the layer under both. The rest of this post is for a small team with robots already out in the world and no good way to see what they are doing.

A robot you can walk up to is a robot you can fix. The ones out in the field are not that. You get a phone call instead, from somebody standing next to the machine, telling you it stopped, and the only question you can usefully ask them is what the lights are doing.

By the time you get a connection to the machine, if you get one at all, the state you wanted is gone. The program restarted itself and took its memory with it. There is no terminal history because nothing was attached to a terminal. Somebody power-cycled the robot before calling you, because that is what people do, and now you are looking at a healthy machine and an unhappy customer.

So you guess. You read code hunting for something that could produce the behaviour described over the phone, a description given by a person who was not watching closely and had no reason to be. Meanwhile the second robot at the other site did something last week that might be the same fault or might be nothing, and you have no way to tell the two apart.

What are the best tools for monitoring a robot in the field?

Health check-ins, shipped logs and event recordings, in that order, plus a fleet platform once you have more machines than one person can hold in their head. Health check-ins are the cheapest thing you will ever build and answer the most common question, which is whether a machine is alive and what mode it thinks it is in. Shipped logs turn a phone call into a search, provided every line carries a timestamp and a machine name so lines from different robots can be sorted into one story. Event recordings are the part teams add last and wish they had added first: a rolling buffer of recent messages that gets written to disk when something goes wrong, so the moments before a fault survive the restart that erases everything else. A fleet platform bundles those three with a map, a device list and alerting, and is worth paying for at the point where keeping track by hand starts costing more than the subscription.

What does monitoring a robot in the field actually mean?

Field monitoring means arranging in advance for the robot to tell you things you will want to know only after it has already gone wrong. That framing matters because it explains why monitoring is never urgent until the day it is far too late. Everything in this category is a bet placed before the fault about which records will turn out to be worth having. On a robot the bet is harder than on a server, because the machine moves, loses its connection, runs on a battery, and gets switched off by people who are not on your team. A server that goes quiet is a problem. A robot that goes quiet may have finished its shift and been unplugged. The category therefore splits into two jobs that people constantly confuse: knowing the state of a fleet right now, and being able to reconstruct one machine's past. Dashboards do the first. Recordings and logs do the second. Teams who buy only the first end up with a beautiful wall display and no answer to why the machine at the second site keeps stopping.

What are the actual options for monitoring a deployed robot?

There are seven families, and a working deployment usually runs four of them. Health check-ins are a small regular message saying the machine is alive and what state it is in. Log shipping copies the machine's own text records somewhere searchable. Metrics and dashboards track a handful of numbers over time, which is the right shape for temperature, storage space and battery level. Event recordings save the recent past when a trigger fires. Remote access, usually a private network overlay, lets you reach into one machine directly. Crash reporting collects the wreckage of programs that died. Commercial fleet platforms package several of those with a device list and alerting. Underneath all of them sits the messaging layer your programs already use to talk to each other, whether that is ROS 2 across a network or a single-machine middleware such as HORUS, because a monitor can only ever record what that layer carried. Choosing among the first seven is about what you want to know. Looking at the eighth happens when the records keep failing to explain the fault.

How do the field monitoring options compare?

The rows run roughly in the order teams adopt them, which is also the order of increasing cost.

Read the last two columns first. Most monitoring money is spent on a family that was never going to answer the question being asked.

Which tools fit a small team with robots already deployed?

Health check-ins and shipped logs, built yourself, in about the time it takes to argue about which platform to buy. A team of a few people supporting a handful of machines is not limited by what a dashboard can display; it is limited by how long it takes one engineer to answer one question about one machine. Both of those tools attack that directly. The check-in tells you which machine to look at. The logs let you look without travelling. Build them so that a single search across every machine is possible from the start, because the moment two robots misbehave in the same way, that search is the whole investigation. Resist the urge to invent a metric system early; a handful of numbers written into the same log stream is enough for a long time. What you should not skip is a version stamp on every record, saying exactly which software the machine was running, because half of all field mysteries turn out to be one machine on an old build. Managing robot software across many deployed units covers the coordination side of the same problem.

What changes when the robots are small boards on a patchy connection?

Storage and bandwidth become the design, and everything else follows from them. On a small board with intermittent connectivity, the correct default is to write everything locally and treat the network as a way of collecting records later rather than a live window into the machine. That single decision removes a whole class of failure, because the records around a network outage are exactly the ones a live-only setup loses, and network outages and robot faults tend to arrive together. Size the on-board storage for the longest stretch anyone thinks the machine could be offline, then increase it, because a full disk on a deployed robot produces symptoms that look like anything except a full disk. Keep recordings on a rolling buffer so old data is discarded automatically rather than by a person who forgot. Watch what monitoring itself costs the machine: streaming video off a small board changes how the board behaves, which means your monitoring can create the stutter you were sent to investigate. Best robotics middleware for Raspberry Pi and Jetson robots covers the rest of what shifts on constrained hardware.

What should you have running before the robots ship next month?

Check-ins, on-board recording and a version stamp, and nothing else, because those three are the ones you cannot add retroactively. A deployment date concentrates the mind on dashboards, which are the part you can build calmly afterwards while looking at data you already have. What you cannot do afterwards is recover the first month of faults from machines that were not recording. So spend the remaining weeks making sure every machine says it is alive, writes its recent past to disk, keeps that recording when something goes wrong, and stamps every record with which build it is running. Then make sure you can actually retrieve those files from a machine at a customer site, which is a separate problem involving access, permissions and a person on the other end, and is best discovered now rather than during your first outage. Practise the retrieval once before shipping. The team that has never pulled a recording off a real machine will discover, at the worst possible moment, that nobody knows the password. Best ways to update software on a deployed robot deals with the other half of shipping.

What skill level does field monitoring assume?

Less than people expect for the basics and considerably more for the parts that look easy. A check-in and a structured log line need only ordinary programming and a decision about format, and any competent developer can build both in a day. The step that catches teams out is operating what they built: storing records somewhere that does not fill up, keeping access working when a certificate expires, making sure alerts reach a person who is awake, and resisting the growth of alerts nobody acts on. That is systems work rather than robotics work, and a robotics team often has nobody who enjoys it. This is the honest argument for a commercial platform, and it has little to do with features. You are paying somebody else to keep the boring parts running. The skill that matters most is neither of those, though: it is deciding what to record, which is a judgement about which future questions are worth answering, and it improves only by being wrong a few times and noticing what you wished you had kept.

What does it look like when field monitoring goes wrong?

It looks like plenty of data and no answers, which is the most common failure and the least obvious. A team with broken monitoring usually has more dashboards than a team with working monitoring. The graphs are green, the check-ins arrive, and when a machine bumps a shelf nothing in the whole system says anything about it, because the records describe the health of the computer rather than the behaviour of the robot. The second failure mode is alert exhaustion: a threshold set optimistically, a channel that fires several times a day, and a team that has learned to swipe the notification away without reading it. The third is silent loss, where recordings stopped being written weeks ago because a disk filled or a service failed to restart, and nobody noticed because nothing monitors the monitoring. The test for all three is simple and worth running quarterly. Take a real fault from last month, and try to reconstruct it using only what your system captured. If you cannot, you have a wall display rather than monitoring.

What do you give up by choosing a monitoring setup?

You give up machine resources, engineering time and, with the hosted options, control over where your data lives. Every record written is work the robot is doing instead of its actual job, and every stream sent is bandwidth and battery. On a small machine this is a real trade rather than a rounding error, and the honest version of it is that a heavily monitored robot does slightly less than an unmonitored one. You also give up engineering time continuously rather than once. Monitoring is not a project that completes; it is a small permanent tax of maintaining formats, storage, access and alerts as the fleet changes. Choosing a commercial platform trades that tax for a fee and a dependency, which includes accepting the vendor's model of what a robot is, and accepting that customer data will travel through somebody else's system, which some customers will have opinions about. Building it yourself trades money for the certainty that nobody else can change it. Neither choice is wrong, but pretending either is free leads to the setup nobody maintains.

When is ROS 2 the better choice?

ROS 2 is the better choice for most deployed robots that already have sensors, a fleet, and people looking at data across a site. The tooling depth is the argument and it is not close: the recording format, the replay tools, the diagnostic conventions, the browser-based viewers that read those recordings, and the years of accumulated practice about what to put in a diagnostic message. If your robots must also talk to each other across a network, ROS 2 is designed for that and gives you discovery between machines for free. And if your team already speaks ROS 2, the monitoring you build on top will be understood by the next person you hire. HORUS is not the answer to that question, because a shared-memory layer for a single computer is about how programs on one machine hand data over, not about fleets, dashboards or recordings. The narrow case where the layer beneath becomes worth examining is one onboard computer running mixed-language code where the fault is when data arrived, not what it contained.

Can a dashboard tell you why a robot stopped?

No, and here is why. A dashboard shows the values somebody decided to display, sampled at whatever interval keeps the graph readable, which means the moment you care about is almost always between two points. A robot stopping is an event lasting less time than the gap in your graph, so the picture shows a normal machine, then a stopped machine, and nothing in between. Worse, the interesting cause is usually not on the dashboard at all: a sensor reading that was briefly implausible, a decision the planner made and immediately reversed, a message that arrived after the moment it was needed. None of those are numbers anyone thought to plot. What does answer the question is a recording that captured the messages themselves around the stop, triggered by the stop, so the sequence survives. Dashboards remain worth having for the thing they genuinely do, which is showing gradual decline: storage filling, a motor running warmer each week, a machine restarting more often than the others. What happens when a robot program crashes mid-motion covers what the stop itself does to the hardware.

Does more monitoring make a fleet easier to support?

Partly, but not the way you think. More recorded data does make individual faults easier to solve, and teams are right to record more than feels necessary at the time. What it does not do is reduce the support burden, because the constraint is almost never a shortage of data; it is the time one engineer spends deciding which machine to look at and finding the relevant part of what was captured. Doubling the recorded data with no change to search makes that worse rather than better. The setups that genuinely reduce support load do something different: they make records comparable. Every machine writing the same shaped log line, with the same names, the same timestamp format and a version stamp, means one search covers the fleet and a fault becomes a pattern rather than an anecdote. That is a formatting discipline, not a tooling purchase, and it is the highest-return thing on this page. Why does my robot behave differently every run covers the faults that only become visible once runs can be compared.

How do you decide what to monitor first?

Take the last three support calls you handled and ask what would have made each one short. That exercise produces a better list than any framework, because it is grounded in the faults your robots actually have rather than the ones the industry writes about. Usually the answer to the first call is a check-in, the answer to the second is a searchable log, and the answer to the third is a recording of the minutes before the fault. Build those, in that order, and stop. Add a metric only when a specific fault would have been caught by watching a specific number decline, and add alerting only for conditions where somebody doing something immediately changes the outcome. If the exercise instead reveals that your calls are about machines behaving differently from each other while running identical software, the question is not what to record but why identical inputs produce different behaviour, and that investigation goes below your own code. Best visualisation and debugging tools for robot projects is where the deeper look usually starts.

Where that leaves you, in five lines:

• If you have one machine at one site -> a check-in and on-board recording, because the phone call is the current monitoring system.
• If you support several sites -> shipped logs with one shared format, because comparison is the whole value.
• If your machines are often offline -> record locally and collect later, because live-only setups lose the outage.
• If tracking the fleet by hand costs more than a subscription -> a commercial platform, because the tax is operational, not technical.
• If the fault survives all of this and only appears when the machine is busy -> look below your code at the messaging layer, because no recording shows what was late.

The HORUS Fit Framework applies the same reasoning to that layer underneath, along five axes: ecosystem size, setup effort, team size fit, deployment target and licence. Field monitoring weighs ecosystem size heavily, which is why teams with sensors and fleets usually keep the widest toolset available to them. For the case where the records exist and still explain nothing, how to debug a robot when you can't add print statements picks up the thread.

HORUS is open source under Apache-2.0 at github.com/softmata/horus. Star it so it is in your list when you start building.