How to Debug a Robot When You Can't Add Print Statements

When printing is not an option, the answers are deferred logging, an in-memory flight recorder, published diagnostic topics and recorded runs you can replay. All four move the work of observing out of the moment being observed, which is the whole trick. That changes when the fault is in how messages travel between programs, a question about ROS 2, HORUS and the layer beneath. The rest of this post is for a ROS 2 developer whose control loop misbehaves the moment anything is added to watch it.

You have a bug that only happens when the robot is really running. You know roughly where it is. So you do the obvious thing and add a line that prints the value, and the bug goes away. You remove the line and the bug comes back.

Or the opposite, which is worse. You add the line and a new problem appears: the arm that was smooth now hesitates, the wheels stutter on the turn, and a watchdog somewhere starts complaining about a loop that used to finish in time. You are now debugging two faults, one of which you created, and you cannot tell them apart.

Then there are the places you cannot print from at all. The interrupt handler. The callback that runs while the joints are waiting for their next command. The board with no terminal attached, running headless in a cabinet, whose output goes nowhere at all. And the whole class of failures that happen once every few hours, where a print statement would produce a wall of text you will never read, containing one line that matters.

How do you debug a robot when you cannot add print statements?

Write the record, but not from the place that must not be disturbed. Every technique in this post is a version of that one move. Deferred logging drops a small record into a queue and lets a separate, less urgent thread do the file work. A flight recorder writes into memory reserved at startup and only touches disk when something goes wrong. Diagnostic topics publish state as an ordinary message, so the viewing happens in another program entirely. Recording captures what actually arrived, and replay lets you re-run it at your desk, where printing is safe again and you can add all the lines you want. Tracing tools go further and record what the whole machine was doing, including the parts of the operating system your program cannot see. None of these is exotic and none requires abandoning the code you have. What they require is accepting that observation must be cheap where it happens and expensive somewhere else.

What does debugging without printing actually mean?

It means separating the act of capturing information from the act of reading it, which on a normal program are the same act. When you print, one statement does both: it decides something is interesting and immediately does the slow work of formatting text and pushing it out. On a machine with nothing waiting, that is fine. On a robot, the code that most needs watching is code that has been promised a slice of time, and the printing spends that slice. So the discipline is to make capture as close to free as you can manage, storing something small and structured and unformatted, and to do the formatting, sorting and reading later or elsewhere. Everything else follows from that split. It is also why the techniques feel unfamiliar at first: they all involve a delay between wanting to know something and finding it out, and that delay is exactly what makes them safe. What is a control loop and why does its timing matter explains why the slice exists in the first place.

What are the actual options for getting information out of a running robot?

There are seven, and a serious project ends up using three or four. Deferred logging keeps your familiar log calls but hands the writing to another thread. A flight recorder holds recent records in memory and dumps them when a trigger fires. Diagnostic topics publish internal state as messages that other programs draw or store. Recording and replay capture a run so it can be studied at leisure. Tracing tools such as the LTTng family and the ROS 2 tracing packages build a timeline of when everything ran. Debuggers and post-mortem dumps examine a program that is stopped or already dead. And there are hardware signals, meaning toggling a pin or a light so an oscilloscope or a camera shows you the timing directly. Underneath all of it is the layer your programs use to hand data to each other, whether that is ROS 2 or a single-machine middleware such as HORUS, which decides what there is to observe in the first place. Pick from the seven by what you need to know; look at the eighth when the seven keep agreeing that nothing is wrong.

How do the ways of getting data out compare?

The rows run from the least disruptive to the most, which is roughly the order to try them in.

Read the last column first. Most lost evenings come from a technique that could never have shown the thing being looked for.

Which approach fits one developer on one ROS 2 robot?

Deferred logging plus recording, and add diagnostic topics for the two or three values you keep wanting to see. A solo developer is not short of techniques; they are short of evenings, and each of these has a setup cost that has to be repaid. Deferred logging repays immediately because it lets you keep writing the log lines you would have written anyway, in the places you would have written them, without the stutter. Recording repays the first time a fault happens once and cannot be reproduced, which will be soon. Diagnostic topics repay when you notice you are restarting the robot repeatedly to see the same number, at which point publishing it and watching it in a plotter is faster forever. Leave tracing until logs and recordings have both failed to explain something, because tracing is a genuine study in itself and is wasted on a fault the other tools would have found. Best visualisation and debugging tools for robot projects covers the viewers those topics feed.

What changes when the robot is a small board with no screen attached?

The split between capture and reading becomes physical, and that reshapes everything. On a headless board the choice is not where in your code to write records but which machine reads them, and the answer should almost always be the laptop rather than the board. Capture on the board, into memory or a local file, and move the reading elsewhere. Streaming records live over the network is pleasant and is also the thing most likely to change the behaviour you are studying, because sending data is work the board is doing instead of driving your robot. Watch storage carefully, since a small card fills quickly and a full card produces symptoms that resemble every fault except a full card. Hardware signals become unusually attractive here: toggling a spare pin at the start and end of a piece of work, then watching the pin on an instrument, tells you about timing without the board doing meaningful extra work. Best robotics middleware for Raspberry Pi and Jetson robots covers what else shifts on small hardware.

What do you do when the bug has to be found before a demo tomorrow?

Record first, then use the recording, and do not install anything new. On a deadline the recording is the only irreplaceable asset, because a fault you captured is one you can still work on after the machine has been packed up, and a fault you merely watched is one you will argue about. So start the recorder, reproduce the problem as many times as the hardware allows, and only then start thinking. Next, use what is already there: if the robot runs ROS 2, the graph and topic tools will tell you in a minute whether the programs are even connected and whether the messages you assume are flowing are flowing. Then replay the recording on your laptop and add every print statement you like, because at a desk with no motors attached the technique that was banned on the robot is suddenly the right one. Save tracing, flight recorders and instrumented builds for after the demo, when there is time to set them up properly. What happens when a robot program crashes mid-motion is worth reading before the demo rather than during it.

What skill level does each of these techniques assume?

They range from an afternoon to a genuine study, and the ordering is not what most people expect. Diagnostic topics assume the least: if you already publish messages, you can publish one more, and the tooling to look at it exists. Recording and replay assume a little discipline about storage and a willingness to build a way of feeding recorded input back through your code, which is more design work than it sounds. Deferred logging assumes you understand threads well enough not to introduce a queue that blocks the very code it was meant to protect, which is the classic way this technique backfires. Flight recorders assume you are comfortable reserving memory up front and writing a trigger that runs correctly while a program is dying. Tracing assumes real familiarity with how the operating system schedules work, and rewards it with answers nothing else provides. Hardware signals assume an instrument and someone who enjoys using one. Start at the cheap end; the expensive techniques are worth their cost only after the cheap ones have failed.

What does it look like when the printing itself is causing the bug?

The signature is a fault that changes character whenever you look at it. You add a line and the problem disappears. You add a second line and it comes back, differently. The behaviour depends on whether the terminal is attached, or on whether output is going to a screen or a file, which makes no sense until you realise the two are not equally slow. A loop that used to complete comfortably starts occasionally overrunning, and the overruns cluster around the interesting moments, because that is where you added the lines. Two threads that both print become entangled, since printing takes a lock that both must wait for, and code that never shared anything now takes turns. The tell that settles it is disabling all the extra output and finding the original fault behaves consistently again. At that point stop adding lines and switch technique, because every further line makes the experiment less trustworthy. Why does my robot behave differently every run covers the neighbouring faults that look similar and are not.

What do you give up by debugging this way?

You give up immediacy, memory and a certain amount of certainty. Immediacy goes first and hurts most: printing tells you now, whereas every technique here tells you afterwards, and afterwards means finishing the run, retrieving the records and reading them before you learn whether your guess was right. That lengthens each cycle of the investigation even as it makes each cycle trustworthy. You give up memory to flight recorders and queues, which must be reserved at startup and are therefore unavailable to the rest of your program forever, whether or not the fault ever occurs. You give up some certainty too, because a deferred record can be lost if the program dies before the writing thread catches up, which is precisely the case you most wanted the record for. That is the reason flight recorders exist alongside deferred logging rather than instead of it. And you give up simplicity: a print statement is one line that anybody understands, while these techniques are structures a new team member has to be taught.

When is ROS 2 the better choice?

ROS 2 is the better choice for this whole problem if your robot has sensors and your team has any intention of looking at data. Almost every technique here is easier inside the ROS 2 ecosystem: the recording format and its replay tools exist and are well worn, the tracing packages are already integrated with the framework so the timeline includes framework events rather than just yours, the diagnostic message conventions save you from designing your own, and the viewers that read all of it are a short install away. There is also a decade of accumulated advice about which panel to open when a transform is missing, and none of that transfers elsewhere. HORUS is not the answer to that question, because a shared-memory layer for a single computer is about how programs on one machine hand data over, not about recorders, viewers or trace tooling. The narrow case where the layer beneath is worth examining is one onboard computer running mixed-language code where the fault is when data arrived rather than what it contained.

Can you just record everything and sort it out later?

No, and here is why. Recording everything is itself an intervention: the work of capturing every message and writing it to storage competes with the work the robot is doing, which means the heaviest possible recording is the most likely to change the timing you were trying to observe. You end up with a complete account of a run that never would have happened otherwise. The second problem is that the interesting state usually was never a message. The planner's rejected options, the reason a candidate was discarded, the intermediate value between two published results: none of that travels anywhere, so no recorder catches it, however thorough. The third is human. A recording of everything is unsearchable unless the records share a format, carry timestamps taken when things happened, and can be sorted together across programs, and teams who record indiscriminately rarely do that groundwork. Record deliberately: the inputs, the decisions and the outputs, with identifiers linking them, and leave the rest. Latency, jitter and determinism explained simply is the vocabulary for describing what you eventually find.

Is attaching a debugger the right tool on a robot?

Partly, but not the way you think. A breakpoint on a moving robot stops your program and stops nothing else. The last command you sent to the joints remains in force, gravity carries on, the wheels keep whatever velocity they were given, and every watchdog in the system starts reacting to a program that has gone silent. So the arm continues into the table while you read a stack trace, and by the time you continue execution the machine is somewhere the code never expected. That is the sense in which the answer is no. The sense in which it is yes is that debuggers are excellent on everything except a live moving machine: on a replayed run at your desk, on the robot with its drives disabled and the wheels off the ground, and above all on a core dump, where the debugger shows you the exact state of a program at the moment it died. Post-mortem inspection is the most underused tool in robotics, and it costs only enabling dumps and keeping the matching build.

How do you decide which technique to reach for?

Ask what the missing information is made of, and let that choose. If the missing thing is a value you would have printed, use deferred logging, and you keep your habits. If the missing thing dies with the program, use a flight recorder, because nothing that writes to disk lazily will survive the crash. If the missing thing is a value you want to watch change while the robot runs, publish it as a diagnostic topic and put a plotter on it. If the missing thing is rare, record and replay, and do the real work at a desk where printing is legal again. If your logs and recordings all say every step completed correctly and the robot still behaved wrongly, the missing thing is ordering or waiting, and that is tracing. And if the answer turns out to be that a message arrived after the moment it was needed, the investigation has moved below your own code into how programs hand data to each other, which is a different question with a different set of answers. Best tools for monitoring a robot in the field covers the version of this problem where you cannot even reach the machine.

Where that leaves you, in five lines:

• If you just want your log lines back -> deferred logging, because the writing moves off the path that must not be delayed.
• If the program dies before it can tell you anything -> a flight recorder, because memory survives what disk writes do not.
• If you keep restarting the robot to see one number -> a diagnostic topic and a plotter, because you only pay the setup once.
• If the fault is rare -> record it now and replay it later, because at a desk you can print freely again.
• If every record says nothing went wrong and the robot disagrees -> tracing, because the fault is in ordering, and no log line describes ordering.

The HORUS Fit Framework applies the same reasoning to the layer under the debugging tools, along five axes: ecosystem size, setup effort, team size fit, deployment target and licence. Debugging weighs ecosystem size hardest, which is why teams who spend real time inside their robots stay where the recorders and viewers already live. For the wider picture of what a stack is made of, what is a robot software stack, layer by layer is the map.

HORUS is open source under Apache-2.0 at github.com/softmata/horus. Star it so it is in your list when you start building.