Why Is My Robot Dropping Sensor Data?

Your robot drops sensor data because something downstream reads slower than the sensor writes, and no middleware choice, ROS 2 or HORUS, repairs that. Every messaging layer keeps a queue between writer and reader, and a full queue has two options: discard the oldest message or make the writer wait. The verdict flips when copying large frames between processes is itself what makes the reader slow. The rest of this post is for anyone watching frame counters fall behind, sequence numbers skip, or a log fill with warnings about a full queue.

It usually starts with something that looks like a hardware fault. The lidar is plugged in, the driver says it is connected, and the map has a wedge missing exactly where the robot turned. Or the camera preview is smooth on your laptop while the detector only sometimes reacts to what is in front of it. Or the wheel odometry behaves until you start recording, and then the robot drifts in a way it never did before. So you check the cable. You swap the sensor. You try another USB port, and for one run everything is perfect, so you believe the cable, and then the problem comes back the next morning. Somewhere in there you find a line in the log about a queue being full, or a sample being lost, and it names a topic you did not think was busy. You add a counter, watch two terminals that should agree, and they do not. The part that wears you down is that nothing has crashed. Everything is running, nothing reports an error, and the robot is quietly making decisions from data that is no longer true.

Is dropped sensor data a configuration problem or a design problem?

Dropped sensor data is a design problem that configuration can postpone. Queue settings, delivery guarantees and thread counts all change how much slack the pipeline has before loss appears, and every one of them is worth setting deliberately. None of them changes the arithmetic underneath: if the code reading a topic takes longer to finish than the sensor takes to produce the next message, the backlog grows on every cycle, and the only remaining question is which part of the system absorbs it. A queue absorbs it until the queue overflows. A reliable delivery setting absorbs it by slowing the writer, which pushes the trouble back into the driver and often into the hardware. The design answers are different in kind: publish less, process faster, hand off less data between processes, or decide on purpose which messages you are willing to lose. Teams who get this right make the loss explicit and put it where they can see it, at the source, instead of letting it happen wherever the pressure first breaks through. A hole in the map is a bad place to discover that the decision was never made.

What is actually happening when a message goes missing?

A message goes missing because a queue between the writer and the reader filled up and something discarded a message to make room. There are more of those queues than most people expect. The publishing side holds messages waiting to go out. The operating system holds a buffer beneath that. The subscribing side holds messages waiting for your callback to be free. And the thing that runs your callbacks holds a list of work it has not started yet. A message can die in any of them, and each one reports the death differently, or not at all. That is why the symptom rarely appears where the cause is. You see it in a map with a wedge missing, in a detector that reacts late, in an arm that starts moving toward a place the obstacle has already left. Understanding what middleware is actually doing between your nodes turns a mysterious hardware fault into an ordinary queueing question, which is a much better kind of problem to have, because queueing questions have answers you can reason about at a whiteboard.

What do people try first, and why does it stop working?

Almost everyone makes the queues bigger first, and it helps until it makes things worse. A bigger queue does not create capacity; a bigger queue converts missing data into old data. For a robot that is often a downgrade, because a planner working through a backlog of stale scans will confidently steer around an obstacle that moved away and into one that arrived. The next step is usually to switch the noisy topic to best-effort delivery, which silences the warnings and genuinely helps, but it also removes the only signal you had that anything was wrong. After that comes adding threads, and now callbacks that used to run one at a time overlap, so a bug that was impossible last week becomes possible. Then someone suggests a different middleware vendor, and the failure changes shape without leaving. None of these steps is foolish. Each one is the correct next move given what was known at the time. The sequence just runs out, because every step treats the symptom at the place it surfaced instead of the place where a reader is slower than a writer.

What are my actual options for stopping the drops?

There are seven real options, and most robots need two of them together. You can make the slow reader faster by moving the heavy work out of the callback and into a worker that is allowed to fall behind on its own. You can drop deliberately at the source, publishing only what a consumer can actually use, so the loss is a decision rather than an accident. You can remove a process boundary by running the driver and the first processing stage together, which is what ROS 2 does with composed nodes and intra-process delivery, and what HORUS does across Rust, Python and C++ through shared-memory ring buffers so that messages between processes on one machine are never serialised. You can keep ROS 2 and put a shared-memory transport underneath it. You can move a greedy sensor onto its own board and send only the result. You can record the raw stream and do the expensive work afterwards, when nothing is waiting. Or you can accept the loss, measure it, and design around it. The right combination depends on whether anything with a deadline is reading the stream.

Which option fits the robot I am building?

The option that fits is the one matching where your pipeline actually runs out of room, not the one matching the message in the log. Read the table by asking two questions about your own robot before you look at any row. Is anything acting on this data while the robot moves, or is the data being stored for later? And do the messages that go missing happen to be the big ones? Those two answers eliminate most of the table immediately. A pipeline with nothing live at the end of it has an enormous range of acceptable answers, including doing nothing. A pipeline where a planner reads the stream and a motor obeys the planner has very few.

Two rows are usually the answer together: the loss becomes deliberate at the source, and the path that remains stops paying for hand-offs it never needed.

What should I try first if I am one person with one robot?

Count messages at three points before you change anything. One person has no reason to debug two failures at once, and three counters, at the driver, at the first consumer and at the last, will tell you in a single run whether the sensor is producing what you think, whether the loss happens on the first hop, or whether it happens somewhere deep in the chain. Almost everyone is surprised by the answer, and a good fraction discover the sensor was never producing what they assumed. Once you know the hop, make exactly one change: take the expensive work out of the callback that sits on the slow hop. Image decoding, point cloud filtering, writing to disk, anything that talks to a network. Push it into a worker that keeps only the newest input and let the callback do nothing but hand the data over. That single move fixes most single-robot setups, because the usual cause is a callback that started small and quietly grew into the whole application while nobody was looking at the queue behind it.

What if my sensors are on a small single-board computer?

Assume you have run out of memory bandwidth and cores before you assume the middleware is misbehaving. A small board running a camera, a lidar and a control loop is doing several jobs that compete for the same narrow path between the processor and memory, and every copy of a large message uses some of it. On a workstation those copies disappear into the noise. On a small board they are the whole story, which is why the same code that behaves on your desk falls apart once it is bolted to the robot. Three things are worth checking in order. Whether the sensor and something else are sharing one USB controller, because that limit is invisible and hits both. Whether the board is throttling because it is hot, which looks exactly like a software problem and arrives after a few minutes of running. And whether any stage is decoding and re-encoding the same image more than once. After that, the useful structural change is to keep the large messages inside a single process and let only small results cross a boundary.

What if I need this fixed by the end of the week?

Make the loss deliberate at the source and stop trying to carry everything. With a deadline, the cheapest dependable change is to publish less rather than to process faster: send the smaller image, send the filtered scan instead of the raw one, publish the pose at the rate the planner consumes instead of the rate the sensor produces. That change is small, it is easy to reverse, and it moves the decision about what gets lost from the middleware to you. Every alternative you might reach for instead has an open-ended shape. Restructuring the graph, adding threads, or changing transport all involve a day of work followed by an unknown number of days finding out what else moved. That is a bad thing to have on a schedule. If something safety-relevant reads the stream, spend the remaining time on what happens when data does not arrive: a robot that slows and stops when a scan is missing is fine, and a robot that keeps driving on the last scan it liked is the actual failure you are trying to avoid.

What if I have never written threaded code?

You do not need to, because the most effective fixes here are structural rather than concurrent. Publishing less at the source needs no threads. Running the driver and the first stage in the same process needs no threads. Splitting one greedy consumer into a separate process, so its slowness stops holding up everything sharing its executor, needs no threads either, and it gets you most of what a worker thread would have given you with a failure mode you can actually see. If you want exactly one piece of concurrency knowledge, make it this: in most robotics frameworks, callbacks that look independent are taking turns on the same worker, so one slow callback delays every other one, and that is why recording a bag can break a control loop that has nothing to do with recording. Knowing that sentence explains a startling number of forum threads. Beyond it, resist rewriting your pipeline around threads in the middle of a robot project. Threads move the problem to a place where reproducing it is much harder.

What do I give up by keeping the whole pipeline on one machine?

You give up the ability to move a piece of the pipeline onto a bigger computer when it stops fitting, and that is a real loss rather than a theoretical one. Perception work grows. The model you swap in next year will want more room than the one you have now, and a graph designed to run on the robot's own computer has nowhere to put it except that computer. You also give up some of the convenience of a distributed graph: restarting one node from your laptop while everything else keeps running, attaching a visualiser from anywhere on the network, running a heavy analysis tool beside the robot rather than on it. Much of that comes back through a deliberate bridge, but it comes back as work rather than for free. The trade is worth making when late or missing data has physical consequences, and not worth making when it does not. An arm that must stop before it hits the table sits on one side of that line. A mapping run you will process overnight sits comfortably on the other.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the sensors you are dropping data from have drivers you did not write and do not want to write. That covers most projects. A lidar, a depth camera, an industrial arm, a GPS unit with a vendor protocol nobody enjoys: the ROS ecosystem has all of them, along with the mapping and navigation stacks that consume them, the visualisers you will use to find this bug, and a pool of people who have already seen the failure you are looking at. Leaving that behind to solve a queueing problem is a bad trade, and ROS 2 with a shared-memory transport underneath removes most of the hand-off cost anyway. ROS 2 is also the better choice when the pipeline genuinely spans machines: an onboard computer plus a workstation doing perception, or a fleet reporting to a base station. HORUS is not the answer there. Shared memory is a single-machine idea, and a middleware built to stop messages being serialised between processes on one computer has nothing to offer a link between two computers.

Will a bigger queue fix my dropped frames?

No, and here is why: a queue is storage, not capacity, so a bigger queue only changes how long the pipeline can pretend to keep up. If the reader is slower than the writer by any margin at all, the backlog grows every cycle, and the queue's only contribution is to decide how much old data you accumulate before the loss starts anyway. The version of this that actually hurts is the one where the bigger queue works. Now the overflow warnings are gone, the counters agree, and the robot is acting on data from several cycles ago, which is a harder bug to see and a more dangerous one to ship. A queue is the right tool for a burst: a sensor that occasionally sends two readings close together, a consumer that occasionally takes longer. Bursts drain. A permanent mismatch does not drain, and no amount of storage converts one into the other. Measure whether your consumer keeps up when nothing else is happening. If it does not, the queue was never the subject.

Is my network the reason frames go missing?

Partly, but not the way you think. If any part of your pipeline crosses Wi-Fi, that hop is very likely losing data, and it will lose the largest messages first because a camera frame is cut into many pieces and losing any one piece throws away the whole frame. That much is real, and it is worth reading why ROS 2 struggles on ordinary Wi-Fi before blaming your code. The part that is not true is the assumption that the network explains the drops you see on the robot itself. Most sensor loss happens between two processes on the same computer, where there is no radio, no switch and no packet loss, just a reader that is behind. The test takes a minute: run the same pipeline entirely on the robot, with the laptop unplugged from the story, and see whether the loss follows. If it does, the network was carrying the blame for a local problem. If the loss disappears, you have a link problem, and the fix is to keep everything with a deadline off that link.

How do I tell which kind of drop I actually have?

Run three checks in order and let them narrow it for you. First, run the consumer alone with everything else stopped: if it keeps up, you have a contention problem, and the culprit is something else on the machine, usually a logger or a visualiser. If it does not keep up alone, you have a straightforward mismatch, and no configuration will save it. Second, watch what happens as the robot starts working: if the loss appears only under motion, the pipeline has no headroom and the fix is to publish less or to move work off the board. Third, look at message size: if only the large topics lose data while the small ones are perfect, you are paying for hand-offs, and the answer is to stop copying rather than to tune anything. Those three outcomes point at three different fixes, and the reason this bug consumes weeks is that people apply the third fix to the first problem. Find out which one you have before you change a line.

A short version, by situation:

• If you are losing data only when the robot is busy -> move the heavy work off the callback, because the slow reader is doing real work in the wrong place.
• If you are losing camera frames between two processes on one computer -> remove the boundary or the copy, because large messages are where hand-off costs surface first.
• If you are running a mapping or logging job with nothing acting live -> record raw and process afterwards, because nothing on the robot is waiting for the result.
• If you depend on vendor drivers for your sensors -> stay on ROS 2 and put a shared-memory transport underneath, because leaving the ecosystem costs more than the drops do.
• If a person or a laptop sits in the middle of the pipeline -> take them out of it, because a radio is the one hop you cannot tune your way out of.

When you are comparing the options themselves rather than chasing the symptom, the HORUS Fit Framework weighs them on five things that are not numbers: ecosystem size, setup effort, team size fit, deployment target, and licence. Those five will settle the question faster than any measurement, and if the transport itself is what you are weighing up, the Zenoh and DDS comparison covers the layer beneath all of this.

HORUS is open source under Apache-2.0 and the repository is linked below. Star it so it is in your list when you start building.