What Is Sensor Fusion, Without the Maths?

Sensor fusion combines disagreeing sensors into one estimate a robot can act on, and most builders should adopt an existing filter rather than write one. The mathematics is largely solved and available; the real work is bookkeeping — when a reading was taken and how far to trust it. Middleware such as ROS 2 or HORUS decides whether readings arrive together; if one sensor answers everything you ask, do not fuse at all. The rest of this post is for someone whose robot knows where it is until suddenly it does not, and who suspects the sensors are arguing.

Your robot was going along fine and then quietly stopped believing in reality. It drove a straight corridor and reported a gentle curve. It sat still on the bench and, according to its own numbers, wandered a hand's width to the left while nobody touched it. You spun it in place and it came back convinced it was facing a different wall.

Nothing crashed. Every sensor reports values, every driver says it is connected, and each graph looks plausible on its own. It is only when you put two of them side by side that you see the argument: the wheels insist the robot moved this far, the camera insists it moved that far, and neither one is obviously lying.

So you pick a favourite. You trust the wheels, because they are smooth, and the robot drifts away from the truth over a long run. You trust the camera, because it is anchored to the room, and the robot jumps sideways whenever the view changes. Then somebody says the word fusion and hands you a page of matrices, and it is not remotely obvious what any of that has to do with your carpet.

Does your robot actually need sensor fusion?

You need sensor fusion the moment two sensors describe the same quantity and disagree in a way that changes what the robot does. Before that point it is work with no payoff. An arm bolted to a table, reading its own joint encoders, already knows where its hand is; nothing needs combining, and a filter would only offer a second opinion about a fact you already had. The dividing line is whether any decision on your machine depends on a quantity that no single sensor reports well. Where the robot is on a floor is the classic case, because wheels know motion but not place, and a camera knows place but not motion. Which way the robot faces is another, and so is whether the gripper is holding anything. If you can name such a quantity, you are already fusing — the only question is whether you are doing it deliberately or by accident, in a chain of if-statements nobody has read for months.

What is sensor fusion in plain language?

Sensor fusion is taking several unreliable witnesses to the same event and producing one account the robot can act on. Every sensor is wrong in its own particular way, and the useful insight is that the ways differ. Wheel encoders are smooth from one instant to the next and slowly, permanently wrong across a journey, because a wheel that slipped is never recovered. A camera fixing itself against the room is roughly right forever and jumpy moment to moment, because it re-decides the answer on every frame. Neither of those is a bad sensor; they are sensors with opposite habits. Fusion keeps the smoothness of the first and the anchoring of the second by trusting each one where it is strong. The bookkeeping underneath is about trust and time: how much confidence a reading deserves, and which moment that reading actually describes. That is the whole idea. The equations people show you are one careful way of doing that arithmetic, and they are not the point — the point is that a robot should believe its sensors unequally.

What are your actual options for combining sensors?

You have more options than writing a filter, and two of the better answers are to fuse nothing and to borrow somebody else's filter. The lightest option is to choose the single sensor that fails least in your environment, then constrain the environment so that stays true. Above that sits a hand-written blend of two sensors, which is a dozen lines of code and covers more first robots than anybody admits. Above that sit the ready-made estimators: a sensor module that reports a fused position directly, or a maintained package that takes wheel odometry, an inertial unit and a map and returns one answer — under ROS 2 that package plus its coordinate-frame bookkeeping is the standard route, while HORUS, an open-source real-time middleware for Rust, Python and C++, does a different job, putting all three languages on the same shared-memory ring buffers so readings from separate processes on one machine reach your estimator without being serialised on the way. Then there is moving the fusion onto a microcontroller sitting beside the sensors. Read the table as a description of situations rather than a ranking.

What should you do if you are one person building a first robot?

Make one sensor honest before you make two sensors agree. The failure I see most in first robots is a fusion layer built on a sensor that was never calibrated, never mounted straight, or never read correctly — and the filter dutifully blends a good reading with a bad one and produces a confident average of the two. Spend the first evening on the unglamorous part. Check that the wheel measurements match a tape measure over a known run. Check that the inertial unit reports level when the robot is level. Check that the units are what you assumed, and that nothing is mounted at an angle nobody wrote down. Then log the raw readings from both sensors, drive a shape you can verify by eye, and put the two records side by side. Much of the time, the argument you were about to solve with mathematics turns out to be a sign flip or a mounting offset. Only once both witnesses are individually credible does combining them mean anything.

What changes when fusion runs on a Raspberry Pi or a small board?

A small board turns the question from which filter into which clock, because the moment a reading arrives is not the moment it describes. Sensors come in over USB, serial or a small bus, each with its own driver and its own habits, and every reading spends an unpredictable while in transit before your code sees it. If your estimator treats arrival as measurement time, then whenever the board gets busy your readings quietly become history, and the estimate lags reality by more exactly when the robot is working hardest. The cure is not a bigger board. It is timestamping as close to the sensor as you can manage, keeping that timestamp attached to the reading for its entire journey, and letting the estimator work from the moment rather than the arrival. A filter chewing on every raw reading is also real work, and if it shares a program with the code driving the wheels, the wheels inherit its every hesitation. Separate those two early.

What if you have only a few weeks before a demo?

Cut the sensor list instead of adding a filter. With a deadline close, the reliable move is to pick the sensor that fails least in the room where the demo happens, then change the room so that sensor keeps working — better light, markers on the floor, a surface the wheels do not slip on, a route that avoids the glass wall. A filter added under deadline pressure is a new failure nobody has watched misbehave yet, and it will misbehave for the first time in front of an audience, because an audience is a room full of moving obstacles your test runs never had. If you must combine something, combine two sensors with a rule you can state in one sentence — use the wheels for motion, and correct the position whenever the camera sees a marker — and accept a small visible jump at each correction. A jump you can predict makes a better demo than a smooth estimate that occasionally decides the robot is in the next room. Save the proper estimator for afterwards, when breaking it costs nothing.

What if you have never written estimation code before?

Answer three questions about every sensor before you learn a single equation. First: how is this sensor wrong — does its error grow as the robot travels, or does it stay small and bounce around? Second: which moment does a reading describe, and how do you know that? Third: what does the robot do when this sensor stops reporting, which every sensor eventually does. Those three answers drive almost every design choice fusion will ask of you, and not one of them is mathematics. Somebody who can answer them will use a library filter well; somebody who cannot will tune a library filter for a week and end up with a confident wrong answer. The other habit worth forming early is treating your estimate as a claim with a confidence attached, rather than as a fact. A robot that knows roughly where it is, and knows that it only knows roughly, can slow down, hunt for a landmark, or ask for help. A robot that believes its own number exactly will drive into the table with total conviction.

What do you give up by fusing sensors?

You give up the ability to point at a value and say where it came from. Before fusion, when the robot is in the wrong place, you can look at each sensor and see which one lied. Afterwards, the position is a blend that no sensor ever reported, shaped by trust settings somebody chose months ago, and debugging it means reasoning about a mechanism instead of reading a number. You also inherit a failure that raw sensors do not have: the confident wrong answer. A single failing sensor usually looks broken, while a fused estimate absorbs a broken input and keeps producing something plausible, which is much harder to notice and much more dangerous to act on. There is a running cost as well. The filter is code inside your timing budget, the trust settings want revisiting whenever you change a wheel or move a camera, and every added sensor is another clock to reconcile. None of that argues against fusion on a robot that needs it. It argues against fusing early, on a machine where one good sensor would have done the job.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever your robot needs a map, and that covers most mobile robots that go somewhere on purpose. The moment you want a machine to know it is in the kitchen, rather than merely that it moved forward, you need mapping, a way to keep coordinate frames straight, and an estimator that ties wheels, an inertial unit and a scan together — all of which already exist and are maintained in that ecosystem, and all of which are months of work to rebuild badly. Take them. The same holds if the sensor you bought ships a ROS 2 driver and nothing else, if the robot spans more than one computer, or if your team already speaks the vocabulary. HORUS is not the answer for those projects, and choosing it there means writing plumbing instead of building a robot. A worked example of the standard pairing is fusing an inertial unit with wheel odometry under ROS 2, and it is the right starting point for a robot that has to know where it is on a floor plan.

Will adding more sensors make the estimate better?

No, and here is why: a sensor only helps when it fails differently from the ones you already have. A third source that goes wrong at the same moments, in the same direction, as your existing two cancels nothing out — it votes with the majority and makes the wrong answer more confident. Two cameras staring at the same featureless white wall will agree beautifully and be equally lost. What improves an estimate is complementary failure: something that stays right while the others drift, or that notices when they have gone astray. A single marker on a doorframe the robot sees once in a while is worth more than a shelf of sensors that all struggle on carpet. There is a cost that grows quicker than the benefit, too — each added sensor is another driver, another mounting offset and another clock. So the question to ask about a candidate sensor is not what it measures. It is when this one still works while the others are failing.

Is sensor fusion mostly a maths problem?

Partly, but not the way you think. The mathematics inside a filter is real, and it is also the part you are least likely to get wrong, because it is written down, tested, and packaged in libraries for every language you might use. What breaks fusion in practice is the bookkeeping around it. Readings arrive with timestamps in different formats, or with none at all. Units are mixed. A sensor sits a few finger-widths from where the model claims, and nobody measured. Coordinate frames disagree about which way is forward. A driver silently repeats its last value when the hardware stops responding, so the estimator is fed a lie wearing the costume of fresh data. Every one of those is clerical, and every one produces symptoms indistinguishable from a badly tuned filter — which is why builders spend weeks adjusting trust values when the actual fault was a sign flip. Spend your effort where it pays: make every reading state clearly what it measured, in what units, at what moment, from what position on the robot.

What does bad sensor fusion look like when it goes wrong?

It looks like a robot that is confident and wrong, which is a worse condition than a robot that is obviously confused. The map shows the machine gliding smoothly through a wall. The position is perfect at first and then slides steadily. The robot copes with a slow careful drive and gets thoroughly lost the one time somebody spins it on the spot. It is fine on tile and hopeless on carpet. It behaves in simulation, where every reading is instant and honest, and falls apart on the real machine where readings are neither. The tell that separates a fusion fault from a broken sensor is that no individual reading looks wrong. Each sensor, viewed alone, is doing something reasonable, and the failure lives in how they were combined — usually in trust granted to a sensor outside its comfort zone, or in a timing assumption that holds on a quiet computer and not on a busy one. If readings are vanishing altogether instead, that is a different fault with a different cure.

What do people try first, and why does it stop working?

Most people average the two readings, and it works well enough to be encouraging. The average of a drifting sensor and a jumpy one is a slightly drifting, slightly jumpy estimate, which for a slow indoor robot on a good day is perfectly serviceable. It stops working at a predictable moment: when the two sensors are no longer equally believable. A wheel spins on a rug and the average inherits half of a lie. The camera loses the room and returns nonsense, and the average is half nonsense. The next attempt is usually to trust whichever reading arrived most recently, which quietly promotes the chattiest sensor to the most important one, whether or not it earned that. Then comes a queue, and the queue raises the question nobody wanted: what should the estimator do with a reading that describes a moment already gone? Answering it well means old readings correct history rather than steer the present. Answering it badly means the robot chases where it used to be, which is the same failure as a control loop acting on stale information.

How do you tell which fusion problem you actually have?

Isolate one cause at a time by driving the robot in ways that provoke each fault on its own. Drive a long straight line and come back: if the error grows with the distance travelled, you have drift or a calibration mistake, and no filter will rescue you from a wheel diameter that is wrong. Spin on the spot: if heading falls apart while position survives, the fault is in whatever reports rotation, not in the blend. Sit perfectly still and watch: if the estimate wanders while the robot does not move, something is reporting motion that is not happening, usually a bias nobody subtracted. Then start the camera pipeline or another heavy program and repeat a test that passed: if the estimate degrades only under load, you have a timing problem rather than a mathematics problem, and the cure is separation, not tuning. Four faults, four different fixes, and only one of them lives inside the filter. Deciding what the robot should then do with its answer is a separate skill, described in motion planning in plain terms.

Decide by situation rather than by instinct:

• If one sensor answers every question your robot asks -> do not fuse, because a filter you did not need is a failure you did not have.
• If two sensors disagree and both are calibrated -> a hand-written blend, because a rule you can explain beats a filter you cannot.
• If your robot must know where it is on a floor plan -> ROS 2 and its maintained estimator, because that stack is years of work you would otherwise repeat.
• If the estimate must survive a busy computer -> move the fusion onto a microcontroller beside the sensors, because a chip with one job cannot be distracted.
• If Python, C++ and Rust parts on one machine all feed one estimator -> a shared-memory middleware, because the language boundary is where readings go stale.

When the foundation question arrives, weigh it on the five axes of the HORUS Fit Framework — ecosystem size, setup effort, team size fit, deployment target, and licence — and take the option that loses on the fewest, with no scores involved. If your robot keeps landing on one machine, more than one language, and sensors that must reach the same place at the same moment, star HORUS on GitHub so it is in your list when you start building.