Should You Simulate Sensors or Fake Them?

Fake sensors with recorded or scripted data while you build the plumbing; simulate them once your policy's decisions depend on what the sensor actually sees. A fake proves your code runs, ships data and does not deadlock, which is most of the early bugs, and costs an afternoon on ROS 2 or on a middleware such as HORUS. The verdict flips when the thing you are tuning is perception itself. The rest of this post is for someone with a model or a policy who needs sensor data flowing this week and cannot lose a month to a simulator.

You have a model that works. It works on a dataset, on your machine, in a notebook you can rerun any time you like. What you do not have is a robot, or you have one and it is on a shelf waiting for a part, and the question in front of you is what to feed the model until then.

The obvious answer is a simulator, and then the second day of the simulator happens. The install wants a specific driver version. The camera renders a black image until somebody adds a light to the world. The laser scan arrives in a coordinate frame nobody can name. Somebody says the word URDF and the room goes quiet.

So you write a stub instead. Twenty lines that publish a plausible reading, and everything works immediately, and a small voice says this is cheating. Two weeks later the stub is still there, three parts of the codebase quietly depend on how well-behaved it is, and nobody can say which of last week's results were about the model and which were about the stub.

Should you simulate sensors or fake them?

Fake them first, and switch to simulation the moment your results depend on what the sensor sees rather than on whether data arrives at all. The reason is that early bugs are almost never perceptual. They are plumbing: a process that starts before the one it depends on, a message shape that changed on one side only, a consumer that blocks and takes the loop down with it, a queue that grows until the machine swaps.

None of those bugs care whether the numbers mean anything. A constant value flushes them out just as well as a rendered scene, and you can have the constant working before lunch.

The switch comes when the question changes. Once you are asking whether the arm reaches the right object, whether the policy slows down for a person, or whether the gripper closes at the right moment, the content of the data is the experiment, and a stand-in that produces the same reading forever answers nothing. Most teams cross that line later than they expect, and then stay on the wrong side of it for a month because the stub still runs and nobody wants to spend the week.

What is the difference between a faked sensor and a simulated one?

A fake produces data of the right shape; a simulation produces data of the right shape by modelling something that could have produced it. That is the entire distinction, and everything else follows from it.

A fake can be a constant, a repeating pattern, random values inside a plausible range, or a recorded run played back at the pace it was captured. Nothing behind a fake knows where the robot is. Move the machine across the room and the readings do not change, because there is no room.

A simulation contains a world with geometry and materials, a model of the device, and a step that computes what the device would report from where the device currently stands. Move the machine and the readings change, which is the whole point and also the whole cost.

Between the two sits replay, which is more honest than a simulation about what a real device does and completely unable to react. A recording of a corridor is a genuine corridor, right up to the moment your robot decides to turn left, at which point the recording keeps going straight and the two of you stop agreeing about reality.

What are the actual ways to get sensor data into your code before hardware arrives?

There are about six, and the first three cost hours rather than weeks. A scripted stub, publishing fixed or patterned values. A stub with deliberate imperfection, adding noise, delays, occasional dropouts and the odd nonsense reading. A recording played back, either captured from your own device or taken from a public dataset. A physics simulator with a simple sensor model, which gives you depth, laser scans and joint readings that change when the robot moves. A rendered simulator with camera models, which gives you images good enough to look at and sometimes good enough to train on. And the real device on a bench, connected to a laptop, with no robot around it at all.

The route the data takes matters as much as the content. In a ROS 2 project a stand-in publishes on the same topic the real driver would use, so swapping means launching something different. On a middleware such as HORUS, where Rust, Python and C++ processes exchange messages through shared memory on one machine, the same trick works across languages: the Python process consuming the readings never learns whether the producer was a driver or a script.

How do the sensor-data options compare?

The comparison comes down to what each option can and cannot tell you, and how much setup stands between you and the first message. Read the last two columns first, because the failure mode here is picking something that answers a question you were not asking.

The rows are not ranked. Most projects use three of them at once and change which is which every couple of months.

Does the answer change if you are training a model rather than writing a controller?

Yes, and it changes in one direction: a learned policy demands far more of a stand-in than a controller does. A controller reading a joint angle mostly needs a number that arrives on time and stays inside a sane range. Give it a constant and the controller behaves stupidly but visibly, and you learn what you wanted to learn about the plumbing.

A policy is different, because a policy will learn whatever your fake happens to do. If the stub returns values that drift in a repeating pattern, the model can discover that pattern and use it, and your training curve will look wonderful for reasons that have nothing to do with the world. This failure is quiet. Nothing crashes. The numbers improve. You only find out on hardware, when the pattern is not there and the policy has nothing to stand on.

The practical rule is that stand-ins are for the code path and real or simulated data is for the learning. Train on recordings, on a simulator, or on both, and reserve stubs for the part of the week where you are checking that processes start, messages arrive and shutdown is clean.

Does the sensor you chose decide whether faking works?

Yes, and the dividing line is between devices that report a number and devices that report a version of the world. Joint encoders, force readings, battery state, wheel odometry, inertial measurements: for all of these, a faked value is close to free and surprisingly useful, because the surrounding code mostly manipulates the number rather than interpreting it.

Cameras and laser scanners are the opposite. A faked image is a placeholder for a pipeline, not a test of it. Everything interesting about a camera lives in what the scene contains, and a stand-in contains nothing, so the detector finds nothing and you have learned only that your code can pass an array around.

There is a middle group people forget. Depth images and laser scans are geometric rather than photographic, which makes them the cheapest realistic thing a simulator can give you. Many robot behaviours that feel like vision problems are actually geometry problems, and switch on depth alone. If your machine mainly needs to avoid furniture and find flat surfaces, you may never need the expensive renderer, a point worth weighing against how realistic your simulation needs to be.

What should you do if the hardware arrives next month?

Fake everything now, and spend the month on the parts that will not change when the device shows up. That means the process structure, the message shapes, startup and shutdown, what happens when a producer dies, and how readings are recorded. All of that is real work, none of it needs a sensor, and all of it is much harder to change later once a model depends on it.

Spend part of the month preparing for the first day of hardware, which is a day teams reliably waste. Write the recording setup before the device exists, so that the moment it produces anything, you are capturing it. Decide where recordings live and how they are named. Nothing is more annoying than a good first session that cannot be replayed because the capture was added afterwards.

Do not spend the month building a detailed simulation of a device you have never held. Modelled noise, guessed field of view and imagined artefacts are all fiction until you can compare them with the thing, and the comparison usually reveals that the real device is strange in a way you would never have guessed.

What if you have never set up a robot simulator before?

Then budget a full week for the first working scene, and do not put that week on the critical path of anything. This is the single most common planning mistake in the area. The install is fiddly, the tutorials assume the previous version, the model of your robot needs a description file somebody has to write, and the first scene renders as a grey void with an object floating somewhere off screen.

The way through is to make the target absurdly small. One floor, one box, one sensor, and success defined as seeing any reading change when the robot moves. Do not model your actual machine yet. Do not import the warehouse. Get one message to change in response to one motion, and then grow the scene from a thing that already works.

While that week runs, keep the fake in place so the rest of the project continues. Teams that stop everything for the simulator lose momentum and often abandon it halfway, ending up with neither a working scene nor an honest stand-in, and choosing between them again a month later with less time.

What goes wrong when a faked sensor stays in too long?

Your code accumulates assumptions the fake happens to satisfy, and they all fail on the same afternoon. It is worth being concrete about those assumptions, because each one is invisible until it breaks.

Readings arrive in order. Readings never arrive twice. Readings never stop for a moment and then resume. The value is always inside range. The timestamp always increases. Two sensors always agree about what time it is. The producer starts before the consumer. Every one of those is true of a script and false of hardware, and code written against a stub quietly depends on all of them without any single line of code saying so.

Then the device is connected and five things break at once, which is the hardest possible debugging situation, because you cannot tell which failure is causing which. The cheap insurance is to make the fake misbehave on purpose long before hardware arrives: drop messages, repeat one, stall for a second, emit one absurd value per minute. Software that survives a rude stand-in tends to survive a real device, and this is exactly the sort of check worth running automatically, as covered in continuous integration for robot software.

What do you give up by faking sensors instead of simulating them?

You give up any claim about whether the robot perceives correctly, and that is a larger surrender than it sounds. With a fake in place, you can demonstrate that a message travels from a producer to a policy to a motor command, and you can demonstrate that nothing crashes. You cannot demonstrate that the machine sees the shelf.

You also give up the class of bug that only appears when the world responds. A robot turning towards a wall should see the wall arrive; a fake never delivers that, so any logic depending on the relationship between motion and observation is untested. Anything that closes a loop through the environment is invisible to a stand-in by construction.

The third thing you give up is the timing shape of real data. Devices are irregular in ways that scripts are not: bursts, gaps, warm-up periods, a scan that takes longer when the room is bright. Software tuned against a metronome tends to be surprised by an instrument, which is one of the standard routes into sim-to-real disappointment.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the stand-in is the smaller half of your problem and the ecosystem around it is the larger half. If you need a laser scanner model, a camera plugin, a bridge into an established simulator, a recorded-run format other people can open, and a viewer that shows what the robot currently believes, ROS 2 has all of those already and they are maintained by people who are not you.

That advantage compounds for anybody doing perception work. Datasets, drivers, calibration tools and visualisation exist for the ecosystem, and the vendor of your next sensor almost certainly ships a driver for it. Reproducing even a fraction of that yourself is a project, not a task.

HORUS is not the answer for those needs. A shared-memory middleware moves messages between processes on one machine and provides no sensor models, no simulated world and none of the ecosystem's tooling. If your central difficulty is making a camera produce believable images, choosing a smaller stack means building the interesting part yourself, and that is rarely what a team with a model to train wants to be doing.

Is a recorded log just as good as a simulator?

Partly, but not the way you think. A recording beats any simulation on realism, since a recording is what a real device really produced, complete with the artefacts nobody would think to model. For anything that only observes, meaning detection, classification, calibration or measuring how a pipeline copes with a difficult scene, a recording is the better tool and costs far less to obtain.

The limit is interaction. A recording cannot respond to what the robot did. The moment your system chooses an action that would have changed the view, the recording carries on describing a world that no longer exists, and every result after that instant is fiction. This is why recordings are excellent for perception and useless for control.

The practical arrangement is to use both, on purpose, for different questions. Recordings answer whether the machine understands what it sees. A simulator answers whether the machine does the right thing about it. Teams that try to make one tool answer both questions end up trusting an answer that the tool was never able to give.

Does a photorealistic simulator remove the need for real data?

No, and here is why. A renderer is built to match how a scene looks to a person, not how a scene looks to your particular device. Real cameras have automatic exposure that hunts when the robot turns, a shutter that smears fast motion, dust on the lens, a colour response that shifts under different lights, and a driver that occasionally hands over a frame from a moment ago. Rendered images have none of that unless somebody deliberately adds each one.

There is a second gap that has nothing to do with pixels. The simulated world contains the objects somebody put in it, while the real site contains a trolley left in the aisle, a floor sign, a person walking through, and a reflective surface that convinces the sensor there is a hole in the ground. Appearance is one gap; content is another, and the second is usually larger.

None of this makes rendering pointless. It makes rendering a way to generate large amounts of nearly-right data cheaply, which is genuinely valuable when it is combined with a smaller quantity of real data rather than used to replace it.

How do you decide which sensors to fake and which to simulate?

Go through your sensors one at a time and ask a single question about each: if the content of this reading were nonsense, would today's experiment still mean anything? If the answer is yes, fake it, permanently, without guilt. If the answer is no, that sensor needs a simulator or a recording, and it is the only one that does this week.

That sorting usually produces three groups. Sensors that never need simulating, because the code treats the number as a number. Sensors that need geometry but not appearance, which a simulator gives you cheaply and which cover more behaviours than most teams expect. And usually one sensor, often a camera, whose content is the actual research question, which deserves the expensive treatment precisely because everything else has been made cheap.

Then write down which is which, in the repository, next to the stand-ins themselves. The dangerous state is not having fakes. It is having fakes nobody has classified, so that six months later no one can say whether a given result was about the robot or about a script somebody wrote in an afternoon and forgot to label.

• If you have no hardware and a demo this month -> fake everything, because plumbing bugs are what will break the demo, not perception quality.
• If you are training a policy on what a camera sees -> use recordings plus a simulator, because a policy learns whatever regularity a stub accidentally contains.
• If your robot mainly needs to avoid obstacles -> use a physics simulator with depth or laser scans, because geometry answers the question and rendering does not.
• If you own the sensor but not the finished robot -> put the device on a bench and record it, because a scrappy real recording outranks a careful guess.
• If your results depend on how the world reacts to the robot -> simulate, because playback stops being true the moment your machine chooses differently.

The HORUS Fit Framework reduces the surrounding stack decision to five axes: ecosystem size, setup effort, team size fit, deployment target, and licence. Score your options on all five, because how easily you can swap a stand-in for a driver is mostly decided by the stack underneath both of them.

If that swap keeps being the awkward part, and your processes are in different languages on one machine, put HORUS on your reading list: star it so it is in your list when you start building.