Why Robots Need So Much Data, and Why That's the Real Bottleneck

Robots need vast data because the world never repeats, and the bottleneck is the pipeline that records it, whether on ROS 2 or HORUS. Models improve with more varied experience, and every hour of that experience must be captured, moved and stored while a robot is trying to stay upright. The verdict flips if your robot is a single arm collecting slowly on a bench, where any pipeline copes. The rest of this post is for someone from machine learning who has trained models before and is now discovering where robot data actually comes from.

You have done this before. You know how to get a model to learn something, you know what a training curve looks like when it is going well, and you know the feeling of a dataset that is finally large enough to be interesting.

So the robot version seemed like it would rhyme. Get a policy, get some episodes, train, iterate.

Then you spent three weeks with a controller in your hands teleoperating an arm, and what you produced would fit in the corner of a text corpus you would once have called small. Half the episodes turned out to be unusable, because the gripper drifted, or the reset was wrong, or the camera and the joint readings do not line up and you cannot tell which frame goes with which action.

Meanwhile the recording script you wrote started taking time away from the thing driving the arm, so the arm moves less smoothly during collection than during evaluation, which means your training data describes a robot you are not going to deploy.

And you are beginning to suspect that the model was never the hard part.

Why do robots need so much data compared with language models?

Because a robot has to learn a mapping the world will not hand over cheaply. Text and images arrive pre-collected in enormous quantity because humans spent decades producing them for their own reasons. Robot experience does not exist until a robot produces it, in real time, at the speed of physics, one attempt at a time, usually with a person nearby to reset the scene.

The data is also narrower than it looks. An hour of a robot picking up mugs teaches a policy about that lighting, that table height, that gripper and those mugs. Move the lamp and a surprising amount of what was learnt stops applying. Language has the same problem in principle and hides it, because the corpus is wide enough that most variations appear somewhere. Robot corpora are not wide.

Then there is the shape that makes counting difficult. A robot's inputs are enormous and its outputs are tiny: every moment carries several camera streams, joint readings and force signals, and the answer is a handful of numbers describing where to move next. What a robot foundation model is covers where this is heading.

What counts as robot data in the first place?

Robot data is the whole synchronised bundle, not the camera feed. A usable episode contains what the robot saw, what the robot felt, what the robot did, and when each of those happened relative to the others, tied together tightly enough that a learning algorithm can say this action followed this observation.

That last clause is where homemade datasets fall apart. A camera frame with no trustworthy notion of when it was taken, sitting beside a joint reading with a different notion, produces episodes where the action is attributed to the wrong moment. A policy trained on that learns a slightly shifted version of the task and behaves strangely in a way that looks exactly like a model problem.

There is also data nobody thinks to keep and everybody later wants: the failures, the resets, the operator's aborted attempts, the moment the object slipped. Successful episodes teach a policy what to do. The other kind teaches what recovery looks like, which is most of what separates a demo from something that works in a kitchen.

What goes wrong when a team starts collecting robot data seriously?

The recording starts competing with the robot. This surprises people from a machine learning background most, because in every previous setting the data was already sitting still in a bucket. On a robot, the program writing your dataset runs on the same board as the program keeping the robot upright, and both want the same cores at the same moment.

The symptoms are consistent. Frames go missing from one stream and not another, so episodes have gaps that appear only when the machine is busy. The control loop becomes less smooth during collection than during evaluation, so the data describes a robot that is subtly not the one you will deploy. A viewer opened on a laptop changes the robot's behaviour, which sounds absurd and is simply the cost of delivering the same pictures to one more consumer.

The second failure is quieter. Storage fills, someone lowers the resolution or the rate to cope, and the dataset silently becomes two datasets with different properties that nobody labels. Why sensor data goes missing covers the diagnosis.

What are your actual options for producing and keeping robot data?

There are seven, and they split into where the data comes from and what carries it. On the source side: public robot datasets, which give breadth immediately and rarely match your gripper or your camera; simulation at scale, which gives unlimited attempts and approximate contact; teleoperated demonstrations, which give exactly your task at the speed a human can perform it; and scripted collection on real hardware, which runs unattended when the scene can be reset automatically.

On the carrying side, the question is what moves those streams around the robot and writes them down. ROS 2 is the usual answer, with recording tools that most published robot datasets already assume. A shared-memory foundation is the other answer, which is where HORUS sits: an open-source real-time robotics middleware for Rust, Python and C++ where all three languages share the same shared-memory ring buffers, so several programs on one machine see the same camera frames without each being handed a separate copy, published under Apache-2.0, validated in simulation, and shipping no navigation or mapping stack of its own.

The seventh option is to buy the data, or partner with someone who already has it.

How do these options compare side by side?

Read the last column first. Most teams can strike three rows immediately, and the real argument is between the two that survive.

Are you a researcher, a startup, or a lab with a fleet?

A researcher chasing a method should borrow data rather than produce it. Public datasets exist so a new algorithm can be compared against others on the same footing, and building your own collection rig is a term of work that proves nothing about your idea. Produce data only when the claim you are making is about data the field does not have.

A startup building a product is in the opposite position. The data that matters comes from your robot, in your customers' environments, doing your task, and no public dataset substitutes for it. This is the group for whom the collection pipeline is the product's foundation rather than a support function, and the group most likely to discover that halfway through.

A lab or company with several robots running has the hardest and best version of the problem. Several robots produce enough data that the pipeline becomes a real engineering system, with storage, indexing and quality checks. The upside is that a fleet collects while people sleep, which is the only known way past what a few operators can produce by hand.

What hardware is doing the recording?

Whatever board is already running the robot, which is the root of the problem. Teams imagine a recording machine and end up with a recording process sharing a compute board with perception, control and everything else, on hardware that was specified for the robot rather than for the dataset.

Camera count decides most of what follows. One camera at a modest resolution is a stream almost any arrangement can write down. Several cameras, especially with depth, means the same large frames are wanted by the policy, the recorder and often a viewer at once. That is the situation where the number of programs reading a stream starts changing how the robot behaves, which is the signature of delivery cost scaling with consumers rather than of a slow disk.

Storage and offload are the unglamorous half. A robot that collects all day and then needs its data moved somewhere has an offload problem nobody plans for, and it limits collection long before anything else does. Decide early whether the robot writes locally and syncs later, or streams as it goes.

How soon do you need a usable dataset?

With weeks, borrow and simulate. Take a public dataset close enough to your setup to sanity-check your training code, generate the rest in simulation, and get the whole pipeline working end to end on data you did not have to collect. Discovering that your training loop has a bug is much cheaper before you have spent a month teleoperating.

With a few months, collect deliberately and narrowly. Pick one task, one environment and one robot configuration, and make the rig good enough that an operator produces clean episodes without thinking about the tooling. Breadth comes later, and a small clean dataset on one task beats a large messy one across five.

With a year, build the pipeline as a system: automatic resets, checks that reject bad episodes before a human sees them, an index that finds the episodes where the object slipped, and enough robots that collection runs overnight. This is also where the foundation carrying the data stops being an implementation detail, and why middleware decisions are so hard to reverse becomes worth reading first.

What does your team already know how to do?

A team fluent in machine learning and new to robots will underestimate the plumbing and overestimate the model, reliably and in that order. The instinct from every previous domain is that data acquisition is somebody else's problem and the interesting work starts once the tensors exist. On a robot the acquisition is the work, and it is systems work: processes, timing, storage, and hardware that fails in the afternoon heat.

A team of robotics engineers with no learning background makes the mirror mistake. The rig is beautiful, the episodes are clean, and nobody asked whether the variation in the dataset resembles the variation the policy will actually meet.

The pairing that works has one person who cares about episode quality and has the authority to reject data. Not to collect it, to reject it. Most datasets that fail to train anything useful were assembled by people who were measured on volume. How to connect a language model to a real robot covers the adjacent version of this gap.

What do you give up by making data collection the priority?

You give up visible progress on everything else, and that is the trade stated honestly. A team that takes collection seriously spends months on rigs, resets, synchronisation and storage, and during those months the model does not improve and the demo does not get more impressive. That is uncomfortable to justify to anyone watching from outside, which is why many teams skip it and stay stuck.

You give up flexibility in the task. A pipeline built around one gripper, one camera arrangement and one table height resists change, and altering any of those invalidates data you already paid for. Teams learn to freeze the hardware earlier than feels comfortable.

You also give up the comforting story that a better model is one experiment away. Once the pipeline is good, the honest constraint becomes visible: your robot can only try things at the speed a real arm moves, and no amount of engineering makes the physical world hurry.

When is ROS 2 the better choice?

ROS 2 is the better choice for most teams collecting robot data today, and HORUS is not the answer when recording tooling is what you actually need. ROS 2 has a recording format that a large share of published robot datasets already use, tools that replay a recording as though the robot were live, and visualisers that let you scrub through an episode and see what the robot saw. Rebuilding that is months of work teaching you nothing about your task.

Use ROS 2 when your sensors ship with vendor drivers written for it, which is most sensors. Use ROS 2 when your rig spans a robot and a workstation, because that link is a network link and shared memory ends at one machine. Use ROS 2 when you want a dataset other people can read, because a format the field already reads is worth more than one that is convenient for you.

And use ROS 2 when the streams are modest. The middleware comparison for on-board models covers where that changes.

Will a larger model make the data problem go away?

No, and here is why: a larger model changes what you can extract from data you already have, and does nothing about the fact that your data describes one table, one gripper and one set of objects. Scale has been generous in language and vision because the corpora were already wide enough to contain the variation that mattered. A robot dataset collected in one room does not contain that variation at any model size, and a bigger network trained on it simply learns the room more thoroughly.

There is a real version of the scaling argument, and it is about breadth rather than size. Pooling episodes across many robots, many labs and many tasks produces something genuinely wide, and models trained on those pools transfer to new setups in a way that models trained on one team's data do not. That is a data result wearing a model result's clothes.

The consequence for a small team is unglamorous. Your gains come from more varied episodes and cleaner labels. Why a language model can write an app but not pick up a cup is the longer argument.

Does simulation solve the data bottleneck?

Partly, but not the way you think: simulation removes the constraint on quantity and leaves the constraint on relevance exactly where it was. You can run many robots in parallel, overnight, with automatic resets and perfect labels, and that genuinely changes what is possible for locomotion, navigation and anything where the physics you care about is physics a simulator handles well.

Contact is where the promise thins. Friction, deformation, slip, cloth, granular material and the exact instant a finger loses grip are the parts simulators approximate, and they are also the parts manipulation policies lean on hardest. A policy trained purely in simulation often behaves confidently and wrongly on a real object, which is the most expensive failure because it looks like success right up until contact.

The arrangement that works is not either-or. Simulate for coverage, use a smaller amount of real data to correct what the simulator got wrong, and design the task so the parts you cannot simulate are the parts you collect. What sim-to-real is and why it keeps failing is the honest treatment.

How do you tell which bottleneck you actually have?

Run the substitution test. Hand your training pipeline a public dataset close to your task and see what comes out. If borrowed data trains a better policy than yours does, your collection is the problem and no model work rescues it. If your data trains something respectable and the policy still fails on the real robot, the gap is between your data and your deployment conditions, which is a variation problem rather than a volume one.

The second test is a stopwatch on your own team rather than on the robot. Count where the weeks actually went last month. Teams convinced they have a modelling problem usually find most of the month went into rigs, resets, file wrangling and chasing episodes that turned out to be corrupt.

The third is the consumer test. Open a viewer during collection and watch whether the robot's motion changes. If one more reader of the same stream changes how the robot moves, the plumbing sits inside your control loop, and that is fixable in a way a data shortage is not.

Here is the whole decision in five lines.

• If you are testing a method rather than a product -> borrow a public dataset, because a collection rig proves nothing about your idea.
• If your task turns on contact and slip -> collect real episodes, because that is exactly what a simulator approximates.
• If your policy trains well and fails on the robot -> collect more varied conditions, because the gap is variation rather than volume.
• If recording changes how the robot moves -> fix the plumbing first, because your data describes a robot you will not deploy.
• If nobody is allowed to reject an episode -> fix that before anything else, because volume without quality trains nothing.

The HORUS Fit Framework compresses the foundation choice onto five axes you can score any option against: ecosystem size, setup effort, team size fit, deployment target, and licence. For data work, ecosystem size and deployment target do most of the work, because recording tools and dataset formats live in the ecosystem.

When your collection rig outgrows the board it runs on, the useful thing is having the alternative already on your shelf. Put HORUS there now: star it so it is in your list when you start building.