Best Robotics Datasets and Where They Come From

The best robotics dataset for most teams is a pooled one like Open X-Embodiment for pretraining, then your own recorded runs for everything that matters. Public data teaches a policy what the world roughly looks like; only your own recordings teach it your gripper and your floor. That flips when your robot is a common arm doing a common task, where borrowed data goes further than a month of collecting. The rest is for people training robot policies who must decide what to download, what to record, and where a stack like ROS 2 or HORUS fits.

You have a model architecture you like and a robot in front of you, and the thing standing between them is data you do not have. So you go looking. Within an hour you have a browser full of tabs: a pooled corpus assembled from dozens of labs, a teleoperation set from a university, a driving benchmark that keeps appearing in papers, and a hub page full of uploads with names that mean nothing to you.

None of those pages tell you what you need to know. They describe how many episodes they contain and what the licence is. What you want to know is whether any of it will make your arm pick up your objects on your bench, and nobody will say. The papers that used the data had different grippers, different cameras and, usually, a different definition of success.

Meanwhile the practical advice splits into two camps that do not talk to each other. One says pretraining on everything is what makes robots general. The other says the only data that ever helped was collected in the room where the robot works.

Which robotics dataset should you actually start with?

Start with a pooled cross-robot corpus if you are pretraining a general policy, and with your own recorded runs if you are trying to make one robot do one job well. Those are different projects that both get called robot learning. The pooled corpora — Open X-Embodiment being the obvious first stop — exist to give a model a broad prior: many arms, many workbenches, many ways of saying "pick up the cloth". They are excellent at that and poor at the last mile, because none of them contain your gripper closing on your objects under your ceiling lights. If your goal is a demonstration of generality, download and begin. If your goal is a robot that reliably clears a specific tray, pooled data is a warm start at best, and the work that decides whether the thing functions is collecting your own demonstrations. Most teams get this backwards, spending months on borrowed data because downloading feels like progress and teleoperating a robot for three days feels like admin.

What counts as a robotics dataset, in plain terms?

A robotics dataset is a recording of a machine acting in the world: what its sensors saw, what its controller commanded, and usually what a human was trying to achieve. That last part is what separates it from ordinary video. A file of camera frames is a perception dataset. A robotics dataset pairs those frames with the actions taken at the same moment, so a model can learn what to do rather than what things are. The pairing is the whole difficulty. Frames arrive from one device on one clock, joint readings from another, and the human's intent — "put the mug in the sink" — is usually written down afterwards by whoever ran the session. When these drift apart, a model learns to associate an image with an action that came from somewhere else, and it will do that quietly and confidently. Datasets also differ in what counts as an episode, whether failures were kept or deleted, and whether the robot was teleoperated or scripted. Two corpora described in identical words can be almost unusable together for those reasons alone.

Where do robotics datasets actually come from?

They come from five places, and knowing which one produced a file tells you more than its size does. The first is pooled academic collections, where many labs contribute recordings of different arms doing loosely comparable tasks and someone converts everything into one format; Open X-Embodiment is the reference example. The second is single-lab teleoperation corpora, where the hardware is consistent and the recordings are cleaner but narrower. The third is human video — egocentric footage of people cooking and assembling things — which contains no robot actions at all. The fourth is simulation, where you generate as much as you can afford and inherit whatever the physics engine gets wrong. The fifth is your own robot, and this is where middleware matters: ROS 2 records with its bag tooling and a large set of converters into training formats, while HORUS, an open-source real-time middleware for Rust, Python and C++ whose three languages share the same shared-memory ring buffers so messages between programs on one machine are not serialised, keeps the recording path and the control path identical under Apache-2.0. Neither is a dataset, a training framework or a model.

How do the main dataset sources compare?

They compare on what you can learn from them and what you must supply yourself, not on how large they are.

Most working projects use three of these rows at once: borrow broadly, generate what is unsafe to collect, and record the last mile yourself.

What goes wrong when you train on somebody else's robot data?

The usual failure is a policy that looks competent in evaluation and cannot touch a real object, caused by mismatches nobody wrote down. Action spaces are the first. One corpus records end-effector poses, another joint velocities, a third a gripper command that means "close" on one robot and "close by this much" on another. Converters paper over this and quietly lose the distinction. Camera placement is the second, and it is worse than it sounds: a policy trained on wrist views learns a completely different strategy from one trained on a shoulder view, and swapping them at test time produces confident nonsense. Control rate is the third — data recorded from a slow teleoperation session teaches a model to move in steps your smooth controller will never produce. Then there is the survivor problem: many published sets keep only successful episodes, so the model never sees a recovery and has no idea what to do the first time it misses. The symptom is always the same. Evaluation looks fine, the arm reaches for the object, and it closes on nothing.

Which dataset fits the kind of team you are?

A research group chasing generality should start pooled; a product team with one robot and a deadline should start with its own recordings. Those two answers cover most people and rarely convince anyone, because each side thinks the other is taking a shortcut. For the research group, breadth is the point: the interesting claim is that a policy trained across many machines transfers to a new one, and that claim needs the pooled corpora by definition. For the product team, breadth is a distraction, because the customer does not care whether the model also works on somebody else's arm. A third kind of team sits between them: a small company building something general enough to sell to several customers. That team should borrow for the prior and collect for each deployment, and should build the collecting apparatus early, because it becomes the thing that makes new sites cheap. Whatever category you are in, the tell is what you would do with a big improvement in your evaluation. If the answer is "write a paper", go pooled. If the answer is "ship", go local.

What hardware do you need to use these datasets?

To train on them you need a machine with a serious graphics card or rented time on one; to collect your own you need a robot, a teleoperation device and somewhere to put the files. Those are different shopping lists, and teams routinely buy the first while their bottleneck is the second. On the training side, the pooled corpora are large enough that storage and disk reads become the practical constraint before compute does, which surprises people who budgeted only for a card. On the collection side, what matters is boringly physical: a way for a person to drive the robot that stays comfortable for hours, cameras mounted so they do not move between sessions, and a recording setup that does not drop frames mid-demonstration. A tripod nudged on Tuesday quietly ruins Tuesday's data. If you are choosing where to spend, spend on making collection repeatable. Compute can be rented by the hour; a session recorded with a shifted camera cannot be recovered.

How long does it take to get from dataset to working robot?

Downloading and training something that moves takes days; getting that motion to work on your own bench takes months, and almost all the surprise is in the second part. The first phase is genuinely quick now. The pooled corpora ship in standard formats, the training recipes are published, and you can have a policy producing plausible actions over a weekend. Then comes the part nobody schedules. Your camera is not where their camera was, so you re-mount or re-train. Your gripper reports its state differently, so the action decoder needs work. The policy is confident and wrong in a way that is unsafe near a table edge, so you build guards. You collect your own demonstrations, discover a good share of them are unusable because teleoperation lag made the operator overshoot, and collect them again. Teams who have been through this describe the balance as inverted: the modelling was the short part. Plan the calendar around collection and evaluation, and treat the first training run as the cheap prologue it is.

What skills do you need to work with robot data?

You need someone who can read a data format carefully and someone who can run a robot patiently, and the second is rarer on machine learning teams. The format skill is unglamorous: opening a corpus, checking what each field actually contains, verifying that timestamps line up across sensors, and noticing that the gripper column is in different units from the one you assumed. Most costly mistakes here are caught by a person who looks at data, not by any tool. The robot skill is different in kind. Collecting good demonstrations is a craft — knowing to vary object placement without making it random, knowing when a session has gone stale because the operator is tired, knowing to record the failures. Teams staffed entirely with model people tend to produce datasets that are technically clean and behaviourally narrow, then blame the architecture. If you can add only one person, add the one who will spend afternoons driving the robot and writing down what happened. The training code is largely already written; the data judgement is not.

What do you give up by using a public dataset instead of your own?

You give up control over what the data contains, and with it the ability to fix a failure by collecting more of the right thing. That last point is the one that hurts. When your robot fails at a specific moment — the mug handle facing away, the cloth folded double — a self-collected pipeline lets you go and record more examples of exactly that on Thursday. With a public corpus you can only hope somebody happened to include it. You also give up knowledge of provenance. You will not know that a particular lab's arm had a sticky joint that month, or that a batch of episodes used a temporary camera mount, and those artefacts are learnable. There are licence questions too: several widely used corpora carry research-only terms, which companies discover late. What you gain is real — breadth no small team could produce, and a starting point that makes your own smaller collection go further. Just do not confuse the starting point with the finish, which is why data remains the real bottleneck.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever your data work depends on tooling that already exists, which for most dataset projects is nearly all of it. Recording, replaying, converting into training formats, visualising a run, checking a transform tree, pushing a recording back into a simulator — those are solved in the ROS ecosystem as nowhere else, and rebuilding any of them to gain something on the recording path is a poor trade. If your robot is a common research arm, if you hire from robotics rather than from web backgrounds, and if you expect to share data with ROS 2 users, use ROS 2 and stop thinking about it. HORUS is not the answer to that problem, and pretending otherwise costs a quarter. The narrow case where something else earns its setup is a machine whose control processes are written in different languages, run on one computer, and cannot afford the copying that crossing process boundaries costs — and even then the offline half of the pipeline will often still be ROS 2 shaped. The logging and replay tooling is the part worth surveying first.

Is more data always better for a robot policy?

No, and here is why: robot data carries provenance in a way text and images do not, and adding data from a different setup can pull a policy away from the one it has to work in. A model trained on many arms learns a compromise, and a compromise is exactly what you do not want when your job is one arm on one bench. Teams see this concretely: adding a large pooled corpus makes the policy look more general in evaluation and slightly worse at the task the company actually sells. The other failure is contamination. Data recorded while the gripper was miscalibrated, or during a week when the lighting rig was broken, teaches the model that the world is sometimes like that. Because nobody labelled those sessions, they are indistinguishable from good ones until someone goes looking. The useful discipline is to treat additions as experiments rather than accumulation: add a source, evaluate on your real task, and be willing to remove it.

Do human videos work as robot training data?

Partly, but not the way you think. Egocentric human video contains no actions a controller can execute — no joint commands, no gripper states, no forces — so nothing in it can be imitated directly. What it does contain is a very large amount of information about how objects behave when handled, what order tasks happen in, and what a person is likely doing next. Used as a pretraining source for the visual and intent parts of a system, that is genuinely valuable, and several strong results lean on it. Used as a substitute for robot demonstrations it fails, in a way that looks like success early: the model describes the scene well and reaches badly. The other honest caveat is embodiment. A human hand does things your two-fingered gripper cannot, so a model that learned human strategies will keep proposing them. Treat human video as a way to make perception smarter and your data requirements smaller, not as a way to avoid putting hours on the robot.

How do you decide which data to build on?

Ask what your robot fails at today, and let the failure choose the data. If it fails because it does not recognise objects it has never seen, you have a perception gap, and broad sources — pooled corpora and human video — are the right medicine. If it recognises everything and still fumbles the grasp, no amount of borrowed data will help, because the fumble is about your gripper, your objects and your controller, and only your own recordings contain those. If it works on the bench and fails in the customer's building, the gap is environmental, and the answer is collection at the customer's site rather than anything you can download. If you have no robot yet, use simulation and public data for the parts that do not depend on hardware, and expect to redo the rest once metal exists, which is where sim-to-real keeps failing. What separates teams who make progress from teams who accumulate downloads is whether they have evaluated on their real task this week.

Decide by situation rather than by ambition:

• If you are pretraining a general policy -> a pooled cross-robot corpus, because breadth is the claim you are making.
• If you are shipping one robot doing one job -> your own recorded demonstrations, because the last mile is in nobody else's data.
• If your robot does not exist yet -> simulation plus public data, because the alternative is waiting.
• If the failure is perception -> broad visual sources including human video, because the gap is about seeing, not acting.
• If the failure is contact and grasping -> collect on the real hardware, because your gripper's physics is written down nowhere else.
• If you cannot evaluate on your real task yet -> build that first, because every data decision after it is a guess.

When two of those lines describe you, weigh the tooling underneath on the five axes of the HORUS Fit Framework — ecosystem size, setup effort, team size fit, deployment target, and licence — and keep whichever loses on the fewest. No scores and no benchmarks, just five honest questions about your situation rather than the software. If your answers keep pointing at one machine, several languages, and recording that must not disturb the control loop it observes, star HORUS on GitHub so it is in your list when you start building.