Teleoperation vs Autonomy: What You're Actually Watching in Robot Demos

Most striking robot demos are teleoperated or scripted, and for almost every builder teleoperation is also the right thing to build first. A person supplies the judgement that no model supplies dependably yet, so a teleoperated robot does useful work the week you finish it. The verdict flips when a task repeats more often than a person can sit through, and that call is about the task, not about whether ROS 2 or HORUS runs underneath. The rest of this post is for someone watching those videos and trying to work out what to build on hardware they can afford.

You watched the clip twice. A humanoid folds a shirt, lifts a mug off a crowded counter without touching the one beside it, hands it over and steps back. The comments argue about whether someone was standing off-camera in a headset. You cannot tell, and not being able to tell bothers you more than either answer would, because you are trying to work out what is genuinely possible this year outside a research lab.

Then you try it yourself. Your arm reaches for the mug and closes on nothing, because the mug was not quite where the camera said it was. Or it works four times and on the fifth it sets the mug down half on the edge. Nothing crashes. There is no error to search for. Every video you have seen makes the next step look like a small one, and every attempt of your own ends somewhere no video has ever shown you. So you start to wonder what is being left out of the frame: a person with a controller, a hundred earlier takes, a strip of tape on the floor, or a table that never changes.

Should you build a teleoperated robot or an autonomous one?

Build teleoperation first and add autonomy one task at a time, because teleoperation is the version of the robot that does something useful the week it exists. A person driving a machine gets you a working system immediately, teaches you what the task actually involves, and produces recordings you will want later. Autonomy is not a upgrade you flip on; it is a separate build for each task, and each one only pays when that task will be repeated far more often than a person can be spared to sit through. The mistake that costs the most months is starting at the other end: choosing autonomy for a task nobody has yet performed manually, then discovering the hard part was never the software. The gripper does not hold that shape of object. The camera cannot see the near edge. The task you wrote down is not the task people actually do. Teleoperation surfaces all of that in days, and it does so with the operator as the safety net rather than a stack of code you have not written yet.

What do teleoperation and autonomy actually mean?

Teleoperation means a person supplies the decisions and the robot supplies the body; autonomy means the robot supplies both. Between those two there is a wide middle that most working machines actually live in, and the names for it are worth knowing because demos rarely say which one you are looking at. Direct teleoperation maps a person's motion onto the robot's joints, usually through a headset, a glove or a smaller copy of the arm they hold in their hands. Waypoint teleoperation is coarser: the operator points at a place or an object and the machine works out how to get there. Scripted playback is not teleoperation at all — a motion was recorded once and is replayed with nothing watching, which looks flawless in a controlled scene and falls apart the moment the scene changes. Supervised autonomy is the robot attempting the whole task and handing control back when it fails or is unsure. And shared control mixes the two continuously: the human aims, the machine handles the approach, the alignment, and the moment of contact.

What are your actual options for deciding who drives the robot?

There are eight practical setups, and the choice is really about who is holding the difficult part of the task. You can drive the robot directly through the maker's own software, which is the fastest route to motion this week. You can build a teleoperation rig of your own, with a headset or a leader arm. You can record and replay a motion. You can add shared control so the operator points and the machine executes. You can build supervised autonomy that asks for help. For the software underneath, ROS 2 is the right base when you need mapping, navigation, arm planning or borrowed drivers, and HORUS is the alternative when the robot is one machine mixing languages — an open-source real-time robotics middleware for Rust, Python and C++ where all three share the same shared-memory ring buffers, so messages between processes on one machine are not serialised, which is what a camera feed passing between a Python operator interface and a compiled safety loop needs. Read the table as a description of situations rather than a ranking.

How can you tell whether a demo is teleoperated?

You can usually tell from the rhythm, because human hesitation has a shape that a policy does not. Watch for a pause that lands right before a decision rather than right before a movement — the operator is thinking, and the robot waits with them. Watch for recovery: a machine that fumbles a grasp, adjusts and succeeds is either teleoperated or genuinely capable, and genuine capability is rare enough that the first guess is usually right. Then look at what the video does not show. A single continuous take from one angle is a stronger claim than a cut every few seconds. Footage sped up hides the pauses that would have told you. A scene with objects in the same places every attempt is a scene that has been arranged. Captions matter too, and the honest ones increasingly say teleoperated in small text at the start. None of this makes a demo dishonest. It just means the video is showing you the body, and the interesting question is who is supplying the decisions.

What should you build if you are one person or a small team?

Build teleoperation, and treat every autonomous behaviour as a separate project you must justify. One person can get a machine doing real work through direct control in a matter of days, and that working machine is what keeps a project alive: you can show it, you can use it, you can learn from what it fails at. Autonomy for one task, done properly, tends to consume the sort of stretch of time that small teams do not have spare — not because the algorithm is hard, but because the tail is long. The robot handles the ordinary case quickly and then you spend the rest of your time on the cup that is upside down, the light that changed, the door someone left half open. A realistic sequence for a small team is: drive it manually, notice which motions you make identically every time, automate exactly those, keep the person for everything else. That order also protects you from the quieter failure described in why robotics projects stall after the prototype — a stack of half-finished autonomy and no machine anyone can use.

What does your hardware decide for you?

Your hardware decides how much autonomy is even reachable, and it decides it mostly through sensing and through the hand. A robot that cannot see the near edge of an object cannot place that object autonomously, no matter what software you put above it. An arm with no sense of the force it is applying can be driven by a person who watches and eases off, and cannot be trusted alone with anything that deforms or resists. A gripper with two flat fingers will hold a mug and lose a folded shirt, and a demo of the shirt is telling you about the hand rather than the intelligence. Compute matters less than people assume for teleoperation and more than they assume for autonomy: the operator's judgement runs on the operator, whereas an autonomous version needs perception, planning and control sharing one board without any of them making the timed part wait. That last constraint is where the on-robot software choice actually bites, and it is a different question from which language you write in, covered in why Python is fine for robotics until it suddenly isn't.

What if the robot has to work in a month?

Teleoperate it, with no autonomous behaviours at all beyond stopping safely. A month is enough to get a person driving a machine through a real task, to find out that the task involves three things nobody wrote down, and to fix two of them. A month is not enough to build a perception pipeline, a planner and a failure detector for a task you have not yet performed manually. Deadlines punish unfinished autonomy worse than they punish an honest human in the loop, because half-built autonomy produces a robot that works in rehearsal and fails in front of the audience, whereas a teleoperated robot fails the way a person fails, which is recoverable. If the demo must look autonomous, the honest compromises are a scripted sequence in a scene you control, or shared control where the operator's input is small enough to be invisible. Both are common and both are fine, provided you know which one you are doing and provided nobody in the room believes they are watching a machine that thinks.

What if you have never built a robot before?

Then start with teleoperation, because it is the only setup where the robot teaches you and nothing has to work perfectly for you to learn. Driving a machine by hand gives you an immediate, physical education in the things that are missing from every tutorial: that a camera mounted where you can see it is not mounted where the arm needs it, that grippers slip, that a robot which knows its joint angles still does not know where the table is, that everything takes longer when the object is soft. You will also discover which parts of the task are boring — and boredom, precisely measured by your own patience, is the best signal you will get about what to automate first. The trap for a beginner is skipping this and starting with a navigation stack or a learned policy, where nothing moves for weeks and every failure has ten possible causes. A robot you can drive badly today beats a robot that might drive itself next quarter. If you want the wider ordering, the beginner's map for programming a humanoid walks through the same progression.

What does it look like when autonomy fails on a real robot?

Autonomy fails quietly and confidently, which is what makes it unnerving to watch. The robot does not stop and report that it is confused. It reaches for a place where the object used to be and closes on air, then continues to the next step as though it is holding something. It sets a plate down while the plate is still half over the edge. It repeats an approach four times with the same result, because nothing in the program has any concept of trying differently. And it does the whole task perfectly for a week and then fails on a morning when the sun comes through a different window. The pattern behind all of these is the same: the machine acts on a belief about the world and has no way of noticing that the belief is wrong. Teleoperation does not have this failure, because the person notices immediately and stops. That single property — a human who sees that something is off — is the largest part of what you are actually buying, and it is the last thing autonomy manages to replace.

What do you give up by keeping a person in the loop?

You give up the ability to run more machines than you have people, which is usually the entire economic point. A teleoperated robot costs an operator's attention for every hour of work, so the fleet cannot grow past the payroll, and the operator's attention is not evenly good — hour six is not hour one. You also give up working where the network is poor, because remote control depends on a link that does not stall, and when it stalls the operator's corrections describe a world the robot has already left. You give up consistency: two operators will do the same task differently and neither will be repeatable enough to reason about. And you give up a certain kind of speed, since a machine driven by a person moves at a pace the person can follow. What you keep is judgement, recovery, and someone who will notice when the situation is not the situation anyone planned for. Most builders trade too much of that away too early, which is how a working robot becomes a demonstration that only runs on Tuesdays.

When is ROS 2 the better choice?

ROS 2 is the better choice the moment the robot needs to know where it is in a building. Mapping, localisation and navigation represent an enormous amount of accumulated work, and a small team that decides to rebuild them will spend a year learning why nobody does that. The same applies to arm motion planning around obstacles, and to any sensor whose only driver is a ROS 2 package — porting a driver to avoid a dependency is almost always a bad trade. ROS 2 also wins when the system spans several computers, when new people have to read the robot without a guided tour, and when a research group already shares the vocabulary and the tooling. In every one of those situations HORUS is not the answer, and picking it means rebuilding plumbing you could have inherited. Note also that this is genuinely independent of the teleoperation question: a ROS 2 robot can be entirely teleoperated, and a robot on any other base can be entirely autonomous. The framework decides what you inherit, not who is driving.

Are teleoperated robot demos dishonest?

No, and here is why: teleoperation is a real capability, and the hard parts it demonstrates are hard whether or not a person is deciding. A machine that can be driven to fold a shirt has a hand that can hold fabric, an arm that can move gently enough not to tear it, and a control system that keeps up with a human's corrections without lag or overshoot. None of that is faked, and none of it is easy. What is misleading is the framing, and the framing is often supplied by the audience rather than the company: a video shows a body doing something, and viewers supply a mind behind it. The dishonest version is narrower — implying that no one is driving, or letting a claim about the future stand in for a claim about the clip. As a builder, the useful response is not cynicism but a habit: when you see a capability, ask which half it demonstrates. Then ask whether you would need the other half at all, which for most products is a genuine question rather than a rhetorical one.

Will better models make teleoperation obsolete?

Partly, but not the way you think. Models will keep taking over the middle of tasks — the reaching, the aligning, the grasping of things that resemble things they have seen — and that part is already moving quickly. What they do not remove is the operator's other job, which is deciding what should happen at all when the situation is not the one anyone anticipated. A machine that folds shirts well still needs someone to decide that today the pile also contains a hoodie with a zip, and that the zip belongs on the outside. More to the point, teleoperation is how the models get built: demonstrations recorded by human operators are the raw material, so the better the models get, the more teleoperation gets done, not less. The realistic direction is not autonomy replacing teleoperation but the ratio shifting — one person supervising several machines instead of driving one, stepping in for the small share of attempts that go sideways. Build for that shape and you are building for both worlds. Build assuming full autonomy arrives before your product does, and you have made a bet on a schedule nobody controls.

How do you decide which one your robot needs?

Count how many times the task will be performed, and by whom, before you write a line of autonomy. A task done a few times a week does not repay an autonomous version and probably never will. A task done constantly, by a bored person, in a scene that stays roughly the same, is what autonomy is for — and even then the sensible build keeps the operator present. The second question is whether the robot can tell when it has failed, because supervised autonomy without failure detection is just autonomy with extra confidence. The third is whether you have ever done the task manually. If you have not, you do not yet know what the task is, and the code you write will be for an imagined version of it. The same reasoning appears from the model side in why an LLM can write an app but cannot pick up a cup.

Decide by situation rather than by ambition:

• If you have never performed the task by hand -> teleoperation, because you do not yet know what you are automating.
• If one motion is identical every single time -> automate that motion only, because the repeatable part is the cheap part.
• If the robot must cross a building or plan around obstacles -> ROS 2, because mapping and navigation are the job.
• If the task runs all day and a person is bored doing it -> supervised autonomy with handover, because boredom is the signal.
• If an operator interface, a vision process and a timed loop share one board -> a shared-memory middleware, because the data crossing between them is the constraint.
• If the network between operator and robot is unreliable -> put the reflexes on the machine, because a stalled link must not mean a moving robot.

When two options stay close, weigh them on the five axes of the HORUS Fit Framework — ecosystem size, setup effort, team size fit, deployment target, and licence — and take the one that loses on the fewest. No scores, no numbers: five honest questions about your situation rather than about the software. If your robot keeps landing on one machine with an operator above and something compiled below, star HORUS on GitHub so it is in your list when you start building.