What Is Manipulation, and Why Is It Harder Than Walking?

Manipulation is changing the world by touching it, and it is harder than walking because contact is unforgiving and every object is new. Walking repeats one motion against a floor that barely changes, so errors wash out; a grasp gets one attempt at an object nobody described. The condition that flips it is a fixed set of known objects, where a jig and a scripted arm beat a learned policy. The rest of this post is for humanoid and arm builders choosing where to start, and it says plainly where ROS 2 wins and where HORUS is merely plumbing.

The demo goes well. The robot walks the length of the lab, steps over a cable, turns, and comes back, and everybody in the room believes something has been achieved. Then somebody puts a mug on the table and asks it to pick the mug up.

What follows is painful to watch. The arm goes out, hesitates, adjusts, closes on the rim instead of the handle, and the mug spins away across the table. On the second attempt the fingers close on nothing at all. On the third the grasp succeeds and then the mug slips a little during the lift and lands on its side, and the robot carries on as if it were still holding something.

The gap is jarring because the walking looked so much more impressive. Walking involves an entire body balancing on one foot at a time; picking up a mug is what a toddler does. And yet the walking works every run and the picking works some of the time, on this table, with this mug, in this light. Nobody in the room can explain why in one sentence, and the honest answer takes rather more than one.

Is manipulation really harder than walking?

Yes, and the reason is structural rather than a matter of how much effort has gone in. Walking is a narrow problem with a closed definition. The goal never changes, the surface is roughly the same from step to step, the only body involved is the robot's own, and that body is described down to the last joint by the people who built it. Best of all, walking repeats: an imperfect step is corrected by the next one a moment later, so the machine is constantly given fresh chances. Manipulation has none of those properties. The goal changes with every task. The object was designed by somebody else, is only partly visible, and may be heavier, slipperier or floppier than it looks. There is no next step to fix the last one, because when a grasp fails the object is on the floor and the situation has changed. Walking is a hard problem that stays the same problem. Manipulation is a hard problem that becomes a different problem with each new thing on the table, and that is what makes it worse rather than merely harder.

What counts as manipulation in robotics?

Manipulation is any deliberate change a robot makes to the world through contact, which is a wider category than picking things up. Grasping and placing are the obvious members. So are opening a drawer, turning a handle, plugging in a connector, wiping a surface, folding a cloth, pushing a heavy box that could never be lifted, and handing an object to a person who is already reaching for it. The field splits these into prehensile work, where the object is held and moves because the hand moves, and non-prehensile work, where the object is nudged, slid or tipped without ever being gripped. Non-prehensile methods matter more than beginners expect, because a great many real objects are too large, too flat or too heavy to grasp. What does not count is an arm moving through empty space to positions somebody taught it. That is motion, and motion is a settled problem. Manipulation starts at the instant two surfaces touch, and everything difficult about it lives in that instant.

What goes wrong when a robot tries to pick something up?

Failures cluster into six stages, and knowing which stage yours is in matters more than any other diagnostic step. Perception fails when the robot sees a mug but not its handle, or invents the far side of an object it has only seen from one direction, or reports a transparent glass as empty air. Grasp choice fails when the chosen contact points are technically valid and practically hopeless, like the smooth curve of a rim. Approach fails when the wrist clips the table on the way in. Closing fails when the fingers meet with force enough to crush a paper cup or too little to hold a full one. Holding fails during transport, when a slow slip rotates the object out of the grip and nothing notices. Placing fails when the object is released a fraction above where the shelf actually is. Teams routinely tune the wrong stage for weeks because the video only shows the final drop. Watch the wrist camera, not the whole robot, and the stage names itself.

What are the actual options for building manipulation?

Seven approaches are in real use, and they are not competing versions of one idea. Taught positions with a fixture is the industrial answer: guarantee the object's location physically and the robot never has to decide anything. A classical pipeline chains perception, grasp selection, motion planning and joint control into stages you can inspect one at a time. Teleoperation puts a person in the loop and works today, at the cost of a person. Imitation learning trains a policy from recorded human demonstrations on the actual robot. Reinforcement learning trains in simulation and then fights to carry the result onto hardware. Vision-language-action models aim at general behaviour across many loosely described tasks. And changing the object or the gripper — a suction cup, an added handle, a printed tag — solves more real problems than any of the above.

Most of those routes are built, not bought, but the surrounding software is not. ROS 2 carries the maintained manipulation ecosystem: motion planning, collision checking, gripper drivers, the coordinate bookkeeping that keeps a wrist camera and an elbow joint talking about the same world, and the viewer that shows what the robot believed at the moment it failed. A middleware such as HORUS — open source under Apache-2.0, with Rust, Python and C++ sharing the same shared-memory ring buffers so nothing is serialised between processes on one machine — sits underneath rather than beside that. It moves camera frames and joint commands between programs, and it plans no motions and grasps nothing.

How do the manipulation options compare?

Read the third column first, because a method your team cannot debug on a bad afternoon is not available to you whatever the papers say. Then strike out every row whose last column matches your situation. Most teams end up running two rows at once rather than one: a scripted, fixtured version that works today and pays the bills, and a learned version being trained on data the first one produces. That pairing is healthier than betting the product on either half, and it is roughly what the better humanoid programmes actually do behind the demo videos.

Which approach fits a small team building a humanoid?

Start with the fixtured version of your task, get it working end to end, and use it to collect data for whatever comes next. This is unpopular advice on a humanoid programme because it feels like a step backwards from the ambition, and it is the fastest route to a robot that does something on a Tuesday. A small team that begins with learned policies typically spends its first months discovering that the hardware drops connections, the wrist camera is mounted a little off, the gripper does not report when it has stalled, and the recordings cannot be lined up in time. Every one of those problems has to be fixed for the learned version to work anyway, and they are far cheaper to find while the task is scripted and the failures are legible. The second reason is data. Demonstrations gathered on a rig that works are worth more than demonstrations gathered on a rig that half works, and a team of a few people cannot afford to collect the second kind twice.

What does your hardware decide for you?

Your hardware has already removed most of the options, and gripper choice removes the most. A two-fingered parallel gripper is easy to control and can hold a mug by its handle, if the handle is found. A suction cup makes a whole class of tasks trivial and is useless on anything porous, curved or dusty. A multi-fingered hand opens up in-hand work and multiplies everything that can be tuned wrongly. Next comes whether the joints can be pushed. An arm that resists all movement will happily press an object through a table it thinks is elsewhere, whereas a backdrivable arm gives way and turns a serious collision into a nudge. If the joints also sense the force they are applying, insertion and assembly become approachable; without that sensing, plan on grasping from above and avoid anything that must be pushed into place. Compliance bought in the hardware saves you from needing it in software, and software compliance demands both fast sensing and a control loop that never hesitates, which is part of why humanoids still move slowly.

How long does it take to get a working pick-and-place?

Days for one object in one place on one table, and considerably longer for anything a stranger might put in front of the robot. The first version is genuinely quick: fixture the object, teach the poses, close the gripper, and there is a video by the end of the week. The distance between that and a robot that picks up whatever is on the counter is where whole company roadmaps go. Each relaxation costs its own stretch of work. Letting the object sit anywhere on the table adds perception. Letting it be any of a handful of objects adds grasp selection. Letting the lighting change adds a fight with the camera. Letting a person stand nearby adds safety behaviour, which is a project of its own. Plan your milestones as a list of restrictions you are removing one at a time, and never as a single leap to general grasping, because the leap has no intermediate state you can demonstrate to anybody.

What skills does a team need before starting manipulation?

Three, and only one of them is what people expect. The obvious skill is a working knowledge of grasping and motion planning, and it matters less than the other two because much of that work already exists in maintained packages. The first underrated skill is debugging geometry and timing: knowing where every camera and joint sits relative to every other, and whether an image and a joint reading that arrived together were actually taken together. Nearly every mysterious manipulation failure is one of those two dressed up as something else, and the same discipline decides whether a mobile robot knows where it is. The second underrated skill is running experiments patiently. Manipulation improves by attempting the same task over and over, recording every attempt including the ugly ones, and being honest about what fraction of the failures share a cause. Teams without somebody who enjoys that work tend to tune by anecdote, fix the failure they saw most recently, and never notice the one that quietly dominates.

What do you give up by choosing one approach?

Each route trades away something you will miss later, so pick the loss you can live with. Fixtures give up generality completely: the station works and it works only there, and the day a customer changes their packaging you rebuild it. A classical pipeline gives up graceful behaviour in messy situations, because every stage assumes the previous one was right, and it repays you with the ability to say exactly which stage broke. Teleoperation gives up the economics, since a person per robot rarely survives contact with a business plan, though it buys real deployment and real data now. Imitation learning gives up explainability; when a policy misses, there is no stage to inspect, only more data to gather. Reinforcement learning in simulation gives up trust in your results, since the simulator is most approximate exactly where contact decides the outcome, which is the argument running through the simulator comparisons for manipulation research. Foundation models give up predictability in exchange for breadth. There is no route that keeps all of it.

When is ROS 2 the better choice?

ROS 2 is the better choice for most manipulation work, and this is not a close call. Motion planning around obstacles, collision checking against the robot's own body, drivers for commercial arms and grippers, the bookkeeping that keeps a wrist camera and a shoulder joint describing the same world, and a viewer that draws what the robot believed at the moment of failure are all years of specialist effort that already exist and are maintained. If your arm is a commercial one, its supported driver is very likely a ROS 2 package, and reimplementing it is not the project you meant to start. If you need to plan a path that avoids the robot's own torso, take that from ROS 2 rather than deriving it. If two engineers must share the work, ROS 2 vocabulary means they already share a language. HORUS is not the answer to any of this, because a messaging layer moves data and plans nothing, and the same holds for most of the software half of a humanoid.

Does a better gripper solve most grasping failures?

No, and here is why: the gripper is the last stage in a chain, and most failures were decided before the fingers moved. A hand with more fingers and better sensing still closes wherever the software told it to close, and if the perception stage put the mug's handle a finger's width from where the handle is, a superior gripper simply misses more precisely. The reason the belief is so common is that gripper upgrades do produce a visible improvement, because forgiving hardware papers over small errors everywhere upstream. That improvement is real and it is also a ceiling: it makes the current failure rarer without removing its cause, and the cause reappears the moment the object changes. Before buying anything, run the diagnostic. Record a long run of attempts, note the stage each one failed at, and count. If most failures are perception or grasp choice, no gripper will save you. If most are slips during the lift, then better contact sensing is genuinely the right purchase.

Have foundation models made manipulation easy?

Partly, but not the way you think. Models trained across many robots and many tasks have made something genuinely new possible: a robot that attempts a task nobody wrote code for, described in ordinary words, and sometimes gets it. That was not available a few years ago and it is not a marketing claim. What has not changed is the last stretch of contact. These models are good at deciding what to do and much weaker at the moment of touching, which is where friction, weight and deformation take over, and they fail without warning rather than reporting doubt. They are also slower than the loop that has to decide what the fingers do next, so a lower layer is always holding the object while the model thinks, and that lower layer is ordinary control code. The practical shape of a modern system is therefore a model choosing the task and a classical controller executing the contact, which is roughly the argument in why a language model can write an app but cannot pick up a cup.

How do you decide where to start with manipulation?

Write down the exact set of objects the robot must handle, then be ruthless about how big that set really is. If it has one member and you control where it sits, the answer is a fixture and taught positions, and no amount of ambition should talk you out of it. If it has a few members and they are rigid, build the classical pipeline on ROS 2, because every stage is inspectable and the parts exist. If the set is genuinely open, you are in learning territory and your real first problem is data collection, not the policy. Then ask the shape questions. Can a human demonstrate this task easily, because that decides whether imitation is even available? Does success depend on friction and deformation, because that decides whether simulation will lie to you? Must nearly every attempt succeed, because that rules out anything that fails without warning? And how many programs, in how many languages, must share the same camera frames and joint readings on one machine?

Decide by situation rather than by principle:

• If the object arrives in the same place every time -> a fixture and taught positions, because there is no decision left to make.
• If you handle a handful of rigid objects -> a classical pipeline on ROS 2, because each stage can be inspected when it breaks.
• If the task must work for customers this quarter -> teleoperation, because a person in the loop beats a policy that nearly works.
• If a person can easily demonstrate the task by hand -> imitation learning, because demonstrations are the cheapest data you will ever get.
• If vision code in Python and joint control in C++ or Rust must share one computer -> a shared-memory middleware, because the language boundary is the part that costs you.

When two of those lines describe you, weigh them on the five axes of the HORUS Fit Framework — ecosystem size, setup effort, team size fit, deployment target and licence — and keep whichever loses on the fewest. No score and no numbers: five honest questions about your situation rather than about the software. If your answers keep landing on one machine, several languages, and a control loop that cannot wait for a model to finish thinking, star HORUS on GitHub so it is in your list when you start building.