Why Your LLM Can Write an App But Can't Pick Up a Cup

Your model cannot pick up a cup because the gap is not intelligence but a body and control software beneath it, ROS 2 or HORUS. Code is text a person reads before it runs, while a cup is physics that continues while the model thinks, so timing belongs to code you wrote. That verdict only flips when the thing being moved cannot hurt anything, such as a simulated arm. The rest of this post is for someone fluent in models who now has a machine in front of them and no idea what to build underneath.

You have shipped real software with a model in the loop. It writes the endpoint, the migration and the tests, you read the diff, and when it invents an API that does not exist you find out before a single user notices. Then someone puts an arm on your desk, or you buy the cheap one with the gripper, and the same model that refactors a service in a minute cannot get a cup off a table. It reaches past the cup. It closes on nothing. It closes on the cup and keeps closing. You go back to the prompt, because the prompt is the lever you know, and a better prompt produces the same confident nonsense in nicer words. You have watched the videos where somebody says clean up this table and an arm does it, so you know this is possible, which makes your own failure feel like a skill issue rather than a category error. Nobody in those demonstrations explains what was actually running underneath, and the repositories you find either name a model you cannot run or skip straight past the layer you are missing.

Should a language model be in charge of the arm, or sit above code that already moves it?

Above, always, and this single decision is what makes the rest of the project tractable. There are two jobs in a robot and they look alike only from a distance. One job repeats without pause: read where the arm is, compare that with where it should be, push it a little closer, and refuse anything that would take it through the table. The other job decides what should happen next — the cup goes in the sink, so grasp it, lift it, move over there, release. A model is genuinely good at the second job, including the parts that are tedious to code by hand, like working out that "the blue one" means the mug, or that a drawer opens before anything goes into it. It is structurally unsuited to the first, not because it is insufficiently clever but because it answers when it answers, and the arm keeps travelling during the wait. Put the model above named actions you wrote and tested and it earns its place on the first afternoon. Put it inside the loop and you have built something that lunges.

What is robot control software in plain terms?

Robot control software is the part that turns an intention into motion and keeps checking reality while the motion happens. Three layers sit under a language model, and confusing them is why the question feels unanswerable. At the bottom is firmware on a small board next to the motors, counting encoder ticks and switching current on and off, with no idea what a cup is. Above that is the control loop on a real computer: it runs at a fixed rhythm, holds the joint limits, and is the reason the arm stops before it hits the table rather than after. Above that sits the plumbing that moves data between the parts — the camera process, the loop, the logger, your Python — which is what people mean by middleware. A language model is a fourth thing, sitting on top of all of it, and it can only use the vocabulary those layers expose. When the loop and the plumbing do not exist, a model has nothing to command, and its plan describes a robot you have not built yet.

What are my actual options for the layer underneath a model?

Five stacks cover nearly everything people build, and they differ mostly in how much is handed to you versus how much you write. ROS 2 is the default and the largest: drivers, navigation, arm planning, visualisation and bridges to agent frameworks, in exchange for a stack you install, learn and keep running. HORUS sits at the leaner end of the same shelf, an open-source Apache-2.0 middleware for Rust, Python and C++ in which all three languages share the same shared-memory ring buffers, so the camera frame your Python agent looked at and the compiled loop acting on it are not serialised between processes on one machine; it hands you no navigation and no drivers, and it runs no models itself. A vendor SDK is the third option, quickest to a first motion and hardest to leave. A single Python script talking to a motor driver is the fourth, and it is the right answer for one machine doing one thing. The fifth is a robot-trained action model that outputs motion directly, replacing some of your actions rather than commanding them.

How do those options compare side by side?

Pick by what you are unwilling to build yourself, not by which stack sounds most serious. Read the last two columns first and treat the middle one as a warning label about the week you will lose before anything moves. None of these rows are exclusive, and most working systems mix them: a vendor SDK for the arm, your own Python for the logic, a model above both. Every row also assumes the actions exist, because a language layer arranges capabilities and cannot invent them. A stack that hands you more ready-made actions shortens the project far more than any property of the transport underneath it.

What do people coming from models try first, and why does it stop working?

They ask the model for joint angles or motor commands directly, and it works in the demonstration and stops working the moment anything is unexpected. That first version is intoxicating: describe the scene in a prompt, ask for a sequence of positions, feed them to the arm, watch it move. It moves. What is missing shows up later. Nothing checks reality between one position and the next, so if the cup is not quite where the model believed, the arm arrives at an empty patch of table and closes anyway. Nothing bounds the path, so a sequence that reads perfectly sends the elbow through your monitor. And the arm is either frozen mid-air or coasting while the model thinks. The fix is not a longer prompt or a larger model. The fix is to stop asking for positions and start asking for verbs — grasp, move to, release — that you implemented and tested, each of which checks its own preconditions and refuses when they do not hold.

Does this change if I am a machine learning engineer rather than a roboticist?

It changes what will surprise you, not what you have to build. Coming from models you already have the two things that matter most: you write Python quickly, and you are comfortable working with a component that is confidently wrong sometimes. What catches people from this direction is that the physical layer has no test set. There is no held-out split telling you whether the grasp generalises; there is a table, and a cup, and whether it is still standing. Retries are not free, and a bad run costs a knocked-over glass or a stripped gear rather than a slightly worse metric. The second surprise is how much of the work is unglamorous plumbing: getting the camera process and the control process to agree about what time it is and what they are looking at. Roboticists coming the other way have exactly the opposite problem, and neither group is behind. If the field itself is still fuzzy, the difference between embodied AI and robotics is worth ten minutes first.

What hardware do I need before a model can move anything?

Less than you expect if the model runs in a data centre, and considerably more if it runs on the robot. With a hosted model the computer on the robot only runs your own code, exactly as it did before: enough machine to hold a camera process and a control loop at once, a network path, and a defined behaviour for the moment the connection disappears mid-plan. With a local model that same computer now runs a model and a control loop together, and under pressure it is the loop that suffers, because the loop is the part with a deadline. The usual escape is a second machine nearby that answers questions while the robot keeps its own timing. Small boards do not participate in this at all: a microcontroller runs firmware, takes commands from a bigger computer, and is unaware a model exists anywhere in the system. If you have not bought anything yet, there is a strong case for learning this without a robot first.

How long does it take to get from a working model to a robot that picks something up?

A convincing weekend, then several months to something you would let a stranger operate. The weekend version works because you asked in roughly the phrasing you had in mind while writing the actions, with the cup roughly where you tested it. Everything after that is the actual project: deciding what happens when the model names an action that does not exist, or an object that is not in the room, or asks for something reasonable that your hardware cannot do. Then the grasp that works on the mug and fails on the glass. Then the run where the plan was correct and the world moved while the model was thinking. Connecting a model to a robot is an afternoon. Making the robot behave predictably when the model is wrong is the whole job, and it is where the time goes on every project that ends up working. The fastest honest route is to spend the first week writing three actions that each end in a known state, and only then hang a model above them.

What do I need to know before wiring a model to a machine?

Enough Python to write a function with clear preconditions, and an honest list of what your robot can already do without help. That is a lower bar than it sounds, because nothing here demands you understand control theory or kinematics on day one. The idea you cannot skip is why the loop underneath cannot wait, which separates people who build this safely from people who build something that works once on camera. If that idea is fuzzy, the explanation of control loops and their timing is the prerequisite, not the model documentation. The second useful thing is knowing where an interpreted language is fine and where it is not, because your agent layer will be Python and your loop may not be able to be; the honest account of Python in robot control covers that line without pretending Python is never the answer. Beyond those, being precise about what each of your functions promises matters more than any framework knowledge.

What do I give up by keeping the model out of the moving part?

You give up the fluidity of the demonstration videos, and it is worth being honest that this is a real loss. A robot built this way can only do what its action list allows, so a request falling between two verbs gets refused rather than improvised. Watching that happen is genuinely frustrating: a person can see obviously how the task should go, and the machine says no because nobody wrote that verb. You also carry the cost of writing and maintaining those actions, which is most of the code in the project and none of the part you find interesting. Every new capability becomes a small engineering task instead of a sentence in a prompt. What you get in exchange is that your failure modes have names. The arm stops before the table whether or not the model made sense. A wrong plan produces a refusal or a stop rather than a swing. That trade is worth making until somebody can explain why it is not.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the hard part of your robot is everything except the language layer. If the machine must build a map of a building, cross it, plan an arm path around an obstacle, or talk to a sensor whose vendor ships only ROS 2 support, the ecosystem is the entire reason to be there and nothing else comes close. If more than one computer is involved, ROS 2 answers questions you would otherwise spend a month answering badly. In a research group it is also how your work gets reproduced by somebody else, which is not a technical argument and is frequently the deciding one. And when a team already knows the tooling, that knowledge outweighs any property of a transport. HORUS is not the answer in those cases and does not pretend to be: no navigation, no mapping, no driver library, validated in simulation rather than across a fleet of shipped machines, and swapping out a working stack to gain something you cannot yet describe is a good way to lose a quarter.

Will a bigger model eventually close this gap?

No, and here is why: the control code is not there because the model is insufficiently clever. It is there because something has to be on time, every time, with a bounded worst case, and that requirement does not soften as models improve. A perfect planner still needs something running continuously that holds the joint limit, notices the person who walked into the workspace between one decision and the next, and brings the machine to a stop when a sensor reading stops arriving. Those are not decisions in the sense a model makes decisions; they are guarantees, and guarantees are properties of code you can reason about line by line. The robot also has to behave when the model is unreachable, which is a Tuesday, not an edge case. A better driving instructor in the passenger seat does not remove the need for brakes. What better models genuinely change is how many actions you must write yourself and how loosely a request can be phrased, and both are real improvements of a different kind.

Is this just a shortage of robot training data?

Partly, but not the way you think. It is true that text is abundant while recordings of robots doing tasks are not, and true that models trained on such recordings do things a chat model cannot. What the framing misses is that data scarcity explains why grasping is hard, not why a model cannot be the whole system. Even a model that grasps beautifully still produces its output when it produces it, still promises nothing about the moment a person's hand enters the workspace, and still cannot tell you afterwards why the gripper closed early. Those properties come from what a model is, not from a shortage of examples. The other half the framing misses is that the code around the model is not a stopgap waiting to be learned away. It is the part that lets you ship something and then sleep. More data will make your actions better and fewer; it will not make the layer underneath optional.

How do I decide what to build the moving part on?

Start by naming who will speak to this robot, because that one answer settles most of it. If the answer is you and two colleagues, you want a better command line more than you want a model, and you will get there in a fraction of the time. If the answer is people who will never read a manual, the language layer is the product and deserves real effort. Then count what a model would have to compose: three or more actions that each end in a known state and that you would trust a stranger to trigger. If you do not have those, build them first, because otherwise you will misdiagnose missing actions as a bad model and lose a month to prompt wording. Then choose the layer underneath by what you refuse to write yourself: navigation and drivers push you towards the large ecosystem, while one machine and a tight loop push you towards something small. Last, write down what the robot does when the model is wrong, slow or unreachable, before you connect anything.

Take the line that matches your situation:

• If you have a model and no robot yet -> a simulator with a real control loop, because the loop is the piece you are missing and it costs nothing to learn.
• If your machine must move around a building -> ROS 2, because navigation and mapping are solved there and effectively nowhere else.
• If you are one person with one machine and one arm -> a small middleware under your own Python, because the week you save goes into writing actions instead.
• If strangers will instruct the robot -> a narrow set of verbs with the model interpreting phrasing, because open-ended language is a support burden you carry forever.
• If the hard part is the grasp itself -> a robot-trained action model, because no amount of prompting fixes a gripper that closes early.

The HORUS Fit Framework is the checklist behind those lines, and not one of its five axes is a number: ecosystem size, setup effort, team size fit, deployment target, and licence. Score every candidate on all five, and the axis you cannot compromise on decides it.

Whatever writes your plans, something underneath still has to be on time. HORUS is open source under Apache-2.0 at github.com/softmata/horus. Star it so it is in your list when you start building.