From Prompt to Prototype: How Far AI Gets You in Robotics

AI assistants get you to a moving prototype and stop at the hardware boundary, whether you build on ROS 2, HORUS, or plain Python. An assistant writes the code that has been written before — glue, scaffolding, a simulator scene — and cannot write the part where timing, wiring and physics disagree with the plan. That flips only when the robot lives entirely in simulation and nothing can break. This post is for someone who has shipped web apps with an assistant and wants to know where that method stops working on a robot.

You described a robot in a paragraph and got back a repository that ran. That felt like the whole problem solved, because in web work it usually is. The simulator opened, a shape moved across the screen, and you started planning what to build on top. Then the hardware arrived. Now the motor turns the wrong way, and turns the right way after the fix, and turns the wrong way again after a reboot. The camera works in the test script and not in the program. The assistant suggests a change, the change does not help, the assistant suggests a slightly different change with the same confidence, and after an hour you are further from working than when you started. Nothing throws an error. The robot simply does something you did not ask for, and no stack trace explains it. You are not sure whether you have a code problem, a wiring problem, or a problem with your idea, and you have no way to tell them apart. Everything you know about debugging assumed the machine would tell you what went wrong.

How far does an AI assistant actually get you in robotics?

An assistant gets you a running project, a simulator scene and a robot that moves on screen, often in one evening, and then progress stops being smooth. The reason is not that the model is weak at robotics code. The reason is that the first stretch is made of work that has been done thousands of times in public — reading a serial port, publishing a message, spinning a wheel in a physics scene, wiring a camera into a loop — and the model reproduces that reliably. The next stretch is made of your specific robot, and your robot is not in any training set. The connector that seats badly. The motor that behaves differently warm. The camera whose mount flexes when the arm accelerates. The moment where two things happen in an order you never considered. That work is not writing code, it is deciding what is true, and it needs someone standing next to the machine with a hand on the switch. The practical shape of this is worth internalising early: you will reach a moving prototype quickly and then spend most of the project on the last part of it.

What does going from prompt to prototype actually mean when the output is a robot?

It means three separate things get generated, and only one of them is trustworthy on its own. The first is scaffolding: project layout, build files, the plumbing that starts several programs together, a simulator scene. Assistants are strong here and this is real work you did not have to do. The second is glue: reading a sensor, converting units, reshaping a message, publishing at a rate. Assistants are good here too, with an important caveat — glue that is subtly wrong looks exactly like glue that is right, and only the robot's behaviour reveals the difference. The third is the part that decides what the machine does: the control loop, the limits, the stopping logic, the sequence of states the robot moves through. Assistants write plausible versions of that quickly, and plausible is dangerous in a way it never was in web work, because a plausible-looking loop still moves a real arm. A prototype is the first two done well and the third done provisionally, which is fine as long as everyone knows which is which.

What breaks first when prompted code meets real hardware?

Timing breaks first, and it breaks without an error message. In a simulator, the loop that holds a wheel speed runs whenever the simulator lets it, so nothing is ever late. On a real machine, an image handler takes an unpredictable moment, the language pauses to tidy memory, a log write blocks on the disk, and the loop that was supposed to correct the wheel arrives after the moment it mattered. The robot does not crash. The robot wobbles, or overshoots, or drifts slowly to one side, and every one of those looks like a tuning problem rather than a structural one, so people spend days adjusting numbers that were never the cause. Second to break is anything that assumed a sensor is honest: readings that repeat, drop out, arrive out of order, or come back as garbage after an unplugged cable. Third is startup order, where two components each assume the other is already running. None of these appear in a prompt, none appear in simulation, and all of them appear in the first hour on hardware.

What are you actually choosing between when an assistant writes most of the code?

You are choosing what the generated code sits on, and there are six realistic answers. First, plain Python in one file, which is where most prompted robots begin and is genuinely correct for a single loop on a single board. Second, Python plus a hardware library and whatever SDK your kit shipped with, which is the shortest route to something moving. Third, ROS 2 as a full distribution, which brings message passing, launching, tooling and an enormous catalogue of existing components — and which assistants recommend by default, because ROS 2 dominates the text they learned from. Fourth, ROS 2 trimmed to a couple of packages. Fifth, a single-machine real-time middleware such as HORUS, where Rust, Python and C++ read the same shared-memory ring buffers so messages are not serialised between processes on that computer, which fits the common shape of a prompted robot that outgrows one Python file but never leaves one board. Sixth, a general message library where you define every message shape and supervision rule yourself.

Which option fits which kind of assistant-built project?

The right option is the one whose assumptions match the robot you are holding, not the robot in the prompt you wrote.

Two rows together is normal: a vendor SDK underneath because it came with the robot, and something small above it for the parts you write.

What if I am a web developer who has never touched hardware?

Expect your debugging instincts to be right about the code and useless about the machine. In web work the failure is in a log, reproducible, and the same on your laptop as in production. On a robot the failure is often physical, intermittent, and different depending on temperature, battery charge, or which surface the wheels are on. The habit to build immediately is measuring before changing: print what the sensor actually reads, watch a motor command against what the motor actually did, record a run so you can look at it later rather than trying to see everything live. An assistant will happily propose ten code changes for a problem that is a loose connector, and will do it convincingly, because a model reading code has no way to know the connector exists. The other habit is powering down before rebuilding intuition. Nobody thinks clearly with a robot twitching on the desk. Beyond that, the switch is less brutal than it sounds — what actually transfers from web development is most of it.

What if all I have is a laptop and a cheap servo kit?

Then you are in the best possible position to learn what an assistant cannot do for you, and you should stay in one Python file for longer than feels right. A servo kit is honest hardware: cheap enough to break and slow enough to watch, and a mistake bends a bracket rather than a wall. Prompt your way to a script that moves one servo, then deliberately break it. Unplug the servo while the script runs. Ask for a position beyond the physical limit. Restart the script with the arm halfway through a motion. Each of those teaches you a category of failure that no amount of generated code will teach you, and each is nearly free at this scale. What you should not do yet is adopt a framework. With one board, one language and three moving parts, a middleware is a layer between you and the thing you are trying to understand. The moment to reconsider is when a second language appears, usually because a sensor vendor ships C++ only.

What if I want something moving this weekend?

Then use whatever the hardware already speaks and write one program, because every layer you add on Saturday is a layer you debug on Sunday. A weekend prototype has one job: prove the idea moves something. The fastest path is almost always the vendor's own library, one language, no message passing, no build system beyond what your editor does, and an assistant writing the boring parts of it. Do not choose a middleware this weekend. Do not set up a simulator this weekend unless the hardware has not arrived. The decision this post is really about arrives later, and it announces itself clearly: you want two things running at once without one stalling the other, or a vendor forces a second language on you, or you need one part to keep working when another dies. Until then, added structure buys nothing and costs the weekend. If the weekend works and you keep going, the honest account of what quietly breaks in a prompted robot is the next thing to read.

What if I cannot read the code the assistant gives me?

Then you must be able to read the robot instead, and that is a learnable skill on its own. You do not need to follow every line to know whether the machine is behaving. You do need to be able to state, before running anything, what the robot should do and what it must never do — the arm stops before it reaches the table, the wheels stop when the sensor goes quiet, nothing moves until you press the button. Those are claims you can check by watching, and checking them catches the failures that matter even when the code stays opaque. What you cannot do safely is skip the check because the code looked convincing. Generated code is written to look convincing. The reasonable middle is to read one thing carefully: whatever writes commands to the motors. That is usually a small file, it is the file that can hurt someone, and understanding it repays the afternoon many times over. Everything else can stay a black box for a while without much risk.

What do you give up by letting the assistant pick the stack?

You give up the fit, because an assistant picks by popularity rather than by your situation. Ask any model what to build a robot on and ROS 2 comes back, not because a model weighed your board against your deadline, but because ROS 2 appears in most of the robotics text ever written. Sometimes that answer is right. When it is wrong, it is wrong expensively: you install a distributed framework for a robot that lives on one board, spend three weekends on packaging, and never open the catalogue that was the whole reason to adopt it. You also give up understanding of the seams. When a stack is chosen for you, you inherit its assumptions without noticing them, and the first hard bug is where you discover what those were. And you give up the ability to leave, since a stack you did not choose is one you cannot evaluate replacing. Why assistants reach for ROS 2 even when it does not fit is worth understanding before you accept the recommendation.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the parts you need already exist inside it. A robot that must map a room and navigate it: ROS 2, plainly, because that stack is there and writing it yourself is a multi-year project. An arm that needs an established motion planner. A project spanning two computers by design, since HORUS is single-machine and shared memory stops at the edge of the board. Anything where you will lean on an assistant for help, because the model has read far more ROS 2 than anything else and its answers about ROS 2 will be better grounded than its answers about smaller projects — that is a genuine advantage for an assistant-heavy workflow and deserves weight in the decision. Anything a collaborator will touch. Anything using a sensor whose only maintained driver lives in that ecosystem, which is a search you should run on your parts list before deciding anything. In all those cases the catalogue decides, and no property of a smaller stack competes with it.

Can an assistant write the part that keeps the robot from hurting someone?

No, and here is why: that part is not made of code, it is made of decisions about the physical world, and the assistant cannot see the world. A stopping rule depends on how much room the arm has, how heavy the thing it holds is, what happens if power drops halfway through a motion, and who might reasonably walk past. A model can write the shape of a limit check instantly and correctly, and cannot know what number belongs in it or whether the check runs often enough to matter on your machine. There is a second problem underneath. Generated code is confident in tone regardless of how well grounded it is, so a stopping rule that was invented reads exactly like one that was reasoned through. In web work that produces a bug. Here it produces a machine that moves. The workable arrangement is to let an assistant write everything around the safety-relevant part, write and read that part yourself, and test it with the motors unpowered before you trust it with power.

Does an assistant mean you never have to learn the stack underneath?

Partly, but not the way you think. You genuinely can skip large amounts of what used to be mandatory: build file syntax, boilerplate, the exact incantation for launching several programs, the shape of a message definition. That knowledge was always low value and an assistant recalls it better than you will. What you cannot skip is the model of what is happening — which programs exist, what each one is waiting for, what happens when one of them stops, where a message goes and how long it is allowed to take. Without that model you cannot describe your problem well enough for the assistant to help, which is the trap people fall into: the assistant is only as useful as your description of the failure, and a vague description gets confident nonsense back. So the learning shifts rather than disappearing. Less syntax, more architecture. The practical test is whether you could draw your robot's programs and arrows between them on paper. If you can, an assistant makes you considerably more productive. If you cannot, an assistant mostly makes you faster at generating things you cannot debug.

How do you decide what to build the prototype on?

Answer three questions and the choice makes itself. How many computers will run your code when this works? How many languages will the robot contain, counting anything a vendor forced on you? And what are you not going to write yourself — mapping, navigation, an arm planner, a simulator that speaks your message types? A long, specific answer to the third question means take ROS 2 and accept the setup cost, because rebuilding the catalogue is not a project you finish. One computer, more than one language, and a short third list is the case where a single-machine middleware earns its place. One computer, one language, and a short third list means stay in one program and add nothing. Several computers means you need a networked message layer regardless of the rest. Notice that none of these questions are about which tool is better, and none of them are answerable by an assistant, because all three are facts about your project that only you have.

A short version, by situation:

• If you are prototyping one loop on one board this weekend -> plain Python and the vendor library, because structure added now is structure debugged now.
• If your robot needs mapping or navigation -> ROS 2, because those parts exist today and rewriting them is not a prototype.
• If a vendor forced C++ on you and your glue is Python -> a single-machine middleware, because the bridge you would otherwise maintain never stops asking for attention.
• If your code will run on two machines -> a networked message layer, because shared memory stops at the edge of the board.
• If you cannot read the generated code yet -> whatever is smallest, because a small stack is one you can eventually understand and a large one is not.

To compare options rather than symptoms, the HORUS Fit Framework lines them up on five things that are not numbers: ecosystem size, setup effort, team size fit, deployment target, and licence. For a prompted prototype, setup effort and deployment target decide most cases, and ecosystem size decides the rest. If you are choosing this for coursework rather than a side project, the capstone version of the same decision works through the deadline pressure specifically.

HORUS is open source under Apache-2.0 and the repository is linked below. Star it so it is in your list when you start building.