How Robots Decide What to Do Next: Planning for AI People

Robots decide in layers: a state machine, behaviour tree or model chooses the next action, and a control loop below it keeps the machine legal. The deciding layer is allowed to be slow and occasionally wrong, the layer below it is not, and plumbing such as ROS 2 or HORUS carries decisions between them. That split only collapses when a machine repeats one fixed motion forever. The rest of this post is for someone who builds agents in software and wants to know what changes when the agent has a body.

You have built agents. A loop, a set of tools, a model choosing which tool to call, retries when a call fails, a transcript afterwards that explains what happened. So when somebody puts a robot in front of you, the shape looks familiar with different tools: instead of sending an email there is a grasp. Then you watch the thing run and the analogy comes apart in a way you cannot quite name. The arm is already moving while the model is still thinking. A failed tool call in software returns an error and costs nothing; here it leaves a gripper half closed around a glass and no obvious place to return to. You search for robot planning and get papers about sampling and configuration space, which is plainly not what you meant, and nobody tells you which words you wanted. Every tutorial either assumes you have a robot working already or assumes you have never written software. What you want is the middle: what decides, where does that live, and why can it not simply be a loop with tools.

What actually chooses a robot's next action?

Three separate things choose it, at three different rhythms, and mixing them up is why the question feels slippery. At the slowest rhythm sits the deciding layer: a state machine, a behaviour tree, a search over actions, or a model, thinking in units like put the mug in the sink and taking as long as it takes. In the middle sits the action currently running — move to that pose, close the gripper — which usually calls a motion planner to find a path that does not travel through the table, and which reports back whether it finished, failed, or was cancelled. At the fastest rhythm, never pausing, sits the control loop, which does not know what a mug is; it knows where each joint is, where each joint should be, and what it is not allowed to do. Every machine that behaves well has all three, even when two of them are hidden inside a library somebody else wrote. Most confusion in this area is a question aimed at one rhythm being answered at another, which is why advice about robot decision-making so often sounds contradictory.

What is robot planning in plain terms?

Robot planning is two unrelated jobs sharing one word, and you almost certainly mean the first one. Task planning is the to-do list: which actions, in which order, given a goal and what is currently believed about the world. Open the drawer before putting socks in the drawer. Put the pan down before opening the fridge. Motion planning is geometry: given a starting pose, an ending pose, and a description of obstacles, find a path the limb can follow without hitting anything. That is where configuration space and sampling live, and for most people it is a library call rather than a research project. A third word, trajectory, means a path with timing attached, which is what actually gets handed to the loop. The reason this matters is practical: newcomers who search the obvious phrase land in the geometry literature, conclude the field is impenetrable mathematics, and never find out that their real problem was a to-do list with honest failure handling.

What are my actual options for the layer that decides?

Six patterns cover almost every deciding layer that ships, and they trade flexibility against your ability to predict behaviour. A scripted sequence runs actions in a fixed order and is the right answer more often than anyone admits. A hand-written state machine names every state and transition, and stays readable while the list is short. A behaviour tree adds priorities and reuse across many behaviours. A symbolic task planner searches for an action order that reaches a goal. A model above named actions turns a freely phrased request into calls on a list you wrote. A learned policy skips the list and produces motion straight from pixels. All six assume something carries decisions to the moving parts and sensor data back, which is where middleware sits: ROS 2 supplies topics, services and an action mechanism for long-running goals that can be cancelled, plus navigation and arm planning; HORUS is the leaner end of the same shelf, open source under Apache-2.0, letting Rust, Python and C++ share the same shared-memory ring buffers so a Python decider and a compiled loop are not serialising camera frames between them on one machine, while shipping no planner and no drivers of its own.

How do those options compare side by side?

Pick by how much of the machine's behaviour you need to be able to predict before it runs, because that single question separates these rows more than anything else. Read the last two columns first, and treat the middle column as the one people skip and later regret, because it names the week that disappears into documentation before anything moves. Two rows describe what sits underneath the decider rather than what decides, and they belong on the same list because that choice constrains every other row. These rows combine freely and most working systems use two or three at once — a scripted sequence inside an action, a state machine above it, a model translating what a person asked for. Nothing here decides anything without an action list underneath, so a machine with three honest actions and a plain state machine will behave better than the same machine with a sophisticated decider on top of actions nobody tested.

What do people from agent work try first, and why does it stop working?

They write an agent loop with the motors exposed as tools, and it is thrilling for one afternoon and then quietly unusable. The first version is genuinely impressive: describe the scene, ask for a sequence of positions, send them, watch the arm move. What is missing shows up on the second day. Nothing checks reality between one position and the next, so when the cup sits slightly left of where it was during testing, the gripper closes on air and the loop records a success. Nothing bounds the path, so a sequence that reads perfectly sends an elbow into a monitor. When a step fails there is no defined state to return to, so the retry — the reflex that saves you in software — starts from somewhere the code has never seen and makes things worse. And between the model answering and the next answer arriving, the arm is either frozen mid-air or still coasting. The fix is not a better prompt or a bigger model. The fix is to stop asking for positions and start asking for verbs you implemented, each of which checks its own preconditions and refuses when they do not hold.

Does this change if I come from agent frameworks rather than robotics?

It changes which instincts help you and which quietly hurt, and both lists are short. Three habits transfer directly and are worth more than they sound: writing tools with narrow contracts, treating a model as a component that will sometimes be confidently wrong, and keeping a transcript you can read afterwards. That third one is undervalued in robotics and you will bring it as an advantage. Two instincts actively hurt. The first is retrying on failure, because a physical retry from an unknown state is how a bad run becomes an expensive one; here the correct reflex is to return to a known state first, then retry. The second is treating latency as a cost rather than a correctness issue — waiting a little longer for a better answer is free in software and is not free when a limb is moving. If you want the underlying reason those instincts break, why a language model can write an app but cannot pick up a cup works through the mechanism rather than the symptoms.

What hardware does the deciding layer actually need?

Almost nothing of its own, which is the genuinely convenient part of this design. The deciding layer can run in a data centre, on a laptop beside the robot, or on the machine itself, because thinking in units of put the mug in the sink does not require being close to the motors. What that arrangement does require is a defined behaviour for the moment the connection disappears halfway through a plan, and that behaviour should be written before anything is connected. The layer underneath has the opposite constraint: the control loop has to be on the machine, next to the hardware, with nothing competing for the processor at the wrong moment. This is why running a model locally changes the hardware conversation completely — a computer running a model and a loop together starves the loop first, and the loop is the part with the deadline, so people who want local models usually add a second computer rather than a bigger one. Small boards next to the motors do not participate in deciding at all; they take commands and count encoder ticks.

How long does it take to get a deciding layer that behaves?

An afternoon for the first version, and most of the project for the version you would leave running unattended. The afternoon version is real: three actions, a state machine with four states, and a machine that does the task while you watch. Everything after that is failure handling, and failure handling is the entire discipline. What happens when a grasp fails halfway. What happens when the deciding layer asks for an action that does not exist or an object that is not in the room. What happens when a plan is correct and someone moves the target while the plan is running. What happens when the machine is switched on mid-task after a crash and has no idea what it was doing. Teams underestimate this consistently because the happy path is so quick to build that it feels like the project is nearly done. The way to compress it is to make every action end in a known state and report honestly, and then to add actions one at a time rather than designing the full list up front.

What do I need to know before writing the layer that decides?

You need to be able to write a function that states its preconditions and ends somewhere known, and that is genuinely most of it. No control theory, no kinematics, no formal methods on day one. The idea you cannot skip is why the layer underneath cannot wait for you, because everything else in this design follows from it; the explanation of control loops and their timing is the prerequisite reading rather than any framework documentation. The second thing worth learning early is what an action should promise: it either reaches the end state, or it stops in a state you named, and it says which. Actions that sometimes return halfway through, or that leave a gripper in an undefined position, poison every layer above them and no clever decider fixes that. The third is a bias towards logging what each part believed at the time, because the interesting bugs here are disagreements between processes about what the world looked like.

What do I give up by keeping the deciding layer separate from the moving part?

You give up the fluidity of the demonstration videos, and that is a real loss worth naming rather than glossing over. A machine built this way can only do what its action list allows, so a request that falls between two verbs gets refused instead of improvised, and watching that happen is genuinely irritating when a person can see obviously how the task should go. You also carry the cost of writing and maintaining those actions, which is most of the code in the project and none of the part you find interesting. Every new capability becomes a small engineering task rather than a sentence in a prompt. What you get in exchange is that your failures have names and locations. The arm stops before the table whether or not the deciding layer made sense. A wrong plan produces a refusal or a halt rather than a swing. And when something does go wrong you can point at one layer, which is the difference between a bug you fix in an evening and a mystery you live with for a month.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the plan involves the parts of robotics other people have already solved well. If the machine has to build a map and cross a building, plan an arm around obstacles, or run a behaviour tree that the wider community maintains and documents, those packages are years of work you are not going to reproduce and the action mechanism for long-running cancellable goals is exactly the shape this problem wants. If more than one computer is involved, ROS 2 answers coordination questions you would otherwise answer badly over a month. In a research group it is also how somebody else reproduces your work, which is a social argument that decides more projects than any technical one. And if your team already knows the tooling, that knowledge beats any property of a transport underneath. HORUS is not the answer in those cases and does not claim to be: no planner, no navigation, no driver library, validated in simulation rather than across shipped fleets, and no community package to install when you hit the part you did not plan for.

Will a large enough model just replace all of this?

No, and here is why: the structure is not there because the deciding layer is not clever enough. It is there because two jobs have incompatible requirements. One must be allowed to think for as long as thinking takes, occasionally be wrong, and be corrected. The other must produce an answer every cycle with a bounded worst case, whatever else is happening on the machine. No single component can have both properties, and that is a statement about scheduling rather than about intelligence. A perfect decider still needs something running continuously that holds the joint limit, notices the person who stepped into the workspace between one decision and the next, and stops the machine when a sensor reading stops arriving. It also needs to behave sensibly when the model is unreachable, which is an ordinary Tuesday. What better models genuinely change is how many actions you write yourself and how loosely a request can be phrased. The layering survives, with a different tenant at the top. Whether an agent can run a robot unsupervised covers where that ceiling currently sits.

Is this only a problem because the robot has no memory of what it did?

Partly, but not the way you think. It is true that most machines that behave badly are machines with no idea what they just did — no record of the last action, no belief about where objects are, nothing to consult when a step fails — and adding that record fixes a startling number of bugs. Knowing which drawer you already opened prevents an entire family of stupid failures. What the framing misses is that memory is a description of the past and the deciding problem is about the present, which has moved. The chair is not where the record says because somebody just moved it. The mug fell over during the last action and no camera was looking. Memory that is trusted without being re-checked is worse than no memory, because it converts a hesitant machine into a confident wrong one. The habit that actually works is cheap: keep the record, and have each action confirm the one fact it depends on immediately before acting on it.

How do I decide which deciding layer to build?

Count the states before you compare any tools, because that number decides this faster than any feature list. Write down every distinct situation the machine can be in and every transition between them. If the list fits on one page and stops growing, write a state machine and stop reading comparisons. If it keeps growing and behaviours are starting to compete for the machine's attention, a behaviour tree pays for itself. If people who will never read a manual are going to phrase requests freely, put a model above your actions — but only after those actions exist, because a language layer arranges capabilities and cannot invent them. If the ordering genuinely needs searching, which is rarer than it sounds, a symbolic planner is the right tool and you will know because you can already write the goal down formally. And if you are still deciding whether to invest in this field at all, whether to learn robotics or wait for foundation models is the question to settle first.

Take the line that matches your situation:

• If your machine does one job in one order -> a scripted sequence, because everything else is machinery you would maintain for nothing.
• If you can draw every state on one page -> a state machine, because readable beats general when something goes wrong at midnight.
• If behaviours compete and priorities shift -> a behaviour tree, because that is the case the extra structure was designed for.
• If strangers will phrase the requests -> a model above named actions, because open-ended language is otherwise a support burden forever.
• If a decider in Python and a loop in a compiled language share one machine -> a lean middleware between them, because that handover is the whole problem.

The HORUS Fit Framework is the checklist underneath those lines, and not one of its five axes is a number: ecosystem size, setup effort, team size fit, deployment target, and licence. Score every candidate on all five, and let the axis you cannot compromise on make the call.

Whatever ends up choosing the next action, something underneath still has to be on time. HORUS is open source under Apache-2.0 at github.com/softmata/horus. Star it so it is in your list when you start building.