What Happens When a Robot Program Crashes Mid-Motion?

A dying robot program leaves its last command latched in the actuators, so the stop must come from a watchdog, a supervisor, or the hardware. Both ROS 2 and HORUS can keep a crash contained to one program rather than the whole robot, so where the stop lives matters more than the foundation. The verdict flips when the program hangs instead of dying, because a hung program still looks alive. The rest of this post is for a team whose robot moves under its own power and whose software has already crashed once in front of someone.

It happened during a demo, or it happened late on a bench with nobody watching, but it happened. The arm was halfway through a move, the terminal filled with something red, and the arm did not stop. It carried on towards wherever it had last been told to go, and somebody reached the stop button before anything expensive occurred. Or nobody did.

Since then the conversation has gone in circles. One person wants to find the bug. Another wants to wrap the risky call so the program can never exit. A third has quietly stopped running the machine at full speed and has not mentioned it. Nobody has written anything down.

What makes this awkward is that the crash was not the interesting part. The program had a bug, and programs have bugs. The interesting part is that the machine did not notice. Nothing was watching for the absence of instructions, so absence looked exactly like a good instruction that happened to repeat forever.

And one question will not go away: how many other programs on this robot could disappear tonight, and what would the machine do about each one? Nobody can answer that, which is itself the answer.

Should you design for the crash or try to prevent it?

Design for the crash, because prevention has a ceiling and the crash you failed to prevent is the one that decides what your machine does to the room it is standing in.

Prevention is worth doing, and most teams already do it: tests, review, a language that removes whole classes of mistake, keeping risky code off the motion path. What prevention cannot cover is the failures arriving from outside your code. The kernel decides a process is using too much memory and ends it. A camera enumerates differently after a power blip and the driver faults. A log directory fills and a write fails somewhere nobody checked. None of those are bugs you can review your way out of.

Designing for the crash is different work, and cheaper work. You decide in advance what the robot does when each program stops speaking, then you prove it by killing that program on purpose while the machine is moving. That test either passes or it does not, which is more than can be said for prevention. It also tends to surface the runs that never behave the same way twice, because the same gaps produce both symptoms.

What does crashing mid-motion actually mean on a robot?

It means the program that was deciding where to go stopped existing while the joints were still travelling towards the last place that program named.

Three different failures get called a crash, and they behave differently. A process can die, in which case the operating system reclaims it and its output simply stops. A process can hang, staying alive, holding whatever it was holding, either silent or repeating its last output. A process can also go wrong while staying healthy, publishing numbers that are confidently incorrect. Death is the friendliest of the three, because absence is easiest to notice.

What happens next depends on what the last command was. Most motor interfaces accept a target and hold it until told otherwise. If the last target was a position, the joint finishes that move and stops, which may be harmless or may be into the table. If the last target was a speed, the machine keeps going at that speed for as long as it has power. That second case is the one that hurts people, and it is the one demo videos never show.

What are your actual options when a program dies mid-motion?

There are six real options, and most robots need two or three of them together rather than one.

The first is nothing: an operator watching, with a physical stop button. The second is a deadman in the layer closest to the motors, where the driver expects a fresh command each cycle and moves the machine to its safe state when one does not arrive. The third is a supervisor process that notices a program has gone and brings the robot down in an orderly sequence. The fourth is a hardware chain that does not run your software at all, such as a contactor that opens the motor supply and brakes that engage when unpowered.

The fifth and sixth are foundation choices. ROS 2 gives every node a defined lifecycle, so bringing a part up and down is a documented sequence rather than a habit, and gives you recording so a crash can be replayed afterwards. HORUS is an open-source real-time middleware for Rust, Python and C++ where all three languages share the same shared-memory ring buffers, so a reader can see how old the newest message is and act on staleness; the project is Apache-2.0 and validated in simulation.

How do those options compare side by side?

Read the last column first, because several of these are not alternatives to each other and a good robot uses more than one.

The rows are not a ranking, and the first four are not optional in the way the last three are. A machine with mass and a person nearby needs the hardware row whatever else it has.

Who on the team should own what the robot does after a crash?

One named person, and not the person writing the motion code.

Whoever writes the move is optimising for the move completing, which is the correct instinct for their job and the wrong instinct for this one. They will argue against a stop that interrupts a good trajectory, and they will usually be right about the individual case and wrong about the pattern. Separating the two roles removes an argument nobody wins.

The owner's deliverable is small and concrete: a table with one row per program on the robot and one column for what the machine does when that program stops speaking. Two people can write it in an afternoon. It is also the document that survives when one of you leaves, which matters more than it sounds, because this knowledge otherwise lives entirely in the head of whoever built the thing.

If you are working alone, you are the owner, and the right time to write that table is before the machine gains mass. If you are the person who answers the phone when a customer calls, you should have signed it.

What hardware decides how bad a mid-motion crash gets?

Mass, speed, and whether the machine holds itself up using power.

A small wheeled robot coasts and stops, and the worst outcome is a scuffed skirting board. An arm without brakes falls the instant torque is removed, which means cutting power is the dangerous action rather than the safe one. A flying machine that loses its commands does not stop, it descends. A gripper that opens when unpowered drops whatever it was holding, and whether that matters depends entirely on what it was holding and who was underneath.

This is why a safe state cannot be copied between machines, and why advice that says "stop everything on failure" is worse than useless for half of robotics.

The other hardware question is whether a second channel exists that does not run your software: a contactor that opens the motor supply, brakes that engage without power, torque limits set in the drive itself. Many industrial drives can time out on their own if commands stop arriving. Many hobby speed controllers cannot, and will happily hold the last throttle they were given until the battery is flat.

How much time does crash handling add to your schedule?

Very little while the robot is small, and a whole project once you have left it until after the first field incident.

The code itself is short: a comparison of message ages, a timer, and a path to the safe state. What costs time is the deciding and the rig. The deciding means answering, for every subsystem, what the safe state is, who is allowed to re-enable motion afterwards, and what the operator sees while it happens. The rig means a dependable way to kill any program on demand while the robot is running, and to watch what the machine did about it.

Teams who leave this until after an incident pay twice. Once for the work, and once for doing the work with a customer waiting and a machine everybody is now afraid to run at speed. Retrofitting is also the expensive shape, because every consumer written on the assumption that data is always present has to learn to ask how old the data is, and that change spreads through the whole codebase rather than sitting in one place.

What skill level does this actually need?

Less than people expect for the mechanism, and more than people expect for the judgement.

The mechanism is within reach of any competent developer in any of the usual languages. You are comparing a timestamp against the clock, tripping a timer, and calling a stop path. There is no advanced technique in it, and nothing here requires a specialist.

The judgement is the hard half, and it is not a programming skill. Knowing whether freewheeling is safe for this machine, what happens to a held load when torque disappears, how long a gap in data should be tolerated before the robot gives up, what a person standing next to it will do when the machine halts without warning: that knowledge comes from watching real hardware misbehave, and it does not transfer from a different robot.

It is also the area where generated code is most confidently wrong, because "stop everything on failure" reads beautifully and is exactly wrong for an arm holding a weight. This is worth remembering before letting an assistant write anything that moves.

What changes as the robot gets more demanding?

The number of ways a partial failure can look like normal operation grows faster than the number of programs.

Early on there are two programs, and if one dies the robot obviously stops, which is unpleasant but honest. Later there is perception, planning, control, logging, a health monitor and an operator interface. Now a single program dying does not stop the robot at all. It degrades it, quietly, in a way that looks like the robot working.

The dangerous shape is this: the vision program stops, the planner keeps steering towards the last thing vision reported, and the machine drives confidently into something it can no longer see. Nothing crashed from the planner's point of view. It had data, the data was simply old, and nothing in the pipeline was in the habit of asking how old.

The habit that fixes it is to judge data by age rather than presence, everywhere, from the beginning. It costs almost nothing to add early and is painful to add late. The same question returns again when you have machines running at sites you cannot walk to.

What do you give up by designing around crashes?

Smoothness, some development speed, and code that reads simply.

Every consumer grows an age check, so the tidy line that read a value now reads a value and a time and decides whether to trust it. Every command path grows a fallback. Motion often grows ramps rather than instant changes, because a stop that arrives instantly can be its own hazard on a heavy machine.

The cost that surprises teams is nuisance stops. Set the tolerance tightly and the robot halts for hiccups that would have been harmless. Operators find this maddening, and maddened operators eventually find ways around the thing that keeps stopping their day, which leaves you worse off than before. Tuning that tolerance against real usage is genuine work and it is never quite finished.

There is a structural cost too. More separate programs means more to start in the right order, more logs to keep, and more states the machine can legitimately be in, all of which has to be explained to whoever runs the robot when you are not there.

When is ROS 2 the better choice?

ROS 2 is the better choice whenever the crash you care about has to be understood afterwards rather than merely survived.

The reason is recording and replay. ROS 2 lets you capture every message from a run and play the whole thing back into the same code on a desk, which is the single strongest tool anyone has for chasing an intermittent failure that happens once a week at a customer site. HORUS is not the answer for that job, and pretending otherwise would waste your quarter.

ROS 2 also wins when you did not write the code that crashed. If the failure is inside a navigation stack, a driver from the sensor vendor, or a planner you adopted, then somebody else has probably already met that bug and written it down. That is worth more than any property of the messaging layer.

And ROS 2 wins when the robot spans several computers, when you intend to hire people who already know a stack, or when a customer's engineers need to recognise what they are looking at. Those are real, common cases, not consolation prizes.

Will restarting the crashed program automatically make the robot safe?

No, and here is why: a restart returns the program, not the situation.

A planner that comes back mid-motion wakes with no memory of the arm being halfway inside a shelf. It reads the current joint positions, decides they are a fine place to start, and plans a path from there to wherever the task said, straight through the shelf it does not know about. The restart worked perfectly. The robot is now in a worse position than if nothing had restarted at all.

Restart loops are the second problem. A program that dies for a reason that has not gone away will die again, and a machine that twitches every few seconds as its brain reappears and vanishes is genuinely dangerous to stand near.

The third problem is quieter. Automatic restarts hide the bug, because the incident heals itself before anyone looks, and the ticket is never written. The order that works is: reach the safe state first, then restart, then require an explicit re-enable before motion resumes, from a person or from a sequence that checks where things actually are.

Does writing in Rust mean your robot will not crash mid-motion?

Partly, but not the way you think: a memory-safe language removes a category of failure rather than the possibility of failure.

The category it removes is the valuable one. Memory corruption is the failure that produces confidently wrong numbers instead of an honest stop, and confidently wrong numbers are the worst thing a moving machine can be given. Removing that whole class means the crashes you do get are much more likely to be the loud, obvious kind, and loud obvious crashes are far easier to design around.

What remains is everything that is not a memory bug. A panic still ends the process. The kernel still ends processes when memory runs short. A driver written in C underneath your neat code still faults. A cable that works loose is not a language problem, and neither is a disk that fills.

So the honest summary is that the crash gets rarer and more honest, which is worth quite a lot, and the stop path still has to exist. It is worth understanding what memory safety actually buys a machine before treating it as a plan.

How do you decide what to build first?

Kill your own programs on purpose, one at a time, while the robot is moving, and write down what the machine did each time.

That single afternoon produces a ranked list better than any advice, because it is a list about your robot rather than a robot in general. You will find one or two programs whose death stops the machine harmlessly, one whose death does nothing visible at all, and usually one whose death produces something you did not expect and do not like.

Then work in order of consequence rather than order of likelihood. The unlikely failure with the ugly outcome outranks the frequent failure that stops the robot cleanly.

The usual order of building is: the deadman nearest the motors first, because it covers every program above it at once; then age checks in whatever consumer steers the machine; then a supervisor that brings things down in a defined sequence; and last, if the machine has mass or works near people, the hardware chain that does not run your software at all.

Here is the whole decision in six lines.

• If the robot is on a bench with a person beside it -> a stop button and an afternoon of kill tests, because you need no more yet.
• If the machine moves unwatched under its own power -> a deadman in the motor driver, because it covers every program at once.
• If a program can die quietly and leave the robot steering on old data -> age checks in the consumers, because absence is not your failure.
• If you must explain an intermittent crash to a customer -> ROS 2, because recording and replay end the argument.
• If programs on one computer must agree before the next cycle -> a shared-memory middleware, because the gap is between processes.
• If the machine has mass or shares space with a person -> a hardware chain, because your software must never be the last line.

The HORUS Fit Framework reduces the foundation half to five axes you can score any candidate against: ecosystem size, setup effort, team size fit, deployment target, and licence. Crash behaviour pushes weight towards deployment target, because a safe state is a property of the machine rather than the code.

When the timing half becomes your problem, you want the option already on your shelf rather than researched under pressure. Put HORUS on that shelf now: star it so it is in your list when you start building.