Best Open-Source Projects for LLM-Driven Robotics in 2026

There is no single best project: ROS 2 plus a robot-learning library is the foundation most LLM-driven robots need, with HORUS covering the single-machine case. The projects worth your time solve what a language model cannot: drivers, planning, recording, and a layer that can refuse a bad instruction. The verdict flips when the model and the control loop share one board and copying between them is what makes motion stutter. The rest of this post is for someone who already builds with models and now has to choose which robotics projects to install first.

Your agent works. It reads a scene, picks a tool, produces something sensible, and you have the transcript to prove it. Then you attach it to a robot arm and spend three days on a driver that builds only against one specific version of Ubuntu.

The advice does not help. Every list of projects reads like a directory: a name, a logo, a sentence of marketing, and no indication of which two you actually need and which six exist for somebody whose situation is nothing like yours. Half turn out to be research code with a good video and a repository that has been quiet for a long time. The other half assume you already have a robot, a dataset and a lab full of people to run it.

Then the arm moves, and the confusion changes shape. It hesitates in the middle of a reach and you cannot say whether the model answered late, the camera dropped a frame, or the gripper driver was waiting on something. A machine that ran your training loop without complaining now cannot hold a wrist steady while a policy is thinking.

So you go looking for the right foundation, and every recommendation you find is a count of stars on GitHub.

Which open-source projects should you actually build an LLM-driven robot on?

Start with ROS 2 and one robot-learning library, add a simulator, and install nothing else until something specific breaks. The reason is unglamorous: almost none of the hard parts of a model-driven robot are the model. The hard parts are a driver for the exact depth camera you bought, odometry that does not drift into a doorframe, motion planning that gets the elbow around a table leg, and a recording tool that lets you replay this morning's failure at your desk this afternoon. All of that exists for ROS 2, usually with somebody's video of it running on hardware close to yours, and none of it is the part you are good at. The learning library matters because the tasks people want a model to supervise — folding, wiping, sorting a bin of mixed objects — are exactly the tasks that resist hand-written rules, and those need demonstrations rather than prompts. The simulator matters because your early attempts should not involve a real gripper. Everything else on the popular lists solves a problem you have not met yet.

What does LLM-driven robotics actually mean in practice?

A language model chooses what the robot should do next, and something underneath decides how. That division is the whole subject. The model reads a camera view and an instruction and produces a decision: pick up the blue mug, go to the kitchen, stop and ask. Underneath sits a layer of skills that already work — a reach, a grasp, a drive to a point — each of which re-checks the world as it begins, runs on a clock the model does not control, and reports honestly whether it finished or gave up. Beneath that sits the code holding the joints steady, which keeps running whether or not anything upstairs has replied. When people say a robot is driven by a language model, they almost never mean the model is emitting motor commands. The projects in this field cluster around those three jobs: helping a model choose, supplying skills worth trusting, and carrying messages between them on time. Confusing the layers is the most common reason a promising build stalls, because a fix applied at the wrong layer never holds for long.

What are the real open-source options in 2026?

Eight projects or project categories turn up repeatedly in working model-driven robots, and they occupy different layers rather than competing for one slot. ROS 2 is the foundation and deserves to be: navigation, motion planning, transforms, recording and the driver ecosystem are the reason most robots exist at all, and nearly every published example assumes ROS 2 underneath. HORUS, an open-source real-time robotics middleware for Rust, Python and C++, answers a narrower question — one machine, several languages, where a Python process holding a model and a C++ or Rust control loop share the same shared-memory ring buffers instead of copying pictures to each other. LeRobot covers demonstrations, training and evaluation. Open-weight robot policies give you a trained starting point instead of an empty one. A behaviour-tree library hands the model a menu rather than a steering wheel. A simulator gives you failure without consequence, and a recording tool gives you an explanation afterwards. Read the table as a set of situations, not a ranking, and expect a serious project to occupy four or five rows at once.

What should you pick if you are one person with one arm on a desk?

Pick whatever the arm's manufacturer already supports, and postpone the project-selection question by a month. A solo builder loses far more time to setup than to any wrong architectural choice, and the shortest path to a moving robot is the path the vendor has tested. If the arm ships with ROS 2 packages, use them. If it ships with a Python SDK and nothing else, write a script and get the thing moving today. The question of which projects to add becomes real at a moment you will recognise without being told: when two programs both need the camera, or when you want to replay a failure you did not watch. Until then, every hour spent comparing repositories is an hour not spent discovering that your gripper cannot hold the object the entire demonstration was designed around. One caution for people arriving from model work: do not build the interesting layer first. Make the arm do one dull task ten times from a script, write down each way it failed, and treat that list as the specification for everything above it.

What hardware do these projects assume you have?

They assume two jobs that must not interrupt each other: keeping the robot steady, and thinking. The steady job wants a modest board running the control loop, the driver and the limits, carrying nothing that pauses at unpredictable moments. The thinking job wants an accelerator for a local model, or a network connection to a hosted one. Both can live on one machine, and on most desk robots they do — which is exactly where the copying problem appears, with a perception process and a control process on the same board moving images around as though they sat on opposite sides of a building. Training is a separate matter and belongs on a separate machine; nothing about a learning library requires the robot to hold the training run. The placement rule is short. Anything that must happen whether or not the model answers stays on the robot and stays below the model: motor control, limits, the stop path, and the code that notices the gripper closed on nothing. Test the split by unplugging the network mid-task and watching what the arm does next.

How long does this take if you want something working this quarter?

A convincing demonstration takes a few weekends; a robot you would leave running with a colleague in the room takes most of a quarter. The demonstration is quick because the pieces exist: a hosted model, two or three skills, a camera, and your existing habit of wiring services together. What consumes the rest is not the model layer at all. It is making every skill report cleanly whether it finished or failed, because a planner handed the bare word error will retry from a position nobody has described to it. Budget the failure handling and the calibration, not the demo. An order that survives contact with reality: one skill driven from a script this week, a second skill and a refusal path next, the model choosing between the two in simulation after that, then hardware with a person on the stop button. Teams who invert that order get a good video in a fortnight and then spend two quarters discovering what the video was hiding, which is usually the same three failures wearing different costumes.

What do you need to know before you start, and what can you skip?

You need Python, patience with Linux, and one honest picture of what the control loop is doing while your model thinks. You can skip kinematics derivations, control theory and C++ for at least the first month. Four things matter early. First, how a program on the robot is started and how you tell whether it is still running, because half of all robotics confusion is a process that quietly died. Second, what coordinate frames mean, so the word left is unambiguous between the camera and the arm. Third, what happens when a skill stops halfway, since a half-open gripper is a state your planner will meet on a Tuesday. Fourth, the habit of treating model output as untrusted input, the way a web service treats a form from a stranger. The instinct that transfers worst from model work is that a retry costs nothing. On hardware a retry moves mass, and whether Python belongs in the timed parts at all is worth settling before you learn it the expensive way.

What does an LLM-driven robot look like when it goes wrong?

It looks like hesitation rather than a crash, which is why it takes so long to diagnose. The arm reaches, pauses a beat too long with the gripper open, and closes on air. Nothing threw an exception. The log shows a decision, and it shows an image, and the image is stamped after the decision that supposedly used it. A second signature: the robot behaves beautifully for the first few minutes of a session and gets progressively less smooth, because a queue somewhere is filling and nobody is draining it. A third: everything works when you run it from your laptop with the recording tool open, and misbehaves when you close the recorder, which means your timing depended on the extra load in a way no one intended. The common thread is that model-driven robots fail late rather than loudly. If your only instrument is a print statement, you will guess for a week. A recorded run with real timestamps turns three of those guesses into one question, which is why the recording tool belongs in the first install rather than the fifth.

What do people try first, and why does it stop working?

Almost everyone starts by asking the model to produce motor commands or a block of code, which works beautifully on a tidy desk and comes apart the instant something moves. The cause is always the same wearing different clothes: the model writes its answer for the world you described a moment ago, with no clock, no feedback and no way to notice the mug shifted while the sentence was being composed. The second attempt stuffs the entire robot state into one enormous prompt, which grows weekly until the model answers confidently about the wrong object. The third asks the model on every cycle, so the robot pauses to think between corrections and smooth motion becomes a stutter that no amount of prompt work removes. Each fix has the same shape: fewer decisions for the model, more structure around them, and a layer underneath that can refuse. The model picks a tested behaviour, the behaviour re-checks the world as it starts, and the loop keeps running whether or not anything upstairs replies. Why these robots look so much better in demos is largely this rearrangement missing.

What do you give up by assembling your own stack from these projects?

You give up the borrowed robot, and for many projects that is the entire argument. Choosing a lean foundation means you write, or find, the navigation, the motion planning, the coordinate transforms, the driver for a camera whose manufacturer publishes a Windows binary and a shrug, and the visualisation tool you did not know you needed until the arm moved somewhere surprising. You also give up the search results: when a ROS 2 user hits an obscure error, somebody has hit it before and written it down, and when you hit an error in a smaller stack you read the source. You give up part of the hiring pool, since a robotics engineer arrives already knowing ROS 2 and knowing nothing about your particular arrangement. And you give up some conference-shaped credibility, which sounds petty until a customer asks what the robot runs on. What comes back is a machine with fewer moving parts underneath, a build you understand end to end, and nothing installed that you never wanted.

When is ROS 2 the better choice?

ROS 2 is the better choice for most robots that navigate, manipulate or map, and HORUS is not the answer for any of them. If the robot must move through a building, ROS 2 hands you localisation, mapping and path planning that many people have already driven into many walls on your behalf. If the robot has an arm that must avoid its own body, motion planning is a package rather than a research project. If you work with a university group or an existing commercial platform, the code you need to read and the people you need to ask live in one ecosystem, and stepping outside it makes every conversation start from nothing. If your team is large enough that people join and leave, ROS 2 is a line on a CV and your bespoke arrangement is not. And if you are learning robotics rather than shipping a product, learn ROS 2, because the concepts carry everywhere and the tutorials assume nothing about you. Choosing anything else in these cases buys a smaller stack and costs the entire body of work that made robotics tractable in the first place.

Is an open-weight robot policy enough on its own?

No, and here is why: a policy produces motion, and it produces motion for the robot, gripper and viewpoint it learned on, with no opinion about anything outside that. Download a published policy and you inherit somebody else's camera placement, somebody else's gripper geometry and somebody else's idea of what a table looks like. Move the camera and the policy sees a different world. Swap the gripper and the grasps that were tuned for two fingers of one width become approximate. None of this makes published weights useless — starting from a trained policy and collecting demonstrations on your own hardware is far quicker than starting from nothing — but it does mean the policy is an ingredient, not the meal. You still need the driver, the frames, the recording, the skill that stops when the force feels wrong, and something that decides which policy to invoke and when to stop invoking it. What a robot foundation model actually is is worth reading before you plan a quarter around one.

Can you let the model talk straight to the motors?

Partly, but not the way you think. A model can emit a target position, and there are narrow cases where that is correct — a slow arm, a controlled workspace, a person watching. What ends the arrangement is not size but timing. A motor wants a new instruction on a steady rhythm, and a model answers when it answers; put one in charge of the other and the joints receive a command describing a world that has already moved on. The second ending is refusal. Something has to be able to say no — the target is outside the workspace, the force is wrong, the object is not where the picture said — and that judgement cannot live in the thing being judged. The third is debugging: when motion and decision are the same act, you cannot tell a bad decision from a late one. Keep a control loop underneath that runs on its own clock, hand the model a set of skills, and let it choose. That structure is what everything in the layer between model and motor exists to provide.

How do you decide which projects to install first?

Answer one question honestly: does your robot need parts other people have already written? If it navigates a building, avoids obstacles with an arm, or has to work alongside an existing platform, the answer is yes and the decision is finished — take ROS 2, add a recording tool on day one, and spend your attention on the model layer, which is the part nobody can write for you. If the robot is one machine where several languages trade data continuously and the borrowed packages hold no interest, you are choosing on a different axis and a shared-memory middleware fits. If you cannot tell which describes you, that is itself the answer: you are early, so write the script, move the robot, and let the second program that needs the camera decide for you. Two checks settle most remaining arguments. If the model were deleted tomorrow, would the robot still do anything useful? A no means you have a demonstration rather than a machine. And which failure would embarrass you in front of whoever pays for this? Make that one impossible underneath, not merely discouraged above.

Decide by situation rather than by ambition:

• If your robot maps, navigates or plans around obstacles -> ROS 2 first, because those packages are the project rather than a garnish.
• If you have one arm, one script and one week -> the vendor SDK in plain Python, because the project-selection question is not real yet.
• If a model process and a control loop share one board -> a shared-memory middleware, because copying is what makes the motion stutter.
• If your task resists being written down -> a robot-learning library and your own demonstrations, because the data pipeline is the work.
• If you are learning rather than shipping -> ROS 2 and a simulator, because the concepts transfer and nothing gets broken.

When the foundation question becomes real, weigh candidates on the five axes of the HORUS Fit Framework — ecosystem size, setup effort, team size fit, deployment target and licence — and take the option that loses on the fewest, with no scoring involved. If your robot is turning into one machine running a model process beside a control loop, star HORUS on GitHub so it is in your list when you start building, and the companies producing these models are the other half of the same decision.