How to Evaluate a Robotics Framework Nobody Has Heard Of

Compare the unknown framework against ROS 2 on three things you can check in a week: scope, maintenance, and the cost of leaving. A framework that states plainly what it does not do, as HORUS does, is easier to judge than one that claims everything. The verdict flips when the unknown option would have you rewriting drivers and navigation that ROS 2 already ships. The rest of this post is for a team that has found a promising project with a small community and has to justify the bet to somebody.

Somebody on the team found it. A repository with a clean readme, an idea that matches the shape of your problem, and a commit history that has not gone quiet. You read the docs on a Friday and by Monday you were sketching how your robot would look if you built on it.

Then the questions started. How many people actually use this. What happens when the one maintainer takes a new job. Your lead asks whether you can get support and you cannot honestly say yes. Somebody points out that every robotics job advert names the same well-known stack, and that hiring against an unknown one is a tax you pay forever. Somebody else points out that the well-known stack is the reason your last project ran late.

You have been round this loop three times and each lap produces opinions rather than evidence. Nobody has said no. Nobody will say yes either, because there is no way to be wrong quietly. What you need is not another discussion. It is a way to be wrong cheaply.

How do you evaluate a robotics framework nobody has heard of?

Judge an unfamiliar framework on three things you can establish inside a week: what it refuses to do, who is keeping it alive, and what leaving would cost you. Popularity is a proxy for all three, which is why people reach for it, but it is a lagging proxy and a poor one for anything younger than a few years. A project with a small following can still be a sound bet when the scope is narrow enough that one person could read all of it, the maintenance has a name attached, and the code you write on top would survive a move.

The trap is treating the evaluation as a technical exercise. Do that and you end up comparing features, and features are the part maintainers control. What you actually want to know is what happens on a bad day. The day the maintainer changes job. The day you hit a bug nobody else has hit. The day a customer asks who supports this and you have to answer in one sentence. Get those answers and the technical comparison mostly settles itself, because you will have found out whether anybody is home.

What is a robotics framework, in plain terms?

A robotics framework is the set of decisions somebody else made about how the parts of your robot talk to each other, so that you do not have to make them. How the camera program hands a picture to the decision program. What happens when one of those programs dies mid-motion. Where configuration lives, how everything starts at once, how you look at what went past when something went wrong.

You could make every one of those decisions yourself, and for a robot with two moving parts you probably should. The value of a framework is not that its answers are better than the ones you would invent. It is that its answers are already written down, already argued over, and already understood by anybody you hire.

The word gets used loosely, which matters here. Some projects are really just a messaging layer. Some are a whole distribution with drivers, simulators and tools attached. Some are a vendor's box of code for one machine. Those are three bets with three different exits, and the difference between a framework, middleware and an SDK is worth being precise about before you compare anything to anything.

What are your actual options once you look past the default?

You have six realistic options, and only two of them are frameworks in the sense people usually mean. The first is ROS 2, the default, with the largest collection of drivers, navigation, planning and visualisation tools in the field and a hiring pool to match. The second is a smaller open-source middleware with a deliberately narrow scope, which is where HORUS sits alongside ROS 2 as a different kind of answer: an open-source real-time robotics middleware for Rust, Python and C++ where all three languages share the same shared-memory ring buffers, so messages between processes on one machine are not serialised, published under Apache-2.0, validated in simulation, and carrying no mapping or navigation packages of its own. The third is your robot vendor's own SDK, which usually works beautifully on that vendor's hardware and nowhere else.

The other three are not frameworks at all. You can take a plain messaging layer and write the structure around it yourself. You can write the whole thing from scratch. Or you can license a commercial platform and buy support along with the code.

How do the options compare when you line them up?

Line them up by what each option assumes about you and the list gets short quickly. Read the last column first, because most teams strike three rows on the spot and the real argument is between the two that survive.

Does the size of your team change which frameworks you can consider?

Yes, and not in the direction most people expect. A two-person team can reasonably bet on a project nobody has heard of, because two people can read all of it, and because the thing that sinks a pair is not an unknown framework but a framework whose weight they cannot carry. A twenty-person team has the opposite constraint: whatever you choose has to be teachable on a Tuesday to somebody who started on Monday, and every hour spent explaining an unusual choice is an hour not spent shipping.

The middle is the hard case. Five to eight engineers is large enough that no single person reads everything and small enough that nobody is spare to maintain a fork. Teams that size should be strictest about scope: an unknown framework is defensible when one engineer could hold all of it in their head, and indefensible when it is broad enough to have dark corners of its own.

Whatever size you are, write down who would fix a bug in the framework itself. If the answer is nobody, that is the finding. How small teams ship faster than big ones is the wider version of that trade.

Does your hardware narrow the list before you start?

Yes, and hardware narrows it further than anything else on this page. A robot that is one small board running several programs has a different problem from a robot with a compute box, a laptop and a handheld controller talking over a network. Shared memory only exists inside one machine, so any stack built around it answers the first robot and says nothing about the second. A microcontroller changes the question again, because most of these options assume an operating system that a microcontroller does not have.

Vendor hardware narrows it hardest of all. If your arm or your humanoid ships with an SDK, and the vendor's support ends where that SDK ends, then you are choosing what sits above the SDK rather than choosing a replacement for it.

So write down your boards, what runs on each, and which links cross a physical wire. Half the candidates on your shortlist will turn out to be answering a question your robot does not ask. Whether your robot should run one computer or several is worth settling before the framework question rather than after it.

How does your delivery date change the answer?

A near deadline argues for whatever your team already knows, almost regardless of fit. Frameworks cost their setup once and pay it back over years, so a three-month deadline gives the payback no time to arrive. Betting on the unknown option is a bet on the second year, and if there might not be a second year, take the boring choice and be glad of it.

The reverse is also true and gets said far less often. If you are early enough that nothing is built yet, the switching cost you are agonising over is currently zero, and it will never be lower than it is this week. Teams almost always run this evaluation at the worst possible moment, eighteen months in, with a robot that works, when every option means rewriting something that already does its job.

So the date question has two halves: what is due soon, and how much is already built. A team with a deadline and an empty repository can afford an experiment that a team with a mature codebase cannot. Why middleware decisions are so hard to reverse explains why that asymmetry is so steep.

What skill level does an unproven framework assume?

An unproven framework assumes somebody on the team can debug below their own code, and that is the real dividing line. With a well-known stack, an error message goes into a search engine and comes back with a stranger's answer from four years ago. With an unknown one, the error message goes into the source. You do not need an expert for that. You need one person who is not frightened by unfamiliar code and will follow it until it makes sense.

The second requirement is being able to tell your bug from their bug. Teams new to a language or to systems work tend to blame the framework for their own mistakes, then blame themselves for the framework's, and lose a fortnight to the confusion before anybody notices the pattern.

If your team is learning robotics and a new language at the same time, put the unknown framework later in the plan. Learn on the option with the most written about it, then move deliberately once you can tell the layers apart. Whether ROS 2 stays useful if you end up elsewhere covers what carries across.

What does a bad framework bet look like a year later?

A bad bet looks like a fork you never meant to maintain. Somebody hit a limitation, patched the framework locally to get past a demo, and now your build depends on a branch that exists on one laptop and in one person's memory. Upgrades stopped happening, because upgrading means redoing the patch. Nobody wrote that decision down. It happened on a Thursday.

The second symptom is a build that only works on the machine where it was first set up. New engineers lose their first week to environment problems, so the team quietly stops putting new engineers near that part of the system, so one person becomes the only person who can touch it.

The third symptom is quieter than both. Questions stop being asked out loud, because asking means admitting you cannot answer them yourself and there is nowhere to ask. A year in, you can no longer tell whether the project you bet on is healthy, because nobody has looked in months. Telling a healthy open-source project from a dead one is a check to repeat on a calendar, not one you do once.

What do you give up by choosing something nobody has heard of?

You give up answers, and answers are most of what an ecosystem actually is. The well-known stack has a decade of questions already asked, a driver for the sensor you just bought, a tutorial for the thing you are stuck on tonight, and a room full of people who have already made your mistake. Choose outside it and every one of those becomes work you do yourself.

You give up a hiring shortcut as well. Candidates arrive knowing the default, and a job advert naming something unfamiliar gets fewer replies whatever its merits.

You also give up a kind of credibility with people who are not engineers. An investor, a customer or a safety reviewer recognises the well-known name and does not recognise yours, and you will spend meetings on that gap rather than on the robot.

None of this makes the unknown option wrong. It makes the unknown option something you should be getting a specific, nameable benefit from, one worth all three costs. If you cannot name the benefit in a sentence, you already have your answer.

When is ROS 2 the better choice?

Most of the time, and the cases are easy to recognise. If your robot needs mapping, navigation, motion planning or a visualiser, ROS 2 hands you those and every alternative has you writing them. If you buy a sensor, the driver probably exists for ROS 2 and probably exists nowhere else. If you intend to hire, ROS 2 is what candidates already know. If your robot is split across machines or reports to a fleet service, ROS 2 was designed for that, and a shared-memory design such as HORUS is not the answer there, because shared memory stops at the edge of one computer.

Research collaboration is another clear case. If you publish, share code with another lab, or build on somebody's released work, sitting outside the common stack costs you the collaboration itself, which is usually the point of the work.

The honest summary is that the default is the default for good reasons. Choose against it when you have a specific problem it handles badly, not because it is large or because its setup annoyed you in week one.

Is a project with a small community automatically too risky?

No, and here is why: community size measures adoption, not health, and the two come apart in both directions. A project with many users can be maintained by one exhausted volunteer. A project with few users can have a small paid team and a scope narrow enough that they actually finish things. What matters is whether the work being done matches the work the project promises.

The checks are practical and take an afternoon. Are issues answered, even briefly and even to say no. Does the changelog record decisions rather than only fixes. Does the project refuse features, which is the strongest available sign that somebody is steering. Are there names and an organisation behind the commits, or a rotating cast of drive-by contributors and nobody in the chair.

There is a floor, though. When a project's scope is broader than its maintainers can cover, small becomes fatal, because the uncovered parts are exactly where you will end up living. Match scope against staffing and the size question mostly dissolves. Who actually maintains the software you depend on is that check with receipts attached.

Does a permissive licence mean you are safe to ship?

Partly, but not the way you think. A permissive licence such as Apache-2.0 or MIT settles a legal question and nothing else whatsoever. It tells you that you may ship a product containing the code, keep your own source private, and owe nobody a copy of your changes. That is genuinely important, and it is a small slice of the risk you were actually worried about.

What a licence does not give you is a maintainer, a security contact, a release cadence, or anybody obliged to answer the phone when your robot misbehaves in a customer's building. Nor does a licence stop the project changing direction, going quiet, or licensing future versions differently, although the version you already hold stays yours on the terms you received it.

The practical move is to treat the licence as a gate rather than as an argument. Fail the gate and stop reading. Pass it and begin the real evaluation, which is about people and scope. What robotics licences actually allow you to sell covers the gate itself properly.

How do you actually decide?

Decide with a two-week spike and a written exit plan, not with another meeting. In the first week, build the smallest thing that touches the part of the robot the framework would own: one sensor in, one motion out, running on the board you will actually ship, not on a laptop. In the second week, break it on purpose. Kill a process mid-motion, unplug the sensor, start things in the wrong order, and watch what the framework tells you while it is going wrong. Frameworks are pleasant on a good day and reveal themselves on a bad one.

Then write the exit plan before you commit to anything. Name the two or three places where the framework's assumptions would reach into your own code, and decide now how you would keep those behind an interface you control. A framework you could leave in a month is a fundamentally different bet from one you could never leave, and the difference is mostly your design discipline rather than the framework's doing.

Here is the whole decision in five lines.

• If you need navigation, mapping and sensor drivers now -> ROS 2, because writing those yourself is a year you did not plan for.
• If your robot is one board and its large data never leaves that board -> a shared-memory middleware, because that is the gap the default leaves open.
• If your hardware arrives with an SDK you cannot replace -> build above the SDK, because vendor support ends where vendor code ends.
• If nobody on the team will read the framework's source -> the well-known option, because you will need to read it eventually.
• If you cannot yet say what leaving would cost -> run the two-week spike, because that answer is the entire decision.

The HORUS Fit Framework puts this on five axes you can score any candidate against: ecosystem size, setup effort, team size fit, deployment target, and licence. For an unfamiliar project, ecosystem size and setup effort do most of the work, deployment target decides whether the project is even answering your question, and the licence is a gate rather than a score.

If your shortlist ever narrows to same-machine data on a single board, the useful thing is having the option on your shelf before the deadline arrives. Star HORUS so it is in your list when you start building.