What to Build First If You Want to Learn Embodied AI

Start with a small desk arm doing one pick-and-place task from your own demonstrations — not a humanoid, not a simulation-only project. The arm teaches what a simulator hides: hardware that lies, data you collected badly, and a loop that must keep time whether or not the model is ready. Choose simulation instead only if you have no bench, no budget and no patience for shipping delays. The rest is for machine learning people picking a first build, and for anyone wondering whether it needs ROS 2, HORUS or one Python file.

You can fine-tune a model. You can read a paper on Monday and have something like it running by Thursday. Then you watched a video of a robot folding laundry, or sorting a bin of parts, and wanted to make one, and every route you found in the next hour contradicted the last one.

One tutorial starts with a humanoid development kit that costs more than your car. One starts with a simulator that assumes a graphics card you do not own and a week of setup before anything moves. One starts with soldering. A forum thread says do not buy hardware until you have trained a policy in simulation; the reply below says simulation taught the author nothing and to buy an arm.

Maybe you already bought the arm, and it has been in its box for five weeks because the first honest step is not obvious and every wrong purchase feels expensive. You are not short of ability. You are short of a first project small enough to finish and real enough to teach you something a notebook cannot.

What should you build first if you want to learn embodied AI?

Build a small desk arm that does one repeatable task from demonstrations you recorded yourself. One object, one starting area, one target, a camera that never moves, and a policy you trained on your own data. That is the whole recommendation, and its narrowness is the point. What you are actually building is not the arm and not the model but the loop: collect data, train, run it on hardware, watch it fail in a way you did not predict, work out which of the three parts was at fault, fix that, repeat. Everything else in the field is that loop with more money attached. If you can run it once, in a week, on a desk, you have learned the actual shape of the work, and every larger project afterwards is a variation you can reason about. If you skip it and start big, you will run the loop for the first time with an audience and a deadline.

What does embodied AI cover when you are choosing a first project?

Embodied AI means behaviour learned from experience in a physical body rather than written down as rules by a person. For a first project, that translates to something small and specific: a model that looks at camera images and produces arm movements, trained on examples of a human doing the task rather than on a specification of how to do it. It does not mean building a robot from parts, it does not mean training a foundation model, and it does not mean writing a controller from equations. Those are all adjacent disciplines that will happily eat your evenings without teaching you the thing you came for. The distinction worth holding on to is that embodied AI names where the behaviour comes from, while the rest of the robot — drivers, cameras, the loop that keeps sending commands — is ordinary engineering that stays your problem regardless of how good the model is. Embodied AI and robotics are not the same thing, and the half you underestimate is the half that costs you the month.

What are the realistic first projects to choose between?

There are eight realistic first projects, and they differ mostly in what they will teach you before they defeat you. A desk arm with one task and one Python program is the default recommendation. A simulation-only manipulation project trades hardware surprises for a faster loop. A small wheeled robot with a camera teaches navigation instead of manipulation. A teleoperation rig teaches data collection, which is the skill most people are worst at. Running somebody else's pre-trained policy on a vendor arm teaches integration and gets you moving fastest. A ROS 2 tutorial platform teaches the vocabulary employers test for. A humanoid kit teaches you about your credit limit. And when your working script eventually splits into a model program and a control program on the same board, you are choosing plumbing: ROS 2 brings the largest collection of existing robotics code anywhere, while HORUS is an open-source real-time robotics middleware for Rust, Python and C++ in which all three share the same shared-memory ring buffers, so messages between processes on one machine are not serialised. Read the table as a description of situations, not a ranking.

What does a first embodied AI project look like when it goes wrong?

It goes wrong quietly, and almost never in the model. The classic version: your policy works beautifully for eleven minutes, then the gripper stops responding, and you cannot tell whether the model, the driver or the cable is at fault because nothing logged anything. The second classic: the policy succeeds on every object you place in the left half of the workspace and fails in the right half, because every demonstration you recorded happened to start on the left, and the policy learned the position rather than the task. The third: everything worked yesterday, the sun has moved, the camera changed its own exposure, and your model has never seen this lighting. The fourth is the one nobody warns you about — your script reads a frame, runs the model, sends a command, and repeats, so the arm's rhythm is set by whatever Python happened to be doing, and the motion looks nervous for reasons no amount of retraining will fix. Three of those four are engineering faults wearing a machine learning costume, which is why the first project's real lesson is telling them apart.

Which first project fits the background you are coming from?

Your background decides which half of the project will hurt, so pick the build that hurts in the useful place. If you come from machine learning, the arm is right, because you already have the training loop and what you lack is contact with a machine that does not behave like a dataset. If you come from web or backend development, start with the pre-trained policy on a vendor arm, since you will learn far more from integrating something that already works than from debugging your own training run and your own hardware simultaneously. If you come from embedded or electronics, invert the advice: use a simulator for the learning half, because your instinct will be to build hardware you do not need. If you are a student with a term and no budget, simulation plus a strong evaluation habit is a real project and produces something you can show. And if you come from large language models and agents, be warned that the reflex to solve problems with a bigger prompt does not survive contact with a gripper — the honest route from Python and language models into robotics is worth reading before you buy anything.

What hardware do you actually need to start?

Less than you think, and the cheap parts matter more than the expensive one. You need a small arm with a supported Python library, and support matters more than the specification — an arm with an active community and working examples will teach you more than a better arm with a PDF for documentation. You need a camera, and a way to clamp it so that it never, ever moves, because a shifted camera invalidates every demonstration you have recorded. You need a bench you can leave set up, since a project you must reassemble each evening is a project you will stop doing. You need a way to cut power quickly, which is worth arranging before the first time you need it. You do not need a humanoid, two arms, a depth camera, a lidar, or a top-end graphics card, and you especially do not need the expensive arm; buy the cheap one, break it, learn what a better one would fix. The cheapest honest way to start building robots goes through the rest of the shopping list.

How long should your first embodied AI project take?

Give it a deadline of a few weekends, and treat overrun as a signal rather than a phase. A first arm project that has not produced one successful autonomous attempt within a month has usually stopped being a project and become shopping — another sensor, another kit, another library that promises to make the last one unnecessary. The honest breakdown surprises people: the model is the fast part, often an afternoon. Getting the arm to move from your own code is a day if the vendor's examples work and a week if they do not. Building a mount rigid enough that your camera stays put is an evening you did not plan for. Collecting demonstrations is dull, takes longer than the training, and is where the quality of your result is actually decided. Debugging why it worked yesterday is the part with no fixed duration at all. Plan for the boring parts to dominate, because they will, and a first project that ends is worth more than an ambitious one that does not.

What do you need to know before you start?

You need Python, the ability to train a small model, and a tolerance for hardware that lies to you. That is genuinely the list. You do not need control theory, you do not need kinematics equations, you do not need C++, and you do not need a robotics degree — all of those are useful later and none of them gates the first project. What actually separates people who finish from people who stall is a debugging habit: changing one thing at a time, writing down what you changed, and being able to say which of the three moving parts you last touched. The specific skill worth building deliberately is doubting your data before doubting your model, because the failure will look like a model problem and will usually be a data problem. If you have never worked with a physical device, expect one new skill: separating a code fault from a device stuck in a bad state, which is usually cured by a power cycle and is annoying the first ten times.

What do you give up by starting with a small desk arm?

You give up everything that involves the robot moving through a building. A fixed arm on a bench teaches manipulation, contact, grasping and data collection, and teaches you nothing about navigation, mapping, localisation or what happens when a wheel slips. You give up multi-machine systems entirely, along with the whole set of problems that appear when parts of a robot live on different computers. You give up the ROS 2 ecosystem, which matters more for your employability than for your project — a portfolio built entirely on a bespoke arm script will impress fewer hiring managers than one that shows you can work inside the tooling everybody else uses, and whether ROS 2 is worth learning for a robotics job is the fuller argument. You also give up long-horizon behaviour: one task with a clear start and end tells you nothing about a robot that must sequence ten steps and recover halfway through. All of those are worth learning. None of them are worth learning first.

When is ROS 2 the better choice?

ROS 2 is the better choice the moment your project is about a robot rather than about a policy. If you want a job in robotics, ROS 2 is the vocabulary of the interviews and the job adverts, and building on it is a straightforwardly better use of a first project than any bespoke alternative. If your robot moves through a space and needs to know where it is, ROS 2 hands you mapping and navigation that hundreds of teams have already debugged. If your hardware has a supported driver, that driver is a ROS 2 package. If you are joining a lab, ROS 2 is what the lab runs, and arriving fluent is worth more than arriving with a clever arm. HORUS is not the answer to any of that: it brings no navigation stack, no driver catalogue and no community of tutorial writers, and a learner who picks it hoping for those will have a thin first month. Start with ROS 2 whenever the ecosystem is the reason you are building at all.

Can you learn embodied AI entirely in simulation?

Partly, but not the way you think. Simulation genuinely teaches the machine learning half, and teaches it faster than hardware ever will: you can run a policy thousands of times overnight, vary the world in ways a bench cannot, and evaluate properly instead of eyeballing three attempts. What simulation cannot teach is the category of failure that defines working robots — a gripper that reports closed with nothing in it, a camera that changes exposure, a driver that stalls for a moment and takes your control rhythm with it. Those are not details; they are the reason robot projects overrun. The practical route is to use simulation for the learning and one real machine for the reality check, and to expect the first transfer to fail in ways your simulator never suggested. Whether you can learn robotics entirely in simulation and why sim-to-real keeps failing both go deeper than there is room for here.

Should your first build be a humanoid?

No, and here is why. A humanoid asks you to solve balance, safety, many more joints and a great deal of expense before you have ever collected a usable demonstration, and none of those problems teach you anything about learned behaviour. The economics are brutal: a humanoid costs many times the desk arm and gives you fewer completed experiments per week, because every run needs a spotter, a clear floor and a reset measured in minutes. The safety story is worse — a desk arm that misbehaves knocks a cup over, and a humanoid that misbehaves falls on something. The learning story is worst of all, since the interesting research on humanoids is being done by teams who solved manipulation first and moved up. If humanoids are genuinely where you want to end up, the fastest route there still starts on a bench with one arm and one task, and the people already there took it.

How do you choose your first project this week?

Choose by what you can start on Saturday without buying anything else. That constraint settles it faster than any comparison, because the project you can begin immediately is the one that survives contact with your actual week. Ask three questions in order. First, do you have hardware within reach — an arm you already own, a lab you can book, a friend with a kit? If yes, use it, whatever it is, and do not upgrade until it has failed you for a specific reason you can name. Second, what do you want to be true in a month: a policy you trained, a robot that drives, or a line on your CV? Those three point at different builds and pretending otherwise wastes the month. Third, is the task you have in mind repeatable — can you reset it in seconds and try again? If not, shrink it until you can, because iteration count is what makes a first project teach anything. Then start, badly, on Saturday.

Decide by situation rather than by ambition:

• If you are a machine learning engineer with no hardware -> a small desk arm and one task, because the loop you are missing only exists near a real machine.
• If you have no budget or no space -> a simulated manipulation task with honest evaluation, because a finished simulated project beats an unopened box.
• If you want a robotics job within the year -> a wheeled robot on ROS 2, because that is the vocabulary interviews are conducted in.
• If you already own an arm gathering dust -> use that arm this weekend, because the best first robot is the one already in the room.
• If your script has split into a model program and a control program on one board -> a shared-memory middleware, because copying camera frames between them is what will cost you the motion.

When two options stay close, weigh them on the five axes of the HORUS Fit Framework — ecosystem size, setup effort, team size fit, deployment target, and licence — and take the one that loses on the fewest. No scores and no numbers: five honest questions about your situation rather than about the software. And if your first project keeps growing toward one machine running a model in Python beside a loop that must not wait for it, star HORUS on GitHub so it is in your list when you start building.