Apple Tests HALO on Unitree G1, Reporting 91% Success in Unmapped Rooms

A robot trained in virtual homes takes on real rooms

Apple researchers say their HALO humanoid control system completed individual physical tasks with an average success rate of 91% across 360 trials in six real environments that had never been scanned or mapped for it. Running on a Unitree Robotics G1, the system used lessons learned in simulation to navigate unfamiliar spaces, sit on furniture, pass through narrow openings and push boxes.

The results appear in Perceptive Humanoid Motion from Simulation, submitted to arXiv on November 4, 2026. The Apple team's paper describes training with about 300 hours of synthetic examples across 1,000 computer generated home environments, then transferring the system directly to a physical robot.

Longer tasks were harder. In a separate real environment, HALO averaged 70% success across three combinations of actions, each attempted 10 times. The sequence requiring it to duck under an obstacle, approach a target and push a box succeeded in five of 10 attempts.

The practical advance is a connection between understanding a request and moving an entire body through unfamiliar furniture. A command such as sitting on a chair requires the robot to identify the seat, find a route, position its body and maintain balance while lowering itself. HALO divides those responsibilities among three cooperating control layers.

The authors are Siyuan Hu, Abulikemu Abuduweili, Yue Meng, Shipeng Liu, Quentin Leboutet, Chen Chen, Matthias Müller and Vladlen Koltun, all listed with Apple affiliations. They also released code covering data generation, training and deployment, including task planning, motion generation and motion tracking in simulation and on physical robots.

What the 91% result covers

The six individual skills had different success rates. Crouching performed best, while sitting was the least successful of the tested actions:

• Going to a named object: 92%.
• Sitting on furniture: 85%.
• Passing through a partly open door: 93%.
• Crouching beneath an obstacle: 100%.
• Crawling beneath an obstacle: 90%.
• Pushing a box: 88%.

The gap between crouching and sitting was 15 percentage points. Sitting involves more than reaching the right place: the robot must align with the furniture and transfer its weight onto a seat. That places demands on both its visual understanding and its physical controller.

All six evaluation environments were unfamiliar to HALO. The researchers had neither scanned them beforehand nor used them during training. The paper calls this zero-shot transfer, meaning the system moved from simulated training to those physical settings without additional training for each destination.

Demonstrations also show the G1 following a person, sitting on a sofa, moving sideways through a partly open door and ducking or crawling beneath a table. These movements illustrate how the system adjusts its posture to the space around it rather than simply walking toward a destination.

Why longer instructions are harder

HALO's three combinations of tasks produced a wider spread of results. Passing through a door and then going to a target succeeded in all 10 trials. Crouching and then sitting succeeded in six. Crouching, approaching a target and pushing a box succeeded in five.

Together, those results amount to 21 successful sequences out of 30 attempts, or 70%. That is 21 percentage points below the average for individual tasks, although the sequence tests used a different setting and a smaller set of trials.

A sequence creates more opportunities for trouble because the outcome of one movement becomes the starting position for the next. Emerging from beneath an obstacle, for example, must leave the robot able to approach its target and make the next movement.

HALO addresses this through visual checks. Its planner watches video from the robot's perspective, judges whether the current action succeeded and decides whether to move on or retry. The results show that connecting useful skills remains a more demanding problem than performing them separately.

Three layers turn a request into movement

The highest control layer handles task planning. It uses a vision-language model, a type of AI that processes both images and words, to interpret a request alongside what the camera sees. The default model is Qwen3.6-35B-A3B, running locally on a Mac mini carried by the robot.

The planner is invoked every two seconds. If someone asks the robot to sit down without naming a seat, it must choose a suitable object in view. If a table obstructs the route to a chair, it can divide the request into smaller actions: duck beneath the obstruction, approach the chair and sit.

The middle layer turns each subtask into a movement plan. It combines Qwen3.5-2B, whose encoder parameters remain fixed during training, with a diffusion transformer decoder that generates whole body motion. An encoder converts the image and instruction into a representation the movement model can use; the decoder produces the planned poses.

This layer looks roughly five seconds ahead and replans about three times per second. It therefore keeps updating its intended movement from fresh camera observations instead of committing to an entire five second action without checking its surroundings.

The lowest layer, called the motion tracker, converts those plans into joint commands 50 times per second. It combines depth images with information about the robot's own position and movement, helping it follow the plan while accounting for balance, contact and motor limits.

A Mac mini rides on the G1

The physical setup puts a Mac mini with an M4 Pro chip and 64 GB of unified memory in a protective enclosure on the G1's back. That computer runs the more demanding task planning and motion generation models. The G1's own onboard processor runs the motion tracker.

A custom head mount carries an OAK-1W camera. Its color video stream has a resolution of 1280 by 800 pixels, runs at 30 frames per second and covers a field of view of 120 degrees. The wide view helps the system observe nearby furniture and openings as the body moves.

The robot operates untethered, with its software components communicating through ZeroMQ, a messaging system. Its default local configuration can function without an internet connection. HALO can also call a cloud model through an application programming interface when configured to do so.

The division of computing work matches the control hierarchy. Understanding a request can happen less frequently, while keeping a moving robot stable requires much faster updates. The motion generator calculates a new plan while the tracker continues executing the current one, avoiding a stop each time the model needs to compute.

How 1,000 virtual homes supply training examples

HALO's training pipeline tackles a specific shortage: examples that connect what a humanoid sees, what it is asked to do and how its whole body should move. Ordinary video contains images of actions, but training this system requires those images to be paired with instructions and detailed robot motion.

The authors describe the time required by some methods that collect demonstrations through human operation or wearable recording devices:

Because a person must manually act out every episode, an hour of data costs at least an hour of human time.

HALO instead builds matched examples in simulation. The process starts with a task divided into motion segments. Sitting in an armchair, for example, can involve walking forward, turning around and lowering into the seat. Text prompts, route waypoints and key poses guide a motion generation model called Kimodo.

Automated geometric checks then score candidate movements for problems such as sliding feet, bodies passing through the floor, sudden jumps and joints moving beyond their limits. The pipeline retains the strongest candidate before placing the movement in a virtual room.

Infinigen, a procedural scene generator, supplies 1,000 environments with varied layouts and furniture, including living rooms, offices and bedrooms. The system searches for placements where the robot's planned movement and target object fit without colliding with surrounding furniture.

A simulator then renders what the robot would see while performing the action, using a simulated fisheye camera matched to the physical setup. Changes to lighting, exposure, color, contrast, blur and sensor noise make the images more varied. Each resulting example connects an instruction, a camera view and the corresponding body movement.

The computing bill and the human work

Generating the main training corpus took about 48 hours using 250 Nvidia H100 graphics processors. These processors, commonly called GPUs, handle many calculations in parallel, allowing large amounts of simulated data to be produced at once.

The action generator was trained through imitation learning using 64 Nvidia B200 GPUs. In imitation learning, a model learns to reproduce example behavior. The motion tracker used four H100 GPUs for training that combined reinforcement learning with policy distillation.

Reinforcement learning trains a controller through attempts and feedback about how well it performs. Policy distillation transfers learned behavior into another model. In HALO, these methods help the tracker turn planned poses into movements the physical robot can execute.

People still define parts of the process. Adding a new physical task requires writing prompts and designing key poses, and some motions draw on human video or motion capture material. Once those task descriptions and constraints exist, the pipeline can reuse them across many generated rooms.

The shift is therefore in how demonstrations are expanded. Instead of requiring a person to perform every episode, HALO spends computing resources generating and checking many variations of a task.

Vision still leaves a contact problem

HALO currently relies mainly on vision and cannot directly sense contact, according to the researchers. Performance can decline during tasks involving substantial physical contact when an object leaves the camera's view or becomes hidden behind something else.

Pushing a box illustrates why this matters. The robot needs to keep its body positioned correctly relative to the box as both move. A camera can provide information about visible position, while touch sensing could provide information about contact that remains useful when the view is blocked.

The researchers identify tactile sensing as one possible direction for further work. They give no timetable for adding it. The current results put a concrete boundary around the challenge: HALO can perform several visually guided skills in unfamiliar rooms, while longer sequences and interactions involving contact remain less reliable.

What the research changes

Apple's study brings training data generation, language based planning and physical control into one system that runs on a humanoid in rooms it has not encountered before. Its strongest result is the transfer from 1,000 simulated environments to 360 physical trials without preparatory mapping of the six test locations.

The released code makes the pipeline available for further research across simulation and hardware. Its scope includes the machinery for producing training examples as well as the models and software needed to generate, track and coordinate movements.

For a person asking a robot to sit down or move a box, reliability depends on the full chain working together: recognizing the right object, reaching it, making contact and completing the movement. HALO's individual task results show progress along that chain. Its sequence results identify where more work is needed before several actions can be joined with the same consistency.

Key Points

• Apple researchers submitted Perceptive Humanoid Motion from Simulation to arXiv on November 4, 2026.
• HALO achieved an average success rate of 91% across 360 individual task trials in six unfamiliar, unmapped real environments.
• Three combinations of tasks averaged 70% success across 30 trials in another real environment.
• The system combines task planning, motion generation and a tracker that issues joint commands 50 times per second.
• Training used about 300 hours of synthetic examples across 1,000 generated environments.
• A Mac mini carried by the Unitree G1 supports local operation without an internet connection.
• Human task design remains part of training, and limited contact sensing remains a weakness.