unitree-g1/phases/phase-2-mocap-balance.md

Phase 2: Motion Capture + Robust Balance Context Expansion

Date: 2026-02-13 Objective: Expand the knowledge base to deeply support development of motion capture retargeting and robust push recovery on the G1, both independently and combined.

User Goal

The user wants to:

1. Push the G1 and have it stay standing — robust push recovery during all activities
2. Replay motion capture data on the G1
3. Combine both — mocap playback with always-on balance

User has an EDU variant with Jetson Orin NX, dexterous hands (currently being repaired), running as 29-DOF body.

Research Conducted

Three explore agents investigated:

Agent 1: Robust Balance & Push Recovery

• Confirmed stock G1 controller has light push recovery (arXiv:2505.20619)
• Perturbation curriculum training is the primary method for enhanced push recovery
• Multiple papers validate push recovery on REAL G1 hardware
• Residual policy approach allows enhancing balance without replacing stock controller
• Control Barrier Functions provide formal safety guarantees (validated on G1)

Agent 2: Motion Capture Retargeting

• Three approaches: IK-based (Pinocchio), optimization-based (CasADi), RL-based (BFM-Zero)
• AMASS dataset pre-retargeted for G1 exists on HuggingFace
• BFM-Zero has open-source motion tracking mode validated on G1
• XR teleoperation provides real-time mocap via Vision Pro / Quest 3
• SMPL body model is the standard format; mapping SMPL→G1 joints is well-defined

Agent 3: Whole-Body Control Frameworks

• GR00T-WBC (NVIDIA) is G1-native, open-source, designed for exactly this use case
• Decouples locomotion (RL policy, lower body) from task (upper body) control
• H2O, OmniH2O, HumanPlus all prove combined mocap + balance works on humanoids
• SoFTA provides slow-fast decoupled control for loco-manipulation
• Pinocchio + TSID offers model-based alternative to RL-based WBC

Key Architectural Decision: Three Approaches

Approach A: Overlay (Residual)
├── Keep stock locomotion controller
├── Control arms via rt/lowcmd
├── Optional: residual corrections on legs
├── Risk: Low | Authority: Limited
└── Best for: Initial experimentation

Approach B: GR00T-WBC (Recommended)
├── Replace stock with trained RL locomotion policy
├── WBC coordinates upper and lower body
├── Integrates with LeRobot for data collection
├── Risk: Medium | Authority: Full
└── Best for: Production mocap + balance system

Approach C: Full Custom RL Policy
├── Single policy handles everything
├── BFM-Zero approach — motion tracking + balance unified
├── Must handle all edge cases
├── Risk: High | Authority: Maximum
└── Best for: Advanced deployment after extensive sim validation

What Changed

New Context Files (3)

• context/whole-body-control.md — WBC paradigms, GR00T-WBC, Pinocchio/TSID, architectural approaches (overlay vs. replacement), H2O/OmniH2O/HumanPlus
• context/motion-retargeting.md — Retargeting problem, IK/optimization/RL approaches, motion sources (AMASS, CMU, XR, video), SMPL joint mapping, full pipeline
• context/push-recovery-balance.md — Ankle/hip/stepping strategies, perturbation curriculum, residual policies, CBFs, always-on architecture, fall recovery, metrics, development roadmap

Updated Context Files (7)

• context/locomotion-control.md — Added §8 Custom Policy Replacement, cross-references to new topics
• context/learning-and-ai.md — Added §5 Motion Tracking Policies, §6 Residual Policy Learning, §7 Perturbation Curriculum
• context/manipulation.md — Expanded §4 with GR00T-WBC reference, added mocap-based manipulation note
• context/simulation.md — Added perturbation testing, motion retargeting validation, AMASS dataset note
• context/safety-limits.md — Added §8 CBFs for Balance, §9 Fall Risk Assessment for Custom Control
• context/equations-and-bounds.md — Added support polygon, CBF, kinematic scaling, residual policy equations
• context/open-questions.md — Added 6 new motion capture + balance open questions

Glossary Expansion

• Added ~20 new terms across 4 new categories:
  • Whole-Body Control: whole_body_control, task_space_inverse_dynamics, operational_space_control, centroidal_dynamics, qp_solver, groot_wbc
  • Motion Retargeting: motion_retargeting, mocap, amass, smpl, kinematic_scaling, inverse_kinematics
  • Push Recovery: push_recovery, ankle_strategy, hip_strategy, stepping_strategy, residual_policy, control_barrier_function, support_polygon, perturbation_curriculum
  • Frameworks: pinocchio, h2o, omnih2o
• Total glossary: ~57 terms (was 37)

Source Documents Archived (~15)

• Research papers: GR00T-WBC, H2O, OmniH2O, HumanPlus, TWIST, SoFTA, residual policy
• GitHub repos: gr00t-wbc, Pinocchio, LeRobot+GR00T integration
• Datasets: AMASS-G1, CMU Mocap
• Community: mocap retargeting tools survey

CLAUDE.md Updates

• Added 3 rows to Quick Topic Lookup (total: 20 topics)
• Updated Key Concepts Quick Map with new topic hierarchy
• Added Phase 2 to History table

Assessment: Achievability

YES — this is an achievable goal. Key evidence:

1. Multiple papers demonstrate push recovery on real G1 hardware
2. GR00T-WBC is designed specifically for G1 WBC
3. BFM-Zero demonstrates motion tracking on real G1
4. AMASS retarget for G1 already exists
5. The combined approach (balance + tracking) is validated by H2O/OmniH2O/HumanPlus on humanoids

What's Next (Phase 3 candidates)

• Parse G1 URDF for exact link masses/inertias (needed for Pinocchio/TSID)
• Clone and document GR00T-WBC setup process
• Create worked examples in examples/ directory:
  • Example: IK-based retargeting of a simple arm gesture
  • Example: Perturbation testing in MuJoCo
  • Example: GR00T-WBC hello world
• Benchmark GR00T-WBC inference latency on Jetson Orin NX
• Test residual policy overlay on stock controller (is it possible?)