6.6 KiB

Raw Blame History

Current State — 2026-02-15

Handoff document for continuing work from another machine.

What's Working

MuJoCo Playground RL Training (Complete)

Location: GB10 (~/mujoco_training/)
Result: 200M-step locomotion policy trained. Reward: -6.4 → +12.1
Time: 3:08:37, ~17.9K steps/sec on Blackwell GPU
Checkpoint: ~/mujoco_training/checkpoints/G1JoystickFlatTerrain-20260215-090148-full/params.pkl
Scripts: train_g1.py, play_g1.py (untested), monitor.sh

GR00T-WBC Simulation (Complete)

Location: GB10 (~/GR00T-WholeBodyControl/)
Status: Walking robot in MuJoCo sim, keyboard controlled (w/s/a/d), 50 Hz, 3.5ms/iter
Launch: ./launch_sim.sh or python3 run_g1_control_loop.py
Viz: NoMachine virtual desktop or web MJPEG viewer (launch_with_web_viewer.py)

Network Bridge GB10 ↔ G1 (90% — needs final verification)

GB10 IP: 10.0.0.68 (primary) + 192.168.123.100/24 (secondary on enP7s7)
Ping: <1ms to 192.168.123.161 (locomotion) and .164 (Jetson)
DDS confirmed alive inside robot: 342 multicast packets/4s from locomotion computer
Root cause found: UFW firewall was blocking incoming DDS multicast
Fix applied: User disabled UFW (sudo ufw disable)
NOT YET VERIFIED: Robot was disconnected before we could confirm rt/lowstate data flows end-to-end

What's Next (In Priority Order)

1. Verify DDS Connection (5 min)

When robot is reconnected to the network:

# On GB10:
sudo ip addr add 192.168.123.100/24 dev enP7s7   # if not persistent
sudo ufw status   # confirm disabled or allows 192.168.123.0/24
cd ~/GR00T-WholeBodyControl && source .venv/bin/activate
python3 /tmp/dds_test.py

This tests ping → DDS multicast → SDK rt/lowstate in sequence.

2. Test GR00T-WBC on Real Robot (30 min)

Once DDS works:

Put robot on harness/stand
Enter debug mode: L2+R2 on wireless remote while robot is in damping state
Launch: ./launch_real.sh (runs --interface real, auto-detects enP7s7)
Keyboard: ] = enable walk, w/s = fwd/back, a/d = strafe, q/e = rotate
WARNING: No wireless remote control under GR00T-WBC — keyboard only

3. Install xr_teleoperate for Vision Pro (1-2 hrs)

Fastest path to telepresence — Vision Pro connects via Safari (WebXR), no app needed:

# On GB10:
cd ~ && git clone https://github.com/unitreerobotics/xr_teleoperate.git
# Follow their install instructions (needs Pinocchio, TeleVuer)
# Generate SSL certs for Vision Pro HTTPS connection
# Vision Pro opens: https://<gb10-ip>:8012

Limitation: xr_teleoperate bypasses GR00T-WBC and uses the stock controller for legs.

4. Build VisionProTeleop → GR00T-WBC Bridge (2-4 hrs)

Higher quality path — native visionOS app with RL-based balance:

Install VisionProTeleop's "Tracking Streamer" app on Vision Pro (App Store)
pip install avp_stream on GB10
Write bridge (~100-200 lines Python): avp_stream → wrist poses → Pinocchio IK → ControlPolicy/upper_body_pose ROS2 topic
Run alongside GR00T-WBC control loop

5. Unified WBC Training (Saved for Later)

Plan at plans/eager-shimmying-raccoon.md. Trains a policy that handles walking + arbitrary arm poses + balance simultaneously. ~400M steps, ~5.5 hrs on GB10.

Key Architecture: GR00T-WBC

LOWER BODY (12 DOF legs)              UPPER BODY (17 DOF waist+arms)
  RL neural network (ONNX)               Open-loop interpolation
  Balance.onnx / Walk.onnx               Targets from external source
  Auto-switches based on                 Receives via ROS2 topic:
  velocity command magnitude             "ControlPolicy/upper_body_pose"
           \                                    /
            --> Merged 29-DOF joint positions --> DDS rt/lowcmd --> Motors

Integration point for ANY telepresence/mocap system: Publish to ControlPolicy/upper_body_pose ROS2 topic with:

target_upper_body_pose: 17 joint angles (3 waist + 7 left arm + 7 right arm)
navigate_cmd: [vx, vy, wz] velocity command (optional, overrides keyboard)

Vision Pro Telepresence Options

Path	What	Needs App?	Uses GR00T-WBC?	Status
xr_teleoperate	WebXR via Safari	No	No (stock SDK)	Not yet installed
VisionProTeleop	Native visionOS app (MIT)	Yes (App Store)	Yes (via bridge)	Not yet installed
iPhone streamer	Socket.IO replication	Custom app	Yes (built-in)	Protocol documented

User has: Apple Vision Pro + Meta Quest 3 Decision: Vision Pro selected as primary device

Files on GB10

File	Purpose
`~/GR00T-WholeBodyControl/`	GR00T-WBC (patched for aarch64)
`~/GR00T-WholeBodyControl/launch_real.sh`	Real robot launcher (`--interface real`)
`~/GR00T-WholeBodyControl/launch_sim.sh`	Simulation launcher
`~/GR00T-WholeBodyControl/launch_with_web_viewer.py`	Sim + MJPEG web viewer
`~/GR00T-WholeBodyControl/gr00t_wbc/.../g1_29dof_gear_wbc_real.yaml`	Config copy with INTERFACE=enP7s7
`~/mujoco_training/train_g1.py`	MuJoCo Playground RL training
`~/mujoco_training/play_g1.py`	Policy visualization (untested)
`~/mujoco_training/checkpoints/...full/params.pkl`	Trained locomotion policy
`/tmp/dds_test.py`	Quick DDS connectivity test
`/tmp/keysender.py`	ROS keyboard publisher for remote terminals

Credentials & Network

Target	Address	User	Password
GB10	10.0.0.68	mitchaiet	(SSH key)
G1 Jetson	192.168.123.164	unitree	123
G1 Locomotion	192.168.123.161	—	Not SSH-accessible

GB10 interface: enP7s7 (primary: 10.0.0.68 DHCP, secondary: 192.168.123.100/24 manual) DDS: Domain 0, CycloneDDS 0.10.2, ChannelFactoryInitialize(0, "enP7s7") Firewall: UFW was disabled. May need re-disabling after reboot.

Context System Updates Made This Session

context/networking-comms.md — Added §6: GB10 ↔ G1 DDS bridge (topology, setup, debugging findings, SSH creds)
context/open-questions.md — Updated DDS relay question, updated Vision Pro mocap question, added "Vision Pro Telepresence Integration (Phase 5)" section, documented wireless remote answer
context/motion-retargeting.md — Added §6: Apple Vision Pro telepresence paths (xr_teleoperate, VisionProTeleop, GR00T-WBC integration point)
context/learning-and-ai.md — §8 MuJoCo Playground pipeline (added in prior session, still current)
CLAUDE.md — Added Phase 4 to history
plans/eager-shimmying-raccoon.md — Unified WBC training plan (from prior session, still current)

6.6 KiB Raw Blame History