epilectrik
/
unitree-g1
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
5 Commits
1 Branch
0 Tags
327 KiB
 
 
Branch: master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'master'
${ noResults }
unitree-g1/context/teleoperation.md

18 KiB
Raw Permalink Blame History

Teleoperation & Telepresence

Status: Active — xr_teleoperate working with Vision Pro (v0.0.60 + patches, verified 2026-02-18) Evidence tier: T1 (code-verified) unless otherwise noted

1. xr_teleoperate (Unitree Official)

Repository: https://github.com/unitreerobotics/xr_teleoperate Location on robot: /home/unitree/xr_teleoperate/ (standalone clone) and /home/unitree/g1-control/repos/xr-teleoperate/ (submodule of friend's g1-control) Conda environment: tv (Python 3.10, pinocchio 3.1.0, casadi 3.6.7, unitree_sdk2py 1.0.1)

Architecture

Apple Vision Pro (Safari WebXR)
    │
    │ HTTPS + WebSocket (port 8012)
    │ Data: wrist 4x4 SE3 + finger joints + head pose
    ▼
Robot Jetson (10.0.0.64): teleop_hand_and_arm.py
    │ TeleVuerWrapper.get_tele_data() → wrist poses
    │ → Pinocchio + CasADi IK → 14 arm joint angles (7 left + 7 right)
    │ → G1_29_ArmController publishes to DDS
    ▼
DDS topic: "rt/arm_sdk" (motion mode) or "rt/lowcmd" (debug mode)
    │ Motor commands at 250 Hz (internal interpolation from 30 Hz IK)
    ▼
Robot motors (arms only in motion mode; all joints in debug mode)

Two Operating Modes

Aspect Debug Mode (no --motion) Motion Mode (--motion)
MotionSwitcher called? Yes — enters Debug Mode No — skipped entirely
DDS topic rt/lowcmd (full control) rt/arm_sdk (arm overlay)
Joint control scope ALL joints (legs locked at current pos) Arms only; legs under built-in locomotion
Robot can walk? No — frozen in place Yes — via R3 physical controller
Blend weight signal Not used motor_cmd[29].q = 1.0 (tells internal controller to apply arm commands)
Required robot state Any (debug mode takes over) Must be in Regular mode (R1+X on R3 controller)
Exit behavior Arms go home Arms go home with gradual weight ramp (1.0→0.0 over ~2s)

Correct Startup Procedure (--motion mode) [T1]

Prerequisites:

  • Robot powered on, standing
  • R3 physical controller available
  • SSL certs at ~/.config/xr_teleoperate/{cert.pem, key.pem} with correct IPs in SAN
  • rootCA.pem installed and trusted on Vision Pro (Settings → General → About → Certificate Trust Settings)
  • Vision Pro on same WiFi network as robot

Step 1: R3 controller → Regular mode The robot must be in Regular mode BEFORE launching teleop. The teleop script does NOT handle mode switching itself.

From power-on: L2+B (Damping) → L2+D-pad UP (Locked Standing) → Regular mode:

Variant Regular Mode Button Waist DOFs Notes
G1 Base (23 DOF) R1 + X 1 waist joint
G1 EDU (29 DOF) R1 + Y 3 waist joints R1+X is disabled on EDU hardware

Running mode (R2+A) is NOT supported — causes lower body joints to lock rigid during teleop (xr_teleoperate Issue #251). [T1 — Verified 2026-02-18]

Step 2: Launch teleop on robot

conda activate tv
cd ~/xr_teleoperate/teleop
python3 teleop_hand_and_arm.py --arm=G1_29 --motion

Wait for: Press [r] to start syncing the robot with your movements.

Step 3: Connect Vision Pro Open Safari → https://10.0.0.64:8012 → tap "Virtual Reality"

Step 4: Align arms Position your arms matching the robot's initial pose (arms at sides) to avoid sudden movement.

Step 5: Press r in terminal Starts the IK control loop. Arms begin tracking hand movements. Velocity ramp-up over 5 seconds.

During operation:

  • Walk the robot using the R3 physical controller
  • Press s to toggle recording (if --record enabled)
  • Press q to exit (arms return to rest over ~5 seconds via weight ramp-down)

Running from GB10 (offboard)

When running on GB10 instead of the robot itself, must specify the network interface:

python3 teleop_hand_and_arm.py \
    --arm G1_29 --motion \
    --network-interface 192.168.123.100 \
    --img-server-ip 192.168.123.164

Also requires LD_LIBRARY_PATH for OpenSSL compatibility:

export LD_LIBRARY_PATH=$HOME/miniforge3/envs/tv/lib:$LD_LIBRARY_PATH

Key CLI Arguments

Argument Default Description
--arm required G1_29, G1_23, H1_2, H1
--ee none End-effector: dex3, dex1, inspire_ftp, inspire_dfx, brainco
--motion false Use rt/arm_sdk with built-in locomotion (vs rt/lowcmd debug)
--frequency 30 IK control loop Hz
--input-mode hand hand (tracking) or controller
--display-mode immersive immersive, ego, pass-through
--network-interface none DDS network interface IP (required when offboard)
--img-server-ip none Camera image server IP (teleimager on PC2)
--sim false Isaac Sim mode (DDS domain 1)
--record false Record episodes for imitation learning
--headless false No XR visualization
--ipc false Enable ZMQ IPC for external control (used by g1-control server.py)

SSL Certificate Setup [T1]

The televuer WebXR server requires HTTPS (WebXR mandates secure context). Certificate resolution order:

  1. Environment variables: XR_TELEOP_CERT and XR_TELEOP_KEY
  2. User config: ~/.config/xr_teleoperate/cert.pem and key.pem
  3. Fallback: bundled in televuer package directory

Current certs on robot (regenerated 2026-02-16):

  • SAN includes: 10.0.0.64 (wlan0), 192.168.123.164 (eth0), 192.168.1.21, 127.0.0.1
  • Issuer: CN = G1 Robot Local CA
  • Valid until: 2028-05-21
  • rootCA.pem served via http://10.0.0.64:9090/rootCA.crt for Vision Pro download

To regenerate certs for a new IP:

# Generate CA
openssl req -x509 -new -nodes -newkey rsa:2048 -keyout rootCA.key -out rootCA.pem -days 825 -subj "/CN=G1 Robot Local CA"
# Generate server key + CSR with SAN
openssl req -new -nodes -newkey rsa:2048 -keyout key.pem -out server.csr -config san.cnf
# Sign with CA
openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out cert.pem -days 825 -extfile v3_ext.cnf

Safari/visionOS Certificate Trust (CRITICAL) [T1 — Verified 2026-02-18]

Safari on visionOS treats HTTPS and WebSocket (wss://) certificate trust separately. Clicking "Accept" on Safari's HTTPS cert warning only trusts the page load — WebSocket connections are silently rejected with no error, no prompt. This is documented Apple/WebKit behavior, not a bug.

The ONLY way to get wss:// working with self-signed certs on Vision Pro:

  1. Generate certs using a CA structure (rootCA.pem → cert.pem), NOT bare self-signed
  2. Serve rootCA.crt via plain HTTP with MIME type application/x-x509-ca-cert
    • python3 -m http.server does NOT work — it serves .pem with wrong MIME type and Safari won't trigger profile install
    • Use a custom handler or serve as .crt with correct Content-Type header
  3. On Vision Pro: download from http://10.0.0.64:9090/rootCA.crt
  4. Install profile: Settings → General → VPN & Device Management
  5. Enable full trust: Settings → General → About → Certificate Trust Settings → toggle ON "G1 Robot Local CA"

If Safari has accumulated stale state from debugging (old certs, cached JS, partial profile installs), a factory reset of the Vision Pro may be necessary. After reset, do a clean cert install on the fresh OS.

Vuer Package Patches (Required for v0.0.60) [T1 — Verified 2026-02-18]

The vuer pip package (v0.0.60) has bugs that must be patched after install. These patches are lost on every pip install.

1. JS Port Fix — hostname → host in WebSocket URI construction

The getDefaultSocketURI() function in the bundled JS client uses window.location.hostname for HTTPS, which strips the port. The WebSocket tries port 443 instead of 8012.

Fix: In ALL chunk files containing wss:// under vuer/client_build/assets/chunks/:

# Find affected files
grep -l "wss://" .../vuer/client_build/assets/chunks/*.js
# In each file, replace:
wss://${window.location.hostname}  →  wss://${window.location.host}

On v0.0.60 this affects 4 chunk files: chunk-Bf98F3Ua.js, chunk-BU6qPyb1.js, chunk-Dd3xtWba.js, chunk-DmvjxeUa.js. Filenames change per version.

2. aiohttp SSL Assertion Fix

In aiohttp/base_protocol.py, the resume_writing() method has assert self._paused which crashes on SSL WebSocket connections.

Fix: assert self._paused → if not self._paused: return

3. Vuer version compatibility

Version visionOS Safari client WebSocket stability Status
v0.0.40 Client JS does NOT work (zero WS connections) N/A Do NOT use
v0.0.60 Client JS works (with port fix) Connects, known disconnect bug (vuer #85) Use this

Community reports (vuer #85, televuer #1, xr_teleoperate #241, #242) confirm v0.0.60 has WebSocket disconnect issues. With a factory-reset VP + clean cert install, connections are stable.

IK Solver Details [T1]

  • Library: Pinocchio 3.1.0 + CasADi 3.6.7 (constrained nonlinear optimization)
  • URDF: ~/xr_teleoperate/assets/g1/g1_body29_hand14.urdf (29 DOF body + 14 DOF Dex3 hands)
  • Reduced model: Locks all non-arm joints → 14 DOF (7 per arm)
  • End-effector frames: 0.05m offset along local x-axis from wrist yaw joint
  • Cost function: 50 × translational_error + rotation_error + 0.02 × regularization + 0.1 × smoothness
  • Max iterations: 30, tolerance 1e-4, warm-start enabled
  • Model cache: g1_29_model_cache.pkl (172 KB, avoids slow URDF parsing)
  • Control rate: 30 Hz outer IK loop, 250 Hz internal motor command interpolation

IK Configuration Flipping (Known Issue) [T1 — Observed 2026-02-18]

During teleoperation, the IK solver (CasADi/IPOPT) can suddenly jump to a different valid joint configuration. Symptoms:

  • Arm suddenly sticks out to the side
  • The "zero" position appears shifted (e.g., arm horizontal becomes the new resting position)
  • User can only move the arm in the "wrong" direction

Root cause: For any given hand position, multiple joint configurations are valid (elbow-up vs elbow-down, different shoulder rotations). The solver uses warm-starting + smoothness cost (0.1 weight) to stay in one configuration. But near singularities (fully extended arm, workspace boundary), it can jump to an alternate solution. Once jumped, the smoothness regularization keeps it in the new (wrong) configuration.

This is NOT an encoder issue. The G1 has dual absolute encoders per joint [T0]. The motors faithfully follow the IK solver's commands — the solver is commanding the wrong configuration.

Recovery: Press q to stop, restart teleop, press r again. The IK solver reinitializes to the default arm configuration.

Potential fixes (not yet implemented):

  1. Null-space bias toward preferred configuration (elbow-down, shoulder neutral)
  2. Discontinuity detection — reject solutions that jump > threshold between frames
  3. Tighter joint-space regularization (increase smoothness weight)
  4. Workspace clamping near singularities

Internal Control Details [T1]

  • Arm velocity limit: Clips to 20–30 rad/s with gradual ramp-up over 5 seconds at start
  • PD gains (arm joints): Shoulder: kp=80–300, kd=3. Wrist: kp=40, kd=1.5
  • State feedback: Always from rt/lowstate regardless of mode
  • mode_machine echo: Reads current mode_machine from robot state, echoes it back in commands
  • Blend weight (motion mode): motor_cmd[kNotUsedJoint0].q set to 1.0 during operation, ramped to 0.0 on exit

2. g1-control (Friend's Custom System)

Location on robot: /home/unitree/g1-control/ Origin: experientialtech/g1-control

A comprehensive web-based control panel built on top of xr_teleoperate submodule. Provides:

Components

File Purpose
server.py (59 KB) aiohttp web server (HTTP 8080, HTTPS 8443) — process management, robot mode control, camera streaming, audio, LiDAR
start_teleop.sh Automated launch: kills videohub → starts teleimager → starts Inspire hand drivers → launches teleop
hand_control.py Inspire FTP hand control via Modbus TCP (left: 192.168.123.210, right: 192.168.123.211)
loco_helper.py Persistent DDS LocoClient subprocess — avoids SIGSEGV from repeated DDS init/cleanup

server.py Features

  • Process management: Start/stop teleop, image_server, hand drivers, voice/vision chat, LiDAR
  • Robot FSM control: Via loco_helper.py — modes: damp (1), stand_up (4), walk (501)
  • TeleopIPC: ZMQ IPC to xr_teleoperate (CMD_START, CMD_STOP, CMD_RECORD_TOGGLE)
  • Camera streaming: MJPEG from ZMQ, single JPEG snapshots
  • Audio: DDS-based mic/speaker, TTS, volume/LED control
  • LiDAR: UDP listener, GLB/USDZ export, WebSocket streaming
  • Web UI: Mobile-friendly on port 8080/8443

start_teleop.sh Launch Sequence

  1. Activates conda tv
  2. Fixes OpenSSL LD_LIBRARY_PATH
  3. Checks SSL certs
  4. Kills videohub_pc4 to free /dev/video2
  5. Starts teleimager image server
  6. Starts Inspire hand Modbus drivers (right=192.168.123.211/LR='l', left=192.168.123.210/LR='r' — labels intentionally swapped)
  7. Launches: python3 teleop_hand_and_arm.py --arm=G1_29 --ee=inspire_ftp
  8. Does NOT handle robot mode switching — assumes robot is already in correct state

3. GR00T-WBC AVP Bridge (Alternative Approach)

Location on GB10: /home/mitchaiet/GR00T-WholeBodyControl/scripts/avp_wbc_bridge.py Status: Code written, not yet tested

Alternative approach using VisionProTeleop's native "Tracking Streamer" app (App Store, free) + avp_stream Python library → bridge to GR00T-WBC's ControlPolicy/upper_body_pose ROS2 topic.

Why This Exists

xr_teleoperate and GR00T-WBC are incompatible when running simultaneously:

  • xr_teleoperate (debug mode) publishes to rt/lowcmd — conflicts with GR00T-WBC's 50 Hz motor commands
  • xr_teleoperate (motion mode) uses rt/arm_sdk with Unitree's built-in locomotion — bypasses GR00T-WBC entirely

The AVP bridge approach uses GR00T-WBC's native upper body topic, allowing:

  • Arm teleoperation via Vision Pro hand tracking (native ARKit quality)
  • GR00T-WBC's RL-based balance and locomotion (trained policy)
  • Joystick walking via GR00T-WBC's wireless_remote integration

Architecture

Vision Pro (Tracking Streamer app, gRPC port 12345)
    ▼
GB10: avp_wbc_bridge.py (avp_stream.VisionProStreamer)
    │ Coordinate transform (AVP → Unitree frame)
    │ Pinocchio damped least-squares IK → 14 arm joints
    │ Prepend 3 waist zeros → 17 upper body DOFs
    ▼
ROS2 topic: "ControlPolicy/upper_body_pose" (msgpack over ByteMultiArray)
    ▼
GB10: GR00T-WBC control loop (merges upper + lower body)
    ▼
DDS rt/lowcmd → Robot

Dependencies (installed on GB10)

  • avp_stream v2.51
  • pinocchio 2.7.0
  • msgpack_numpy
  • URDF: ~/xr_teleoperate/assets/g1/g1_body29_hand14.urdf

4. FSM Mode Reference [T1]

Robot FSM states (from unitree_sdk2py):

FSM ID Name Description
0 ZeroTorque Motors off
1 Damp Soft stop, motors damped
3 Sit Sit down
4 Stand up Robot stands from sitting
5 Locked standing Standing, position held (observed) [T2]
200 Start Start locomotion
501 Walk/AI Full AI locomotion active
702 Lie2StandUp Stand up from lying
706 Squat2StandUp Toggle squat/stand

MotionSwitcher modes (separate from FSM):

  • "ai" — AI/RL locomotion
  • "normal" — Normal locomotion
  • "advanced" — Advanced mode
  • Released (empty) — Debug mode (full low-level access)

For xr_teleoperate --motion: Robot must be in Regular mode via the R3 physical controller — R1+X (1-DOF waist, base G1) or R1+Y (3-DOF waist, EDU G1). Running mode (R2+A) is NOT supported and will lock legs. This is distinct from the FSM IDs above.

5. Pitfalls & Known Issues

  1. --motion requires R3 controller Regular mode (R1+X) FIRST — the teleop script does NOT handle mode switching [T1]
  2. Running mode (R2+A) is NOT supported — only Regular mode works with xr_teleoperate [T1]
  3. xr_teleoperate + GR00T-WBC conflict — both publish motor commands, cannot run simultaneously [T1]
  4. SSL cert SAN must include robot's IP — Vision Pro will reject connection if IP not in cert SAN [T1]
  5. rootCA.pem must be explicitly trusted on Vision Pro — installing the profile is not enough, must enable trust in Certificate Trust Settings [T1]
  6. LD_LIBRARY_PATH needed for conda OpenSSL — without it, SSL handshake may fail with version mismatch [T1]
  7. Vuer WebSocket session mismatch — if teleop is started before Vision Pro connects, may need to restart teleop for clean session [T2]
  8. Inspire hand labels are swapped — right hand at 192.168.123.211 uses DDS label 'l', left uses 'r' [T1]
  9. Vuer v0.0.40 client JS incompatible with visionOS — zero WebSocket connections. Must use v0.0.60 (with patches). [T1 — Verified 2026-02-18]
  10. pip install overwrites patches — reinstalling vuer resets JS port fix and aiohttp SSL fix. Re-apply ALL patches after every pip install. [T1]
  11. Must launch from ~/xr_teleoperate/teleop/ — URDF path is ../assets/g1/g1_body29_hand14.urdf (relative). Launching from parent directory causes FileNotFoundError. [T1]
  12. Safari caches aggressively on visionOS — after server restart, the VP may resume an old cached page with a dead WebSocket. Hard-reload (Aa menu → Reload) required. [T1 — Observed 2026-02-18]
  13. Running mode (R2+A) locks legs during teleop — by design, rt/arm_sdk in Running mode causes lower body joints to lock rigid. Walking only works in Regular mode (R1+X base / R1+Y EDU). [T1 — xr_teleoperate Issue #251]
  14. EDU units use R1+Y, not R1+X — R1+X enters 1-DOF waist Regular mode (disabled on 29-DOF EDU hardware). R1+Y enters 3-DOF waist Regular mode. [T1 — Verified on G1 EDU Ultimate E, 2026-02-18]

Key Relationships