# Teleoperation & Telepresence

**Status:** Active — xr_teleoperate working with Vision Pro (v0.0.60 + patches, verified 2026-02-18)
**Evidence tier:** T1 (code-verified) unless otherwise noted

---

## 1. xr_teleoperate (Unitree Official)

**Repository:** https://github.com/unitreerobotics/xr_teleoperate
**Location on robot:** `/home/unitree/xr_teleoperate/` (standalone clone) and `/home/unitree/g1-control/repos/xr-teleoperate/` (submodule of friend's g1-control)
**Conda environment:** `tv` (Python 3.10, pinocchio 3.1.0, casadi 3.6.7, unitree_sdk2py 1.0.1)

### Architecture

```
Apple Vision Pro (Safari WebXR)
    │
    │ HTTPS + WebSocket (port 8012)
    │ Data: wrist 4x4 SE3 + finger joints + head pose
    ▼
Robot Jetson (10.0.0.64): teleop_hand_and_arm.py
    │ TeleVuerWrapper.get_tele_data() → wrist poses
    │ → Pinocchio + CasADi IK → 14 arm joint angles (7 left + 7 right)
    │ → G1_29_ArmController publishes to DDS
    ▼
DDS topic: "rt/arm_sdk" (motion mode) or "rt/lowcmd" (debug mode)
    │ Motor commands at 250 Hz (internal interpolation from 30 Hz IK)
    ▼
Robot motors (arms only in motion mode; all joints in debug mode)
```

### Two Operating Modes

| Aspect | Debug Mode (no `--motion`) | Motion Mode (`--motion`) |
|--------|---------------------------|--------------------------|
| MotionSwitcher called? | Yes — enters Debug Mode | No — skipped entirely |
| DDS topic | `rt/lowcmd` (full control) | `rt/arm_sdk` (arm overlay) |
| Joint control scope | ALL joints (legs locked at current pos) | Arms only; legs under built-in locomotion |
| Robot can walk? | No — frozen in place | Yes — via R3 physical controller |
| Blend weight signal | Not used | `motor_cmd[29].q = 1.0` (tells internal controller to apply arm commands) |
| Required robot state | Any (debug mode takes over) | **Must be in Regular mode (R1+X on R3 controller)** |
| Exit behavior | Arms go home | Arms go home with gradual weight ramp (1.0→0.0 over ~2s) |

### Correct Startup Procedure (`--motion` mode) [T1]

**Prerequisites:**
- Robot powered on, standing
- R3 physical controller available
- SSL certs at `~/.config/xr_teleoperate/{cert.pem, key.pem}` with correct IPs in SAN
- `rootCA.pem` installed and trusted on Vision Pro (Settings → General → About → Certificate Trust Settings)
- Vision Pro on same WiFi network as robot

**Step 1: R3 controller → Regular mode**
The robot must be in Regular mode BEFORE launching teleop. The teleop script does NOT handle mode switching itself.

From power-on: **L2+B** (Damping) → **L2+D-pad UP** (Locked Standing) → **Regular mode:**

| Variant | Regular Mode Button | Waist DOFs | Notes |
|---------|-------------------|------------|-------|
| G1 Base (23 DOF) | **R1 + X** | 1 waist joint | |
| G1 EDU (29 DOF) | **R1 + Y** | 3 waist joints | R1+X is disabled on EDU hardware |

**Running mode (R2+A) is NOT supported** — causes lower body joints to lock rigid during teleop (xr_teleoperate Issue #251). [T1 — Verified 2026-02-18]

**Step 2: Launch teleop on robot**
```bash
conda activate tv
cd ~/xr_teleoperate/teleop
python3 teleop_hand_and_arm.py --arm=G1_29 --motion
```
Wait for: `Press [r] to start syncing the robot with your movements.`

**Step 3: Connect Vision Pro**
Open Safari → `https://10.0.0.64:8012` → tap "Virtual Reality"

**Step 4: Align arms**
Position your arms matching the robot's initial pose (arms at sides) to avoid sudden movement.

**Step 5: Press `r` in terminal**
Starts the IK control loop. Arms begin tracking hand movements. Velocity ramp-up over 5 seconds.

**During operation:**
- Walk the robot using the R3 physical controller
- Press `s` to toggle recording (if `--record` enabled)
- Press `q` to exit (arms return to rest over ~5 seconds via weight ramp-down)

### Running from GB10 (offboard)

When running on GB10 instead of the robot itself, must specify the network interface:
```bash
python3 teleop_hand_and_arm.py \
    --arm G1_29 --motion \
    --network-interface 192.168.123.100 \
    --img-server-ip 192.168.123.164
```
Also requires `LD_LIBRARY_PATH` for OpenSSL compatibility:
```bash
export LD_LIBRARY_PATH=$HOME/miniforge3/envs/tv/lib:$LD_LIBRARY_PATH
```

### Key CLI Arguments

| Argument | Default | Description |
|----------|---------|-------------|
| `--arm` | required | `G1_29`, `G1_23`, `H1_2`, `H1` |
| `--ee` | none | End-effector: `dex3`, `dex1`, `inspire_ftp`, `inspire_dfx`, `brainco` |
| `--motion` | false | Use `rt/arm_sdk` with built-in locomotion (vs `rt/lowcmd` debug) |
| `--frequency` | 30 | IK control loop Hz |
| `--input-mode` | `hand` | `hand` (tracking) or `controller` |
| `--display-mode` | `immersive` | `immersive`, `ego`, `pass-through` |
| `--network-interface` | none | DDS network interface IP (required when offboard) |
| `--img-server-ip` | none | Camera image server IP (teleimager on PC2) |
| `--sim` | false | Isaac Sim mode (DDS domain 1) |
| `--record` | false | Record episodes for imitation learning |
| `--headless` | false | No XR visualization |
| `--ipc` | false | Enable ZMQ IPC for external control (used by g1-control server.py) |

### SSL Certificate Setup [T1]

The televuer WebXR server requires HTTPS (WebXR mandates secure context). Certificate resolution order:
1. Environment variables: `XR_TELEOP_CERT` and `XR_TELEOP_KEY`
2. User config: `~/.config/xr_teleoperate/cert.pem` and `key.pem`
3. Fallback: bundled in televuer package directory

**Current certs on robot** (regenerated 2026-02-16):
- SAN includes: `10.0.0.64` (wlan0), `192.168.123.164` (eth0), `192.168.1.21`, `127.0.0.1`
- Issuer: `CN = G1 Robot Local CA`
- Valid until: 2028-05-21
- `rootCA.pem` served via `http://10.0.0.64:9090/rootCA.crt` for Vision Pro download

**To regenerate certs for a new IP:**
```bash
# Generate CA
openssl req -x509 -new -nodes -newkey rsa:2048 -keyout rootCA.key -out rootCA.pem -days 825 -subj "/CN=G1 Robot Local CA"
# Generate server key + CSR with SAN
openssl req -new -nodes -newkey rsa:2048 -keyout key.pem -out server.csr -config san.cnf
# Sign with CA
openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out cert.pem -days 825 -extfile v3_ext.cnf
```

### Safari/visionOS Certificate Trust (CRITICAL) [T1 — Verified 2026-02-18]

Safari on visionOS treats HTTPS and WebSocket (`wss://`) certificate trust **separately**. Clicking "Accept" on Safari's HTTPS cert warning only trusts the page load — **WebSocket connections are silently rejected** with no error, no prompt. This is documented Apple/WebKit behavior, not a bug.

**The ONLY way to get `wss://` working with self-signed certs on Vision Pro:**

1. Generate certs using a CA structure (rootCA.pem → cert.pem), NOT bare self-signed
2. Serve `rootCA.crt` via plain HTTP with MIME type `application/x-x509-ca-cert`
   - **`python3 -m http.server` does NOT work** — it serves `.pem` with wrong MIME type and Safari won't trigger profile install
   - Use a custom handler or serve as `.crt` with correct Content-Type header
3. On Vision Pro: download from `http://10.0.0.64:9090/rootCA.crt`
4. Install profile: Settings → General → VPN & Device Management
5. **Enable full trust**: Settings → General → About → Certificate Trust Settings → toggle ON "G1 Robot Local CA"

**If Safari has accumulated stale state** from debugging (old certs, cached JS, partial profile installs), a **factory reset of the Vision Pro** may be necessary. After reset, do a clean cert install on the fresh OS.

### Vuer Package Patches (Required for v0.0.60) [T1 — Verified 2026-02-18]

The `vuer` pip package (v0.0.60) has bugs that must be patched after install. **These patches are lost on every `pip install`.**

**1. JS Port Fix — `hostname` → `host` in WebSocket URI construction**

The `getDefaultSocketURI()` function in the bundled JS client uses `window.location.hostname` for HTTPS, which strips the port. The WebSocket tries port 443 instead of 8012.

Fix: In ALL chunk files containing `wss://` under `vuer/client_build/assets/chunks/`:
```
# Find affected files
grep -l "wss://" .../vuer/client_build/assets/chunks/*.js
# In each file, replace:
wss://${window.location.hostname}  →  wss://${window.location.host}
```

On v0.0.60 this affects 4 chunk files: `chunk-Bf98F3Ua.js`, `chunk-BU6qPyb1.js`, `chunk-Dd3xtWba.js`, `chunk-DmvjxeUa.js`. Filenames change per version.

**2. aiohttp SSL Assertion Fix**

In `aiohttp/base_protocol.py`, the `resume_writing()` method has `assert self._paused` which crashes on SSL WebSocket connections.

Fix: `assert self._paused` → `if not self._paused: return`

**3. Vuer version compatibility**

| Version | visionOS Safari client | WebSocket stability | Status |
|---------|----------------------|---------------------|--------|
| v0.0.40 | Client JS does NOT work (zero WS connections) | N/A | Do NOT use |
| v0.0.60 | Client JS works (with port fix) | Connects, known disconnect bug (vuer #85) | **Use this** |

Community reports (vuer #85, televuer #1, xr_teleoperate #241, #242) confirm v0.0.60 has WebSocket disconnect issues. With a factory-reset VP + clean cert install, connections are stable.

### IK Solver Details [T1]

- **Library:** Pinocchio 3.1.0 + CasADi 3.6.7 (constrained nonlinear optimization)
- **URDF:** `~/xr_teleoperate/assets/g1/g1_body29_hand14.urdf` (29 DOF body + 14 DOF Dex3 hands)
- **Reduced model:** Locks all non-arm joints → 14 DOF (7 per arm)
- **End-effector frames:** 0.05m offset along local x-axis from wrist yaw joint
- **Cost function:** 50 × translational_error + rotation_error + 0.02 × regularization + 0.1 × smoothness
- **Max iterations:** 30, tolerance 1e-4, warm-start enabled
- **Model cache:** `g1_29_model_cache.pkl` (172 KB, avoids slow URDF parsing)
- **Control rate:** 30 Hz outer IK loop, 250 Hz internal motor command interpolation

### IK Configuration Flipping (Known Issue) [T1 — Observed 2026-02-18]

During teleoperation, the IK solver (CasADi/IPOPT) can suddenly jump to a different valid joint configuration. Symptoms:
- Arm suddenly sticks out to the side
- The "zero" position appears shifted (e.g., arm horizontal becomes the new resting position)
- User can only move the arm in the "wrong" direction

**Root cause:** For any given hand position, multiple joint configurations are valid (elbow-up vs elbow-down, different shoulder rotations). The solver uses warm-starting + smoothness cost (0.1 weight) to stay in one configuration. But near singularities (fully extended arm, workspace boundary), it can jump to an alternate solution. Once jumped, the smoothness regularization keeps it in the new (wrong) configuration.

**This is NOT an encoder issue.** The G1 has dual absolute encoders per joint [T0]. The motors faithfully follow the IK solver's commands — the solver is commanding the wrong configuration.

**Recovery:** Press `q` to stop, restart teleop, press `r` again. The IK solver reinitializes to the default arm configuration.

**Potential fixes (not yet implemented):**
1. Null-space bias toward preferred configuration (elbow-down, shoulder neutral)
2. Discontinuity detection — reject solutions that jump > threshold between frames
3. Tighter joint-space regularization (increase smoothness weight)
4. Workspace clamping near singularities

### Internal Control Details [T1]

- **Arm velocity limit:** Clips to 20–30 rad/s with gradual ramp-up over 5 seconds at start
- **PD gains (arm joints):** Shoulder: kp=80–300, kd=3. Wrist: kp=40, kd=1.5
- **State feedback:** Always from `rt/lowstate` regardless of mode
- **mode_machine echo:** Reads current `mode_machine` from robot state, echoes it back in commands
- **Blend weight (motion mode):** `motor_cmd[kNotUsedJoint0].q` set to 1.0 during operation, ramped to 0.0 on exit

---

## 2. g1-control (Friend's Custom System)

**Location on robot:** `/home/unitree/g1-control/`
**Origin:** `experientialtech/g1-control`

A comprehensive web-based control panel built on top of xr_teleoperate submodule. Provides:

### Components

| File | Purpose |
|------|---------|
| `server.py` (59 KB) | aiohttp web server (HTTP 8080, HTTPS 8443) — process management, robot mode control, camera streaming, audio, LiDAR |
| `start_teleop.sh` | Automated launch: kills videohub → starts teleimager → starts Inspire hand drivers → launches teleop |
| `hand_control.py` | Inspire FTP hand control via Modbus TCP (left: 192.168.123.210, right: 192.168.123.211) |
| `loco_helper.py` | Persistent DDS LocoClient subprocess — avoids SIGSEGV from repeated DDS init/cleanup |

### server.py Features
- **Process management:** Start/stop teleop, image_server, hand drivers, voice/vision chat, LiDAR
- **Robot FSM control:** Via loco_helper.py — modes: damp (1), stand_up (4), walk (501)
- **TeleopIPC:** ZMQ IPC to xr_teleoperate (CMD_START, CMD_STOP, CMD_RECORD_TOGGLE)
- **Camera streaming:** MJPEG from ZMQ, single JPEG snapshots
- **Audio:** DDS-based mic/speaker, TTS, volume/LED control
- **LiDAR:** UDP listener, GLB/USDZ export, WebSocket streaming
- **Web UI:** Mobile-friendly on port 8080/8443

### start_teleop.sh Launch Sequence
1. Activates conda `tv`
2. Fixes OpenSSL LD_LIBRARY_PATH
3. Checks SSL certs
4. Kills `videohub_pc4` to free `/dev/video2`
5. Starts teleimager image server
6. Starts Inspire hand Modbus drivers (right=192.168.123.211/LR='l', left=192.168.123.210/LR='r' — labels intentionally swapped)
7. Launches: `python3 teleop_hand_and_arm.py --arm=G1_29 --ee=inspire_ftp`
8. **Does NOT handle robot mode switching** — assumes robot is already in correct state

---

## 3. GR00T-WBC AVP Bridge (Alternative Approach)

**Location on GB10:** `/home/mitchaiet/GR00T-WholeBodyControl/scripts/avp_wbc_bridge.py`
**Status:** Code written, not yet tested

Alternative approach using VisionProTeleop's native "Tracking Streamer" app (App Store, free) + `avp_stream` Python library → bridge to GR00T-WBC's `ControlPolicy/upper_body_pose` ROS2 topic.

### Why This Exists

xr_teleoperate and GR00T-WBC are **incompatible** when running simultaneously:
- xr_teleoperate (debug mode) publishes to `rt/lowcmd` — conflicts with GR00T-WBC's 50 Hz motor commands
- xr_teleoperate (motion mode) uses `rt/arm_sdk` with Unitree's built-in locomotion — bypasses GR00T-WBC entirely

The AVP bridge approach uses GR00T-WBC's native upper body topic, allowing:
- Arm teleoperation via Vision Pro hand tracking (native ARKit quality)
- GR00T-WBC's RL-based balance and locomotion (trained policy)
- Joystick walking via GR00T-WBC's wireless_remote integration

### Architecture

```
Vision Pro (Tracking Streamer app, gRPC port 12345)
    ▼
GB10: avp_wbc_bridge.py (avp_stream.VisionProStreamer)
    │ Coordinate transform (AVP → Unitree frame)
    │ Pinocchio damped least-squares IK → 14 arm joints
    │ Prepend 3 waist zeros → 17 upper body DOFs
    ▼
ROS2 topic: "ControlPolicy/upper_body_pose" (msgpack over ByteMultiArray)
    ▼
GB10: GR00T-WBC control loop (merges upper + lower body)
    ▼
DDS rt/lowcmd → Robot
```

### Dependencies (installed on GB10)
- `avp_stream` v2.51
- `pinocchio` 2.7.0
- `msgpack_numpy`
- URDF: `~/xr_teleoperate/assets/g1/g1_body29_hand14.urdf`

---

## 4. FSM Mode Reference [T1]

Robot FSM states (from unitree_sdk2py):

| FSM ID | Name | Description |
|--------|------|-------------|
| 0 | ZeroTorque | Motors off |
| 1 | Damp | Soft stop, motors damped |
| 3 | Sit | Sit down |
| 4 | Stand up | Robot stands from sitting |
| 5 | Locked standing | Standing, position held (observed) [T2] |
| 200 | Start | Start locomotion |
| 501 | Walk/AI | Full AI locomotion active |
| 702 | Lie2StandUp | Stand up from lying |
| 706 | Squat2StandUp | Toggle squat/stand |

MotionSwitcher modes (separate from FSM):
- `"ai"` — AI/RL locomotion
- `"normal"` — Normal locomotion
- `"advanced"` — Advanced mode
- Released (empty) — Debug mode (full low-level access)

**For xr_teleoperate `--motion`:** Robot must be in Regular mode via the R3 physical controller — **R1+X** (1-DOF waist, base G1) or **R1+Y** (3-DOF waist, EDU G1). Running mode (R2+A) is NOT supported and will lock legs. This is distinct from the FSM IDs above.

---

## 5. Pitfalls & Known Issues

1. **`--motion` requires R3 controller Regular mode (R1+X) FIRST** — the teleop script does NOT handle mode switching [T1]
2. **Running mode (R2+A) is NOT supported** — only Regular mode works with xr_teleoperate [T1]
3. **xr_teleoperate + GR00T-WBC conflict** — both publish motor commands, cannot run simultaneously [T1]
4. **SSL cert SAN must include robot's IP** — Vision Pro will reject connection if IP not in cert SAN [T1]
5. **`rootCA.pem` must be explicitly trusted on Vision Pro** — installing the profile is not enough, must enable trust in Certificate Trust Settings [T1]
6. **`LD_LIBRARY_PATH` needed for conda OpenSSL** — without it, SSL handshake may fail with version mismatch [T1]
7. **Vuer WebSocket session mismatch** — if teleop is started before Vision Pro connects, may need to restart teleop for clean session [T2]
8. **Inspire hand labels are swapped** — right hand at 192.168.123.211 uses DDS label 'l', left uses 'r' [T1]
9. **Vuer v0.0.40 client JS incompatible with visionOS** — zero WebSocket connections. Must use v0.0.60 (with patches). [T1 — Verified 2026-02-18]
10. **`pip install` overwrites patches** — reinstalling vuer resets JS port fix and aiohttp SSL fix. Re-apply ALL patches after every pip install. [T1]
11. **Must launch from `~/xr_teleoperate/teleop/`** — URDF path is `../assets/g1/g1_body29_hand14.urdf` (relative). Launching from parent directory causes FileNotFoundError. [T1]
12. **Safari caches aggressively on visionOS** — after server restart, the VP may resume an old cached page with a dead WebSocket. Hard-reload (`Aa` menu → Reload) required. [T1 — Observed 2026-02-18]
13. **Running mode (R2+A) locks legs during teleop** — by design, `rt/arm_sdk` in Running mode causes lower body joints to lock rigid. Walking only works in Regular mode (R1+X base / R1+Y EDU). [T1 — xr_teleoperate Issue #251]
14. **EDU units use R1+Y, not R1+X** — R1+X enters 1-DOF waist Regular mode (disabled on 29-DOF EDU hardware). R1+Y enters 3-DOF waist Regular mode. [T1 — Verified on G1 EDU Ultimate E, 2026-02-18]

---

## Key Relationships
- Hardware: [[joint-configuration]] (arm DOFs, joint limits)
- Control: [[whole-body-control]] (GR00T-WBC upper body integration)
- Control: [[manipulation]] (arm IK, end-effector control)
- Retargeting: [[motion-retargeting]] (human → robot pose mapping)
- Network: [[networking-comms]] (DDS topics, robot IPs, SSL)
- Compute: [[gb10-offboard-compute]] (offboard teleop hosting)
- Safety: [[safety-limits]] (joint velocity limits, mode switching)