Browse Source
Phase 5.5+: VP teleop working, R3 buttons, calibration, deploy scripts
Phase 5.5+: VP teleop working, R3 buttons, calibration, deploy scripts
Context system updates: - New: teleoperation.md (VP WebXR pipeline, xr_teleoperate architecture) - New: robot-modifications-log.md (tracking all changes made to robot) - Updated: deployment-operations, networking, open-questions, sensors, sdk-programming, simulation, whole-body-control, glossary - CLAUDE.md: added Phase 5.5 history, SSH safety rules, Gitea workflow Deploy tooling: - scripts/pull_on_robot.py: pull xr_teleoperate from Gitea via paramiko - scripts/launch_teleop.py: launch teleop in screen session via SSH - scripts/diagnose_r3_buttons.py: R3 controller button bitmask diagnostic Key milestones this phase: - VP WebXR arm tracking verified end-to-end - Integral drift correction (Ki, clamp, decay, toggle) - R3 controller buttons mapped to teleop commands (A/B/X/Y/Start) - Start-time calibration for relative-mode arm tracking - Arm reset with auto-pause and re-calibration workflow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>master
17 changed files with 1141 additions and 34 deletions
-
2.gitignore
-
25CLAUDE.md
-
143context/deployment-operations.md
-
20context/gb10-offboard-compute.md
-
15context/learning-and-ai.md
-
26context/networking-comms.md
-
42context/open-questions.md
-
91context/robot-modifications-log.md
-
35context/sdk-programming.md
-
38context/sensors-perception.md
-
11context/simulation.md
-
363context/teleoperation.md
-
41context/whole-body-control.md
-
88reference/glossary.yaml
-
112scripts/diagnose_r3_buttons.py
-
76scripts/launch_teleop.py
-
47scripts/pull_on_robot.py
@ -0,0 +1,2 @@ |
|||
# Local copies of upstream repos (not part of this project) |
|||
apps/ |
|||
@ -0,0 +1,91 @@ |
|||
# Robot Modifications Log — Pre-Wipe Snapshot |
|||
|
|||
**Date:** 2026-02-17 |
|||
**Purpose:** Document all changes made to the robot before resetting to clean upstream code. |
|||
|
|||
--- |
|||
|
|||
## 1. xr_teleoperate (~/xr_teleoperate on robot) |
|||
|
|||
**Upstream:** https://github.com/unitreerobotics/xr_teleoperate.git |
|||
**Robot commit:** 9fadc51 (upstream HEAD) |
|||
|
|||
### Modified files (tracked, dirty): |
|||
|
|||
#### teleop/teleop_hand_and_arm.py |
|||
- Added `import numpy as np` at top |
|||
- Added `--avp-ip` argparse flag (Vision Pro IP for avp_stream bypass) |
|||
- Added `--debug` argparse flag (enables per-frame file logging) |
|||
- Added `teleop_logger` import and `setup_teleop_logging()` call |
|||
- Added conditional: if `--avp-ip` given, uses `AVPStreamWrapper` instead of `TeleVuerWrapper` |
|||
- Added per-frame IK debug logging block (writes to `/tmp/teleop_debug_*.log`) |
|||
- Added `[IK_DBG]` print every 30 frames (sol_q, cur_q, delta) |
|||
|
|||
#### teleop/robot_control/robot_arm_ik.py |
|||
- **IK cost function:** zeroed rotation cost (`0 * self.rotation_cost`) for position-only mode — applied to ALL 4 IK classes (G1_29, G1_23, H1_2, H1) |
|||
- Added `self.last_solve_info = {}` dict to all 4 classes |
|||
- Added solve info capture in try/except blocks (status, raw_sol, cost on success; status, error on fail) |
|||
|
|||
### Untracked files (our additions): |
|||
- `teleop/avp_stream_wrapper.py` — Drop-in AVPStream wrapper using gRPC (Tracking Streamer app). Full coordinate transform pipeline matching TeleVuerWrapper. |
|||
- `teleop/teleop_logger.py` — Debug logging utility (setup_teleop_logging, fmt_arr, fmt_pos, fmt_rpy, rot_to_euler) |
|||
- `teleop/rotation_diagnostic.py` — Diagnostic script for rotation analysis |
|||
- `teleop/test_avp_stream.py` — Test script for avp_stream connectivity |
|||
- `teleop/robot_control/robot_arm_ik.py.bak_debug` — Backup before debug changes |
|||
- `teleop/teleop_hand_and_arm.py.bak_debug` — Backup before debug changes |
|||
|
|||
--- |
|||
|
|||
## 2. vuer package (~/miniforge3/envs/tv/lib/python3.10/site-packages/vuer/) |
|||
|
|||
**Package:** vuer 0.0.60 (pip install) |
|||
**Upstream:** https://github.com/vuer-ai/vuer |
|||
|
|||
### Patches applied (with .bak backups): |
|||
|
|||
#### base_protocol.py (aiohttp, NOT vuer) |
|||
**File:** `~/miniforge3/envs/tv/lib/python3.10/site-packages/aiohttp/base_protocol.py` |
|||
**Bug:** `assert self._paused` in `resume_writing()` crashes on SSL WebSocket connections (aiohttp 3.10.5, Python 3.10, aarch64) |
|||
**Fix:** Changed line 36 from `assert self._paused` to `if not self._paused: return` |
|||
**Backup:** `.bak` exists |
|||
|
|||
#### chunk-Dd3xtWba.js (Vuer client JS bundle) |
|||
**File:** `~/miniforge3/envs/tv/lib/python3.10/site-packages/vuer/client_build/assets/chunks/chunk-Dd3xtWba.js` |
|||
**Bug:** `getSocketURI()` uses `window.location.hostname` (no port) for HTTPS case, causing WebSocket to connect to port 443 instead of 8012 |
|||
**Fix:** Changed `wss://${window.location.hostname}` to `wss://${window.location.host}` |
|||
**Backup:** `.bak` exists |
|||
|
|||
#### index.html (Vuer client) |
|||
**File:** `~/miniforge3/envs/tv/lib/python3.10/site-packages/vuer/client_build/index.html` |
|||
**Change:** Added `?cb=1771359809` cache-busting to all JS/CSS references |
|||
**Backup:** `.bak` exists |
|||
|
|||
#### server.py (Vuer server) |
|||
**File:** `~/miniforge3/envs/tv/lib/python3.10/site-packages/vuer/server.py` |
|||
**Change:** Added request logging line at start of `socket_index()`: `print(f"[REQ] {request.method} {request.path_qs} upgrade=...")` |
|||
**Backup:** No .bak (logging only, can be dropped) |
|||
|
|||
--- |
|||
|
|||
## 3. Current Issue (at time of wipe) |
|||
|
|||
WebSocket connects then immediately disconnects in pass-through mode. 3 connect/disconnect cycles (react-use-websocket retries 3x then gives up). User sees hands (client-side WebXR) but no data flows to robot. Root cause investigation was in progress — examining Vuer's `downlink()` handler in `server.py` lines 597-670. |
|||
|
|||
The `downlink()` handler: |
|||
1. Creates a VuerProxy session |
|||
2. Calls `self.bound_fn(vuer_proxy)` which returns a generator |
|||
3. Calls `await generator.__anext__()` to get the first server event |
|||
4. Enters `async for msg in ws:` loop to process incoming client messages |
|||
5. When client stops sending → "websocket is now disconnected" |
|||
|
|||
**Hypothesis:** The pass-through handler pushes `Hands(stream=True)` then sleeps forever. The Vuer server's downlink expects the `socket_handler` (set by `@app.add_handler`) to be an async generator that yields events. If the handler is a plain async function (not a generator), the `hasattr(generator, "__anext__")` check may fail, causing it to call `next(generator)` on a coroutine, which could raise TypeError silently. |
|||
|
|||
--- |
|||
|
|||
## 4. Key Findings to Preserve |
|||
|
|||
1. **Vuer JS WebSocket port bug** — MUST fix in any vuer version used with HTTPS on non-443 port |
|||
2. **aiohttp SSL assertion bug** — MUST fix for aiohttp 3.10.5 on aarch64 |
|||
3. **display-mode pass-through** is required when Vision Pro can't reach robot's internal network (192.168.123.164) for WebRTC |
|||
4. **IK rotation cost = 0** was set for position-only tracking during debugging — should be restored to original values once rotation corrections are working |
|||
5. **avp_stream_wrapper.py** is a complete working wrapper for the native Tracking Streamer pipeline (gRPC, port 12345) — should be committed as a proper feature |
|||
@ -0,0 +1,363 @@ |
|||
# Teleoperation & Telepresence |
|||
|
|||
**Status:** Active — xr_teleoperate working with Vision Pro (v0.0.60 + patches, verified 2026-02-18) |
|||
**Evidence tier:** T1 (code-verified) unless otherwise noted |
|||
|
|||
--- |
|||
|
|||
## 1. xr_teleoperate (Unitree Official) |
|||
|
|||
**Repository:** https://github.com/unitreerobotics/xr_teleoperate |
|||
**Location on robot:** `/home/unitree/xr_teleoperate/` (standalone clone) and `/home/unitree/g1-control/repos/xr-teleoperate/` (submodule of friend's g1-control) |
|||
**Conda environment:** `tv` (Python 3.10, pinocchio 3.1.0, casadi 3.6.7, unitree_sdk2py 1.0.1) |
|||
|
|||
### Architecture |
|||
|
|||
``` |
|||
Apple Vision Pro (Safari WebXR) |
|||
│ |
|||
│ HTTPS + WebSocket (port 8012) |
|||
│ Data: wrist 4x4 SE3 + finger joints + head pose |
|||
▼ |
|||
Robot Jetson (10.0.0.64): teleop_hand_and_arm.py |
|||
│ TeleVuerWrapper.get_tele_data() → wrist poses |
|||
│ → Pinocchio + CasADi IK → 14 arm joint angles (7 left + 7 right) |
|||
│ → G1_29_ArmController publishes to DDS |
|||
▼ |
|||
DDS topic: "rt/arm_sdk" (motion mode) or "rt/lowcmd" (debug mode) |
|||
│ Motor commands at 250 Hz (internal interpolation from 30 Hz IK) |
|||
▼ |
|||
Robot motors (arms only in motion mode; all joints in debug mode) |
|||
``` |
|||
|
|||
### Two Operating Modes |
|||
|
|||
| Aspect | Debug Mode (no `--motion`) | Motion Mode (`--motion`) | |
|||
|--------|---------------------------|--------------------------| |
|||
| MotionSwitcher called? | Yes — enters Debug Mode | No — skipped entirely | |
|||
| DDS topic | `rt/lowcmd` (full control) | `rt/arm_sdk` (arm overlay) | |
|||
| Joint control scope | ALL joints (legs locked at current pos) | Arms only; legs under built-in locomotion | |
|||
| Robot can walk? | No — frozen in place | Yes — via R3 physical controller | |
|||
| Blend weight signal | Not used | `motor_cmd[29].q = 1.0` (tells internal controller to apply arm commands) | |
|||
| Required robot state | Any (debug mode takes over) | **Must be in Regular mode (R1+X on R3 controller)** | |
|||
| Exit behavior | Arms go home | Arms go home with gradual weight ramp (1.0→0.0 over ~2s) | |
|||
|
|||
### Correct Startup Procedure (`--motion` mode) [T1] |
|||
|
|||
**Prerequisites:** |
|||
- Robot powered on, standing |
|||
- R3 physical controller available |
|||
- SSL certs at `~/.config/xr_teleoperate/{cert.pem, key.pem}` with correct IPs in SAN |
|||
- `rootCA.pem` installed and trusted on Vision Pro (Settings → General → About → Certificate Trust Settings) |
|||
- Vision Pro on same WiFi network as robot |
|||
|
|||
**Step 1: R3 controller → Regular mode** |
|||
The robot must be in Regular mode BEFORE launching teleop. The teleop script does NOT handle mode switching itself. |
|||
|
|||
From power-on: **L2+B** (Damping) → **L2+D-pad UP** (Locked Standing) → **Regular mode:** |
|||
|
|||
| Variant | Regular Mode Button | Waist DOFs | Notes | |
|||
|---------|-------------------|------------|-------| |
|||
| G1 Base (23 DOF) | **R1 + X** | 1 waist joint | | |
|||
| G1 EDU (29 DOF) | **R1 + Y** | 3 waist joints | R1+X is disabled on EDU hardware | |
|||
|
|||
**Running mode (R2+A) is NOT supported** — causes lower body joints to lock rigid during teleop (xr_teleoperate Issue #251). [T1 — Verified 2026-02-18] |
|||
|
|||
**Step 2: Launch teleop on robot** |
|||
```bash |
|||
conda activate tv |
|||
cd ~/xr_teleoperate/teleop |
|||
python3 teleop_hand_and_arm.py --arm=G1_29 --motion |
|||
``` |
|||
Wait for: `Press [r] to start syncing the robot with your movements.` |
|||
|
|||
**Step 3: Connect Vision Pro** |
|||
Open Safari → `https://10.0.0.64:8012` → tap "Virtual Reality" |
|||
|
|||
**Step 4: Align arms** |
|||
Position your arms matching the robot's initial pose (arms at sides) to avoid sudden movement. |
|||
|
|||
**Step 5: Press `r` in terminal** |
|||
Starts the IK control loop. Arms begin tracking hand movements. Velocity ramp-up over 5 seconds. |
|||
|
|||
**During operation:** |
|||
- Walk the robot using the R3 physical controller |
|||
- Press `s` to toggle recording (if `--record` enabled) |
|||
- Press `q` to exit (arms return to rest over ~5 seconds via weight ramp-down) |
|||
|
|||
### Running from GB10 (offboard) |
|||
|
|||
When running on GB10 instead of the robot itself, must specify the network interface: |
|||
```bash |
|||
python3 teleop_hand_and_arm.py \ |
|||
--arm G1_29 --motion \ |
|||
--network-interface 192.168.123.100 \ |
|||
--img-server-ip 192.168.123.164 |
|||
``` |
|||
Also requires `LD_LIBRARY_PATH` for OpenSSL compatibility: |
|||
```bash |
|||
export LD_LIBRARY_PATH=$HOME/miniforge3/envs/tv/lib:$LD_LIBRARY_PATH |
|||
``` |
|||
|
|||
### Key CLI Arguments |
|||
|
|||
| Argument | Default | Description | |
|||
|----------|---------|-------------| |
|||
| `--arm` | required | `G1_29`, `G1_23`, `H1_2`, `H1` | |
|||
| `--ee` | none | End-effector: `dex3`, `dex1`, `inspire_ftp`, `inspire_dfx`, `brainco` | |
|||
| `--motion` | false | Use `rt/arm_sdk` with built-in locomotion (vs `rt/lowcmd` debug) | |
|||
| `--frequency` | 30 | IK control loop Hz | |
|||
| `--input-mode` | `hand` | `hand` (tracking) or `controller` | |
|||
| `--display-mode` | `immersive` | `immersive`, `ego`, `pass-through` | |
|||
| `--network-interface` | none | DDS network interface IP (required when offboard) | |
|||
| `--img-server-ip` | none | Camera image server IP (teleimager on PC2) | |
|||
| `--sim` | false | Isaac Sim mode (DDS domain 1) | |
|||
| `--record` | false | Record episodes for imitation learning | |
|||
| `--headless` | false | No XR visualization | |
|||
| `--ipc` | false | Enable ZMQ IPC for external control (used by g1-control server.py) | |
|||
|
|||
### SSL Certificate Setup [T1] |
|||
|
|||
The televuer WebXR server requires HTTPS (WebXR mandates secure context). Certificate resolution order: |
|||
1. Environment variables: `XR_TELEOP_CERT` and `XR_TELEOP_KEY` |
|||
2. User config: `~/.config/xr_teleoperate/cert.pem` and `key.pem` |
|||
3. Fallback: bundled in televuer package directory |
|||
|
|||
**Current certs on robot** (regenerated 2026-02-16): |
|||
- SAN includes: `10.0.0.64` (wlan0), `192.168.123.164` (eth0), `192.168.1.21`, `127.0.0.1` |
|||
- Issuer: `CN = G1 Robot Local CA` |
|||
- Valid until: 2028-05-21 |
|||
- `rootCA.pem` served via `http://10.0.0.64:9090/rootCA.crt` for Vision Pro download |
|||
|
|||
**To regenerate certs for a new IP:** |
|||
```bash |
|||
# Generate CA |
|||
openssl req -x509 -new -nodes -newkey rsa:2048 -keyout rootCA.key -out rootCA.pem -days 825 -subj "/CN=G1 Robot Local CA" |
|||
# Generate server key + CSR with SAN |
|||
openssl req -new -nodes -newkey rsa:2048 -keyout key.pem -out server.csr -config san.cnf |
|||
# Sign with CA |
|||
openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out cert.pem -days 825 -extfile v3_ext.cnf |
|||
``` |
|||
|
|||
### Safari/visionOS Certificate Trust (CRITICAL) [T1 — Verified 2026-02-18] |
|||
|
|||
Safari on visionOS treats HTTPS and WebSocket (`wss://`) certificate trust **separately**. Clicking "Accept" on Safari's HTTPS cert warning only trusts the page load — **WebSocket connections are silently rejected** with no error, no prompt. This is documented Apple/WebKit behavior, not a bug. |
|||
|
|||
**The ONLY way to get `wss://` working with self-signed certs on Vision Pro:** |
|||
|
|||
1. Generate certs using a CA structure (rootCA.pem → cert.pem), NOT bare self-signed |
|||
2. Serve `rootCA.crt` via plain HTTP with MIME type `application/x-x509-ca-cert` |
|||
- **`python3 -m http.server` does NOT work** — it serves `.pem` with wrong MIME type and Safari won't trigger profile install |
|||
- Use a custom handler or serve as `.crt` with correct Content-Type header |
|||
3. On Vision Pro: download from `http://10.0.0.64:9090/rootCA.crt` |
|||
4. Install profile: Settings → General → VPN & Device Management |
|||
5. **Enable full trust**: Settings → General → About → Certificate Trust Settings → toggle ON "G1 Robot Local CA" |
|||
|
|||
**If Safari has accumulated stale state** from debugging (old certs, cached JS, partial profile installs), a **factory reset of the Vision Pro** may be necessary. After reset, do a clean cert install on the fresh OS. |
|||
|
|||
### Vuer Package Patches (Required for v0.0.60) [T1 — Verified 2026-02-18] |
|||
|
|||
The `vuer` pip package (v0.0.60) has bugs that must be patched after install. **These patches are lost on every `pip install`.** |
|||
|
|||
**1. JS Port Fix — `hostname` → `host` in WebSocket URI construction** |
|||
|
|||
The `getDefaultSocketURI()` function in the bundled JS client uses `window.location.hostname` for HTTPS, which strips the port. The WebSocket tries port 443 instead of 8012. |
|||
|
|||
Fix: In ALL chunk files containing `wss://` under `vuer/client_build/assets/chunks/`: |
|||
``` |
|||
# Find affected files |
|||
grep -l "wss://" .../vuer/client_build/assets/chunks/*.js |
|||
# In each file, replace: |
|||
wss://${window.location.hostname} → wss://${window.location.host} |
|||
``` |
|||
|
|||
On v0.0.60 this affects 4 chunk files: `chunk-Bf98F3Ua.js`, `chunk-BU6qPyb1.js`, `chunk-Dd3xtWba.js`, `chunk-DmvjxeUa.js`. Filenames change per version. |
|||
|
|||
**2. aiohttp SSL Assertion Fix** |
|||
|
|||
In `aiohttp/base_protocol.py`, the `resume_writing()` method has `assert self._paused` which crashes on SSL WebSocket connections. |
|||
|
|||
Fix: `assert self._paused` → `if not self._paused: return` |
|||
|
|||
**3. Vuer version compatibility** |
|||
|
|||
| Version | visionOS Safari client | WebSocket stability | Status | |
|||
|---------|----------------------|---------------------|--------| |
|||
| v0.0.40 | Client JS does NOT work (zero WS connections) | N/A | Do NOT use | |
|||
| v0.0.60 | Client JS works (with port fix) | Connects, known disconnect bug (vuer #85) | **Use this** | |
|||
|
|||
Community reports (vuer #85, televuer #1, xr_teleoperate #241, #242) confirm v0.0.60 has WebSocket disconnect issues. With a factory-reset VP + clean cert install, connections are stable. |
|||
|
|||
### IK Solver Details [T1] |
|||
|
|||
- **Library:** Pinocchio 3.1.0 + CasADi 3.6.7 (constrained nonlinear optimization) |
|||
- **URDF:** `~/xr_teleoperate/assets/g1/g1_body29_hand14.urdf` (29 DOF body + 14 DOF Dex3 hands) |
|||
- **Reduced model:** Locks all non-arm joints → 14 DOF (7 per arm) |
|||
- **End-effector frames:** 0.05m offset along local x-axis from wrist yaw joint |
|||
- **Cost function:** 50 × translational_error + rotation_error + 0.02 × regularization + 0.1 × smoothness |
|||
- **Max iterations:** 30, tolerance 1e-4, warm-start enabled |
|||
- **Model cache:** `g1_29_model_cache.pkl` (172 KB, avoids slow URDF parsing) |
|||
- **Control rate:** 30 Hz outer IK loop, 250 Hz internal motor command interpolation |
|||
|
|||
### IK Configuration Flipping (Known Issue) [T1 — Observed 2026-02-18] |
|||
|
|||
During teleoperation, the IK solver (CasADi/IPOPT) can suddenly jump to a different valid joint configuration. Symptoms: |
|||
- Arm suddenly sticks out to the side |
|||
- The "zero" position appears shifted (e.g., arm horizontal becomes the new resting position) |
|||
- User can only move the arm in the "wrong" direction |
|||
|
|||
**Root cause:** For any given hand position, multiple joint configurations are valid (elbow-up vs elbow-down, different shoulder rotations). The solver uses warm-starting + smoothness cost (0.1 weight) to stay in one configuration. But near singularities (fully extended arm, workspace boundary), it can jump to an alternate solution. Once jumped, the smoothness regularization keeps it in the new (wrong) configuration. |
|||
|
|||
**This is NOT an encoder issue.** The G1 has dual absolute encoders per joint [T0]. The motors faithfully follow the IK solver's commands — the solver is commanding the wrong configuration. |
|||
|
|||
**Recovery:** Press `q` to stop, restart teleop, press `r` again. The IK solver reinitializes to the default arm configuration. |
|||
|
|||
**Potential fixes (not yet implemented):** |
|||
1. Null-space bias toward preferred configuration (elbow-down, shoulder neutral) |
|||
2. Discontinuity detection — reject solutions that jump > threshold between frames |
|||
3. Tighter joint-space regularization (increase smoothness weight) |
|||
4. Workspace clamping near singularities |
|||
|
|||
### Internal Control Details [T1] |
|||
|
|||
- **Arm velocity limit:** Clips to 20–30 rad/s with gradual ramp-up over 5 seconds at start |
|||
- **PD gains (arm joints):** Shoulder: kp=80–300, kd=3. Wrist: kp=40, kd=1.5 |
|||
- **State feedback:** Always from `rt/lowstate` regardless of mode |
|||
- **mode_machine echo:** Reads current `mode_machine` from robot state, echoes it back in commands |
|||
- **Blend weight (motion mode):** `motor_cmd[kNotUsedJoint0].q` set to 1.0 during operation, ramped to 0.0 on exit |
|||
|
|||
--- |
|||
|
|||
## 2. g1-control (Friend's Custom System) |
|||
|
|||
**Location on robot:** `/home/unitree/g1-control/` |
|||
**Origin:** `experientialtech/g1-control` |
|||
|
|||
A comprehensive web-based control panel built on top of xr_teleoperate submodule. Provides: |
|||
|
|||
### Components |
|||
|
|||
| File | Purpose | |
|||
|------|---------| |
|||
| `server.py` (59 KB) | aiohttp web server (HTTP 8080, HTTPS 8443) — process management, robot mode control, camera streaming, audio, LiDAR | |
|||
| `start_teleop.sh` | Automated launch: kills videohub → starts teleimager → starts Inspire hand drivers → launches teleop | |
|||
| `hand_control.py` | Inspire FTP hand control via Modbus TCP (left: 192.168.123.210, right: 192.168.123.211) | |
|||
| `loco_helper.py` | Persistent DDS LocoClient subprocess — avoids SIGSEGV from repeated DDS init/cleanup | |
|||
|
|||
### server.py Features |
|||
- **Process management:** Start/stop teleop, image_server, hand drivers, voice/vision chat, LiDAR |
|||
- **Robot FSM control:** Via loco_helper.py — modes: damp (1), stand_up (4), walk (501) |
|||
- **TeleopIPC:** ZMQ IPC to xr_teleoperate (CMD_START, CMD_STOP, CMD_RECORD_TOGGLE) |
|||
- **Camera streaming:** MJPEG from ZMQ, single JPEG snapshots |
|||
- **Audio:** DDS-based mic/speaker, TTS, volume/LED control |
|||
- **LiDAR:** UDP listener, GLB/USDZ export, WebSocket streaming |
|||
- **Web UI:** Mobile-friendly on port 8080/8443 |
|||
|
|||
### start_teleop.sh Launch Sequence |
|||
1. Activates conda `tv` |
|||
2. Fixes OpenSSL LD_LIBRARY_PATH |
|||
3. Checks SSL certs |
|||
4. Kills `videohub_pc4` to free `/dev/video2` |
|||
5. Starts teleimager image server |
|||
6. Starts Inspire hand Modbus drivers (right=192.168.123.211/LR='l', left=192.168.123.210/LR='r' — labels intentionally swapped) |
|||
7. Launches: `python3 teleop_hand_and_arm.py --arm=G1_29 --ee=inspire_ftp` |
|||
8. **Does NOT handle robot mode switching** — assumes robot is already in correct state |
|||
|
|||
--- |
|||
|
|||
## 3. GR00T-WBC AVP Bridge (Alternative Approach) |
|||
|
|||
**Location on GB10:** `/home/mitchaiet/GR00T-WholeBodyControl/scripts/avp_wbc_bridge.py` |
|||
**Status:** Code written, not yet tested |
|||
|
|||
Alternative approach using VisionProTeleop's native "Tracking Streamer" app (App Store, free) + `avp_stream` Python library → bridge to GR00T-WBC's `ControlPolicy/upper_body_pose` ROS2 topic. |
|||
|
|||
### Why This Exists |
|||
|
|||
xr_teleoperate and GR00T-WBC are **incompatible** when running simultaneously: |
|||
- xr_teleoperate (debug mode) publishes to `rt/lowcmd` — conflicts with GR00T-WBC's 50 Hz motor commands |
|||
- xr_teleoperate (motion mode) uses `rt/arm_sdk` with Unitree's built-in locomotion — bypasses GR00T-WBC entirely |
|||
|
|||
The AVP bridge approach uses GR00T-WBC's native upper body topic, allowing: |
|||
- Arm teleoperation via Vision Pro hand tracking (native ARKit quality) |
|||
- GR00T-WBC's RL-based balance and locomotion (trained policy) |
|||
- Joystick walking via GR00T-WBC's wireless_remote integration |
|||
|
|||
### Architecture |
|||
|
|||
``` |
|||
Vision Pro (Tracking Streamer app, gRPC port 12345) |
|||
▼ |
|||
GB10: avp_wbc_bridge.py (avp_stream.VisionProStreamer) |
|||
│ Coordinate transform (AVP → Unitree frame) |
|||
│ Pinocchio damped least-squares IK → 14 arm joints |
|||
│ Prepend 3 waist zeros → 17 upper body DOFs |
|||
▼ |
|||
ROS2 topic: "ControlPolicy/upper_body_pose" (msgpack over ByteMultiArray) |
|||
▼ |
|||
GB10: GR00T-WBC control loop (merges upper + lower body) |
|||
▼ |
|||
DDS rt/lowcmd → Robot |
|||
``` |
|||
|
|||
### Dependencies (installed on GB10) |
|||
- `avp_stream` v2.51 |
|||
- `pinocchio` 2.7.0 |
|||
- `msgpack_numpy` |
|||
- URDF: `~/xr_teleoperate/assets/g1/g1_body29_hand14.urdf` |
|||
|
|||
--- |
|||
|
|||
## 4. FSM Mode Reference [T1] |
|||
|
|||
Robot FSM states (from unitree_sdk2py): |
|||
|
|||
| FSM ID | Name | Description | |
|||
|--------|------|-------------| |
|||
| 0 | ZeroTorque | Motors off | |
|||
| 1 | Damp | Soft stop, motors damped | |
|||
| 3 | Sit | Sit down | |
|||
| 4 | Stand up | Robot stands from sitting | |
|||
| 5 | Locked standing | Standing, position held (observed) [T2] | |
|||
| 200 | Start | Start locomotion | |
|||
| 501 | Walk/AI | Full AI locomotion active | |
|||
| 702 | Lie2StandUp | Stand up from lying | |
|||
| 706 | Squat2StandUp | Toggle squat/stand | |
|||
|
|||
MotionSwitcher modes (separate from FSM): |
|||
- `"ai"` — AI/RL locomotion |
|||
- `"normal"` — Normal locomotion |
|||
- `"advanced"` — Advanced mode |
|||
- Released (empty) — Debug mode (full low-level access) |
|||
|
|||
**For xr_teleoperate `--motion`:** Robot must be in Regular mode via the R3 physical controller — **R1+X** (1-DOF waist, base G1) or **R1+Y** (3-DOF waist, EDU G1). Running mode (R2+A) is NOT supported and will lock legs. This is distinct from the FSM IDs above. |
|||
|
|||
--- |
|||
|
|||
## 5. Pitfalls & Known Issues |
|||
|
|||
1. **`--motion` requires R3 controller Regular mode (R1+X) FIRST** — the teleop script does NOT handle mode switching [T1] |
|||
2. **Running mode (R2+A) is NOT supported** — only Regular mode works with xr_teleoperate [T1] |
|||
3. **xr_teleoperate + GR00T-WBC conflict** — both publish motor commands, cannot run simultaneously [T1] |
|||
4. **SSL cert SAN must include robot's IP** — Vision Pro will reject connection if IP not in cert SAN [T1] |
|||
5. **`rootCA.pem` must be explicitly trusted on Vision Pro** — installing the profile is not enough, must enable trust in Certificate Trust Settings [T1] |
|||
6. **`LD_LIBRARY_PATH` needed for conda OpenSSL** — without it, SSL handshake may fail with version mismatch [T1] |
|||
7. **Vuer WebSocket session mismatch** — if teleop is started before Vision Pro connects, may need to restart teleop for clean session [T2] |
|||
8. **Inspire hand labels are swapped** — right hand at 192.168.123.211 uses DDS label 'l', left uses 'r' [T1] |
|||
9. **Vuer v0.0.40 client JS incompatible with visionOS** — zero WebSocket connections. Must use v0.0.60 (with patches). [T1 — Verified 2026-02-18] |
|||
10. **`pip install` overwrites patches** — reinstalling vuer resets JS port fix and aiohttp SSL fix. Re-apply ALL patches after every pip install. [T1] |
|||
11. **Must launch from `~/xr_teleoperate/teleop/`** — URDF path is `../assets/g1/g1_body29_hand14.urdf` (relative). Launching from parent directory causes FileNotFoundError. [T1] |
|||
12. **Safari caches aggressively on visionOS** — after server restart, the VP may resume an old cached page with a dead WebSocket. Hard-reload (`Aa` menu → Reload) required. [T1 — Observed 2026-02-18] |
|||
13. **Running mode (R2+A) locks legs during teleop** — by design, `rt/arm_sdk` in Running mode causes lower body joints to lock rigid. Walking only works in Regular mode (R1+X base / R1+Y EDU). [T1 — xr_teleoperate Issue #251] |
|||
14. **EDU units use R1+Y, not R1+X** — R1+X enters 1-DOF waist Regular mode (disabled on 29-DOF EDU hardware). R1+Y enters 3-DOF waist Regular mode. [T1 — Verified on G1 EDU Ultimate E, 2026-02-18] |
|||
|
|||
--- |
|||
|
|||
## Key Relationships |
|||
- Hardware: [[joint-configuration]] (arm DOFs, joint limits) |
|||
- Control: [[whole-body-control]] (GR00T-WBC upper body integration) |
|||
- Control: [[manipulation]] (arm IK, end-effector control) |
|||
- Retargeting: [[motion-retargeting]] (human → robot pose mapping) |
|||
- Network: [[networking-comms]] (DDS topics, robot IPs, SSL) |
|||
- Compute: [[gb10-offboard-compute]] (offboard teleop hosting) |
|||
- Safety: [[safety-limits]] (joint velocity limits, mode switching) |
|||
@ -0,0 +1,112 @@ |
|||
"""Diagnose R3 controller button bitmask positions via wireless_remote in rt/lowstate. |
|||
|
|||
Tries different network interfaces and LowState types to find where |
|||
wireless_remote data flows. |
|||
""" |
|||
import paramiko, sys, time |
|||
sys.stdout.reconfigure(encoding='utf-8', errors='replace') |
|||
|
|||
ssh = paramiko.SSHClient() |
|||
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) |
|||
ssh.connect('10.0.0.64', username='unitree', password='123', |
|||
timeout=15, look_for_keys=False, allow_agent=False) |
|||
|
|||
# Deploy diagnostic script |
|||
script = r''' |
|||
import sys, time, struct |
|||
sys.path.insert(0, "/home/unitree/miniforge3/envs/tv/lib/python3.11/site-packages") |
|||
from unitree_sdk2py.core.channel import ChannelSubscriber, ChannelFactoryInitialize |
|||
from unitree_sdk2py.idl.unitree_hg.msg.dds_ import LowState_ as hg_LowState |
|||
|
|||
# Try different interfaces |
|||
import subprocess |
|||
result = subprocess.run(['ip', 'link', 'show'], capture_output=True, text=True) |
|||
print("=== Network interfaces ===") |
|||
for line in result.stdout.split('\n'): |
|||
if ': ' in line and 'state' in line: |
|||
print(f" {line.strip()}") |
|||
|
|||
# Use eth0 which connects to the internal bus |
|||
iface = "eth0" |
|||
print(f"\nUsing interface: {iface}") |
|||
|
|||
ChannelFactoryInitialize(0, iface) |
|||
|
|||
count = [0] |
|||
last_wr = [None] |
|||
|
|||
def handler(msg): |
|||
count[0] += 1 |
|||
# Check if wireless_remote exists and has data |
|||
try: |
|||
wr = msg.wireless_remote |
|||
data = bytes(wr) |
|||
|
|||
# Only print if changed or first time or periodic |
|||
if data != last_wr[0] or count[0] % 200 == 1: |
|||
btn1 = data[2] if len(data) > 2 else 0 |
|||
btn2 = data[3] if len(data) > 3 else 0 |
|||
|
|||
btn1_names = [] |
|||
if btn1 & 0x01: btn1_names.append("R1") |
|||
if btn1 & 0x02: btn1_names.append("L1") |
|||
if btn1 & 0x04: btn1_names.append("Start") |
|||
if btn1 & 0x08: btn1_names.append("Select") |
|||
if btn1 & 0x10: btn1_names.append("R2") |
|||
if btn1 & 0x20: btn1_names.append("L2") |
|||
if btn1 & 0x40: btn1_names.append("F1") |
|||
if btn1 & 0x80: btn1_names.append("F3") |
|||
|
|||
btn2_names = [] |
|||
if btn2 & 0x01: btn2_names.append("A") |
|||
if btn2 & 0x02: btn2_names.append("B") |
|||
if btn2 & 0x04: btn2_names.append("X") |
|||
if btn2 & 0x08: btn2_names.append("Y") |
|||
if btn2 & 0x10: btn2_names.append("Up") |
|||
if btn2 & 0x20: btn2_names.append("Right") |
|||
if btn2 & 0x40: btn2_names.append("Down") |
|||
if btn2 & 0x80: btn2_names.append("Left") |
|||
|
|||
any_nonzero = any(b != 0 for b in data[:4]) |
|||
ts = time.strftime('%H:%M:%S') |
|||
print(f"[{ts}] #{count[0]:4d} len={len(data)} " |
|||
f"btn1=0x{btn1:02x}[{','.join(btn1_names) or '-'}] " |
|||
f"btn2=0x{btn2:02x}[{','.join(btn2_names) or '-'}] " |
|||
f"raw={data[:8].hex()} " |
|||
f"{'*** BUTTON ***' if any_nonzero else ''}", flush=True) |
|||
last_wr[0] = data |
|||
except Exception as e: |
|||
if count[0] <= 3: |
|||
print(f"Error reading wireless_remote: {e}", flush=True) |
|||
|
|||
sub = ChannelSubscriber("rt/lowstate", hg_LowState) |
|||
sub.Init(handler, 10) |
|||
|
|||
print("\n=== R3 Button Diagnostic (hg_LowState on eth0) ===", flush=True) |
|||
print("Press buttons on R3 controller. Running 15 seconds...\n", flush=True) |
|||
|
|||
time.sleep(15) |
|||
print(f"\nTotal messages received: {count[0]}", flush=True) |
|||
''' |
|||
|
|||
print("Deploying R3 button diagnostic to robot...") |
|||
sftp = ssh.open_sftp() |
|||
with sftp.file('/tmp/r3_diag.py', 'w') as f: |
|||
f.write(script) |
|||
sftp.close() |
|||
print("Deployed.") |
|||
|
|||
print("\n=== Running diagnostic (15 seconds) ===") |
|||
print("Press each R3 button now!\n") |
|||
|
|||
_, o, e = ssh.exec_command( |
|||
'/home/unitree/miniforge3/envs/tv/bin/python /tmp/r3_diag.py', |
|||
timeout=30) |
|||
|
|||
output = o.read().decode('utf-8', errors='replace') |
|||
print(output) |
|||
err = e.read().decode('utf-8', errors='replace') |
|||
if err: |
|||
print(f"Stderr: {err}") |
|||
|
|||
ssh.close() |
|||
@ -0,0 +1,76 @@ |
|||
"""Launch teleop server from correct directory + ensure cert server running.""" |
|||
import paramiko, sys, time |
|||
sys.stdout.reconfigure(encoding='utf-8', errors='replace') |
|||
ssh = paramiko.SSHClient() |
|||
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) |
|||
ssh.connect('10.0.0.64', username='unitree', password='123', |
|||
timeout=15, look_for_keys=False, allow_agent=False) |
|||
|
|||
# Kill any existing teleop screen |
|||
ssh.exec_command('screen -S teleop -X quit 2>/dev/null; sleep 1', timeout=5) |
|||
time.sleep(2) |
|||
|
|||
# Ensure cert server is running on 9090 |
|||
_, o, _ = ssh.exec_command('ss -tlnp | grep 9090', timeout=5) |
|||
if not o.read().decode().strip(): |
|||
print('Starting cert server on port 9090...') |
|||
ssh.exec_command( |
|||
'cd ~/.config/xr_teleoperate && nohup python3 -m http.server 9090 --bind 0.0.0.0 > /dev/null 2>&1 &', |
|||
timeout=5) |
|||
time.sleep(2) |
|||
else: |
|||
print('Cert server already running on 9090') |
|||
|
|||
# Launch teleop from ~/xr_teleoperate/teleop (correct dir for URDF paths) |
|||
cmd = ( |
|||
'screen -dmS teleop bash -c "' |
|||
'source ~/miniforge3/etc/profile.d/conda.sh && conda activate tv && ' |
|||
'cd ~/xr_teleoperate/teleop && ' |
|||
'python3 teleop_hand_and_arm.py --arm=G1_29 --input-mode hand --motion --display-mode pass-through ' |
|||
'2>&1 | tee /tmp/teleop_v60.log' |
|||
'"' |
|||
) |
|||
print('Starting teleop server...') |
|||
ssh.exec_command(cmd, timeout=10) |
|||
time.sleep(1) |
|||
|
|||
# Wait for server to start |
|||
for i in range(10): |
|||
time.sleep(2) |
|||
_, o, _ = ssh.exec_command('ss -tlnp | grep 8012', timeout=5) |
|||
port = o.read().decode().strip() |
|||
if port: |
|||
print(f'Server listening on port 8012! ({(i+1)*2}s)') |
|||
break |
|||
# Check if it crashed |
|||
_, o, _ = ssh.exec_command('tail -3 /tmp/teleop_v60.log', timeout=5) |
|||
log = o.read().decode('utf-8', errors='replace').strip() |
|||
if 'exiting program' in log.lower() or 'error' in log.lower(): |
|||
print(f'Server crashed! Log:') |
|||
_, o, _ = ssh.exec_command('tail -20 /tmp/teleop_v60.log', timeout=5) |
|||
print(o.read().decode('utf-8', errors='replace')) |
|||
ssh.close() |
|||
sys.exit(1) |
|||
print(f' waiting... ({(i+1)*2}s)') |
|||
else: |
|||
print('Server did not start in 20s. Log:') |
|||
_, o, _ = ssh.exec_command('tail -20 /tmp/teleop_v60.log', timeout=5) |
|||
print(o.read().decode('utf-8', errors='replace')) |
|||
ssh.close() |
|||
sys.exit(1) |
|||
|
|||
# Show final state |
|||
_, o, _ = ssh.exec_command('tail -5 /tmp/teleop_v60.log', timeout=5) |
|||
print('\nLog (last 5 lines):') |
|||
print(o.read().decode('utf-8', errors='replace')) |
|||
|
|||
# Verify cert server accessible |
|||
_, o, _ = ssh.exec_command('curl -s http://localhost:9090/rootCA.pem | head -1', timeout=5) |
|||
cert = o.read().decode().strip() |
|||
print(f'\nCert server: {"OK" if "BEGIN" in cert else "NOT WORKING"}') |
|||
|
|||
print('\n=== READY ===') |
|||
print('Cert install: http://10.0.0.64:9090/rootCA.pem') |
|||
print('Teleop page: https://10.0.0.64:8012') |
|||
|
|||
ssh.close() |
|||
@ -0,0 +1,47 @@ |
|||
"""Pull xr_teleoperate changes from Gitea on the robot.""" |
|||
import paramiko, sys |
|||
sys.stdout.reconfigure(encoding='utf-8', errors='replace') |
|||
ssh = paramiko.SSHClient() |
|||
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) |
|||
ssh.connect('10.0.0.64', username='unitree', password='123', |
|||
timeout=15, look_for_keys=False, allow_agent=False) |
|||
|
|||
# Check current remotes - origin already points to gitea |
|||
print('=== Current remotes ===') |
|||
_, o, _ = ssh.exec_command('cd ~/xr_teleoperate && git remote -v', timeout=10) |
|||
print(o.read().decode().strip()) |
|||
|
|||
# Check for local changes |
|||
print('\n=== Git status ===') |
|||
_, o, _ = ssh.exec_command('cd ~/xr_teleoperate && git status --short', timeout=10) |
|||
status = o.read().decode().strip() |
|||
print(status if status else '(clean)') |
|||
|
|||
# Pull from origin (which is already gitea) |
|||
print('\n=== Pulling from origin main ===') |
|||
_, o, e = ssh.exec_command('cd ~/xr_teleoperate && git pull origin main', timeout=30) |
|||
pull_out = o.read().decode().strip() |
|||
pull_err = e.read().decode().strip() |
|||
print(pull_out) |
|||
if pull_err: |
|||
print(pull_err) |
|||
|
|||
# Verify the changes are present |
|||
print('\n=== Verifying compute_fk exists ===') |
|||
_, o, _ = ssh.exec_command('grep -n "def compute_fk" ~/xr_teleoperate/teleop/robot_control/robot_arm_ik.py', timeout=5) |
|||
print(o.read().decode().strip() or 'NOT FOUND') |
|||
|
|||
print('\n=== Verifying --ki arg exists ===') |
|||
_, o, _ = ssh.exec_command('grep -n "\\-\\-ki" ~/xr_teleoperate/teleop/teleop_hand_and_arm.py', timeout=5) |
|||
print(o.read().decode().strip() or 'NOT FOUND') |
|||
|
|||
print('\n=== Verifying I-term logic exists ===') |
|||
_, o, _ = ssh.exec_command('grep -n "I-term" ~/xr_teleoperate/teleop/teleop_hand_and_arm.py', timeout=5) |
|||
print(o.read().decode().strip() or 'NOT FOUND') |
|||
|
|||
print('\n=== Latest commit ===') |
|||
_, o, _ = ssh.exec_command('cd ~/xr_teleoperate && git log --oneline -3', timeout=5) |
|||
print(o.read().decode().strip()) |
|||
|
|||
ssh.close() |
|||
print('\nDone!') |
|||
Write
Preview
Loading…
Cancel
Save
Reference in new issue