@ -1,63 +1,110 @@
# G1 Teleop — Quest 3 Native App Setup# G1 Teleop — Quest 3 Native App Setup
## Overview
Godot 4.6.1 Quest 3 VR app for teleoperating a Unitree G1 humanoid robot. Uses OpenXR with Meta Quest hand tracking, body tracking, and passthrough.
## Prerequisites## Prerequisites
- **Godot 4.3+** (download from https://godotengine.org)
- **Android SDK + NDK** (install via Android Studio or standalone)
- **Meta Quest Developer Hub** (for sideloading) or `adb`
- **Godot 4.6.1** — located at `Godot4_6_1/Godot_v4.6.1-stable_win64_console.exe`
- **Android SDK + NDK** — located at `android-sdk/`
- **JDK 17** — located at `jdk17/`
- **OpenXR Vendors plugin v4.3.0** — located at `addons/godotopenxrvendors/`
- **adb** — located at `C:\Users\John\Downloads\platform-tools\adb.exe`
- Quest 3 in **developer mode**- Quest 3 in **developer mode**
## Step 1: Install Meta OpenXR Vendors Plugin
## Build Pipeline
1. Open project in Godot: `godot --editor --path C:\git\g1-teleop`
2. Go to **AssetLib** tab (top center)
3. Search for "Godot OpenXR Vendors"
4. Download and install (includes Meta Quest support)
5. Enable the plugin: **Project → Project Settings → Plugins → godotopenxrvendors** → Enable
6. Restart Godot when prompted
## Step 2: Configure Android Export
```bash
# 1. Setup vendor plugin (copies AARs and .so files)
python build/setup_vendor_plugin.py
1. Go to **Editor → Editor Settings → Export → Android**
2. Set the path to your Android SDK
3. Set Java SDK path
4. Go to **Project → Export → Quest 3**
5. Download Android export templates if prompted (Editor → Manage Export Templates → Download)
# 2. Build APK (headless Godot export)
build\build_461.bat
## Step 3: Enable Body Tracking
# 3. Install to Quest 3
adb install -r build/g1-teleop.apk
```
Body tracking is configured via:
- `export_presets.cfg` : `xr_features/hand_tracking=2` (required)
- `android/AndroidManifest.xml` : body tracking features and metadata
The build script runs: `Godot_v4.6.1-stable_win64_console.exe --headless --quit --path . --export-debug "Quest 3" build/g1-teleop.apk`
The Meta OpenXR Vendors plugin v4.1.1+ exposes `XRBodyTracker` in GDScript.
## Architecture
## Step 4: Build and Sideload
### Two-Phase Startup
- **CONFIG phase** : Dark VR environment with start screen UI, G1 robot models spinning on each side, hand laser pointer with pinch-to-click for menu interaction
- **AR phase** : Meta Quest passthrough with body tracking, webcam quad, gaze-activated exit balls
1. Connect Quest 3 via USB (or Wi-Fi ADB)
2. In Godot: **Project → Export → Quest 3 → Export Project** (or one-click deploy)
3. APK will be at `build/g1-teleop.apk`
4. Sideload: `adb install build/g1-teleop.apk`
### Key Files
## Step 5: Robot Server
| File | Purpose |
|------|---------|
| `Main.gd` | Main controller — phase management, passthrough, gaze balls, G1 models, recenter |
| `Main.tscn` | Scene tree — XROrigin3D, camera, controllers, start screen, webcam quad |
| `scripts/vr_ui_pointer.gd` | Hand laser pointer, pinch-to-click, hand poke, controller ray interaction |
| `scripts/body_tracker.gd` | XRBodyTracker joint visualization and tracking data |
| `scripts/teleop_client.gd` | WebSocket client for robot communication |
| `scripts/start_screen.gd` | Config UI panel (connect, launch AR buttons) |
| `scripts/webcam_quad.gd` | Webcam video display |
| `project.godot` | Project settings including OpenXR extensions |
| `export_presets.cfg` | Android export config with Meta XR features |
| `models/g1_full.obj` | 3D model of G1 robot (76MB OBJ, Z-up) |
On the robot (or dev machine for testing):
### Critical Project Settings (`project.godot`)
```bash
cd server/
pip install -r requirements.txt
python teleop_server.py --port 8765
```ini
openxr/extensions/meta/passthrough=true # Required for Quest passthrough
openxr/extensions/meta/body_tracking=true # Required for body tracking
openxr/environment_blend_mode=0 # Must stay 0, passthrough enabled via code
``````
## Step 6: Configure App
### Critical Export Settings (`export_presets.cfg`)
Edit `scripts/teleop_client.gd` to set the robot's IP:
```gdscript
@export var server_host: String = "10.0.0.64" # Robot IP
@export var server_port: int = 8765
```ini
meta_xr_features/hand_tracking=2
meta_xr_features/passthrough=2
meta_xr_features/body_tracking=2
``````
Or configure at runtime via the Godot editor's export properties.
## Technical Notes
### Meta Quest Passthrough
- Requires `openxr/extensions/meta/passthrough=true` in project.godot AND `meta_xr_features/passthrough=2` in export presets
- `get_supported_environment_blend_modes()` only returns `[0]` (opaque) on Quest — must force `set_environment_blend_mode(XR_ENV_BLEND_MODE_ALPHA_BLEND)` without checking support
- The Meta vendor plugin intercepts this call via `XR_FB_PASSTHROUGH` extension
- Also requires `transparent_bg = true` on the viewport and `clear_color = Color(0,0,0,0)`
- Do NOT add a WorldEnvironment with BG_COLOR black — it makes everything black in VR
### Hand Tracking
- Palm joint's `-basis.z` points perpendicular to palm (upward when hand flat), NOT forward
- Ray direction must be computed geometrically: wrist to middle finger metacarpal
- XRHandTracker joints: PALM=0, WRIST=1, THUMB_TIP=5, INDEX_TIP=10, MIDDLE_META=11
- Pinch detection: thumb tip to index tip distance with hysteresis (press < 2.5cm , release > 3.5cm)
### Scene Parenting
- Nodes under `XRCamera3D` follow head tracking
- Nodes under `XROrigin3D` stay stationary in tracking space
- Webcam quad is under `XROrigin3D` (stationary), not `XRCamera3D`
### OBJ Model Loading
- `load("res://models/g1_full.obj")` returns a `Mesh` resource directly (not PackedScene)
- OBJ files use Z-up; Godot uses Y-up — rotate -90 degrees on X axis to stand upright
- G1 model scale: `Vector3(0.0015, 0.0015, 0.0015)`
- Needs explicit material override and a DirectionalLight3D for shading
### GDScript Gotchas
- Untyped Arrays return Variant — use explicit type: `var ball: MeshInstance3D = _gaze_balls[i]`
- Cannot infer types from Variant expressions — annotate variables explicitly
### Gaze Ball System
- 4 balls above field of view: red (exit AR), yellow (reserved), green (reserved), blue (quit app)
- Head gaze detection via angle threshold (8 degrees)
- 5-second stare to activate with visual feedback (opacity, color, scale 1.0x to 1.8x)
- Laser beam from head to gazed ball
### Recenter Support
- Connected to `XRInterface.pose_recentered` signal
- Must connect in both `is_initialized()` and `elif` branches of `_ready()`
- Repositions start screen + G1 models (CONFIG) or webcam + gaze balls (AR)
## Network## Network
@ -77,3 +124,10 @@ Or configure at runtime via the Godot editor's export properties.
- Verify robot IP is correct and reachable: `ping 10.0.0.64` - Verify robot IP is correct and reachable: `ping 10.0.0.64`
- Check server is running: `ss -tlnp | grep 8765` - Check server is running: `ss -tlnp | grep 8765`
- Check firewall: `sudo ufw allow 8765/tcp` - Check firewall: `sudo ufw allow 8765/tcp`
### Passthrough shows black/gray
- Verify `openxr/extensions/meta/passthrough=true` in project.godot
- Verify `meta_xr_features/passthrough=2` in export_presets.cfg
- Do NOT set `openxr/environment_blend_mode=2` in project.godot (keep it 0)
- Force alpha blend mode via code, not project settings
- Remove any WorldEnvironment node with black background
