g1-teleop/SETUP.md

G1 Teleop — Quest 3 Native App Setup

Overview

Godot 4.6.1 Quest 3 VR app for teleoperating a Unitree G1 humanoid robot. Uses OpenXR with Meta Quest hand tracking, body tracking, and passthrough.

Prerequisites

• Godot 4.6.1 — located at Godot4_6_1/Godot_v4.6.1-stable_win64_console.exe
• Android SDK + NDK — located at android-sdk/
• JDK 17 — located at jdk17/
• OpenXR Vendors plugin v4.3.0 — located at addons/godotopenxrvendors/
• adb — located at C:\Users\John\Downloads\platform-tools\adb.exe
• Quest 3 in developer mode

Build Pipeline

# 1. Setup vendor plugin (copies AARs and .so files)
python build/setup_vendor_plugin.py

# 2. Build APK (headless Godot export)
build\build_461.bat

# 3. Install to Quest 3
adb install -r build/g1-teleop.apk

The build script runs: Godot_v4.6.1-stable_win64_console.exe --headless --quit --path . --export-debug "Quest 3" build/g1-teleop.apk

Architecture

Two-Phase Startup

• CONFIG phase: Dark VR environment with start screen UI, G1 robot models spinning on each side, hand laser pointer with pinch-to-click for menu interaction
• AR phase: Meta Quest passthrough with body tracking, webcam quad, gaze-activated exit balls

Key Files

File	Purpose
Main.gd	Main controller — phase management, passthrough, gaze balls, G1 models, recenter
Main.tscn	Scene tree — XROrigin3D, camera, controllers, start screen, webcam quad
scripts/vr_ui_pointer.gd	Hand laser pointer, pinch-to-click, hand poke, controller ray interaction
scripts/body_tracker.gd	XRBodyTracker joint visualization and tracking data
scripts/teleop_client.gd	WebSocket client for robot communication
scripts/start_screen.gd	Config UI panel (connect, launch AR buttons)
scripts/webcam_quad.gd	Webcam video display
project.godot	Project settings including OpenXR extensions
export_presets.cfg	Android export config with Meta XR features
models/g1_full.obj	3D model of G1 robot (76MB OBJ, Z-up)

Critical Project Settings (project.godot)

openxr/extensions/meta/passthrough=true    # Required for Quest passthrough
openxr/extensions/meta/body_tracking=true  # Required for body tracking
openxr/environment_blend_mode=0            # Must stay 0, passthrough enabled via code

Critical Export Settings (export_presets.cfg)

meta_xr_features/hand_tracking=2
meta_xr_features/passthrough=2
meta_xr_features/body_tracking=2

Technical Notes

Meta Quest Passthrough

• Requires openxr/extensions/meta/passthrough=true in project.godot AND meta_xr_features/passthrough=2 in export presets
• get_supported_environment_blend_modes() only returns [0] (opaque) on Quest — must force set_environment_blend_mode(XR_ENV_BLEND_MODE_ALPHA_BLEND) without checking support
• The Meta vendor plugin intercepts this call via XR_FB_PASSTHROUGH extension
• Also requires transparent_bg = true on the viewport and clear_color = Color(0,0,0,0)
• Do NOT add a WorldEnvironment with BG_COLOR black — it makes everything black in VR

Hand Tracking

• Palm joint's -basis.z points perpendicular to palm (upward when hand flat), NOT forward
• Ray direction must be computed geometrically: wrist to middle finger metacarpal
• XRHandTracker joints: PALM=0, WRIST=1, THUMB_TIP=5, INDEX_TIP=10, MIDDLE_META=11
• Pinch detection: thumb tip to index tip distance with hysteresis (press <2.5cm, release >3.5cm)

Scene Parenting

• Nodes under XRCamera3D follow head tracking
• Nodes under XROrigin3D stay stationary in tracking space
• Webcam quad is under XROrigin3D (stationary), not XRCamera3D

OBJ Model Loading

• load("res://models/g1_full.obj") returns a Mesh resource directly (not PackedScene)
• OBJ files use Z-up; Godot uses Y-up — rotate -90 degrees on X axis to stand upright
• G1 model scale: Vector3(0.0015, 0.0015, 0.0015)
• Needs explicit material override and a DirectionalLight3D for shading

GDScript Gotchas

• Untyped Arrays return Variant — use explicit type: var ball: MeshInstance3D = _gaze_balls[i]
• Cannot infer types from Variant expressions — annotate variables explicitly

Gaze Ball System

• 4 balls above field of view: red (exit AR), yellow (reserved), green (reserved), blue (quit app)
• Head gaze detection via angle threshold (8 degrees)
• 5-second stare to activate with visual feedback (opacity, color, scale 1.0x to 1.8x)
• Laser beam from head to gazed ball

Recenter Support

• Connected to XRInterface.pose_recentered signal
• Must connect in both is_initialized() and elif branches of _ready()
• Repositions start screen + G1 models (CONFIG) or webcam + gaze balls (AR)

Network

• Quest 3 and robot must be on the same network
• No SSL required (raw WebSocket)
• Default port: 8765
• Firewall: allow TCP 8765 on the robot

Troubleshooting

Body tracking not activating

• Ensure Quest 3 system software is up to date
• Check Settings → Movement Tracking → Body Tracking is enabled
• Ensure the Meta OpenXR Vendors plugin version is ≥ 4.1.1

WebSocket connection fails

• Verify robot IP is correct and reachable: ping 10.0.0.64
• Check server is running: ss -tlnp | grep 8765
• Check firewall: sudo ufw allow 8765/tcp

Passthrough shows black/gray

• Verify openxr/extensions/meta/passthrough=true in project.godot
• Verify meta_xr_features/passthrough=2 in export_presets.cfg
• Do NOT set openxr/environment_blend_mode=2 in project.godot (keep it 0)
• Force alpha blend mode via code, not project settings
• Remove any WorldEnvironment node with black background