Looking at the yellow gaze ball for 5 seconds sends a recalibrate
command to the bridge, which resets calibration state — arms return
to zero and after 10s delay the user's neutral pose is recaptured.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wrist cameras are mounted sideways on the robot, so the raw images
need rotation to display upright in the Quest VR camera dashboard.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reads server host/port from start screen inputs and auto-connects
if not already connected when entering AR mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Godot 4 shares sub_resource materials across scene instances by default,
so all 3 camera quads were showing the same (last updated) texture.
Duplicating the material in _ready() gives each instance its own texture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Server: reads all 3 cameras from Isaac Sim SHM, encodes JPEG
(head Q85, wrists Q70), sends with 1-byte camera ID prefix.
Head at ~10fps, wrist cameras at ~5fps for bandwidth control.
Godot: teleop_client parses camera ID byte and emits per-camera
signals. Three webcam quads arranged in a row (left|head|right)
in front of the user in AR mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reads head camera from Isaac Sim shared memory, encodes as JPEG (quality 85),
and relays to Quest 3 over the existing WebSocket connection at ~15fps.
Falls back to left/right wrist cameras if head SHM unavailable.
Prevents Python resource_tracker from destroying SHM on bridge exit.
Also increases WebSocket write buffer and reduces send rate for stability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
R_FIX_HAND rotation corrects hand positions from body tracker frame (+X fingers)
to Inspire URDF frame (-Y fingers). Inverts normalization formula so open hand
maps to open robot hand. Adjusts inspire_hand.yml scaling/alpha for smoother tracking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dex-retargeting target_link_human_indices expect WebXR convention
(joint 0 = WRIST) but we were outputting OpenXR convention (joint 0 = PALM).
This caused the 5 difference vectors referencing index 0 to compute
palm-to-fingertip instead of wrist-to-fingertip, degrading tracking accuracy.
Also updates Inspire hand ranges and normalization formula.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Compute R_CORR empirically from measured rotation matrices:
R_CORR_L = Rz(+90°) @ Rx(180°), R_CORR_R = Rz(-90°) @ Rx(180°)
Body tracking wrist frame differs from hand tracking frame used
by the reference xr_teleoperate code.
- Switch finger joint data from XRBodyTracker to XRHandTracker.
XRBodyTracker doesn't provide articulated finger data on Quest 3;
XRHandTracker does via camera-based hand tracking.
25 joints: PALM + 24 finger joints (skip WRIST), matching
dex-retargeting's expected format.
- Re-enable hand retargeting in bridge main loop (was disabled
during arm IK testing).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
body_tracker.gd used Meta's raw XR_FB_body_tracking indices but Godot's
XRBodyTracker.get_joint_transform() expects Godot's own enum values.
The OpenXR vendors plugin remaps between them, so e.g. index 16
(Meta RIGHT_ARM_LOWER) was actually returning LEFT_FOOT data in Godot,
explaining why right elbow tracking showed floor-level positions.
Also switches retarget_bridge.py from Euler angle decomposition to IK
solver (Pinocchio/CasADi) with elbow position constraints from body
tracking l_lower/r_lower joints.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace flat JOINT_GAIN with per-joint JOINT_SCALE/JOINT_OFFSET linear mapping
for accurate retargeting (l_shoulder_pitch/roll/yaw, l_elbow calibrated)
- Switch shoulder parent from chest to hips to avoid head rotation contamination
- Add solo-joint mode (--solo-joint) for isolating single joints during testing
- Use XRHandTracker wrist transforms in body_tracker.gd for better wrist rotation
- Add joint_test.py for single-joint animation with pause-at-peak support
- Add joint_calibrate.py for XR-to-robot range calibration with scale+offset output
- Add calibration_log.md documenting extent tuning and axis mappings
- Disable waist/right arm/hands temporarily for left-arm-only testing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Quest system keyboard doesn't work in immersive XR mode, so added a
numpad overlay that appears when tapping input fields.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a joint retargeting bridge that converts Quest 3 body tracking
data into G1 robot joint commands for Isaac Sim via CycloneDDS.
- Extend body_tracker.gd with 8 body joint transforms (56 floats)
- Extend teleop_server.py with body_joints_shared memory
- Add retarget_bridge.py: body retargeting, hand mapping, DDS publishing
- Add launch_bridge.py: one-command launcher for sim + bridge
- Update requirements.txt with unitree-sdk2py dependency
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Hand tracking laser pointer with pinch-to-click gesture for menu interaction
- Meta Quest passthrough (alpha blend + XR_FB_PASSTHROUGH extension)
- Stationary webcam quad (moved from XRCamera3D to XROrigin3D)
- Gaze-activated exit balls (4 balls: exit AR, 2 reserved, quit app)
- Gaze laser beam and ball growth visual feedback
- Spinning G1 3D model flanking the start screen menu
- Recenter support via pose_recentered signal
- Exit AR mode returns to CONFIG phase with UI repositioning
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Two-phase startup: CONFIG phase (VR start screen) -> AR phase (passthrough)
- Start screen with server host/port input, connect button, and launch AR button
- VR UI pointer with controller ray-pointing and hand poke interaction
- Body tracking visualization with colored spheres (spine, head, arms)
- Hand tracking with camera-based tracking (no controllers required)
- Android flavor manifest for Quest hand/body tracking permissions
- Upgrade OpenXR Vendors plugin from v3.1.2 to v4.3.0-stable
- Export presets updated for Godot 4.6.1 Meta plugin format
- Build scripts for Godot 4.6.1 with vendor plugin setup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bundle godotopenxrvendors addon (Meta, Khronos, Pico, MagicLeap)
for self-contained repo — no manual plugin download needed
- Remove stale editor_plugins section from project.godot
(GDExtension auto-loads via plugin.gdextension, not plugin.cfg)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Godot 4.3 project for Quest 3 that replaces the browser-based WebXR
teleoperation with a native app using XR_FB_body_tracking (70 joints).
Key advantage: chest-relative wrist tracking eliminates head-position
subtraction artifacts and decouples body rotation from arm control.
Godot app: body_tracker.gd, teleop_client.gd, webcam_display.gd
Robot server: teleop_server.py (WebSocket, replaces Vuer ~900→~250 lines)
Drop-in wrapper: native_tv_wrapper.py (compatible with teleop_hand_and_arm.py)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
melancholytron
melancholytron
Joe DiPrima