How it runs
The real inference stack, with the hardware swapped for physics.
Calibrated RGB-D camera
A virtual RealSense D405 at the real hand-eye calibration renders 640×480 RGB-D, cropped exactly like CamRS.
Released perception
Masks → KptsSelector → multi-frame depth lift → PCA object frames → VisualKpts. The arm is removed and a virtual gripper is drawn, as on the robot.
Released policy
ICTPolicy loads the public checkpoint and solves the flow ODE (20 Euler steps) for a 50-step hand trajectory. The left arm sees a mirrored world.
Real-rig controller
Temporal ensembling, safety cage, EMA/Slerp, look-ahead → a Trossen-driver emulator (quintic re-targeting + IK) → MuJoCo, where the bread is a soft body that squashes in the gripper and is held by friction alone. Each robot runs its self-tuned settings + grasp skill (table below).
Success rates
Randomized initial conditions following the paper's protocol.
Self-tuning loop
Simulator and low-level controller parameters, optimized by an agent-in-the-loop population search.
Run it yourself
Same code, any rig: swap robot or camera with one flag.
# one-time setup git clone https://github.com/TX-Leo/HumanEgo.git && cd HumanEgo pip install -r requirements.txt -r simulation/requirements.txt huggingface-cli download Leo-TX/HumanEgo --include "serve_bread/*" --local-dir ./checkpoints python simulation/scripts/build_assets.py # one closed-loop episode (+ video): real-rig controller, or the shipped tuned one + grasp skill python simulation/scripts/run_episode.py --video python simulation/scripts/run_episode.py --profile tuned --robot franka_panda --camera zed2i --mode single # success rate over randomized scenes, and the self-tuning loop python simulation/scripts/evaluate.py --profile tuned --n 40 --workers 8 python simulation/scripts/autotune.py --name my_rig # this page: build every config, pack, record the videos (never deploys) python simulation/scripts/rebuild_site.py