From one human video to a robot simulation
The four DexAgent stages, each run by an LLM agent with a tool library and judged by independent verifiers. Every artifact below is the actual output of this run.
Human demonstration
inputHumanEgo serve_bread episode 040 — Aria glasses, 15.1 s, right hand. Converted to DexAgent's EgoDex format:
per-frame head-camera pose (MPS SLAM) and the 3D hand skeleton (MPS hand tracking) — the head moves only 2 cm / 3°.
Semantic understanding
passed
The agent reads the video through contact sheets and writes scene_semantics.json: two rigid objects and three subgoals with measurable goals.
Property-based reconstruction
passed
SAM 3 masks, the EgoDex camera, then objects built by library skills and placed where the video shows them on the Tianji bench. The croissant needed a shape no skill had: the agent wrote and tested a new one.
Robot trajectory generation
passedGrasp synthesis, 7-DOF arm IK, and execution in contact physics with motor-torque arms and position-servoed fingers. The human's whole-hand wrap did not survive the static screen; the agent converged on a top-down thumb–index pinch over the croissant's centre of mass and removed a 30–85 N release spike by tracing six finger-geometry causes.
Robot data generation
running at scaleDexAgent Stage 4 re-draws the object layout (here ±7 cm, ±35°), anchors each hand segment to the object it acts on, re-solves the arm, and keeps a layout only if a full contact-physics replay passes every subgoal check. The accepted episodes are rendered from the robot's ZED and wrist cameras with domain randomization — see Generated data.
The agent at work
Rounds, tool calls and verifier verdicts per stage, for the original (right-arm) and the mirrored (left-arm, real-rig) runs. No API key: a Claude subagent answered DexAgent's own harness through its file transport.
Self-evolving tool library
Tools the agent created and tested through library_create during these runs; they are now part of the library.
A second task: water the flowers, with both arms
One more human video, the same pipeline: the left hand pulls the faucet's spray head out of its dock and holds it over the potted flowers, the right hand swings the lever on. Replay it in 3D above (Run → Water the flowers).
The human demonstration
Aria · 19.4 sReconstruction
passedThe setup's table top became the sink floor (the sink sits below the counter in the video); the faucet is held by a keyed shank, the spray head by a 6 N magnet in its dock (comes off at 4.7 N), the lever is a hinge with friction so it stays where it is pushed; an open pot of tulips stands in the sink.
The robot does it in contact physics
all gates passedLeft: a side pinch pulls the head 5.5 cm off the magnet and holds it 0.2 mm from its target over the pot's mouth; right: it rises out of the basin, pinches the lever tip from the front and swings it to 23.6° (on from 17.5°) while the left keeps holding. Both hold to the end.
Photoreal
The verified episodes rendered with Blender Cycles: polished chrome faucet, brushed-steel sink, stone counter, the pull-down hose redrawn every frame, a glazed pot of tulips with translucent petals — and the agent's episode composited back into the human's kitchen from the head camera.
How the agent got there
Every stage session, including DexAgent's cross-stage repair loops (a later stage re-opens an earlier one with evidence) and two gaps in the library's own audits that this task exposed and that were fixed with regression tests.
Tools the agent wrote for this task
Created and tested through library_create; now part of the library.
Generated data
Stage 4 randomizes the pot inside the sink (the faucet and its docked head stay mounted); every layout is re-simulated in contact physics, then rendered from the ZED, both wrists and a randomized third-person camera.
Generated data
Every episode is a physics replay that passed DexAgent's checks; each is rendered twice with a different appearance draw. ZED 2i (robot head) + wrist camera, 640×480 @ 15 fps, with depth, 58-joint states and actions, hand and object poses, phase labels.
Start layouts
Croissant start rotation
Final distance to plate centre
Action → next state
# one episode of datasets/serve_bread_040_tianji/ ep_shard_01_0007_r0/ ego.mp4 # ZED 2i, 640x480 @ 15 fps wrist_right.mp4 # wrist camera of the acting hand depth_ego.npz # uint16 depth (mm) data.npz # qpos/qvel/action [T, 58] by joint name (14 arm + 44 hand), ee/* and object/* poses, labels meta.json # source sample + layout, randomization draw, camera intrinsics/extrinsics, instruction
Photoreal robot-camera data
The same physics-verified episodes re-rendered with Blender Cycles from the robot's ZED and wrist cameras: PBR robot (white satin paint, black anodised wrist, graphite torso, rubber fingertip pads), a glazed ceramic plate in the colour measured from the human video, a baked-crust croissant with subsurface scattering and a contact dent where the fingers press, real indoor HDRI rooms or a studio cyclorama. The robot stays exactly where the physics put it.
Same frames, three renderers
Photoreal episodes
ZED 2i + wrist, 640×480, with depth and the full state/action streams (same format as the fast dataset); every appearance knob drawn per episode.
Agent-designed diversity
An agent decides which axes of variation matter for the task and how wide each may be, then every draw is rebuilt and re-verified in contact physics.
The spec the agent wrote
What survived physics
Diverse episodes
Each with its own instruction, bread, plate, table height, grasp, motion and speed — and the look the designer drew for it (table finish, lamp colour temperature and direction, room colour, ZED offset).
The same diverse samples, photoreal
Every diverse sample re-rendered with Blender Cycles in a real HDRI room, the bread's own shape rebuilt at high resolution with a crust per type (scored roll, glossy bun, cut baguette), the plate in the variant's glaze, and the designed table finish, lamps and ZED offset.
All 71 photoreal diverse episodes
Soft bread
Softness is a physical property of the simulated bread, the DexAgent way: compliant contact on the loaf (construct_tapered_loaf softness), calibrated per bread type in the real Tianji/Sharpa pinch, checked by soft_squeeze_test and a new soft_pinch_test. The photoreal dent at each fingertip is the recorded penetration depth itself.
All 71 diverse samples
Faster and cheaper, same data
Engineering only: every component, check, audit and the contact-physics replay are kept. The speed work leaves Stage 4's outputs bitwise identical; two bugs in its checks were fixed, so more of the drawn layouts survive.
Photoreal composites
DexAgent's own Stage 4 render path: Blender Cycles renders of the robot and objects composited onto the human video's clean background plate, plus the wrist and ZED views.
Porting DexAgent to our rig
DexAgent ships for a 6-DOF YAM bench; everything needed for a 7-DOF Tianji torso with SharpaWave hands and a ZED head camera.
Aria → EgoDex
MPS SLAM poses rotated z-up → y-up, the 21 MPS hand landmarks mapped to the 25 ARKit joints. A mirrored copy turns the right-hand demo into a left-hand one for the real rig.
Tianji + Sharpa + ZED
Generated from far_dexbot.urdf: arms in DexAgent's naming, the hand clocked 180° on the flange as on the real robot, the ZED from the hand-eye calibration, home pose by IK.
7-DOF arms
Motor caps 108/108/66/66/18/18/18 N·m, gains and inertias from the setup; a computed-torque feed-forward removes the 18–80 N contact spikes the heavy arm caused; MPC/RL residuals sized per arm.
Robot-camera renders
Stage 4 samples rendered from the ZED and wrist with MuJoCo (ms per frame) and domain randomization; Cycles composites for the photoreal subset; a clean plate without VOID.