The moments the camera cannot see
Vision-conditioned policies go blind exactly when precision matters: at contact, the gripper and the object occlude each other. Sound is the cheap way back in, but it is a sparse signal in a dense trajectory. In our target task, the audio that distinguishes the two objects lasts 2 to 3 seconds out of a 60 to 70 second episode, so the policy's easiest route to low training loss is to ignore the microphone entirely.
Vision, sound and joint states in one policy
- Extended Diffusion Policy conditioning from vision alone to three streams: a 512-dimensional ResNet-18 image feature over an observation history of 2, a 264-dimensional audio embedding from 1D convolutions over the spectrogram of the last two seconds from a gripper-mounted microphone, and an 8-dimensional joint state (7 joint positions plus gripper width). All fused into the noise prediction network with FiLM.
- Receding-horizon execution: predict Tp = 16 actions, execute Ta = 8. DDIM with about 10 denoising steps at inference to run in real time on the Franka.
- Three tasks of increasing difficulty: fixed-location pick and place, vision-based pick and place from a random start pose, and identifying which of two visually identical bags holds coins from the sound it makes when dropped.
- Data collected by a finite state machine rather than teleoperation: 100, 185 and 120 trajectories at 10 Hz for the three tasks. Trained 400 epochs, batch size 128, Adam at 1e-4, on an RTX 3090.
- A standalone audio classifier (coin bag drop, empty bag drop, no drop) used to initialize the audio encoder. It reached about 95% accuracy on its own.
On pick and place from a random start pose the diffusion policy succeeded 85% of the time, against 15% for a fully connected behavior-cloning baseline trained on the same demonstrations. Both reached 100% on the fixed-location task, where vision is not needed.
What the diffusion policy avoided
- The failure mode the diffusion policy avoids is getting stuck. The fully connected baseline had lower validation MSE and still froze mid-trajectory; when it did not freeze it finished the task. Validation loss was the wrong signal for rollout success, which is the practical argument for action-sequence prediction over single-step regression.
- Spectrograms made the discrimination easy in isolation: a bag of coins rings noticeably longer on impact than one of paper, and a small 1D convolutional classifier picked that up at about 95% accuracy.
- Scripting demonstrations with a state machine instead of teleoperating produced smoother, more consistent trajectories and made datasets of 100 to 185 trajectories a 5 to 6 hour job rather than a skill-intensive one.