Spatial and Temporal Beliefs for Mistake Detection in Assembly Tasks

Computer Vision and Image Understanding (CVIU) 2025 · Workshop version: Ordering Mistake Detection in Assembly Tasks, CVPR Workshops 2024

Guodong Ding¹, Fadime Sener², Shugao Ma² and Angela Yao¹

¹National University of Singapore   ²Reality Labs, Meta

Abstract

One promising use case of AI assistants is to help with complex procedures like cooking, home repair, and assembly tasks. Can we teach the assistant to interject after the user makes a mistake? This paper targets the problem of identifying ordering mistakes in assembly procedures. We propose a system that can detect ordering mistakes by utilizing a learned knowledge base. Our framework constructs a knowledge base with spatial and temporal beliefs based on observed mistakes. Spatial beliefs depict the topological relationship of the assembling components, while temporal beliefs aggregate prerequisite actions as ordering constraints. With an episodic memory design, our algorithm can dynamically update and construct the belief sets as more actions are observed, all in an online fashion. We demonstrate experimentally that our inferred spatial and temporal beliefs are capable of identifying incorrect orderings in real-world action sequences. To construct the spatial beliefs, we collect a new set of coarse-level action annotations for Assembly101 based on the positioning of the toy parts. Finally, we demonstrate the superior performance of our belief inference algorithm in detecting ordering mistakes on the Assembly101 dataset.

Citation

@article{ding2025spatial,
  title   = {Spatial and Temporal Beliefs for Mistake Detection in Assembly Tasks},
  author  = {Ding, Guodong and Sener, Fadime and Ma, Shugao and Yao, Angela},
  journal = {Computer Vision and Image Understanding},
  year    = {2025}
}

@inproceedings{ding2024ordering,
  title     = {Ordering Mistake Detection in Assembly Tasks},
  author    = {Ding, Guodong and Sener, Fadime and Ma, Shugao and Yao, Angela},
  booktitle = {CVPR Workshops},
  year      = {2024}
}