OnlineTAS: An Online Baseline for Temporal Action Segmentation

Conference on Neural Information Processing Systems (NeurIPS) 2024

Qing Zhong^1,2, Guodong Ding² and Angela Yao²

¹University of Adelaide   ²National University of Singapore

Abstract

Temporal context plays a significant role in temporal action segmentation. In an offline setting, the context is typically captured by the segmentation network after observing the entire sequence. However, capturing and using such context information in an online setting remains an under-explored problem. This work presents the first online framework for temporal action segmentation. At the core of the framework is an adaptive memory designed to accommodate dynamic changes in context over time, alongside a feature augmentation module that enhances the frames with the memory. In addition, we propose a post-processing approach to mitigate the severe over-segmentation in the online setting. On three common segmentation benchmarks, our approach achieves state-of-the-art performance.

Citation

@inproceedings{zhong2024online,
  title     = {OnlineTAS: An Online Baseline for Temporal Action Segmentation},
  author    = {Zhong, Qing and Ding, Guodong and Yao, Angela},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2024}
}