Recurrent Vision Transformers for Object Detection with Event Cameras (CVPR 2023)
HTML-код
- Опубликовано: 5 сен 2024
- We present Recurrent Vision Transformers (RVTs), a novel backbone for object detection with event cameras. Event cameras provide visual information with sub-millisecond latency at a high-dynamic range and with strong robustness against motion blur. These unique properties offer great potential for low-latency object detection and tracking in time-critical scenarios. Prior work in event-based vision has achieved outstanding detection performance but at the cost of substantial inference time, typically beyond 40 milliseconds. By revisiting the high-level design of recurrent vision backbones, we reduce inference time by a factor of 6 while retaining similar performance. To achieve this, we explore a multi-stage design that utilizes three key concepts in each stage: First, a convolutional prior that can be regarded as a conditional positional embedding. Second, local and dilated global self-attention for spatial feature interaction. Third, recurrent temporal feature aggregation to minimize latency while retaining temporal information. RVTs can be trained from scratch to reach state-of-the-art performance on event-based object detection - achieving an mAP of 47.2% on the Gen1 automotive dataset. At the same time, RVTs offer fast inference (less than 12 ms on a T4 GPU) and favorable parameter efficiency (5 times fewer than prior art). Our study brings new insights into effective design choices that can be fruitful for research beyond event-based vision.
Reference:
M. Gehrig, D. Scaramuzza
"Recurrent Vision Transformers for Object Detection with Event Cameras"
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, 2023
PDF: arxiv.org/abs/...
Code: github.com/uzh...
For more information about our research, visit these pages:
1. Event-based Vision: rpg.ifi.uzh.ch/...
2. Machine Learning: rpg.ifi.uzh.ch/...
Affiliations:
M. Gehrig and D. Scaramuzza are with the Robotics and Perception Group, Dep. of Informatics, University of Zurich, Switzerland.
Nice work! Thanks for the video!
Good work
can RVT fine tuning on other DVS dataset ?
WSD789i Toaccou nt