Recurrent Vision Transformers for Object Detection with Event Cameras (CVPR 2023)

We present Recurrent Vision Transformers (RVTs), a novel backbone for object detection with event cameras. Event cameras provide visual information with sub-millisecond latency at a high-dynamic range and with strong robustness against motion blur. These unique properties offer great potential for low-latency object detection and tracking in time-critical scenarios. Prior work in event-based vision has achieved outstanding detection performance but at the cost of substantial inference time, typically beyond 40 milliseconds. By revisiting the high-level design of recurrent vision backbones, we reduce inference time by a factor of 6 while retaining similar performance. To achieve this, we explore a multi-stage design that utilizes three key concepts in each stage: First, a convolutional prior that can be regarded as a conditional positional embedding. Second, local and dilated global self-attention for spatial feature interaction. Third, recurrent temporal feature aggregation to minimize latency while retaining temporal information. RVTs can be trained from scratch to reach state-of-the-art performance on event-based object detection - achieving an mAP of 47.2% on the Gen1 automotive dataset. At the same time, RVTs offer fast inference (less than 12 ms on a T4 GPU) and favorable parameter efficiency (5 times fewer than prior art). Our study brings new insights into effective design choices that can be fruitful for research beyond event-based vision.
Reference:
M. Gehrig, D. Scaramuzza
"Recurrent Vision Transformers for Object Detection with Event Cameras"
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, 2023
PDF: arxiv.org/abs/...
Code: github.com/uzh...
For more information about our research, visit these pages:
1. Event-based Vision: rpg.ifi.uzh.ch/...
2. Machine Learning: rpg.ifi.uzh.ch/...
Affiliations:
M. Gehrig and D. Scaramuzza are with the Robotics and Perception Group, Dep. of Informatics, University of Zurich, Switzerland.

[@user-kd1eo5lh5v]

Nice work! Thanks for the video!

[@betanapallisandeepra]

Good work

[@bowenjing3674]

can RVT fine tuning on other DVS dataset ?

[@weerachaithongsaart8948]

WSD789​i Toaccou nt