Low Latency Automotive Vision with Event Cameras (Nature, 2024)

The computer vision algorithms used in today’s advanced driver assistance systems rely on image-based RGB cameras, leading to a critical bandwidth-latency trade-off for delivering safe driving experiences. To address this, event cameras have emerged as alternative vision sensors. Event cameras measure changes in intensity asynchronously, offering high temporal resolution and sparsity, drastically reducing bandwidth and latency requirements. Despite these advantages, event camera-based algorithms are either highly efficient but lag behind image-based ones in terms of accuracy or sacrifice the sparsity and efficiency of events to achieve comparable results. To overcome this, we propose a novel hybrid event- and frame-based object detector that preserves the advantages of each modality and thus does not suffer from this tradeoff. Our method exploits the high temporal resolution and sparsity of events and the rich but low temporal resolution information in standard images to generate efficient, high-rate object detections, reducing perceptual and computational latency. We show that the use of a 20 Hz RGB camera plus an event camera can achieve the same latency as a 5,000 Hz camera with the bandwidth of a 45 Hz camera without compromising accuracy. Our approach paves the way for efficient and robust perception in edge-case scenarios by uncovering the potential of event cameras.
Daniel Gehrig, Davide Scaramuzza
Low Latency Automotive Vision with Event Cameras.
Nature, May 29th, 2024
PDF Open Access: www.nature.com...
Code: github.com/uzh...
Dataset: dsec.ifi.uzh.c...
Dataset Helper Tools: github.com/uzh...
For an event camera simulator: rpg.ifi.uzh.ch/...
For a survey paper on event cameras, see here: rpg.ifi.uzh.ch...
Other resources on event cameras (publications, software, drivers, where to buy, etc.): github.com/uzh...
Affiliation:
D. Gehrig and D. Scaramuzza are with the Robotics and Perception Group, University of Zurich, Switzerland rpg.ifi.uzh.ch/

[@zhuweikai6867]

so great work！

[@benjaminsawicki29]

great work!

[@karun427]

Impressive work

[@nicolascapetillo4086]

Very nice!

[@user-rg1jp2us4o]

Audio is not working on right side.

[@lightjarvis363]

ALARMING! This work can't be repeated now , all the issue pulled on github hasn't been processed, I wonder wether this programm is real.

[@JariVasell]

Congrats to the team! Did Elon call already? 😄

[@The-Dark-Tower]

Where can we order such Event Cameras?

[@Graveness4920]

this system is bound to fail because you cant predict whats you havent seen yet.

[@robertopopitanu]

Our brain work just like this event cameras workflow

[@RaspberryDrone-s8h]

excellent work!