Event-based Asynchronous Sparse Convolutional Networks (ECCV 2020 Presentation)

Event cameras are bio-inspired sensors that respond to per-pixel brightness changes in the form of asynchronous and sparse "events". Recently, pattern recognition algorithms, such as learning-based methods, have made significant progress with event cameras by converting events into synchronous dense, image-like representations and applying traditional machine learning methods developed for standard cameras. However, these approaches discard the spatial and temporal sparsity inherent in event data at the cost of higher computational complexity and latency. In this work, we present a general framework for converting models trained on synchronous image-like event representations into asynchronous models with identical output, thus directly leveraging the intrinsic asynchronous and sparse nature of the event data. We show both theoretically and experimentally that this drastically reduces the computational complexity and latency of high-capacity, synchronous neural networks without sacrificing accuracy. In addition, our framework has several desirable characteristics: (i) it exploits spatio-temporal sparsity of events explicitly, (ii) it is agnostic to the event representation, network architecture, and task, and (iii) it does not require any train-time change, since it is compatible with the standard neural networks' training process. We thoroughly validate the proposed framework on two computer vision tasks: object detection and object recognition. In these tasks, we reduce the computational complexity up to 20 times with respect to high-latency neural networks. At the same time, we outperform state-of-the-art asynchronous approaches up to 24% in prediction accuracy.
Reference:
Nico Messikommer*, Daniel Gehrig*, Antonio Loquercio, and Davide Scaramuzza
European Conference on Computer Vision (ECCV), 2020.
PDF: rpg.ifi.uzh.ch/...
Code: github.com/uzh...
Our research page on event based vision: rpg.ifi.uzh.ch/...
Our research page on machine learning: rpg.ifi.uzh.ch/...
For event-camera datasets, see here:
rpg.ifi.uzh.ch/...
and here: github.com/uzh...
For an event camera simulator: rpg.ifi.uzh.ch/...
For a survey paper on event cameras, see here:
rpg.ifi.uzh.ch/...
Other resources on event cameras (publications, software, drivers, where to buy, etc.):
github.com/uzh...
Affiliations:
The authors are with the Robotics and Perception Group, Dep. of Informatics, University of Zurich, and Dep. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland
rpg.ifi.uzh.ch/

[@MaksymCzech]

Great stuff!

[@paulkirkland9860]

Amazing work!.. I'm a researcher of spiking neural networks so I obv see the parallels between the two. I'd be very interested to know what the weights and feature maps are active in the network and what they look like now that the operations have a large amount of sparsity. As when using hebbian update rules the networks tend to learn object shapes not textures as in normal CNNs. It would be good to know if the training methodology might need to alter to take into account the sparse nature of the input.

[@gptty]

So exciting 😃