Customizable Decay: How to Maximize Suricata Event Utility in Finite Space

Presented at SuriCon 2022 by Sascha Steinbiss, Matthias Vallentin & Benno Evers
In addition to its rule-based alerting, Suricata provides a rich metadata feed that summarizes network activity as structured events. This observed activity proves invaluable for post-hoc incident response, proactive threat hunting, and alert contextualization. In large environments, it is not trivial to back-haul this data to a central location. Large links generate terabytes of data daily, and data residency requirements often restrict data shipping. Given finite space at the edge, how do we maximize metadata retention span?
In this talk, we present design, implementation, and empirical analysis of metadata compaction of Suricata events in the VAST telemetry engine. We demonstrate how operators can flexibly configure incremental data aging to reduce the storage footprint of events gracefully. In contrast to the naive approach of deleting the oldest data points, compaction still retains key details for security analysis, preserving as much informational value as possible while gradually stripping expendable content from events.
In a case study, we show using concrete examples how this notion of event decay increases retention periods considerably. We report on our experience of running compaction in a live production environment at DCSO, where Suricata-based sensors feed a high-volume stream of EVE-JSON into VAST instances that store and continuously compact metadata.