A Needle in a Haystack: How to Find a Threat Hidden in Over 6 Billion Logs Per Day – Brian Davis
This video features Brian Davis from Red Canary discussing their approach to detecting security threats within the massive volume of cloud environment logs. He explains their six-stage pipeline: Ingest, Standardize, Combine, Detect, Suppress, and Respond, which processes billions of logs daily by focusing on the cloud control plane and its telemetry. This modular strategy, built on the Unix philosophy and leveraging cloud-native services like S3, SNS, SQS, and Kubernetes, allows Red Canary to efficiently identify malicious activities and adapt to evolving cloud threats.