Adding Spark to Accelerate Security Management
SIEM
Written By David O’Hara, Security Engineer at Securonix
Many organizations today are faced with a common challenge when handling potential threats within their environments, and that is the time to execute various security management steps from detection to resolution. The security management process in most organizations involves at least 3 steps:
- Threat detection – results in security alerts
- Forensic analysis – process for incident validation (removal of false positives)
- Incident response – including remediation, clean up and documentation
The longer this process takes, the greater the chances that the organization will suffer a major cyber security incident. Current tools in use for security incident and event management were not designed for and IT landscape that includes multiple devices for every user, mobile workforce, and petabytes of IT data. Because of this mismatch, SOC analysts now face thousands of alerts every day, quickly succumbing to alert fatigue and corresponding backlogs.
The fundamental reason current tools hinder this process is because the log data and events from these systems do not have the necessary context needed today. This forces a high degree of manual labor in each stage. Starting with a very inaccurate detection process compounds the issue at later stages.
The challenge is the same whether handling employee related incidents or cybersecurity related incidents, and there are several common factors that impact the security management process:
- Limited visibility across data sources and security landscape
- Segmented tools or solutions
- Limited staff or skill sets
- Limited or no analytics
User and entity behavior analytics, or UEBA, solutions help bring together the data sources we need to quickly investigate an alert to determine if it’s a benign event or a true incident. The standard data sources that are commonly leveraged for these solutions include:
- AD Security Logs
- Endpoint Logs (Windows or EDR)
- DNS
- Firewall Logs
- VPN Logs
- IDS\IPS Logs
- AV Logs
These data sources will typically provide visibility into who users are, where they are on our network, what systems they’ve logged into, are there any signature-based threats running on the systems, and finally where they’ve gone outside of our network or come into it.
We can potentially answer the questions of who, when, and where, but to answer the what, as in what was accessed or taken, then we need to add more security related data sources, such as:
- DLP Logs
- Identity Access Management Logs
- Application Logs
The challenge for many organizations at this point becomes not only the number of data sources collected but the size of the data being collected that must now be processed and indexed for search and review. Also note that I didn’t include cloud solutions in the equation, which adds another data source layer, but also adds more visibility into our security landscape.
The Securonix SNYPR solution has addressed this challenge by moving to a highly scalable and open data platform based on Hadoop. This next-generation security analytics platform combines log management, SIEM, UEBA, and fraud detection to quickly provide actionable intelligence.
At the core of the technology stack is Apache Spark. Spark is a robust processing engine that is designed to perform real-time processing and has been used to perform ETL (Extract, Transform and Load) and data analysis on data sizes upwards of a Petabyte. By enabling real-time streaming enrichment and analysis of incoming data, the platform is able to dramatically improve the detection, and more importantly the rejection of irrelevant events, and therefore accelerating every downstream step in the security management process.
To learn more about Securonix SNYPR platform and how it leverages this technology please watch our webinar, SPARK On Hadoop Is The Next Gen SIEM.
