Blog

Adding Spark to Accelerate Security Management

SIEM

Written By David O’Hara, Security Engineer at Securonix

Many organizations today are faced with a common challenge when handling potential threats within their environments, and that is the time to execute various security management steps from detection to resolution. The security management process in most organizations involves at least 3 steps:

Threat detection – results in security alerts
Forensic analysis – process for incident validation (removal of false positives)
Incident response – including remediation, clean up and documentation

The longer this process takes, the greater the chances that the organization will suffer a major cyber security incident. Current tools in use for security incident and event management were not designed for and IT landscape that includes multiple devices for every user, mobile workforce, and petabytes of IT data. Because of this mismatch, SOC analysts now face thousands of alerts every day, quickly succumbing to alert fatigue and corresponding backlogs.

The fundamental reason current tools hinder this process is because the log data and events from these systems do not have the necessary context needed today. This forces a high degree of manual labor in each stage. Starting with a very inaccurate detection process compounds the issue at later stages.

The challenge is the same whether handling employee related incidents or cybersecurity related incidents, and there are several common factors that impact the security management process:

Limited visibility across data sources and security landscape
Segmented tools or solutions
Limited staff or skill sets
Limited or no analytics

User and entity behavior analytics, or UEBA, solutions help bring together the data sources we need to quickly investigate an alert to determine if it’s a benign event or a true incident. The standard data sources that are commonly leveraged for these solutions include:

AD Security Logs
Endpoint Logs (Windows or EDR)
DNS
Firewall Logs
VPN Logs
IDS\IPS Logs
AV Logs

These data sources will typically provide visibility into who users are, where they are on our network, what systems they’ve logged into, are there any signature-based threats running on the systems, and finally where they’ve gone outside of our network or come into it.

We can potentially answer the questions of who, when, and where, but to answer the what, as in what was accessed or taken, then we need to add more security related data sources, such as:

DLP Logs
Identity Access Management Logs
Application Logs

The challenge for many organizations at this point becomes not only the number of data sources collected but the size of the data being collected that must now be processed and indexed for search and review. Also note that I didn’t include cloud solutions in the equation, which adds another data source layer, but also adds more visibility into our security landscape.

The Securonix SNYPR solution has addressed this challenge by moving to a highly scalable and open data platform based on Hadoop. This next-generation security analytics platform combines log management, SIEM, UEBA, and fraud detection to quickly provide actionable intelligence.

At the core of the technology stack is Apache Spark. Spark is a robust processing engine that is designed to perform real-time processing and has been used to perform ETL (Extract, Transform and Load) and data analysis on data sizes upwards of a Petabyte. By enabling real-time streaming enrichment and analysis of incoming data, the platform is able to dramatically improve the detection, and more importantly the rejection of irrelevant events, and therefore accelerating every downstream step in the security management process.

To learn more about Securonix SNYPR platform and how it leverages this technology please watch our webinar, SPARK On Hadoop Is The Next Gen SIEM.