Blog

Top 5 Tips and Tricks to Measure, Manage, and Improve your Detection Portfolio

SIEM

By Josh Neil, Chief Data Scientist, Leigh Shlomovich, Senior Data Scientist

Receiving too many alerts has burdened security operations teams for many years, and the problem only seems to increase as more data and alerts are ingested and processed. In this post, we share some tips and tricks to lessen the number of alerts per day and ensure that your analysts are not missing out on any real attacks.

If your analysts are exposed to more than ~100 alerts per day, it might be time to tune your detection portfolio. Failure to tune could lead to miserable analysts, wasted money, and worst of all, missed attacks. Here are some ways to reduce alerts without missing true positives:

Tune up your detectors: Your detectors are probably throwing too many false positives (FPs) at your security operations team. By tuning your detection portfolio, you can reduce the number of FPs, making it easier for analysts to focus on the actual incidents.
Combine alerts together: Instead of showing analysts individual alerts, combine them to show more holistic incidents. This is something analysts are doing manually anyway, so why not automate the process? Securonix provides this through our Threat Models capability.
Ask your analysts: Your analysts are the ones dealing with these alerts on a daily basis, so why not ask them what’s noisy and what’s not? They might have insights that you haven’t considered.
Risk scoring: Use risk scoring to help analysts prioritize alerts. However, it’s best to get rid of the noise before you start quantifying what’s left. This capability is provided out of the box for Securonix customers,
Grading your detectors: If you’re feeling ambitious, you can grade your detectors with humans. To calculate precision, you need to take a sample of alerts generated by the detector and assign them True Positive (TP) / False Positive (FP) labels. Then you can use the following formula: precision = TP / (TP + FP)

For example, suppose you take a sample of 100 alerts generated by a detector and find that 60 of them are true positives and 40 of them are false positives. Then the precision of the detector would be:

precision = 60 / (60 + 40) = 60%

”You can’t manage what you can’t measure” — Peter Drucker

Figures 1 and 2: An example showing how bad policies can make your analysts miserable. On the left, we see the time series of a specific policy firing on a repeated basis, and generating nearly a million alerts in a period of three weeks. On the right, we see that one user account is causing almost all of these alerts (in orange), and upon further investigation was determined to be a poorly configured policy.

We should emphasize that maintaining your detection portfolio is crucial and requires ongoing efforts. Your detectors can decrease in precision and recall, and it’s important to have a good detection management process in place to mitigate this. The best detection management processes enable teams to quantify performance on a regular basis and have a review team empowered to enable and disable detectors.

Reducing the number of alerts is crucial for the well-being of your analysts and the security of your company. By tuning your detectors, combining alerts, asking your analysts, using risk scoring, and grading your detectors, you can ensure that your analysts are focusing on the real incidents and not missing any potential attacks.