SIEM and Log Correlation: The Principles

One log line rarely tells a story. A failed sign-in could be a typo. A new process could be an update. A transfer at night could be a backup. But put them together, in order, and you may see a break-in unfold. A SIEM exists to do exactly that: gather events from everywhere and connect the dots. In this lesson you will understand how it works and write your first correlation rules, in plain language.

What is a SIEM?

SIEM stands for Security Information and Event Management. It is a platform that:

• Collects logs from many sources: servers, workstations, firewalls, cloud services, applications, security tools.
• Normalizes them, which means converting different formats into a common structure.
• Stores them so that analysts can search quickly, including events from the past.
• Correlates them, by applying rules and analytics that raise alerts.
• Presents them in dashboards and case views for analysts.

Think of a detective's wall covered with photos, notes and red strings. Each source is a witness; the SIEM is the wall that lets you see how their stories connect.

The journey of a log

1. Generation: a system writes an event, for instance "user alice failed to sign in".
2. Collection: an agent or a forwarding service sends it to a central point.
3. Parsing and normalization: the SIEM extracts fields such as time, user, source address and result, and names them consistently across sources.
4. Enrichment: adds context, such as the department of the user, the criticality of the machine, or whether an address is internal or external.
5. Detection: rules and analytics evaluate the event, alone or with others.
6. Alerting and response: an alert is created, assigned and investigated. Automated actions may run.
7. Retention: logs are kept for the period required by law, contracts or investigation needs, then deleted.

Every step can fail silently. If a source stops sending logs, your detections for it become blind. Monitoring the health of the log pipeline is as important as writing rules.

Time is everything

Correlation depends on timestamps. If one server's clock is five minutes ahead, events appear in the wrong order and your reconstruction becomes nonsense. That is why all systems must synchronize their clocks with a reliable time source and why logs should record time zones clearly. A good habit: store and compare in a single reference time (UTC) and convert only for display.

What to collect first

Collecting everything is expensive and noisy. Start with sources that give the best value for security:

• Identity and authentication: sign-ins, failures, password changes, group membership changes. This includes the directory events from chapter 1.
• Endpoints: process creation, security tool alerts.
• Network boundary: firewall, proxy, DNS and VPN logs.
• Critical servers and applications: especially those that hold sensitive data.
• Cloud control plane: logs of administrative actions in cloud accounts (chapter 5).

A principle used by many teams: first decide the questions you want to answer ("who signed in to the finance server last night?") and then collect the logs that answer them.

Correlation: from events to meaning

A correlation rule describes a pattern of events. Here are three types, in plain language:

• Threshold: "More than ten failed sign-ins for one account within five minutes."
• Sequence: "A burst of failed sign-ins from one source, followed by a success from the same source, within ten minutes."
• Rarity or anomaly: "An administrator signs in from a country never seen before for that account."

Rules are usually written in the query language of the platform, but the logic is the same everywhere. A fictional pseudocode version of the sequence rule looks like this:

WHEN
  count(event = "signin_failed", by source, within 5 minutes) >= 10
  AND THEN event = "signin_success" from the same source
  within 10 minutes
RAISE alert "Possible password guessing succeeded"
  severity = high
  include: account, source, first_failure_time, success_time

It is the same idea as your chapter 2 script, but with a time window, a sequence and an alert.

False positives and how to tame them

A rule that fires on legitimate behavior is useless noise. Typical causes and fixes:

• A scanner or backup tool that behaves like an attacker: add a documented exception by source or account.
• A threshold too low: look at normal behavior for a few weeks, then set it above it.
• Missing context: enrich with asset criticality so the same event on a test server and a payroll server get different priorities.
• Duplicates: group alerts that describe the same activity into one case.

Every exception is a small hole in your detection. Document each one with an owner and a review date.

Use cases: the unit of detection work

Mature teams manage their detections as use cases. Each one has a name, the threat it addresses, the data it needs, the logic, expected false positives, the severity, the response steps and a test. Writing this down turns a rule from someone's private idea into a maintained asset. It also connects naturally with the ATT&CK mapping that comes in lesson 4.3.

Limits of a SIEM

• It only sees what is logged. Attackers prefer places with poor logging.
• Logs can be disabled or tampered with on compromised machines, which is why logs are forwarded quickly to a separate system.
• Poor rules produce poor results at great cost: storage, licenses and analyst time.
• It supports people but does not replace them. Investigation still needs judgment.

Privacy and law

Logs contain personal data: user names, addresses, sometimes browsing information. In Europe, the GDPR requires a legitimate purpose, proportionality, access control and retention limits. Employees should be informed about monitoring. Involve the data protection officer and the legal team when designing a collection plan. This is not an obstacle; it is part of professional security.

Your turn: write two rules

For your lab, write in plain language or pseudocode two correlation rules: (1) a sequence rule for a user who is added to a very powerful group and, within 30 minutes, signs in to a domain controller; (2) a threshold rule for a single account requesting service tickets for many different services in a short time. For each, say one likely false positive and how you would reduce it.

RULE 1: Privileged group change followed by sensitive sign-in
WHEN event = "member_added_to_privileged_group"
  THEN event = "signin_success" on a domain controller
       by the same account, within 30 minutes
RAISE high

RULE 2: Many service ticket requests by one account
WHEN count(distinct service, by account, within 10 minutes) >= 10
RAISE medium
include: account, source, list of services

Portfolio step

Create a folder 04-detection-response in your portfolio and add use-cases.md with your two rules written as full use cases: name, threat, data needed, logic, false positives, severity and response steps. Detection engineering is a sought-after skill, and a few well-documented use cases are an excellent sample.

Next, you will learn what exactly a detection looks for: indicators and behaviors.