Parsing Logs and Tool Results

Every system, firewall and security tool produces logs: thousands of lines no one can read by eye. A few lines of Python can summarize them in seconds and reveal patterns such as "one address failed to sign in two hundred times". This is a core skill for defenders and a great way to practice for the detection chapter that comes next.

What a log line looks like

Here are fictional authentication log lines from a Linux server in your lab. The addresses are private lab addresses.

Oct 29 10:15:02 lab-srv01 sshd[1201]: Failed password for invalid user admin from 192.168.56.20 port 52114 ssh2
Oct 29 10:15:04 lab-srv01 sshd[1201]: Failed password for invalid user admin from 192.168.56.20 port 52116 ssh2
Oct 29 10:15:09 lab-srv01 sshd[1205]: Failed password for alice from 192.168.56.30 port 40222 ssh2
Oct 29 10:16:41 lab-srv01 sshd[1210]: Accepted password for alice from 192.168.56.30 port 40230 ssh2
Oct 29 10:17:12 lab-srv01 sshd[1214]: Failed password for invalid user test from 192.168.56.20 port 52210 ssh2

Each line has a timestamp, a host, a program, and a message. The message is free text, but it follows a pattern that we can describe with a regular expression.

Regular expressions in five minutes

A regular expression (regex) is a compact way to describe a text pattern. You only need a few symbols:

• \S+: one or more characters that are not spaces (a "word").
• \d+: one or more digits.
• (...): a group, meaning "capture this part".
• (?:...): a group that is used for structure but not captured.
• ? after something: that thing is optional.

Our pattern is: "Failed password for", optionally "invalid user", then the user name, then "from", then an address.

import re

FAILED = re.compile(
    r"Failed password for (?:invalid user )?(\S+) from (\d+\.\d+\.\d+\.\d+)"
)

line = "Oct 29 10:15:02 lab-srv01 sshd[1201]: Failed password for invalid user admin from 192.168.56.20 port 52114 ssh2"
match = FAILED.search(line)
if match:
    user, source = match.groups()
    print(user, source)    # admin 192.168.56.20

A tip: write the pattern in the REPL, test it on one line, then extend it. Never write a long regex in one go.

Counting with Counter

The standard library provides collections.Counter, a dictionary specialized in counting.

from collections import Counter

def count_failures(path):
    by_source = Counter()
    by_user = Counter()
    with open(path, encoding="utf-8", errors="replace") as f:
        for line in f:
            match = FAILED.search(line)
            if match:
                user, source = match.groups()
                by_source[source] += 1
                by_user[user] += 1
    return by_source, by_user

by_source, by_user = count_failures("auth.log")
print("Top sources:")
for source, n in by_source.most_common(5):
    print(f"  {source}: {n} failures")

The argument errors="replace" prevents a strange byte from crashing the script, which happens often with real logs. Reading the file line by line keeps memory use low even for huge files.

From counting to a simple alert

A single failure is normal (a user mistyped). Many failures from one source in a short time is a signal. Here is a readable version that uses a threshold:

THRESHOLD = 3

def suspicious_sources(by_source, threshold=THRESHOLD):
    return {src: n for src, n in by_source.items() if n >= threshold}

This is the seed of what a SIEM does in chapter 4: collect, count, compare to a threshold, raise an alert. In real life you would also add a time window, otherwise three failures spread over a year would count the same as three failures in ten seconds.

Parsing structured results from tools

Many tools can export JSON, a text format for structured data. JSON is much easier to process than free text, because you do not need a regex. Imagine a fictional scanner result saved in your own lab:

{
  "hosts": [
    {"address": "192.168.56.10", "ports": [{"port": 389, "service": "ldap"}, {"port": 445, "service": "smb"}]},
    {"address": "192.168.56.30", "ports": [{"port": 80, "service": "http"}]}
  ]
}

Reading it takes only a few lines with the standard json module:

import json

with open("lab-results.json", encoding="utf-8") as f:
    data = json.load(f)

for host in data["hosts"]:
    services = ", ".join(f"{p['port']}/{p['service']}" for p in host["ports"])
    print(f"{host['address']}: {services}")

This prints one line per host: 192.168.56.10: 389/ldap, 445/smb. The part inside join(...) is a generator expression, a compact loop that builds the text for each port. Do not worry if it looks dense: write it as a normal loop first, then compress it later.

Be careful with real data

• Real logs contain personal data: user names, addresses, sometimes more. Handle them as sensitive information and follow the retention rules agreed with the client.
• Never publish real logs in your portfolio. Generate fictional ones or use logs from your own lab.
• Check your regular expressions on several samples. A pattern that matches 90 percent of lines silently loses the rest, so count the lines that did not match as well.
• Logs can be forged, and timestamps may be wrong if clocks are not synchronized. A script reads what is written, it does not prove that it is true.

Your turn: a log summary

Write log_summary.py that reads auth.log (use the five lines above as a start, and add more of your own), and prints: (1) the total number of failed attempts, (2) the number of lines that did not match the pattern, (3) for each source that reached three or more failures, one line like ALERT 192.168.56.20: 3 failures.

import re
from collections import Counter

FAILED = re.compile(r"Failed password for (?:invalid user )?(\S+) from (\d+\.\d+\.\d+\.\d+)")
THRESHOLD = 3

def main():
    by_source = Counter()
    unmatched = 0
    total = 0
    with open("auth.log", encoding="utf-8", errors="replace") as f:
        for line in f:
            match = FAILED.search(line)
            if match:
                total += 1
                by_source[match.group(2)] += 1
            else:
                unmatched += 1
    print(f"Failed attempts: {total}")
    print(f"Lines not matching: {unmatched}")
    for source, n in by_source.most_common():
        if n >= THRESHOLD:
            print(f"ALERT {source}: {n} failures")

if __name__ == "__main__":
    main()

Portfolio step

Add log_summary.py, a small fictional auth.log and a README to your 02-python-security folder. In the README, explain in two sentences how the threshold could be tuned and why a time window would improve it. That shows you already think like a detection engineer.

In the last lesson of the chapter you will build a small audit script that actually talks to a web server, with strict guard-rails.