Incident Response and Forensics

The Clock Starts at Detection#

Incident response (IR) is the organised handling of a security event: working out what happened, stopping it, removing the attacker, restoring service and learning from it. It is the discipline you hope never to use, and the one that most decides the final cost. The difference between a quiet two-hour containment and a three-week outage with regulators involved is usually the quality of the preparation that existed before the alert, not the heroics of whoever responds.

The Lifecycle#

Two classic models dominate, and they describe the same work at different granularity:

NIST's current guidance, SP 800-61 Rev. 3 (April 2025), replaced the four-phase lifecycle with a model built on the six Functions of the NIST Cybersecurity Framework (CSF) 2.0. Govern, Identify and Protect are the broader risk-management work that prepares for incidents; Detect, Respond and Recover are the incident response itself; and lessons learned from every Function feed the Improvement category (under Identify) and flow back into all of them. The point of the change is that incident response is part of everyday risk management, not a separate procedure pulled out during a crisis. The classic phases are still the vocabulary most playbooks and training use, so the rest of this lesson is organised around them.

In practice the phases overlap and loop. You contain a host, discover a second compromised account while scoping, and go back to analysis. The model is a checklist to make sure nothing is skipped, not a strict sequence.

Preparation Decides Everything#

Before an incident, have these in place:

• A written plan with named roles: incident commander (runs the response, makes decisions), technical lead, communications lead, scribe (keeps the timeline), and contacts for legal, leadership and any external IR retainer.
• Playbooks for likely scenarios: phishing with credential theft, ransomware, leaked cloud key, compromised web server, insider data theft.
• Logging that already exists: you cannot investigate what you did not record. Centralised, retained, tamper-resistant logs are a preparation item. See detection and SIEM.
• Out-of-band communication: if the attacker is in your email or chat workspace, they can read your response. Agree an alternative channel in advance.
• Tested backups and a known-good way to rebuild systems.
• Access for responders: break-glass accounts, EDR console access, permission to isolate hosts, cloud read access to audit logs.

Identification and Triage#

An alert, a user report or a third-party notification arrives. Triage answers quickly: is this real, how bad is it, and who needs to know?

Useful triage questions: What systems and accounts are involved? When did it start (the first evidence, not the first alert)? Is it still active? What data could be reached from there? Write down every answer with a timestamp and a source; the timeline starts now.

Containment: Choose Deliberately#

Containment stops the damage from spreading. Every option trades speed against evidence and business disruption:

Two rules of thumb. First, isolate rather than power off when you may need memory. Second, when the attacker is active and aware, contain everything you know about at the same time. Disabling one account while leaving their other footholds alone teaches them to hide better.

Evidence and Forensics#

Forensics is the evidence half of IR: capturing what happened so you can reconstruct the attacker's actions and, if needed, prove them. The rules:

• Collect in order of volatility (the principle in RFC 3227): memory and running state first, then network connections, then disk, then logs and backups. A reboot erases the first two.
• Hash everything at acquisition and record who collected it, when, how and where it is stored. This is the chain of custody. See hashing.
• Work on copies, never the original disk or image.
• Record your own actions, because responder activity also shows up in the logs you will later analyse.

A minimal Linux live-response collection, run from trusted tools on external media where possible:

#!/bin/sh
OUT=/mnt/usb/ir_$(hostname)_$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$OUT"
date -u                      > "$OUT/collected_at.txt"
ps auxwwf                    > "$OUT/processes.txt"
ss -tunap                    > "$OUT/connections.txt"
{ who; last -n 50; }         > "$OUT/logins.txt" 2>&1
ls -la /tmp /dev/shm         > "$OUT/tmp_listing.txt"
crontab -l                   > "$OUT/crontab.txt" 2>&1
cp /var/log/auth.log* "$OUT/" 2>/dev/null
sha256sum "$OUT"/*           > "$OUT/SHA256SUMS"

For memory, dedicated tools such as AVML or LiME (Linux) and WinPmem (Windows) produce a raw image that Volatility 3 can analyse (for example vol -f mem.raw windows.pslist to list processes, including ones hidden from the live OS). In cloud environments, snapshot the disk volumes, export the relevant audit logs, and preserve object-store version history before anyone "cleans up".

Building the Timeline#

The timeline is the spine of the investigation. Merging several log sources by time is often the first real analysis step:

import csv, re
from datetime import datetime, timezone

events = []

# SSH logins from auth.log (syslog format, year not included)
ssh = re.compile(r"^(\w{3} +\d+ [\d:]+) \S+ sshd\[\d+\]: Accepted (\S+) for (\S+) from (\S+)")
for line in open("auth.log", encoding="utf-8", errors="replace"):
    m = ssh.match(line)
    if m:
        ts = datetime.strptime(f"2026 {m.group(1)}", "%Y %b %d %H:%M:%S").replace(tzinfo=timezone.utc)
        events.append((ts, "ssh", f"{m.group(3)} logged in via {m.group(2)} from {m.group(4)}"))

# Web requests from a combined-format access log
web = re.compile(r'^(\S+) \S+ \S+ \[([^\]]+)\] "(\S+) (\S+)[^"]*" (\d{3})')
for line in open("access.log", encoding="utf-8", errors="replace"):
    m = web.match(line)
    if m:
        ts = datetime.strptime(m.group(2), "%d/%b/%Y:%H:%M:%S %z").astimezone(timezone.utc)
        events.append((ts, "web", f"{m.group(1)} {m.group(3)} {m.group(4)} -> {m.group(5)}"))

events.sort()
with open("timeline.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["utc_time", "source", "event"])
    for ts, src, text in events:
        w.writerow([ts.isoformat(), src, text])

Reading the merged file, you look for the story: a burst of 404s probing admin paths, a POST to an upload endpoint returning 200, and eight minutes later an SSH login for www-data from the same IP. Three pitfalls: syslog lines omit the year and may be in local time (check the server's timezone rather than assuming UTC as this sketch does), clocks on different hosts drift if NTP is broken, and log rotation may have already deleted the earliest evidence.

Eradication and Recovery#

• Eradicate every foothold found during scoping: malware, persistence entries, web shells, new accounts, added SSH keys, OAuth app grants, cloud access keys created by the attacker.
• Rotate credentials that were exposed or reachable, including service accounts and API keys. Rotation order matters: rotate the ones the attacker could use to regain access first. See secrets management.
• Close the entry point: patch, reconfigure or remove the vulnerable component. Recovering without this invites the same attacker back through the same door.
• Rebuild instead of clean where trust is lost, restoring data from backups taken before the compromise began (your timeline tells you when that was).
• Monitor closely afterwards for the indicators you found. Attackers frequently try to return.

Communication and Legal Obligations#

Decide in advance who talks to whom. Many regulations impose notification deadlines once a breach of personal data is confirmed; for example, the GDPR generally requires notifying the supervisory authority within 72 hours of becoming aware of a qualifying breach. Contracts, sector rules and other jurisdictions add their own. Involve legal counsel early, keep statements factual, and avoid speculating about scope before the evidence supports it.

Lessons Learned#

Hold a blameless review within a couple of weeks: the timeline, what worked, what slowed the team down, and concrete actions with owners. Good outputs are specific: "add alerting on new OAuth app consents", "reduce log retention gaps on the VPN appliance", "make the isolation playbook usable without domain admin". A review that ends in "be more careful" changes nothing.

Practice#

Run a tabletop exercise: take the scenario "a developer's cloud access key was committed to a public repository six hours ago". Walk through who is paged, how you find what the key did (hint: the audit log in cloud security), what you rotate, and who decides whether customers are notified. Note every step where the answer was "we do not know"; those are your preparation tasks. For malware-specific handling, see malware types.