Linux Endpoint Triage: A Practical DFIR Workflow

Cybersecurity

Linux DFIR gets less attention than Windows, which is exactly why attackers love Linux servers. This is the triage workflow we run with UAC and AVML when a containerized host or VPS goes hot.

By Arjun Raghavan, Security & Systems Lead, BIPI · March 5, 2024 · 8 min read

Linux servers carry the weight of most modern infrastructure and far less of the DFIR mindshare. When a production Ubuntu box gets popped, the team often improvises with tar and grep. There are better tools, and the order matters.

Trigger signals worth taking seriously

Unexpected outbound traffic to a low-reputation ASN, an SSH session from a country you do not operate in, a sudden CPU spike on a host that runs a steady workload, a new file in /tmp or /var/tmp with execute bits. Any one of these is worth a triage collection. Two together is an incident.

First-hour collection with UAC

Unix-like Artifacts Collector (UAC) by Tclahr is the closest thing Linux has to KAPE. One binary, one config file, runs across Linux, macOS, BSD, Solaris, AIX. Drop it on the host, run it with the ir_triage profile, and walk away with a hashed archive of everything useful. Capture memory with AVML from Microsoft first, since memory is volatile and AVML works across kernels without needing to compile LiME.

• avml /mnt/evidence/host.lime, captures full memory in LiME format
• uac -p ir_triage /mnt/evidence/, pulls logs, persistence, history, network state
• Hash both outputs with sha256sum and store the hash off-host
• If the host is in a container orchestrator, snapshot the container before killing it

The log files that tell the story

Linux logging is fragmented and that works in the attacker's favor when you do not know where to look. The triage list is shorter than people think. auth.log shows SSH authentication, sudo usage, and session start. syslog and messages cover the kernel and most services. journalctl with --since gives you a unified view if the host runs systemd. Bash history is per-user and trivially evaded but still worth reading.

1. /var/log/auth.log, SSH logins, sudo invocations, failed auths, source IPs
2. /var/log/syslog and /var/log/messages, service-level events, cron runs, kernel messages
3. journalctl --since '7 days ago', full systemd journal with reverse chronology
4. ~/.bash_history for every user, plus /root/.bash_history, plus zsh and fish equivalents
5. /var/log/audit/audit.log if auditd was running, which catches syscalls bash history cannot

Persistence: the Linux flavors

Linux persistence is a long buffet. systemd unit files in /etc/systemd/system and /usr/lib/systemd/system are the modern favorite. Cron survives via /etc/cron.* and per-user crontabs in /var/spool/cron. SSH keys in ~/.ssh/authorized_keys are the quietest backdoor in the world. SUID binaries planted in /usr/local/bin give an attacker a way back from a user shell. LD_PRELOAD hooks in /etc/ld.so.preload are rare but devastating.

Quick triage commands that punch above their weight

Before you run UAC, you can answer most questions with a handful of commands. Run them as root from a tmux session you can attach to from a clean workstation.

• ss -tunap shows live listeners and connections with PID, faster than netstat
• ps auxf gives the process tree, look for orphans and processes with deleted binaries
• ls -la /proc/*/exe 2>/dev/null | grep deleted finds processes whose binary was unlinked
• lsof -p PID for any suspicious process gives you open files and sockets
• stat on suspicious files reveals birth time, which attackers rarely think to forge

Memory analysis without a custom kernel module

Volatility 3 supports Linux memory analysis as long as you have a matching symbol table. The Linux symbol generation has gotten easier but still trips teams up. Generate the ISF JSON with dwarf2json against the running kernel before the incident if you can. If not, Volexity maintains a public repo of common symbol tables for popular distributions and kernel versions.

Where Linux IR goes wrong

Teams forget that containers are not isolation. A compromised container with a runtime CVE can escape to the host. When you triage a container compromise, also triage the host. Check kernel logs for capability use you did not expect, audit the docker.sock mount, and look at the kubelet logs if you are on Kubernetes. The investigation does not stop at the container boundary.

Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.