Container Forensics: When the Evidence Disappears with the Pod

Cybersecurity

Containers are designed to be ephemeral. That is a feature for ops and a problem for IR. Falco runtime alerts, CRIU checkpoints, kubectl debug, overlay filesystem inspection, and the sysdig captures that save the case before the pod restarts.

By Arjun Raghavan, Security & Systems Lead, BIPI · July 8, 2024 · 9 min read

Container forensics is fundamentally harder than endpoint forensics because the artifact you want to image will be gone before you finish reading the alert. A crashed pod restarts, a scaled-down deployment evaporates, and the overlay filesystem that held the attacker's tooling gets garbage collected. The trick is to design the runtime such that you have evidence already, not to scramble for it after the fact.

Falco is the cheap insurance policy

Falco is the open-source runtime detection engine that watches syscalls and emits alerts on rule matches. The default rule pack flags shell spawned inside a container, write below /etc, package management run in container, and unexpected outbound connection. None of those are sophisticated detections; all of them catch real intrusion behaviour. Crucially, Falco emits the event before the container exits, so you have a record even when the pod is gone.

Run Falco as a DaemonSet, ship its output to a log pipeline that lives outside the cluster, and treat the alerts as evidence rather than just notifications. The first time you investigate a pod that no longer exists, the Falco event log will be the only thing you have.

Checkpoint with CRIU before you kill

When you do catch a live container that you suspect, your first instinct should not be to kubectl delete it. CRIU (Checkpoint Restore In Userspace) lets you snapshot the running process tree, including memory, into a set of image files on disk. containerd and CRI-O both expose checkpoint APIs in 2024, and kubectl checkpoint exists as an alpha feature in recent Kubernetes versions. The result is a forensic image you can analyse offline.

Ephemeral containers and kubectl debug

If the pod is still running and you do not want to disturb the workload, kubectl debug attaches a new container into the same pod's namespaces. You get a shell with your own tooling image (busybox, alpine, a forensics image with strace, lsof, tcpdump) that shares network, PID, and sometimes mount namespaces with the target. This is how you observe a live container without modifying it. ephemeral containers were stable in Kubernetes 1.25; if you are still on older clusters, kubectl exec into a sidecar is a poor substitute that you should plan to retire.

Inspecting the overlay filesystem

Each container has a writable layer on top of its image, usually under /var/lib/docker/overlay2/ or /var/lib/containerd/. If you know the container ID, you can find its upper directory and inspect the files the attacker dropped, even after the container has stopped, as long as you have not yet pruned. Pull a tar of the upper directory before any cleanup. crictl inspect gives you the container ID and the runtime-specific path to the layer.

For images, treat the image registry as forensic evidence too. If the attacker pushed a malicious image, the registry layer manifests, the build provenance (if you use Sigstore or in-toto attestations), and the registry audit logs will tell you when and from where the push happened.

sysdig captures for the deep dive

sysdig (the open-source tool, not just the company) records every syscall to a capture file you can replay offline. Running sysdig -w capture.scap on the node for the duration of an active incident gives you a complete behavioural trace of every container on that node. Replay it with sysdig -r capture.scap and you can ask after-the-fact questions you did not think of at the time. It is heavy at scale but invaluable for the duration of an active investigation on a specific node.

Why containers are forensics-hard

The list is short and the mitigations are obvious once you write them down:

The runbook to write before you need it

Decide ahead of time who has cluster-admin or break-glass access to checkpoint pods, take captures, and pull layer tarballs. Decide what your standard forensic image is and pre-pull it to every node, so you do not have to wait for a registry fetch during an incident. Decide where the evidence lands (an S3 bucket with object lock is fine) and how long you keep it. None of this is technically hard. All of it is impossible to organise at 3 a.m. on the night of an actual intrusion.

Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.