Watch the cluster at runtime with Falco

Admission and scanners look at objects before they run. Falco looks at what the process actually does: a shell in a container that should not have one, a write under /bin, a read of /etc/shadow.

It is a userspace engine plus a kernel probe (eBPF, or the older kernel module). Rules match syscall fields. Output goes to stdout, a file, syslog, or an HTTP webhook.

Install

DaemonSet via Helm is the usual cluster install:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
  --namespace falco-system \
  --create-namespace

kubectl get pods -n falco-system
kubectl get daemonset -n falco-system

On a node with the package:

systemctl status falco
journalctl -fu falco

Files that matter

Path Role
/etc/falco/falco.yaml Engine config, outputs, which rule files to load
/etc/falco/falco_rules.yaml Default rules. Do not edit these.
/etc/falco/falco_rules.local.yaml Your rules and overrides
/etc/falco/k8s_audit_rules.yaml Kubernetes audit-driven rules

In-cluster, that config is usually a ConfigMap. After you change it:

kubectl rollout restart daemonset/falco -n falco-system
kubectl logs -f -l app.kubernetes.io/name=falco -n falco-system

Turn on a file you can tail

# falco.yaml
file_output:
  enabled: true
  keep_alive: false
  filename: /var/log/falco_events.txt

stdout_output:
  enabled: true

json_output: true

Restart, then tail -f /var/log/falco_events.txt.

A local rule

# /etc/falco/falco_rules.local.yaml
- rule: Package Management in Container
  desc: Detect package managers running in containers
  condition: >
    spawned_process and container and
    proc.name in (apt, apt-get, yum, rpm, apk, pip)
  output: >
    Package manager in container (proc=%proc.name container=%container.name)
  priority: WARNING
  tags: [process, container]

Disable a noisy default without touching the shipped file:

- rule: Some Default Rule Name
  enabled: false

Validate before restart:

falco --validate /etc/falco/falco.yaml
falco --list-rules

Prove it

kubectl exec -it <pod> -- /bin/bash
kubectl exec -it <pod> -- touch /bin/test-file
kubectl exec -it <pod> -- cat /etc/shadow

You should see the matching rule in the Falco log. If you see nothing, the DaemonSet is not on that node, the rule did not load, or you edited falco_rules.yaml and the next upgrade wiped it.

Keep conditions short. Restart after every config change. Watch journalctl -fu falco while you test.