Using Falco to listen for runtime security

1. What is Falco

Falco is a cloud-native runtime security-related project contributed to CNCF by Sysdig.

Falco implements a scalable event rule filtering engine that finds security issues in a system by fetching events, matching security rules, and generating alert notifications. The events come from system calls and also support ebpf probes, and the rules are open source and can be extended by your own definitions. The following diagram shows its architecture.

Typical events that Falco can detect include:

• Running a shell in a container
• Running a container in a privileged form
• Reading sensitive data, such as `/etc/shadow
• Sensitive paths to container mount hosts
• Outbound network connections

2. Generating certificates

Falco’s gRPC requires two-way TLS authentication. falco exporter exposes relevant events via gRPC, and self-developed systems can also integrate Falco directly via gRPC. the following steps are used to generate the certificates required for the interaction. falco’s official documentation is somewhat outdated and some operations may cause exceptions.

• Create a certificate directory

  1 2	mkdir /root/falco cd /root/falco

• Create CA Certificate

  1 2 3 4 5 6

  openssl genrsa -out ca.key 4096 openssl req -x509 -new -nodes -sha512 -days 3650 \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=dev.chenshaowen.com" \ -key ca.key \ -out ca.crt

• Create server-side certificate

  1 2 3 4 5 6 7 8 9 10 11 12 13 14

  openssl genrsa -out server.key 4096 openssl req -sha512 -new \ -key server.key \ -out server.csr \ -subj "/C=SP/ST=Italy/L=Ornavasso/O=Test/OU=Server/CN=localhost" openssl x509 -req -sha512 \ -days 3650 \ -CA ca.crt \ -CAkey ca.key \ -in server.csr \ -out server.crt \ -set_serial 01

• Create client certificates

  1 2 3 4 5 6 7 8 9 10 11 12 13 14

  openssl genrsa -out client.key 4096 openssl req -sha512 -new \ -key client.key \ -out client.csr \ -subj "/C=SP/ST=Italy/L=Ornavasso/O=Test/OU=client/CN=localhost" openssl x509 -req -sha512 \ -days 3650 \ -CA ca.crt \ -CAkey ca.key \ -in client.csr \ -out client.crt \ -set_serial 01

• View all generated certificates

  1 2 3

  ls /root/falco ca.crt ca.key client.crt client.csr client.key server.crt server.csr server.key

3. Installing Falco on Kubernetes

• Add Helm source

  1 2	helm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update

• Install falco

  1 2 3 4 5 6 7 8 9 10 11

  helm install falco falcosecurity/falco \ --namespace falco --create-namespace \ --version 3.0.0 \ --set-file certs.ca.crt=/root/falco/ca.crt,certs.server.key=/root/falco/server.key,certs.server.crt=/root/falco/server.crt \ --set ebpf.enabled=true \ --set falco.grpc.enabled=true \ --set falco.grpc_output.enabled=true \ --set falcosidekick.enabled=true \ --set falcosidekick.webui.enabled=true \ --set falcosidekick.webui.user="admin:admin" \ --set falco.grpc.unixSocketPath=""

• Install falco-exporter

  1 2 3 4 5

  helm install falco-exporter falcosecurity/falco-exporter \ --namespace falco --create-namespace \ --version 0.9.1 \ --set falco.grpcTimeout=3m --set-file certs.ca.crt=/root/falco/ca.crt,certs.client.key=/root/falco/client.key,certs.client.crt=/root/falco/client.crt

• Check out the services

  During runtime it will request ghcr.io to download the default rules falco_rules.yaml.tar.gz, which may fail to download in network-constrained environments.

  1 2 3 4 5 6 7 8

  kubectl -n falco get pod -w NAME READY STATUS RESTARTS AGE falco-5bbl6 2/2 Running 0 152s falco-exporter-26gfz 1/1 Running 0 124s falco-falcosidekick-5c8bf5d7fb-kx778 1/1 Running 0 111s falco-falcosidekick-ui-5b56bbd7cb-5wdwl 1/1 Running 3 111s ...

• [Optional] Uninstall Falco

  1 2	helm uninstall falco --namespace falco helm uninstall falco-exporter --namespace falco

4. View Falco event data using Grafana panel

By looking at the falco-exporter svc you can see that it has exposed the metrics to Prometheus.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

kubectl -n falco get svc falco-exporter  -o yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9376"
    prometheus.io/scrape: "true"
  name: falco-exporter
  namespace: falco
spec:
  ports:
  - name: metrics
    port: 9376
    protocol: TCP
    targetPort: 9376
  selector:
    app.kubernetes.io/instance: falco-exporter
    app.kubernetes.io/name: falco-exporter

Next, just add the Grafana panel. Import 11914, which is https://grafana.com/grafana/dashboards/11914-falco-dashboard/. View the data as shown below.

However, the information exposed in Metrics is more limited. Query falco_events{rule="Read sensitive file untrusted"} in Prometheus and get the result.

1

falco_events{app_kubernetes_io_instance="falco-exporter", app_kubernetes_io_managed_by="Helm", app_kubernetes_io_name="falco-exporter", app_kubernetes_io_version="0.8.0", helm_sh_chart="falco-exporter-0.9.1", hostname="falco-h57xg", instance="1.1.1.1:9376", job="kubernetes-service-endpoints", k8s_ns_name="<NA>", k8s_pod_name="<NA>", namespace="falco", node="node1", priority="4", rule="Read sensitive file untrusted", service="falco-exporter", source="syscall", tags=",T1020,T1083,T1212,T1552,T1555,container,filesystem,host,mitre_credential_access,mitre_discovery,"} 

It does not show detailed information such as the user who executed it and the command executed, but only limited information such as the priority of the event and the rules triggered.

5. View events with falcosidekick-ui

falcosidekick mainly implements centralized management of events and provides rich alert channel capability, capable of sending alerts to slack, rocketchat, elasticsearch, etc.

And falco-falcosidekick-ui provides the ability to view falco events.

• Installing falco-falcosidekick-ui

  The following parameters were added when installing falco, so falcosidekick and falcosidekick-ui are already installed.

  1 2 3

  --set falcosidekick.enabled=true \ --set falcosidekick.webui.enabled=true \ --set falcosidekick.webui.user="admin:admin"

• Exposing the service port

  1 2	kubectl -n falco patch svc falco-falcosidekick-ui --patch \ '{"spec": { "type": "NodePort", "ports": [ { "nodePort": 32000, "port": 2802, "protocol": "TCP", "targetPort": 2802 } ] } }'

• Login page to view UI

  The falcosidekick ui page can be opened through the host IP:32000 port. The default account is admin and the default password is admin.

  After reading the sensitive file cat /etc/shadow on the host, you can see the events on the falcosidekick ui page, as shown below:

  

  Output content, since it is an operation on the host, will be missing cluster-related fields, but the information on files, commands, etc. will be quite rich.

  1	Warning Sensitive file opened for reading by non-trusted program (user=root user_loginuid=1001 program=cat command=cat /etc/shadow pid=54909 file=/etc/shadow parent=bash gparent=sudo ggparent=bash gggparent=sshd container_id=host image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=host

  The Dashboard page provides a view of global statistics that allows you to evaluate the security of clusters and hosts as a whole, with the following view.