Shipping nginx-ingress logs with filebeat on Kubernetes

This took me a good afternoon to figure out.

The main difficulty was figuring out how to make the nginx module process the log output of the container input correctly, and also finding out that the ingress_controller fileset for nginx does not actually exist in v7.6.2, which was the latest at time of writing.

I'm not going to cover setting up Elasticsearch and Kibana here.

Deploying filebeat with Helm

Use the following values.yml as a base. This will send all the nginx-ingress logs to your Elasticsearh cluster. Once v7.7 is out, it should contain the ingress_controller fileset which will extract better metadata from the log output.

---
filebeatConfig:
  filebeat.yml: |
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition:
                equals:
                  kubernetes.container.name: "nginx-ingress-controller"
              config:
              - module: nginx
                access:
                  enabled: true
                  input:
                    type: container
                    paths:
                    - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
      username: '${ELASTICSEARCH_USERNAME}'
      password: '${ELASTICSEARCH_PASSWORD}'
    setup.kibana:
      host: "<kibana path>"