Skip to content

Logging and Monitoring

ESS Pro integrates with industry standard Kubernetes logging and monitoring systems. It does not deploy its own monitoring stack, rather it makes it easy to integrate into existing monitoring stacks via the Prometheus Operator ServiceMonitor (or other compatible implementations).

By default, the ESS Pro Helm chart detects the presence of ServiceMonitors in the cluster and constructs appropriate instances for each backend component.

Example Monitoring Stack

To demonstrate the logging and monitoring capabilities an example stack is documented below. This is not a production ready stack.

Loki for storing logs

Loki is a log aggregation system.

An example Loki stack can be installed in the cluster as follows:

  1. Create loki.yml:

    ---
loki:
  commonConfig:
    replication_factor: 1
  schemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: index_
          period: 24h
  storage:
    type: 'filesystem'

deploymentMode: Monolithic

singleBinary:
  replicas: 1
read:
  replicas: 0
write:
  replicas: 0
backend:
  replicas: 0

  2. Install Loki: helm upgrade -i --create-namespace -n loki loki oci://ghcr.io/grafana-community/helm-charts/loki -f loki.yml

Loki is now available for logs to be shipped to and logs to be queried from

Alloy for shipping logs

Alloy is used to collect logs from the Kubernetes Pods and ship them to Loki.

An example setup of Alloy can be installed in the cluster as follows:

  1. Create alloy.yml

    alloy:
  configMap:
    content: |-
      logging {
        level = "info"
        format = "logfmt"
      }

      loki.write "default" {
        endpoint {
          url = "http://loki-gateway.loki.svc.cluster.local./loki/api/v1/push"
          tenant_id = "default"
        }
      }

      discovery.kubernetes "pod" {
        role = "pod"
        selectors {
          role = "pod"
          field = "spec.nodeName=" + coalesce(sys.env("HOSTNAME"), constants.hostname)
        }
      }

      discovery.relabel "pod_logs" {
        targets = discovery.kubernetes.pod.targets

        // Label creation - "namespace" field from "__meta_kubernetes_namespace"
        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          action = "replace"
          target_label = "namespace"
        }

        // Label creation - "pod" field from "__meta_kubernetes_pod_name"
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          action = "replace"
          target_label = "pod"
        }

        // Label creation - "container" field from "__meta_kubernetes_pod_container_name"
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          action = "replace"
          target_label = "container"
        }

        // Label creation -  "app" field from "__meta_kubernetes_pod_label_app_kubernetes_io_name"
        rule {
          source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
          action = "replace"
          target_label = "app"
        }

        // Label creation -  "job" field from "__meta_kubernetes_namespace" and "__meta_kubernetes_pod_container_name"
        // Concatenate values __meta_kubernetes_namespace/__meta_kubernetes_pod_container_name
        rule {
          source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
          action = "replace"
          target_label = "job"
          separator = "/"
          replacement = "$1"
        }

        // Label creation - "__path__" field from "__meta_kubernetes_pod_uid" and "__meta_kubernetes_pod_container_name"
        // Concatenate values __meta_kubernetes_pod_uid/__meta_kubernetes_pod_container_name.log
        rule {
          source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
          action = "replace"
          target_label = "__path__"
          separator = "/"
          replacement = "/var/log/pods/*$1/*.log"
        }
      }

      // loki.source.kubernetes tails logs from Kubernetes containers using the Kubernetes API.
      loki.source.kubernetes "pod_logs" {
        targets    = discovery.relabel.pod_logs.output
        forward_to = [loki.write.default.receiver]
      }

  2. Add the Helm repository: helm repo add grafana https://grafana.github.io/helm-charts

  3. Install Alloy: helm upgrade -i --create-namespace -n alloy alloy grafana/alloy -f alloy.yml

Alloy will now start reading logs from all Pods in the cluster and shipping them to Loki.

Prometheus Operator and Grafana

The kube-prometheus-stack provides an installation of the Prometheus Operator and Grafana for the cluster.

This installs components to scrape metrics from ServiceMonitors in cluster, store them and expose them through Grafana.

An example setup can be installed in the cluster as follows:

  1. Create monitoring.yml

    ---
grafana:
  ingress:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    hosts:
    - grafana.<server name>
    tls:
    - secretName: grafana
      hosts:
      - grafana.<server name>

  datasources:
    loki.yaml:
      apiVersion: 1
      datasources:
        - name: Loki
          url: http://loki-gateway.loki.svc.cluster.local.
          type: 'loki'
          jsonData:
            httpHeaderName1: 'X-Scope-OrgID'
          secureJsonData:
            httpHeaderValue1: "default"

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    storageSpec:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 10Gi
    1. Replace grafana.<server name> with an appropriate DNS entry that points at your server's IP
    2. letsencrypt-prod is the ClusterIssuer created in the ESS Pro setup instructions but can be adjusted as appropriate

  2. Install the stack: helm upgrade -i --create-namespace -n monitoring monitoring oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack -f monitoring.yml

  3. Obtain the Grafana admin password: kubectl --namespace monitoring get secrets monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo
  4. Login to Grafana at https://grafana.<server name> using admin as the username and the password obtained above

The Grafana instance will have a set of dashboards for health of the cluster. Additional dashboards can be added, for example:

Arbitrary metrics and logs can be viewing using the "Explore" tab.