Kubernetes15 June 2023Updated 12 June 2026

Grafana with Prometheus for Traefik in minikube

If services run behind a Traefik ingress controller, the metrics are already there — Traefik counts every request, status code and response time. This guide wires those metrics into Prometheus and visualizes them in Grafana, all inside a local minikube cluster. Every manifest you need is below.

Exposing Traefik metrics

Prometheus scrapes metrics over HTTP, so the first step is making Traefik's metrics endpoint reachable inside the cluster. A small Service in the traefik namespace exposes the metrics port; after deploying it, the endpoint is available via Kubernetes DNS at traefik-metrics.traefik.svc.cluster.local:9100.

apiVersion: v1
kind: Service
metadata:
  name: traefik-metrics
  namespace: traefik
spec:
  ports:
    - name: metrics
      protocol: TCP
      port: 9100
      targetPort: metrics
  selector:
    app.kubernetes.io/instance: traefik-traefik
    app.kubernetes.io/name: traefik
  type: ClusterIP

The selector labels must match your Traefik installation — check yours with kubectl get pods -n traefik --show-labels if the service finds no endpoints.

Prometheus: collecting the metrics

Prometheus stores metrics as time series and exposes a query language (PromQL) that Grafana uses for dashboards. We deploy it into its own namespace:

kubectl create ns prometheus

Scrape configuration

A ConfigMap holds the Prometheus configuration. One scrape job pointed at the metrics service from the previous step is all it takes:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: prometheus
data:
  prometheus.yml: |
    global:
      scrape_interval: 10s
    scrape_configs:
    - job_name: 'traefik'
      static_configs:
      - targets: ['traefik-metrics.traefik.svc.cluster.local:9100']

Persistent storage

Prometheus writes its time series to /prometheus. Without a volume, every pod restart wipes your history — so we claim persistent storage first. 10Gi is comfortable for a local cluster; adjust to taste.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-storage-persistence
  namespace: prometheus
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Deployment

The deployment mounts the configuration into /etc/prometheus, the volume into /prometheus, and exposes port 9090. Pin the image to a specific version (and bump it deliberately) instead of relying on latest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: prometheus
spec:
  selector:
    matchLabels:
      app: prometheus
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:v2.44.0
        ports:
        - containerPort: 9090
          name: default
        volumeMounts:
        - name: prometheus-storage
          mountPath: /prometheus
        - name: config-volume
          mountPath: /etc/prometheus
      volumes:
        - name: prometheus-storage
          persistentVolumeClaim:
            claimName: prometheus-storage-persistence
        - name: config-volume
          configMap:
            name: prometheus-config

Exposing Prometheus

Grafana lives in the same cluster, so a NodePort service is enough — inside the cluster Prometheus answers at http://prometheus.prometheus.svc.cluster.local:9090.

kind: Service
apiVersion: v1
metadata:
  name: prometheus
  namespace: prometheus
spec:
  selector:
    app: prometheus
  type: NodePort
  ports:
  - protocol: TCP
    port: 9090
    targetPort: 9090
    nodePort: 30909

Optionally, expose the Prometheus UI outside the cluster with a Traefik IngressRoute — handy for ad-hoc PromQL queries:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: prometheus
  namespace: prometheus
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`prometheus.test`)
      services:
        - kind: Service
          name: prometheus
          port: 9090

Grafana: visualizing the metrics

Grafana turns the Prometheus data into dashboards, charts and alerts. Same drill — its own namespace first:

kubectl create ns grafana

Provisioning the datasource

Instead of clicking the datasource together in the UI, provision it declaratively with a ConfigMap — the connection survives reinstalls and code review:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: grafana
data:
  prometheus.yaml: |-
    {
        "apiVersion": 1,
        "datasources": [
            {
               "access":"proxy",
                "editable": true,
                "name": "prometheus",
                "orgId": 1,
                "type": "prometheus",
                "url": "http://prometheus.prometheus.svc.cluster.local:9090",
                "version": 1
            }
        ]
    }

Persistence and deployment

Grafana keeps users, dashboards and settings in /var/lib/grafana — persist it so a restart doesn't reset your work:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-storage-persistence
  namespace: grafana
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

The deployment mounts both the datasource provisioning and the storage volume:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      securityContext:
        runAsUser: 472
        runAsGroup: 472
        fsGroup: 472
      containers:
      - name: grafana
        image: grafana/grafana:10.0.0
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "1Gi"
            cpu: "1000m"
          requests:
            memory: 500M
            cpu: "500m"
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
          - mountPath: /etc/grafana/provisioning/datasources
            name: grafana-datasources
            readOnly: false
      volumes:
        - name: grafana-storage
          persistentVolumeClaim:
            claimName: grafana-storage-persistence
        - name: grafana-datasources
          configMap:
              defaultMode: 420
              name: grafana-datasources

Exposing Grafana

A NodePort service plus a Traefik IngressRoute makes the Grafana UI reachable from your browser:

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: grafana
spec:
  selector:
    app: grafana
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 32000

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: grafana
  namespace: grafana
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`grafana.test`)
      services:
        - kind: Service
          name: grafana
          port: 3000

Testing the setup

Run minikube tunnel in a terminal to route traffic into the cluster, then point the test hostnames at localhost in /etc/hosts (sudo nano /etc/hosts):

127.0.0.1 grafana.test
127.0.0.1 prometheus.test

Open https://prometheus.test — the Targets page should show the traefik job as UP.
Open https://grafana.test and log in with the default admin/admin account (change it immediately).
Go to Dashboards > Import and enter dashboard ID 4475 — the official Traefik dashboard from grafana.com.

Within a few scrape intervals the dashboard fills with request rates, status codes and response times for everything running behind your Traefik ingress.

Conclusion

With four manifests for Prometheus and four for Grafana you get a complete, persistent monitoring stack for Traefik in minikube — the same building blocks scale up to a production cluster with an operator or Helm chart on top.

Setting up observability for a real production cluster — metrics, alerting, dashboards, on-call — is part of our DevOps and cloud services. If your team runs Kubernetes without visibility, we can fix that.