Skip to content

This is the Experimental version (Latest). It is under active development and may change. For the most reliable documentation, use the version selector in the top-right to switch to Stable, or click here to go to the Stable version's homepage.

Grafana Mimir Installation

Last updated:

Grafana Mimir is a Metrics Aggregation for LGTM Observability Stack


Grafana Mimir will be installed on 🟢 Management Kubernetes Cluster

  • 📥Ingress Service provided as Kubernetes Ingress Class (IngressClass)
  • 🛡️TLS Certificate for Grafana Mimir provided as Kubernetes Secret
    • Grafana Mimir will be exposed as HTTPS with Kubernetes Ingress.
  • 📦S3 API-compatible Object Storage ; For Metrics Storage
    • 🪣S3 Buckets: A Unit of Logical Storage with 🌏Region specified.
      • Grafana Mimir uses 3 separated buckets.
        • Metrics Storage (TSDB)
        • Alert Manager Component
        • Ruler Component
    • 🔑Credentials to Access S3 Bucket: Access Key, Secret Key.
      • Create/Gather a Dedicated Access Key/Secret Key for Grafana Mimir to access to its buckets.

Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

  1. Connect to 🟢 Management Kubernetes Cluster ; i.e w/ Kubeconfig File

    Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

    Terminal window
    source $HOME/opstella-installation/shell-values/kubernetes/management_cluster.vars.sh
  2. Set 🟢 Management Kubernetes Cluster Information

    Ensure GRAFANA_MIMIR_DOMAIN, K8S_INTERNAL_DOMAIN, K8S_INGRESSCLASS_NAME, K8S_STORAGECLASS_NAME, K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME are defined as per the Shell Variables guide.

  3. Create Kubernetes Secret for 🛡️ TLS Certificate for Grafana Mimir in Namespace observability-system.

    Kubernetes Ingress for Grafana Mimir will associate TLS Certificate with Kubernetes Secret named wildcard-${BASE_DOMAIN}-tls.

    (Ensure K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME is defined in your variables)

    Create one using from .crt and .key file.

    Terminal window
    kubectl create secret tls $K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME \
    --cert=/path/to/cert/file --key=/path/to/key/file \
    --namespace observability-system

    💡 Should return secret/wildcard-${BASE_DOMAIN}-tls created message.

Set S3 API-compatible Object Storage Information for Grafana Mimir.

  1. Set S3 Connection with Domain

    (Ensure your S3 domain, multiple bucket names, regions, and credentials are all loaded from your tool variables in observability.vars.sh)

  1. Set Grafana Mimir Entrypoint Domain

    (Ensure Grafana Mimir Entrypoint Domain is loaded from your variables)

  2. Create Helm Values Configuration

    Terminal window
    cat <<EOF > $BASE_WORKING_DIR/helm-values/grafana-mimir-full-values.yaml
    ## OPSTELLA_CUSTOMIZE: Global Variables
    global:
    ## -- Definitions to set up nginx resolver (nginx gateway that proxied within microservices)
    ## OPSTELLA_CUSTOMIZE/RKE2: Defaults was 'kube-dns'/Change for RKE2
    # -- Definitions to set up nginx resolver
    # -- configures DNS service name
    dnsService: ${K8S_INTERNAL_DNS_SERVICE}
    clusterDomain: "${K8S_INTERNAL_DOMAIN}."
    ## OPSTELLA_CUSTOMIZE: Disable Built-in MinIO (it's not intended for Production uses!)
    minio:
    enabled: false
    # -- A reverse proxy deployment that is meant to receive traffic for Mimir or GEM.
    # When enterprise.enabled is true the GEM gateway is deployed. Otherwise, it is an nginx.
    # Options except those under gateway.nginx apply to both versions - nginx and GEM gateway.
    gateway:
    # -- Whether to render the manifests related to the gateway component.
    # This controls the behaviour both when enterprise.enabled=true and when enterprise.enabled=false.
    ## OPSTELLA_CUSTOMIZE: Utilise Ingress, Disable NGINX Configurations
    enabled: false
    ## OPSTELLA_CUSTOMIZE: Utilise Ingress, Disable NGINX Deployment
    nginx:
    enabled: false
    ## OPSTELLA_CUSTOMIZE: Expose to K8s Ingress
    ## -- Use either this ingress or the gateway, but not both at once.
    ## If you enable this, make sure to disable the gateway's ingress.
    ingress:
    enabled: true
    ingressClassName: ${K8S_INGRESSCLASS_NAME}
    hosts:
    - ${GRAFANA_MIMIR_DOMAIN}
    tls:
    - hosts:
    - ${GRAFANA_MIMIR_DOMAIN}
    secretName: ${K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME}
    ## OPSTELLA_CUSTOMIZE: Configure Mimir
    mimir:
    structuredConfig:
    ## Utilize Object Storage : S3(-compatible)
    common:
    storage:
    backend: s3
    s3:
    bucket_lookup_type: auto ## Bucket lookup style type, used to access bucket in S3-compatible service. Default is auto. Supported values are: auto, path, virtual-hosted.
    endpoint: ${GRAFANA_MIMIR_S3_DOMAIN}
    region: ${GRAFANA_MIMIR_S3_BUCKET_REGION}
    access_key_id: ${GRAFANA_MIMIR_S3_ACCESS_KEY}
    secret_access_key: ${GRAFANA_MIMIR_S3_ACCESS_SECRET}
    blocks_storage:
    s3:
    bucket_name: ${GRAFANA_MIMIR_S3_BLOCKS_STORAGE_BUCKET_NAME}
    alertmanager_storage:
    s3:
    bucket_name: ${GRAFANA_MIMIR_S3_ALERTMANAGER_STORAGE_BUCKET_NAME}
    ruler_storage:
    s3:
    bucket_name: ${GRAFANA_MIMIR_S3_RULER_STORAGE_BUCKET_NAME}
    alertmanager:
    ## TODO: OPSTELLA_CUSTOMIZE/TEMP: Disable Persistence until we can measure the workload
    ## OPSTELLA_NOTE: Alertmanager persist the alerts for long-term
    persistentVolume:
    ## It is advisable to enable volume persistence in ingester to avoid losing metrics
    enabled: false ## true: DEFAULT FROM CHART ##
    ingester:
    ## OPSTELLA_CUSTOMIZE: Disable Zone-Aware Replication for Non-Cloud Environment
    ## -- Zone-aware replication for ingester
    zoneAwareReplication:
    enabled: false
    ## TODO: OPSTELLA_CUSTOMIZE/TEMP: Disable Persistence until we can measure the workload
    persistentVolume:
    ## CHART_IMPORTANT_NOTE: It is advisable to enable volume persistence in ingester to avoid losing metrics
    enabled: false ## true: DEFAULT FROM CHART ##
    compactor:
    ## TODO: OPSTELLA_CUSTOMIZE/TEMP: Disable Persistence until we can measure the workload
    ## OPSTELLA_NOTE: Compactor requires a TSDB Block Caching
    persistentVolume:
    enabled: false ## true: DEFAULT FROM CHART ##
    store_gateway:
    ## OPSTELLA_CUSTOMIZE: Disable Zone-Aware Replication for Non-Cloud Environment
    ## -- Zone-aware replication for store-gateway
    zoneAwareReplication:
    enabled: true
    ## TODO: OPSTELLA_CUSTOMIZE/TEMP: Disable Persistence until we can measure the workload
    ## OPSTELLA_NOTE: Store-Gateway component requires fetching a portion of TSDB then compact data.
    persistentVolume:
    enabled: false ## true: DEFAULT FROM CHART ##
    EOF
  1. Add Grafana Helm Repository

    Terminal window
    helm repo add grafana https://grafana.github.io/helm-charts
    helm repo update
  1. Install Grafana Mimir

    • Install a Helm Release with specific Helm Chart Version --version 5.6.0 (App Version: 2.15.0)

      Terminal window
      helm upgrade --install grafana-mimir grafana/mimir-distributed --version 5.6.0 \
      --namespace observability-system \
      -f $HOME/opstella-installation/helm-values/grafana-mimir-full-values.yaml
  1. Get Pods Status

    Terminal window
    kubectl get pods -n observability-system

    💡 Grafana Mimir (Distributed Deployment Mode Components) Pods should be Running

    NAME READY STATUS RESTARTS AGE
    ... (deducted)
    grafana-mimir-alertmanager-0 1/1 Running 0 Xd
    grafana-mimir-chunks-cache-0 1/1 Running 0 Xd
    grafana-mimir-compactor-0 1/1 Running 0 Xd
    grafana-mimir-distributor-XXXXXXX-YYYYY 1/1 Running 0 Xd
    grafana-mimir-index-cache-0 1/1 Running 0 Xd
    grafana-mimir-ingester-0 1/1 Running 0 Xd
    grafana-mimir-ingester-0 1/1 Running 0 Xd
    grafana-mimir-ingester-0 1/1 Running 0 Xd
    grafana-mimir-metadata-cache-0 1/1 Running 0 Xd
    grafana-mimir-nginx-XXXXXXX-YYYYY 1/1 Running 0 Xd
    grafana-mimir-overrides-exporter-XXXXXXX-YYYYY 1/1 Running 0 Xd
    grafana-mimir-querier-XXXXXXX-YYYYY 1/1 Running 0 Xd
    grafana-mimir-querier-XXXXXXX-YYYYY 1/1 Running 0 Xd
    grafana-mimir-query-frontend-XXXXXXX-YYYYY 1/1 Running 0 Xd
    grafana-mimir-query-scheduler-XXXXXXX-YYYYY 1/1 Running 0 Xd
    grafana-mimir-query-scheduler-XXXXXXX-YYYYY 1/1 Running 0 Xd
    grafana-mimir-results-cache-0 1/1 Running 0 Xd
    grafana-mimir-ruler-XXXXXXX-YYYYY 1/1 Running 0 Xd
    grafana-mimir-store-gateway-0 1/1 Running 0 Xd

Finished?

Use the below navigation to proceed