Grafana Mimir Installation

Grafana Mimir is a Metrics Aggregation for LGTM Observability Stack

Prerequisites

Kubernetes Cluster

Grafana Mimir will be installed on 🟢 Management Kubernetes Cluster

📥Ingress Service provided as Kubernetes Ingress Class (IngressClass)
🛡️TLS Certificate for Grafana Mimir provided as Kubernetes Secret
- Grafana Mimir will be exposed as HTTPS with Kubernetes Ingress.

Application Dependencies

📦S3 API-compatible Object Storage ; For Metrics Storage
- 🪣S3 Buckets: A Unit of Logical Storage with 🌏Region specified.
  - Grafana Mimir uses 3 separated buckets.
    - Metrics Storage (TSDB)
    - Alert Manager Component
    - Ruler Component
- 🔑Credentials to Access S3 Bucket: Access Key, Secret Key.
  - Create/Gather a Dedicated Access Key/Secret Key for Grafana Mimir to access to its buckets.

Complete Prerequisites

Kubernetes Cluster

Prepare Shell Variables

Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

Connect to 🟢 Management Kubernetes Cluster ; i.e w/ Kubeconfig File

Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

source $HOME/opstella-installation/shell-values/kubernetes/management_cluster.vars.sh
source $HOME/opstella-installation/shell-values/tools/observability.vars.sh

export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/management_cluster.yaml"

Set 🟢 Management Kubernetes Cluster Information

Ensure GRAFANA_MIMIR_DOMAIN, K8S_INTERNAL_DOMAIN, K8S_INGRESSCLASS_NAME, K8S_STORAGECLASS_NAME, K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME are defined as per the Shell Variables guide.
Create Kubernetes Secret for 🛡️ TLS Certificate for Grafana Mimir in Namespace observability-system.

Kubernetes Ingress for Grafana Mimir will associate TLS Certificate with Kubernetes Secret named wildcard-${BASE_DOMAIN}-tls.
```
export K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME="wildcard-${BASE_DOMAIN}-tls"
```
Create one using from .crt and .key file.
Terminal window
```
kubectl create secret tls $K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME \
  --cert=/path/to/cert/file --key=/path/to/key/file \
  --namespace observability-system
```
💡 Should return secret/wildcard-${BASE_DOMAIN}-tls created message.

Application Dependencies

S3 API-compatible Object Storage

Set S3 API-compatible Object Storage Information for Grafana Mimir.

Set S3 Connection with Domain

export GRAFANA_MIMIR_S3_DOMAIN="http://seaweedfs-s3.apps-supporting-services.svc:9000"

Set 🪣S3 Buckets

Grafana Mimir uses 3 separated buckets.

Metrics Storage (TSDB) named grafana-mimir-tsdb

export GRAFANA_MIMIR_S3_BLOCKS_STORAGE_BUCKET_NAME="grafana-mimir-tsdb"

Alert Manager Component named grafana-mimir-alertmanager

export GRAFANA_MIMIR_S3_ALERTMANAGER_STORAGE_BUCKET_NAME="grafana-mimir-alertmanager"

Ruler Component named grafana-mimir-ruler

export GRAFANA_MIMIR_S3_RULER_STORAGE_BUCKET_NAME="grafana-mimir-ruler"

Set 🌏S3 Region

export GRAFANA_MIMIR_S3_BUCKET_REGION="us-east-1"

Set 🔑Credentials to Access S3 Bucket

Access Key

export GRAFANA_MIMIR_S3_ACCESS_KEY="grafana-mimir"

Secret Key

export GRAFANA_MIMIR_S3_ACCESS_SECRET="${SEAWEEDFS_HA_S3_GRAFANA_MIMIR_PASSWORD}"

Pre-Installation

Grafana Mimir Preparation

Set Grafana Mimir Entrypoint Domain

export GRAFANA_MIMIR_DOMAIN="mimir.${BASE_DOMAIN}"

Create Helm Values Configuration

cat <<EOF > $HOME/opstella-installation/helm-values/grafana-mimir-full-values.yaml
## OPSTELLA_CUSTOMIZE: Global Variables
global:
  ## -- Definitions to set up nginx resolver (nginx gateway that proxied within microservices)
  ## OPSTELLA_CUSTOMIZE/RKE2: Defaults was 'kube-dns'/Change for RKE2
  # -- Definitions to set up nginx resolver
  # -- configures DNS service name
  dnsService: ${K8S_INTERNAL_DNS_SERVICE}
  clusterDomain: "${K8S_INTERNAL_DOMAIN}."

## OPSTELLA_CUSTOMIZE: Disable Built-in MinIO (it's not intended for Production uses!)
minio:
  enabled: false

# -- A reverse proxy deployment that is meant to receive traffic for Mimir or GEM.
# When enterprise.enabled is true the GEM gateway is deployed. Otherwise, it is an nginx.
# Options except those under gateway.nginx apply to both versions - nginx and GEM gateway.
gateway:
  # -- Whether to render the manifests related to the gateway component.
  # This controls the behaviour both when enterprise.enabled=true and when enterprise.enabled=false.
  ## OPSTELLA_CUSTOMIZE: Utilise Ingress, Disable NGINX Configurations
  enabled: false
## OPSTELLA_CUSTOMIZE: Utilise Ingress, Disable NGINX Deployment
nginx:
  enabled: false

## OPSTELLA_CUSTOMIZE: Expose to K8s Ingress
## -- Use either this ingress or the gateway, but not both at once.
## If you enable this, make sure to disable the gateway's ingress.
ingress:
  enabled: true
  ingressClassName: ${K8S_INGRESSCLASS_NAME}
  hosts:
    - ${GRAFANA_MIMIR_DOMAIN}
  tls:
    - hosts:
        - ${GRAFANA_MIMIR_DOMAIN}
      secretName: ${K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME}

## OPSTELLA_CUSTOMIZE: Configure Mimir
mimir:
  structuredConfig:
    ## Utilize Object Storage : S3(-compatible)
    common:
      storage:
        backend: s3
        s3:
          bucket_lookup_type: auto ## Bucket lookup style type, used to access bucket in S3-compatible service. Default is auto. Supported values are: auto, path, virtual-hosted.
          endpoint: ${GRAFANA_MIMIR_S3_DOMAIN}
          region: ${GRAFANA_MIMIR_S3_BUCKET_REGION}
          access_key_id: ${GRAFANA_MIMIR_S3_ACCESS_KEY}
          secret_access_key: ${GRAFANA_MIMIR_S3_ACCESS_SECRET}
    blocks_storage:
      s3:
        bucket_name: ${GRAFANA_MIMIR_S3_BLOCKS_STORAGE_BUCKET_NAME}
    alertmanager_storage:
      s3:
        bucket_name: ${GRAFANA_MIMIR_S3_ALERTMANAGER_STORAGE_BUCKET_NAME}
    ruler_storage:
      s3:
        bucket_name: ${GRAFANA_MIMIR_S3_RULER_STORAGE_BUCKET_NAME}

alertmanager:
  ## TODO: OPSTELLA_CUSTOMIZE/TEMP: Disable Persistence until we can measure the workload
  ## OPSTELLA_NOTE: Alertmanager persist the alerts for long-term
  persistentVolume:
    ## It is advisable to enable volume persistence in ingester to avoid losing metrics
    enabled: false ## true: DEFAULT FROM CHART ##

ingester:
  ## OPSTELLA_CUSTOMIZE: Disable Zone-Aware Replication for Non-Cloud Environment
  ## -- Zone-aware replication for ingester
  zoneAwareReplication:
    enabled: false
  ## TODO: OPSTELLA_CUSTOMIZE/TEMP: Disable Persistence until we can measure the workload
  persistentVolume:
    ## CHART_IMPORTANT_NOTE: It is advisable to enable volume persistence in ingester to avoid losing metrics
    enabled: false ## true: DEFAULT FROM CHART ##

compactor:
  ## TODO: OPSTELLA_CUSTOMIZE/TEMP: Disable Persistence until we can measure the workload
  ## OPSTELLA_NOTE: Compactor requires a TSDB Block Caching
  persistentVolume:
    enabled: false ## true: DEFAULT FROM CHART ##

store_gateway:
  ## OPSTELLA_CUSTOMIZE: Disable Zone-Aware Replication for Non-Cloud Environment
  ## -- Zone-aware replication for store-gateway
  zoneAwareReplication:
    enabled: true
  ## TODO: OPSTELLA_CUSTOMIZE/TEMP: Disable Persistence until we can measure the workload
  ## OPSTELLA_NOTE: Store-Gateway component requires fetching a portion of TSDB then compact data.
  persistentVolume:
    enabled: false ## true: DEFAULT FROM CHART ##
EOF

Helm Chart Preparation

Add Grafana Helm Repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Installation

Install Grafana Mimir

Install a Helm Release with specific Helm Chart Version --version 5.6.0 (App Version: 2.15.0)

helm install grafana-mimir grafana/mimir-distributed --version 5.6.0 \
  --namespace observability-system \
  -f $HOME/opstella-installation/helm-values/grafana-mimir-full-values.yaml

Post-Installation

Grafana Mimir Testing

Get Pods Status

kubectl get pods -n observability-system

💡 Grafana Mimir (Distributed Deployment Mode Components) Pods should be Running

NAME                                             READY   STATUS    RESTARTS  AGE
... (deducted)
grafana-mimir-alertmanager-0                     1/1     Running   0         Xd
grafana-mimir-chunks-cache-0                     1/1     Running   0         Xd
grafana-mimir-compactor-0                        1/1     Running   0         Xd
grafana-mimir-distributor-XXXXXXX-YYYYY          1/1     Running   0         Xd
grafana-mimir-index-cache-0                      1/1     Running   0         Xd
grafana-mimir-ingester-0                         1/1     Running   0         Xd
grafana-mimir-ingester-0                         1/1     Running   0         Xd
grafana-mimir-ingester-0                         1/1     Running   0         Xd
grafana-mimir-metadata-cache-0                   1/1     Running   0         Xd
grafana-mimir-nginx-XXXXXXX-YYYYY                1/1     Running   0         Xd
grafana-mimir-overrides-exporter-XXXXXXX-YYYYY   1/1     Running   0         Xd
grafana-mimir-querier-XXXXXXX-YYYYY              1/1     Running   0         Xd
grafana-mimir-querier-XXXXXXX-YYYYY              1/1     Running   0         Xd
grafana-mimir-query-frontend-XXXXXXX-YYYYY       1/1     Running   0         Xd
grafana-mimir-query-scheduler-XXXXXXX-YYYYY      1/1     Running   0         Xd
grafana-mimir-query-scheduler-XXXXXXX-YYYYY      1/1     Running   0         Xd
grafana-mimir-results-cache-0                    1/1     Running   0         Xd
grafana-mimir-ruler-XXXXXXX-YYYYY                1/1     Running   0         Xd
grafana-mimir-store-gateway-0                    1/1     Running   0         Xd

Finished?

Use the below navigation to proceed

Previous
Grafana Dashboard Next
Grafana Loki