Grafana Tempo Installation

Grafana Tempo is a Tracing Aggregation for LGTM Observability Stack

---

Prerequisites

Kubernetes Cluster

Grafana Tempo will be installed on 🟢 Management Kubernetes Cluster

• 📥Ingress Service provided as Kubernetes Ingress Class (IngressClass)
• 🛡️TLS Certificate for Grafana Tempo provided as Kubernetes Secret
  • Grafana Tempo will be exposed as HTTPS with Kubernetes Ingress.

Application Dependencies

• 📦S3 API-compatible Object Storage ; For Metrics Storage
  • 🪣S3 Buckets: A Unit of Logical Storage with 🌏Region specified.
    • Grafana Tempo uses 1 bucket.
      • Tracing Storage (TSDB)
  • 🔑Credentials to Access S3 Bucket: Access Key, Secret Key.
    • Create/Gather a Dedicated Access Key/Secret Key for Grafana Tempo to access to its buckets.

Complete Prerequisites

Kubernetes Cluster

Prepare Shell Variables

Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

1. Connect to 🟢 Management Kubernetes Cluster ; i.e w/ Kubeconfig File

   Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

   source $HOME/opstella-installation/shell-values/kubernetes/management_cluster.vars.sh

2. Set 🟢 Management Kubernetes Cluster Information

   Ensure GRAFANA_TEMPO_DOMAIN, K8S_INTERNAL_DOMAIN, K8S_INGRESSCLASS_NAME, K8S_STORAGECLASS_NAME, K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME are defined as per the Shell Variables guide.

3. Create Kubernetes Secret for 🛡️ TLS Certificate for Grafana Tempo in Namespace observability-system.

   Kubernetes Ingress for Grafana Tempo will associate TLS Certificate with Kubernetes Secret named wildcard-${BASE_DOMAIN}-tls.

   (Ensure K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME is defined in your variables)

   Create one using from .crt and .key file.

   kubectl create secret tls $K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME \
     --cert=/path/to/cert/file --key=/path/to/key/file \
     --namespace observability-system

   💡 Should return secret/wildcard-${BASE_DOMAIN}-tls created message.

Application Dependencies

S3 API-compatible Object Storage

Set S3 API-compatible Object Storage Information for Grafana Tempo.

1. Set S3 Connection with Domain

   (Ensure your S3 domain, bucket names, regions, and credentials are all loaded from your tool variables in observability.vars.sh)

Pre-Installation

Grafana Tempo Preparation

1. Set Grafana Tempo Entrypoint Domain

   (Ensure Grafana Tempo Domain and Mimir Target are loaded from your variables)

2. Create Helm Values Configurations

   cat <<EOF > $BASE_WORKING_DIR/helm-values/grafana-tempo-full-values.yaml
   global:
     ## -- Definitions to set up nginx resolver (nginx gateway that proxied within microservices)
     ## OPSTELLA_CUSTOMIZE/RKE2: Defaults was 'kube-dns'/Change for RKE2
     # -- Definitions to set up nginx resolver
     # -- configures DNS service name
     dnsService: ${K8S_INTERNAL_DNS_SERVICE} # was defaults to `kube-dns`
     # -- configures DNS service namespace
     dnsNamespace: "kube-system"
     # -- configures cluster domain ("cluster.local" by default)
     clusterDomain: "${K8S_INTERNAL_DOMAIN}"
   
   ## OPSTELLA_CUSTOMIZE: Disable Built-in MinIO (it's not intended for Production uses!)
   minio:
     enabled: false
   
   ## OPSTELLA_CUSTOMIZE: Enable Multi-Tenancy
   multitenancyEnabled: true
   
   ## OPSTELLA_CUSTOMIZE: Disable Telemetry to Grafana Labs???
   # -- If true, Tempo will report anonymous usage data about the shape of a deployment to Grafana Labs
   reportingEnabled: false
   
   ## OPSTELLA_CUSTOMIZE: XXXXXX
   # Configuration for the gateway
   gateway:
     # -- Specifies whether the gateway should be enabled
     enabled: true
     ingress:
       enabled: true
       ingressClassName: ${K8S_INGRESSCLASS_NAME}
       hosts:
         - host: ${GRAFANA_TEMPO_DOMAIN}
           paths:
             - path: /
               pathType: Prefix
       tls:
         - secretName: ${K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME}
           hosts:
             - ${GRAFANA_TEMPO_DOMAIN}
   
   ## OPSTELLA_CUSTOMIZE: Tempo Storage Configuration
   ## Utilize S3(-compatible) Object Storage by Default
   storage:
     trace:
       backend: s3
       s3:
         endpoint: ${GRAFANA_TEMPO_S3_DOMAIN}
         bucket: ${GRAFANA_TEMPO_S3_TSDB_BUCKET_NAME}
         region: ${GRAFANA_TEMPO_S3_BUCKET_REGION}
         access_key: ${GRAFANA_TEMPO_S3_ACCESS_KEY}
         secret_key: ${GRAFANA_TEMPO_S3_ACCESS_SECRET}
         forcepathstyle: true
   
   tempo:
     ## OPSTELLA_CUSTOMIZE: SecurityContext
     # -- The SecurityContext for Tempo pods
     podSecurityContext:
       fsGroup: 10001
     # -- The SecurityContext for Tempo containers
     containerSecurityContext:
       runAsUser: 10001
       runAsGroup: 10001
       runAsNonRoot: true
       privileged: false
       allowPrivilegeEscalation: false
       capabilities:
         drop: ["ALL"]
       seccompProfile:
         type: RuntimeDefault
       readOnlyRootFilesystem: true
   
   EOF

Helm Chart Preparation

1. Add Grafana Helm Repository

   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update

Installation

1. Install Grafana Tempo

   • Install a Helm Release with specific Helm Chart Version --version 1.32.5 (App Version: 2.7.1)

   helm upgrade --install grafana-tempo grafana/tempo-distributed --version 1.32.5 \
     --namespace observability-system \
     -f $HOME/opstella-installation/helm-values/grafana-tempo-full-values.yaml

Post-Installation

Grafana Tempo Testing

1. Get Pods Status

   kubectl get pods -n observability-system

   💡 Grafana Tempo (Distributed Deployment Mode Components) Pods should be Running

   NAME                                           READY   STATUS    RESTARTS
   ... (deducted)
   grafana-tempo-compactor-XXXXXXX-YYYYY          1/1     Running   0          Xd
   grafana-tempo-distributor-XXXXXXX-YYYYY        1/1     Running   0          Xd
   grafana-tempo-gateway-XXXXXXX-YYYYY            1/1     Running   0          Xd
   grafana-tempo-ingester-0                       1/1     Running   0          Xd
   grafana-tempo-ingester-1                       1/1     Running   0          Xd
   grafana-tempo-ingester-2                       1/1     Running   0          Xd
   grafana-tempo-memcached-0                      1/1     Running   0          Xd
   grafana-tempo-metrics-generator-XXXXXXX-YYYYY  1/1     Running   0          Xd
   grafana-tempo-querier-XXXXXXX-YYYYY            1/1     Running   0          Xd
   grafana-tempo-query-frontend-XXXXXXX-YYYYY     1/1     Running   0          Xd