Grafana Tempo Installation

This content is not available in your language yet.

Grafana Tempo is a Tracing Aggregation for LGTM Observability Stack

Prerequisites

Kubernetes Cluster

Grafana Tempo will be installed on 🟢 Management Kubernetes Cluster

📥Ingress Service provided as Kubernetes Ingress Class (IngressClass)
🛡️TLS Certificate for Grafana Tempo provided as Kubernetes Secret
- Grafana Tempo will be exposed as HTTPS with Kubernetes Ingress.

Application Dependencies

📦S3 API-compatible Object Storage ; For Metrics Storage
- 🪣S3 Buckets: A Unit of Logical Storage with 🌏Region specified.
  - Grafana Tempo uses 1 bucket.
    - Tracing Storage (TSDB)
- 🔑Credentials to Access S3 Bucket: Access Key, Secret Key.
  - Create/Gather a Dedicated Access Key/Secret Key for Grafana Tempo to access to its buckets.

Complete Prerequisites

Kubernetes Cluster

Prepare Shell Variables

Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

Connect to 🟢 Management Kubernetes Cluster ; i.e w/ Kubeconfig File

Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

source $HOME/opstella-installation/shell-values/kubernetes/management_cluster.vars.sh
source $HOME/opstella-installation/shell-values/tools/observability.vars.sh

export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/management_cluster.yaml"

Set 🟢 Management Kubernetes Cluster Information

Ensure GRAFANA_TEMPO_DOMAIN, K8S_INTERNAL_DOMAIN, K8S_INGRESSCLASS_NAME, K8S_STORAGECLASS_NAME, K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME are defined as per the Shell Variables guide.
Create Kubernetes Secret for 🛡️ TLS Certificate for Grafana Tempo in Namespace observability-system.

Kubernetes Ingress for Grafana Tempo will associate TLS Certificate with Kubernetes Secret named wildcard-${BASE_DOMAIN}-tls.
```
export K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME="wildcard-${BASE_DOMAIN}-tls"
```
Create one using from .crt and .key file.
Terminal window
```
kubectl create secret tls $K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME \
  --cert=/path/to/cert/file --key=/path/to/key/file \
  --namespace observability-system
```
💡 Should return secret/wildcard-${BASE_DOMAIN}-tls created message.

Application Dependencies

S3 API-compatible Object Storage

Set S3 API-compatible Object Storage Information for Grafana Tempo.

Set S3 Connection with Domain

export GRAFANA_TEMPO_S3_DOMAIN="http://seaweedfs-s3.apps-supporting-services.svc:9000"

Set 🪣S3 Bucket

Grafana Tempo uses 1 bucket.
- Tracing Storage (TSDB) named grafana-tempo-tsdb
```
export GRAFANA_TEMPO_S3_TSDB_BUCKET_NAME="grafana-tempo-tsdb"
```

Set 🌏S3 Region

export GRAFANA_TEMPO_S3_BUCKET_REGION="us-east-1"

Set 🔑Credentials to Access S3 Bucket

Access Key

export GRAFANA_TEMPO_S3_ACCESS_KEY="grafana-tempo"

Secret Key

export GRAFANA_TEMPO_S3_ACCESS_SECRET="${SEAWEEDFS_HA_S3_GRAFANA_TEMPO_PASSWORD}"

Pre-Installation

Grafana Tempo Preparation

Set Grafana Tempo Entrypoint Domain

export GRAFANA_TEMPO_DOMAIN="tempo.${BASE_DOMAIN}"

Set Grafana Tempo Extra Configurations
- Specify Grafana Mimir Target Host for Metrics Generator Feature
Terminal window
```
export GRAFANA_TEMPO_METRICS_GENERATOR_TARGET="https://mimir.${BASE_DOMAIN}/api/v1/push"
```

Create Helm Values Configurations

cat <<EOF > $HOME/opstella-installation/helm-values/grafana-tempo-full-values.yaml
global:
  ## -- Definitions to set up nginx resolver (nginx gateway that proxied within microservices)
  ## OPSTELLA_CUSTOMIZE/RKE2: Defaults was 'kube-dns'/Change for RKE2
  # -- Definitions to set up nginx resolver
  # -- configures DNS service name
  dnsService: ${K8S_INTERNAL_DNS_SERVICE} # was defaults to `kube-dns`
  # -- configures DNS service namespace
  dnsNamespace: "kube-system"
  # -- configures cluster domain ("cluster.local" by default)
  clusterDomain: "${K8S_INTERNAL_DOMAIN}"

## OPSTELLA_CUSTOMIZE: Disable Built-in MinIO (it's not intended for Production uses!)
minio:
  enabled: false

## OPSTELLA_CUSTOMIZE: Enable Multi-Tenancy
multitenancyEnabled: true

## OPSTELLA_CUSTOMIZE: Disable Telemetry to Grafana Labs???
# -- If true, Tempo will report anonymous usage data about the shape of a deployment to Grafana Labs
reportingEnabled: false

## OPSTELLA_CUSTOMIZE: XXXXXX
# Configuration for the gateway
gateway:
  # -- Specifies whether the gateway should be enabled
  enabled: true
  ingress:
    enabled: true
    ingressClassName: ${K8S_INGRESSCLASS_NAME}
    hosts:
      - host: ${GRAFANA_TEMPO_DOMAIN}
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: ${K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME}
        hosts:
          - ${GRAFANA_TEMPO_DOMAIN}

## OPSTELLA_CUSTOMIZE: Tempo Storage Configuration
## Utilize S3(-compatible) Object Storage by Default
storage:
  trace:
    backend: s3
    s3:
      endpoint: ${GRAFANA_TEMPO_S3_DOMAIN}
      bucket: ${GRAFANA_TEMPO_S3_TSDB_BUCKET_NAME}
      region: ${GRAFANA_TEMPO_S3_BUCKET_REGION}
      access_key: ${GRAFANA_TEMPO_S3_ACCESS_KEY}
      secret_key: ${GRAFANA_TEMPO_S3_ACCESS_SECRET}
      forcepathstyle: true

tempo:
  ## OPSTELLA_CUSTOMIZE: SecurityContext
  # -- The SecurityContext for Tempo pods
  podSecurityContext:
    fsGroup: 10001
  # -- The SecurityContext for Tempo containers
  containerSecurityContext:
    runAsUser: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    privileged: false
    allowPrivilegeEscalation: false
    capabilities:
      drop: ["ALL"]
    seccompProfile:
      type: RuntimeDefault
    readOnlyRootFilesystem: true

EOF

Helm Chart Preparation

Add Grafana Helm Repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Installation

Install Grafana Tempo

Install a Helm Release with specific Helm Chart Version --version 1.32.5 (App Version: 2.7.1)

helm install grafana-tempo grafana/tempo-distributed --version 1.32.5 \
  --namespace observability-system \
  -f $HOME/opstella-installation/helm-values/grafana-tempo-full-values.yaml

Post-Installation

Grafana Tempo Testing

Get Pods Status

kubectl get pods -n observability-system

💡 Grafana Tempo (Distributed Deployment Mode Components) Pods should be Running

NAME                                           READY   STATUS    RESTARTS
... (deducted)
grafana-tempo-compactor-XXXXXXX-YYYYY          1/1     Running   0          Xd
grafana-tempo-distributor-XXXXXXX-YYYYY        1/1     Running   0          Xd
grafana-tempo-gateway-XXXXXXX-YYYYY            1/1     Running   0          Xd
grafana-tempo-ingester-0                       1/1     Running   0          Xd
grafana-tempo-ingester-1                       1/1     Running   0          Xd
grafana-tempo-ingester-2                       1/1     Running   0          Xd
grafana-tempo-memcached-0                      1/1     Running   0          Xd
grafana-tempo-metrics-generator-XXXXXXX-YYYYY  1/1     Running   0          Xd
grafana-tempo-querier-XXXXXXX-YYYYY            1/1     Running   0          Xd
grafana-tempo-query-frontend-XXXXXXX-YYYYY     1/1     Running   0          Xd

Finished?

Use the below navigation to proceed

Previous
Grafana Loki Next
Health Checks for Opstella