This is the Stable version of the documentation. The latest version is experimental and under active development. You can use the version selector in the top-right menu to switch versions for this specific page, or click here to go to the latest version's homepage.

Grafana Tempo Installation

Grafana Tempo is a Tracing Aggregation for LGTM Observability Stack

Prerequisites

Kubernetes Cluster

Grafana Tempo will be installed on 🟢 Management Kubernetes Cluster

📥Ingress Service provided as Kubernetes Ingress Class (IngressClass)
🛡️TLS Certificate for Grafana Tempo provided as Kubernetes Secret
- Grafana Tempo will be exposed as HTTPS with Kubernetes Ingress.

Application Dependencies

📦S3 API-compatible Object Storage ; For Metrics Storage
- 🪣S3 Buckets: A Unit of Logical Storage with 🌏Region specified.
  - Grafana Tempo uses 1 bucket.
    - Tracing Storage (TSDB)
- 🔑Credentials to Access S3 Bucket: Access Key, Secret Key.
  - Create/Gather a Dedicated Access Key/Secret Key for Grafana Tempo to access to its buckets.

Complete Prerequisites

Kubernetes Cluster

Prepare Shell Variables

Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

Connect to 🟢 Management Kubernetes Cluster ; i.e w/ Kubeconfig File

Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

source $HOME/opstella-installation/shell-values/kubernetes/management_cluster.vars.sh
source $HOME/opstella-installation/shell-values/tools/observability.vars.sh

export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/management_cluster.yaml"

Set 🟢 Management Kubernetes Cluster Information

Ensure GRAFANA_TEMPO_DOMAIN, K8S_INTERNAL_DOMAIN, K8S_INGRESSCLASS_NAME, K8S_STORAGECLASS_NAME, K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME are defined as per the Shell Variables guide.
Create Kubernetes Secret for 🛡️ TLS Certificate for Grafana Tempo in Namespace observability-system.

Kubernetes Ingress for Grafana Tempo will associate TLS Certificate with Kubernetes Secret named wildcard-${BASE_DOMAIN}-tls.
```
export K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME="wildcard-${BASE_DOMAIN}-tls"
```
Create one using from .crt and .key file.
Terminal window
```
kubectl create secret tls $K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME \
  --cert=/path/to/cert/file --key=/path/to/key/file \
  --namespace observability-system
```
💡 Should return secret/wildcard-${BASE_DOMAIN}-tls created message.

Application Dependencies

S3 API-compatible Object Storage

Set S3 API-compatible Object Storage Information for Grafana Tempo.

Set S3 Connection with Domain

Tempo passes this value straight to the MinIO client, which rejects a URL scheme with Endpoint url cannot have fully qualified paths. — use a bare host:port.
```
export GRAFANA_TEMPO_S3_DOMAIN="seaweedfs-s3.apps-supporting-services.svc:9000"
```
When the S3 endpoint speaks plain HTTP, as an in-cluster SeaweedFS does, also set:
```
export GRAFANA_TEMPO_S3_INSECURE="true"
```
Set 🪣S3 Bucket

Grafana Tempo uses 1 bucket.
- Tracing Storage (TSDB) named grafana-tempo-tsdb
```
export GRAFANA_TEMPO_S3_TSDB_BUCKET_NAME="grafana-tempo-tsdb"
```

Set 🌏S3 Region

export GRAFANA_TEMPO_S3_BUCKET_REGION="us-east-1"

Set 🔑Credentials to Access S3 Bucket

Access Key

export GRAFANA_TEMPO_S3_ACCESS_KEY="grafana-tempo"

Secret Key

export GRAFANA_TEMPO_S3_ACCESS_SECRET="${SEAWEEDFS_HA_S3_GRAFANA_TEMPO_PASSWORD}"

Pre-Installation

Grafana Tempo Preparation

Set Grafana Tempo Entrypoint Domain

export GRAFANA_TEMPO_DOMAIN="tempo.${BASE_DOMAIN}"

Create Helm Values Configurations

cat <<EOF > $BASE_WORKING_DIR/helm-values/grafana-tempo-full-values.yaml
global:
  ## -- Definitions to set up nginx resolver (nginx gateway that proxied within microservices)
  ## OPSTELLA_CUSTOMIZE/RKE2: Defaults was 'kube-dns'/Change for RKE2
  # -- Definitions to set up nginx resolver
  # -- configures DNS service name
  dnsService: ${K8S_INTERNAL_DNS_SERVICE} # was defaults to kube-dns
  # -- configures DNS service namespace
  dnsNamespace: "kube-system"
  # -- configures cluster domain ("cluster.local" by default)
  clusterDomain: "${K8S_INTERNAL_DOMAIN}"

## OPSTELLA_CUSTOMIZE: Disable Built-in MinIO (it's not intended for Production uses!)
minio:
  enabled: false

## OPSTELLA_CUSTOMIZE: Enable Multi-Tenancy
multitenancyEnabled: true

## OPSTELLA_CUSTOMIZE: Disable Telemetry to Grafana Labs???
# -- If true, Tempo will report anonymous usage data about the shape of a deployment to Grafana Labs
reportingEnabled: false

## OPSTELLA_CUSTOMIZE: XXXXXX
# Configuration for the gateway
gateway:
  # -- Specifies whether the gateway should be enabled
  enabled: true
  ingress:
    enabled: true
    ingressClassName: ${K8S_INGRESSCLASS_NAME}
    hosts:
      - host: ${GRAFANA_TEMPO_DOMAIN}
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: ${K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME}
        hosts:
          - ${GRAFANA_TEMPO_DOMAIN}

## OPSTELLA_CUSTOMIZE: Tempo Storage Configuration
## Utilize S3(-compatible) Object Storage by Default
storage:
  trace:
    backend: s3
    s3:
      endpoint: ${GRAFANA_TEMPO_S3_DOMAIN}
      bucket: ${GRAFANA_TEMPO_S3_TSDB_BUCKET_NAME}
      region: ${GRAFANA_TEMPO_S3_BUCKET_REGION}
      access_key: ${GRAFANA_TEMPO_S3_ACCESS_KEY}
      secret_key: ${GRAFANA_TEMPO_S3_ACCESS_SECRET}
      forcepathstyle: true
      # Plain-HTTP S3 (e.g. in-cluster SeaweedFS): export GRAFANA_TEMPO_S3_INSECURE=true
      # and set GRAFANA_TEMPO_S3_DOMAIN to a bare host:port (no http:// scheme).
      # Defaults to false so external TLS S3 endpoints stay secure.
      insecure: ${GRAFANA_TEMPO_S3_INSECURE:-false}

## OPSTELLA_CUSTOMIZE: Trace Receivers
## The chart enables no receiver by default, which renders "receivers: null" into the
## distributor config and leaves Tempo unable to ingest any span. OTLP over gRPC and HTTP
## are the protocols Grafana Alloy and the OpenTelemetry SDKs use.
traces:
  otlp:
    grpc:
      enabled: true
    http:
      enabled: true

tempo:
  ## OPSTELLA_CUSTOMIZE: SecurityContext
  # -- The SecurityContext for Tempo pods
  podSecurityContext:
    fsGroup: 10001
  # -- The SecurityContext for Tempo containers
  securityContext:
    runAsUser: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    privileged: false
    allowPrivilegeEscalation: false
    capabilities:
      drop: ["ALL"]
    seccompProfile:
      type: RuntimeDefault
    readOnlyRootFilesystem: true

EOF

Helm Chart Preparation

Add Grafana Helm Repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Installation

Install Grafana Tempo

Install a Helm Release with specific Helm Chart Version --version 1.32.5 (App Version: 2.7.1)

helm install grafana-tempo grafana/tempo-distributed --version 1.32.5 \
  --namespace observability-system \
  -f $HOME/opstella-installation/helm-values/grafana-tempo-full-values.yaml

Post-Installation

Grafana Tempo Testing

Get Pods Status

kubectl get pods -n observability-system

💡 Grafana Tempo (Distributed Deployment Mode Components) Pods should be Running

NAME                                           READY   STATUS    RESTARTS
... (deducted)
grafana-tempo-compactor-XXXXXXX-YYYYY          1/1     Running   0          Xd
grafana-tempo-distributor-XXXXXXX-YYYYY        1/1     Running   0          Xd
grafana-tempo-gateway-XXXXXXX-YYYYY            1/1     Running   0          Xd
grafana-tempo-ingester-0                       1/1     Running   0          Xd
grafana-tempo-ingester-1                       1/1     Running   0          Xd
grafana-tempo-ingester-2                       1/1     Running   0          Xd
grafana-tempo-memcached-0                      1/1     Running   0          Xd
grafana-tempo-querier-XXXXXXX-YYYYY            1/1     Running   0          Xd
grafana-tempo-query-frontend-XXXXXXX-YYYYY     1/1     Running   0          Xd

Finished?

Use the below navigation to proceed

Previous
Grafana Loki Next
Health Checks for Opstella