This is the Experimental version (Latest). It is under active development and may change. For the most reliable documentation, use the version selector in the top-right to switch to Stable, or click here to go to the Stable version's homepage.
Grafana Tempo Installation
Last updated:
Grafana Tempo is a Tracing Aggregation for LGTM Observability Stack
Prerequisites
Section titled “Prerequisites”Kubernetes Cluster
Section titled “Kubernetes Cluster”Grafana Tempo will be installed on 🟢 Management Kubernetes Cluster
- 📥Ingress Service provided as Kubernetes Ingress Class (
IngressClass) - 🛡️TLS Certificate for Grafana Tempo provided as Kubernetes Secret
- Grafana Tempo will be exposed as HTTPS with Kubernetes Ingress.
Application Dependencies
Section titled “Application Dependencies”- 📦S3 API-compatible Object Storage ; For Metrics Storage
- 🪣S3 Buckets: A Unit of Logical Storage with 🌏Region specified.
- Grafana Tempo uses 1 bucket.
- Tracing Storage (TSDB)
- Grafana Tempo uses 1 bucket.
- 🔑Credentials to Access S3 Bucket: Access Key, Secret Key.
- Create/Gather a Dedicated Access Key/Secret Key for Grafana Tempo to access to its buckets.
- 🪣S3 Buckets: A Unit of Logical Storage with 🌏Region specified.
Complete Prerequisites
Section titled “Complete Prerequisites”Kubernetes Cluster
Section titled “Kubernetes Cluster”Prepare Shell Variables
Section titled “Prepare Shell Variables”Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.
-
Connect to
🟢 ManagementKubernetes Cluster ; i.e w/ Kubeconfig FileEnsure you have defined and loaded your Global Shell Variables as described in Shell Variables.
Terminal window source $HOME/opstella-installation/shell-values/kubernetes/management_cluster.vars.sh -
Set
🟢 ManagementKubernetes Cluster InformationEnsure
GRAFANA_TEMPO_DOMAIN,K8S_INTERNAL_DOMAIN,K8S_INGRESSCLASS_NAME,K8S_STORAGECLASS_NAME,K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAMEare defined as per the Shell Variables guide. -
Create Kubernetes Secret for 🛡️ TLS Certificate for Grafana Tempo in Namespace
observability-system.Kubernetes Ingress for Grafana Tempo will associate TLS Certificate with Kubernetes Secret named
wildcard-${BASE_DOMAIN}-tls.(Ensure
K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAMEis defined in your variables)Create one using from
.crtand.keyfile.Terminal window kubectl create secret tls $K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME \--cert=/path/to/cert/file --key=/path/to/key/file \--namespace observability-system💡 Should return
secret/wildcard-${BASE_DOMAIN}-tls createdmessage.
Application Dependencies
Section titled “Application Dependencies”S3 API-compatible Object Storage
Section titled “S3 API-compatible Object Storage”Set S3 API-compatible Object Storage Information for Grafana Tempo.
-
Set S3 Connection with Domain
(Ensure your S3 domain, bucket names, regions, and credentials are all loaded from your tool variables in
observability.vars.sh)
Pre-Installation
Section titled “Pre-Installation”Grafana Tempo Preparation
Section titled “Grafana Tempo Preparation”-
Set Grafana Tempo Entrypoint Domain
(Ensure Grafana Tempo Domain and Mimir Target are loaded from your variables)
-
Create Helm Values Configurations
Terminal window cat <<EOF > $BASE_WORKING_DIR/helm-values/grafana-tempo-full-values.yamlglobal:## -- Definitions to set up nginx resolver (nginx gateway that proxied within microservices)## OPSTELLA_CUSTOMIZE/RKE2: Defaults was 'kube-dns'/Change for RKE2# -- Definitions to set up nginx resolver# -- configures DNS service namednsService: ${K8S_INTERNAL_DNS_SERVICE} # was defaults to `kube-dns`# -- configures DNS service namespacednsNamespace: "kube-system"# -- configures cluster domain ("cluster.local" by default)clusterDomain: "${K8S_INTERNAL_DOMAIN}"## OPSTELLA_CUSTOMIZE: Disable Built-in MinIO (it's not intended for Production uses!)minio:enabled: false## OPSTELLA_CUSTOMIZE: Enable Multi-TenancymultitenancyEnabled: true## OPSTELLA_CUSTOMIZE: Disable Telemetry to Grafana Labs???# -- If true, Tempo will report anonymous usage data about the shape of a deployment to Grafana LabsreportingEnabled: false## OPSTELLA_CUSTOMIZE: XXXXXX# Configuration for the gatewaygateway:# -- Specifies whether the gateway should be enabledenabled: trueingress:enabled: trueingressClassName: ${K8S_INGRESSCLASS_NAME}hosts:- host: ${GRAFANA_TEMPO_DOMAIN}paths:- path: /pathType: Prefixtls:- secretName: ${K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME}hosts:- ${GRAFANA_TEMPO_DOMAIN}## OPSTELLA_CUSTOMIZE: Tempo Storage Configuration## Utilize S3(-compatible) Object Storage by Defaultstorage:trace:backend: s3s3:endpoint: ${GRAFANA_TEMPO_S3_DOMAIN}bucket: ${GRAFANA_TEMPO_S3_TSDB_BUCKET_NAME}region: ${GRAFANA_TEMPO_S3_BUCKET_REGION}access_key: ${GRAFANA_TEMPO_S3_ACCESS_KEY}secret_key: ${GRAFANA_TEMPO_S3_ACCESS_SECRET}forcepathstyle: truetempo:## OPSTELLA_CUSTOMIZE: SecurityContext# -- The SecurityContext for Tempo podspodSecurityContext:fsGroup: 10001# -- The SecurityContext for Tempo containerscontainerSecurityContext:runAsUser: 10001runAsGroup: 10001runAsNonRoot: trueprivileged: falseallowPrivilegeEscalation: falsecapabilities:drop: ["ALL"]seccompProfile:type: RuntimeDefaultreadOnlyRootFilesystem: trueEOF
Helm Chart Preparation
Section titled “Helm Chart Preparation”-
Add Grafana Helm Repository
Terminal window helm repo add grafana https://grafana.github.io/helm-chartshelm repo update
Installation
Section titled “Installation”-
Install Grafana Tempo
- Install a Helm Release with specific Helm Chart Version
--version 1.32.5(App Version: 2.7.1)
Terminal window helm upgrade --install grafana-tempo grafana/tempo-distributed --version 1.32.5 \--namespace observability-system \-f $HOME/opstella-installation/helm-values/grafana-tempo-full-values.yaml - Install a Helm Release with specific Helm Chart Version
Post-Installation
Section titled “Post-Installation”Grafana Tempo Testing
Section titled “Grafana Tempo Testing”-
Get Pods Status
Terminal window kubectl get pods -n observability-system💡 Grafana Tempo (Distributed Deployment Mode Components) Pods should be
RunningNAME READY STATUS RESTARTS... (deducted)grafana-tempo-compactor-XXXXXXX-YYYYY 1/1 Running 0 Xdgrafana-tempo-distributor-XXXXXXX-YYYYY 1/1 Running 0 Xdgrafana-tempo-gateway-XXXXXXX-YYYYY 1/1 Running 0 Xdgrafana-tempo-ingester-0 1/1 Running 0 Xdgrafana-tempo-ingester-1 1/1 Running 0 Xdgrafana-tempo-ingester-2 1/1 Running 0 Xdgrafana-tempo-memcached-0 1/1 Running 0 Xdgrafana-tempo-metrics-generator-XXXXXXX-YYYYY 1/1 Running 0 Xdgrafana-tempo-querier-XXXXXXX-YYYYY 1/1 Running 0 Xdgrafana-tempo-query-frontend-XXXXXXX-YYYYY 1/1 Running 0 Xd
Finished?
Use the below navigation to proceed