Observability Installation Overview

Observability System that you need to provision and integrate with Opstella which are

1. Grafana Dashboard: An interface visually presents real-time metrics, logs, and traces from various data sources.
2. Grafana Mimir: A highly scalable and performant backend solution for metrics data.
3. Grafana Loki: A log aggregation system that stores and queries logs efficiently.
4. Grafana Tempo: A distributed tracing backend for analyzing application performance and dependencies.
5. Grafana Alloy: A metrics, logs, and traces push-based collector and exporter

Prerequisites

Kubernetes Cluster

Observability will be on 🟢 Management Kubernetes Cluster.

Where Observability Agent Services are span across Kubernetes Cluster(s) which are

• 🟢 Management Kubernetes Cluster
• 🟦 Non-Production Workload Kubernetes Cluster
• 🟥 Production Workload Kubernetes Cluster

Provisioning Summary

Once finished everything, these are systems that up and running.

* Not Including any of application dependencies.

No.	Tool	Description	Agent Service	Kubernetes Namespace	Kubernetes Cluster
1.	Grafana Dashboard	Centralised Observability Dashboard	❌	observability-system	🟢 Management
2.	Grafana Loki	Application/System Logs Backend	❌	observability-system	🟢 Management
3.	Grafana Mimir	Application/System Metrics Backend	❌	observability-system	🟢 Management
4.	Grafana Tempo	Application Tracing Backend	❌	observability-system	🟢 Management
5.	Grafana Alloy (Platform)	Metrics/Logs Gathering from Kubernetes Cluster and its Workloads	✅	observability-agents	🟢 Management 🟦 Non-Production Workload 🟥 Production Workload
6.	Grafana Alloy (Workload)	Metrics/Logs/Traces Gathering from Opstella-managed Application (OpenTelemetry Enabled Applications)	✅	observability-agents	🟦 Non-Production Workload 🟥 Production Workload

Complete Prerequisites

Create Namespace across Kubernetes Clusters

---

🟢 Management Kubernetes Cluster

1. Connect to 🟢 Management Kubernetes Cluster ; i.e w/ Kubeconfig File

   Set Kubeconfig File

   Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

   source $HOME/opstella-installation/shell-values/kubernetes/management_cluster.vars.sh

   export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/management_cluster.yaml"

2. Create Kubernetes Namespace(s)

   2.1 Create Kubernetes Namespace nameed observability-system

   kubectl create namespace observability-system

   2.2 Create Kubernetes Namespace nameed observability-agents

   kubectl create namespace observability-agents

   2.2 Add Label for observability-agents namespace for PSA Exception

   Prometheus Node Exporter requires host-level access which violates PSA restricted profile, and require to make an exception

   kubectl label namespace/observability-agents pod-security.kubernetes.io/enforce=privileged
   kubectl label namespace/observability-agents pod-security.kubernetes.io/enforce-version=latest

---

🟦 Non-Production Workload Kubernetes Cluster

1. Connect to 🟦 Non-Production Workload Kubernetes Cluster ; i.e w/ Kubeconfig File

   Set Kubeconfig File

   Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

   source $HOME/opstella-installation/shell-values/kubernetes/nonprod_cluster.vars.sh

   export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/nonprod_cluster.yaml"

2. Create Kubernetes Namespace(s)

   2.1 Create Kubernetes Namespace named observability-agents

   kubectl create namespace observability-agents

   2.2 Add Label for observability-agents namespace for PSA Exception

   Prometheus Node Exporter will be installed and requires host-level access which violates PSA restricted profile, and require to make an exception

   kubectl label namespace/observability-agents pod-security.kubernetes.io/enforce=privileged
   kubectl label namespace/observability-agents pod-security.kubernetes.io/enforce-version=latest

---

🟥 Production Workload Kubernetes Cluster

1. Connect to 🟥 Production Workload Kubernetes Cluster ; i.e w/ Kubeconfig File

   Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

   source $HOME/opstella-installation/shell-values/kubernetes/prod_cluster.vars.sh

   export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/prod_cluster.yaml"

2. Create Kubernetes Namespaces

   2.1 Create Kubernetes Namespace named observability-agents

   kubectl create namespace observability-agents

   2.2 Add Label for observability-agents namespace for PSA Exception

   Prometheus Node Exporter will be installed and requires host-level access which violates PSA restricted profile, and require to make an exception

   kubectl label namespace/observability-agents pod-security.kubernetes.io/enforce=privileged
   kubectl label namespace/observability-agents pod-security.kubernetes.io/enforce-version=latest