Grafana Alloy (Workload) Installation

Grafana Alloy (Workload) is a Metrics/Logs/Traces Gathering from Opstella-managed Application (OpenTelemetry Enabled Applications)

---

Prerequisites

Kubernetes Cluster

Grafana Alloy (Workload) will be installed on 🟦 Non-Production Workload and 🟥 Production Workload Kubernetes Cluster

• 📥Ingress Service provided as Kubernetes Ingress Class (IngressClass)
• 🛡️TLS Certificate for Grafana Alloy (Workload) provided as Kubernetes Secret
  • Grafana Alloy (Workload) will be exposed as HTTPS with Kubernetes Ingress.

Complete Prerequisites

Kubernetes Cluster

Prepare Shell Variables

Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

1. Connect to 🟢 Management Kubernetes Cluster ; i.e w/ Kubeconfig File

   Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

   source $HOME/opstella-installation/shell-values/kubernetes/management_cluster.vars.sh
   source $HOME/opstella-installation/shell-values/tools/observability.vars.sh

   export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/management_cluster.yaml"

2. Set 🟢 Management Kubernetes Cluster Information

   Ensure K8S_INTERNAL_DOMAIN, K8S_INGRESSCLASS_NAME, K8S_STORAGECLASS_NAME are defined as per the Shell Variables guide.

3. Create Kubernetes Secret for 🛡️ TLS Certificate for Grafana Alloy (Workload) in Namespace observability-system.

   Kubernetes Ingress for Grafana Alloy (Workload) will associate TLS Certificate with Kubernetes Secret named wildcard-${BASE_DOMAIN}-tls.

   export K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME="wildcard-${BASE_DOMAIN}-tls"

   Create one using from .crt and .key file.

   kubectl create secret tls $K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME \
     --cert=/path/to/cert/file --key=/path/to/key/file \
     --namespace observability-system

   💡 Should return secret/wildcard-${BASE_DOMAIN}-tls created message.

Pre-Installation

Grafana Alloy (Workload) Preparation

1. Create Helm Values Configuration, create for 🟦 Non-Production Workload Kubernetes Cluster

   cat <<EOF > $HOME/opstella-installation/helm-values/grafana-alloy-workload-nonprod-full-values.yaml
   ## Global properties for image pulling override the values defined under 'image.registry' and 'configReloader.image.registry'.
   ## If you want to override only one image registry, use the specific fields but if you want to override them all, use 'global.image.registry'
   global:
     # -- Security context to apply to the Grafana Alloy pod.
     podSecurityContext:
       fsGroup: 473
   
   # Options for the extra controller used for config reloading.
   configReloader:
     securityContext:
       runAsUser: 473
       runAsGroup: 473
       runAsNonRoot: true
       privileged: false
       allowPrivilegeEscalation: false
       capabilities:
         drop: ["ALL"]
       seccompProfile:
         type: "RuntimeDefault"
   
   # Options for the extra controller used for config reloading.
   alloy:
   
     ## OPSTELLA_CUSTOMIZE: SecuityContext
     securityContext:
       runAsUser: 473
       runAsGroup: 473
       runAsNonRoot: true
       privileged: false
       allowPrivilegeEscalation: false
       capabilities:
         drop: ["ALL"]
       seccompProfile:
         type: "RuntimeDefault"
   
     configMap:
       create: true
   
     ## OPSTELLA_CUSTOMIZE: Enable Alloy Clustering
     clustering:
       # -- Deploy Alloy in a cluster to allow for load distribution.
       enabled: true
   
     ## OPSTELLA_CUSTOMIZE: Enable Alloy to be OTel gRPC Receiver
     extraPorts:
       - name: otlp-grpc
         port: 4317
         targetPort: 4317
         protocol: "TCP"
   
   EOF

2. Create Helm Values Configuration, create for 🟥 Production Workload Kubernetes Cluster

   cat <<EOF > $HOME/opstella-installation/helm-values/grafana-alloy-workload-prod-full-values.yaml
   ## Global properties for image pulling override the values defined under 'image.registry' and 'configReloader.image.registry'.
   ## If you want to override only one image registry, use the specific fields but if you want to override them all, use 'global.image.registry'
   global:
     # -- Security context to apply to the Grafana Alloy pod.
     podSecurityContext:
       fsGroup: 473
   
   # Options for the extra controller used for config reloading.
   configReloader:
     securityContext:
       runAsUser: 473
       runAsGroup: 473
       runAsNonRoot: true
       privileged: false
       allowPrivilegeEscalation: false
       capabilities:
         drop: ["ALL"]
       seccompProfile:
         type: "RuntimeDefault"
   
   # Options for the extra controller used for config reloading.
   alloy:
   
     ## OPSTELLA_CUSTOMIZE: SecuityContext
     securityContext:
       runAsUser: 473
       runAsGroup: 473
       runAsNonRoot: true
       privileged: false
       allowPrivilegeEscalation: false
       capabilities:
         drop: ["ALL"]
       seccompProfile:
         type: "RuntimeDefault"
   
     configMap:
       create: true
   
     ## OPSTELLA_CUSTOMIZE: Enable Alloy Clustering
     clustering:
       # -- Deploy Alloy in a cluster to allow for load distribution.
       enabled: true
   
     ## OPSTELLA_CUSTOMIZE: Enable Alloy to be OTel gRPC Receiver
     extraPorts:
       - name: otlp-grpc
         port: 4317
         targetPort: 4317
         protocol: "TCP"
   
   EOF

3. Create Kubernetes Manifest Configuration for Grafana Alloy Health Checking for 🟦 Non-Production Workload

   Set Domain for Grafana Alloy

   export GRAFANA_ALLOY_WORKLOAD_DOMAIN="alloy-nonprod.${BASE_DOMAIN}"

   kubectl apply --namespace observability-agents -f - <<EOF
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: grafana-alloy-workload-healthcheck
     namespace: observability-agents
     annotations:
       nginx.ingress.kubernetes.io/rewrite-target: /-/ready
   spec:
     ingressClassName: ${K8S_INGRESSCLASS_NAME}
     tls:
       - secretName: ${K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME}
     rules:
       - host: ${GRAFANA_ALLOY_WORKLOAD_DOMAIN}
         http:
           paths:
             - path: /workload/ready
               pathType: ImplementationSpecific
               backend:
                 service:
                   name: grafana-alloy-workload
                   port:
                     number: 12345
   EOF

4. Create Kubernetes Manifest Configuration for Grafana Alloy Health Checking for 🟥 Production Workload

   Set Domain for Grafana Alloy

   export GRAFANA_ALLOY_WORKLOAD_DOMAIN="alloy-prod.${BASE_DOMAIN}"

   kubectl apply --namespace observability-agents -f - <<EOF
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: grafana-alloy-workload-healthcheck
     namespace: observability-agents
     annotations:
       nginx.ingress.kubernetes.io/rewrite-target: /-/ready
   spec:
     ingressClassName: ${K8S_INGRESSCLASS_NAME}
     tls:
       - secretName: ${K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME}
     rules:
       - host: ${GRAFANA_ALLOY_WORKLOAD_DOMAIN}
         http:
           paths:
             - path: /workload/ready
               pathType: ImplementationSpecific
               backend:
                 service:
                   name: grafana-alloy-workload
                   port:
                     number: 12345
   EOF

Installation

Caution

If you have been following Grafana Alloy (Workload) Preparation section, you’ve just create,

• grafana-alloy-workload-nonprod-full-values.yaml
• grafana-alloy-workload-prod-full-values.yaml

Which are FULL Helm Values that’s everything you need to install Grafana Alloy (Workload) for each of Kubernetes Cluster(s)!

If you need to add anything for extra configuration, feel free to add it there (Please see Default Helm Values from Chart)

1. Connect to 🟦 Non-Production Workload Kubernetes Cluster ; i.e w/ Kubeconfig File

   export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/nonprod_cluster.yaml"

2. Install Grafana Alloy (Workload) on 🟦 Non-Production Workload

   • Install a Helm Release with specific Helm Chart Version --version 0.12.3 (App Version: v1.7.2)

     helm install grafana-alloy-workload grafana/alloy --version 0.12.3 \
       --namespace observability-agents \
       -f $HOME/opstella-installation/helm-values/grafana-alloy-workload-nonprod-full-values.yaml

3. Apply Kubernetes Manifest on 🟦 Non-Production Workload

   kubectl apply -f $HOME/opstella-installation/kubernetes-manifests/grafana-alloy-nonprod-healthcheck-ingress.yaml

4. Connect to 🟥 Production Workload Kubernetes Cluster ; i.e w/ Kubeconfig File

   export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/prod_cluster.yaml"

5. Install Grafana Alloy (Workload) on 🟥 Production Workload

   helm install grafana-alloy-workload grafana/alloy --version 0.12.3 \
     --namespace observability-agents \
     -f $HOME/opstella-installation/helm-values/grafana-alloy-workload-prod-full-values.yaml

6. Apply Kubernetes Manifest on 🟥 Production Workload

   kubectl apply -f $HOME/opstella-installation/kubernetes-manifests/grafana-alloy-prod-healthcheck-ingress.yaml

Post-Installation

Caution

The Grafana Alloy (Platform) Installation instructions here is repetitive, instructions providing here are steps for ONE Kubernetes Cluster.

It is meant for you to repeat the steps for each of Kubernetes Cluster(s). Please change the Kubernetes Cluster accordingly.

Grafana Alloy (Workload) Testing

1. Get Pods Status

   kubectl get pods -n observability-agents

   Grafana Alloy should be Running

   NAME                                READY   STATUS    RESTARTS  AGE
   ... (deducted)
   grafana-alloy-workload-0            1/1     Running   0         Xd
   grafana-alloy-workload-1            1/1     Running   0         Xd
   grafana-alloy-workload-2            1/1     Running   0         Xd