Grafana Health Checks for Opstella Installation

Prerequisites

Kubernetes Cluster

Grafana Health Checks for Opstella will be installed on 🟢 Management Kubernetes Cluster

Expose /ready Path from Mimir, Loki, and Tempo (All in Distributed Deployment Mode) for Opstella to Check for their Health through Ingress.

Re-using Grafana Dashboard Domain and separate traffic with query paths as the following,

Grafana Mimir:

• $GRAFANA_DASHBOARD_DOMAIN/mimir/read/ready - Check querier service
• $GRAFANA_DASHBOARD_DOMAIN/mimir/write/ready - Check distributor service
• $GRAFANA_DASHBOARD_DOMAIN/mimir/backend/ready - Check ingester service

Optionally Learn More about sub-components (Micro-Services) from Grafana Mimir Architecture

Grafana Loki:

• $GRAFANA_DASHBOARD_DOMAIN/loki/read/ready - Check querier service
• $GRAFANA_DASHBOARD_DOMAIN/loki/write/ready - Check distributor service
• $GRAFANA_DASHBOARD_DOMAIN/loki/backend/ready - Check ingester service

Optionally Learn More about sub-components (Micro-Services) from Grafana Loki Architecture

Grafana Tempo:

• $GRAFANA_DASHBOARD_DOMAIN/tempo/read/ready - Check querier service
• $GRAFANA_DASHBOARD_DOMAIN/tempo/write/ready - Check distributor service
• $GRAFANA_DASHBOARD_DOMAIN/tempo/backend/ready - Check ingester service

Optionally Learn More about sub-components (Micro-Services) from Grafana Tempo Architecture

• 📥Ingress Service provided as Kubernetes Ingress Class (IngressClass)
• 🛡️TLS Certificate for Grafana Dashboard provided as Kubernetes Secret
  • Grafana Dashboard will be exposed as HTTPS with Kubernetes Ingress.

Kubernetes Cluster Prerequisites

• 🛡️TLS Certificate for Grafana Health Checks for Opstella
  • Grafana Health Checks for Opstella will be exposed through Web with HTTPS with Kubernetes Ingress configured with TLS Certificate located within Kubernetes Cluster.

Complete Prerequisites

Kubernetes Cluster

1. Connect to 🟢 Management Kubernetes Cluster ; i.e w/ Kubeconfig File

   Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

   source $HOME/opstella-installation/shell-values/kubernetes/management_cluster.vars.sh

   export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/management_cluster.yaml"

2. Create Kubernetes Secret for TLS Certificate in Namespace observability-system. (If you not have it done.)

   Kubernetes Ingress for DefectDojo will associate TLS Certificate with Kubernetes Secret named wildcard-${BASE_DOMAIN}-tls.

   Ensure K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME is defined.

   If not, create one using from .crt and .key file.

   kubectl create secret tls $K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME \
   --cert=/path/to/cert/file --key=/path/to/key/file \
   --namespace observability-system

3. Prepare 🟢 Management Kubernetes Cluster Information

   Ensure GRAFANA_DASHBOARD_DOMAIN, K8S_INGRESSCLASS_NAME, K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME are defined as per the Shell Variables guide.

Installation Preparations

Kubernetes Manifest Preparation

1. Create Kubernetes Manifest Configuration

   cat <<EOF >> $HOME/opstella-installation/kubernetes-manifests/grafana-ltm-healthcheck-ingress.yaml
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: grafana-ltm-healthcheck
     namespace: observability-system
     annotations:
       nginx.ingress.kubernetes.io/rewrite-target: /ready
   spec:
     ingressClassName: ${K8S_INGRESSCLASS_NAME}
     tls:
       - secretName: ${K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME}
         hosts:
           - ${GRAFANA_DASHBOARD_DOMAIN}
     rules:
       - host: ${GRAFANA_DASHBOARD_DOMAIN}
         http:
           paths:
             ## LOKI
             - path: /loki/read/ready
               pathType: Prefix
               backend:
                 service:
                   name: grafana-loki-querier
                   port:
                     number: 3100
             - path: /loki/write/ready
               pathType: Prefix
               backend:
                 service:
                   name: grafana-loki-distributor
                   port:
                     number: 3100
             - path: /loki/backend/ready
               pathType: Prefix
               backend:
                 service:
                   name: grafana-loki-ingester
                   port:
                     number: 3100
   
             ## MIMIR
             - path: /mimir/read/ready
               pathType: Prefix
               backend:
                 service:
                   name: grafana-mimir-querier
                   port:
                     number: 8080
             - path: /mimir/write/ready
               pathType: Prefix
               backend:
                 service:
                   name: grafana-mimir-distributor
                   port:
                     number: 8080
             - path: /mimir/backend/ready
               pathType: Prefix
               backend:
                 service:
                   name: grafana-mimir-ingester
                   port:
                     number: 8080
   
             ## TEMPO
             - path: /tempo/read/ready
               pathType: Prefix
               backend:
                 service:
                   name: grafana-tempo-querier
                   port:
                     number: 3200
             - path: /tempo/write/ready
               pathType: Prefix
               backend:
                 service:
                   name: grafana-tempo-distributor
                   port:
                     number: 3200
             - path: /tempo/backend/ready
               pathType: Prefix
               backend:
                 service:
                   name: grafana-tempo-ingester
                   port:
                     number: 3200
   
   EOF

Installation

1. Apply Kubernetes Manifests

   kubectl apply --namespace observability-system \
     -f $HOME/opstella-installation/kubernetes-manifests/grafana-ltm-healthcheck-ingress.yaml