Grafana Dashboard Installation

Grafana Dashboard is a Centralised Observability Dashboard

---

Prerequisites

Kubernetes Cluster

Grafana Dashboard will be installed on 🟢 Management Kubernetes Cluster

• 📥Ingress Service provided as Kubernetes Ingress Class (IngressClass)
• 🛡️TLS Certificate for Grafana Dashboard provided as Kubernetes Secret
  • Grafana Dashboard will be exposed as HTTPS with Kubernetes Ingress.

Application Dependencies

Grafana Dashboard depends on multiple dependencies to be up and running but only the following will be provisioned and/or managed by you.

• 📦PostgreSQL SQL Database ; Deploy external/dedicated instance of Database.
  • In case will be installed on Kubernetes Cluster.
    • Will require 💿Persistence Storage as Kubernetes Storage Class (StorageClass)

To Setup Single Sign-On with Opstella (MUST be performed at the time of installation), you need

• 📦Opstella Keycloak
  • Your dedicated Keycloak Realm. foobar-opstella ; Please change accordingly
• 🔑OpenID Connect Credentials: Client ID, Client Secret.
  • Gather Client ID, Client Secret - from Opstella Keycloak Appendix - Gathering Client ID, Client Secret

Provisioning Summary

Once finished everything, these are systems that up and running.

No.	Tool	Description	Kubernetes Namespace	Kubernetes Cluster
1.	PostgreSQL for Grafana Dashboard	Database System for Grafana Dashboard	observability-system	🟢 Management
2.	Grafana Dashboard	Centralised Observability Dashboard	observability-system	🟢 Management

Complete Prerequisites

Kubernetes Cluster

Prepare Shell Variables

Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

1. Connect to 🟢 Management Kubernetes Cluster ; i.e w/ Kubeconfig File

   Ensure you have defined and loaded your Global Shell Variables as described in Shell Variables.

   source $HOME/opstella-installation/shell-values/kubernetes/management_cluster.vars.sh
   source $HOME/opstella-installation/shell-values/tools/observability.vars.sh

   export KUBECONFIG="$HOME/opstella-installation/kubeconfigs/management_cluster.yaml"

2. Set 🟢 Management Kubernetes Cluster Information

   Ensure GRAFANA_DASHBOARD_DOMAIN, K8S_INTERNAL_DOMAIN, K8S_INGRESSCLASS_NAME, K8S_STORAGECLASS_NAME, K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME are defined as per the Shell Variables guide.

3. Create Kubernetes Secret for 🛡️ TLS Certificate for Grafana Dashboard in Namespace observability-system.

   Kubernetes Ingress for Grafana Dashboard will associate TLS Certificate with Kubernetes Secret named wildcard-${BASE_DOMAIN}-tls.

   export K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME="wildcard-${BASE_DOMAIN}-tls"

   Create one using from .crt and .key file.

   kubectl create secret tls $K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME \
     --cert=/path/to/cert/file --key=/path/to/key/file \
     --namespace observability-system

   💡 Should return secret/wildcard-${BASE_DOMAIN}-tls created message.

Application Dependencies

PostgreSQL SQL Database

Note

Provided guide adheres to utilising CloudnativePG Operator to provide PostgreSQL instance.

1. Set PostgreSQL Information

   • Database Name: grafana_dashboard
   • Database User: grafana_dashboard_user
   • Database Password: CHANGEME
     • NOTE: Only Allowed Special Characters are !-_=+?@

   export GRAFANA_DASHBOARD_DB_NAME="grafana_dashboard"

   export GRAFANA_DASHBOARD_DB_USER="grafana_dashboard_user"

   export GRAFANA_DASHBOARD_POSTGRES_USER_PASSWORD="CHANGEME"
   
   export GRAFANA_DASHBOARD_POSTGRES_SUPERUSER_PASSWORD="CHANGEME"
   export SEAWEEDFS_HA_S3_POSTGRES_BACKUP_PASSWORD="CHANGEME"

   Note

   SEAWEEDFS_HA_S3_POSTGRES_BACKUP_PASSWORD is assumed to be exported as defined in the SeaweedFS (HA) Installation guide.

2. Provision PostgreSQL for Grafana Dashboard

   The following manifest defines the entire PostgreSQL system, including credentials and the CNPG cluster.

   cat <<EOF > $HOME/opstella-installation/kubernetes-manifests/grafana-dashboard-postgres.yaml
   ---
   # Postgres Superuser (root) Credentials
   apiVersion: v1
   kind: Secret
   type: kubernetes.io/basic-auth
   metadata:
     name: grafana-dashboard-postgres-superuser
     namespace: observability-system
   stringData:
     username: postgres
     password: "${GRAFANA_DASHBOARD_POSTGRES_SUPERUSER_PASSWORD}"
   ---
   # Grafana Dashboard Application User Credentials
   apiVersion: v1
   kind: Secret
   type: kubernetes.io/basic-auth
   metadata:
     name: grafana-dashboard-postgres-user
     namespace: observability-system
   stringData:
     username: "${GRAFANA_DASHBOARD_DB_USER}"
     password: "${GRAFANA_DASHBOARD_POSTGRES_USER_PASSWORD}"
   ---
   # S3 Credentials for Postgres Backups (to SeaweedFS HA)
   apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     name: grafana-dashboard-postgres-s3-secret
     namespace: observability-system
   stringData:
     S3_ACCESS_KEY: "postgres-backup"
     S3_SECRET_KEY: "${SEAWEEDFS_HA_S3_POSTGRES_BACKUP_PASSWORD}"
   ---
   apiVersion: barmancloud.cnpg.io/v1
   kind: ObjectStore
   metadata:
     name: grafana-dashboard-postgres-backup
     namespace: observability-system
   spec:
     # This resource defines the backup destination for the Barman Cloud Plugin
     configuration:
       destinationPath: s3://postgres-backups/
       endpointURL: http://seaweedfs-s3.apps-supporting-services.svc:9000
       s3Credentials:
         accessKeyId:
           name: grafana-dashboard-postgres-s3-secret
           key: S3_ACCESS_KEY
         secretAccessKey:
           name: grafana-dashboard-postgres-s3-secret
           key: S3_SECRET_KEY
       wal:
         compression: gzip
       data:
         compression: gzip
   
     # Retention policy for backups and WALs handled by the plugin
     retentionPolicy: "30d"
   
   ---
   apiVersion: postgresql.cnpg.io/v1
   kind: Cluster
   metadata:
     name: grafana-dashboard-postgres
     namespace: observability-system
     labels:
       app.kubernetes.io/name: grafana-dashboard-postgres
       app.kubernetes.io/part-of: grafana-dashboard
   spec:
     instances: 1
   
     # Image configuration (Postgres 15)
     imageName: ghcr.io/cloudnative-pg/postgresql:15
   
     # Enable the Barman Cloud Plugin for backups
     plugins:
       - name: barman-cloud.cloudnative-pg.io
         isWALArchiver: true
         parameters:
           serverName: "grafana-dashboard-postgres"
           barmanObjectName: "grafana-dashboard-postgres-backup"
   
     # Storage Configuration
     storage:
       size: 5Gi
       storageClass: "${K8S_STORAGECLASS_NAME}"
   
     walStorage:
       size: 2Gi
       storageClass: "${K8S_STORAGECLASS_NAME}"
   
     # Bootstrap Configuration
     bootstrap:
       initdb:
         database: "${GRAFANA_DASHBOARD_DB_NAME}"
         owner: "${GRAFANA_DASHBOARD_DB_USER}"
         # Secret containing password for the Application User
         secret:
           name: grafana-dashboard-postgres-user
   
     # Superuser credentials (root/postgres)
     superuserSecret:
       name: grafana-dashboard-postgres-superuser
   
     # Backup Configuration (Barman Cloud Plugin will automatically detect matching ObjectStore)
     backup:
       {}
   
     # Monitoring (disabled for now)
     monitoring:
       enablePodMonitor: false
   
     # Resources
     resources:
       requests:
         memory: "256Mi"
         cpu: "200m"
       limits:
         memory: "1Gi"
         cpu: "1000m"
   
   EOF

   Apply the configuration:

   kubectl apply --namespace observability-system \
       -f grafana-dashboard-postgres.yaml

   💡 Should return secret/grafana-dashboard-postgres-superuser, secret/grafana-dashboard-postgres-user, and cluster.postgresql.cnpg.io/grafana-dashboard-postgres created messages.

3. Check for PostgreSQL for Grafana Dashboard Readiness - MUST be Ready and Running.

   Get Pod Status - PostgreSQL for Grafana Dashboard

   kubectl get pods -n observability-system

   💡 PostgreSQL for Grafana Dashboard Pods MUST be Running

   NAME                        READY   STATUS    RESTARTS   AGE
   ... (deducted)
   postgresql-grafana-dashboard-1     2/2     Running   0          XdXh
   postgresql-grafana-dashboard-2     2/2     Running   0          XdXh
   postgresql-grafana-dashboard-3     2/2     Running   0          XdXh

Pre-Installation

Prepare Grafana Dashboard Configurations

1. Set Grafana Dashboard Entrypoint Domain

   export GRAFANA_DASHBOARD_DOMAIN="grafana.${BASE_DOMAIN}"

2. Create/Store Grafana Dashboard Credentials for Initial Admin Account

   • Username: admin (Cannot be changed)
   • Password: CHANGEME

   export GRAFANA_DASHBOARD_PASSWORD="CHANGEME"

   Create Kubernetes Secret named grafana-dashboard-admin-credentials

   kubectl apply --namespace observability-system -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: grafana-dashboard-admin-credentials
   stringData:
     admin-user: admin
     admin-password: ${GRAFANA_DASHBOARD_PASSWORD}
   EOF

   • PostgreSQL Connection and Credentials
     • Use previously mentioned in Application Dependencies/PostgreSQL for Grafana Dashboard

   export GRAFANA_DASHBOARD_DB_HOST="postgresql-grafana-dashboard-rw.observability-system.svc.${K8S_INTERNAL_DOMAIN}"

Prepare Grafana Dashboard Single Sign-On Configurations

1. Specify OIDC Authentication Information

   Using Opstella Keycloak Information

   • Opstella Keycloak Domain: idp.${BASE_DOMAIN}

     export KEYCLOAK_DOMAIN="idp.${BASE_DOMAIN}"

   • Opstella Keycloak Realm Name: foobar-opstella

     💡 Your dedicated Keycloak Realm. foobar-opstella ; Please change accordingly

     export KEYCLOAK_REALM="foobar-opstella"

   • OIDC Issuer Endpoint from Opstella Keycloak Information

     export OIDC_ISSUER_ENDPOINT="https://${KEYCLOAK_DOMAIN}/realms/${KEYCLOAK_REALM}"

   • Client ID: grafana

     export GRAFANA_DASHBOARD_OIDC_CLIENT_ID="grafana"

   • Client secret: CHANGEME

     export GRAFANA_DASHBOARD_OIDC_CLIENT_SECRET="CHANGEME"

2. Create Kubernetes Secret named grafana-dashboard-idp-credentials

   kubectl apply --namespace observability-system -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: grafana-dashboard-idp-credentials
   stringData:
     client_id: ${GRAFANA_DASHBOARD_OIDC_CLIENT_ID}
     client_secret: ${GRAFANA_DASHBOARD_OIDC_CLIENT_SECRET}
   EOF

Create Grafana Dashboard Configurations

1. Create Helm Values Configurations

   cat <<EOF > $HOME/opstella-installation/helm-values/grafana-dashboard-full-values.yaml
   ingress:
     enabled: true
     ingressClassName: ${K8S_INGRESSCLASS_NAME}
     path: /
     pathType: Prefix
     hosts:
       - ${GRAFANA_DASHBOARD_DOMAIN}
     tls:
       - secretName: ${K8S_INGRESS_TLS_CERTIFICATE_SECRET_NAME}
         hosts:
           - ${GRAFANA_DASHBOARD_DOMAIN}
   
   admin:
     existingSecret: grafana-dashboard-admin-credentials
   
   persistence:
     enabled: false
   
   extraSecretMounts:
     ## OPSTELLA_CUSTOMIZE: Single Sign-On Integration
     - name: auth-generic-oauth-secret-mount
       secretName: grafana-dashboard-oidc-credentials
       defaultMode: 0440
       mountPath: /etc/secrets/grafana-dashboard-idp-credentials
       readOnly: true
     - name: db-auth-secret-mount
       secretName: grafana-dashboard-db-credentials
       defaultMode: 0440
       mountPath: /etc/secrets/grafana-dashboard-db-credentials
       readOnly: true
   
   ## Grafana's primary configuration
   ## NOTE: values in map will be converted to ini format
   ## ref: http://docs.grafana.org/installation/configuration/
   ##
   grafana.ini:
     server:
       ## OPSTELLA_CUSTOMIZE: Enable gzip for Optimized Front-end Performance
       enable_gzip: true
       ## OPSTELLA_CUSTOMIZE: Single Sign-On Integration/Required for OAuth Protocol ##
       root_url: https://${GRAFANA_DASHBOARD_DOMAIN}
       domain: ${GRAFANA_DASHBOARD_DOMAIN}
     ## OPSTELLA_CUSTOMIZE: Utilize External Database
     database:
       type: postgres
       host: ${GRAFANA_DASHBOARD_DB_HOST}
       name: ${GRAFANA_DASHBOARD_DB_NAME}
       user: \$__file{/etc/secrets/grafana-dashboard-db-credentials/username}
       password: \$__file{/etc/secrets/grafana-dashboard-db-credentials/password}
       ssl_mode: disable
     auth:
       ## OPSTELLA_CUSTOMIZE: Enable Matching Users with their email
       ## See: https://github.com/grafana/grafana/issues/74154
       oauth_allow_insecure_email_lookup: true
     auth.generic_oauth:
       ## OPSTELLA_CUSTOMIZE: Single Sign-On Integration
       ## See: https://github.com/grafana/grafana/issues/74154
       enabled: true
       name: Opstella
       client_id: \$__file{/etc/secrets/grafana-dashboard-idp-credentials/client_id}
       client_secret: \$__file{/etc/secrets/grafana-dashboard-idp-credentials/client_secret}
       auth_url: ${OIDC_ISSUER_ENDPOINT}/protocol/openid-connect/auth
       token_url: ${OIDC_ISSUER_ENDPOINT}/protocol/openid-connect/token
       api_url: ${OIDC_ISSUER_ENDPOINT}/protocol/openid-connect/userinfo
       email_attribute_name: email:primary
       scopes: openid profile email
       ## OPSTELLA_CUSTOMIZE: Opstella Platform will manage Roles of a User within Organzation
       skip_org_role_sync: true
   
   EOF

Helm Chart Preparation

1. Add Grafana Helm Repository

   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update

Installation

1. Check for PostgreSQL for Grafana Dashboard Readiness - MUST be Running and Ready.

   Get Pod Status - PostgreSQL for Grafana Dashboard

   kubectl get pods -n devsecops-system

   💡 PostgreSQL for Grafana Dashboard Pods MUST be Running

   NAME                   READY   STATUS    RESTARTS   AGE
   ... (deducted)
   postgresql-grafana-dashboard-1    2/2     Running   0          XdXh
   postgresql-grafana-dashboard-2    2/2     Running   0          XdXh
   postgresql-grafana-dashboard-3    2/2     Running   0          XdXh

2. Install Grafana Dashboard

   • Install a Helm Release with specific Helm Chart Version --version 8.10.3 (App Version: 11.5.2)

     helm install grafana grafana/grafana --version 8.10.3 \
       --namespace observability-system \
       -f $HOME/opstella-installation/helm-values/grafana-dashboard-full-values.yaml

Post-Installation

Grafana Dashboard Testing

1. Get Pod Status - Grafana Dashboard

   kubectl get pods -n observability-system

   💡 Grafana Dashboard Pod should be Running

   NAME                              READY   STATUS    RESTARTS    AGE
   ... (deducted)
   grafana-dashboard-XXXXXXX-YYYY    1/1     Running   0           XdXh

2. Visit https://grafana.${BASE_DOMAIN}

   • It should be accessible.
   • TLS Certificate should be valid and coresponding to your TLS Certificate Installed

   

3. Login with Users

   • Try to login with Initial Admin Account (admin, $GRAFANA_DASHBOARD_PASSWORD), if success then it works.