Opstella GitLab CI Standard CI/CD Pipeline - Detailed Guide

Table of Contents

How to Use
Pipeline Compatibility
Pipeline Defaults
Pipeline Features

How to Use

As GitLab CI with Default configuration for Pipeline will use .gitlab-ci.yml file.

Your repository should contain .gitlab-ci.yml like the following to utilise the pipeline as a starter. (Create one if not exist)

The Standard `.gitlab-ci.yml`

include:
  - project: "opstella-cicd/pipeline-template"
    ref: "latest"
    file: "/pipelines/01-default-opstella/template.yml"

Pipeline Compatibility

GitLab Server/Runner Requires 17.0 or later
- Must be Linux-based Runner
- Requires Kubernetes as Executor
Mimir Requires Multi-Tenant enabled
DefectDojo Requires 2.33.7 or later
SonarQube Requires 9.9 LTS or later
ArgoCD Requires 2.12.0 or later
Opstella/Generic Deployment Chart Requires 0.3.15 or later
Gimlet/OneChart Chart Requires 0.66.0 or later

Pipeline Defaults

Source Code Directory Path: src

If your repository does store Source Code differently, please adjust SOURCE_CODE_PATH variable accordingly.

For Example; Your repository stored in source path

include:
  - project: "opstella-cicd/pipeline-template"
    ref: "latest"
    file: "/pipelines/01-default-opstella/template.yml"

### ADD THIS ###
variables:
  SOURCE_CODE_PATH: "source"

Acceptance Test Path: /
- If your application has different web query path i.e /health, please adjust ACCEPTANCE_PATH variable accordingly.
```
  ... (deducted)
variables:
  ACCEPTANCE_PATH: "/health"
```

Expected Directory Structure

Shown example is Go Project

.
├── src/
│   ├── pkg
│   ├── internal
│   ├── ...
│   ├── cmd/
│   │   └── main.go
│   └── ...
├── Dockerfile
├── .gitlab-ci.yml
├── Dockerfile.baseimage (optional)
├── Dockerfile.testimage (optional)
└── ...

Git Taging

Git Tagging will be the indicator for Your Application that pair with Git History

Versioning for Your Application as Semantic Versioning Pattern
- v0.0.0 **Recommended
- v.0.0.0 **Leave Available ONLY for Backward-compatibility
Docker Special Images with Semantic Versioning Pattern (See More)
- XXXimage-v0.0.0 **Recommended
  - baseimage-v0.0.0, testimage-v0.0.0
- XXXimage-v.0.0.0 **Leave Available ONLY for Backward-compatibility
  - baseimage-v.0.0.0, testimage-v.0.0.0

Pipeline Features

Pipeline Profiles/Presets and Versioning

Pipelines are categorised as Profiles (or you can call it Presets),

Within .gitlab-ci.yml file,

in include to file syntax of GitLab CI, will indicate which Profile/Preset is using.

in include to ref syntax of GitLab CI, will indicate which Version is using.

include:
  - project: "opstella-cicd/pipeline-template"
    ref: "latest"                                        ##    THIS   ##
    file: "/pipelines/01-default-opstella/template.yml"  ## ALSO THIS ##

Ready to use Pipeline Profiles/Presets

00-base-pipeline: /pipelines/00-base-pipeline/template.yml
- Minimal Build, Push, and Deploy.
01-default-opstella: /pipelines/01-default-opstella/template.yml
- Opstella Standard Procedure ; includes Most of the Testing.
- Tests
  - Pre-build Security Tests
    - SAST, SCA, IaC with SonarQube, Trivy
  - Post-build Security Tests
    - DAST with OWASP ZAP
    - Container Vulnerability with Trivy
    - Basic Acceptance Test for Service Reachability
- Build and Push Application as Container
- Deploy
  - With Opstella GitOps and ArgoCD

Ready to use Pipeline Profiles/Presets by Languages/Frameworks

10-javascript-nodejs-opstella: /pipelines/10-javascript-nodejs-opstella/template.yml
- Same as Opstella Standard Procedure (01-default-opstella)
- Tailored for JavaScript/NodeJS with NPM Projects
  - Run Unit Test with Jest
11-java-springboot-opstella: /pipelines/11-java-springboot-opstella/template.yml
- Same as Opstella Standard Procedure (01-default-opstella)
- Tailored for JavaScript/NodeJS with Maven Projects
  - Run Unit Test with Maven Test Goal
  - SonarQube with Maven Integration
12-python-opstella: /pipelines/12-python-opstella/template.yml
- Same as Opstella Standard Procedure (01-default-opstella)
- Tailored for Python Projects
  - Run Unit Test with Pytest

Mirrored Registry/Proxy Cache

Prefix available for container images used across jobs and procedure which will be pulled through Harbor Mirrored Registry/Proxy Cache.

This help for

Your network utilisation for the Internet access.
Alleviate Registries Access Limits i.e docker.io (Docker Hub)

Please Create Harbor Proxy Cache Projects named the following:

mirror-docker.io
mirror-gcr.io
mirror-quay.io

You need to have MIRROR_REGISTRY variable created and set in GitLab CI/CD Variable to be: harbor.example/mirror-

Dockerfile Specials

With Image Building as Container (Docker), you can have special images to reduce build time which mostly spent on downloading external dependencies.

Base Image: An image that contains application dependencies that really required for application to run and Run in Production Environment.
Test Image: An image that contains application dependencies that ONLY required in non-production environments, mostly used while testing/evaluating your source code in non-production.
- For instance, You are writing JavaScript/NodeJS. it has devDependencies which separate application dependencies for unit test frameworks.
  - You don’t need unit test libraries for run in production and you can have test images to cache those dependencies devDependencies.

Note that this also include any dependencies not only from Package Managers but also from OS as well.

FROM node:lts AS builder

  ... (deducted)

### APPLICATION DEPENDENCIES

RUN npm install

### OS DEPENDENCIES

RUN apt install git

## OR ; Depends on your image used base OS##

RUN apk add git

Instruction

Requires Dockerfile.XXXimage at the root of your repository and that Dockerfile.XXXimage should run only just installing application dependencies.

For example,

Your typical Dockerfile (You should already have Dockerfile that contains all of typical Docker building instructions)

## Dockerfile.baseimage Example

FROM node:lts AS builder

  ... (deducted)

RUN npm install --production
RUN apt install git

RUN npm run build

  ... (deducted)

CMD ["node", "server.js"]

Create Dockerfile.baseimage or Dockerfile.testimage or Dockerfile.XXXimage in your repository (At develop branch)

## Dockerfile.baseimage Example

FROM node:lts AS builder

  ... (deducted)

RUN npm install --production
RUN apt install git

## Dockerfile.testimage Example

FROM node:lts AS builder

  ... (deducted)

RUN npm install --dev
RUN apt install git

Triggering Special Image Builds

You need to use Git Tagging to indicate that a certain point of time (from Git History) that you would freeze your application dependencies state/version.
- XXXimage-v0.0.0 **Recommended
  - baseimage-v0.0.0, testimage-v0.0.0
- XXXimage-v.0.0.0 **Leave Available ONLY for Backward-compatibility
  - baseimage-v.0.0.0, testimage-v.0.0.0
Once built and version is progressed, Image tags will contain the version and the latest one will also be tagged XXXimage-latest

Also once that is done, your Dockerfile can be replace with the special image in builder section.

## Dockerfile.baseimage Example

FROM harbor.example.com/<project-name>/<image-repository-name>:baseimage-<vX.Y.Z/latest> AS builder

RUN npm run build

  ... (deducted)

CMD ["node", "server.js"]

Dockerfile Build Arguments (Optional)

To take advantages from certain automation from this pipeline which requires Dockerfile modifications by adding extra Dockerfile Arguments.