Setup GitLab Backup

This content is not available in your language yet.

🟢 Management/GitLab

This will be optional task, provided for references only.

Reference: Official GitLab Backup Document

** Backups are assumed that you’re not snapshotting or use any solution that backup at the VM instance level

Understanding Backup Strategy

There are 2 type of Backup that you need to concern, Backup managed by GitLab and Unmanaged by GitLab

Managed by GitLab are Git Repositories Data, Database Dumps
Unmanaged by GitLab are SSH Keys, GitLab Configuration Files, Secret Files
- As GitLab stated from Official GitLab Backup Document
  
  The primary reason for this is that your database contains items including encrypted information for two-factor authentication and the CI/CD secure variables. Storing encrypted information in the same location as its key defeats the purpose of using encryption in the first place.
  
  For example, the secrets file contains your database encryption key. If you lose it, then the GitLab application will not be able to decrypt any encrypted values in the database.
  
  You may also want to back up any TLS keys and certificates (/etc/gitlab/ssl, /etc/gitlab/trusted-certs), and your SSH host keys (/etc/ssh) to avoid man-in-the-middle attack warnings if you have to perform a full machine restore.

Managed by GitLab Backup

Managed by GitLab Backup will be done by by running /opt/gitlab/bin/gitlab-backup create

This will dump database and Git Repositories data by archiving (using tar)

/opt/gitlab/bin/gitlab-backup is a GitLab utility to do backup and it follows backup configuration by /etc/gitlab/gitlab.rb file which will configure to upload backed up data to a remote (cloud) storage. (S3(-compatible) one)

Reference: Upload backups to a remote (cloud) storage

Steps

Become Root User
Terminal window
```
sudo -i
```

Configure /etc/gitlab/gitlab.rb

Assume that you’re

Using S3(-compatible) Object Storage with domain seaweedfs-api.${BASE_DOMAIN}
aws_access_key_id is Username and aws_secret_access_key is Password for authentication with API
Upload to gitlab-backups Bucket

GITLAB_BACKUP_S3_API_DOMAIN="seaweedfs-api.${BASE_DOMAIN}"

GITLAB_BACKUP_S3_BUCKET_NAME="gitlab-backups"

GITLAB_BACKUP_S3_BUCKET_REGION="us-east-1"

GITLAB_BACKUP_S3_ACCESS_KEY="gitlab-backup"

GITLAB_BACKUP_S3_ACCESS_SECRET="CHANGEME"

Add these to /etc/gitlab/gitlab.rb

cat <<EOF >> /etc/gitlab/gitlab.rb
# OPSTELLA_CUSTOMIZE: Add GitLab Backup to S3(-compatible)
gitlab_rails['backup_upload_connection'] = {
  'provider' => 'AWS',
  'region' => '${GITLAB_BACKUP_S3_BUCKET_REGION}',
  'aws_access_key_id' => '${GITLAB_BACKUP_S3_ACCESS_KEY}',
  'aws_secret_access_key' => '${GITLAB_BACKUP_S3_ACCESS_SECRET}',
  'endpoint' => 'https://${GITLAB_BACKUP_S3_API_DOMAIN}',
  'path_style' => true, #
}

# OPSTELLA_CUSTOMIZE: Add GitLab Backup to an S3(-compatible) Bucket
gitlab_rails['backup_upload_remote_directory'] = '${GITLAB_BACKUP_S3_BUCKET_NAME}'

# Consider using multipart uploads when file size reaches 100MB. Enter a number in bytes.
gitlab_rails['backup_multipart_chunk_size'] = 104857600

# Limit backup lifetime to 7 days - 604800 seconds
# To prevent regular backups from using all your disk space, you may want to set a limited lifetime for backups. The next time the backup task runs, backups older than the backup_keep_time are pruned.
gitlab_rails['backup_keep_time'] = 604800
EOF

This will upload archived (tar) files to S3(-compatible) Object Storage and follow all of rentention strategy in gitlab.rb configuration (Default for Opstella is Retention for 7 Days)

Setup Cron Job

Instead, configure crontab to run every Midnight as a Daily Backup, this will be stored on to your disk to backup path /var/opt/gitlab/backups (Default Path or gitlab_rails['backup_path'] value if you’ve configure differently)
Terminal window
```
sudo crontab -e
```
```
0 0 * * * /opt/gitlab/bin/gitlab-backup create CRON=1
```

Unmanaged by GitLab Backup

Unmanaged Backup will be done through Shell Scripts and crontab configuration

Steps

Install Object Storage CLI

You need to install MinIO CLI mc to use the following shell script

sudo curl https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/bin/mc
sudo chmod u+x /usr/bin/mc

Install Backup Script

This script is taring files in /etc/gitlab and /etc/ssh and Upload to S3(-compatible) Object Storage

This requires Access Key, Secret Key for upload files to gitlab-backups Bucket

sudo -i
cat <<EOF >> /usr/bin/gitlab-unmanaged-backup
#!/bin/bash
set -xe

# Object Storage keys
s3_access_key=CHANGEME
s3_secret_key=CHANGEME
bucket_name=gitlab-backups

# Set MinIO CLI Alias
/usr/bin/mc alias set s3-store https://seaweedfs-api.${BASE_DOMAIN} ${s3_access_key} ${s3_secret_key}

# Initial GitLab Config Backup
echo "<== RUN: TARING GITLAB CONFIG FOLDER ==>"
BACKUP_CONFIG_TIME=$(date "+%s")
CONFIG_BACKUP_NAME=/var/opt/gitlab/backups/$(date "+gitlab_config_${BACKUP_CONFIG_TIME}_%Y_%m_%d.tar")
tar cfz $CONFIG_BACKUP_NAME -C / etc/gitlab

echo $CONFIG_BACKUP_NAME > /var/opt/gitlab/backups/.last_backup

# Initial SSH Backup
echo "<== RUN: TARING SSH FOLDER ==>"
BACKUP_SSH_TIME=$(date "+%s")
SSH_BACKUP_NAME=/var/opt/gitlab/backups_ssh/$(date "+gitlab_ssh_${BACKUP_SSH_TIME}_%Y_%m_%d.tar")
tar cfz $SSH_BACKUP_NAME -C / etc/ssh

echo $SSH_BACKUP_NAME > /var/opt/gitlab/backups_ssh/.last_backup

/usr/bin/mc cp $CONFIG_BACKUP_NAME s3-store/${bucket_name}/backups_config/
/usr/bin/mc cp $SSH_BACKUP_NAME s3-store/${bucket_name}/backups_ssh/

EOF

Apply Appropriate File Permission

sudo chmod u+x /usr/bin/gitlab-unmanaged-backup

Install Retention Backup Script

Like Backups managed by GitLab, it will store file on the local disk before upload.

From Backup Script, have left a breadcrumb for which is the latest backup . last_backup

For this we only keep the last one and remove all of the older backup as this assume that the older ones have been backed up to remote storage.

sudo -i
cat <<EOF >> /usr/bin/gitlab-unmanaged-retention
#!/bin/bash
set -xe

# GitLab Config Backup Rentention
LAST_CONFIG_BACKUP=$(cat /var/opt/gitlab/backups/.last_backup)
for file in /var/opt/gitlab/backups/*.tar;do
  if [ $file != $LAST_CONFIG_BACKUP ];then
    rm $file
  fi
done

# GitLab SSH Backup Rentention
LAST_SSH_BACKUP=$(cat /var/opt/gitlab/backups_ssh/.last_backup)
for file in /var/opt/gitlab/backups_ssh/*.tar;do
  if [ $file != $LAST_SSH_BACKUP ];then
    rm $file
  fi
done

EOF

Apply Appropriate File Permission

sudo chmod u+x /usr/bin/gitlab-unmanaged-retention

Create Cron Task

Running Backup at Midnight as a Daily Backup and Runing Rentention at 3.00 AM as a Daily Task
Terminal window
```
sudo crontab -e
```
```
0 0 * * * /usr/bin/gitlab-unmanaged-backup

0 3 * * 0 /usr/bin/gitlab-unmanaged-retention
```

Retention on Remote Storage

As we backup to S3(-compatible) Storage, Set Bucket Object Lifecycle to be 7 Days for Backup Retention