Barman Cloud CNPG-I plugin

Welcome to the codebase of the barman-cloud CNPG-I plugin for CloudNativePG.

Table of contents

• Features
• Prerequisites
• Installation
• Usage
  • WAL Archiving
  • Backup
  • Restore
  • Replica clusters

Features

This plugin enables continuous backup to object storage for a PostgreSQL cluster using the barman-cloud tool suite.

The features provided by this plugin are:

• Data Directory Backup
• Data Directory Restore
• WAL Archiving
• WAL Restoring
• Point-in-Time Recovery (PITR)
• Replica Clusters

This plugin is compatible with all object storage services supported by barman-cloud, including:

• Amazon AWS S3
• Google Cloud Storage
• Microsoft Azure Blob Storage

The following storage solutions have been tested and confirmed to work with this implementation:

• MinIO – An S3-compatible object storage solution.
• Azurite – A simulator for Microsoft Azure Blob Storage.
• fake-gcs-server – A simulator for Google Cloud Storage.

Backups created with in-tree object store support can be restored using this plugin, ensuring compatibility and reliability across environments.

Prerequisites

To use this plugin, ensure the following prerequisites are met:

• CloudNativePG version 1.25 or newer.
• cert-manager for enabling TLS communication between the plugin and the operator.

Installation

Important Notes:

1. The plugin must be installed in the same namespace where the operator is installed (typically cnpg-system).

2. Be aware that the operator's listening namespaces may differ from its installation namespace. Ensure you verify this distinction to avoid configuration issues.

Here’s an enhanced version of your instructions for verifying the prerequisites:

Step 1 - Verify the Prerequisites

If CloudNativePG is installed in the default cnpg-system namespace, verify its version using the following command:

kubectl get deployment -n cnpg-system cnpg-controller-manager \
  | grep ghcr.io/cloudnative-pg/cloudnative-pg

Example output:

image: ghcr.io/cloudnative-pg/cloudnative-pg:1.25.0-rc1

Ensure that the version displayed is 1.25 or newer.

Then, use the cmctl tool to confirm that cert-manager is correctly installed:

cmctl check api

Example output:

The cert-manager API is ready

Both checks are necessary to proceed with the installation.

Step 2 - install the barman-cloud plugin

TODO: temporary section - to be rewritten when manifests are available

The plugin can be installed via its manifest:

# Download the plugin-barman-cloud codebase, including its manifest
$ curl -Lo plugin-barman-cloud.tgz https://api.github.com/repos/cloudnative-pg/plugin-barman-cloud/tarball/main

# Expand it in a temporary folder (this can be deleted after the plugin is installed)
$ tar xvzf plugin-barman-cloud.tgz

# From now on, the proposed commands are supposed to be invoked from
# the repository root directory
$ cd cloudnative-pg-plugin-barman-cloud*

# Apply the manifest for the latest commit in the `main` branch
$ kubectl apply -k kubernetes/

customresourcedefinition.apiextensions.k8s.io/objectstores.barmancloud.cnpg.io created
serviceaccount/plugin-barman-cloud created
role.rbac.authorization.k8s.io/leader-election-role created
clusterrole.rbac.authorization.k8s.io/metrics-auth-role created
clusterrole.rbac.authorization.k8s.io/metrics-reader created
clusterrole.rbac.authorization.k8s.io/objectstore-editor-role created
clusterrole.rbac.authorization.k8s.io/objectstore-viewer-role created
clusterrole.rbac.authorization.k8s.io/plugin-barman-cloud created
rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/metrics-auth-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/plugin-barman-cloud-binding created
secret/plugin-barman-cloud-8tfddg42gf created
service/barman-cloud created
deployment.apps/barman-cloud configured
certificate.cert-manager.io/barman-cloud-client created
certificate.cert-manager.io/barman-cloud-server created
issuer.cert-manager.io/selfsigned-issuer created

Once the plugin is installed, the following command will wait until the plugin is ready to be used:

$ kubectl rollout status deployment -n cnpg-system barman-cloud

deployment "barman-cloud" successfully rolled out

Usage

The BarmanObjectStore object

A BarmanObjectStore object should be created for each object stored used by the PostgreSQL architecture. This is an example:

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: minio-store
spec:
  configuration:
    destinationPath: s3://backups/
    endpointURL: http://minio:9000
    s3Credentials:
      accessKeyId:
        name: minio
        key: ACCESS_KEY_ID
      secretAccessKey:
        name: minio
        key: ACCESS_SECRET_KEY
    wal:
      compression: gzip

The objectstore.spec.configuration API is the same api used by the in-tree barman-cloud support and can be used like discussed in the relative documentation page.

WAL Archiving

Once the BarmanObjectStore has been defined, a cluster using it to archive WALs will reference it in the .spec.plugins section, like in the following example:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-example
spec:
  instances: 3
  imagePullPolicy: Always
  plugins:
  - name: barman-cloud.cloudnative-pg.io
    parameters:
      barmanObjectName: minio-store

  storage:
    size: 1Gi

This will enable WAL archiving and data directory backups, as discussed in the next section.

Backup

Once the transaction log files are being archived, the cluster is ready to be backed up.

To request a backup, the backup.spec.pluginConfiguration stanza must be set with the name of this plugin like in the following example:

apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: backup-example
spec:
  method: plugin

  cluster:
    name: cluster-example

  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

Restore

To recover a cluster from an object store, the user should create a new Cluster resource referring to the object store containing the backup, like in the following example:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-restore
spec:
  instances: 3
  imagePullPolicy: IfNotPresent

  bootstrap:
    recovery:
      source: source

  externalClusters:
  - name: source
    plugin:
      name: barman-cloud.cloudnative-pg.io
      parameters:
        barmanObjectName: minio-store
        serverName: cluster-example

  storage:
    size: 1Gi

IMPORTANT recovering a cluster like in the previous example do not enable WAL archiving for the cluster being recovered.

The latter can be configured by combining the .spec.plugins section with the externalClusters.plugin section, like in the following example:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-restore
spec:
  instances: 3
  imagePullPolicy: IfNotPresent

  bootstrap:
    recovery:
      source: source

  plugins:
  - name: barman-cloud.cloudnative-pg.io
    parameters:
      barmanObjectName: minio-store-bis

  externalClusters:
  - name: source
    plugin:
      name: barman-cloud.cloudnative-pg.io
      parameters:
        barmanObjectName: minio-store
        serverName: cluster-example

  storage:
    size: 1Gi

The object store that is used to archive the transaction log may be the same object store that is being used to restore a cluster or a different one.

Replica clusters

The previous definition can be combined to setup a distributed topology using the .spec.replica section like in the following example:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-dc-a
spec:
  instances: 3
  primaryUpdateStrategy: unsupervised

  storage:
    storageClass: csi-hostpath-sc
    size: 1Gi

  plugins:
  - name: barman-cloud.cloudnative-pg.io
    parameters:
      barmanObjectName: minio-store-a

  replica:
    self: cluster-dc-a
    primary: cluster-dc-a
    source: cluster-dc-b

  externalClusters:
  - name: cluster-dc-a
    plugin:
      name: barman-cloud.cloudnative-pg.io
      parameters:
        barmanObjectName: minio-store-a

  - name: cluster-dc-b
    plugin:
      name: barman-cloud.cloudnative-pg.io
      parameters:
        barmanObjectName: minio-store-b