diff --git a/.wordlist.txt b/.wordlist.txt index 9c7e135..a634c07 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -3,7 +3,9 @@ Azurite BarmanObjectStore BarmanObjectStoreConfiguration BarmanObjectStores +CRD CloudNativePG +DataBackupConfiguration DigitalOcean Docusaurus EKS @@ -24,17 +26,24 @@ ObjectStore ObjectStoreSpec ObjectStoreStatus PITR +PoR +PostgreSQL +Postgres README +RPO +RTO RecoveryWindow RetentionPolicy SAS SFO SPDX +SPDX ServerRecoveryWindow TLS TODO Uncomment WAL +WALBackupConfiguration WALs api apiVersion @@ -46,6 +55,7 @@ backend backends barmanObjectName barmancloud +bzip cd cloudnative clusterrole @@ -64,6 +74,7 @@ externalClusters gcs gf github +gzip hostpath http https @@ -89,6 +100,7 @@ rc recoverability repos retentionCheckInterval +retentionPolicy rolebinding sc secretKeyRef diff --git a/web/README.md b/web/README.md index d939a7f..77a8930 100644 --- a/web/README.md +++ b/web/README.md @@ -3,6 +3,13 @@ This website is built using [Docusaurus](https://docusaurus.io/), a modern static website generator. +### Requirements + +- Docker +- [Yarn](https://yarnpkg.com/) +- [Dagger](https://dagger.io/) +- [Task](https://taskfile.dev/) + ### Installation ```shell @@ -35,6 +42,14 @@ $ yarn serve By default, this will load your site at http://localhost:3000/. +### Spellchecking + +From the top directory: + +```shell +task spellcheck +``` + ### Versioning Docusaurus allows versioning of the documentation to maintain separate sets of diff --git a/web/docs/compression.md b/web/docs/compression.md new file mode 100644 index 0000000..2abbede --- /dev/null +++ b/web/docs/compression.md @@ -0,0 +1,43 @@ +--- +sidebar_position: 80 +--- + +# Compression + + + +By default, backups and WAL files are archived **uncompressed**. However, the +Barman Cloud Plugin supports multiple compression algorithms via +`barman-cloud-backup` and `barman-cloud-wal-archive`, allowing you to optimize +for space, speed, or a balance of both. + +### Supported Compression Algorithms + +- `bzip2` +- `gzip` +- `lz4` (WAL only) +- `snappy` +- `xz` (WAL only) +- `zstd` (WAL only) + +Compression settings for base backups and WAL archives are configured +independently. For implementation details, refer to the corresponding API +definitions: + +- [`DataBackupConfiguration`](https://pkg.go.dev/github.com/cloudnative-pg/barman-cloud/pkg/api#DataBackupConfiguration) +- [`WALBackupConfiguration`](https://pkg.go.dev/github.com/cloudnative-pg/barman-cloud/pkg/api#WalBackupConfiguration) + +:::important +Compression impacts both performance and storage efficiency. Choose the right +algorithm based on your recovery time objectives (RTO), storage capacity, and +network throughput. +::: + +## Compression Benchmark (on MinIO) + +| Compression | Backup Time (ms) | Restore Time (ms) | Uncompressed Size (MB) | Compressed Size (MB) | Ratio | +| ----------- | ---------------- | ----------------- | ---------------------- | -------------------- | ----- | +| None | 10,927 | 7,553 | 395 | 395 | 1.0:1 | +| bzip2 | 25,404 | 13,886 | 395 | 67 | 5.9:1 | +| gzip | 116,281 | 3,077 | 395 | 91 | 4.3:1 | +| snappy | 8,134 | 8,341 | 395 | 166 | 2.4:1 | diff --git a/web/docs/concepts.md b/web/docs/concepts.md new file mode 100644 index 0000000..42249ed --- /dev/null +++ b/web/docs/concepts.md @@ -0,0 +1,173 @@ +--- +sidebar_position: 10 +--- + +# Main Concepts + + + +:::important +Before proceeding, make sure to review the following sections of the +CloudNativePG documentation: + +- [**Backup**](https://cloudnative-pg.io/documentation/current/backup/) +- [**WAL Archiving**](https://cloudnative-pg.io/documentation/current/wal_archiving/) +- [**Recovery**](https://cloudnative-pg.io/documentation/current/recovery/) +::: + +The **Barman Cloud Plugin** enables **hot (online) backups** of PostgreSQL +clusters in CloudNativePG through [`barman-cloud`](https://pgbarman.org), +supporting continuous physical backups and WAL archiving to an **object +store**—without interrupting write operations. + +It also supports both **full recovery** and **Point-in-Time Recovery (PITR)** +of a PostgreSQL cluster. + +## The Object Store + +At the core is the [`ObjectStore` custom resource (CRD)](plugin-barman-cloud.v1.md#objectstorespec), +which acts as the interface between the PostgreSQL cluster and the target +object storage system. It allows you to configure: + +- **Authentication and bucket location** via the `.spec.configuration` section +- **WAL archiving** settings—such as compression type, parallelism, and + server-side encryption—under `.spec.configuration.wal` +- **Base backup options**—with similar settings for compression, concurrency, + and encryption—under `.spec.configuration.data` +- **Retention policies** to manage the life-cycle of archived WALs and backups + via `.spec.configuration.retentionPolicy` + +WAL files are archived in the `wals` directory, while base backups are stored +as **tarballs** in the `base` directory, following the +[Barman Cloud convention](https://docs.pgbarman.org/cloud/latest/usage/#object-store-layout). + +:::tip +For details, refer to the +[API reference for the `ObjectStore` resource](plugin-barman-cloud.v1.md#objectstorespec). +::: + +## Integration with a CloudNativePG Cluster + +CloudNativePG can delegate continuous backup and recovery responsibilities to +the **Barman Cloud Plugin** by configuring the `.spec.plugins` section of a +`Cluster` resource. This setup requires a corresponding `ObjectStore` resource +to be defined. + +:::important +While it is technically possible to reuse the same `ObjectStore` for multiple +`Cluster` resources within the same namespace, it is strongly recommended to +dedicate one object store per PostgreSQL cluster to ensure data isolation and +operational clarity. +::: + +The following example demonstrates how to configure a CloudNativePG cluster +named `cluster-example` to use a previously defined `ObjectStore` (also named +`cluster-example`) in the same namespace. Setting `isWALArchiver: true` enables +WAL archiving through the plugin: + +```yaml +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: cluster-example +spec: + # Other cluster settings... + plugins: + - name: barman-cloud.cloudnative-pg.io + isWALArchiver: true + parameters: + barmanObjectName: cluster-example +``` + +## Backup of a Postgres Cluster + +Once the object store is defined and the `Cluster` is configured to use the +Barman Cloud Plugin, **WAL archiving is activated immediately** on the +PostgreSQL primary. + +Physical base backups are seamlessly managed by CloudNativePG using the +`Backup` and `ScheduledBackup` resources, respectively for +[on-demand](https://cloudnative-pg.io/documentation/current/backup/#on-demand-backups) +and +[scheduled](https://cloudnative-pg.io/documentation/current/backup/#scheduled-backups) +backups. + +To use the Barman Cloud Plugin, you must set the `method` to `plugin` and +configure the `pluginConfiguration` section as shown: + +```yaml +[...] +spec: + method: plugin + pluginConfiguration: + name: barman-cloud.cloudnative-pg.io + [...] +``` + +With this configuration, CloudNativePG supports: + +- Backups from both **primary** and **standby** instances +- Backups from **designated primaries** in a distributed topology using + [replica clusters](https://cloudnative-pg.io/documentation/current/replica_cluster/) + +:::tip +For details on how to back up from a standby, refer to the official documentation: +[Backup from a standby](https://cloudnative-pg.io/documentation/current/backup/#backup-from-a-standby). +::: + +:::important +Both backup and WAL archiving operations are executed by sidecar containers +running in the same pod as the PostgreSQL `Cluster` primary instance—except +when backups are taken from a standby, in which case the sidecar runs alongside +the standby pod. +The sidecar containers use a [dedicated container image](images.md) that +includes only the supported version of Barman Cloud. +::: + +## Recovery of a Postgres Cluster + +In PostgreSQL, *recovery* refers to the process of starting a database instance +from an existing backup. The Barman Cloud Plugin integrates with CloudNativePG +to support both **full recovery** and **Point-in-Time Recovery (PITR)** from an +object store. + +Recovery in this context is *not in-place*: it bootstraps a brand-new +PostgreSQL cluster from a backup and replays the necessary WAL files to reach +the desired recovery target. + +To perform a recovery, define an *external cluster* that references the +appropriate `ObjectStore`, and use it as the source in the `bootstrap` section +of the target cluster: + +```yaml +[...] +spec: + [...] + bootstrap: + recovery: + source: source + externalClusters: + - name: source + plugin: + name: barman-cloud.cloudnative-pg.io + parameters: + barmanObjectName: cluster-example + serverName: cluster-example + [...] +``` + +The critical element here is the `externalClusters` section of the `Cluster` +resource, where the `plugin` stanza instructs CloudNativePG to use the Barman +Cloud Plugin to access the object store for recovery. + +This same mechanism can be used for a variety of scenarios enabled by the +CloudNativePG API, including: + +* **Full cluster recovery** from the latest backup +* **Point-in-Time Recovery (PITR)** +* Bootstrapping **replica clusters** in a distributed topology + +:::tip +For complete instructions and advanced use cases, refer to the official +[Recovery documentation](https://cloudnative-pg.io/documentation/current/recovery/). +::: diff --git a/web/docs/features.md b/web/docs/features.md deleted file mode 100644 index 04f70a9..0000000 --- a/web/docs/features.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -sidebar_position: 2 ---- - -# Features - -This plugin enables continuous backup to object storage for a PostgreSQL -cluster using the [barman-cloud](https://pgbarman.org/) tool suite. - -The features provided by this plugin are: - -- Data Directory Backup -- Data Directory Restore -- WAL Archiving -- WAL Restoring -- Point-in-Time Recovery (PITR) -- Replica Clusters - -This plugin is compatible with all object storage services supported by -barman-cloud, including: - -- Amazon AWS S3 -- Google Cloud Storage -- Microsoft Azure Blob Storage - -The following storage solutions have been tested and confirmed to work with -this implementation: - -- [MinIO](https://min.io/) – An S3-compatible object storage solution. -- [Azurite](https://github.com/Azure/Azurite) – A simulator for Microsoft Azure Blob Storage. -- [fake-gcs-server](https://github.com/fsouza/fake-gcs-server) – A simulator for Google Cloud Storage. - -Backups created with in-tree object store support can be restored using this -plugin, ensuring compatibility and reliability across environments. - diff --git a/web/docs/images.md b/web/docs/images.md new file mode 100644 index 0000000..ed1bdbb --- /dev/null +++ b/web/docs/images.md @@ -0,0 +1,9 @@ +--- +sidebar_position: 40 +--- + +# Container Images + + + +TODO diff --git a/web/docs/installation.md b/web/docs/installation.md index 88cba3a..323e9af 100644 --- a/web/docs/installation.md +++ b/web/docs/installation.md @@ -1,23 +1,28 @@ --- -sidebar_position: 4 +sidebar_position: 20 --- # Installation -**IMPORTANT NOTES:** + -1. The plugin **must** be installed in the same namespace where the operator is - installed (typically `cnpg-system`). +:::important +1. The plugin **must** be installed in the same namespace as the CloudNativePG + operator (typically `cnpg-system`). -2. Be aware that the operator's **listening namespaces** may differ from its - installation namespace. Ensure you verify this distinction to avoid - configuration issues. +2. Keep in mind that the operator's **listening namespaces** may differ from its + installation namespace. Double-check this to avoid configuration issues. +::: -Here’s an enhanced version of your instructions for verifying the prerequisites: +## Verifying the Requirements -## Step 1 - Verify the Prerequisites +Before installing the plugin, make sure the [requirements](intro.md#requirements) are met. -If CloudNativePG is installed in the default `cnpg-system` namespace, verify its version using the following command: +### CloudNativePG Version + +Ensure you're running a version of CloudNativePG that is compatible with the +plugin. If installed in the default `cnpg-system` namespace, you can verify the +version with: ```sh kubectl get deployment -n cnpg-system cnpg-controller-manager -o yaml \ @@ -30,10 +35,12 @@ Example output: image: ghcr.io/cloudnative-pg/cloudnative-pg:1.26.0 ``` -Ensure that the version displayed is **1.26** or newer. +The version **must be 1.26 or newer**. -Then, use the [cmctl](https://cert-manager.io/docs/reference/cmctl/#installation) -tool to confirm that `cert-manager` is correctly installed: +### cert-manager + +Use the [cmctl](https://cert-manager.io/docs/reference/cmctl/#installation) +tool to confirm that `cert-manager` is installed and available: ```sh cmctl check api @@ -45,17 +52,20 @@ Example output: The cert-manager API is ready ``` -Both checks are necessary to proceed with the installation. +Both checks are required before proceeding with the installation. -## Step 2 - Install the barman-cloud Plugin +## Installing the Barman Cloud Plugin -Use `kubectl` to apply the manifest for the latest commit in the `main` branch: +Install the plugin using `kubectl` by applying the manifest for the latest +release: + ```sh kubectl apply -f \ https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.3.0/manifest.yaml ``` + Example output: @@ -80,8 +90,7 @@ certificate.cert-manager.io/barman-cloud-server created issuer.cert-manager.io/selfsigned-issuer created ``` -After these steps, the plugin will be successfully installed. Make sure it is -ready to use by checking the deployment status as follows: +Finally, check that the deployment is up and running: ```sh kubectl rollout status deployment \ @@ -94,4 +103,4 @@ Example output: deployment "barman-cloud" successfully rolled out ``` -This confirms that the plugin is deployed and operational. +This confirms that the plugin is deployed and ready to use. diff --git a/web/docs/intro.md b/web/docs/intro.md new file mode 100644 index 0000000..9e2eb5c --- /dev/null +++ b/web/docs/intro.md @@ -0,0 +1,60 @@ +--- +sidebar_position: 1 +sidebar_label: "Introduction" +--- + +# Barman Cloud Plugin + + + +The **Barman Cloud Plugin** for [CloudNativePG](https://cloudnative-pg.io/) +enables online continuous physical backups of PostgreSQL clusters to object storage +using the `barman-cloud` suite from the [Barman](https://docs.pgbarman.org/release/latest/) +project. + +## Requirements + +To use the Barman Cloud Plugin, you need: + +- [CloudNativePG](https://cloudnative-pg.io) version **1.26** +- [cert-manager](https://cert-manager.io/) to enable TLS communication between + the plugin and the operator + +## Key Features + +This plugin provides the following capabilities: + +- Physical online backup of the data directory +- Physical restore of the data directory +- Write-Ahead Log (WAL) archiving +- WAL restore +- Full cluster recovery +- Point-in-Time Recovery (PITR) +- Seamless integration with replica clusters for bootstrap and WAL restore from archive + +:::important +The Barman Cloud Plugin is designed to **replace the in-tree object storage support** +previously provided via the `.spec.backup.barmanObjectStore` section in the +`Cluster` resource. +Backups created using the in-tree approach are fully supported and compatible +with this plugin. +::: + +## Supported Object Storage Providers + +The plugin works with all storage backends supported by `barman-cloud`, including: + +- **Amazon S3** +- **Google Cloud Storage** +- **Microsoft Azure Blob Storage** + +In addition, the following S3-compatible and simulator solutions have been +tested and verified: + +- [MinIO](https://min.io/) – An S3-compatible storage solution +- [Azurite](https://github.com/Azure/Azurite) – A simulator for Azure Blob Storage +- [fake-gcs-server](https://github.com/fsouza/fake-gcs-server) – A simulator for Google Cloud Storage + +:::tip +For more details, refer to [Object Store Providers](object_stores.md). +::: diff --git a/web/docs/migration.md b/web/docs/migration.md new file mode 100644 index 0000000..481705d --- /dev/null +++ b/web/docs/migration.md @@ -0,0 +1,9 @@ +--- +sidebar_position: 40 +--- + +# Migrating from CloudNativePG + + + +TODO diff --git a/web/docs/misc.md b/web/docs/misc.md new file mode 100644 index 0000000..4d3cefc --- /dev/null +++ b/web/docs/misc.md @@ -0,0 +1,76 @@ +--- +sidebar_position: 90 +--- + +# Miscellaneous + + + +## Backup Object Tagging + +You can attach key-value metadata tags to backup artifacts—such as base +backups, WAL files, and history files—via the `.spec.configuration` section of +the `ObjectStore` resource. + +- `tags`: applied to base backups and WAL files +- `historyTags`: applied to history files only + +### Example + +```yaml +apiVersion: barmancloud.cnpg.io/v1 +kind: ObjectStore +metadata: + name: my-store +spec: + configuration: + [...] + tags: + backupRetentionPolicy: "expire" + historyTags: + backupRetentionPolicy: "keep" + [...] +``` + +## Extra Options for Backup and WAL Archiving + +You can pass additional command-line arguments to `barman-cloud-backup` and +`barman-cloud-wal-archive` using the `additionalCommandArgs` field in the +`ObjectStore` configuration. + +- `.spec.configuration.data.additionalCommandArgs`: for `barman-cloud-backup` +- `.spec.configuration.wal.additionalCommandArgs`: for `barman-cloud-wal-archive` + +Each field accepts a list of string arguments. If an argument is already +configured elsewhere in the plugin, the duplicate will be ignored. + +### Example: Extra Backup Options + +```yaml +kind: ObjectStore +metadata: + name: my-store +spec: + configuration: + data: + additionalCommandArgs: + - "--min-chunk-size=5MB" + - "--read-timeout=60" +``` + +### Example: Extra WAL Archive Options + +```yaml +kind: ObjectStore +metadata: + name: my-store +spec: + configuration: + wal: + additionalCommandArgs: + - "--max-concurrency=1" + - "--read-timeout=60" +``` + +For a complete list of supported options, refer to the +[official Barman Cloud documentation](https://docs.pgbarman.org/release/latest/). diff --git a/web/docs/object_stores.md b/web/docs/object_stores.md index 7e2e735..9ca5a2a 100644 --- a/web/docs/object_stores.md +++ b/web/docs/object_stores.md @@ -1,5 +1,5 @@ --- -sidebar_position: 99 +sidebar_position: 50 --- # Object Store Providers @@ -359,7 +359,7 @@ write permissions to the bucket. ## MinIO Gateway MinIO Gateway can proxy requests to cloud object storage providers like S3 or GCS. -For more information, please refer to [MinIO official documentation](https://docs.min.io/). +For more information, refer to [MinIO official documentation](https://docs.min.io/). ### Setup diff --git a/web/docs/prerequisites.md b/web/docs/prerequisites.md deleted file mode 100644 index afa181e..0000000 --- a/web/docs/prerequisites.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -sidebar_position: 3 ---- - -# Prerequisites - -To use this plugin, ensure the following prerequisites are met: - -- [**CloudNativePG**](https://cloudnative-pg.io) version **1.25** or newer. -- [**cert-manager**](https://cert-manager.io/) for enabling **TLS communication** between the plugin and the operator. diff --git a/web/docs/retention.md b/web/docs/retention.md new file mode 100644 index 0000000..fefbd08 --- /dev/null +++ b/web/docs/retention.md @@ -0,0 +1,38 @@ +--- +sidebar_position: 60 +--- + +# Retention Policies + + + +The Barman Cloud Plugin supports **automated cleanup of obsolete backups** via +retention policies, configured in the `.spec.retentionPolicy` field of the +`ObjectStore` resource. + +:::note +This feature uses the `barman-cloud-backup-delete` command with the +`--retention-policy "RECOVERY WINDOW OF {{ value }} {{ unit }}"` syntax. +::: + +#### Example: 30-Day Retention Policy + +```yaml +apiVersion: barmancloud.cnpg.io/v1 +kind: ObjectStore +metadata: + name: my-store +spec: + [...] + retentionPolicy: "30d" +```` + +:::note +A **recovery window retention policy** ensures the cluster can be restored to +any point in time between the calculated *Point of Recoverability* (PoR) and +the latest WAL archive. The PoR is defined as `current time - recovery window`. +The **first valid backup** is the most recent backup completed before the PoR. +Backups older than that are marked as *obsolete* and deleted after the next +backup completes. +::: + diff --git a/web/docs/usage.md b/web/docs/usage.md index 61483bd..3c22b2a 100644 --- a/web/docs/usage.md +++ b/web/docs/usage.md @@ -1,14 +1,28 @@ --- -sidebar_position: 5 +sidebar_position: 30 --- -# Usage +# Using the Barman Cloud Plugin -## Defining the `BarmanObjectStore` + -A `BarmanObjectStore` object should be created for each object store used in -your PostgreSQL architecture. Below is an example configuration for using -MinIO: +After [installing the plugin](installation.md) in the same namespace as the +CloudNativePG operator, enabling your PostgreSQL cluster to use the Barman +Cloud Plugin involves just a few steps: + +- Defining the object store containing your WAL archive and base backups, using + your preferred [provider](object_stores.md) +- Instructing the Postgres cluster to use the Barman Cloud Plugin + +From that moment, you’ll be able to issue on-demand backups or define a backup +schedule, as well as rely on the object store for recovery operations. + +The rest of this page details each step, using MinIO as object store provider. + +## Defining the `ObjectStore` + +An `ObjectStore` resource must be created for each object store used in your +PostgreSQL architecture. Here's an example configuration using MinIO: ```yaml apiVersion: barmancloud.cnpg.io/v1 @@ -30,16 +44,15 @@ spec: compression: gzip ``` -The `.spec.configuration` API follows the same schema as the +The `.spec.configuration` schema follows the same format as the [in-tree barman-cloud support](https://pkg.go.dev/github.com/cloudnative-pg/barman-cloud/pkg/api#BarmanObjectStoreConfiguration). Refer to [the CloudNativePG documentation](https://cloudnative-pg.io/documentation/preview/backup_barmanobjectstore/) -for detailed usage. +for additional details. ## Configuring WAL Archiving -Once the `BarmanObjectStore` is defined, you can configure a PostgreSQL cluster -to archive WALs by referencing the store in the `.spec.plugins` section, as -shown below: +Once the `ObjectStore` is defined, you can configure your PostgreSQL cluster +to archive WALs by referencing the store in the `.spec.plugins` section: ```yaml apiVersion: postgresql.cnpg.io/v1 @@ -62,9 +75,8 @@ This configuration enables both WAL archiving and data directory backups. ## Performing a Base Backup -Once WAL archiving is enabled, the cluster is ready for backups. To create a -backup, configure the `backup.spec.pluginConfiguration` section to specify this -plugin: +Once WAL archiving is enabled, the cluster is ready for backups. To issue an +on-demand backup, use the following configuration with the plugin method: ```yaml apiVersion: postgresql.cnpg.io/v1 @@ -72,13 +84,17 @@ kind: Backup metadata: name: backup-example spec: - method: plugin cluster: name: cluster-example + method: plugin pluginConfiguration: name: barman-cloud.cloudnative-pg.io ``` +:::note +You can apply the same concept to the `ScheduledBackup` resource. +::: + ## Restoring a Cluster To restore a cluster from an object store, create a new `Cluster` resource that @@ -106,8 +122,9 @@ spec: size: 1Gi ``` -**NOTE:** The above configuration does **not** enable WAL archiving for the -restored cluster. +:::important +The above configuration does **not** enable WAL archiving for the restored cluster. +::: To enable WAL archiving for the restored cluster, include the `.spec.plugins` section alongside the `externalClusters.plugin` section, as shown below: diff --git a/web/docs/welcome.md b/web/docs/welcome.md deleted file mode 100644 index 5c88ae6..0000000 --- a/web/docs/welcome.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -sidebar_position: 1 -sidebar_label: "Welcome" ---- - -# Barman Cloud CNPG-I plugin - -**Status:** EXPERIMENTAL - -Welcome to the documentation of the [barman-cloud](https://pgbarman.org/) CNPG-I -plugin for [CloudNativePG](https://cloudnative-pg.io/). - -Test