Archive() and the restore job hook each re-derived, on their own,
whether to verify the WAL archive destination is empty, by reading a
Cluster annotation and, for Archive, an on-disk marker file. That
decision belongs to the operator, which already tracks both the
annotation and the marker file's lifecycle.
Honor cnpg-i's new WALArchiveRequest/RestoreRequest field
CheckEmptyWalArchive when the operator sets it: obey it directly,
without re-inspecting the marker file. Only fall back to the previous
annotation-and-marker-file logic when talking to an operator that
predates this field.
Related: cloudnative-pg/cnpg-i#353 adds the field this depends on;
cloudnative-pg/cloudnative-pg#11216 is the operator-side counterpart.
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
When a demoted primary rejoins the cluster after a failover, the
instance manager runs `pg_rewind --restore-target-wal`, and the
resulting `restore_command` invocations are served by this sidecar
exactly as if an instance in recovery were asking. pg_rewind walks the
timeline backwards, fetches every WAL file it needs exactly once, and
treats any restore failure as fatal, so two optimizations meant for
recovery break it: prefetching the following segments is wasted work
that, past the end of the timeline, is guaranteed to end in an archive
miss, and the end-of-wal-stream flag recorded by that miss makes a later
invocation fail without contacting the object store, on a segment the
archive actually has, aborting the whole rewind.
The restore request now carries the context it is made in
(cloudnative-pg/cnpg-i#351). When it says `MODE_REWIND`, the sidecar
restores exactly the requested file: no prefetching, no
end-of-wal-stream flag check, no flag recording. Requests from operators
predating the field keep the current behavior.
This is the plugin-side counterpart of
cloudnative-pg/cloudnative-pg#11204. The cnpg-i dependency points to a
pseudo-version of the protocol pull request and will move to the next
tagged release once it is available.
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
During a replica cluster failover, the designated primary could
incorrectly attempt to restore WALs from its own object store instead of
the replica source, causing recovery to fail. This happened because the
previous logic relied on IsReplica() returning true, but that flag can
already be false while PostgreSQL is still in recovery and needs WALs
from the source cluster.
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Map each restorer sentinel to the gRPC status that best reflects whether
the operator should retry:
- ErrWALNotFound -> NotFound (terminal)
- ErrInvalidWalName -> InvalidArgument (terminal)
- ErrConnectivity -> Unavailable (retry)
- ErrGeneric -> Unavailable (retry)
- anything else -> Internal (terminal)
Previously every non-NotFound failure was returned verbatim, leaving the
operator unable to tell transient blips apart from terminal conditions
like a malformed WAL name.
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Tao Li <tao.li@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Co-authored-by: Tao Li <tao.li@enterprisedb.com>
When the end of the WAL stream is reached, the parallel WAL restore
feature attempts to predict the names of subsequent WAL files to restore
and records the first missing WAL file.
On high-availability (HA) replicas, if PostgreSQL requests the first
missing WAL file, the code returns an error status that prompts
PostgreSQL to switch to streaming replication.
Currently, the code assumes a `wal_segment_size` of 16MB for predicting
the next WAL file names. If the configured WAL segment size exceeds
16MB, it may request non-existent WAL files. For instance, with 16MB
segments, the names would range from `000000010000000100000000` to
`0000000100000001000000FF` before moving to the next segment. For 1GB
segments, they would range from `000000010000000100000000` to
`000000010000000100000003`.
With the assumption of a 16MB segment size, the code will not find the
WALs from `000000010000000100000004` to `0000000100000001000000FF`.
While this assumption does not affect HA replicas - which can shift to
streaming mode - it's problematic for a PostgreSQL instance seeking
consistency after a restore, as the restore process will fail.
This patch disables end-of-wal file marker management during
replication, addressing restore issues for backups that were:
1. using a custom WAL file segment size
2. utilizing parallel WAL recovery
3. initiated on one WAL segment and concluded on a different one
Fixes: #603
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
The plugin now returns a 404 status code when a requested WAL file does
not exist in the object store.
This prevents misleading log entries such as "Error while handling gRPC
request" for expected missing-file scenarios.
The `ErrEndOfWALStreamReached` now returns `OutOfRange` error.
The `ErrMissingPermissions` now returns `FailedPrecondition` error.
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
In the in-tree barman-cloud implementation, the check for an empty WAL
archive is performed both immediately after the restore process and when the
first WAL file is archived.
Previously, the plugin-based implementation only performed this check after
restore, skipping it during archiving of the first WAL. This patch restores
parity with the in-tree behavior by ensuring the check is also performed
during WAL archiving.
Closes: #457
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
This patch enables the use of custom CA certificates when connecting
to the object store in the barman-cloud plugin. The certificates are
injected into the sidecar via a projected volume and used by the
barman-cloud tool suite.
If the barman object name or the key name changes, users must trigger
a Pod rollout to apply the new values.
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
