Backup and restore

Everything Bookend knows lives in three places. Back up all three together; a restore is only consistent when they come from the same point in time.

What	Where	Why
Database	dbdata volume (bundled Postgres) or the bank’s instance	loans, extracted values, findings, approvals, the evidence chain, settings (secrets encrypted), scheduler state, audit log, refresh tokens
Documents	documents volume (/data/documents, content-addressed by SHA-256) and exports (/data/exports: boarding files, wire requests)	the PDFs every evidence link hashes; a chain verifies only if the bytes are present
Master key and env	deploy/.env (BOOKEND_MASTER_KEY, database passwords) and your compose override files	without the master key the encrypted settings, including the RS256 signing key, cannot be read

The commands below are for the bundled Postgres and the production override from the Installation Guide. Volume names carry the Compose project prefix (bookend_documents, bookend_exports). Replace bookend in -U bookend -d bookend if you changed POSTGRES_USER or POSTGRES_DB.

Backup (nightly, and before every upgrade)

DC="docker compose -f deploy/compose.yml -f deploy/compose.production.yml"
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p backup
# 1. Database: custom-format dump as the owner. Roles are not in a pg_dump; the application role is recreated
#    by hand only when restoring onto a new database server (see Restore).
$DC exec -T db pg_dump -U bookend -d bookend -Fc > backup/bookend-$STAMP.dump
# 2. Document and export volumes: tar from a throwaway container.
docker run --rm -v bookend_documents:/data/documents:ro -v bookend_exports:/data/exports:ro -v "$PWD/backup":/backup alpine \
  tar czf /backup/volumes-$STAMP.tgz /data/documents /data/exports
# 3. Environment: copy deploy/.env and the override files into the bank's secret store (never into the same share as the dump).

With an external database, use the bank’s native backup for step 1 (Postgres pg_dump or point-in-time recovery, SQL Server full and log backups) and keep steps 2 and 3. Keep at least the retention the bank’s records policy requires (documents.retention_days, default 2,555 days, about seven years).

Verify a backup by restoring it into a scratch stack at least quarterly. The drill procedure, the measured timings and the availability posture are in resilience.md.

Restore

DC="docker compose -f deploy/compose.yml -f deploy/compose.production.yml"
$DC down                                              # keeps volumes
$DC up -d db --wait
$DC exec -T db psql -U bookend -d postgres -c "DROP DATABASE bookend;" -c "CREATE DATABASE bookend OWNER bookend;"
# Only on a new database server or an empty dbdata volume, where the application role does not exist yet:
#   $DC exec -T db psql -U bookend -d postgres -c "CREATE ROLE bookend_app LOGIN PASSWORD '<DB_APP_PASSWORD>';"
$DC exec -T db pg_restore -U bookend -d bookend --no-owner < backup/bookend-<stamp>.dump
docker run --rm -v bookend_documents:/data/documents -v bookend_exports:/data/exports -v "$PWD/backup":/backup alpine \
  sh -c "rm -rf /data/documents/* /data/exports/* && tar xzf /backup/volumes-<stamp>.tgz -C /"
docker run --rm -v bookend_documents:/data/documents -v bookend_exports:/data/exports alpine chown -R 1654:1654 /data   # the api's `app` uid
$DC up -d --wait                                      # migrator checks the ledger and applies any newer scripts; api starts after it

Restore with the same release version the backup was taken on, or a newer one: the migrator applies scripts a backup predates, but refuses (exit 4) a ledger that holds scripts its own release does not ship.

Then prove the restore:

1. GET /readyz healthy, GET /v1/diagnostics shows the expected schema version and license.
2. GET /v1/loans/{ref}/evidence/verify on a sealed loan returns intact: true with the full chain length: the bytes and the chain came back together. The nightly evidence sweep repeats this for every loan and records the result, shown in Diagnostics.
3. Sign in with an existing user. This proves the restored master key decrypts the signing key. Sessions started after the backup point are not in the restored database, so those users sign in again, which is expected.

What is not needed

Scheduler tables are in the dump; in-flight pipeline stages resume with their retry schedule. The watch volume is transient (packages move to processed/ or rejected/ after ingest) and the modelcache volume is rebuilt by the inference container.