Safety-gated upgrades are in Beta. Every command on this page ships in a released image and the recovery paths are supported, but we're still refining the workflow on our own instances, so some sequences and outputs will change. Open an issue if a step doesn't fit your deployment.

Upgrade safety tells you whether an upgrade is safe to roll and whether there are required stops on the way. This page tells you how to run it: the exact sequence for each deployment shape, the commands that inspect and drive migrations, and what to do when something gets stuck.

Read them in that order. Decide first, then execute.

No Qyra account is needed to upgrade safely

Nothing in this runbook requires a Qyra login, a personal access token, or an authenticated instance:

• qyra upgrade-check reads the public release-safety index over plain HTTPS. It never contacts your instance and never asks who you are.
• The migrate commands run inside your own Qyra container and authenticate with the same database environment variables the server already uses (PGHOST, PGUSER, PGPASSWORD, and friends). There is no second credential to provision.

That matters for air-gapped and locked-down deployments: the decision layer runs in CI with no secrets, and the execution layer runs in your cluster with credentials that already exist.

What shipped when

Every command on this page is in a released image. Version-fence your runbook accordingly:

If you are upgrading from something older, that is fine. These are properties of the image you are upgrading to, and of the CLI you run the check with. The one place the old world still shows up is rolling back across the 1.123.0 boundary.

The command surface

qyra upgrade-check

Answers the span question from the public index, with no login and no instance access. Full detail, including the JSON shape and the exit-code contract, is on upgrade safety.

qyra upgrade-check --from 1.130.0 --to 1.138.0
qyra upgrade-check --from 1.130.0 --to 1.138.0 --json

Exit 0 means the whole span is proven safe to roll. Anything else, including a version the index cannot see, exits non-zero. Both --from and --to are required and must be X.Y.Z release versions.

The migrate commands

These ship inside the Qyra image and are the runtime execution layer. Invoke them the same way the image's own entrypoint does:

pnpm -F backend migrate-production <command> [flags]

You do not need to change directory first. The image's working directory is /usr/app/packages/backend, which is where the entrypoint runs this command, and both kubectl exec and docker compose exec inherit it. The chart's migration Job runs the same command from /usr/app instead. Either directory works.

Running them in context

The commands need the deployment's database environment, so run them where that environment already exists.

kubectl exec deploy/qyra-backend -- \
  pnpm -F backend migrate-production status

kubectl get deploy,job -l app.kubernetes.io/instance=<release>

Reading preflight

Preflight probes the live database and reports one line per check, then a decision:

[RED PASS] version-path: The migration ledger structurally matches the target artifact direct-predecessor or up-to-date path
[RED PASS] postgres-version: ...
[RED PASS] migration-privileges: ...
[YELLOW WARN] long-transactions: ...
[INFO INFO] pending-migrations: ...
Preflight decision: proceed-with-warnings (0 red, 1 yellow)

The decision is one of proceed, proceed-with-warnings, abort, or force-proceed. Any red failure aborts; --strict makes yellow warnings abort too; --force turns an abort into force-proceed and prints a loud override banner. A standalone preflight that aborts exits non-zero, which makes it a usable CI gate.

Reading status

status reports one of four states:

Alongside the state it prints the lease holder (hostname, pod, app version, current migration, last heartbeat), the parked details if any, the completed and pending Knex migration counts, the ledger classification, and recent migration runs with their outcomes. Any prior unlock is recorded against the run that followed it, so the audit trail survives.

--json gives you the same payload for automation.

When you need wait

wait blocks until the database is migrated. While another process holds a live lease, it follows that process and never migrates alongside it. If migrations are still pending and the lease is unheld or has expired, wait claims the lease and runs them itself. It gives up after 30 minutes by default; change that with --timeout-ms or MIGRATION_WAIT_TIMEOUT_MS.

So wait is not a read-only command. It will not race a live migrator, but it will become the migrator when there is work to do and nobody is doing it. Use status when you only want to look.

You rarely need to run wait by hand. On the non-Job path, up already falls back to this exact behaviour when it loses the race for the lease, which is how the pod that does not migrate ends up waiting for the pod that does.

Kubernetes and Helm

qyra upgrade-check --from 1.130.0 --to 1.138.0

upgrade:
  mode: Recreate
migrationJob:
  enabled: true

kubectl get hpa -l app.kubernetes.io/instance=qyra
kubectl delete hpa -l app.kubernetes.io/instance=qyra

kubectl get deployments -l app.kubernetes.io/instance=qyra
kubectl scale deployments -l app.kubernetes.io/instance=qyra --replicas=0

kubectl get pods -l app.kubernetes.io/instance=qyra

helm repo update qyra
helm upgrade -f values.yml qyra quanvio/qyra

kubectl logs -f job/qyra-migrate     # when migrationJob is enabled
kubectl exec deploy/qyra-backend -- \
  pnpm -F backend migrate-production status

curl -sS -o /dev/null -w '%{http_code}\n' https://qyra.example.com/api/v1/readyz

Docker compose

qyra upgrade-check --from 1.130.0 --to 1.138.0

docker compose pull qyra
docker compose run --rm --entrypoint pnpm qyra \
  -F backend migrate-production preflight

docker compose pull qyra
docker compose up --detach --remove-orphans

docker compose exec qyra \
  pnpm -F backend migrate-production status
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8080/api/v1/readyz

Automated upgrades

The Qyra repository ships a generic reference automation at examples/upgrade-automation: a GitHub Actions workflow plus two composite actions that keep a deployment on the newest release the public safety gate can reach. It is deliberately generic, sends no telemetry, and keeps all of its evidence in your own repository. Copy it and point it at the file that pins your image tag.

The loop it implements is the sequence to copy even if you build your own:

Pass a checked mode to Helm

The reference automation only promotes green-reachable targets. Use the route below when your deployment pipeline intentionally upgrades to a checked target that can require Recreate. The chart does not fetch the verdict. The pipeline passes the checked mode to Helm.

set -euo pipefail

: "${CURRENT_VERSION:?Set CURRENT_VERSION}"
: "${TARGET_VERSION:?Set TARGET_VERSION}"
: "${TARGET_IMAGE_TAG:=$TARGET_VERSION}"
: "${VALUES_FILE:=values.yml}"

version_gte() {
  local current_major current_minor current_patch minimum_major minimum_minor minimum_patch
  IFS=. read -r current_major current_minor current_patch <<<"$1"
  IFS=. read -r minimum_major minimum_minor minimum_patch <<<"$2"
  if (( 10#$current_major != 10#$minimum_major )); then
    (( 10#$current_major > 10#$minimum_major ))
    return
  fi
  if (( 10#$current_minor != 10#$minimum_minor )); then
    (( 10#$current_minor > 10#$minimum_minor ))
    return
  fi
  (( 10#$current_patch >= 10#$minimum_patch ))
}

set +e
CHECK_JSON="$(qyra upgrade-check --from "$CURRENT_VERSION" --to "$TARGET_VERSION" --json)"
CHECK_EXIT=$?
set -e

if ! jq -e \
  --arg from "$CURRENT_VERSION" \
  --arg to "$TARGET_VERSION" \
  '
    (.fromVersion == $from) and
    (.toVersion == $to) and
    (.safe | type == "boolean") and
    (.verdict == true or .verdict == false or .verdict == "unknown") and
    (.requiredStops | type == "array" and all(.[]; type == "string")) and
    (.minPreviousVersion == null or (.minPreviousVersion | type == "string")) and
    (.missingRanges | type == "array" and all(.[]; type == "object" and (.afterVersion | type == "string") and (.beforeVersion | type == "string")))
  ' >/dev/null <<<"$CHECK_JSON"; then
  printf '%s\n' 'upgrade-check returned no usable verdict; aborting.' >&2
  exit 1
fi

MINIMUM_VERSION="$(jq -r '.minPreviousVersion // empty' <<<"$CHECK_JSON")"
if [[ -n "$MINIMUM_VERSION" ]] && ! version_gte "$CURRENT_VERSION" "$MINIMUM_VERSION"; then
  printf 'Current version %s is below the minimum direct-upgrade version %s.\n' "$CURRENT_VERSION" "$MINIMUM_VERSION" >&2
  exit 1
fi

EARLIEST_INTERMEDIATE_STOP="$(jq -r --arg target "$TARGET_VERSION" '
  [.requiredStops[] | select(. != $target) | {version: ., parts: (split(".") | map(tonumber))}]
  | sort_by(.parts)
  | .[0].version // empty
' <<<"$CHECK_JSON")"
if [[ -n "$EARLIEST_INTERMEDIATE_STOP" ]]; then
  printf 'Upgrade first to required stop %s, then check the next hop.\n' "$EARLIEST_INTERMEDIATE_STOP" >&2
  exit 1
fi

REQUIRED_STOP_COUNT="$(jq -r '.requiredStops | length' <<<"$CHECK_JSON")"
if [[ "$REQUIRED_STOP_COUNT" != "0" ]]; then
  STOP_TARGET_VERDICT="$(jq -r --arg target "$TARGET_VERSION" '
    if .requiredStops | length == 1 and .[0] == $target then .verdict else "invalid" end
  ' <<<"$CHECK_JSON")"
  case "$STOP_TARGET_VERDICT:$CHECK_EXIT" in
    true:1) UPGRADE_MODE=RollingUpdate ;;
    false:1|unknown:1) UPGRADE_MODE=Recreate ;;
    *)
      printf '%s\n' 'The required-stop target has an unusable verdict; aborting.' >&2
      exit 1
      ;;
  esac
else
  CHECK_STATE="$(jq -r '[.safe, .verdict] | @tsv' <<<"$CHECK_JSON")"
  case "$CHECK_STATE:$CHECK_EXIT" in
    $'true\ttrue:0') UPGRADE_MODE=RollingUpdate ;;
    $'false\tfalse:1'|$'false\tunknown:1') UPGRADE_MODE=Recreate ;;
    *)
      printf '%s\n' 'upgrade-check returned an unusable result; aborting.' >&2
      exit 1
      ;;
  esac
fi

helm upgrade -f "$VALUES_FILE" \
  --set-string upgrade.mode="$UPGRADE_MODE" \
  --set-string image.tag="$TARGET_IMAGE_TAG" \
  qyra quanvio/qyra

--set-string overrides any static upgrade.mode in the values file. Derive the mode for every upgrade. An earlier required stop is a separate target. When requiredStops is exactly [TARGET_VERSION] and the minimum version is satisfied, true selects RollingUpdate; false or unknown selects Recreate. A current version below minPreviousVersion aborts the direct hop. Malformed output, an execution failure, or a fetch failure has no usable verdict and aborts the pipeline.

Recommended default policy

Auto-apply when green, hold when not. A proven-safe hop is exactly the case where human review adds latency and no information; everything else is exactly the case where it adds both. Keep the freeze switch manual and obvious, so disarming upgrades during an incident is one action rather than a code change.

Recovery

The migration Job failed

On the migrationJob.enabled path, a failed upgrade shows up as a failed Job. Read its logs before you reach for anything else. The migration's own error says what broke; the commands below only tell you the state it left behind.

kubectl logs job/qyra-migrate
kubectl describe job/qyra-migrate

The Job retries before it gives up. migrationJob.backoffLimit defaults to 10, so Kubernetes marks the Job failed only after 11 attempts. Each attempt runs in its own pod, so kubectl logs job/... shows you one of them and kubectl get pods shows the rest.

If you missed the window, the database still has the durable record. migrate status reports the parked details and recent migration runs with their outcomes, and that record outlives the Job object.

To retry after you have fixed the cause, re-run helm upgrade to recreate the hook Job. See what resumes on its own.

A migration is stuck

Start by looking, not by fixing:

kubectl exec deploy/qyra-backend -- \
  pnpm -F backend migrate-production status

• migrating with a recent heartbeat: it is working. Migrations on large tables can take a long time. Leave it alone.
• stale: the holder died. The lease expires 75 seconds after its last heartbeat, and another process takes it over automatically. No action needed in most cases.
• parked: the migration failed its retries (three attempts with backoff) and stopped deliberately. The same app version will refuse to retry, which is what stops a crash-looping pod from hammering a half-applied migration. Fix the cause, then deploy a fixed version, or unlock with attribution and retry.

Releasing a lock

kubectl exec deploy/qyra-backend -- \
  pnpm -F backend migrate-production unlock --actor "alex@example.com"

--actor is mandatory and is recorded against the next migration run, so an unlock is always attributable afterwards.

unlock refuses, by design, when the lease is actively held by a live process, or when a pre-lease Knex lock is still held. Both refusals mean "something may still be running". Terminate the holder first. Only then reach for --force, which overrides the refusal.

After an unlock: what resumes on its own

Rebuilding an instance from scratch

Three things are needed to stand an instance back up, so keep all three recoverable:

• the Postgres backup,
• the QYRA_SECRET (it decrypts data at rest — losing it means losing access to encrypted data), and
• your Helm values.

Rolling back

Rolling back means redeploying an older image. It does not unwind the database, and Qyra does not run down-migrations in production.

• Back up first. Always, and before the upgrade rather than after you need it. A backup restore is the only path that undoes a schema change, and it costs you everything written since the snapshot.
• Prefer small spans. One release back is a decision. Ten releases back is an archaeology project. Frequent, small upgrades keep the rollback target close.
• Roll back promptly. Schema compatibility is not data compatibility. The new version may have written values the old code mishandles or cannot read, and that risk grows every hour the new version serves traffic. A rollback ten minutes in is a very different proposition from one ten days in.
• upgrade-check will not help here. It answers forward spans only; reverse spans are an error, not a verdict. Use the guidance on this page instead.

The 1.123.0 fence

The image you roll back to determines what happens when it meets a database that is ahead of it:

• 1.123.0 and later: the migrate command classifies the ledger itself. A database carrying migrations the image does not have is recognised as database-ahead and the image starts normally. ALLOW_MISSING_MIGRATIONS is a deprecated no-op on this path and logs a warning saying so.
• Before 1.123.0: the image validates the migration directory at boot and treats any database-only migration as a corrupt migration directory. It will refuse to start. Set ALLOW_MISSING_MIGRATIONS=true on that deployment so it can start against the newer database.

So a rollback from 1.130.0 to 1.124.0 needs nothing extra, while a rollback from 1.130.0 to 1.122.0 needs ALLOW_MISSING_MIGRATIONS=true.

Migration batch granularity

The lease runtime applies each migration as its own Knex batch, rather than grouping a whole deploy into one batch as stock Knex does. That changes the granularity of the development-tooling rollback: knex migrate:rollback unwinds one migration per invocation, not one deploy per invocation.

This matters mid-incident, when someone reaches for a rollback expecting a whole deploy to come off in one command. It will not. Production recovery remains forward-only regardless.

For contributors

If you write migrations, the safety verdict this runbook depends on is generated from declarations in the migration files themselves. A migration containing a detected breaking operation must declare it in the same file:

export const breaking = {
    reason: 'old pods read legacy_column',
    requiredStop: true,
};

Raw SQL that the static lint cannot classify needs an explicit export const classification = { kind: 'safe' | 'breaking', reason: '...' }. Declaring a break is not a way to make CI pass: it flips the release to not rolling-safe and advises every self-hosted deployment to use Recreate. Try an expand-only redesign first.

The full rules, including the idempotency contract for transaction: false migrations and the down() requirements, live in packages/backend/src/database/migrations/CLAUDE.md in the Qyra repository.