When Three Kubernetes Pods Try to Run One Database Migration

This startup command looked fine while our application had one replica:

prisma migrate deploy && node server.js

Then we scaled the Deployment to three pods.

All three pods started together. All three ran the migration command. Prisma used a PostgreSQL advisory lock to protect the migration, so the database was not corrupted, but the losing pods waited for the lock and eventually timed out. Kubernetes saw failed containers, restarted them, and filled the rollout logs with errors.

One pod was doing useful work. The other two were turning expected contention into application failures.

The clean solution is a migration Job that runs before the Deployment. Our release platform could only start one application workload, so we used a small PostgreSQL-backed coordinator instead.

This post shows the complete pattern.

The problem in one picture

A Deployment creates copies of a long-running process. A database migration is one-shot work.

The coordinator gives every pod the same startup protocol:

1. Try to acquire one application-specific advisory lock.
2. If the lock is busy, close the connection, wait with jitter, and try again.
3. If the lock is acquired, run prisma migrate deploy.
4. Release the lock and start the application.

Every pod runs the migration command after acquiring the lock. For the first pod it applies pending migrations. For later pods it is a no-op. This matters because a released lock only proves that the previous database session ended. It does not prove the previous migration succeeded.

Use a Job when your platform supports one

Before adding coordination code, check whether the delivery system can run an ordered pre-deployment Job.

A Job is the better default because it gives the migration its own status, logs, image and database credentials. The application Deployment does not need schema-changing permissions, and a failed migration stops the rollout at the correct boundary.

The coordinator below is for the awkward case where the platform can only deploy the application workload. That was our constraint.

Add the coordinator

The application already contained Prisma. The coordinator only needed the PostgreSQL client:

npm install pg

Create prisma/migration-coordinator.mjs:

import { spawn } from "node:child_process";
import { createHash } from "node:crypto";
import { fileURLToPath } from "node:url";
import pg from "pg";

const { Client } = pg;

const connectionString = process.env.DIRECT_DATABASE_URL;
const lockName = process.env.MIGRATION_LOCK_NAME ?? "web:public:prisma-migrations";
const lockId = createHash("sha256")
  .update(lockName)
  .digest()
  .readBigInt64BE(0)
  .toString();
const waitTimeoutMs = Number(process.env.MIGRATION_WAIT_TIMEOUT_MS ?? "300000");
const retryDelayMs = Number(process.env.MIGRATION_RETRY_DELAY_MS ?? "2000");
const prismaCli = fileURLToPath(new URL("../node_modules/.bin/prisma", import.meta.url));

if (!connectionString) {
  throw new Error("DIRECT_DATABASE_URL is required");
}

function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

function runPrismaMigrateDeploy() {
  return new Promise((resolve, reject) => {
    const child = spawn(prismaCli, ["migrate", "deploy"], {
      stdio: "inherit",
      env: {
        ...process.env,
        DATABASE_URL: connectionString,
      },
    });

    child.once("error", reject);
    child.once("exit", (code) => {
      if (code === 0) {
        resolve();
        return;
      }

      reject(new Error(`prisma migrate deploy exited with code ${code}`));
    });
  });
}

const deadline = Date.now() + waitTimeoutMs;
let migrationComplete = false;

while (!migrationComplete) {
  const client = new Client({
    connectionString,
    connectionTimeoutMillis: 10_000,
  });
  let acquired = false;

  try {
    await client.connect();

    const result = await client.query(
      "SELECT pg_try_advisory_lock($1::bigint) AS acquired",
      [lockId],
    );
    acquired = result.rows[0].acquired === true;

    if (acquired) {
      console.log("migration lock acquired");
      await runPrismaMigrateDeploy();
      console.log("database migrations are current");
      migrationComplete = true;
    }
  } finally {
    if (acquired) {
      await client
        .query("SELECT pg_advisory_unlock($1::bigint)", [lockId])
        .catch((error) => console.error("failed to release migration lock", error));
    }

    await client.end().catch(() => {});
  }

  if (migrationComplete) {
    break;
  }

  if (Date.now() >= deadline) {
    throw new Error("timed out waiting for the migration lock");
  }

  const jitterMs = Math.floor(Math.random() * 500);
  console.log("migration lock busy; waiting before retry");
  await sleep(retryDelayMs + jitterMs);
}

There are four details worth keeping:

• pg_try_advisory_lock returns immediately instead of keeping a waiting connection open.
• The same database connection stays open while the Prisma subprocess runs. Session-level advisory locks belong to the connection that acquired them.
• Every lock owner runs migrate deploy. Already-applied migrations make it a no-op.
• A failed migration rejects the script, so the application does not start against an unknown schema.

Use a direct PostgreSQL connection for DIRECT_DATABASE_URL. Transaction-mode poolers do not preserve session state, so they are the wrong boundary for a session-level advisory lock.

The lock name is not a secret, but it must be stable. The script hashes the readable name into the signed 64-bit integer PostgreSQL expects. Every revision of this application should use the same name for the same database and schema. Different applications sharing a database should use different names.

After adding the coordinator, expected contention becomes an ordinary waiting path:

Run it before the server

Update the container command so the server starts only after coordination succeeds:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: registry.example.com/web:abc123
          ports:
            - name: http
              containerPort: 3000
          command: ["sh", "-c"]
          args:
            - node prisma/migration-coordinator.mjs && exec node server.js
          env:
            - name: DIRECT_DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: web-database
                  key: direct-url
            - name: MIGRATION_LOCK_NAME
              value: "web:public:prisma-migrations"
            - name: MIGRATION_WAIT_TIMEOUT_MS
              value: "300000"
          startupProbe:
            httpGet:
              path: /health/startup
              port: 3000
            periodSeconds: 5
            failureThreshold: 72
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 3000
            periodSeconds: 5

exec replaces the shell with the Node process after the migration finishes. That lets the application receive termination signals directly as PID 1.

The coordinator calls the Prisma binary already installed in the image. Keep the Prisma CLI in the production image rather than downloading it during startup. If your image uses a different project layout, adjust prismaCli to match it.

The startup probe allows six minutes for lock waiting, migration and application startup. Set that budget from your slowest expected migration rather than copying this number. Readiness still decides when the running pod can receive traffic.

What happens during a rollout

With three new pods, the rollout now looks like this:

pod-a  migration lock acquired
pod-b  migration lock busy; waiting before retry
pod-c  migration lock busy; waiting before retry

pod-a  database migrations are current
pod-a  application starts

pod-b  migration lock acquired
pod-b  no pending migrations to apply
pod-b  application starts

pod-c  migration lock acquired
pod-c  no pending migrations to apply
pod-c  application starts

Expected contention is logged as waiting, not as a migration failure. If the winning pod disappears, PostgreSQL releases its session lock when the connection closes. Another pod acquires the lock and runs migrate deploy again.

This has the same shape as the shared-state problem in Scaling an Agentic Coding SDK: What Concurrency Actually Costs. Sequential execution had hidden an ownership problem. Adding replicas made it visible.

The lock does not make breaking migrations safe

The advisory lock serialises migration commands. It does not stop old pods from serving traffic while a new pod changes the schema.

Rolling deployments still need expand-and-contract migrations:

1. Add the new schema in a backward-compatible form.
2. Deploy code that works with both representations.
3. Backfill data if required.
4. Switch reads and writes.
5. Remove the old schema after old pods are gone.

I covered the same compatibility problem in more detail in Replacing a Critical Data Path Without a Flag Day.

Before shipping it

Check these six things:

1. A pre-deployment Job really is unavailable.
2. The lock uses a direct PostgreSQL session.
3. The lock name is stable and unique to the application and schema.
4. Waiting has jitter and an overall deadline.
5. A failed migration prevents the application from starting.
6. Schema changes remain compatible with old pods during the rollout.

That is the whole pattern. One pod applies the migration. The others wait, check the same desired state, and then start normally.

The code is small because PostgreSQL already owns the hard guarantee: only one session can hold the advisory lock at a time.

Thanks for reading ✌️

References

• Prisma Migrate in development and production
• Prisma migrate deploy reference
• PostgreSQL advisory lock functions
• Kubernetes Jobs
• Kubernetes liveness, readiness and startup probes