Key Rotation Without Downtime: Never Generate a Key in a Migration
Third post from the compliance-ready backend kit, after nested JWTs and append-only audit logs.
Key rotation is the control everybody agrees with and almost nobody exercises. It sits in the policy document as “keys are rotated periodically”, and the first time anyone runs it for real is during an incident, which is the worst possible moment to discover that rotating logs every user out, or that it quietly did nothing.
Two questions decide whether you have it. Can you rotate without an outage? And does the old key actually stop working afterwards?
The second one is where I found a bug that no amount of reasoning would have surfaced.
Never generate a key in a migration
Start here, because it is the mistake with the worst blast radius and it looks tidy.
You need a signing key before the service can issue tokens. There is a migration system right there. Generating the key in a migration seems like exactly the kind of one-time setup migrations are for.
A migration is committed to git. It is also replayed against Prisma’s shadow database on every schema diff. So any key a migration generated is byte-identical in every environment that ever ran it, sitting in your repository, present in every clone and every CI log. Your production signing key would be public, and it would look provisioned.
Key generation is an operational act, not a schema change. So it lives in an operator command:
pnpm keys:init # bootstrap: one signing + one encryption key
pnpm keys:rotate --purpose signing # mint a new active key, retire the previous one
pnpm keys:list # every key, its state, and any overlap window
pnpm keys:revoke --kid <kid> --reason "..."That has a consequence worth stating plainly, because it is a confusing first-run experience otherwise: the table starts empty, and a service with no active signing key boots fine and fails every login. So keys:init is a required setup step, not an optional one. The error says so directly rather than surfacing as a generic 500.
Where the keys live
In a config_keys table in the master database, holding wrapped material and never plaintext.
The only key in configuration is the KEK that wraps everything else. That is the whole point of envelope encryption: a key in config is a key in every deploy manifest, every CI secret store and every developer shell, whereas one KEK can be moved into a KMS, an HSM or an enclave without touching the rest of the system.
Keys are deployment-wide, not per tenant, and that surprised me until I tried to design it the other way. Which tenant a token belongs to is the tid claim inside the ciphertext. A per-tenant key would require knowing the tenant before the token can be decrypted, and every route to that is broken: reading an unverified claim is circular, putting the tenant in a header leaks it on every request, and letting the caller name the key is attacker-directed key selection.
kid is TEXT, not uuid
A small column decision with a real failure mode.
RFC 7517 section 4.5 makes kid an arbitrary case-sensitive string. A uuid column would normalise case, which contradicts the spec. That part is theoretical.
This part is not: an attacker sending a non-uuid kid against a uuid column raises Postgres error 22P02, which surfaces as a 500 and an error-log flood instead of a clean 401. You have handed anyone a cheap way to fill your logs and your error tracker, using a value they fully control, on an unauthenticated path.
Any place you accept an attacker-supplied identifier and hand it straight to a typed column has this shape. Worth grepping for.
The lifecycle is enforced by Postgres, not by code
“Exactly one active key per purpose” checked in application code is a race between two operators, or two pods, in one maintenance window. So the invariants live in the database.
Prisma cannot express any of this: @@unique takes no predicate and there is no CHECK syntax. So it is hand-written in the migration, where the shadow database still reproduces it and it does not later read as drift.
-- THE invariant. As a partial unique index, Postgres simply refuses the second row.
CREATE UNIQUE INDEX "config_keys_one_active_per_purpose"
ON "config_keys" ("purpose") WHERE "state" = 'active';Then four CHECK constraints, each closing a specific hole:
Purpose paired to algorithm. A mismatched row is an algorithm-confusion primitive: token_signing with A256KW would offer 32 raw symmetric bytes to a signature verification, which is RFC 8725 section 2.1 territory. Neither a Prisma enum nor a foreign key can express that two columns must agree.
A public JWK required for asymmetric keys, refused for symmetric ones. A NULL public_jwk on a signing key makes it unverifiable by anyone but us. A non-NULL one on an encryption key is a row the JWKS endpoint could accidentally publish.
A revoked row must hold no material, plus a timestamp and a reason. Revocation destroys the key or it is not revocation. A revoked row still holding its ciphertext means the revocation did not happen, and the row is now lying to whoever reads it.
A retiring row must have an end. A retiring key with no not_after verifies forever, which is not a rotation. Enforced here rather than trusted to the rotate command.
That last one is the pattern worth internalising. The rotate command already sets not_after. The constraint exists because the command might not, in some future edit, and the failure would be invisible: everything keeps working, the old key just never expires.
Rotation: mint pending, then swap atomically
Three states do the work, and pending is the one that makes it safe.
A new key is created as pending, wrapped, outside any transaction. It affects nothing. Then a single transaction does the swap:
return master.$transaction(async (tx) => {
const current = await tx.configKey.findFirst({ where: { purpose, state: "active" } });
if (current) {
await tx.configKey.update({
where: { kid: current.kid },
data: { state: "retiring", retiringAt: new Date(), notAfter },
});
}
await tx.configKey.update({
where: { kid },
data: { state: "active", activatedAt: new Date() },
});
return { retired: current?.kid };
});The partial unique index is what forces this into a transaction: two active keys for one purpose cannot coexist, so the retire and the activate happen together or the second statement fails. And a crash between them cannot leave the deployment with no active key, which would be a total authentication outage.
The overlap window is arithmetic, not a guess
notAfter = now + JWT_ACCESS_TTL_SECONDS + JWT_CLOCK_TOLERANCE_SECONDS // 900 + 5 by defaultA token signed a moment before rotation must still verify until it expires. That is the access-token TTL. Add the clock tolerance because verifiers may disagree with the issuer about what time it is.
Both directions of getting this wrong are bad, which is why it is computed rather than configured. Shorter, and rotation logs out every currently valid session. Absent, and the overlap never ends: a key you meant to retire keeps verifying forever, which defeats the reason you rotated.
Revoking the active key is refused outright, because it would leave the service unable to issue tokens at all. Revoking a retiring key is allowed and destroys its material. CI asserts that revoking the active key fails, because what makes rotation correct here is the constraints rather than the code, and the only way to know a constraint works is to try to violate it.
not_after is honoured at the point of use
private usable(state: KeyState, notAfter: Date | null): boolean {
if (state === "revoked" || state === "pending") return false;
if (notAfter && notAfter.getTime() <= Date.now()) return false;
return true;
}Not delegated to a sweeper job. A sweeper that has not run yet, or that failed silently, would otherwise leave an expired key verifying tokens indefinitely. Expiry that depends on a cron running is not expiry.
The bug: refresh-on-miss silently misses a rotation
This is the part I would not have got right by thinking about it.
The resolvers handed to jose must be synchronous, because jose calls them with an attacker-controlled kid before anything has been verified. I covered why in the nested JWT post: an async resolver turns a forged kid into a database query or a KMS unwrap per unauthenticated request. So the service keeps an in-memory snapshot and the resolvers only ever read it.
An operator rotating a key still has to take effect without a restart. The obvious mechanism is refresh-on-miss: an unrecognised kid triggers a reload, rate-limited to at most one per 30 seconds. That is the same shape jose’s own createRemoteJWKSet uses, and it bounds what a forged kid can cause to one query per window.
Then I rotated a key against a running server.
The instance carried on signing with the key it had loaded at boot, and published a JWKS containing only that key. Because nothing had failed. A miss only happens for a kid the snapshot does not know. A stale-but-still-valid active key produces no misses at all: the old key keeps verifying every token it sees, so the service never learns there is a newer one. Rotation had silently not taken effect, and would not until a restart.
The fix is unglamorous: also reload on a 60 second timer, regardless of traffic.
const REFRESH_COOLDOWN_MS = 30_000; // at most one reload per window, however many unknown kids
const REFRESH_INTERVAL_MS = 60_000; // reload regardless of traffic, because a miss may never comeLISTEN/NOTIFY would make it immediate, at the cost of a dedicated connection per instance and a reconnect path to get wrong. A minute of staleness on a rotation is an acceptable trade, especially since the retiring key stays valid for the whole overlap window anyway.
The generalisable lesson: an event-driven cache invalidation that only fires on failure cannot detect a change that does not cause a failure. If your invalidation is triggered by a miss, ask what happens when the stale value still works. Ours worked perfectly, which is exactly why nothing fired.
And note how it was found. Not by review, not by a unit test. By running the operator command against a live server and looking at what the server then did.
The JWKS, and two things that had to be special-cased
Public signing keys are served at /.well-known/jwks.json, at the origin root, per RFC 8615.
The route is excluded from the global api prefix. A well-known URI nested under a path prefix is not a well-known URI. A verifier that expects to find it by convention looks at the root, finds nothing, and you get to debug that instead.
The route returns a raw response, bypassing the success envelope. This is the one worth remembering, because of how it hid. Every response in the kit is wrapped as { success, data }. Applied to a JWK Set, that produces:
{"success":true,"data":{"keys":[...]}}which breaks every standard consumer, including jose’s own createRemoteJWKSet. The handler was correct. A unit test on the handler passed, because the handler returned the right object and the wrapping happened downstream in an interceptor. The bug existed only in the bytes on the wire, so nothing that inspected the return value could see it.
There is now a smoke assertion on the wire format, and a second one asserting the document contains no private scalar d. Both are the kind of test that feels redundant right up until it is the only thing that would have caught the problem.
A published JWKS does not mean anyone can verify your tokens
Worth being blunt, because the presence of a JWKS invites the opposite conclusion.
These are nested tokens: the outer layer is a symmetric A256KW JWE. So the JWKS alone gets a third party exactly nowhere, and specifically ERR_JWS_INVALID, because they cannot get past the encryption to reach the signature. Verification requires being handed the A256KW key too, and that key grants decryption of every token the deployment has ever issued, though still not the ability to mint one.
ECDH-ES for the outer layer would fix this properly and is not implemented. Until it is, treat token verification as something only the issuing deployment can do, and do not hand out the outer key thinking of it as a verification credential.
The honest gap
Envelope encryption, the lifecycle, graceful rotation and the published JWKS all exist. The KEK is still in configuration, because no KMS or HSM adapter is written. So the KEK is only as protected as the process and its config store.
That is why the kit marks key management Partial rather than Implemented, and why that row must not be cited against a requirement calling for a secure cryptographic device, which PCI Req 3.6 and 3.7 do. The difference between “we encrypt keys at rest with a key in an environment variable” and “keys are protected by a secure cryptographic device” is exactly the difference between a KMS and an HSM, and an assessor knows it.
What makes that gap closeable rather than structural is a port:
interface KeyProvider {
wrap(plaintext: Uint8Array, ctx: KeyContext): Promise<Uint8Array>;
unwrap(ciphertext: Uint8Array, ctx: KeyContext): Promise<Uint8Array>;
}The local implementation uses AES-256-GCM with the AAD bound to purpose and kid, so a wrapped signing key cannot be replayed as an encryption key or under a different kid. A KMS, HSM or enclave adapter is a new file implementing this interface, not a refactor. That was the entire reason for defining the seam before needing it.
Also missing, stated rather than buried: no automatic rotation schedule, since rotation is an operator action, and no documented periodic cryptographic inventory review.
The part worth taking with you
- Never generate a key in a migration. It is committed, replayed, and identical everywhere, which makes it public.
- Put the lifecycle in database constraints. “One active key” in application code is a race, and the only way to know a constraint works is to try to violate it in CI.
- Compute the overlap window as TTL plus clock tolerance. Too short logs everyone out; missing means the old key never dies.
- A cache invalidation that only fires on failure cannot see a change that does not fail. Rotate against a running server and watch what it does next, because this class of bug is invisible to reasoning and to unit tests.
The code is in packages/db/src/keys/manage-keys.ts, services/auth/src/keys/key-registry.service.ts and the config_keys migration in the kit repo. Next in this series: one response contract, and why RFC 9457 Problem Details beats whatever error shape you invented.