Emergency Kill-Switch & Instant-Rollback Runbook

This how-to is part of Managing Flag Deprecation & Cleanup. It is the runbook you reach for when a release behind a flag is actively causing an incident and you need the blast radius gone in seconds — not after a revert, rebuild, and redeploy.

The scenario: a flagged feature (checkout.payments.express-pay) is throwing errors in production, error budget is burning, and you are on call. A kill switch flips the feature to its safe variant for every request instantly, bypassing all targeting logic, because the change lives in the control plane rather than in your deploy pipeline. This runbook covers flipping it, proving it propagated, and recovering cleanly.

The reason a kill switch beats a code revert during an incident is arithmetic: a revert has to pass CI, build an artifact, roll through your deploy stages, and drain connections — a fifteen-to-forty-minute path on most pipelines, and every one of those minutes burns error budget and customer trust. A control-plane flip removes the feature from the request path in the time it takes your sync transport to fan the change out to every replica, which is sub-second on streaming and one poll interval otherwise. Treat this document as muscle memory to rehearse in a game day, not as reference material you read for the first time at 03:00 with a pager screaming — the whole value of a kill switch evaporates if you have to think about the commands while the incident is live.

Prerequisites

• Write access to the production flag control plane (or a break-glass credential)
• The flag’s safe defaultVariant known and documented in the flag metadata
• flagctl (or your provider CLI) authenticated against production
• A way to query each replica’s resolved variant (a debug endpoint or trace query)
• Sync transport propagation budget known — see polling vs streaming

Step-by-Step Procedure

Three moves, in order: identify the flag and its safe variant, force that variant for all traffic, and confirm every replica actually flipped. The critical nuance is force, not disable.

Step 1 — Identify the flag and its safe variant

Confirm the exact flag key and the variant that disables the failing behavior before touching anything.

flagctl get checkout.payments.express-pay --env prod -o json | jq '{state, defaultVariant, variants}'

The defaultVariant is your target state. If the safe value isn’t obvious, the flag taxonomy metadata should record which variant is fail-safe. Reading the current state first also tells you whether someone has already touched the flag — if it is already forced to some other variant, you may be looking at a mid-flight change from a teammate rather than the natural targeting state, and blindly overwriting it can mask what actually triggered the incident. Copy the full JSON output into your incident channel before you change anything; that snapshot is the “before” half of the diff you will want during the retro, and it is the fastest way to prove exactly what production was serving at the moment the kill switch fired.

Step 2 — Force the safe variant for all traffic

Override targeting entirely so every evaluation returns the safe variant, regardless of context.

flagctl set checkout.payments.express-pay \
  --env prod --force-variant off --reason "INC-4821 express-pay 5xx" --actor "$USER"

Forcing the variant (rather than disabling the flag) keeps the flag object intact for audit and makes recovery a single inverse command. The --reason and --actor land in the audit trail. Put the incident ticket ID in --reason unconditionally — six months later, when someone runs a cleanup audit and finds a flag pinned to off, that string is the only thread connecting the override to the outage that justified it, and a flag pinned “just in case” with no ticket behind it is exactly the kind of debt that never gets paid down. If your control plane supports it, prefer a targeting override that pins the variant at the top of the rule stack rather than editing the existing rules in place; the override is a single object you delete to recover, whereas hand-edited rules have to be reconstructed from memory, and reconstruction under pressure is where second incidents are born.

Step 3 — Confirm propagation across every replica

A kill switch you can’t confirm is a guess. Query each replica until they all report the safe variant.

for host in $(cat replicas.txt); do
  printf '%s ' "$host"; curl -s "$host/debug/flags/checkout.payments.express-pay" | jq -r '.variant'
done | sort | uniq -c     # expect every line to read "off"

If some replicas lag, you are watching your sync transport’s propagation window — streaming clears in under a second, polling within one interval. The sort | uniq -c at the tail is deliberate: it collapses hundreds of hosts into a two-line summary so you can see at a glance whether the fleet is uniform, and the moment one line still reads the old variant you have a concrete list of hosts to investigate rather than a vague suspicion. Do not stop querying the instant the first replica flips — a single healthy node proves the change reached the control plane, not that it reached the fleet, and the whole point of Step 3 is to distinguish “I sent the command” from “every request now honours it.” If you run behind a CDN or edge worker that evaluates flags at the edge, remember those points of presence are replicas too; a debug endpoint that only reaches your origin fleet will report all-clear while edge caches keep serving the failing variant to real users.

Verification Step

Confirm the error signal actually stops. Watch the service’s 5xx rate or the failing metric for one full propagation window plus a safety margin:

# Error rate should fall to baseline within the propagation window
watch -n 5 'curl -s http://metrics.internal/q?expr=rate_5xx{service="checkout"} | jq .value'

The incident is mitigated — not resolved — once the metric returns to baseline. Recovery (Step: restore targeting) happens only after the root-cause fix ships and is verified in staging.

Watch a second signal alongside the error rate, because a kill switch can move the symptom without fixing the user. If express-pay was failing, forcing it off will stop the 5xx spike, but customers now fall back to the standard checkout path — confirm that path is actually carrying the traffic and not silently dropping conversions, or you have swapped a loud failure for a quiet one. Give the metric a full propagation window plus a margin of at least one more window before you declare mitigation; latency percentiles and cache-warmed error counters both lag the underlying change, and calling it too early means you announce “resolved” in the incident channel just as a delayed metric ticks back up. If the error rate does not fall after a full window, do not assume the switch failed — first re-run Step 3, because an unpropagated switch and an ineffective switch look identical on the dashboard and demand opposite responses.

Gotchas & Edge Cases

• Forced variant vs. flag disable: disabling a flag falls back to the SDK-supplied code default, which may differ from the control-plane defaultVariant. Force the explicit safe variant so behavior is deterministic. Worse, the in-code default is set by whoever wrote the client.getBooleanValue("...", false) call, often years ago and often as a placeholder — betting the incident on that literal matching today’s safe state is a bet you will eventually lose.
• Cached evaluations: a long local TTL can keep a node serving the old variant past the propagation window. Confirm per-replica (Step 3) rather than trusting a single healthy node. The same trap lives one layer up: if your application memoizes evaluation results per request or per session, a user mid-session may keep hitting the failing variant until their session turns over, so measure recovery against new sessions, not a synthetic probe that opens a fresh connection every time.
• Restoring too early: flipping targeting back before the fix is verified re-triggers the incident and burns trust in the kill switch. Gate recovery on a verified fix, and record it in the audit trail.
• Flag dependencies: if express-pay is a prerequisite for a downstream flag — say a rewards feature that only renders when express-pay is on — forcing it off can trip that dependent flag into an unexpected branch. Know the dependency graph before you flip, or you turn a single-feature incident into a cascade across features that shared the toggle.
• The kill switch is itself a change: a forced override is a production write, and on a locked-down control plane it can require the same approval gate as any other change. Break-glass credentials exist precisely to bypass that gate under a declared incident — provision them, scope them tightly, and alert on every use, because an emergency path that needs a second approver is not an emergency path.

Troubleshooting & FAQ

The kill switch fired but errors continue on a few hosts — why?

Those hosts are still serving a cached or pre-switch rule set. Check their resolved variant directly (Step 3); if they lag, your local cache TTL or a dropped streaming connection is the cause. A failed sync connection is the usual culprit — verify each replica’s connection state.

Should I disable the flag or force a variant?

Force the safe variant. Disabling reverts to the in-code default, which is not guaranteed to match the control-plane safe state, and it loses the explicit intent in the audit log.

How do I make sure a kill switch is always fast enough?

Keep the failing-feature flags on a streaming transport with a tight fallback poll, and rehearse the runbook so propagation latency is a known number before an incident, not a discovery during one.

What if the control plane itself is down when I need the kill switch?

Then you fall back to the last value each replica cached and to the in-code default, which is exactly why the code default should always be the safe state for high-risk flags. Design the SDK initialization so a control-plane outage fails static on the safe variant rather than blocking startup or flapping, and keep a documented secondary path — a config-map override or an environment flag the deploy can set — for the case where you cannot reach the control plane at all.

Can I automate the kill switch instead of paging a human?

Yes, and for well-understood failure signals you should: wire an alert on the feature’s own error rate to a webhook that forces the safe variant automatically, so mitigation happens in seconds without a human in the loop. Keep the automation narrow — one flag, one clear signal, one safe variant — require the same audit metadata a human would supply, and always leave the manual runbook intact for the cases the automation was never scoped to cover.

Who is allowed to fire the kill switch, and how do I prevent misuse?

Scope kill-switch authority to the on-call rotation and incident commanders through a break-glass role, not to everyone with control-plane read access. Every forced override should carry an actor and a reason, alert a shared channel on use, and be reviewed in the incident retro — the goal is a fast path that is fully attributable after the fact, not an unlogged back door.

How is a kill switch different from a normal gradual rollback?

A gradual rollback walks a percentage down over minutes to limit churn and watch metrics between steps; a kill switch is the opposite — it forces the safe variant for 100% of traffic in one move because the feature is actively harmful and there is nothing to gain by easing off. Reserve the instant, all-traffic flip for genuine incidents, and use a staged rollback for the routine “this isn’t performing as hoped” cases.