Implementing Progressive Delivery Workflows

This guide is part of the Feature Flag Architecture & Lifecycle Management series. Progressive delivery decouples a code deployment from a user-facing release — the binary lands in production but only a fraction of traffic sees it, and that fraction grows only when measured signals stay healthy. Feature flags are the control knobs: they define who sees what variant, when exposure shifts, and what triggers an instant rollback without a redeploy.

This guide covers canary releases, ring deployments, percentage ramps, automated traffic shifting, and rollback triggers. It does not cover the mechanics of how flags are stored, versioned, or named — see designing a scalable flag taxonomy for that — nor the specifics of stale-flag retirement after a rollout completes, which is covered in managing flag deprecation and cleanup.

The reason this pattern earns its complexity is arithmetic: a defect shipped to 100% of traffic in a single deploy has a blast radius of every active user, and your mean time to detect is bounded only by how fast someone notices and pages someone else. Ramp the same defect to 1% first and the worst case is one user in a hundred hitting the fault while an automated monitor watches a per-variant error rate that will breach its threshold in seconds, not minutes. Progressive delivery does not make your code more correct — it makes the cost of being wrong proportional to how far you have ramped, and it makes that cost recoverable without a rollback deploy, a revert commit, or a pipeline run. Everything below is in service of that single property: keep exposure small until measured evidence says it is safe to widen it.

Prerequisites

• OpenFeature server SDK ≥ 1.x installed and connected to a provider (flagd or vendor)
• Flag definitions include owner, expiry, and defaultVariant per your flag taxonomy
• Evaluation telemetry (error rate, p95 latency, conversion) flowing to an observable sink
• Automated health checks or monitors that can call a flag-update API or webhook
• A documented propagation latency budget (determines how fast a rollback is visible)

Core Concept & Architecture

Progressive delivery sits at the intersection of deployment automation and flag evaluation. The deployment pipeline brings new code to every host simultaneously; the flag controls what percentage of requests activate it. This separates two previously coupled events — “code reaches production” and “users see the change” — and makes the second one a continuous, measurable process rather than a binary switch.

That separation has a consequence people underestimate: the new code path is live on every replica the moment the deploy completes, even when the flag resolves off for 99% of traffic. Both branches must be present, compiling, and correct simultaneously — the old behaviour that most requests still take and the new behaviour the ramp is exposing. This is why progressive delivery pairs naturally with trunk-based development and short-lived branches: you are no longer gating risk at merge time, you are gating it at evaluation time. It also means a flag that resolves off is not dead code. If the new branch allocates a connection pool, registers a background timer, or mutates shared state on module load, that cost is paid on every host regardless of the rollout percentage, so keep the expensive work behind the evaluation call, not beside it.

Three patterns dominate production use:

All three use the same underlying mechanism: a flag that maps a targetingKey to a variant based on a bucketing rule. The key requirement is stickiness — a given user must see the same variant on every request as the percentage climbs, across all replicas. The how-to for that is covered in percentage-based rollout with sticky bucketing.

The distinction between the three patterns is not academic — it changes what kind of risk you can catch. A canary catches defects that show up under real traffic shape: the malformed input a synthetic test never produces, the cache key collision that only appears at production concurrency, the third-party timeout that your staging mock never simulates. A ring deployment catches defects that are population-specific: a rendering bug that only affects users with a particular locale, a permission regression that only bites accounts on a legacy plan. A percentage ramp is the vehicle that carries you from one to the other — it is how a canary becomes a full release without a discontinuity. In practice you compose them: rings define who is eligible at each stage, and a percentage ramp governs how much of the eligible population is exposed within a stage. Treat them as orthogonal dials on the same flag rather than competing strategies, and the targeting rule in Step 3 falls out naturally.

Step-by-Step Implementation

The four steps build a self-driving ramp: define the fractional rule, gate each advance on guardrail metrics, layer ring targeting for population segmentation, and wire an automated rollback that fires without a human.

Step 1 — Define the flag with a percentage rollout rule

Start with 1% and make the ramp schedule explicit in flag metadata so operators know what phase they’re reading.

# flagd-format definition — namespace.service.feature key convention
flags:
  checkout.payments.express-pay:
    state: ENABLED
    variants: { "on": true, "off": false }
    defaultVariant: "off"
    targeting:
      fractionalEvaluation:
        - { "var": "targetingKey" }
        - ["on", 1]     # 1% get "on"
        - ["off", 99]   # 99% get "off"

Note the ordering of the fractionalEvaluation weights: they are relative shares, not cumulative thresholds, so ["on", 1] and ["off", 99] mean “one part in a hundred,” and to advance you change only those two integers. Keeping the schedule in metadata — a comment, a label, or a structured annotation the flag store supports — matters because the raw percentage tells an on-call engineer where the ramp is but not where it is headed. An operator who pages in at 3 a.m. and sees ["on", 8] needs to know whether 8% is the target hold point of an experiment or the fourth rung of a ladder to 100%, and those two situations call for opposite responses to the same metric wobble.

Pitfall: using a non-deterministic bucketing source (random UUID per request, server timestamp) breaks stickiness — a user can flip between variants on consecutive requests. Always derive the bucket from a stable identity attribute like userId or sessionId.

Pitfall: reusing the same targetingKey across two unrelated flags that ramp at the same time correlates their buckets — the same users land in the on cohort for both. If you need independent assignment, most providers let you salt the hash with the flag key so identical keys still bucket differently per flag; verify your provider does this rather than assuming it.

Step 2 — Gate progression on guardrail metrics

Automate the ramp: measure, compare to thresholds, and advance only when metrics are healthy. Never advance manually on a schedule without a metric check.

import { OpenFeature } from '@openfeature/server-sdk';

async function advanceRollout(flagKey: string, currentPct: number): Promise<void> {
  const errorRate = await metrics.query(`rate_5xx{flag="${flagKey}"}`);
  const p95Latency = await metrics.query(`p95_ms{flag="${flagKey}"}`);

  if (errorRate > 0.01 || p95Latency > 200) {
    await rollbackFlag(flagKey, currentPct);
    alerting.fire(`rollback.${flagKey}`, { errorRate, p95Latency });
    return;
  }

  const nextPct = Math.min(currentPct * 2, 100); // exponential: 1→2→4→8→16→32→64→100
  await flagAPI.setRolloutPercentage(flagKey, nextPct);
  console.log(`checkout.payments.express-pay advanced to ${nextPct}%`);
}

Two design choices in this loop deserve scrutiny. First, the bake time — the interval the ramp holds at each step before the check runs. Advance too fast and you accumulate no statistically meaningful sample at low percentages; a 1% cohort on a low-traffic endpoint might see a handful of requests per minute, and a single unlucky timeout would trip a naive threshold. Advance too slow and a genuinely broken release lingers in front of users longer than it needs to. Size the bake time so each step collects enough requests to make the guardrail comparison trustworthy — a few hundred requests per variant is a reasonable floor — and lengthen it, not shorten it, for the lower rungs where the sample is thin. Second, the comparison basis: querying an absolute threshold like errorRate > 0.01 assumes you know the healthy baseline in advance. A more robust gate compares the on variant against the concurrently-measured off variant, so a site-wide latency spike that hits both cohorts equally does not falsely roll back a blameless release.

Pitfall: measuring error rate globally rather than per-variant conflates control-group errors with variant errors. Attribute metrics to the resolved variant (available from the evaluation context) so the signal is clean.

Pitfall: exponential doubling is aggressive at the top of the ramp — 32% → 64% exposes an additional third of your users in one step, which is where a slow-burn defect that needed a larger sample to surface finally does, at the worst possible moment. Consider capping the multiplier or switching to linear increments above 50% so the final stretch to 100% stays gated rather than sprinting.

Step 3 — Configure ring deployments for user population segmentation

Ring deployments restrict early exposure to low-risk populations (internal employees, beta opt-ins) before exposing the general user base.

flags:
  checkout.payments.express-pay:
    state: ENABLED
    variants: { "on": true, "off": false }
    defaultVariant: "off"
    targeting:
      if:
        - { "in": [ { "var": "ring" }, ["internal", "beta"] ] }
        - "on"
        - { "fractionalEvaluation":
            - { "var": "targetingKey" }
            - ["on", 5]
            - ["off", 95] }

This rule resolves on for internal/beta users unconditionally, and gives 5% of the general population the new variant. Promote to the next ring by updating the fractionalEvaluation percentages after the internal ring shows no regressions. The rule reads top to bottom: the if short-circuits, so the fractional evaluation is only reached by requests whose ring is neither internal nor beta. That ordering is deliberate — it guarantees your dogfooding population always sees the newest variant regardless of what the general-traffic percentage is doing, which is exactly what you want when internal users are your fastest, highest-signal bug reporters.

The ring attribute itself has to come from somewhere trustworthy. Deriving it from an employee SSO claim or a signed beta-enrollment token is safe; deriving it from a client-supplied header or query parameter is not, because any user can then promote themselves into the internal ring and see unfinished features. Treat ring membership as an authorization decision and source it server-side from an attribute you already trust for access control.

Pitfall: defining rings as separate flags rather than targeting rules in one flag creates attribution confusion — you can’t compare variant outcomes across rings when the flag key differs.

Pitfall: forgetting that ring members are a biased sample. Internal users click through flows faster, tolerate rough edges, and rarely exercise the edge cases that real customers hit — a clean internal ring is necessary but not sufficient evidence to widen the general percentage. Weight the general-population canary metrics more heavily than the internal ring when deciding to advance.

Step 4 — Wire automated rollback triggers

The kill switch for a rollout should fire without human intervention when a guardrail breaches. Wire a webhook from your alerting system to a flag-update endpoint.

# FastAPI webhook: automated rollback on SLO breach
from fastapi import FastAPI, Request
import httpx, os

app = FastAPI()

@app.post("/webhook/slo-alert")
async def rollback_on_breach(request: Request):
    payload = await request.json()
    flag_key = payload.get("labels", {}).get("flag_key", "")
    metric = payload.get("metric", "")
    value = payload.get("value", 0.0)

    if flag_key == "checkout.payments.express-pay" and metric == "error_rate" and value > 0.01:
        async with httpx.AsyncClient() as http:
            await http.patch(
                f"https://flags.internal/v1/flags/{flag_key}/rollout",
                json={"percentage": 0},
                headers={"Authorization": f"Bearer {os.environ['FLAG_API_TOKEN']}"},
            )
        return {"status": "rolled_back", "flag": flag_key}
    return {"status": "ignored"}

Two properties make an automated rollback safe to trust with no human in the loop. It must be idempotent — a flapping alert that fires the webhook five times in ten seconds must converge on percentage 0, not thrash — and the PATCH above is naturally idempotent because it sets an absolute value rather than decrementing. It must also be debounced against its own recovery: once rolled back, the ramp automation should not immediately re-advance because the error rate dropped the instant traffic drained off the broken variant. Gate re-entry behind a human acknowledgement or a cool-off window, otherwise you build an oscillator that ramps into the fault, rolls back, sees green, and ramps straight back in.

Guard the webhook endpoint itself, too. A POST that can zero out any flag’s rollout is a denial-of-service primitive if it is unauthenticated — verify the alert payload’s signature (most alerting systems sign webhooks with a shared secret) and constrain which flag keys a given alert source is allowed to touch, so a misconfigured monitor cannot roll back an unrelated critical flag.

Pitfall: a rollback that sets the percentage to 0 is not the same as forcing the safe variant. If the SDK default in code differs from defaultVariant, behaviour is ambiguous. Force the explicit variant — see the emergency kill-switch runbook for the production-safe approach.

Verification & Testing

After each ring advance, confirm both the exposure percentage and the metric signal:

# Confirm fraction of requests seeing "on" matches the target percentage
flagctl get checkout.payments.express-pay --env prod -o json | jq '.targeting'

# Spot-check variant resolution across replicas
for host in $(cat replicas.txt); do
  curl -s "$host/debug/flags/checkout.payments.express-pay" | jq -r '.variant'
done | sort | uniq -c
# expect ~1 out of every 100 lines to show "on" at 1% rollout

The sort | uniq -c histogram is the cheapest sanity check you have, and it catches two distinct failure modes at once. If no replica ever returns on, the targeting rule did not deploy or the flag is disabled. If the observed fraction is wildly off target — 20% resolving on when you set 1% — the bucketing input is not uniformly distributed, which usually means the targetingKey you sampled is skewed (all your test identities happen to hash into the same slice) rather than the percentage being wrong. Sample with real, varied keys, not a loop over emp-1 through emp-10, or the histogram lies to you.

For ring deployments, also verify that internal users resolve on regardless of percentage:

curl -s -X POST http://flagd.internal:8013/schema.v1.Service/ResolveBoolean \
  -H 'Content-Type: application/json' \
  -d '{"flagKey":"checkout.payments.express-pay","context":{"targetingKey":"emp-42","ring":"internal"}}' \
  | jq '.value'   # must be true

Troubleshooting & FAQ

Why do users flip between variants as the percentage increases?

The bucketing hash is not stable. Either the targetingKey changes between requests (re-generated session ID, anonymous → logged-in transition) or the provider is not using a deterministic hash. Confirm the key is stable for the session lifetime and that the provider uses a repeatable hashing function. See percentage-based rollout with sticky bucketing for the full fix.

My rollback webhook fired, but some replicas still serve the new variant — why?

The flag change has not propagated to all replicas yet. Propagation speed depends on your sync transport: streaming delivers within a second, polling within one interval. Check the sync connection state on lagging replicas and verify the transport is healthy. For transport selection guidance see polling vs streaming flag synchronization.

How do I attribute a metric regression to a specific variant without changing the data model?

Resolve the variant in-process and attach it as a structured attribute on the trace or log event at the point of the flag call. Most OpenFeature hooks let you tap into the afterEvaluation hook to add the resolved variant to the active span without adding a separate network call.

Should I use one flag per ring or one flag with ring-based targeting?

One flag. Multiple flags for the same feature make cross-ring attribution difficult: you can’t compare variant metrics when the flag key differs, and the audit trail is fragmented. Keep all ring rules in a single flag’s targeting configuration and promote by updating that configuration.

How long should I hold at each percentage before advancing?

Long enough that each variant accumulates a statistically meaningful sample — a few hundred requests per variant is a practical floor — and long enough to span at least one full cycle of the behaviour you’re watching. If a defect only manifests on an hourly batch job or a cache that expires every fifteen minutes, a two-minute bake time will wave it through. Hold the low rungs longer than the high ones: at 1% your sample is thinnest and your evidence weakest, so that is precisely where haste costs you.

What is the difference between progressive delivery and a blue-green deployment?

Blue-green swaps all traffic from an old environment to a new one at once, so it is a binary cutover with a fast rollback but no graduated exposure — every user flips together. Progressive delivery holds a single environment and shifts a fraction of traffic onto the new code path, growing that fraction only as metrics stay healthy. Blue-green isolates risk by infrastructure; progressive delivery isolates it by measured audience size, which lets you catch a defect at 1% rather than discovering it after a 100% flip.

Can I run a progressive rollout and an A/B experiment on the same feature at once?

Not on the same flag without care, because the two have conflicting goals: a rollout wants to reach 100% as fast as guardrails allow, while an experiment wants to hold a fixed split long enough to reach statistical significance. Run the experiment on a stable percentage first, decide the winner, and only then start ramping the winning variant to 100%. If you must overlap them, keep them on separate flags with independent targeting keys so the rollout’s changing percentage does not disturb the experiment’s fixed allocation. See experimentation and A/B testing guardrails for the split-stability requirement.

How do I ramp a flag that has more than two variants?

Fractional evaluation takes any number of weighted variants, so a three-way rollout is [["a", 1], ["b", 1], ["control", 98]] and you widen by adjusting the weights. The complication is your guardrail logic: each treatment variant needs its own per-variant metric series and its own rollback threshold, because one arm can regress while another is healthy. Roll back only the offending variant by zeroing its weight and redistributing to control, rather than aborting the whole rollout, so a single bad arm does not cost you the progress the others earned.

Performance & Scale Considerations

Percentage evaluation adds negligible overhead — it is a deterministic hash and a modulo comparison. The cost that scales with ring complexity is the evaluation context assembly: building the ring attribute requires a lookup or claim parse, which should happen once per request boundary and be cached on the request context, not re-fetched per flag call.

At high replica counts, the exponential ramp schedule (1% → 2% → 4% → 8% …) naturally limits blast radius: a defect at 1% affects 1 in 100 users, giving early warning before any significant population is exposed. For experimentation and A/B testing guardrails, the same ramp structure applies — the difference is that experiment variants hold at a fixed percentage long enough to accumulate statistical significance rather than ramping to 100%.

The scaling factor that actually bites is not evaluation CPU but change propagation and metric fan-out. Every percentage advance is a flag mutation that must reach every replica, and every replica emits per-variant telemetry that your metric backend must ingest and aggregate before the guardrail query can answer. At a few dozen replicas this is invisible; at several thousand, a naive design that queries the metric store on a tight loop and pushes flag updates individually to each host becomes the bottleneck, and your rollback latency is dominated by that fan-out rather than by the flag evaluation itself. This is where the streaming-versus-polling transport choice stops being a preference and becomes the thing that determines whether your automated rollback lands in one second or one polling interval — quantify it against your propagation budget, and see polling vs streaming flag synchronization for how the two transports trade freshness against connection overhead.

One more cost is easy to miss: the cardinality your per-variant telemetry adds. Tagging every metric with the flag key and resolved variant multiplies your time-series count by the number of variants for every metric you attribute, and if you also tag by ring you multiply again. On a metrics backend billed by active series, an enthusiastic rollout instrumented across dozens of flags can balloon cardinality without anyone noticing until the bill or the query latency does. Attribute the variant on the handful of metrics your guardrails actually read — error rate, p95, the primary conversion signal — not on every counter in the service.