Multi-Environment Flag Promotion Pipelines

This guide is part of the Feature Flag Architecture & Lifecycle Management series.

Flag configuration is code. The same discipline you apply to application releases — version control, schema validation, peer review, automated gates — must govern how feature flags travel from a developer’s branch through staging into production. Without a formal promotion pipeline, flag state becomes a source of invisible coupling: a percentage rollout misconfigured in production, a targeting rule that passed manual QA in staging but was never re-verified after the merge, or a flag key that quietly diverged across environments until it caused an incident.

This guide shows you how to build a deterministic, auditable promotion pipeline using Git as the source of truth, CI gates for schema validation and dry-run evaluation, and pull-request merges as the sole promotion mechanism. The pattern integrates with OpenFeature-compatible backends and pairs directly with the progressive delivery workflow you may already have in place for gradual rollouts.

What this guide covers: Structuring flag configuration as environment-scoped YAML files, running schema and evaluation gates in CI, performing GitOps promotions via PR, and post-promotion verification.

What this guide does NOT cover: Real-time flag evaluation logic, SDK integration patterns, A/B test statistical analysis, or runtime kill-switch mechanisms. For flag naming and taxonomic organisation, see designing a scalable flag taxonomy and the naming conventions reference.

Prerequisites

• Git repository with branch protection enabled on main; staging promotion target is a long-lived staging branch or directory-based environment separation
• CI/CD system with YAML pipeline support (GitHub Actions shown; adapt for GitLab CI, Buildkite, or similar)
• flagd installed locally or available as a container image (ghcr.io/open-feature/flagd:latest) for dry-run evaluation
• A JSON Schema file describing your flag definition format (starter schema provided in Step 2)
• jq and curl available in CI runners
• RBAC configured in your Git host: a flag-promoters team or role with merge rights to production config paths; developers restricted to flags/dev/ and PR authorship only
• Familiarity with the flag taxonomy and key namespace schema used by your organisation

Core Concept & Architecture

A promotion pipeline treats each environment’s flag configuration as a discrete artefact that must pass validation gates before it can advance. The pipeline is unidirectional: config flows dev → staging → production, never sideways or backwards outside of an explicit rollback commit.

Key structural properties

The pipeline does not replace your flag evaluation backend — it governs the configuration files that the backend reads. When flagd (or your equivalent provider) polls for updated configuration, it reads from the file path corresponding to its deployed environment, which is the file that just passed validation and was merged.

Step-by-Step Implementation

Step 1 — Define per-environment flag config as code

Organise your flag definitions into a directory tree rooted at flags/. A _base/ directory holds defaults that apply everywhere; per-environment directories override only the fields that differ for that environment.

flags/
├── _base/
│   └── platform.billing.yaml          # canonical definition, defaults
├── dev/
│   └── platform.billing.yaml          # dev overrides: all users ON
├── staging/
│   └── platform.billing.yaml          # staging: 50% rollout, test accounts
└── prod/
    └── platform.billing.yaml          # prod: 5% rollout, strict targeting

flags/_base/platform.billing.yaml — the canonical flag definition with conservative defaults:

# Base flag definition — inherited by all environments unless overridden.
# Key follows namespace.service.feature schema.
flags:
  platform.billing.new-invoicing:
    state: DISABLED
    variants:
      "on": true
      "off": false
    defaultVariant: "off"
    description: >
      Enables the redesigned invoicing engine for the billing service.
      Toggle independently of UI feature flags.
    owner: team-billing
    tags:
      - billing
      - invoicing
    targeting: {}

flags/dev/platform.billing.yaml — development environment enables the flag for all users so engineers can iterate without targeting rules:

flags:
  platform.billing.new-invoicing:
    state: ENABLED
    defaultVariant: "on"
    targeting: {}

flags/staging/platform.billing.yaml — staging runs a 50% fractional rollout scoped to internal test accounts, verifying the flag evaluation logic before any real traffic is affected:

flags:
  platform.billing.new-invoicing:
    state: ENABLED
    defaultVariant: "off"
    targeting:
      if:
        - in:
            - var: account_type
            - ["internal", "beta"]
        - "on"
        - fractionalEvaluation:
            - "platform.billing.new-invoicing"
            - - - "on"
                - 50
              - - "off"
                - 50

flags/prod/platform.billing.yaml — production starts at 5% and is gated to non-enterprise accounts only, limiting blast radius:

flags:
  platform.billing.new-invoicing:
    state: ENABLED
    defaultVariant: "off"
    targeting:
      if:
        - and:
            - "!":
                - in:
                    - var: account_type
                    - ["enterprise"]
            - fractionalEvaluation:
                - "platform.billing.new-invoicing"
                - - - "on"
                    - 5
                  - - "off"
                    - 95
        - "on"
        - "off"

A merge script (used in Step 3) deep-merges base into the environment file at pipeline time, so the final artefact sent to flagd is complete. Store only the delta in environment files — do not duplicate unchanged fields.

Pitfall: If you store complete definitions in every environment file, divergence is guaranteed. A field updated in _base/ will not propagate to files that shadow it entirely. Always store only overrides in per-environment files and merge at pipeline time.

Step 2 — Add CI schema validation and dry-run gate

Every push that modifies a file under flags/ triggers two checks before any merge is allowed: structural schema validation and a behavioural dry-run against synthetic evaluation contexts.

JSON Schema for flag definitions — save as flags/.schema/flag-definition.schema.json:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "FlagDefinitionFile",
  "type": "object",
  "required": ["flags"],
  "additionalProperties": false,
  "properties": {
    "flags": {
      "type": "object",
      "patternProperties": {
        "^[a-z][a-z0-9-]+\\.[a-z][a-z0-9-]+\\.[a-z][a-z0-9-]+$": {
          "type": "object",
          "required": ["state", "variants", "defaultVariant"],
          "properties": {
            "state": { "type": "string", "enum": ["ENABLED", "DISABLED"] },
            "variants": { "type": "object", "minProperties": 1 },
            "defaultVariant": { "type": "string" },
            "description": { "type": "string" },
            "owner": { "type": "string" },
            "tags": { "type": "array", "items": { "type": "string" } },
            "targeting": { "type": "object" }
          },
          "additionalProperties": false
        }
      },
      "additionalProperties": false
    }
  }
}

GitHub Actions workflow — .github/workflows/flag-promotion-gate.yml:

name: Flag Promotion Gate

on:
  pull_request:
    paths:
      - "flags/**"

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install tooling
        run: |
          sudo apt-get install -y jq python3-pip
          pip3 install check-jsonschema

      - name: Detect changed environment directories
        id: changed
        run: |
          git fetch origin ${{ github.base_ref }}
          DIRS=$(git diff --name-only origin/${{ github.base_ref }}...HEAD \
            | grep '^flags/' \
            | awk -F/ '{print $1"/"$2}' \
            | sort -u \
            | grep -v '_base')
          echo "dirs=$DIRS" >> "$GITHUB_OUTPUT"

      - name: Merge base + env overrides into merged artefacts
        run: |
          mkdir -p /tmp/merged
          for envdir in ${{ steps.changed.outputs.dirs }}; do
            env=$(basename "$envdir")
            for basefile in flags/_base/*.yaml; do
              key=$(basename "$basefile")
              envfile="flags/${env}/${key}"
              if [ -f "$envfile" ]; then
                python3 - <<'PYEOF'
          import yaml, sys, copy
          base = yaml.safe_load(open("$basefile"))
          override = yaml.safe_load(open("$envfile"))
          merged = copy.deepcopy(base)
          for flag, cfg in override.get("flags", {}).items():
              if flag in merged["flags"]:
                  merged["flags"][flag].update(cfg)
              else:
                  merged["flags"][flag] = cfg
          yaml.dump(merged, open(f"/tmp/merged/${env}-${key}", "w"))
          PYEOF
              else
                cp "$basefile" "/tmp/merged/${env}-${key}"
              fi
            done
          done

      - name: Schema validation
        run: |
          EXIT=0
          for f in /tmp/merged/*.yaml; do
            echo "Validating $f"
            check-jsonschema --schemafile flags/.schema/flag-definition.schema.json "$f" || EXIT=1
          done
          exit $EXIT

      - name: flagd dry-run evaluation
        run: |
          docker pull ghcr.io/open-feature/flagd:latest

          EXIT=0
          for f in /tmp/merged/*.yaml; do
            echo "Dry-run: $f"

            # Write synthetic evaluation contexts
            cat > /tmp/ctx.json <<'JSON'
          [
            {"account_type": "internal", "user_id": "test-001"},
            {"account_type": "enterprise", "user_id": "test-002"},
            {"account_type": "standard", "user_id": "test-003"}
          ]
          JSON

            docker run --rm \
              -v "$f:/etc/flagd/flags.yaml:ro" \
              -v "/tmp/ctx.json:/etc/flagd/ctx.json:ro" \
              ghcr.io/open-feature/flagd:latest \
              flagd evaluate \
                --uri "file:/etc/flagd/flags.yaml" \
                --context-file /etc/flagd/ctx.json \
              | tee /tmp/eval-output.json

            # Assert no "ERROR" variants appear in dry-run output
            if jq -e '.[] | select(.reason == "ERROR")' /tmp/eval-output.json > /dev/null 2>&1; then
              echo "ERROR: flagd returned ERROR reason for one or more contexts in $f"
              EXIT=1
            fi
          done
          exit $EXIT

Pitfall: The flagd evaluate dry-run mode exposes mis-typed variant names and broken JsonLogic targeting rules that JSON Schema cannot catch — JSON Schema validates structure, not semantics. Both gates are required; one without the other leaves a class of production-breaking defects undetected.

Step 3 — Promote via PR / GitOps merge

Promotion from staging to production is a pull request that copies the validated staging config into the production directory. There is no automated push to prod — the PR is the gate, and only members of the flag-promoters team can merge it.

Promotion script — scripts/promote-flags.sh:

#!/usr/bin/env bash
# promote-flags.sh
# Usage: ./scripts/promote-flags.sh <source-env> <target-env> [flag-key-pattern]
# Example: ./scripts/promote-flags.sh staging prod platform.billing

set -euo pipefail

SOURCE="${1:?Source environment required (e.g. staging)}"
TARGET="${2:?Target environment required (e.g. prod)}"
PATTERN="${3:-}"   # Optional: restrict to files matching glob pattern

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
FLAGS_DIR="${REPO_ROOT}/flags"

SOURCE_DIR="${FLAGS_DIR}/${SOURCE}"
TARGET_DIR="${FLAGS_DIR}/${TARGET}"

if [ ! -d "$SOURCE_DIR" ]; then
  echo "ERROR: Source directory not found: $SOURCE_DIR" >&2
  exit 1
fi

echo "=== Flag promotion: ${SOURCE} → ${TARGET} ==="
echo ""

# Show diff before promoting so the promoter knows exactly what will change
echo "--- Diff (${SOURCE} → ${TARGET}) ---"
if [ -n "$PATTERN" ]; then
  FILES=$(find "$SOURCE_DIR" -name "${PATTERN}*.yaml" -type f)
else
  FILES=$(find "$SOURCE_DIR" -name "*.yaml" -type f)
fi

CHANGED=0
for src_file in $FILES; do
  filename=$(basename "$src_file")
  tgt_file="${TARGET_DIR}/${filename}"

  if [ -f "$tgt_file" ]; then
    if ! diff -u "$tgt_file" "$src_file"; then
      CHANGED=$((CHANGED + 1))
    fi
  else
    echo "(NEW) $filename — not present in ${TARGET}, will be added"
    CHANGED=$((CHANGED + 1))
  fi
done

if [ "$CHANGED" -eq 0 ]; then
  echo "No differences detected between ${SOURCE} and ${TARGET}. Nothing to promote."
  exit 0
fi

echo ""
echo "${CHANGED} file(s) will change in ${TARGET}."
read -r -p "Proceed with promotion? [y/N] " confirm
if [[ ! "$confirm" =~ ^[Yy]$ ]]; then
  echo "Promotion aborted."
  exit 1
fi

# Copy source files into target directory
mkdir -p "$TARGET_DIR"
for src_file in $FILES; do
  filename=$(basename "$src_file")
  cp "$src_file" "${TARGET_DIR}/${filename}"
  git -C "$REPO_ROOT" add "${TARGET_DIR}/${filename}"
done

# Propose a commit message and branch name
BRANCH="promote/${SOURCE}-to-${TARGET}/$(date +%Y%m%d-%H%M%S)"
git -C "$REPO_ROOT" checkout -b "$BRANCH"
git -C "$REPO_ROOT" commit -m "feat(flags): promote ${PATTERN:-all} flags from ${SOURCE} to ${TARGET}

Automated promotion via promote-flags.sh.
Diff reviewed and confirmed by $(git config user.name).
Run post-promotion verification after merge: scripts/verify-promotion.sh ${TARGET}"

echo ""
echo "Branch '${BRANCH}' committed. Push and open a PR targeting main (or your ${TARGET} config branch)."
echo "Only flag-promoters team members may merge this PR."

Restrict the flags/prod/ path in your Git host’s CODEOWNERS file:

# .github/CODEOWNERS
flags/prod/    @your-org/flag-promoters
flags/staging/ @your-org/flag-promoters @your-org/developers
flags/dev/     @your-org/developers

Pitfall: Without CODEOWNERS (or an equivalent path-based approval requirement), any developer with repository write access can self-approve a PR that modifies production flag config. RBAC at the file-path level is the enforcement boundary — branch protection rules alone are insufficient if they allow the PR author to be their own approver.

Step 4 — Verify post-promotion state

After the promotion PR merges and flagd reloads its configuration (typically within one poll interval, default 60 seconds), run a verification script that queries each environment’s flagd endpoint and asserts the resolved variant matches the expected post-promotion value.

scripts/verify-promotion.sh:

#!/usr/bin/env bash
# verify-promotion.sh
# Usage: ./scripts/verify-promotion.sh <env> <flag-key> <expected-variant>
# Example: ./scripts/verify-promotion.sh prod platform.billing.new-invoicing "off"
#
# Requires environment variables:
#   FLAGD_PROD_URL   e.g. http://flagd.internal.prod:8013
#   FLAGD_STAGING_URL
#   FLAGD_DEV_URL

set -euo pipefail

ENV="${1:?Environment required}"
FLAG_KEY="${2:?Flag key required}"
EXPECTED_VARIANT="${3:?Expected variant required}"

ENV_UPPER="${ENV^^}"
FLAGD_URL_VAR="FLAGD_${ENV_UPPER}_URL"
FLAGD_URL="${!FLAGD_URL_VAR:?Variable ${FLAGD_URL_VAR} must be set}"

echo "=== Post-promotion verification ==="
echo "Environment : ${ENV}"
echo "Flag key    : ${FLAG_KEY}"
echo "Expected    : ${EXPECTED_VARIANT}"
echo "Endpoint    : ${FLAGD_URL}"
echo ""

# Test with multiple synthetic contexts representative of each targeting branch
CONTEXTS=(
  '{"account_type":"internal","user_id":"verify-internal-001"}'
  '{"account_type":"standard","user_id":"verify-standard-001"}'
  '{"account_type":"enterprise","user_id":"verify-enterprise-001"}'
)

PASS=0
FAIL=0

for ctx in "${CONTEXTS[@]}"; do
  RESPONSE=$(curl -sf \
    -X POST \
    -H "Content-Type: application/json" \
    -d "{\"flagKey\":\"${FLAG_KEY}\",\"context\":${ctx}}" \
    "${FLAGD_URL}/schema.v1.Service/ResolveBoolean" 2>&1) || {
      echo "FAIL: flagd endpoint unreachable or returned error for context ${ctx}"
      FAIL=$((FAIL + 1))
      continue
    }

  RESOLVED=$(echo "$RESPONSE" | jq -r '.variant // empty')
  REASON=$(echo "$RESPONSE" | jq -r '.reason // "UNKNOWN"')

  if [ "$RESOLVED" = "$EXPECTED_VARIANT" ]; then
    echo "PASS [${REASON}] context=${ctx} variant=${RESOLVED}"
    PASS=$((PASS + 1))
  else
    echo "FAIL context=${ctx} expected=${EXPECTED_VARIANT} got=${RESOLVED} reason=${REASON}"
    FAIL=$((FAIL + 1))
  fi
done

echo ""
echo "Results: ${PASS} passed, ${FAIL} failed"

if [ "$FAIL" -gt 0 ]; then
  echo "ERROR: Post-promotion verification failed. Check flag configuration and flagd reload status."
  echo "For ongoing drift detection, see the drift detection guide:"
  echo "  /feature-flag-architecture-lifecycle-management/multi-environment-flag-promotion-pipelines/detecting-flag-configuration-drift-across-environments/"
  exit 1
fi

echo "Verification passed. Flag is serving expected variant in ${ENV}."

Run this script as a post-deployment step in your CD pipeline immediately after flagd restarts or the config reload is confirmed. For environments where flagd polls rather than receiving a push notification, add a wait loop before verification:

# Wait for flagd to reload (poll interval default: 60s; adjust to your config)
echo "Waiting for flagd to reload configuration..."
until curl -sf "${FLAGD_URL}/healthz" | jq -e '.status == "SERVING"' > /dev/null 2>&1; do
  sleep 5
done
echo "flagd ready."
./scripts/verify-promotion.sh prod platform.billing.new-invoicing "off"

For automated drift detection running on a schedule — catching divergence that can accumulate after manual hotfixes or out-of-band changes — see detecting flag configuration drift across environments.

Pitfall: Verifying only the default variant misses targeting-rule failures. A flag whose defaultVariant is "off" may still erroneously serve "on" to users who match a broken targeting rule. Test at least one context per targeting branch in your verification suite, not just an uncategorised context.

Verification & Testing

Run the full validation stack locally before pushing a branch:

# 1. Merge base + env overrides for the environment you are modifying
python3 scripts/merge-flags.py --env staging --output /tmp/merged-staging/

# 2. Schema validation
check-jsonschema \
  --schemafile flags/.schema/flag-definition.schema.json \
  /tmp/merged-staging/*.yaml

# 3. flagd dry-run
docker run --rm \
  -v /tmp/merged-staging/platform.billing.yaml:/etc/flagd/flags.yaml:ro \
  ghcr.io/open-feature/flagd:latest \
  flagd evaluate \
    --uri "file:/etc/flagd/flags.yaml" \
    --context '{"account_type":"internal","user_id":"local-test"}'

# 4. Promotion diff (before opening PR)
./scripts/promote-flags.sh staging prod platform.billing
# Review the diff output — do not confirm; exit without committing when running locally to inspect

Add integration tests that import your flag YAML directly into a test harness and assert variant output for a table of contexts. This catches targeting rule regressions before the schema validation layer, which is purely structural.

Troubleshooting & FAQ {#faq}

What happens if a schema validation gate fails mid-promotion? {#faq-schema-failure}

The CI job exits non-zero and the PR is blocked from merging. No partial state is applied — Git-based promotion is atomic at the PR level. Fix the schema error in the source environment’s YAML, push to the same branch, and the gate re-runs automatically. Common causes: a flag key that does not match the namespace.service.feature pattern (check for uppercase letters or underscores), a defaultVariant value that references a variant name not present in the variants map, or a targeting rule that references a non-existent context attribute name (caught by dry-run, not schema validation). Check the CI log for the specific check-jsonschema output — it includes the JSON Pointer to the failing field.

How do environment overrides interact with percentage rollouts? {#faq-percentage-rollouts}

The merge step (Step 1) deep-merges the targeting object from the environment file on top of the base definition. If the base file defines no targeting and the staging file defines a 50% fractional rollout, the merged staging artefact carries that rollout rule. When promoting to prod, the prod file should define its own targeting block (for example, 5% rollout) — it will replace the staging targeting block entirely, not add to it. Do not rely on partial targeting overrides that depend on base targeting being present; write each environment’s targeting block as a self-contained rule. If you need to inherit a base targeting rule and extend it, flatten the composition in the base merge script rather than relying on implicit inheritance of nested objects. For percentage rollouts that interact with progressive delivery stages, treat each stage transition as a new promotion with a new PR.

Who should have permission to promote flags to production? {#faq-rbac}

The flag-promoters role should be a small, named group — typically the senior engineer on the owning team plus a rotation of on-call engineers — not a broad “developers” group. Treat production flag promotion with the same access control as a production deployment: require a second approver from outside the authoring team for flags that affect billing, authentication, or data-plane behaviour. For flags attached to stale or deprecated features, the promoter should confirm that the deprecation owner has signed off before promoting a permanent-on or permanent-off state to production. The audit trail — Git commit history with author and approver metadata — is your evidence record for compliance reviews.

Performance & Scale

Large flag inventories. When the flags/ directory contains hundreds of files, CI validation time grows linearly. Scope the validation job to changed files only (the workflow in Step 2 already does this via git diff). Do not re-validate unchanged environment files on every PR.

Monorepo layouts. If flag config lives alongside application code, use path filters on your CI trigger (paths: ["flags/**"]) to avoid running the promotion gate on every commit. Consider separating flag config into its own repository if it receives more than a few promotions per day — the noise in a shared repo’s PR list increases coordination costs.

flagd reload latency. flagd polls its file source on a configurable interval. For time-sensitive promotions (incident response, kill-switch activation), configure the poll interval to 10–15 seconds in staging and prod. For routine promotions, 60 seconds is adequate. The post-promotion verification script should wait for at least one full poll cycle before asserting state.

Environment count. The two-gate pipeline (dev → staging → prod) covers most organisations. If you operate additional environments (QA, pre-prod, canary), add intermediate directories and gate jobs following the same pattern. Do not add environments without adding corresponding validation gates — an ungated environment breaks the invariant that every artefact reaching production has been schema-validated and dry-run evaluated.

Rollback speed. A rollback is a Git revert commit. Because the pipeline re-runs on every merge, the revert commit will trigger another CI run before it can merge. For emergency rollbacks where CI speed matters, pre-stage a revert PR immediately after any high-risk production promotion so that the CI run can complete in parallel with your observation window. If the promotion causes an incident, merge the pre-staged revert PR immediately — the CI should already be green.