Backend Evaluation & Server-Side SDKs

Server-side evaluation keeps targeting logic, segmentation rules, and sensitive context inside your trust boundary, resolving flags deterministically before a byte reaches the client. This overview owns the backend half of a controlled rollout system: how a server-side SDK bootstraps, how it assembles an evaluation context, how rules are compiled and cached for sub-millisecond resolution, and how the whole path stays safe when the control plane degrades.

The reason this half of the system earns its own treatment is that a flag read is not a normal function call — it sits on the hottest path in your service, inside request handlers that run thousands of times a second, and it depends on state that is owned by an external system you do not control. Every design choice below flows from that tension: you want the freshness of a remote source of truth and the latency and availability of a local computation, and you cannot have both at once. The engineering is in deciding, per flag and per service, where on that spectrum to sit, and then building the caching, synchronization, and fallback machinery that makes the compromise invisible to the request. Get it right and a flag read costs a few microseconds and never fails; get it wrong and you have coupled your checkout path to the uptime of a configuration service.

Architecture Overview & Client-Server Boundaries

Strict client-server boundaries keep sensitive targeting logic out of untrusted environments. Backend evaluation guarantees deterministic resolution while minimizing frontend payload — the browser never sees the rule tree, only the resolved variant. The first architectural decision is where resolution happens. Stateless (remote) evaluation calls a control-plane endpoint per request and trades network latency for zero local state; stateful (local) evaluation downloads the rule set once and resolves in-process for sub-millisecond reads. Most production services choose local evaluation and treat the network only as a sync channel, because a per-request round trip couples every flag read to the availability and latency of an external service — a coupling that turns a flag outage into an application outage.

That boundary decision also shapes your blast radius. When rules live in-process, a control-plane incident degrades freshness (you keep serving the last-known-good rule set) rather than availability (you stop resolving entirely). The trade-off is memory and warm-up: the service must hold the compiled rule set and reach a ready state before it accepts traffic, which is why readiness probes and warm-up ordering matter as much as the evaluation call itself.

OpenFeature standardizes the provider abstraction so application code calls one vendor-neutral API while the provider behind it can be flagd, a SaaS vendor, or a custom backend. A declarative flag definition is the contract between the control plane and the engine:

# flagd-format definition — keys use namespace.service.feature
flags:
  checkout.payments.express-pay:
    state: ENABLED
    variants: { "on": true, "off": false }
    defaultVariant: "off"
    targeting:
      if:
        - { "==": [ { var: "tenantTier" }, "enterprise" ] }
        - "on"
        - "off"

Keeping the definition declarative is what lets the same artifact drive both the runtime engine and the CI validation gate. The rule tree is data, not code, so it can be diffed, linted, and dry-run evaluated before it is ever loaded by a live provider — the foundation for the governance and integration practices below.

There is a subtler benefit to the data-not-code framing that only shows up under load. Because the targeting expression is a structured tree rather than an arbitrary predicate, the engine can compile it once into an executable form — an abstract syntax tree or a flattened bytecode — and reuse that compilation across every evaluation until the definition changes. A per-request interpreter that re-parses JSON logic on every call will happily consume a millisecond of CPU per flag; the same rule set compiled ahead of time resolves in tens of microseconds. That gap is why the rule-engine performance work below treats compilation as a first-class step rather than an optimization to reach for later, and why a service that reads dozens of flags per request can still hold a single-digit-millisecond p99. The declarative contract is not a convenience for humans reviewing diffs — it is the property that makes fast evaluation mechanically possible.

Lifecycle & Governance

A structured lifecycle enforces governance across development, staging, and production. Creation requires schema validation; rollout phases mandate gradual exposure thresholds; retirement is driven by usage telemetry rather than guesswork. Every flag should carry metadata that makes its stage machine-readable: an owner, a type (release, ops, experiment, or permission), a creation date, and an expiry or review date. Without that metadata, flags accumulate silently and the rule set that the engine compiles grows without bound, quietly inflating evaluation cost and audit surface.

Promotion between environments should be automated and diffable — see multi-environment flag promotion pipelines for drift detection and GitOps gates, and designing a scalable flag taxonomy for the metadata schema that makes ownership and expiry enforceable. Mandatory deprecation windows and a scheduled flag cleanup pass prevent the configuration sprawl that turns a healthy rule set into a liability.

Governance is most effective when it is enforced by the same pipeline that ships the flags rather than by convention. A pull request that adds a flag should fail CI unless the flag carries its required metadata — an owner who can be paged, a type that classifies its blast radius, and a review date that turns “temporary” into a tracked commitment. The classification matters more than it appears: a release flag that gates an unfinished feature and an operational kill switch that disables a payment path have opposite safe defaults and opposite retirement timelines, and conflating them is how a cleanup script deletes the switch you needed during an incident. Encoding the type in metadata lets the evaluation strategy and the operational safety layers treat each flag according to what it actually controls.

The other half of lifecycle discipline is measuring decay. A flag that has resolved to the same variant for every evaluation over the last month is almost certainly dead code wearing a runtime disguise, and it costs you twice: once in the rule set the engine compiles on every sync, and again in the cognitive load of every engineer who has to reason about a branch that never branches. Track a staleness signal per flag — age, time since last change, and evaluation skew toward a single variant — and route the worst offenders to their owners for removal. Turning that decay into a single, trackable number is what converts “we should clean up flags someday” into a backlog that actually shrinks.

Retirement is also where most teams underestimate the risk, because deleting a flag is a code change with the same blast radius as adding one. Removing the flag key from the definition is the easy part; removing the branch it guarded is where regressions hide. If the losing side of the branch was never exercised in production — because the flag shipped at 100% for months — then deleting it is the first time that code path runs for real users, and any latent bug in the “cleanup” is now a production incident with no flag to roll it back. Treat flag removal as a normal, reviewed, gradually rolled-out change: land the code deletion behind its own short-lived flag if the branch is load-bearing, watch the same guardrails you would for a feature, and only then retire the guarding flag once the simplified path has proven itself. A cleanup that skips this discipline is how a well-intentioned tidy-up becomes the root cause on a post-mortem.

Ecosystem Integration & DevOps Alignment

Treating flag configuration as code enables version control, peer review, and automated validation. Declarative definitions get linted in CI; dry-run evaluations verify targeting against synthetic contexts before a deploy reaches production. Webhook orchestration propagates config changes to caches and warms them ahead of a rollout, and observability correlation ties each flag decision to a trace so you can answer “which variant did this request see?” during an incident.

The integration points form a pipeline: a pull request changes a flag definition, CI validates the schema and dry-runs the targeting, a merge triggers a webhook that promotes the change and warms downstream caches, and every subsequent evaluation is stamped onto a distributed trace. Wiring these together turns flag changes from an out-of-band operation into an auditable, reviewable part of the same delivery pipeline that ships code.

Observability is the integration point that pays back fastest during an incident. When a flag drives a regression, the question is never abstract — it is “which cohort saw which variant, and when did the split shift?” A backend that emits an evaluation metric per variant, attaches each decision to the request’s trace, and writes a structured, PII-safe log line answers all three from a query rather than a code read. The discipline that keeps this affordable is cardinality routing: low-cardinality facts (flag key, variant, reason) belong in metrics where they aggregate cheaply, while high-cardinality detail (a hashed context, a request id) belongs in traces and logs where it is queryable but never explodes a time-series database. The observability for flag evaluations guide develops this into a single instrumentation hook that covers every flag automatically.

Cache warming deserves its own note because it is where a clean pipeline still surprises teams. A freshly deployed instance starts with an empty evaluation cache, so the first requests each pay a cold miss and stampede the control plane at once — a self-inflicted latency spike at exactly the moment a new version is most fragile. Warming the rule set during startup, behind the readiness probe, moves that unavoidable first load off the user’s request and staggers it across the fleet with jitter so the provider is never hit by a synchronized wave. The same staggering principle reappears whenever many clients act in lockstep, from deploy-time warmup to reconnection after a dropped stream.

The failure mode worth internalizing here is the thundering herd, and it is not limited to cold starts. Any event that invalidates a shared cache entry at the same instant across the fleet — a config push, a TTL that was set to a round number so thousands of entries expire on the same second, a control-plane restart that drops every streaming connection at once — funnels a synchronized burst of traffic at the source of truth. The defenses are the same three every time: add random jitter to TTLs and reconnect backoffs so expirations spread across a window instead of landing together; serve stale-while-revalidate so a request that finds an expired entry uses the old value and triggers exactly one background refresh rather than blocking; and cap reconnection with exponential backoff plus a ceiling so a control-plane outage does not turn into a retry storm that keeps it down. A pipeline that promotes config cleanly can still take out its own control plane if it forgets that “notify every node at once” and “every node reacts at once” are the same event. The distributed caching guide works through the topologies and the invalidation strategies that keep these bursts survivable.

# CI gate: validate definitions, then dry-run against flagd before deploy
npx ajv-cli validate -s ./flags/schema.json -d './flags/*.json'
curl -sf -X POST http://localhost:8013/schema.v1.Service/ResolveBoolean \
  -H 'Content-Type: application/json' \
  -d '{"flagKey":"checkout.payments.express-pay","context":{"targetingKey":"ci-123","tenantTier":"enterprise"}}' \
  | jq -e '.reason=="TARGETING_MATCH" or .reason=="DEFAULT"' || { echo "flag validation failed"; exit 1; }

Progressive Delivery & Experimentation

Server-side resolution is where progressive delivery actually executes: deterministic bucketing assigns each targetingKey to a stable variant so a user does not flip cohorts between replicas, and traffic is shifted by percentage rather than by redeploy. Because the bucketing hash is computed from a stable key, every replica in a horizontally scaled fleet reaches the same assignment for the same user without coordinating — the property that makes percentage rollouts with sticky bucketing safe under autoscaling.

Experiment integrity depends on guardrail metrics that can auto-halt a rollout when error rate or latency regresses, and on attribution that records the resolved variant alongside the outcome event without adding blocking I/O to the request path. The pattern is to ramp exposure in stages, watch a small set of guardrails at each step, and roll back automatically the moment a guardrail breaches its threshold — surfacing regressions in minutes instead of after a full rollout.

The statistics behind that automation are what separate a real guardrail from a nervous one. A single interval where the treatment cohort looks worse is usually noise or delayed metric ingestion, not a regression, so a monitor that halts on the first bad sample will cry wolf until the team disables it. The durable design requires a difference that is both statistically significant and practically large — a change big enough to act on, sustained across consecutive evaluation windows — before it pulls the rollback. The same comparison, run before the experiment rather than during it, is a sample-size calculation: decide the smallest effect worth detecting, compute how many subjects each variant needs, and commit to a decision date up front so the experiment is not stopped the moment the numbers happen to look good. Reusing one guardrail definition for both the pre-registered experiment and each step of the automated canary analysis keeps the safety logic consistent across the whole delivery path.

Deterministic bucketing has a correctness requirement that is easy to get subtly wrong: the hash must be salted with the flag key, not just the targetingKey. If every flag hashes the same user id through the same function, a user who lands in the first 5% for one rollout lands in the first 5% for every rollout, so your “5% canary” cohorts are all the same unlucky users carrying the blast radius of every experiment at once. Mixing the flag key into the hash input decorrelates the buckets so that being in the treatment group for one flag says nothing about your group for the next. The second requirement is that the assignment survives a change in fleet size — the bucket must be a pure function of (flagKey, targetingKey) and the rollout percentage, computed identically on every replica, never a function of which instance happened to serve the request. When both hold, you can scale from three pods to three hundred mid-rollout and no user crosses cohorts, which is exactly the invariant an experiment’s validity depends on.

Operational Safety

Every evaluation call needs a strict timeout and a safe default — an unhandled null variant is how flag infrastructure takes down a request path it was supposed to protect. A circuit breaker around the provider isolates a failing control plane, and a kill switch bypasses complex targeting to force a known-good state instantly, without a code deploy. Choosing the right sync transport matters here too: polling versus streaming determines how fast a kill switch actually propagates to every node.

The circuit breaker’s three states encode the safety contract. Closed is the normal path: calls flow through and successes and failures are counted. When failures cross a threshold the breaker trips open, and every call short-circuits to the safe default without touching the failing provider — protecting both the request path and the struggling control plane. After a cooldown it moves to half-open, letting a probe request through; a success closes the breaker and a failure re-opens it. This is what keeps a control-plane incident from cascading into a request-path outage.

The safe default is not a throwaway argument — it is a per-flag policy decision that deserves the same review as the flag’s targeting. The question to answer for every flag is “what value is safe when we cannot know the real one?”, and the answer flips depending on what the flag controls. A flag that enables a new feature defaults off, so an outage serves the known-good path. But a flag that guards a fallback — one that keeps a read replica or a degraded mode available — must default on, because defaulting it off disables the safety net at precisely the moment the primary is failing. Deciding this in advance, recording it as required metadata, and asserting it under a simulated provider outage is what turns “fail safe” from a slogan into a tested property. The safe fallback defaults guide works through the classification.

None of this resilience is trustworthy until it has been rehearsed. A circuit breaker that has never tripped in anger, a kill switch that shares a failure domain with the control plane it is meant to rescue, a safe default that was typed once and never verified — each is a hypothesis, not a guarantee. A periodic game day that deliberately cuts the provider, confirms the service degrades to defaults instead of erroring, and times how long the kill switch takes to propagate to every node is what converts those hypotheses into evidence. The operational safety and incident response guide covers both the automated defenses and the human runbook that coordinates the response when they are exercised for real.

The kill switch’s failure-domain independence is the constraint teams most often violate without noticing. A switch that lives in the same control plane, behind the same targeting engine, and reached over the same network path as the flags it is meant to override is not a rescue lever — it is another passenger on the same sinking boat, and it will be unreachable in exactly the incident it exists for. A dependable kill switch resolves through the simplest, most independent mechanism you can build: a value the SDK reads without evaluating targeting, cached locally so it survives a control-plane outage, and ideally settable through a second path — an environment variable, a config file baked into the deploy, a separate low-dependency endpoint — so that “force everything to the safe state” works even when the primary configuration service is the thing on fire. The emergency kill-switch runbook walks through building that independent path and rehearsing the human steps around it.

func evaluateWithResilience(ctx context.Context, c openfeature.IClient, key string) bool {
    evalCtx, cancel := context.WithTimeout(ctx, 50*time.Millisecond)
    defer cancel()
    val, err := c.BooleanValue(evalCtx, key, false, openfeature.EvaluationContext{})
    if err != nil { breaker.RecordFailure(); return false } // safe default
    breaker.RecordSuccess()
    return val
}

Compliance & Audit

Immutable audit trails capture every flag mutation, the actor, the evaluator context, and a resolution timestamp — the raw material for an incident post-mortem and a security review alike. Role-based access control restricts who can change production targeting, and retention policies align evaluation telemetry with regulatory boundaries. A well-formed audit record answers who changed what, when, and to what value, and pairs the mutation log with an evaluation log that records which variant a given request saw and why (the reason and any matched rule).

The mechanics of structuring those logs, satisfying SOC2 evidence requests, and masking personal data in the evaluation context are covered in the compliance and context-enrichment guides. The controlling principle is separation of duties enforced by RBAC: the people who author targeting rules, those who approve promotion to production, and those who read the audit trail are distinct roles, and every boundary crossing is logged.

Access control has to be proportional or it becomes theater. A flag console democratizes production changes, which is its value and its danger: without governance, a cosmetic toggle and a payment kill switch carry the same one-click ease and the same blast radius. The workable model scales the weight of the controls to the risk of the flag — a low-risk toggle needs only an authenticated actor and a logged change, while a high-risk flag requires a scoped role, a second approver, and a recorded justification. Crucially, the policy must be enforced where the change is applied, at the API layer, not only in the console UI, because a governance rule that lives only in the front end is bypassed by the first script that calls the API directly. The RBAC and flag change governance guide develops the approval workflows and least-privilege scoping that make this enforceable across every entry point.

Retention is the other half of the compliance story and it cuts both ways. Keeping audit records too long can violate data-minimization rules; deleting them too soon can fail an audit. The defensible answer classifies each record by what compels keeping it — compliance evidence, security forensics, or operational history — assigns the shortest lawful window per class, and enforces deletion on a schedule that is itself logged, so the log stays immutable within its lifetime and provably gone after it. Immutability and retention are not in tension: a record cannot be altered while it lives, and its deletion at window expiry is a distinct, evidenced operation, not a silent edit.

Key Concepts at a Glance

These five clusters make up the backend evaluation surface; each links to a deep-dive guide, and together they form the in-process path from a raw request to a resolved, audited variant.

• Server-side SDK integration — initialization lifecycle, dependency injection, and resilience for high-throughput services.
• Context enrichment — assembling sanitized user, tenant, and request attributes for precise targeting.
• Distributed caching — cache topologies and consistency that keep evaluation latency flat across nodes.
• Rule-engine performance — AST compilation and latency budgets for sub-5ms resolution.
• Flag synchronization transport — the polling-versus-streaming decision and its propagation guarantees.

Troubleshooting & FAQ