Feature Flags for AI Prompts With Canaries and Rollbacks

We ship AI features weekly and prompts regress. A one-line tweak can flip refusal rates, latency, or tool routing. I stopped shipping prompts as code-only changes and started shipping them behind feature flags with the same discipline we use for risky backend changes. This is how we wire LaunchDarkly or Flipt, run small canaries, and auto roll back without waking up the team at 3 a.m.

What To Flag In An AI System

Prompts are not a string. Treat the model call as a contract with multiple parameters. The things I put behind flags:

• System and instruction prompts, including safety and style layers
• Toolchain configuration, for example which functions the model can call and the default tool
• RAG parameters, topK, min score, context truncation strategy
• Model identity, fallback order, temperature and penalties
• Output schema, for example JSON mode with a strict schema vs freeform
• Post-processing and validation rules

Flags decide which configuration variant a user sees. A single flag can drive a compound variant, for example stable vs v3 which maps to prompt text, tool list, and model. That keeps rollout atomic. If you split flags too finely, you create combinatorial risk and you cannot reason about UX impact per user.

We also flag by audience. New accounts can default to the canary. Enterprise accounts get explicit allow-lists and sticky bucketing. Per account stickiness matters because LLM behavior adapts to prior interactions.

A Minimal Schema That Works

Keep variants in versioned config, not embedded in code. We use YAML and compute a checksum for reproducibility. Version is in Git, and we log the applied checksum with every inference.

# prompts/assistant.yml
variants:
  stable:
    id: assistant_v2
    checksum: e9c796e
    model: anthropic/claude-3-5-sonnet
    temperature: 0.2
    tools: [search, quote_price]
    rag:
      topK: 6
      minScore: 0.35
      truncate: middle
    system: |
      You are a precise assistant for sales reps. Always cite source URLs.
  v3:
    id: assistant_v3
    checksum: a41b22d
    model: openai/gpt-4o-mini
    temperature: 0.1
    tools: [search, quote_price, check_inventory]
    rag:
      topK: 8
      minScore: 0.42
      truncate: from_end
    system: |
      You are a precise assistant. Always cite source URLs. Prefer inventory checks.

Flag naming follows product scope. Example, ai.assistant.variant, ai.rag.scorer, ai.safety.mode. We resolve the flag to a named variant, then hydrate the model call from YAML. That makes local testing trivial and audit trails clear.

We track guardrail SLOs per variant. Pick three to five metrics that map to UX and cost. For most assistants we use:

• 95th percentile first token latency less than 2.5 s, end-to-end P95 less than 6 s
• Refusal or safety block rate less than 2 percent on benign prompts
• Tool call failure rate less than 1 percent
• Hallucination classifier alert rate less than 3 percent
• Cost per session less than 1.2x stable

If any SLO breaks for the canary, we roll back automatically. Do not argue with the robot at 2 a.m., flip the switch and analyze daylight hours.

If you need to get deeper on how we layer these pieces into product, our AI product engineering approach describes the philosophy in more detail in the context of delivery, see our AI product engineering approach.

Wiring LaunchDarkly or Flipt

Use a server-side SDK. Do not evaluate flags in the browser for AI behavior, you will leak internal prompt names and make rollbacks laggy.

Here is a minimal TypeScript wrapper that works with LaunchDarkly or Flipt. It selects a prompt variant with sticky bucketing, assembles the config, calls the model, and emits metrics. Replace the model call with your provider of choice.

// src/promptRouter.ts
import * as fs from 'node:fs';
import * as path from 'node:path';
import { LDClient, init as ldInit } from 'launchdarkly-node-server-sdk';
import fetch from 'node-fetch'; // used for Flipt

type VariantName = 'stable' | 'v3';

type PromptConfig = {
  id: string;
  checksum: string;
  model: string;
  temperature: number;
  tools: string[];
  rag: { topK: number; minScore: number; truncate: 'middle' | 'from_end' };
  system: string;
};

type Variants = Record<VariantName, PromptConfig>;

const variants: Variants = JSON.parse(
  JSON.stringify(
    // Load YAML at startup in your app, simplified here
    // In prod we parse YAML and validate with zod or joi
    {
      stable: {
        id: 'assistant_v2', checksum: 'e9c796e', model: 'anthropic/claude-3-5-sonnet',
        temperature: 0.2, tools: ['search', 'quote_price'], rag: { topK: 6, minScore: 0.35, truncate: 'middle' },
        system: 'You are a precise assistant for sales reps. Always cite source URLs.'
      },
      v3: {
        id: 'assistant_v3', checksum: 'a41b22d', model: 'openai/gpt-4o-mini',
        temperature: 0.1, tools: ['search', 'quote_price', 'check_inventory'], rag: { topK: 8, minScore: 0.42, truncate: 'from_end' },
        system: 'You are a precise assistant. Always cite source URLs. Prefer inventory checks.'
      }
    }
  )
);

export type Context = { userKey: string; accountKey?: string; tier?: 'free'|'pro'|'enterprise' };

export class PromptRouter {
  private ld?: LDClient;
  private fliptUrl?: string;

  async init() {
    if (process.env.LAUNCHDARKLY_SDK_KEY) {
      this.ld = ldInit(process.env.LAUNCHDARKLY_SDK_KEY, { application: { id: 'ai-app', version: '1.0.0' } });
      await this.ld.waitForInitialization();
    } else if (process.env.FLIPT_URL) {
      this.fliptUrl = process.env.FLIPT_URL;
    } else {
      throw new Error('Set LAUNCHDARKLY_SDK_KEY or FLIPT_URL');
    }
  }

  async resolveVariant(ctx: Context): Promise<VariantName> {
    const key = ctx.accountKey || ctx.userKey; // sticky by account when possible

    if (this.ld) {
      const ldUser = { key, custom: { tier: ctx.tier } };
      // Flag returns string variation name, default to stable
      const v = await this.ld.variation('ai.assistant.variant', ldUser, 'stable');
      return (v === 'v3' ? 'v3' : 'stable');
    }

    // Flipt evaluation via REST. Namespace and flag must exist.
    const body = { entityId: key, context: { tier: ctx.tier || 'free' } };
    const resp = await fetch(`${this.fliptUrl}/api/v1/evaluate/flags/ai.assistant.variant`, {
      method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body)
    });
    if (!resp.ok) return 'stable';
    const data = await resp.json() as { enabled: boolean; value: string };
    return (data.value === 'v3' ? 'v3' : 'stable');
  }

  getConfig(variant: VariantName): PromptConfig { return variants[variant]; }
}

// Example usage in a handler
export async function handleChat(ctx: Context, userMessage: string) {
  const router = new PromptRouter();
  await router.init();
  const variant = await router.resolveVariant(ctx);
  const cfg = router.getConfig(variant);

  const t0 = Date.now();
  try {
    const result = await callModel(cfg, userMessage); // implement provider call, stream or not
    const ms = Date.now() - t0;
    emitMetric('ai.latency.ms', ms, { variant: cfg.id });
    return { text: result.text, variant: cfg.id, checksum: cfg.checksum };
  } catch (e) {
    emitMetric('ai.error', 1, { variant: cfg.id, type: String((e as Error).name) });
    throw e;
  }
}

function emitMetric(name: string, value: number, tags: Record<string,string>) {
  // send to StatsD, Prometheus, or your warehouse
}

async function callModel(cfg: PromptConfig, msg: string) {
  // Map cfg.model to provider client and pass cfg.system, tools, rag params
  return { text: `stub for ${cfg.id}` };
}

Two practical notes:

• Keep flag keys stable across environments, drive behavior via targeting rules. Duplicate keys create blind spots in dashboards.
• Make evaluation sticky by account or conversation. Do not switch variants mid thread. Cache the chosen variant in session state.

If you want to pressure test this approach against your architecture choices, we typically run it alongside an architecture review for AI systems to verify where flags evaluate and where to emit metrics.

Canary Strategy and Auto Rollback

Rollouts are simple if you pick defaults now, not during an incident.

• Target 1 percent of eligible traffic for 2 to 4 hours. Stickiness by account.
• Escalate to 5 percent for a day, then 25 percent, then 50 percent.
• Hold out 10 percent of traffic on stable for a week after full rollout so you catch regressions that show up only with new user behavior.

Eligibility rules I like:

• Exclude enterprise and high revenue accounts until 25 percent stage.
• Exclude accounts with open severity 1 bugs.
• Exclude unsupported locales if prompt contains locale-specific instructions.

Automated rollback needs two pieces. Streaming metrics with short windows, and a write path that can change the flag automatically. I use a five minute sliding window with 2 to 3 consecutive window breaches to avoid flapping. Below is a minimal in-process guard. In production we run this as a background worker and hit the provider API to dial weights.

// src/rollbackGuard.ts
import fetch from 'node-fetch';

type WindowStats = { errors: number; requests: number; p95ms: number; costUsd: number };
let windows: Record<string /*variant*/, WindowStats[]> = { v3: [], stable: [] };

export function recordSample(variant: 'stable'|'v3', ok: boolean, ms: number, costUsd: number) {
  // Push into a time-bucketed structure. Simplified, no actual time slicing here.
  const w = windows[variant];
  const last = w[w.length - 1] || { errors: 0, requests: 0, p95ms: 0, costUsd: 0 };
  last.requests += 1; if (!ok) last.errors += 1; last.p95ms = Math.max(last.p95ms, ms); last.costUsd += costUsd;
  if (!w.length) w.push(last);
}

export async function maybeRollback() {
  const recent = summarize(windows['v3']);
  // Guardrails
  const errorRate = recent.errors / Math.max(1, recent.requests);
  const latencyBad = recent.p95ms > 6000; // 6 s end-to-end
  const errorBad = errorRate > 0.02; // 2 percent
  const costBad = recent.costUsd / Math.max(1, recent.requests) > 1.2 * baselineCentsPerReq();

  const breaches = [latencyBad, errorBad, costBad].filter(Boolean).length;
  if (breaches >= 2 && canaryWeight() > 0) {
    console.warn('Rolling back v3, guardrails breached', { errorRate, p95: recent.p95ms });
    await setCanaryWeight(0); // call LaunchDarkly or Flipt admin API
  }
}

function summarize(w: WindowStats[]): WindowStats {
  return w.reduce((a, b) => ({
    errors: a.errors + b.errors,
    requests: a.requests + b.requests,
    p95ms: Math.max(a.p95ms, b.p95ms),
    costUsd: a.costUsd + b.costUsd,
  }), { errors: 0, requests: 0, p95ms: 0, costUsd: 0 });
}

function baselineCentsPerReq() { return 0.01; } // replace with a real baseline
function canaryWeight() { return 0.05; } // track from config

async function setCanaryWeight(weight: number) {
  // Example for Flipt. For LaunchDarkly, use their Flags API with a patch op.
  await fetch(process.env.FLIPT_ADMIN_URL + '/api/v1/flags/ai.assistant.variant', {
    method: 'PUT', headers: { 'content-type': 'application/json', authorization: `Bearer ${process.env.FLIPT_TOKEN}` },
    body: JSON.stringify({
      // Pseudo payload. Consult your provider schema for exact fields.
      rollout: { variations: [{ value: 'stable', weight: 1 - weight }, { value: 'v3', weight }] }
    })
  });
}

You can keep the same shape and call LaunchDarkly with a PATCH that adjusts the percentage rollout for the v3 variation. The principle matters more than the exact client. Have a robot that watches guardrails and can turn the canary to zero. Log the decision with metrics and an audit message in your incident channel.

To keep this honest, build the allow-list that can bypass the robot. There are always cases where support needs to leave a single account on stable during investigation.

For the pipeline that feeds these decisions, stable and canary both must emit the same metrics. I rely on a shared event for start, first token, end, refusal, tool call status, and token counts. If you already have Cloud and DevOps pipelines for metrics and secrets management, reuse them. If not, set that up first. Our teams fold this into CI and infra using the same patterns we use for services, which are similar to what we describe for Cloud and DevOps pipelines.

Testing, Shadowing, and Cost

Offline tests catch easy stuff. I keep a golden set of 200 to 1000 prompts with expected properties, for example must call check_inventory on these 50, must not hallucinate price on these 20, must answer in 2 seconds on these 30. I grade with simple heuristics and a secondary LLM when needed.

Shadow inference is useful before the first 1 percent canary. Run the new variant on live prompts but do not show it to users. Store the output and grade it. Watch cost and latency because you are paying twice.

Here is a tiny Python harness that compares stable vs v3 on a CSV of prompts and emits a CSV with scores. We run this in CI on every prompt PR and again nightly on a rotating sample of real traffic.

# evals/grade_variants.py
import csv, time
from typing import Dict

# wire your provider here

def call_model(cfg: Dict, prompt: str) -> Dict:
    # return { 'text': '...', 'ms': 1234, 'cost_usd': 0.002 }
    return { 'text': f"stub {cfg['id']}", 'ms': 1000, 'cost_usd': 0.001 }

stable = { 'id': 'assistant_v2' }
v3 = { 'id': 'assistant_v3' }

def grade(row, out):
    p = row['prompt']
    s = call_model(stable, p)
    n = call_model(v3, p)
    # toy rubric
    must_tool = 'inventory' in p
    s_tool = int('inventory' in s['text'])
    n_tool = int('inventory' in n['text'])
    return {
        'prompt_id': row['id'],
        's_ms': s['ms'], 'n_ms': n['ms'],
        's_cost': s['cost_usd'], 'n_cost': n['cost_usd'],
        's_tool': s_tool, 'n_tool': n_tool,
        'win': int(n_tool > s_tool) - int(n_tool < s_tool)
    }

with open('prompts.csv') as f, open('results.csv', 'w', newline='') as o:
    rdr = csv.DictReader(f)
    wr = csv.DictWriter(o, fieldnames=['prompt_id','s_ms','n_ms','s_cost','n_cost','s_tool','n_tool','win'])
    wr.writeheader()
    for row in rdr:
        wr.writerow(grade(row, o))

For cost control during shadowing, cap concurrent shadow calls and sample at 5 percent of traffic. If your cost unexpectedly doubles, stop. That is usually a prompt explosion issue. I have seen a change to include longer context references add 500 tokens per call. That was a silent 3x cost increase.

We also pin model versions where possible. Providers change behavior across minor revisions. Log model version, provider region, and prompt checksum for every call. When your canary wins, you want to know exactly what you shipped.

If your team wants to formalize this end to end, we often document the flow in our internal runbooks and mirror it during delivery. The same patterns show up in our Claude code workflow practices when we gate AI-assisted code changes behind staged tooling rollouts.

Common Failure Modes and What I Would Do Differently Next Time

• Sticky context drift. Users in the canary teach the assistant new behavior that bleeds into stable when you rotate. Reset conversation state or pin by thread ID, not just user ID.
• Tool availability mismatch. You enable a new tool only for the canary, but the UI shows the tool to all users. Gate UI affordances on the same flag.
• Latency regressions come from upstream dependencies. New prompt uses check_inventory more often, but that service has poor P95. Your rollback triggers for the wrong reason. Add per-tool latency metrics and budget.
• Safety regressions in obscure locales. Run a language detector and exclude languages you do not test in your golden set.
• Logging overload. Prompt debugging logs can spike storage. Sample logs for stable at 1 percent, canary at 10 percent, and redact.
• CI drift. Devs update prompt YAML without updating the offline eval rubric. Fail the PR if rubric or golden set does not change during certain types of prompt edits. This is easy to wire into your CI. If you need help operationalizing this, treat it like any other pipeline change and fold it into your engineering process setup.
• Provider limits. A successful canary can double your RPS to an LLM endpoint. Reserve capacity or add a circuit breaker. Retry budgets can also explode cost. Track retry count and cap per session.

Three things I would do differently on a greenfield build now:

1. Put prompt variants in their own repository early, with a schema and code owners. It improved review quality.
2. Add a simple refusal classifier at the edge. It catches safety regressions before they hit users.
3. Bake a manual override UI on day one. Incidents feel smaller when anyone on call can force stable for an account.

A Short Checklist You Can Lift

• One flag per AI surface that selects a compound variant
• Sticky evaluation by account or thread
• YAML or JSON registry with variant checksum, model, tools, RAG params
• Golden set offline eval in CI on each prompt PR
• Shadow run in staging or sampled prod before 1 percent canary
• Guardrail SLOs and a robot that can dial canary to zero
• Shared metrics schema and dashboards split by variant
• Manual override UI and allow-lists
• Holdout group for a week after 100 percent rollout

This is not complicated, but it is specific. Once the scaffolding is in, shipping prompts behind flags feels like shipping code again. Your incident load drops, your UX is stable, and you can move faster without fear.

If you want to apply the same approach across the rest of your stack, we use the same control points we would for service evolution, traffic shaping, and release management. That looks the same whether we are touching prompts or endpoints, and it ties back to our habits in Cloud and DevOps pipelines and the product-first mindset outlined in our AI product engineering approach.