You ask an LLM to generate synthetic order records and it returns "status": "currently being shipped" instead of "IN_TRANSIT". Your JSON Schema validator passes it — you didn't constrain the field. Your downstream Kafka consumer silently drops the record. No exception, no alert. This is semantic drift: the model understands the intent of an enum but substitutes a natural-language paraphrase for the canonical value, and your pipeline has no idea.

The problem compounds at scale. A single LLM call might return 98% valid enums; at 50,000 synthetic records, that 2% drift produces 1,000 invalid rows — enough to corrupt aggregate test metrics, skew cardinality, and produce false-green integration tests. The failure mode is quiet, intermittent, and almost never caught in prompt review.

By the end of this article you'll have a concrete validation layer, a constrained prompting pattern, and a Hypothesis-based property test that catches enum drift before it reaches your fixtures.

Why LLMs Paraphrase Enums Instead of Reproducing Them

Language models are trained to produce fluent, contextually appropriate text — not to treat string literals as sacred tokens. When a prompt says "generate an order with a realistic status", the model's sampling distribution covers the entire semantic neighborhood of "order status": SHIPPED, Shipped, shipped, "out for delivery", "dispatched". All are plausible completions. Without an explicit constraint, the model has no reason to prefer your canonical enum over an equally probable synonym. This is a token-prediction artifact, not a reasoning failure — the model isn't confused, it's just not anchored.

The issue is distinct from token boundary drift in compound identifiers, where the model splits a structured string at the wrong character boundary. Semantic drift on enums is a vocabulary problem: the output is well-formed text, it just isn't in your allowed set. It survives JSON parsing, passes type checks, and only fails at the business-logic layer — which is exactly where you don't want surprises in test data.

Constraining, Validating, and Testing Enum Fidelity

The first line of defense is prompt-level constraint. Embed the allowed values directly in the system prompt and instruct the model to treat them as literals. Vague instructions like "use a valid status" don't work — models interpret "valid" generously. Be explicit:

SYSTEM:
You are a test data generator. For the `status` field you MUST use
ONLY one of these exact strings — no paraphrasing, no case changes:
["PENDING", "IN_TRANSIT", "DELIVERED", "CANCELLED", "RETURNED"]
Any other value is a hard error.

USER:
Generate 20 order records as a JSON array. Each record must include
`order_id`, `status`, and `created_at` (ISO-8601).

This alone drops drift from ~4% to under 0.5% in practice with GPT-4o and Claude 3.5 Sonnet on medium-cardinality enums (5–15 values). For high-cardinality enums (50+ values), the rate climbs again — the model starts interpolating. At that threshold, generate the enum value separately with a deterministic sampler (Faker's random_element, Mimesis's choice) and inject it into the LLM context rather than asking the model to choose it.

The second layer is schema validation at generation time, not consumption time. Use JSON Schema 2020-12 with a strict enum keyword and validate every batch before it touches a fixture store:

from jsonschema import validate, ValidationError
import json

ORDER_SCHEMA = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "order_id": {"type": "string", "pattern": "^ORD-[0-9]{8}$"},
        "status": {
            "type": "string",
            "enum": ["PENDING", "IN_TRANSIT", "DELIVERED", "CANCELLED", "RETURNED"]
        },
        "created_at": {"type": "string", "format": "date-time"}
    },
    "required": ["order_id", "status", "created_at"],
    "additionalProperties": False
}

def validate_batch(records: list[dict]) -> tuple[list, list]:
    valid, invalid = [], []
    for r in records:
        try:
            validate(instance=r, schema=ORDER_SCHEMA)
            valid.append(r)
        except ValidationError as e:
            invalid.append({"record": r, "error": e.message})
    return valid, invalid

Reject and log the invalid batch — don't silently drop records, because a high rejection rate signals prompt regression. In a GitHub Actions pipeline, surface the drift rate as a job output and fail the workflow when it exceeds a threshold (1% is a reasonable starting point). The third layer is a Hypothesis property test that runs against your LLM generation function directly, treating it as a black box:

from hypothesis import given, settings
from hypothesis import strategies as st
import pytest

VALID_STATUSES = {"PENDING", "IN_TRANSIT", "DELIVERED", "CANCELLED", "RETURNED"}

@given(seed=st.integers(min_value=0, max_value=10_000))
@settings(max_examples=50)
def test_no_enum_drift(seed, llm_order_generator):
    # llm_order_generator is a pytest fixture wrapping your generation call
    records = llm_order_generator(seed=seed, count=10)
    for r in records:
        assert r["status"] in VALID_STATUSES, (
            f"Enum drift detected: got '{r['status']}'"
        )

Fifty examples at 10 records each means 500 LLM-generated records exercised per test run. With caching at the fixture level this completes in under 30 seconds. Pairing this with property-based testing for data validation gives you a systematic coverage model rather than a handful of manually checked samples.

Where Senior Engineers Still Get Burned

Trusting one-time prompt verification. Engineers test the prompt manually, see clean output, and ship it. LLM outputs are non-deterministic; a prompt that returns perfect enums at temperature 0.7 during review will drift at temperature 1.0 under load or when the model version rolls. Always validate programmatically on every generation call, not just during prompt development. Model provider silent upgrades (OpenAI's model aliasing, for example) have broken stable prompts overnight.

Conflating case normalization with enum validation. A common "fix" is to call .upper() on the returned value before storing it. This masks the symptom — "delivered".upper() becomes "DELIVERED" — but it also silently coerces "out for delivery".upper() to "OUT FOR DELIVERY", which still isn't in your enum. Worse, it trains the team to assume the normalization layer catches everything, so the real validator never gets written. Normalize only after strict enum membership is confirmed. Also watch for cardinality skew — even when enum values are correct, LLMs over-represent semantically "interesting" values like CANCELLED and under-represent PENDING.

Myths That Let Enum Drift Survive Code Review

"The model follows the schema if I include it in the prompt." Providing a JSON Schema in the prompt is not the same as enforcing it. Schema-in-prompt is a hint; the model may still generate outside it, especially for fields where natural language is more probable than a code-style constant. Use structured output APIs (OpenAI's response_format: json_schema, Instructor with Pydantic, or Outlines for local models) to enforce schema at the decoding layer, not the prompting layer. This is the difference between asking someone to stay in bounds and building a fence.

"This only matters for string enums." Boolean and integer enums drift too — a model asked for a priority field with values 1 | 2 | 3 will occasionally return 4 or "high". Numeric range drift is harder to spot because it passes type validation. The same constrained-generation and post-validation pattern applies. And don't assume that because your consumer fixtures are schema-aligned today, they'll catch a new enum variant an LLM invents — fixtures are point-in-time and won't know about values that were never in the allowed set to begin with.

Semantic enum drift is a generation-time correctness problem masquerading as a data quality problem. Fix it at the source: constrain the model's output vocabulary with structured decoding APIs, validate every batch against a strict JSON Schema 2020-12 enum definition, and run Hypothesis property tests against your generation layer in CI. If you're building a repeatable generation pipeline, the custom test data generator walkthrough covers the scaffolding that makes these validation hooks easy to wire in.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.