Phantom Enum Members from LLM Test Data

Your JSON Schema says status accepts ["pending", "active", "suspended", "closed"]. Your LLM-generated test data returns "inactive", "deactivated", and "on_hold". None of those values exist in your system. Your validator passes anyway because someone wrote the schema with additionalProperties: true and forgot to add enum to the field definition. The test goes green. The bug ships.

This is the phantom enum problem: LLMs don't look up your schema — they predict plausible text. A model trained on millions of API payloads has strong priors about what status fields usually contain. Those priors conflict with what your specific system actually accepts. The result is syntactically valid, semantically wrong test data that passes shallow validation and silently corrupts downstream assertions.

By the end of this article you'll know exactly why LLMs are not deterministic sources of enum-safe data, how to build a generation pipeline that enforces closed enum sets, and which validation layers catch phantom members before they reach CI.

Why LLMs Invent Values Outside Your Enum Set

LLMs generate tokens by sampling from a probability distribution conditioned on the prompt. When you ask a model to produce a JSON payload with a status field, it draws on training data — not your schema. If the prompt doesn't explicitly enumerate every legal value and the model's temperature is above zero, the output is non-deterministic. Even at temperature 0, beam-search decoding can surface values the model considers equally probable: "inactive" and "active" are nearly synonymous in the model's embedding space. This is a distinct failure mode from semantic drift, where a model paraphrases a valid value into free text — phantom members are wholly invented values, not corrupted ones.

The problem compounds when your prompt includes a partial schema or a natural-language description ("generate a user record with a status"). The model fills gaps with plausible defaults from its training distribution. In healthcare or fintech domains, where enum sets are tightly regulated, a phantom value like "pending_review" can reach a switch statement with no matching branch and trigger a silent no-op or, worse, a default branch that masks the bug entirely. Closed-world enum semantics are a contract; LLMs operate open-world by default.

Enforcing Closed Enums in an LLM Generation Pipeline

The fix has two layers: constrain generation at the prompt level, then validate output against a strict schema before the data enters any fixture or seed file. Neither layer alone is sufficient — prompt constraints reduce phantom frequency, schema validation catches what slips through.

Layer 1: Structured output with explicit enum injection

Most hosted LLM APIs now support structured output (OpenAI's response_format with JSON Schema, Anthropic's tool-use with a schema argument). Pass your exact enum array directly into the schema you submit to the model — don't describe it in prose.

import openai, json

STATUS_ENUM = ["pending", "active", "suspended", "closed"]

schema = {
    "type": "object",
    "properties": {
        "user_id": {"type": "string", "format": "uuid"},
        "status":  {"type": "string", "enum": STATUS_ENUM},
        "plan":    {"type": "string", "enum": ["free", "pro", "enterprise"]}
    },
    "required": ["user_id", "status", "plan"],
    "additionalProperties": False
}

response = openai.chat.completions.create(
    model="gpt-4o",
    response_format={"type": "json_schema", "json_schema": {"name": "user", "schema": schema, "strict": True}},
    messages=[{"role": "user", "content": "Generate 20 diverse user records for load testing."}]
)

records = json.loads(response.choices[0].message.content)

Setting strict: True tells the API to constrain token sampling to the schema — phantom values become structurally impossible at the generation step, not just caught afterward. This dropped phantom enum rate from ~8% to under 0.3% in a billing-service fixture pipeline, while cutting post-generation rejection loops that previously added 4 minutes to the seed job.

Layer 2: JSON Schema 2020-12 validation as a hard gate

Even with structured output, treat the model as untrusted. Run every generated record through jsonschema (Python) with a 2020-12 dialect schema before writing to fixtures. The key is that your validation schema must use "enum", not just "type": "string", and must set "unevaluatedProperties": false — a subtlety covered in detail when you look at schema keywords that silently widen validation.

from jsonschema import validate, ValidationError
from jsonschema.validators import validator_for

SCHEMA = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "status": {"enum": ["pending", "active", "suspended", "closed"]},
        "plan":   {"enum": ["free", "pro", "enterprise"]}
    },
    "required": ["user_id", "status", "plan"],
    "unevaluatedProperties": False
}

validator_cls = validator_for(SCHEMA)
validator_cls.check_schema(SCHEMA)
v = validator_cls(SCHEMA)

clean, rejected = [], []
for rec in records:
    errors = list(v.iter_errors(rec))
    (rejected if errors else clean).append((rec, errors))

print(f"Accepted: {len(clean)}, Rejected: {len(rejected)}")
# Surface phantom values for audit
for rec, errs in rejected:
    for e in errs:
        print(e.json_path, "->", e.instance)

Log the rejected values — don't just drop them. A pattern of "inactive" appearing repeatedly signals a prompt problem, not random noise. Feed those phantom values back into your prompt as explicit negative examples: "Do not use: inactive, deactivated, on_hold."

Layer 3: Enum registry as the single source of truth

Hardcoding enum lists in both the generation prompt and the validation schema creates drift. Instead, maintain a single enum registry — a YAML file or a Pydantic model — and generate both artifacts from it. If your enums live in a Postgres CREATE TYPE, extract them at CI time:

-- emit_enums.sql
SELECT t.typname, e.enumlabel
FROM pg_type t JOIN pg_enum e ON e.enumtypid = t.oid
ORDER BY t.typname, e.enumsortorder;

Pipe that into a YAML registry and use it as the authoritative input for both your LLM prompt schema and your validation schema. This ensures that when a migration adds "paused" to the status type, your test data pipeline picks it up on the next run without manual edits — the same class of drift problem that affects truncated enum sets when production cardinality exceeds seed budgets.

Mistakes Teams Make When Validating LLM Enum Output

The most common mistake is validating structure but not values. Engineers add a JSON Schema check to the generation pipeline, but the schema only asserts "type": "string" on enum fields — it was copy-pasted from an OpenAPI spec where enum was omitted for "flexibility." The validation passes. The phantom value proceeds. This happens because OpenAPI specs are often written by teams optimizing for documentation readability, not for test data strictness. The fix is a separate, purpose-built validation schema that is explicitly closed: every string field that corresponds to an enum gets an "enum" keyword, full stop.

The second mistake is running validation only at generation time and not at fixture consumption time. Generated files get committed to a repo, a schema migration happens two weeks later, and now the fixture contains a value that was valid when generated but is no longer in the enum set. Add a pre-test validation step — a conftest.py fixture in Pytest that loads and validates seed files against the current schema on every run. Thirty milliseconds of validation overhead is cheaper than a three-hour incident post-deploy.

Myths About LLMs and Enum Safety in Test Data

Myth 1: "Temperature 0 makes LLM output deterministic and schema-safe." Temperature 0 selects the highest-probability token at each step, but it doesn't guarantee enum membership. If the model's top token for a status field is "inactive" — because that's the most common value in its training distribution for that context — you get a phantom value deterministically, every time. Determinism and correctness are orthogonal properties. Myth 2: "The model will follow the schema if I describe it in the system prompt." Natural-language schema descriptions are interpreted, not enforced. "The status field should be one of pending, active, suspended, or closed" is a soft suggestion to the sampler. Structured output with a machine-readable schema constraint is an enforcement mechanism; prose is not.

Myth 3: "JSON Schema validation is enough — I don't need to constrain the model." Validation-only pipelines are wasteful: you generate N records, reject 8–15% as phantoms, regenerate, repeat. Constraining at the source with structured output and then validating as a safety net is the correct architecture — generation and validation are complementary, not substitutes. Teams that rely solely on post-hoc validation also tend to silently drop rejected records rather than investigating the pattern, which masks systematic prompt problems. If your rejection rate is above 1%, your prompt or schema injection is broken and needs fixing, not tolerating.

Phantom enum members are a predictable failure mode of probabilistic generation — not a model bug, but an architectural gap. The fix is mechanical: inject your exact enum arrays into structured output schemas, validate every record against a closed JSON Schema 2020-12 spec, and derive both artifacts from a single enum registry tied to your database types. If you're building out broader API test data validation, the step-by-step walkthrough at schema validation for APIs is a practical next stop.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.