Your LLM-generated test data looks fine in the preview. Then a downstream service rejects ORDER-2024_US because it arrived as ORDER -2024_US — a single space injected at a token seam. The model wasn't hallucinating; it was doing exactly what it was trained to do: predict the next token. Compound identifiers like ORDER-2024_US, user_id:acct-00123, or TXN/2024/04/EUR are not atomic to a tokenizer. They're sequences, and sequences get split.

This is token boundary drift: the silent corruption of structured identifiers when an LLM's tokenization vocabulary doesn't align with your identifier grammar. It doesn't throw an error. It produces plausible-looking strings that fail regex validation, break foreign-key lookups, or — worst — pass shallow string-equality checks while carrying the wrong semantics.

By the end of this article you'll understand why drift happens at the tokenizer level, how to reproduce and measure it, and which generation-time and validation-time controls actually stop it from reaching your test suite.

Why Tokenizers Fracture Compound Identifiers

GPT-4's cl100k_base tokenizer and the tokenizers used by Claude models both operate on byte-pair encoding (BPE). BPE merges frequent character sequences greedily during training, so common English subwords get single tokens while domain-specific separators — hyphens inside IDs, slashes in path-style identifiers, underscores in snake_case keys — often land on boundaries between two or three tokens. Run tiktoken.encode("ORDER-2024_US") and you'll see five tokens, not one. Each token is an independent prediction site, which means each separator character is a point where the model can drift: inserting whitespace, changing case, or substituting a visually similar character.

This matters most when you're using LLMs for privacy-safe synthetic data generation at scale — the use case where you're asking the model to emit hundreds of records in a single prompt. Batch generation amplifies drift because the model's attention window shifts across records, and later records in a long completion are statistically more likely to show boundary artifacts than the first few. A 500-record batch that looks clean in spot-checks can carry a 3–8% drift rate in compound ID fields, which is enough to corrupt FK chains or break contract tests.

Reproducing, Measuring, and Constraining Drift

Before you can fix drift you need to measure it. The following snippet uses tiktoken to flag any identifier field whose token count exceeds a threshold — a cheap static signal that a field is drift-prone before you even call the model.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def drift_risk(identifier: str, threshold: int = 3) -> bool:
    """Return True if the identifier spans more tokens than threshold."""
    tokens = enc.encode(identifier)
    return len(tokens) > threshold

# Examples
print(drift_risk("ORDER-2024_US"))   # True  — 5 tokens
print(drift_risk("acct00123"))       # False — 2 tokens
print(drift_risk("TXN/2024/04/EUR")) # True  — 6 tokens

Fields flagged as drift-prone should never be free-generated. Instead, generate the structural components separately and assemble them in application code. Ask the model only for the variable parts — a region code, a year, a sequence number — then join them with a Python format string your test harness controls. This keeps the model away from the separator characters entirely.

import random, string

def make_order_id(region: str, year: int) -> str:
    seq = ''.join(random.choices(string.digits, k=5))
    return f"ORDER-{year}_{region}-{seq}"

# LLM provides region="US", year=2024 — nothing else
order_id = make_order_id(region="US", year=2024)
# → "ORDER-2024_US-08312"  — separator characters never touched by the model

For cases where you must have the model emit full identifiers — legacy prompt designs, third-party pipelines — constrain output with a JSON Schema pattern and validate before the data enters your fixture store. JSON Schema 2020-12 pattern is your last line of defense here. Pair it with Pydantic v2 for in-process validation:

from pydantic import BaseModel, Field

class OrderRecord(BaseModel):
    order_id: str = Field(pattern=r"^ORDER-\d{4}_[A-Z]{2}-\d{5}$")
    account_id: str = Field(pattern=r"^acct-\d{5}$")
    txn_ref: str = Field(pattern=r"^TXN/\d{4}/\d{2}/[A-Z]{3}$")

In a real pipeline, validation failures should be logged with the raw model output, not silently retried. Retrying without logging hides the true drift rate. In one batch-generation pipeline processing 10,000 synthetic order records, adding Pydantic boundary validation surfaced a 6.2% drift rate in order_id fields that had been invisible for two sprint cycles — because downstream tests were doing substring matches, not full-pattern assertions. Fixing the generation strategy (component assembly, not free generation) dropped that rate to zero. The same cross-service consistency validation pattern applies: validate at the boundary where data is produced, not only where it's consumed.

Where Senior Engineers Still Get Burned

Trusting few-shot examples as a sufficient constraint. Adding two or three well-formed IDs to the prompt does reduce drift, but it doesn't eliminate it. Few-shot examples bias the distribution; they don't enforce a grammar. Under long completions or temperature settings above 0.3, the model will eventually deviate. Engineers who validated their prompt on a 20-record test and shipped it to generate 5,000 records have learned this the hard way. The fix is structural: schema validation in the pipeline, not prompt engineering alone.

Ignoring tokenizer differences across model versions. cl100k_base (GPT-4, GPT-3.5-turbo) and the tokenizer used by Claude 3.x encode the same identifier differently. A generation pipeline tuned against one model and then pointed at another — common when teams switch providers for cost reasons — can see drift rates jump with no code change. If your pipeline is model-agnostic by design, your validation layer must be model-agnostic too. This is also where schema drift in test infrastructure compounds the problem: a stale regex in a fixture validator won't catch a new identifier format the model started emitting after a model upgrade.

Myths That Let Drift Accumulate Undetected

"Structured output mode (JSON mode) prevents identifier drift." JSON mode guarantees syntactically valid JSON — it does not guarantee that string values inside that JSON conform to your identifier grammar. A model in JSON mode will happily emit "order_id": "ORDER -2024_US". The structural envelope is clean; the value is corrupt. JSON mode solves a different problem. Your identifier constraints still need explicit pattern validation on top of it.

"We only use LLMs for realistic names and addresses, not IDs — so this doesn't apply to us." Compound identifiers appear in more fields than teams realize: email addresses, SKU codes, IBAN-style account numbers, ISO 8601 timestamps with timezone offsets, semantic version strings, and any field that concatenates two domain concepts with a separator. If you're asking an LLM to generate realistic-looking records for a financial or logistics domain, you almost certainly have compound identifiers in the payload. The same tokenization mechanics that corrupt ORDER-2024_US will corrupt 2024-04-15T09:30:00+05:30. Audit your schema for any field whose valid values require a non-alphanumeric separator before assuming you're safe.

Token boundary drift is a deterministic consequence of BPE tokenization applied to identifier grammars that were never part of the training corpus. The mitigation stack is straightforward: flag drift-prone fields with a token-count heuristic, assemble compound identifiers in application code rather than prompting for them wholesale, and enforce Pydantic or JSON Schema 2020-12 pattern constraints at the generation boundary. Start by running tiktoken.encode() against every identifier field in your most-used test fixtures — the results will tell you exactly where to focus.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.