Hotspot Bias: Weighted Factories Miss Tail Enums

Most enum-related test failures aren't caught in CI — they surface in production when a rare status transition hits a code path that was never exercised. Your weighted factory assigns 80% of records to ACTIVE, 15% to PENDING, and the remaining 5% is split across six tail values. At 1,000 generated records, statistically you might get zero SUSPENDED_FRAUD rows. The branch that handles that status has never run under test.

This is hotspot bias at the enum layer: your factory's weight distribution reflects business frequency, not coverage intent. Those are different goals, and conflating them is how you ship a payment processor that silently swallows CHARGEBACK_REVERSED events.

By the end of this article you'll be able to instrument your factories to detect tail starvation, apply coverage-aware weighting strategies, and enforce minimum enum path cardinality in CI — without abandoning the realistic distributions your load tests depend on.

Why Weighted Factories Produce Structurally Incomplete Enum Coverage

A weighted factory is a record generator that assigns probability mass to each possible value of a field according to some distribution — usually derived from production analytics or a product manager's intuition. The intent is realism: if 82% of orders in prod are FULFILLED, the factory should reflect that. The problem is that coverage and realism are orthogonal properties. A factory optimized purely for distributional fidelity will, by design, under-represent tail enum members — and at typical CI dataset sizes (hundreds to low thousands of records), those tail members may not appear at all.

This isn't a Faker-specific problem. It manifests in factory_boy factories, FactoryBot sequences, Pydantic model builders, and dbt seed files alike. The bias is structural: any generator that samples from a weighted distribution without a coverage floor will starve low-probability paths. It's closely related to the broader hotspot bias problem in weighted Faker distributions, but the enum variant is particularly dangerous because enum branches map directly to conditionals in application logic — each missing value is a missing test.

Detecting and Fixing Tail Starvation in factory_boy and Pydantic Builders

Start by measuring. Before you change any factory, instrument your existing generation pipeline to emit an enum frequency report. A one-liner with collections.Counter against your generated dataset tells you immediately which members are starved:

from collections import Counter
import json

with open("generated_orders.jsonl") as f:
    statuses = [json.loads(line)["status"] for line in f]

freq = Counter(statuses)
total = sum(freq.values())
for val, count in sorted(freq.items(), key=lambda x: x[1]):
    print(f"{val:30s} {count:5d}  ({100*count/total:.2f}%)")

If any enum member shows 0% at your standard dataset size, you have a coverage gap. The fix is a coverage floor: guarantee at least N records for every enum member before applying weighted sampling for the remainder. Here's the pattern in factory_boy:

import factory
from factory.fuzzy import FuzzyChoice
from myapp.models import Order, OrderStatus

TAIL_GUARANTEE = 3  # minimum records per enum member

def build_coverage_aware_batch(size: int) -> list[Order]:
    all_statuses = list(OrderStatus)
    guaranteed = [
        Order(status=s)
        for s in all_statuses
        for _ in range(TAIL_GUARANTEE)
    ]
    remainder_size = max(0, size - len(guaranteed))
    weighted_statuses = (
        [OrderStatus.FULFILLED] * 82 +
        [OrderStatus.PENDING] * 10 +
        [OrderStatus.CANCELLED] * 5 +
        [OrderStatus.REFUNDED] * 2 +
        [OrderStatus.SUSPENDED_FRAUD] * 1
    )
    remainder = [
        Order(status=factory.fuzzy.FuzzyChoice(weighted_statuses).fuzz())
        for _ in range(remainder_size)
    ]
    import random
    combined = guaranteed + remainder
    random.shuffle(combined)
    return combined

The guaranteed slice ensures every branch is exercised; the weighted remainder preserves distributional realism for load and performance tests. At 500 records with 8 enum members and TAIL_GUARANTEE=3, tail coverage went from a ~34% miss rate (Poisson expectation at p=0.005) to 100% — and the overall distribution still passes a chi-squared test against production proportions for the weighted segment.

For schema-enforced pipelines, encode the coverage floor in a JSON Schema 2020-12 annotation so downstream generators can read it programmatically:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "status": {
      "type": "string",
      "enum": ["FULFILLED","PENDING","CANCELLED","REFUNDED","SUSPENDED_FRAUD",
               "CHARGEBACK_REVERSED","DISPUTED","CLOSED_DORMANT"],
      "x-coverage-floor": 3,
      "x-weight-distribution": {
        "FULFILLED": 0.82, "PENDING": 0.10, "CANCELLED": 0.05,
        "REFUNDED": 0.02, "SUSPENDED_FRAUD": 0.003,
        "CHARGEBACK_REVERSED": 0.003, "DISPUTED": 0.002,
        "CLOSED_DORMANT": 0.002
      }
    }
  }
}

Pair this with a Pytest fixture that reads x-coverage-floor and asserts it post-generation. Schemathesis can consume the same schema for property-based API testing, and it will exercise those tail values automatically if your enum is declared correctly. If you're also seeing gaps because your seed files simply don't list all enum members, the problem compounds — that's the truncated enum set problem and it needs its own remediation pass.

Where Senior Engineers Still Get Burned by Enum Weight Assumptions

Copying production weights directly into factories is the most common mistake, and it's an org-level failure as much as a tooling one. Analytics teams produce frequency tables; engineers paste them into factories without asking whether coverage is the goal. Production weights are correct for load tests and performance benchmarks. They are wrong for functional test suites where every branch must fire. Maintain two factory profiles — a realistic profile and a coverage profile — and be explicit in CI config about which one each job uses.

Assuming large datasets solve the problem is the other trap. Engineers scale up to 10,000 records thinking the tail will fill in. At a weight of 0.003, you expect 30 records — but variance means you might get 14 or 48, and a specific sub-combination (e.g., SUSPENDED_FRAUD + currency=JPY + payment_method=CRYPTO) may still be zero. Sparse seeds starving low-frequency enum paths is a related failure mode where the seed space itself is too thin to generate the combination, regardless of dataset size. Volume is not a substitute for intentional coverage.

Three Myths About Enum Weights That Lead to Silent Coverage Gaps

Myth 1: "Randomness equals coverage." Random sampling from a weighted distribution is not coverage-complete by definition. Coverage requires that every member appears at least once (or N times for combinatorial paths). Randomness is a sampling strategy; coverage is a guarantee. Hypothesis with explicit enum strategies gets you closer, but you still need to assert post-generation that all members are present — Hypothesis shrinks toward failure cases, not toward completeness. Myth 2: "Our enum only has four values, so this doesn't matter." Four values with three binary flags is 32 combinations. At realistic weights, a CI dataset of 200 records will miss a double-digit percentage of those. The cardinality math compounds fast — see the analysis of cardinality skew in synthetic enum data for the combinatorial breakdown.

Myth 3: "LLM-generated test data solves this because the model knows all the enum values." LLMs don't reliably enumerate every member of a schema-defined enum, and they introduce a different failure mode: inventing plausible-sounding values that aren't in the schema at all. The weight distribution problem remains, and you add a validity problem on top. Use LLMs for seed prose and edge-case narrative generation; use schema-driven generators with explicit coverage floors for enum fields. The two approaches are complementary, not substitutes.

Weighted factories are the right tool for realistic data — but realism and coverage are separate contracts that need separate enforcement. Add a coverage-floor assertion to your generation pipeline today: count enum member frequencies post-build, fail CI if any member falls below your floor, and keep your realistic-weight profile for load tests only. If you want to go deeper on the seed-side of this problem, the analysis of sparse seeds and low-frequency enum paths is the natural next read.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.