Parallel bulk inserts expose a class of test data failure that sequential fixture loading hides completely. Your factory_boy suite runs clean at 50 rows; you scale to 500,000 rows across four worker threads and suddenly 3–8% of inserts violate foreign key constraints — not because your schema changed, but because your generation strategy never accounted for concurrent ID space. The FK values Faker emitted in worker 2 reference parent rows that worker 3 hasn't committed yet.

The root cause is a mismatch between how synthetic data libraries produce IDs and how databases enforce referential integrity under transaction isolation. Faker and Mimesis generate values in isolation — they have no shared state, no awareness of commit order, and no concept of a dependency graph. At low volume and single-thread, the insertion order happens to be correct. Under parallelism, that luck evaporates.

By the end of this article you'll understand exactly where the race condition lives, how to restructure your generation pipeline to guarantee FK safety at scale, and which failure modes senior engineers keep reintroducing after they think they've fixed it.

The Race Condition Inside Parallel FK Generation

Referential integrity collapse under parallelism isn't a database bug — it's a generation-order contract violation. A parent row must exist and be committed before any child row referencing it is inserted. In a single-threaded pipeline, generation and insertion happen in topological order: parents first, children second. Introduce a thread pool or multiprocessing pool and that order is no longer guaranteed unless you explicitly enforce it. Each worker pulls a batch, generates FKs by sampling from a locally-held ID list, and fires inserts — but "locally-held" means the ID list was snapshotted before the other workers committed their parents.

This problem is distinct from simple ID collision. The IDs are unique; the referenced rows just don't exist yet at insert time. Postgres with SET CONSTRAINTS DEFERRED will mask it during a transaction, then blow up at commit. MySQL with FOREIGN_KEY_CHECKS=0 will silently accept orphaned rows, poisoning downstream query results rather than throwing an error. Both behaviors make the failure harder to diagnose than a straightforward constraint violation. Understanding how a referential integrity graph drives insertion order is the prerequisite for fixing this at the pipeline level rather than papering over it per-table.

Building a Commit-Safe Parallel Generation Pipeline

The fix has two parts: generate and commit all parent IDs before any child worker starts, then distribute the committed ID space to workers as a read-only shared structure. Don't sample FKs from a pre-generation list — sample from a post-commit query. Here's a minimal pattern in Python using concurrent.futures and Psycopg3:

import psycopg
from concurrent.futures import ThreadPoolExecutor
from faker import Faker
import random

fake = Faker()
DSN = "postgresql://user:pass@localhost/testdb"

def insert_users_batch(batch_size: int, offset: int) -> list[int]:
    with psycopg.connect(DSN, autocommit=False) as conn:
        rows = [(fake.name(), fake.email()) for _ in range(batch_size)]
        result = conn.execute(
            "INSERT INTO users (name, email) VALUES (unnest(%s::text[]), unnest(%s::text[])) RETURNING id",
            ([r[0] for r in rows], [r[1] for r in rows])
        )
        ids = [row[0] for row in result.fetchall()]
        conn.commit()
        return ids

# Phase 1: commit all parent rows, collect real PKs
with ThreadPoolExecutor(max_workers=4) as pool:
    futures = [pool.submit(insert_users_batch, 25_000, i) for i in range(4)]
    committed_user_ids = []
    for f in futures:
        committed_user_ids.extend(f.result())

# Phase 2: child workers sample only from committed IDs
def insert_orders_batch(user_ids: list[int], batch_size: int):
    with psycopg.connect(DSN, autocommit=False) as conn:
        rows = [(random.choice(user_ids), fake.bothify("ORD-####")) for _ in range(batch_size)]
        conn.executemany(
            "INSERT INTO orders (user_id, ref) VALUES (%s, %s)", rows
        )
        conn.commit()

with ThreadPoolExecutor(max_workers=4) as pool:
    futs = [pool.submit(insert_orders_batch, committed_user_ids, 50_000) for _ in range(4)]
    for f in futs: f.result()

The critical discipline is the phase barrier between the two ThreadPoolExecutor blocks. f.result() on every parent future before spawning child workers is what makes this safe — it's a synchronization point, not just error handling. Skipping it and merging both phases into one pool is the most common way this pattern gets broken during refactoring.

At 200,000 parent rows and 800,000 child rows across four Postgres connections, this two-phase approach runs in roughly 11 seconds on a local Postgres 16 instance (M2 MacBook, default config). The naive single-threaded Faker loop with factory_boy SubFactory chaining for the same volume takes around 9 minutes — the per-row round-trip cost dominates. Batching with executemany or COPY is the actual performance win; the parallelism is secondary.

For schemas with three or more FK levels (e.g., tenants → users → orders → line_items), encode the dependency graph explicitly rather than relying on insertion order by convention. A simple adjacency list or a networkx DAG with topological sort is enough — don't manage this in your head or in comments. When generating test data with Faker at scale, the library itself is rarely the bottleneck; it's the missing coordination layer around it.

Where Senior Engineers Still Get This Wrong

Sampling from pre-generation ID lists instead of post-commit queries. Engineers generate 100,000 user IDs in memory using uuid4() or sequential integers, pass that list to child workers, and assume insertion will succeed because the IDs are "planned." The list is a promise, not a fact. Any batch that fails a partial insert (network blip, constraint on another column, deadlock) leaves gaps in the committed ID space. Child rows referencing those gaps will fail or — worse — succeed silently if FK checks are disabled. Always query committed PKs back from the database; never trust the generated list.

Disabling FK constraints for speed and forgetting to re-enable them. SET session_replication_role = replica in Postgres or SET FOREIGN_KEY_CHECKS=0 in MySQL is a legitimate bulk-load optimization, but it requires a post-load validation pass. Teams skip the validation, the orphaned rows survive into the test dataset, and integration tests start returning results that would never occur in production. If you're going to disable constraints, pair it with a Great Expectations suite or a post-load SQL assertion block that verifies FK cardinality before the dataset is marked ready. Property-based testing for data validation can catch these cardinality invariants automatically if you model them as properties.

Myths That Keep This Problem Alive

"factory_boy's SubFactory handles the dependency for me." It does — at one row at a time. SubFactory issues a separate INSERT per parent, which is correct but catastrophically slow at bulk scale and not parallelism-safe without explicit session management. Teams reach for SubFactory because it's the path of least resistance, then wonder why their data generation job takes 40 minutes. The abstraction is correct for unit-test fixtures; it's the wrong tool for seeding a 10-million-row integration dataset. "Random IDs mean realistic coverage." Random FK sampling produces a uniform distribution across parent IDs, which is almost never how production data looks. A real e-commerce dataset has power-law order distribution — a small percentage of users account for most orders. Uniform FK distribution will miss join-skew bugs, partition-pruning edge cases, and query planner regressions that only appear with realistic cardinality ratios.

"If the schema has FK constraints, the database will catch bad data." Only if the constraints are enabled, deferred correctly, and the transaction actually commits atomically. Production-mirrored test environments frequently have constraints disabled for load performance and never re-enabled. The database becomes a passive store with no integrity guarantees, and the test data gradually drifts into a state that would be impossible in production. Auditing constraint status with a schema-inspection query before any generation run is a five-line check that eliminates an entire class of silent corruption.

Referential integrity collapse under parallelism is a coordination problem, not a Faker problem. The fix is architectural: enforce a phase barrier between parent and child generation, sample FKs from committed PKs only, and encode your dependency graph explicitly rather than relying on insertion order by convention. If you're building a pipeline that needs to scale beyond single-table seeding, the test data lake architecture covers how to operationalize these patterns across environments without re-solving them per test suite.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.