Null Batch Items Slipping Past GraphQL Lists

A GraphQL query returns a list. Your contract test asserts the list is non-null, the schema validates, and CI goes green. Two days later, a downstream consumer crashes because one element inside that list was null — and every layer of your test pipeline missed it. This isn't a schema gap; it's a test data gap. You never generated a batch where a subset of items resolve to null, so you never exercised the code path that breaks.

The problem compounds in batch operations: nodes arrays, edges in Relay-style pagination, and multi-entity loaders all return lists where individual items can legally be null under GraphQL's type system. A field typed [User] is distinct from [User!], and that difference — one exclamation mark — is what separates "the list may contain nulls" from "every item is guaranteed non-null." Most teams write their test data to the happy path and never seed a batch with a null member.

By the end of this article you'll know how to generate batches that include deliberate null items, write assertions that catch item-level nulls specifically, and instrument your contract tests so a single null member in a 50-item list fails loudly instead of silently passing.

Build an API Automation Framework in Python

Learn Python, Behave, GitHub Copilot, APIs, and CI/CD by building a real framework you can finish in a weekend.

Learn more

Why GraphQL's List Nullability Is a Test Data Problem

GraphQL distinguishes four nullability shapes for lists: [T], [T!], [T]!, and [T!]!. The outer bang controls whether the list itself can be null; the inner bang controls whether each item can be null. A resolver returning a list of order records might type it as [Order] — meaning the list exists but individual Order items may be null if, say, a DataLoader fetch fails for one ID in a batch. This is valid GraphQL. It is also a contract surface that almost no test data strategy explicitly exercises.

Standard contract testing tools — Pact, Schemathesis, even hand-rolled JSON Schema checks — validate the envelope. They confirm the field is present, the list is an array, and sampled items match their type. What they don't do by default is assert that every item in the list is non-null when the schema permits nulls. That's a test data responsibility, not a tooling responsibility. If your seed batches never include a null member, the assertion is never triggered. This is a close cousin of the assertion gaps that appear when optional JSON fields collapse to null — the schema says "this can be absent," so nobody tests the absent case.

Generating and Asserting Null-Inclusive Batch Payloads

The first step is seeding batches that actually contain null items. Using factory_boy or a plain Faker-backed fixture, inject nulls at a configurable rate. A 5–10% null rate across a 20-item list is enough to surface most consumer bugs without drowning signal in noise.

import random
from faker import Faker

fake = Faker()

def make_order(nullable: bool = False):
    if nullable:
        return None
    return {
        "id": fake.uuid4(),
        "status": random.choice(["PENDING", "SHIPPED", "DELIVERED"]),
        "total": round(random.uniform(10.0, 500.0), 2),
    }

def make_order_batch(size: int = 20, null_rate: float = 0.1):
    return [
        make_order(nullable=(random.random() < null_rate))
        for _ in range(size)
    ]

That null_rate parameter is the key. Fix it to a non-zero value in your contract test fixtures so the null path is always covered, not just when randomness happens to produce it. Relying on pure randomness for coverage is a trap — see the misunderstanding section below.

On the assertion side, a JSON Schema items check with "type": "object" will fail if an item is null — but only if you haven't also set "nullable": true or left the item schema open. Here's a tighter approach using JMESPath to count nulls explicitly, which works well in Pytest fixtures and in Postman test scripts:

import jmespath

def assert_no_null_items(response_data: dict, list_path: str):
    items = jmespath.search(list_path, response_data) or []
    null_items = [i for i, v in enumerate(items) if v is None]
    assert not null_items, (
        f"Null items at indices {null_items} in '{list_path}'. "
        f"Total items: {len(items)}"
    )

# Usage
assert_no_null_items(response, "data.orders.nodes")

Be aware that JMESPath silently coerces absent keys to null in predicate expressions — a subtlety covered in depth when looking at how JMESPath casts absent keys to null. Use the explicit list comprehension above rather than a filtered projection if you need to distinguish "item is null" from "item exists but a field inside it is null." In a batch of 50 orders seeded with a 10% null rate, this assertion catches failures in under 50ms per test run — compared to debugging a downstream crash that previously took hours to trace back to a null batch member.

For Schemathesis users, override the response strategy to inject nulls into the list before Schemathesis validates the response schema. A custom response_conformance hook lets you mutate the fixture mid-flight without forking the library. Pair this with a broader GraphQL test data strategy that covers DataLoader batching, deferred fields, and error boundary shapes.

Where Senior Engineers Still Get Burned

Asserting only on list length or first item. A common pattern is to check len(response["data"]["orders"]["nodes"]) > 0 and then spot-check nodes[0]["id"]. This passes even when nodes[1] through nodes[4] are null. The fix is a full-sweep null check (the JMESPath loop above) as a mandatory assertion in every list-returning contract test — not just the ones you suspect are risky.

Trusting the GraphQL schema alone to enforce item non-nullability. If the schema says [Order], the server is within spec returning nulls. Teams often assume that because their resolvers "should never" return null items, they don't need to test for them. Resolvers fail. DataLoaders return partial results. A [Order!]! schema change that should have been a [Order]! slips through review. Schema drift is real, and it's the same class of problem as JSON Schema validation passing while the data contract silently breaks. Your test data must cover the gap between what the schema permits and what your consumers can tolerate.

Myths That Keep Null Batch Items Invisible

Myth 1: Random test data gives you null coverage automatically. It doesn't, unless your factories explicitly model the null-item scenario and fix its probability. A Faker-backed factory that generates valid objects 100% of the time will run for months without ever producing a null list member. Coverage from randomness is coverage you can't reproduce, version, or bisect. Deterministic seeds with explicit null injection are the correct model.

Myth 2: GraphQL aliases and field renames can't affect list nullability assertions. They can. When a consumer aliases orders as recentOrders, your JMESPath path data.orders.nodes silently returns null — not because the list has null items, but because the path is wrong. Your assertion passes vacuously on an empty result. This is a distinct but related failure mode: GraphQL aliases shadowing contract assertions can make a null-item check appear to pass when it's actually checking nothing. Always validate the path resolves to a non-empty list before asserting on item contents.

Null batch items are a test data problem before they're a code problem. Add a null_rate parameter to every list-producing factory, sweep every list response with an explicit null-item assertion, and validate your JMESPath paths resolve before trusting a green result. The next concrete step: audit your existing GraphQL contract tests for any list assertion that checks only length or first-item shape — those are the ones hiding this failure right now.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.

Understanding how systems actually work is the first step toward navigating them effectively.

Browse all articles