End-to-End Test Data for Product Tribes

Most CI failures in a product tribe aren't bugs in the code—they're bugs in the test data. The same suite that's green at 9am goes red at noon because a fixture got mutated three days ago in an upstream service and nobody noticed. When you own six services across three teams and a shared Kafka cluster, test data becomes a distributed systems problem, not a unit test problem.

The challenge is threefold: generating coherent data across service boundaries, cleaning up test artifacts before they leak into production deployments, and ensuring that the same logical entity—say, a customer order—remains consistent whether you're testing the order service, the fulfillment service, or the analytics pipeline. This article shows how to build an end-to-end test data pipeline for a product tribe using schema-driven generation, property-based validation, and automated cleanup strategies.

By the end, you'll have a working pattern for cross-service test data that survives deployments, integrates with CI, and doesn't require manual coordination between teams.

What Cross-Service Test Data Actually Requires

End-to-end test data for a product tribe is not a collection of isolated fixtures—it's a dependency graph. When the order service writes an event to Kafka, the fulfillment service expects a schema-valid payload with a customer ID that exists in the identity service. If your test data doesn't respect those constraints, your integration tests will pass locally and fail in staging, or worse, pollute production with orphaned records.

The pattern requires three layers: a schema registry (JSON Schema 2020-12, Protobuf, or Avro) that defines contracts between services; a generation layer that produces data satisfying those schemas and their cross-service invariants; and a cleanup layer that marks test entities for deletion before they reach production. This sits between your unit test fixtures and your production data pipelines, typically orchestrated in CI via GitHub Actions or a dbt test run.

Building a Schema-Driven Generation and Cleanup Pipeline

Start with a shared schema repository. Each service publishes its contract as JSON Schema or Protobuf. Here's a minimal JSON Schema for an order event:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "order_id": { "type": "string", "format": "uuid" },
    "customer_id": { "type": "string", "pattern": "^CUST-[0-9]{8}$" },
    "items": {
      "type": "array",
      "items": { "$ref": "#/$defs/item" },
      "minItems": 1
    },
    "test_marker": { "type": "string", "const": "CI_RUN" }
  },
  "required": ["order_id", "customer_id", "items"],
  "$defs": {
    "item": {
      "type": "object",
      "properties": {
        "sku": { "type": "string" },
        "quantity": { "type": "integer", "minimum": 1 }
      },
      "required": ["sku", "quantity"]
    }
  }
}

Use property-based testing to generate test data from JSON Schema with Hypothesis and the hypothesis-jsonschema plugin. This produces random-but-valid payloads:

from hypothesis import given
from hypothesis_jsonschema import from_schema
import json

schema = json.load(open("order_schema.json"))

@given(from_schema(schema))
def test_order_processing(order_payload):
    # Inject test_marker to flag cleanup
    order_payload["test_marker"] = "CI_RUN"
    response = order_service.create(order_payload)
    assert response.status_code == 201

For cross-service consistency, maintain a canonical ID registry in a lightweight Postgres table or Redis cache. Before generating an order, the test harness queries the registry for a valid customer_id. If none exists, it generates one via the identity service's API and registers it. This ensures referential integrity without hardcoding fixtures. A complete end-to-end test data pipeline example with Kafka and dbt is available in the projects section.

Cleanup happens in two phases. First, mark test entities at creation time with a test_marker field or a created_by: ci-bot tag. Second, run a scheduled cleanup job (cron or GitHub Actions workflow) that deletes or archives records older than 24 hours:

DELETE FROM orders
WHERE test_marker = 'CI_RUN'
  AND created_at < NOW() - INTERVAL '24 hours';

This prevents accidental production deployment of test data. One team reduced staging database bloat from 4.2 GB to 180 MB after implementing marker-based cleanup, cutting query times by 60% in their analytics pipeline.

Production Deployment Cleanup and Candidate Application Pitfalls

The most common mistake is assuming test markers propagate automatically. They don't. If the order service writes an event to Kafka with test_marker: CI_RUN, but the fulfillment service strips unknown fields during deserialization, the marker is lost. Your cleanup job never sees it, and the test order ships to a real warehouse. The fix: include the marker in your Protobuf or Avro schema as an optional field, and enforce its presence in API contract tests.

The second pitfall is cleanup racing with test execution. If your cleanup cron runs every hour and your integration suite takes 45 minutes, you'll delete entities mid-test. Use a grace period (e.g., 2 hours minimum age) and tag each CI run with a unique run_id. Only delete entities from completed runs, verified by querying your CI system's API for finished workflows.

What Teams Get Wrong About Cross-Service Test Data

Many teams believe that snapshotting a staging database is sufficient for end-to-end tests. It's not. Snapshots capture state, not behavior. When the identity service changes its customer ID format from numeric to alphanumeric, your snapshot still has the old format, and your tests pass until production breaks. Schema-driven generation catches this immediately because the regex pattern in your JSON Schema fails. Modern environment sync strategies use schema registries and streaming replication, not static snapshots.

Another myth: randomness equals coverage. Hypothesis generates random data, but without constraints it produces edge cases your services will never see (e.g., an order with 10,000 line items). Use hypothesis.strategies.integers(min_value=1, max_value=20) to bound your generators to realistic ranges. One team found that 80% of their Hypothesis-generated failures were impossible in production; tightening constraints reduced false positives by 70% and uncovered two real bugs in pagination logic.

Start by auditing which services in your tribe share entities, then publish their schemas to a central repository. Implement marker-based cleanup for your highest-volume test entity—usually users or orders—and measure the reduction in staging database size after one week. For schema-driven generation patterns and property-based workflows, see JSON Schema and Test Data: A Complete Guide.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.