Test Data Management for QA Teams

Test data is where compliance, reproducibility and test reliability meet. Copy production data and you have moved personal data into a less protected environment. Generate it randomly at run time and your tests become non-reproducible. This page covers the middle path.

Why production data in test is the wrong default

It is the easiest option, which is why it is the common one. Two separate regimes make it a bad idea, and they apply independently.

PCI DSS states that live account numbers are not used for testing or development — Requirement 6.5.5 in version 4, previously 6.4.3. There is no threshold and no exception for “just a few rows”; the PCI guide covers what else follows from having card data in a system.

Data protection law applies to everything else in that dump. A production copy in staging is still personal data under the GDPR, with the same lawful basis, the same retention limits, and the same breach notification duties — in an environment that typically has looser access control, shared credentials, copies on laptops, and screenshots in tickets.

That last sentence is the practical argument. The regulatory position and the engineering position agree here: test environments are less protected than production by design, because protecting them properly would make them useless for testing. Putting your most sensitive data in your least defended environment is the trade nobody would make deliberately.

Four approaches, compared

Most teams end up with the third, supported by a little of the fourth for the cases that matter most.

The masking trap

Masking looks like the best of both worlds and frequently is not, because the failure mode is silent. Replacing names and email addresses leaves the rest of the row intact, and rare combinations identify people even when every obvious identifier is gone — a postcode, a date of birth and a gender is enough to single out individuals in a surprisingly large share of cases. The UK ICO’s guidance on anonymisation is worth reading before committing to this route.

Done properly, masking is a real engineering project with ongoing maintenance as the schema changes. Done casually, it produces a dataset that is legally personal data while everyone involved believes it is not. Synthetic data has no re-identification risk at all, because there is nobody to re-identify.

Designing synthetic data that finds bugs

Synthetic data that does not resemble the real world will not find real bugs. Six things to design in deliberately:

Distribution. The mix of card networks, countries and order sizes should approximate your actual traffic. A suite that is half American Express when your traffic is mostly Visa spends its effort on a path your users rarely take.

Extremes. The shortest and longest name you will accept, a 12-digit and a 19-digit card, a single-word name, an order with one item and one with two hundred.

International cases. Accented characters, non-Latin scripts, postcodes that are not five digits, countries with no state field. The identity generator produces these deliberately.

Negative cases. A broken checksum, an expired date, a security code of the wrong length. Validation you have never watched fail is validation you have not tested.

Boundaries. A card expiring in the current month, the minimum and maximum order amount, the exact threshold of any rule you have.

Time. Generate dates relative to now. A fixture pinned to 12/25 becomes an expired-card fixture on a date nobody chose, and the resulting failure looks like a code regression.

For card-specific fixtures, the bulk generator handles the distribution and negative-case shares directly, and the payment form checklist lists what each of them is for.

Determinism and seeding

Random data makes flaky tests, and flaky tests destroy a team’s trust in the suite faster than bugs do. The fix is a seeded generator: the same seed produces the same sequence, therefore the same data, therefore the same result.

// A small seeded PRNG. Same seed in, same sequence out — every run, every machine.
function mulberry32(seed) {
  return function () {
    seed |= 0;
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

// Take the seed from the environment so CI can pin it and a developer can
// reproduce a failure with the value printed in the log.
const seed = Number(process.env.TEST_SEED ?? 20260804);
const random = mulberry32(seed);

console.log(`test data seed: ${seed}`);   // print it on every run, especially failures

const pick = (list) => list[Math.floor(random() * list.length)];

Three rules that make seeding actually pay off:

1. Print the seed on every run. A failure you cannot reproduce is a failure you will close as flaky.
2. Pin the seed in CI, and run a scheduled job with fresh seeds to catch what the pinned one never generates.
3. Record the settings, not just the seed. A seed fixes the sequence of random numbers, not the code consuming them — change the generator options and the same seed yields a different set. That is correct behaviour and it surprises people.

Fixtures and factories

Both, for different jobs:

Fixtures are fixed files, committed to the repository and reviewed like code. Their strength is that the input is exactly what it was last time, which is what a regression test needs. Their weakness is maintenance: a schema change means editing files.

Factories build objects at run time from sensible defaults, letting a test override the one field it cares about. Their strength is expressiveness — buildOrder({ total: 0 }) says what the test is about. Their weakness is that the other twenty fields are decided somewhere else, so a factory change can alter tests that never mentioned the field.

The rule of thumb: fixtures for the scenarios you must never break, factories for everything else. The mistake is picking one and forcing it everywhere.

Data lifecycle

• Where it lives. Small fixture sets belong in the repository, next to the tests. Large ones belong in an artefact store with a version, referenced by the test setup.
• When it is refreshed. Set a cadence, because a fixture set slowly diverges from the schema and from reality until one day it tests nothing.
• Cleanup. Every test run should leave the database in a known state, whether by transaction rollback, truncation, or a fresh schema per run.
• No leakage between environments. Staging data should not appear in development and neither should reach production.
• Never move test data into production. Synthetic records that arrive in a production database get treated as real by reports, exports, support tooling and marketing lists, and there is rarely a reliable way to tell them apart afterwards. The traffic is one way.

Test data in CI

• Isolate per build. A database or schema per run, so parallel jobs cannot collide on the same rows.
• Version the seed script with the code it seeds. A seed script that drifts from the schema is a broken build waiting for a slow week.
• Namespace parallel runs. Where full isolation is too expensive, prefix generated identifiers per worker so two runs never claim the same record.
• Watch the seeding time. A setup step that takes two minutes on a suite that runs a hundred times a day is three hours of machine time daily, and it is the first thing to profile when CI feels slow.

A practical setup

For a checkout suite, concretely:

Cards. A committed fixture file of a few hundred generated numbers, covering every network you accept, both American Express and 2-series Mastercard, all five lengths, and a deliberate share of Luhn-invalid rows for negative tests. Generated once with a recorded seed, with the seed in a comment at the top of the file so it can be rebuilt.

Identities. Names and addresses generated per test from a factory, seeded from the same run seed, so international characters and unusual postcodes appear throughout rather than only where someone remembered to add them.

Gateway responses. Your provider’s sandbox cards, not generated ones — approvals, declines and 3-D Secure challenges need a real processor response, which the sandbox reference collects. Mock the provider entirely for unit tests and use the sandbox for the end-to-end path.

Stored cards. Provider tokens rather than card numbers, exactly as in production — tokenised flows have their own failure modes and should be exercised the way they actually run.

Nothing in that setup contains a real person’s data, every part of it is reproducible from a recorded seed, and none of it needs a compliance conversation before a new developer can run the suite on their laptop. That combination is the whole point.

The last property is the one that quietly decides whether any of this survives contact with a deadline. A test data strategy that requires an approval, a VPN, or a request to another team will be routed around the first time someone is in a hurry, and what they route around it with is a copy of production. Making the compliant path the fastest path is not a nicety; it is the only version of this that holds up over a year.