Synthetic Data vs. Real Data: Best Practices for Dev & Staging

A scrubbed copy of production is still personal data under the GDPR. Pseudonymised data stays in scope; only true anonymisation leaves it. Most "anonymised" test databases are pseudonymised at best, which means your staging environment carries the same legal obligations as production and usually none of the controls.

The distinction the regulation actually draws

GDPR Article 4(5) defines pseudonymisation as processing personal data so it can no longer be attributed to a person "without the use of additional information". Recital 26 then makes the consequence explicit: pseudonymised data remains personal data, because the additional information exists somewhere. Anonymous data — where re-identification is not reasonably possible by anyone, with any means — is outside the regulation entirely.

The test is not "did we remove the names". It is whether re-identification is reasonably possible, accounting for the data you hold elsewhere, data a third party holds, and techniques that will exist over the lifetime of the dataset.

	pseudonymised	anonymised
Names replaced with tokens	yes	yes
A mapping back to the person exists	yes, somewhere	no
Still personal data under GDPR	yes	no
Needs a lawful basis, retention limit, breach notification	yes	no
Counts as a safeguard under Art. 32	yes	n/a

Pseudonymisation is a genuine security measure and the regulation credits it as one. It is not an exemption.

Why scrubbing production fails the test

A typical scrub replaces names and emails, truncates card numbers, and leaves everything else. What is left is usually enough.

Quasi-identifiers. Latanya Sweeney's work established that ZIP code, date of birth and sex together uniquely identify a large majority of the US population. Scrubbing the name while keeping those three leaves records that can be re-identified against a voter roll — which is exactly how the Massachusetts hospital discharge data was re-identified in the 1990s, including the state governor's record.

Structure that survives substitution. Replace every name with a random one and you keep: how many orders each customer placed, when, for how much, from which IP range, with which support tickets attached. A single externally-known fact — a date and an amount — picks the row out.

Free text. Support ticket bodies, delivery instructions, internal notes. Nobody scrubs these properly, and they contain names, addresses and phone numbers in prose.

High-cardinality fields nobody thinks of. Precise timestamps, lat/long to six decimals, device identifiers, referral URLs with embedded account IDs.

Satisfying the anonymisation test properly means k-anonymity or differential privacy over the whole record — generalising ZIP to three digits, birth dates to years, suppressing rare combinations. By the time you have done that, your test data no longer behaves like production and has lost the realism you copied it for. The honest options are a proper anonymisation programme or synthetic data. Partial scrubbing gets you the obligations of the first with the realism of neither.

The Patreon breach: production data on a debug server

In September 2015 Patreon was breached, and roughly 2.3 million email addresses plus home addresses and encrypted passwords were published. Patreon's own security notice stated that the unauthorised access took place "via a debug version of our website that was visible to the public", and that "the development server included a snapshot of our production database".

Note the shape of it. Production servers were not breached. A development server was — one with debug mode on, reachable from the internet, holding a copy of real customer data. Researchers at Detectify had reported a publicly exposed Werkzeug debugger to Patreon before the breach, and that is the generally accepted entry point, though Patreon's own notice does not name it.

Every control that would have mattered was a production control. The database was the production database; the environment was not. That asymmetry is the entire risk, and it is why "it's only staging" is not a security posture.

Referential integrity is the hard part

Generating a plausible single record is easy. Generating a *dataset* that behaves like production is not, and this is where naive synthetic data falls over.

Three kinds of consistency you have to maintain:

Within a record. A city, state, ZIP and county that actually go together. Random independent draws produce Miami, WY 90210, which every address validator in your stack will reject — so your tests pass or fail on the validator rejecting your fixtures rather than on the behaviour you meant to test.

Across tables. An order's customer_id must exist in customers. A shipment's address must be the order's address. A refund must not exceed its payment. Generate tables independently and every foreign key is broken.

Distributional. Production has a few customers with thousands of orders and a long tail with one. Uniform random generation gives every customer the same number, so you never exercise the pagination, the N+1 query, or the report that times out on the biggest account. Realistic skew is what finds performance bugs.

A workable pattern: generate the graph top-down, deriving dependent values rather than drawing them.

import random

def make_customer(rng, address_pool):
    # One draw from a pool of internally consistent addresses, not four
    # independent draws of city, state, ZIP and county.
    return {"id": f"cus_{rng.getrandbits(32):08x}", "address": rng.choice(address_pool)}

def make_orders(rng, customer):
    # Zipf-ish skew: most customers have one order, a few have many.
    count = min(int(rng.paretovariate(1.3)), 400)
    return [
        {
            "id": f"ord_{rng.getrandbits(32):08x}",
            "customer_id": customer["id"],          # never invented
            "ship_to": customer["address"],         # derived, so it always matches
            "total_cents": rng.randrange(500, 50_000),
        }
        for _ in range(count)
    ]

"We already use Faker" is not the same thing

Faker and its ports are good at the single-field problem and do nothing about the dataset problem. faker.address.city(), faker.address.state() and faker.address.zipCode() are three independent draws — the same bug as rolling your own, with a nicer API.

// faker-js: three independent draws, no relationship between them
const bad = {
  city: faker.location.city(),      // 'Lake Alexandria'
  state: faker.location.state(),    // 'Wyoming'
  zip: faker.location.zipCode()     // '09876'
};

What Faker gives you is a plausible *value*. What a test fixture needs is a plausible *record*, and then a plausible *graph* of records. Use Faker for the invented parts — names, street numbers, company names, lorem text — and a registry lookup for anything with a real-world relationship: city to state to county to ZIP prefix, area code to state, currency to country.

Seed everything

An unseeded generator makes failures unreproducible. A test that fails on one run in fifty because of a particular generated value is worse than no test: it trains the team to re-run CI until it passes.

Seed from a value you record, and print the seed on failure.

import random, os

seed = int(os.environ.get("TEST_SEED", random.randrange(2**32)))
rng = random.Random(seed)
print(f"TEST_SEED={seed}")   # paste this to reproduce exactly

customers = [make_customer(rng, ADDRESS_POOL) for _ in range(1_000)]

// Node has no seedable Math.random; use an explicit PRNG.
function mulberry32(seed) {
  return function () {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

const seed = Number(process.env.TEST_SEED ?? Date.now() % 2 ** 32);
console.log(`TEST_SEED=${seed}`);
const rand = mulberry32(seed);

Two rules that make this pay off: one seed per test *file*, not per process, so tests stay independent; and never seed from the clock without recording the value.

A checklist worth keeping

• No production dump leaves production. Not to staging, not to a laptop, not to a debug server.
• Treat a scrubbed dump as personal data. If it is in your staging database, staging is in scope for your retention policy, your access controls and your breach procedure.
• Generate test data instead, with referential integrity maintained by derivation.
• Seed it, record the seed, print it on failure.
• Skew the distributions deliberately.
• Keep a small set of deliberately awful fixtures: APO addresses, apostrophes in names, five-line addresses, unique ZIPs, Unicode. Real data has these and uniform generators do not.
• Check debug mode is off and non-production environments are not publicly reachable. That is the Patreon lesson and it costs nothing.

For addresses that hold together across city, county, ZIP and state, the test data generator emits them in bulk as CSV, JSON or SQL inserts.