A user pastes a customer list into your assistant to ask for a summary.
That text has now travelled to a third-party provider, been written into your trace store, embedded into your vector index, possibly cached as a semantic entry, and captured in the conversation history you persist.
One request, five copies of personal data — most in places nobody intended as a data store. AI systems accumulate personal data in more locations than conventional applications, and they do it by default.
Where it accumulates
Diagram, as text
- Customer record leads to Prompt sent to provider
- Customer record leads to Index chunks and embeddings
- Prompt sent to provider leads to Prompt and trace logs
- Prompt sent to provider leads to Semantic cache
- Prompt and trace logs leads to Evaluation set
- Erasure request leads to Customer record
- Erasure request leads to Index chunks and embeddings
- Erasure request leads to Prompt and trace logs
- Erasure request leads to Semantic cache
- Erasure request leads to Evaluation set
| Location | How it gets there | Usually overlooked? |
|---|---|---|
| Provider API | Every prompt you send | No |
| Trace and log store | Observability capturing prompts | Yes |
| Vector index | Documents embedded for retrieval | Sometimes |
| Semantic cache | Questions stored as keys | Yes |
| Conversation history | Session persistence | No |
| Long-term memory | Extracted facts about users | Sometimes |
| Fine-tuning datasets | Curated from real interactions | Yes |
| Evaluation sets | Built from production traffic | Yes |
The bolded rows are where teams get caught. Traces, caches, training sets and eval sets are built by engineers for engineering reasons, and rarely enter the data inventory — which is exactly what a privacy review asks for.
Start by mapping this for your own system. You cannot protect data whose locations you cannot list.
What leaves your network
The first question any review asks, and it needs a precise answer.
Which provider, in which region? Data residency requirements may forbid processing in another jurisdiction.
Is it retained? Providers differ, and terms differ by plan. Zero-retention arrangements exist; assuming one you do not have is a common and serious error.
Is it used for training? Business tiers generally exclude this. Verify rather than assume.
Are subprocessors involved? Your provider's providers are in scope for most compliance regimes.
If the answers are unacceptable for some data, the options are to redact before sending, route sensitive workloads to a self-hosted model, or refuse to process that category at all. That is a real argument for self-hosting — often stronger than the cost argument.
Redaction
Detect and replace personal data before it reaches the model, restoring it afterwards where needed.
in: "Email Priya Sharma at priya@acme.com about invoice 8842"
sent: "Email [PERSON_1] at [EMAIL_1] about invoice 8842"
out: restore [PERSON_1] -> "Priya Sharma"Reversible tokenisation keeps the response useful while the provider never sees the values.
Three honest limitations:
Detection is imperfect. Names in unusual formats, identifiers you did not anticipate, and personal data embedded in free text all slip through. Treat it as risk reduction, not a guarantee.
It degrades quality. Heavily redacted text loses meaning, and answers get worse.
It cannot help when the task needs the data. Summarising a customer record requires the record.
Deploy it for logs and traces — where it is nearly free and highly effective — and for provider calls where the task tolerates it.
Deletion
The requirement that exposes weak architecture. A user asks you to delete their data — a legal right in many jurisdictions.
Conventional systems delete rows. AI systems must also handle:
Vector indexes. Documents were embedded; those vectors persist. The retrieval lesson noted deletions must propagate — here it is a legal obligation, not a stale-cache annoyance.
Semantic caches. Their questions are stored as searchable keys, and answers may contain their data.
Conversation history and long-term memory. Extracted facts about them.
Traces. Every request they ever made, with content.
Training and evaluation sets. If their data was curated into a fine-tuning dataset, it is now in a model's weights, and you cannot delete it from a trained model. Retraining without it is the only remedy.
That last point should shape design: prefer not to train on personal data. If you fine-tune on interactions, redact thoroughly during dataset construction, and record provenance so you know which examples came from whom.
Design for deletion up front. Tag every derived record with the user it came from. Retrofitting that across a vector index, a cache and a trace store is significantly harder than including it from the start.
Access control at retrieval
Covered in the RAG lesson and worth restating as a privacy control.
Filter by permission during retrieval, not after. Once a passage is in the prompt it has been disclosed, and instructing the model to ignore it is not a control. This is the same reasoning as the vector search lesson's filtering discussion — and the same trap, since post-filtering both starves small tenants and leaks data.
For multi-tenant systems, ensure the tenant boundary is enforced in the query itself.
Practical measures
Minimise what you send. Retrieve the relevant passage rather than the whole document.
Set retention on everything. Traces, caches, conversations and memories all need a policy and an enforced expiry. Indefinite retention is a decision, usually an accidental one.
Separate identifiers from content. Store user_id with a trace; do not store their name in it.
Encrypt at rest, restrict access, audit reads. Standard controls, applied to stores engineers do not think of as sensitive.
Give users visibility. Let them see and delete stored memories. This is a product feature, a debugging aid, and in several jurisdictions a requirement — the same argument as the memory lesson.
Check your understanding
Common mistakes
Traces stored raw. The most common exposure, and the least inventoried.
Assuming provider terms. Retention and training-use vary by plan; verify.
Deletion that misses derived stores. Vectors, caches, memories and traces are separate copies.
Training on unredacted interactions. Creates data you cannot delete.
Post-filtering permissions. Disclosure has already happened.
No retention policy on caches. They accumulate personal data indefinitely.
Treating redaction as a guarantee. It reduces risk; it does not eliminate it.
Practice
Design the data-protection measures for an assistant that answers questions using customer records. Cover what leaves your network, what is logged, and how deletion works.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Follow one customer's data through every system it touches.
Show a strong answer
What leaves your network.
- Minimise before sending. The model needs the facts relevant to the question, not the full record. Send the order status, not the customer's address, date of birth and payment history.
- Redact identifiers where the task does not need them. Replace names and account numbers with placeholders, and restore them in the response — the model can reason about "CUSTOMER_A" perfectly well.
- Know the provider's terms. Whether inputs are retained, for how long, and whether they may be used for training is a contractual question that determines what may be sent at all. Zero-retention endpoints exist and should be used for anything sensitive.
- Record the legal basis and the data-processing agreement, because a model provider is a processor and that relationship must be documented.
What is logged.
- Log identifiers and shapes, not contents, on paths handling personal data: the chunk IDs retrieved, the token counts, the outcome. Not the full prompt — which is exactly the field observability guidance says to capture, and this is a genuine tension to resolve deliberately rather than by default.
- Where full prompts are needed for debugging, redact before writing, keep a short retention, and restrict access separately from ordinary logs.
How deletion works — designed in, not retrofitted:
- Tag every derived artefact with the subject IDs that produced it: index chunks, cache entries, evaluation cases, trace spans. Deletion then becomes a query rather than an archaeology exercise.
- Delete from derived stores first, then the primary record, so nothing can be re-derived from a source that still exists mid-process.
- Reconcile. A periodic job checking that deleted subjects have no surviving artefacts, because a pipeline added later will forget.
- Exclude personal data from evaluation sets, or the set becomes a permanent copy that no deletion process will ever reach.
The measure that prevents most of this: do not collect what the feature does not need. Every field not sent is a field that cannot leak, cannot be logged and does not need deleting.
A colleague proposes building the evaluation set from real production conversations. Assess the privacy implications and propose a workable approach.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
An evaluation set is permanent by design, which is the crux.
Show a strong answer
The core problem: an evaluation set is designed to be permanent and stable. That is what makes it useful — comparing scores across months requires the same cases. It is also what makes it the worst possible place for personal data, because permanence is directly opposed to retention limits and erasure rights.
The specific issues:
- Retention. The set outlives any reasonable retention period for the underlying conversations.
- Erasure. A deletion request should remove the person's data, but removing a case changes the set and breaks comparability — so there is pressure to quietly not do it.
- Purpose. Data collected to answer a customer's question is being reused to test software, which usually needs a separate basis and may need disclosure.
- Access. Evaluation sets get shared with contractors, pasted into tickets, and committed to repositories, so the access controls around production data do not follow them.
A workable approach:
- Anonymise at capture, not later. Replace names, addresses, account numbers and anything identifying with realistic synthetic substitutes as the case is created. The structure of the conversation — which is what the evaluation tests — is preserved.
- Paraphrase free text where the wording itself could identify someone.
- Keep a documented review step so a human confirms each case is clean before it enters the set. Automated redaction misses things, and the set is permanent.
- Generate synthetic cases for the sensitive categories, informed by real patterns rather than copied from them.
- Where a real case genuinely cannot be anonymised — its specificity is the point — hold it in a restricted set with its own retention and access controls, and accept that it will eventually be removed.
The framing I would offer: the value is in the shape of real traffic, not in the specific people. Anonymisation keeps the value and removes the liability, and doing it at capture is far cheaper than retrofitting it under a deletion request.
Key takeaways
- AI systems scatter personal data into many derived stores — traces, vector indexes, caches, memories, training and eval sets. Map them.
- Traces are the largest and least-inventoried exposure. Redact at capture, not at query time.
- Know precisely what leaves your network: provider, region, retention, training use, subprocessors. Data residency is a legitimate reason to self-host.
- Redaction is risk reduction, not a guarantee, and it degrades quality when the task needs the data.
- Deletion must reach every derived store, and data absorbed into fine-tuned weights cannot be deleted without retraining — so avoid training on personal data.
- Tag derived records with their source user from the start; retrofitting is far harder.
- Enforce permissions during retrieval, because a passage in the prompt has already been disclosed.
Further reading
- Regulation (EU) 2016/679 (GDPR), Articles 5, 17 and 25 — minimisation, erasure, and data protection by design.
- Cynthia Dwork and Aaron Roth, "The Algorithmic Foundations of Differential Privacy" (Foundations and Trends, 2014).
- Nicholas Carlini et al., "Extracting Training Data from Large Language Models" (USENIX Security 2021) — why derived artefacts can leak their sources.
- Nicholas Carlini et al., "Quantifying Memorization Across Neural Language Models" (ICLR 2023).
- NIST Privacy Framework (2020) — mapping data flows, which is what the first exercise above is.