Data leakage through AI assistants: where your pasted text goes

Someone on your team pastes a client contract into a chatbot and asks for a summary. It works, they save twenty minutes, and nobody thinks about it again. But where did that contract go? Is it training the next model? Could a stranger at the provider read it? Is it still sitting on a server somewhere a year from now?

The real answer is "it depends on which product, which plan and which settings", and the details are in the providers' own policy pages. I read the current ones from OpenAI, Anthropic and Google (as of October 2026) so you don't have to guess. Policies change, so treat this as a map of what to check, not a permanent answer.

The four ways pasted text can travel#

When people say "data leakage" with AI assistants, they usually mean one of four different things, and each one has a different fix.

The first is training. Your conversation becomes part of the data used to improve future models. The OWASP entry on sensitive information disclosure warns users about the risk of "unintentionally providing sensitive data, which may later be disclosed in the model's output." That's the scenario most people picture, though it's not the only one.

The second is human review. Some providers have people read a sample of conversations to check quality or safety. The third is retention: how long the text is stored, including after you press delete. The fourth is plain old access inside your own organisation. On a business plan, your workspace admins may be able to see or export what staff typed.

Keep those four apart when you read a policy. A product can promise "we don't train on your data" and still keep it for 30 days, have it reviewed if it's flagged, or hold it longer when the law requires.

Consumer plans: training is usually on unless you turn it off#

The biggest practical difference is between personal accounts and business accounts. Personal accounts are where most surprises live.

OpenAI's help article on how your data is used to improve model performance says that for services for individuals, such as ChatGPT and Codex, "we may use your content to train our models." You can opt out by turning off "Improve the model for everyone" under Settings > Data controls, or through OpenAI's privacy portal. After that, new conversations aren't used for training. Temporary chats aren't used to improve the models while they remain temporary, and they don't show up in your history.

Anthropic changed its consumer terms in August 2025. Its announcement says Claude Free, Pro and Max users (including Claude Code used from those accounts) now choose whether their chats are used for training, and can change that choice in Privacy Settings at any time. The choice also changes retention. If you allow training, Anthropic keeps the data for five years. If you don't, the existing 30-day retention period applies. A deleted conversation won't be used for future training.

Google's Gemini Apps Privacy Hub is controlled by a setting called Keep Activity. With it on, activity is auto-deleted after 18 months by default, and you can change that to 3 months, 36 months or keep it indefinitely. Temporary chats aren't used to train Google's AI models. Chats you have with Keep Activity off are still kept with your account for 72 hours.

Human review is the part people miss#

Training gets the headlines, but a person reading your text is the more immediate risk for anything confidential.

Google is the most direct about this. The privacy hub says "Human reviewers (including trained reviewers from our service providers) review some of the data", and it asks you not to enter confidential information you wouldn't want a reviewer to see. Chats are disconnected from your account before they go to reviewers. Reviewed chats are kept for up to three years, and because they're no longer linked to you, they aren't deleted when you delete your activity. Even with Keep Activity off, Google says it still uses chats to respond to you and to protect Google, its users and the public, including with help from human reviewers.

Feedback buttons are a quiet trap on every platform. OpenAI's help article says that if you give feedback, the entire conversation attached to it may be used to improve its models, even if you've opted out of training. Anthropic's consumer retention page says it keeps data associated with a feedback submission for five years. Google says reviewed feedback and the associated conversations are kept for up to three years. A thumbs-down on a chat containing a client's details can undo the careful setting you chose.

Deleting a chat isn't instant, and safety flags extend retention#

"Delete" on these products means removed from your view straight away and from back-end storage later. Anthropic's consumer page says deleted chats are removed from your history immediately and from its back-end systems within 30 days.

There are exceptions. Anthropic says that if automated trust and safety systems flag a chat, it keeps the inputs and outputs for up to two years and the classification scores for up to seven. OpenAI's enterprise privacy page says deleted Enterprise conversations are removed within 30 days unless it's legally required to retain them. Anthropic's commercial retention page lists legal requirements as an exception too. In other words, a court order or legal hold can keep data around longer than the headline number, and that's outside your control.

Business and API plans: different defaults, different risks#

Business plans flip the training default. OpenAI says it doesn't use inputs or outputs from ChatGPT Business, Enterprise, Edu or the API to improve its models by default. Anthropic's consumer terms update says it doesn't apply to services under its Commercial Terms, which covers Claude for Work, Claude Gov, Claude for Education and the API, including API use through Amazon Bedrock and Google Cloud's Vertex AI. Google's hub notes that using Gemini with a work or school account may come with different data handling terms, so check your Workspace agreement rather than the consumer page.

Retention on business plans is shorter or configurable. OpenAI says it may retain API inputs and outputs for up to 30 days to provide the service and spot abuse, and eligible customers can request zero data retention for some endpoints. For ChatGPT Enterprise and Edu, the customer controls how long data is kept. Anthropic says it automatically deletes API inputs and outputs on its back end within 30 days, with exceptions such as files you've stored, agreed alternatives and policy enforcement.

Access by the provider's own staff is limited but not zero. OpenAI says authorised employees only access Enterprise conversations to resolve incidents, recover conversations with your explicit permission, or where the law requires it.

The catch with business plans is internal. The data now belongs to a workspace, and admins on your side can usually see it. That's mostly a good thing for governance, but staff should know it.

What this means under Australian privacy law#

If you're an Australian business covered by the Privacy Act, the OAIC has already given a clear steer. Its guidance on commercially available AI products recommends that organisations "do not enter personal information, and particularly sensitive information, into publicly available generative AI tools." It also says privacy obligations apply to any personal information you put into an AI system, and to output that contains it. We covered the wider obligations in our earlier post on the Privacy Act and AI tools.

A staff member using a free personal account for client work is the worst combination here. Training may be on, feedback may override the opt-out, and the business has no admin control or audit trail.

A short policy you can actually adopt#

You don't need a 20-page AI policy to fix most of this. Start with a few rules people can remember.

1. Work data goes only into accounts the business controls, on a plan where training is off by default.
2. Personal and client information stays out of consumer accounts entirely, in line with the OAIC's advice.
3. If someone must use a personal account, they turn training off and use temporary chats.
4. Nobody clicks feedback buttons on conversations containing client or staff details.
5. Before adopting a tool, someone reads its current retention and review terms and writes down the answers.

That last step matters most because these pages change. Anthropic's consumer policy changed in 2025 and Google's hub was updated in October 2026. Set a calendar reminder to re-read them every six months. When we set up AI tools for clients at TheAILAB, that written summary of "what happens to our data" is the first thing we hand over, because it's the question staff ask first and the one nobody can answer from memory.

Sources

1. How your data is used to improve model performance — OpenAI Help Center (checked 2026-10-17)
2. Enterprise privacy at OpenAI — OpenAI (checked 2026-10-17)
3. Updates to Consumer Terms and Privacy Policy — Anthropic (28 August 2025) (checked 2026-10-17)
4. How long do you store my data? — Anthropic Privacy Center (checked 2026-10-17)
5. How long do you store my organization's data? — Anthropic Privacy Center (checked 2026-10-17)
6. Gemini Apps Privacy Hub — Google, Gemini Apps Help (last updated 13 October 2026) (checked 2026-10-17)
7. Guidance on privacy and the use of commercially available AI products — Office of the Australian Information Commissioner (October 2024) (checked 2026-10-17)
8. LLM02:2025 Sensitive Information Disclosure — OWASP Gen AI Security Project (checked 2026-10-17)