OpenAI contractors review ChatGPT chats and memory summaries for Project Lily

Why it matters

ChatGPT's polished responses depend partly on contractors reviewing real conversations, creating a privacy boundary many users will encounter only after opening the settings menu.

OpenAI is using hundreds of outside contractors to read and evaluate real ChatGPT conversations, including chats containing sensitive personal information, according to a 404 Media investigation published September 14th.

The work, identified in internal documents by the codename "Project Lily," gives reviewers access to anonymized prompts and, in some cases, entire conversations. Usernames are removed, but the text itself can expose personal details. Some tasks also include a "user memories summary" describing previous interactions, location information and other context ChatGPT has retained about the user, 404 Media reported.

OpenAI says it runs the conversations through its Privacy Filter before contractors receive them. That filter is built to detect and mask names, contact details, addresses, account numbers, passwords and other identifiers. OpenAI's own technical description of Privacy Filter says the model can miss uncommon identifiers or ambiguous private references and can over-redact or under-redact text when context is limited.

That limitation matters because removing an account name does not necessarily make a conversation anonymous. A medical history, workplace dispute, home address or detailed family problem can identify someone from context, even after obvious fields such as email addresses and phone numbers have been stripped.

The sharper issue is that ChatGPT's usefulness increasingly depends on context, while OpenAI's privacy protections depend on removing enough context to prevent identification. Memory summaries make that tension explicit: the same accumulated detail that helps the chatbot personalize an answer can make a supposedly anonymized conversation easier to connect to a real person.

The documents do not establish how many users' chats were reviewed, how often the Privacy Filter failed or whether a contractor identified any user. They do establish the access design: OpenAI can retain conversations for model improvement, remove selected identifiers and give the resulting text to outside reviewers. That is narrower than evidence of a data breach, but broader than a filtering error.

How Project Lily works

Reviewers first read a ChatGPT user's prompt and summarize what the user is trying to accomplish. They then compare four responses generated by ChatGPT, highlight at least three passages they consider aligned or misaligned with OpenAI's instructions, and explain their decisions.

Each response receives a score from one to seven. A one represents an unusable answer; a seven means the reviewer believes the response would be difficult to improve meaningfully. Contractors are told to grade for accuracy, usefulness, length, style and whether the model matches the user's tone without copying its intensity.

The guidelines focus heavily on behaviors OpenAI has been trying to suppress. Reviewers are instructed to flag sycophancy, forced mimicry, patronizing assumptions, claims of personal experience and responses that imply the model has human emotions. They also penalize clutter, "AI-speak" and inappropriate emoji use.

The work shows how much human judgment remains behind a product presented as an automated conversation partner. OpenAI needs examples drawn from real use because synthetic tests and benchmark prompts cannot capture the full range of requests, emotional cues and stylistic failures encountered by ChatGPT in production. That access gives reviewers unusually intimate material to work with.

The documents identify the training effort only as Project Lily. They do not name the model being evaluated or establish whether the results are intended for a currently available system or a future release.

The opt-out is buried in data controls

OpenAI's current consumer data FAQ says authorized personnel and trusted service providers may access user content to improve model performance unless the user opts out. It also warns users against entering sensitive information they would not want reviewed or used.

Users can opt out by opening ChatGPT's settings, selecting Data Controls and disabling "Improve the model for everyone," according to OpenAI's Data Controls instructions. The change prevents new conversations from being used for training. It does not retroactively withdraw conversations already de-identified and separated from an account.

Temporary Chats are excluded from model training and are deleted after 30 days, although OpenAI says they may be reviewed for abuse monitoring. ChatGPT Business, Enterprise and Edu content is not used to improve models by default.

OpenAI's privacy policy says deleted personal data is generally removed within 30 days, with exceptions for legal, safety and security obligations or material already de-identified and disassociated from an account for model improvement.

404 Media reported that OpenAI expanded its opt-out help page after the outlet contacted OpenAI. The page now explicitly says humans at OpenAI and trusted service providers may view content for model improvement. That disclosure sits in support documentation rather than in the conversational interface where users are encouraged to treat ChatGPT as an assistant, adviser or confidant.

OpenAI is not alone in using human review. Anthropic's consumer privacy guidance says a limited number of personnel may review de-linked Claude conversations when users enable model improvement. Anthropic also advises users to avoid sharing financial information, health records, passwords and confidential documents.

For founders building AI assistants, Project Lily exposes a product decision that privacy policies tend to flatten into a checkbox. Using production conversations for improvement is not only a retention choice; it is an access-control choice. Operators must decide not just which identifiers to remove, but whether contextual histories and memory summaries should enter a human review queue at all.

The contractors behind Project Lily are doing the manual work required to make ChatGPT sound less mechanical, less flattering and more restrained. Users supply the conversations. OpenAI's settings determine whether those conversations can become training material, and the default leaves that pipeline open until a user closes it. By then, earlier conversations may already be outside the user's control.