AI Observatory Exposes Blind Spots in ChatGPT and Claude Usage Reports
AI Observatory analysis finds company usage reports miss personal and sensitive chats, exposing data gaps that could distort AI policy and risk decisions.
Summary
Published August 18, 2026, the public AI Observatory, co-led by Stanford Trustworthy AI Research Lab PhD candidate Anka Reuel and recent MIT Media Lab PhD Shayne Longpre, aggregates 85,633 conversational turns across 24,521 conversations, seven consented real-world datasets, 5,000 users and 52 models, including ChatGPT, Gemini, Claude and Grok, from 2023 to 2025. Built with researchers from MIT, Stanford, the Data Provenance Initiative and other institutions, it will open data to researchers and expand over time. Voluntary submissions probably underrepresent sensitive use, so findings do not represent all AI activity.
Applying Anthropic Economic Index methods, which exclude nonwork Claude chats, would remove 48% of Observatory conversations. Those excluded chats contained more health and relationship material, 44.2% versus Anthropic's 31.2%; adult or illicit topics, 7.9% versus 2.1%; harassment and hate, 27.5% versus 5.66%; and sexual content, 16.7% versus 2.4%. OpenAI's 2025 ChatGPT report likewise found only 30% of consumer use was work-related. Anthropic's latest index analyzed 1 million Claude conversations and OpenAI analyzed 1.5 million, dwarfing the Observatory sample. Anthropic said its reports follow specific research questions and supported independent work; OpenAI did not comment.
WildChat conversations grew longer through rising prompt tokens, response tokens and turns; small talk increased as assistant self-disclosure declined, suggesting more companionship, while sensitive exchanges fell, potentially reflecting better safeguards. Grok and Gemini favored information retrieval; Grok concentrated news, politics and misinformation, and xAI did not comment. Anthropic led coding, Gemini social and roleplay use, and ChatGPT homework. GPT-3.5 chats were shorter than GPT-4o's longer, iterative exchanges, a version associated with emotional addiction. University of Texas at Austin assistant professor David Widder said a unified view reveals patterns siloed in Anthropic posts on companionship and CSAM; researchers warn proprietary, selectively released company data cannot independently support consequential benefit, risk and policy judgments.
Positives
- 85,633 conversational turns across 24,521 chats provide an independent view spanning 5,000 users and 52 models.
- Sensitive exchanges became less frequent from 2023 to 2025, potentially indicating that platform safeguards improved.
- The AI Observatory will release its data to researchers and plans to expand its datasets over time.
- Model comparisons identify distinct uses for Anthropic, Gemini, ChatGPT and Grok that company reports often miss.
- Anthropic said external independent research is important, despite limiting its own publications to specific research questions.
Risks & concerns
- Anthropic's methodology would exclude 48% of Observatory conversations, disproportionately omitting personal, sexual, illicit, hateful and harassing content.
- The Observatory's voluntary datasets probably underrepresent sensitive behavior and cannot characterize all generative AI use.
- Anthropic analyzed 1 million Claude conversations and OpenAI analyzed 1.5 million, far exceeding the Observatory's 24,521-conversation sample.
- Misinformation concentrated on Grok, which users frequently consulted for news and politics; xAI did not respond.
- WildChat showed increasing small talk alongside declining chatbot self-disclosure, suggesting growing companionship without equally visible reminders that assistants are artificial.
- Proprietary chat data leaves researchers and policymakers unable to independently verify company narratives about AI benefits and harms.