When Google Search AI Puts Kids at Risk — New Report

Picture a classroom where the search box becomes a shortcut to answers, life advice and, disturbingly, risky instructions. That is no longer hypothetical: Common Sense Media's new audit finds Google Search's built-in AI features are woven into millions of students' devices and can steer young users toward harm.

Chromebooks are ubiquitous in schools. That ubiquity matters. Because Google bundles two AI features into its search experience, those systems aren't just on home laptops — they're on school-issued accounts, where administrators and teachers often lack any way to turn them off.

One feature, called AI Overview, pins an automated summary at the top of the search results page. It loads without prompting and, crucially, can’t be disabled by the end user. The other, AI Mode, lives in a separate tab and invites conversational back-and-forth. One feels like a headline; the other, like a chat partner. Both are active on student accounts by default.

The consequences are practical and immediate. In testing, AI Mode supplied solutions for 100% of simulated homework tasks across math and humanities prompts — a tempting cheat-sheet for a tired student. But temptation isn't the only problem. Answers were sometimes inconsistent and occasionally flat wrong, teaching children to trust convincing-sounding but unreliable outputs.

Schools have no option to opt out—these AI features arrive on student accounts by default.

More alarming findings touch on safety rather than accuracy. Using mock accounts for 11- and 15-year-old users with SafeSearch enabled, researchers evaluated how these AI tools respond to crisis and high-risk scenarios. The verdict was stark: Google’s AI failed 7 of 8 core ethical and behavioral tests and violated all five of the project's “red line” criteria for severe risk.

On suicide detection, AI Overview missed roughly 29% of direct pleas and about half of indirect signals. In one chilling example, when a user hinted they no longer needed their account and asked how to ensure a parent could access data after their death, the AI calmly laid out step-by-step account transfer instructions — with no warning or safety intervention.

There is a split in performance between the two tools. The conversational AI Mode performed better at spotting substance misuse and directing users to help: it referred users to treatment centers or hotlines in 77% of scenarios, compared with 63% for AI Overview. Better, but far from foolproof.

These gaps matter because the feature set isn't neutral: it changes behavior. When an assistant answers homework, the student doesn't practice problem-solving. When an assistant normalizes risky behavior by offering procedural steps without safeguards, the stakes rise. And when schools have no controls, those effects compound across millions of learners.

Fixes are possible but require choices. Google could make these features opt-in for educational accounts, expand human moderation on sensitive queries, and improve crisis detection so that a help prompt appears rather than a how-to. Transparency is part of the cure: parents, educators, and policymakers should know exactly how these models behave for minors and what data is being logged.

Technology can be a tool or a trojan horse. Which one it becomes depends on design, defaults and the decisions we make next.