(Credit: PCMag/Michael Kan)
A Google security feature that promises to detect and warn users about scam phone calls is facing concerns it could be abused for spying purposes.
“This is incredibly dangerous,” says Meredith Whittaker, the president of a foundation for the end-to-end encrypted messaging app Signal.
Google previewed the security feature at its I/O developer conference on Tuesday; it uses the company’s Gemini Nano AI model to detect and flag scam phone calls in real time. In a demo, the tech giant simulated a scam call involving a fraudster impersonating a bank. A pop-up message appeared, encouraging the user to hang up.
So far, Google hasn't fully detailed how the system works. In a blog post, it says the feature is designed to detect "conversation patterns commonly associated with scams," suggesting Google’s AI model will potentially listen in on all your phone calls — or at least ones it suspects are coming from a fraudster.
To protect the user’s privacy, the company says Gemini Nano operates locally, without connecting to the internet. “This protection all happens on-device, so your conversation stays private to you. We’ll share more about this opt-in feature later this year,” the company says.
However, Whittaker —a former Google employee— argues that the entire premise of the anti-scam call feature poses a potential threat. That’s because Google could potentially program the same technology to scan for other keywords, like asking for access to abortion services.
“It lays the path for centralized, device-level client-side scanning,” she said in a post on Twitter/X. “From detecting 'scams' it's a short step to ‘detecting patterns commonly associated w/ seeking reproductive care’ or ‘commonly associated w/ providing LGBTQ resources' or ‘commonly associated with tech worker whistleblowing.’”
Cryptography expert Matthew Green also raised the alarm bells, tweeting: "Imagine it [Google’s Gemini Nano] listens to your conversations but it works for and reports to the police.”
Google didn’t immediately respond to a request for comment. But the company is likely aware of the privacy concerns, hence it plans on making the security feature an opt-in function. In 2021, Apple faced a similar backlash after proposing a plan to use consumer iPhones to flag child sexual abuse material being uploaded to iCloud. Apple later abandoned the idea.
In the meantime, other users on social media have expressed support for Google's effort to harness AI to stop scams, which often prey on less tech-savvy users. “Many don’t understand how unprecedentedly privacy-preserving this is. Gemini Nano is sitting on the phone listening and then forgets everything after the call,” tweeted David Dalrymple, a program director for ARIA, a UK research funding agency.