(Credit: NurPhoto/Contributor via Getty Images)
Reddit has struggled to block tech firms like Microsoft, Perplexity AI, and Anthropic from scraping its content—and wants them to understand that its social platform is pay-to-play.
In a recent interview with The Verge, Reddit CEO Steve Huffman doubled down on Reddit's stance against unlicensed AI scraping. Unlike the (convenient) view held by some firms that they should be able to actively scrape any "publicly available" data without permission and for free, Reddit is firmly opposed to this stance.
Microsoft didn't notify Reddit that it was scraping Reddit's content and using it for its AI features in Bing, Huffman says. To make matters worse, Huffman also alleges Microsoft resold Reddit's data after scraping it for free to other search engines via Bing's API.
Reddit has since blocked Microsoft from scraping its site, an effort Huffman calls "a real pain in the ass."
"We’ve had Microsoft, Anthropic, and Perplexity act as though all of the content on the internet is free for them to use," Huffman tells The Verge. "That’s their real position."
Other tech giants have taken this position, too. Apple previously confirmed to PCMag last month that it has used publicly available data to train its upcoming Apple Intelligence. Similarly, Salesforce previously defended its use of YouTube video transcripts and other unlicensed data by arguing that what it used was "publicly available." In all of these cases, however, it's unclear what "publicly available" really means.
Reached for comment, a Microsoft spokesperson tells PCMag: "Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site."
In recent months, Reddit has been making changes to try to control how AI firms use Reddit users' data, posts, and communities. Earlier this year, it struck a $60 million AI licensing deal with Google and also made a deal with OpenAI in May. Individual Reddit users won't be paid if their posts are used, though.
Would Microsoft strike a deal with Reddit? It's unlikely. In response to a Twitter user begging Microsoft to pay Reddit to compete with Google, Microsoft's Head of Search Jordi Ribas rejected the idea. "Bing needs to stand on its own feet like other products in our company," Ribas said.
Reddit frames its licensing agreements as its way of knowing what these companies do with its users' content. “Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used,” Huffman says.
It's unlikely big tech firms like Microsoft will change their tune on "public" data until substantial AI regulation is passed in the US—or some of the high-profile copyright lawsuits against AI firms set a precedent.