Reddit Gives Google a Search Monopoly
“Reddit” has quickly become one of my favorite search engine keywords. When I want to know what real people are saying about restaurants near me or a hotel I might be reviewing (as opposed to a bunch of ads or questionable AI garbage), adding “Reddit” to the end of my search is a good way to go. way to get it. But going forward, you’ll only be able to use this trick with Google: Microsoft today confirmed that Reddit is blocking Bing’s access to it, and a 404 Media investigation also found that most non-Google sites no longer work with Reddit.
The news comes after an update to Reddit’s robots.txt file, which controls which search engines are allowed to crawl the social media site for information. Search experts later reached out to X (formerly Twitter) to report that they had noticed that Reddit seemed to be blocking Bing and other search engines, after which Microsoft confirmed to Search Engine Land that “Bing stopped crawling Reddit after they implemented an updated robots.txt file as of July 1st, which prevents any crawling of their site.” If it seems like the news took a while to spread, that’s because it can take time to catch on to such behind-the-scenes changes. Bing and other affected search engines will still be able to show old links that their robots have already crawled, but they won’t be able to collect new ones going forward. When reporters noticed something fishy was going on, they asked for details, although Microsoft is the only company to provide official confirmation so far.
However, according to 404 Media, Reddit is not adding new links to other popular Google alternatives such as DuckDuckGo. The publication reported that links on independent paid search engine Kagi are still being updated, but it’s worth noting that Kagi buys part of its search index from Google.
There is currently no official reason for the lockout, but the news coincidentally follows a $60 million-a-year deal that allows Google to train its AI on Reddit user posts . In a statement to The Verge, Reddit spokesperson Tim Rathschmidt said, “This has nothing to do with our recent partnership with Google,” before clarifying, “We have been in discussions with several search engines. We were unable to reach an agreement with all of them because some of them are unable or unwilling to make enforceable promises regarding their use of Reddit content, including its use for AI.”
It’s true that more search engines are getting into the AI game , but reading between the lines, it seems that Reddit’s concerns about its users’ content may actually be a hint that companies will now have to pay to play with the giant social networks, and that at the moment Google is the only one willing to spend money.