Prairie Labs
Our Relationship With Search Engines in the Age of Artificial Intelligence
Can search be saved in the age of AI?
The history of search engines is not just a story about better results. It is a story about the layers that stand between us and the web we traverse. In the primitive days of the internet there were directories and portals. Then Google intelligently ranked the web. Next, advertisements began crowding the results page. Now AI is beginning to summarize the web before we reach it.
Searching before Google
Before Google, search was fragmented. Yahoo, AltaVista, Lycos, Excite, Infoseek, HotBot, and others were all trying to become the front door to the internet. Some were directories, others crawler-based search engines. There was no single clean answer to where do I search the web?
Google did not simply arrive as another search box. It arrived in a world where the web was growing faster than people could organize it, and where sorting by relevance was still an unsolved problem.
Google's clever algorithm
Google's early breakthrough was PageRank. Instead of ranking pages only by keyword match, Google treated the web as a network of links. If important pages linked to a page, that page likely mattered more. Instead of just reading the words on the page, it was reading the structure of the web itself. Google made search feel smarter because it used the web's own link structure as a signal of trust.
The brilliance comes partially from Markov chains, which I cannot explain, though this video by Veritasium does a great job explaining them.
The injection of mass advertising
Over time, the search page became more and more of an advertising surface. Google launched AdWords in 2000, placing sponsored text ads alongside search results. At first, the distinction was clear: organic results here, sponsored links there.
But the business model progressed. By Google's 2004 annual report, advertising represented 99% of the company's total revenue. Search was still useful, but it was no longer only a relevance machine. It had also become one of the most valuable markets for user intent in the world.
Over the years the volume and placement of advertisements have increasingly introduced more noise to the equation to a point where generally you must scroll to get past the sponsored links.
Now AI is being inserted into search
Search is changing again, though this time in a way which seems contrary to the quality of received information. Google's AI Overviews are not simply a separate chatbot sitting next to search. They are AI-generated summaries placed above the ordinary Google Search results. In 2024, Google announced that AI Overviews were rolling out to everyone in the United States.
As Google's own AI summary states:
Unfortunately, Google has integrated AI Overviews as a core part of its search engine, and there is no official off switch.
Mind you, Gemini added the highlight touch, not me.
DuckDuckGo's swift response
Recently, in response to Google's imposition of AI on searches, DuckDuckGo, who makes a privacy-first search engine, made a very clear statement by presenting the option to disable AI from the search entirely. Week-over-week installations increased by 30%. I can tell you subjectively that I have started seeing DuckDuckGo ads for the first time in a while, and they all pertain to the non-AI search, though perhaps that's just my algorithm.
They did not reject AI altogether. Their position is that AI should be optional. Users can turn AI features off, tune them down, or use noai.duckduckgo.com for a version of search where AI features are turned off by default. The question is not simply whether search engines should use AI. They undoubtedly will.
Is there a way out?
There may be. Instead of having AI explain the results to you, it may be fruitful for it to instead simply evaluate the quality of each retrieved link and rank them by relevance. Such a system also allows a user to insert their own criteria, meaning that the LLM's analysis may direct the results away from commercial or advertising content.
Theoretically.
References
Wired, Search Engine Shake-Out
Sergey Brin and Lawrence Page, The Anatomy of a Large-Scale Hypertextual Web Search Engine
Google Press Blog, Google Launches Self-Service Advertising Program
Google, Generative AI in Search
Tom's Guide, DuckDuckGo CEO says installs are surging after Google I/O