For years, Google reigned as the undisputed king of online search, shaping not only the content we create for websites but also how we code them – all in the pursuit of higher rankings in search results. However, with the rise of artificial intelligence, this monopoly is weakening. So, what does this mean for developers?
This very topic was extensively discussed at the eCommerce Expo 2024 in London, which I attended in September. Experts unanimously agreed that the future of search lies in relevance and real-time information. Users no longer want a simple list of links; they expect direct answers to their questions – delivered quickly, accurately, and with full context.
The Rise of AI Tools: Context, Relevance, and Real-Time Responses
New AI-powered tools like large language models (LLMs) are responding to the demand for real-time, relevant information by delivering precise, immediate answers rather than a list of results. Among them, ChatGPT and the increasingly popular Perplexity AI stand out, blending traditional search with the power of generative AI. While ChatGPT focuses on providing full-text answers, Perplexity AI goes a step further by offering relevant sources, related topics, and even allowing users to copy, share, or rephrase the generated content.
The TikTok Trend: Search, Simplified
A fascinating trend is the rise of TikTok as a go-to search engine for nearly 40% of internet users born after 1996. Originally a social platform, TikTok now provides fast, easy-to-consume answers through short videos, cutting out the need for lengthy articles or endless clicks. Its visual format not only makes information easier to absorb but also boosts retention.
How is Google Responding?
In response to these shifts, Google has introduced Search Generative Experience (SGE), designed to offer more comprehensive and informative answers directly in search results. However, this innovation has faced significant criticism, with many pointing out that the responses are often inaccurate and lack proper context or verification.
Google’s leadership has responded by noting that SGE is still in its experimental phase and will undergo significant optimization by the end of 2024, thanks to the integration of the new PaLM 2 language model. However, the real question remains: can Google catch up to its rapidly advancing competitors, who continue to improve with every user interaction?
Optimizing Your Website for AI
While Google may be losing some of its dominance, SEO is far from obsolete. AI tools like ChatGPT and Perplexity still rely on crawling websites, tracking keywords, backlinks, and overall page structure. But if you want these tools to display your website in response to relevant queries, you can take a few extra steps:
1. Enable AI Crawlers to Access Your Website
Ensure your website isn’t blocking access to AI crawlers. You can check this by reviewing your robots.txt file, which should not include any Disallow: / commands. If you want to explicitly allow AI crawlers, you can add lines like:
User-agent: GPTBot
Disallow:
Other crawlers to welcome include BingBot, GoogleBot, and CognitiveSearchBot used by Microsoft Azure.
2. Utilize FAQ Sections
AI tools are great at processing structured content. One of the best ways to do this is through FAQ sections. By including clear questions and answers directly on your site, AI tools can quickly recognize the context and offer responses from your website. Moreover, you can further enhance clarity by assigning a specific heading level to each question, which helps to highlight its importance and improve the overall structure.
3. Link to Relevant Sources
AI tools favor well-cited, authoritative content. When sharing information sourced from external sites, ensure that it’s properly linked and easy to trace. The higher the quality of your sources, the more likely AI systems will view your website as authoritative and prioritize it in search results.
A Positive Outlook on the Future
Though the way we search is changing dramatically, the core principles remain the same. This means that the role of web developers remains largely unchanged. However, optimizing for AI-powered search engines opens up exciting new opportunities. Tools like ziptie.dev are emerging, helping businesses analyze their visibility in AI search results and offering insights into how to improve it. Established platforms like Semrush are also expanding their features to accommodate AI-friendly functionalities.
As emphasized at eCommerce Expo, artificial intelligence is now an integral part of our world. The sooner businesses adapt to and leverage these technologies, the better prepared they’ll be for the future.
Sources:
- Why AI Search Engines Failed: Bing and Google
- AI and the Future of Search
- The SEO Mindset Podcast
- Google Search vs. Perplexity AI
- Perplexity AI vs. Google
- Perplexity vs. Google
- Robots.txt: Allow/Disallow
