The share of web traffic generated by AI has eclipsed human-generated traffic this year (57.7% vs. 42.3%), according to Cloudflare. Cisco recently shared that AI agents generate 450% more network traffic than humans, driven by their ability to gather, create, and send information. With Agentic AI traffic increasing, there’s an emerging problem online: a new race for your business’s discoverability. Do AI agents even know your business exists? And do they surface your website when a user asks for related information? It is like Search Engine Optimization (SEO) all over again, but this time, it’s different.
Yesterday —> Humans browsing the web. Today —> Humans using AI agents to find information. Tomorrow —> AI agents making decisions on a user’s or company’s behalf.
What you optimize for (AI agents or humans) can be a simple economic decision. Who is going to find you, and how? If raising your visibility toward AI bots yields greater returns than optimizing it for humans, the decision is easy. The more strategic question is how you can (still) provide value beyond what AI agents can scrape and process. That’s the part you want to monetize as a business. The rest is part of a new discovery funnel. You can implement three fixes this week that move the needle.
Your website was built for humans to browse, find information, and transact on. Since the 1990s, web designers have turned text-heavy websites into beautiful, effective pieces of online real estate. With the emergence of AI agents and assistants like Claude and Perplexity, there’s just one problem: AI tools are unimpressed by the visual appeal of your website. They can process structured information, such as plain text, more easily and quickly. But few businesses provide it. In fact, they might even block AI vendors from indexing their website, which would make them discoverable by AI agents.
For the past 20+ years, so-called web crawlers have indexed websites to be listed in Google’s search results. A text file, called robots.txt, tells crawlers whether and which pages of a website they are allowed to index. Now, AI crawlers are doing the same, surfacing your website and its content via AI agents. (AI vendors might also scrape your website’s public content in the process to train and improve their AI models.) Extending robots.txt with information about whether and which AI crawlers may index your website gives you granular control over the pages that you want AI agents to surface to users.
robots.txt
A plain-text protocol file at your domain root that instructs web crawlers (Googlebot, GPTBot, etc.) which pages or directories to crawl or skip. It’s a permission layer — not a security control, but a signal crawlers are expected to honor. Critical for controlling AI training data scraping and search indexing scope.
For example, you might extend your robots.txt file using one of the following three options, defining whether and which AI crawlers can index your website in robots.txt:
# (1) Prevent AI crawlers from indexing your site User-agent: * Disallow: /
# (2) Allow select AI crawlers to index your site User-agent: GPTBot Allow: /
User-agent: ClaudeBot Allow: /
User-agent: PerplexityBot Allow: /
# (3) Allow any AI crawler to index your site User-agent: * Allow: /
Key decisions to make as a business owner or marketing leader:
Which pages of your website best position your business and expertise? Consider surfacing those to AI?
Are you okay with AI labs (potentially) training their models on public data from your website?
What do you anticipate to be a greater risk to your business—being or not being discoverable via AI agents?
Thank you for reading The AI MEMO. Feel free to share it with your network.
AI agents navigate websites differently from humans. Structured information in plain text or Markdown format makes it easier for agents to understand what you do and offer when they encounter your brand. But what are the relevant pages or pieces of information they should know about your business? Which pages do you want AI agents to visit and process to surface your website to users (or other AI agents)?
That is the kind of information you can define and manage in a plain-text file called llms.txt. For example, you provide AI agents with key information about your business, products, and services, along with the links to the pages where to find it on your website. The llms.txt file should contain the most important information concisely, without creating a bloated file.
llms.txt
A proposed standard (analogous to robots.txt but for LLMs) that gives AI agents a structured, Markdown-formatted summary of your site. Includes information and links on who you are, what you do, and where key content lives. Designed for agent discoverability: when an AI assistant researches you, llms.txt is the curated front door you control.
You can include additional information about your website and key pages that AI agents should be aware of by creating a more detailed version, called llms-full.txt.
llms-full.txt
The expanded version of llms.txt serves the same purpose but contains the complete content of your site in a single LLM-readable file. Include biographical detail, product and service descriptions, publications, etc. Optimized for deep-context retrieval when an agent or RAG pipeline needs the full picture, not just the summary.
For example, the llms.txt file on my website contains the following information about me in addition to the products and services I provide:
## About
- [About Andreas Welsch](https://intelligence-briefing.com/about/): Founder & Chief Human Agentic AI Officer. 25+ years in global AI leadership at SAP; 2 best-selling books (*AI Leadership Handbook*, *The HUMAN Agentic AI Edge*); 100,000+ LinkedIn Learning students; quoted in CIO, VentureBeat, CNBC, Forbes, InformationWeek, Computer Weekly.
Key decisions to make as a business owner or marketer:
What information do you want anyone to know about your business and its offerings?
Which specific pages do you want surfaced? Include a brief title and the URL for each item.
Get your copy: The HUMAN Agentic AI Edge
Organizations are racing to deploy Agentic AI, yet few are ready for the risks that emerge when employees use AI without structure, standards, or oversight.
The HUMAN Agentic AI Edge offers leaders a practical blueprint for building accountable AI-ready teams that consistently produce high-quality results. Drawing on real-world knowledge and insights from interviews with more than 50 AI leaders and experts, Andreas Welsch shows how to combine human judgment with Agentic AI capabilities to reach the level of performance many organizations expect but rarely achieve. This book prepares you to shape the next generation of AI-ready teams delivering high-quality results with high accountability.
Once an AI agent has found your website and processed the llms.txt or llms-full.txt file, it visits the page that has the relevant information it is looking for. AI agents can process information more easily as pairs of information, such as “title”: “The HUMAN Agentic AI Edge” and “author”: “Andreas Welsch”. You can define this information by adding a schema to your website’s <head> tag.
For example, you can learn more about common schemas that AI agents can process, such as WebSite, Organization, or Product on Schema.org. After you’ve added the schema to your website, validate it via the online tool on Schema.org.
Schema (Schema.org / JSON-LD)
Structured data markup embedded in your site’s HTML that tags entities, such as Person, Product, Organization, and Service. This is a format that search engines and AI systems understand natively. It’s semantic labeling: rather than inferring that a page is about metal fabrication, the markup explicitly declares product, company, SKU, dimensions, and more. JSON-LD is the dominant implementation format.
For example, the schema on my website contains the following information in addition to product- and service-specific information on the respective subpages:
# JSON-LD Schema (example from intelligence-briefing.com)
<script type="application/ld+json"> { "@context":"https://schema.org", "@type":"WebSite", "@id":"https://intelligence-briefing.com/#website", "url":"https://intelligence-briefing.com/", "name":"Intelligence Briefing", "alternateName":"Intelligence Briefing — AI Advisory by Andreas Welsch", "description":"AI advisory, training, and keynote speaking for enterprise leaders by Andreas Welsch — turning AI hype into measurable business outcomes."
/* …and more */
} </script>
Key decisions to make as a business owner or marketer:
What key entities related to your business do you want AI agents to know about? (Review the information on Schema.org.)
What information should AI agents surface about each entity?
Is the generated schema valid? (Use the online validator tool.)
Conclusion
Whether you are a business owner or a seasoned marketer, implementing the three structural additions to your website can make it more discoverable by AI agents: robots.txt controls access to individual pages, llms.txt and llms-full.txt control representation, and schema controls interpretation. Key decisions you will need to make include which pages you want AI agents to access or process to surface results to users and other AI agents.
A word of caution:As your content on the website version for humans changes, you will need to keep your llms.txt, llms-full.txt, and schema in sync as well. For good hygiene, establish a cadence to review and update it at least once a quarter to avoid significant drift between them.
This is just the beginning and the most foundational improvements you can make in order to make your website discoverable by AI agents. Next up is optimizing your website content for AI agents…
Do you want your business to be discoverable by AI agents? Let’s connect to audit your website and implement the key fixes together.
Join my bi-weekly podcast for leaders. Each episode features a different guest who shares their AI journey and actionable insights. Learn from your peers how you can lead artificial intelligence, agentic AI, generative AI & automation in business with confidence.
August 04 - Grant Ecker (Founder & CEO of Chief Architect Network) will share how IT executives can prepare their technology stack for agentic transactions and volume. [More details to follow on my LinkedIn profile…]
August 18 - Michael Carroll (Fellow at LNS Research) will share how to adapt traditional approaches for AI agents to recommend and buy your product. [More details to follow on my LinkedIn profile…]
Upcoming events
Join me or say hello at these sessions and appearances over the coming weeks:
