Guides to files bots and agents read
Practical, opinionated writing on llms.txt, llms-full.txt,
sitemap.xml and robots.txt and the wider craft of making your site securely legible to AI Agents and Search bots.
One line for AI training
Anthropic, Microsoft, Google and OpenAI back a shared robots.txt group,
__usage_ai-training. What it does, the two catches that can undo it,
and what's new in Lab451.
The hidden risks in llms.txt
Hallucinated packages, lapsed domains and invisible text can turn a helpful file into instructions an attacker wrote. How each risk works, and how we catch them.
Read article →robots.txt for SEO and GEO
Spend crawl budget on pages that matter, choose which AI systems can cite you, and point every bot straight at your sitemap and llms.txt.
Read guide →The complete llms.txt guide for 2026
What llms.txt is, why it exists, who actually reads it, and how to write one correctly — including the mistakes that quietly tank the whole exercise.
Read guide →llms.txt vs llms-full.txt
One is a map, one is the territory. Here's when to use each, how they work together, and which AI models read which file.
Read comparison →How to add llms.txt to WordPress
Four working methods, ranked from least-technical to most. SFTP, plugin, functions.php, or native WordPress.com — pick what matches your hosting.
Read tutorial →How to add llms.txt to Shopify
Shopify now serves the file natively (and redirects to /agents.md).
Here's how to override the default with your own content.