Under the hood
From URL to four production-ready files in about 30 seconds. Here's exactly what happens in between.
Enter any publicly accessible website URL. Lab451 accepts root domains, subdomains, and specific path prefixes. No authentication, no API key, no installs required on your end.
/blog), you can specify a path prefix and we'll limit the crawl scope accordingly.
We send a polite crawl bot (identifiable by its user-agent Lab451Bot/1.0) to walk your site's link graph. It respects your existing robots.txt — even the one it's about to replace.
Follows internal links to discover all public pages
Strips nav/footer/boilerplate, extracts main content
Reads title tags, meta descriptions, Open Graph data
Finds publish dates for accurate sitemap lastmod values
With your site fully mapped, Lab451 compiles each file individually. Each is built to a specific spec:
llms.txt
llmstxt.org spec
A concise, Markdown-formatted document describing your site: its purpose, main sections, key topics, and preferred AI interaction guidelines. Think of it as a README for AI models.
llms-full.txt
Extended spec
A full-content companion that includes the actual text of every page (cleaned and structured). Used by RAG pipelines and AI systems that do deep document retrieval rather than just crawling.
sitemap.xml
Sitemaps.org protocol
A standards-compliant XML sitemap with accurate lastmod dates, appropriate changefreq hints, and canonical URLs. Submitted-ready for Google Search Console.
robots.txt
RFC 9309
Configured to welcome major AI crawlers (GPTBot, ClaudeBot, Googlebot, etc.) while blocking known scrapers. Auto-references your new sitemap.xml location.
Download all four files in a single ZIP, then upload them to your web server's root directory. That's literally it. No ongoing maintenance required on the free plan.
AI companies run their crawlers on their own schedules — typically every few weeks. Once your files are live, the next crawl will pick them up. There's no "submit to AI" button (yet) — this is the right way to do it at scale.
When significant content changes, such as new product features, updated API documentation, or updated FAQs. For active sites, a weekly or bi-weekly update cadence is recommended.
Compatibility
The proposed standard that helps large language models understand what your site is about. Presently supported by:
GPTBotClaudeBotGooglebotPerplexityBotxAI-BotBingbotMistralBotMeta-ExternalAgent