It decides which search engines and AI agents may read everything else, including your sitemap.xml, llms.txt and llms-full.txt.
Lab451 generates all four visibility-first, every package, hostname and hidden line inside them, and monitors them for drift. Essential for every site, free for most.
Visibility-first
/robots.txt before anything else on your domain. One stray Disallow can hide your sitemap and llms.txt from the very search engines and AI agents they were written for, while a Sitemap: line is how crawlers find the rest of your site.
Lab451 writes robots.txt to be found: search engines and AI search and retrieval agents allowed, your live sitemap declared, private paths closed, and AI training crawlers handled as a separate, deliberate choice.
User-agent: __usage_ai-training, proposed in an IETF draft. It says how your content may be used for training, not whether it may be crawled, so it sits alongside your per-crawler rules rather than replacing them. Lab451 adds it only if you choose to, flags a misspelled or contradicted token in the Checker, and alerts you when it changes.
Fast, and then careful
Drop in any public website URL. No plugins, no code, no OAuth dance.
One crawl maps your site and builds all four files to spec: robots.txt from your detected platform, a sitemap with real dates, and llms files from the content a human actually sees.
Every reference is checked before you publish. You get a severity report on anything that doesn't resolve: a sitemap that isn't live, a package nobody registered, a domain that lapsed, text hidden from human readers.
Plugins, CMS updates and CDN settings rewrite robots.txt without asking. Checker and Monitoring catch a newly blocked agent or a broken file before it costs you visibility.
What you get
robots.txt and sitemap.xml are read by every search engine and AI crawler on the web. llms.txt and llms-full.txt are read by AI agents, coding assistants and retrieval pipelines, but only the ones robots.txt lets in. We build all four to work together, then check what's inside them.
robots.txtThe gatekeeper every crawler reads first. Built from your site: platform detected, sitemap confirmed live before it's listed, search and AI retrieval agents allowed, training crawlers in their own group - File example:
sitemap.xmlDeclared in robots.txt so every search engine finds it. Real lastmod dates read from your server, not today's date stamped on every URL - File example:
llms.txtA short Markdown map of your site for AI agents. Coding assistants like Cursor and Claude Code fetch it when you point them at your docs - File example:
llms-full.txtThe full text of every page in one document, for retrieval pipelines and agents that want the whole picture, not just a link list - File example:
robots.txt FAQ
A robots.txt written to be found first. Search engines and AI search and retrieval agents are allowed in, your live sitemap is declared with a Sitemap: line, and only private paths such as admin, cart and internal search are closed. Whether AI training crawlers may use your content is handled as a separate, deliberate choice rather than as a side effect of one blanket rule.
Crawlers that follow robots.txt read it before fetching anything else on your domain, including /sitemap.xml, /llms.txt and /llms-full.txt. A Disallow: / aimed at an agent hides those files from that agent too. The Sitemap: line is the standard way robots.txt points search engines to your sitemap. llms.txt has no robots.txt directive of its own; agents look for it at its fixed path, so it only helps if robots.txt lets them fetch it.
It depends on which ones. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content for model training. Search, retrieval and user-requested agents such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot fetch pages to answer questions and cite sources. Blocking the second group removes your site from AI answers. Lab451 writes these as separate groups, so you can allow one and refuse the other. robots.txt is honoured voluntarily by well-behaved crawlers; it is not access control.
Yes, as a proposal. An IETF draft defines a shared group, User-agent: __usage_ai-training, that Anthropic, Microsoft, Google and OpenAI have said they will honour (Microsoft from 1 February 2027). It states how your content may be used, not whether it may be crawled, and User-agent: * does not count as one. Lab451's generator can add it as an opt-in, and the Checker reports what yours says. How the AI training token works →
Every package name is checked against public registries, every hostname is resolved, and text hidden from human readers is stripped. In robots.txt, sitemaps are confirmed live before they are listed. Anything that doesn't resolve appears in a severity report next to the file, before you publish it.
CMS and SEO plugin updates, hosting migrations and CDN settings can all rewrite robots.txt. Run the free Checker on your domain at any time, or use Monitoring (included with Pro and Agency), which re-checks robots.txt and the files it governs six times a day and emails you when an agent is newly blocked or a file breaks.
Yes for most sites: up to 50 pages per crawl and 3 sites a day, with verification included. Pro and Agency add deeper crawls, higher daily limits, history and monitoring. See pricing.