Free tool

Robots.txt generator

Write a robots.txt that says exactly which crawlers may read which paths, including the AI crawlers.

Your siteUsed for the Sitemap linePaths to keep crawlers out of, one per line

AI crawlersBlocking them keeps you out of AI answers, which is where a lot of buyers now ask first. Most sites should allow.Crawl delay (seconds)Usually empty; Google ignores it

robots.txt

User-agent: *
Disallow: /admin
Disallow: /api
Disallow: /account

The tools are free. The agents are the product.

Paste your URL and get a marketing brief in a minute: positioning, competitors, the communities where your buyers ask, and a reply it would write.

Get my free brief

More free tools

UTM builder Sitemap.xml generator Meta tag checker Meta description generator Schema markup generator LLMs.txt generator