SEO / FREE ONLINE TOOL

Robots.txt Generator & Tester

Generate a robots.txt file and test URLs against it online. Block AI and SEO crawlers, find syntax errors, and see which rule allows or blocks each URL for Googlebot, Bingbot, GPTBot and other crawlers.

Processed in your browser

Generate a robots.txt

WebsiteAll crawlersAI crawlers

Block SEO backlink crawlers (Ahrefs, Semrush, Majestic…)

Block paths (one per line)Allow exceptions (one per line)Block other crawlers (one name per line)Sitemap URLs (one per line)

Crawl-delay (seconds, optional)

robots.txt

0 characters

Test URLs

Paste URLs or paths, one per line. Results update as you edit the file.

Crawler

URLs to test

Ready when you are.

QUICK GUIDE

How to use this tool

Build a file with the generator or paste your robots.txt, then enter URLs and choose a crawler. Matching follows RFC 9309 and Google: the longest matching rule wins, Allow wins a tie, * matches any characters and $ marks the end of the URL. Each host, protocol and port needs its own robots.txt at the site root. robots.txt controls crawling, not indexing: a blocked URL can still appear in search results.

Your input is processed locally. Use Copy or Download to keep your result.

Keep your workflow moving

Need another step? Try a related tool.