Free, runs in your browser

Robots.txt tester and generator: see what Google and AI crawlers can fetch

Paste your robots.txt, type a page address and pick a crawler. The tool shows whether that page is allowed or blocked and which line decided it, using the matching rules Google documents. You can also build a file from an example and choose which AI crawlers to allow.

robots-txt-tester v1.0 by media87

1. Your robots.txt

Build a file with AI crawler choices

Block these AI crawlers:

This replaces the text in the box above. Everyone else stays allowed.

2. Test pages

A full address works too; only the part after the domain is tested.

Result

PageVerdictDecided by
Same first page, every crawler in the list
CrawlerVerdictGroup used

3. Checks on the file

    Nothing you paste leaves your browser. The text is kept on this device only, so you can come back to it. The tool does not fetch your live robots.txt: copy it from yourdomain.com/robots.txt.

    How it works

    The rules the tester follows

    1. One group per crawler

    A crawler obeys only the most specific group that names it, such as User-agent: Googlebot-Image. If none does, it uses User-agent: *. Groups are not combined with the * group.

    2. Longest rule wins

    Of all the Allow and Disallow lines that match the address, the one with the longest path is used. If an allow and a disallow are equally long, the less restrictive one, allow, wins.

    3. Wildcards and case

    * matches any characters and $ marks the end of the address. Paths are case-sensitive, so /Admin/ and /admin/ are different. Crawler names are not.

    Blocking a page is not the same as hiding it

    A robots.txt rule asks crawlers not to fetch a page. Google says a blocked page's content cannot be read, but the address may still be indexed and shown in results without a description. To keep a page out of search results, Google's separate guidance is to let it be crawled and mark it noindex, or to put it behind a login. Never use robots.txt for private files: the file itself is public, and anyone can read the paths you list.

    Blocking the files a page needs to display, such as its stylesheets and scripts, can stop Google from seeing the page the way a visitor does. That warning in the checks is our advice. If your site is built with a framework that loads content with JavaScript, test the page with Search Console's URL Inspection as well.

    AI crawlers: training, search and visits on request

    The big AI companies publish separate crawler names, so you can allow one purpose and block another. From their own help pages:

    NameCompanyWhat it is for
    GPTBotOpenAICollects content that may be used to train OpenAI's models. Respects robots.txt.
    OAI-SearchBotOpenAIShows websites in ChatGPT search. Respects robots.txt; blocked sites do not appear in ChatGPT search answers, though they can still appear as plain links.
    ChatGPT-UserOpenAIOpens a page when a person asks ChatGPT about it. OpenAI says robots.txt rules may not apply because the person starts the visit.
    ClaudeBotAnthropicCollects web content that could help train Anthropic's models.
    Claude-SearchBotAnthropicImproves the relevance of search results. Anthropic says blocking it may reduce your visibility in them.
    Claude-UserAnthropicOpens a page when a Claude user asks a question that needs it. Blocking it may reduce visibility in user-directed search.
    Google-ExtendedGoogleA robots.txt token, not a separate crawler. Controls whether content Google has crawled can be used for Gemini model training and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search.

    Whether to block is your business decision. If you want to be cited by AI answer tools, blocking their search crawlers works against that; blocking only the training crawlers is the common middle path. Nobody can promise that allowing a crawler leads to a citation. See AI search visibility for how Media87 approaches that work.

    Method and sources

    The matching rules come from Google's page How Google interprets the robots.txt specification (page updated 2026-10-23, checked 2026-10-30): the supported fields (user-agent, allow, disallow, sitemap), the 500 KiB limit with later content ignored, crawl-delay not supported, * and $, longest-match precedence with the less restrictive rule winning ties, case-sensitive paths, case-insensitive crawler names, and one group per crawler. Google's common crawlers page (updated 2026-10-15, checked 2026-10-30) covers Googlebot-Image, Googlebot-Video, Googlebot-News, Storebot-Google and Google-Extended. For the AI crawlers: OpenAI's bots page and Anthropic's help article (dated 2026-04-07), both checked 2026-10-30.

    Choices we made: Googlebot-Image, Googlebot-Video and Googlebot-News fall back to a Googlebot group when they have none of their own; any other crawler name falls back to * after an exact name match. Other search engines and bots may read edge cases differently, so treat a result for a non-Google crawler as a close guide, not a guarantee. Percent-encoded characters are compared as typed, and the tool never visits your site, so it cannot tell you whether a page exists or is already indexed. Use Search Console's robots.txt report and URL Inspection for the live view.

    Examples like the WordPress and online-store files are starting points we wrote, not rules Google requires. Check every line against your own site before you upload the file to your site's top-level folder.

    Questions

    Robots.txt questions

    In the top-level folder of the site, so it opens at https://yourdomain.com/robots.txt. Crawlers do not look for it in subfolders, and each subdomain needs its own file.

    No. It stops crawling, not indexing. The address can still appear in results without a description. To keep a page out of results, allow crawling and use a noindex tag, or protect the page with a login.

    No. Google supports only user-agent, allow, disallow and sitemap. Anthropic says its crawlers do honour Crawl-delay. A Noindex line in robots.txt is ignored by Google.

    Nothing is blocked. A Disallow: with no path does not match any page, so it is the same as allowing everything for that group.

    These are not Google's crawlers, so blocking them does not change how Googlebot crawls. Google-Extended is also not used for Search ranking, according to Google. Blocking search-type AI crawlers such as OAI-SearchBot can reduce how often your pages show in those tools.

    It is supported and helps other crawlers find it. Use the full address starting with https://. It does not replace submitting the sitemap in Search Console.

    Only in your own browser, so you can come back to it. Nothing is sent to Media87.

    Crawlable, then findable

    Fix the basics first,
    then grow the traffic.

    Crawl rules are one small part of technical SEO. Media87 SEO starts from $500 a month and covers on-page and off-page work. Pair this tool with the snippet preview and the hreflang generator.

    See SEO services in Dubai

    Also: SEO guide for Dubai businesses · AI search visibility · all Media87 tools