Question 1

What is a robots.txt file?

Accepted Answer

A robots.txt file is a plain text file placed at the root of a website (e.g., example.com/robots.txt) that tells search engine crawlers and other bots which pages or sections of the site they are allowed or not allowed to access. It follows the Robots Exclusion Protocol standard.

Question 2

How do search engines use robots.txt?

Accepted Answer

When a search engine crawler visits a website, it first checks for a robots.txt file at the site root. The crawler reads the directives (Allow, Disallow, Crawl-delay) for its specific user-agent or the wildcard (*) user-agent, and then follows those rules when deciding which URLs to crawl. Note that robots.txt is advisory — well-behaved bots honor it, but malicious bots may ignore it.

Question 3

Can robots.txt block AI crawlers like GPTBot and ChatGPT?

Accepted Answer

Yes. AI companies like OpenAI (GPTBot, ChatGPT-User), Google (Google-Extended), Common Crawl (CCBot), and Anthropic (anthropic-ai) respect robots.txt directives. You can add specific user-agent blocks for these bots and set Disallow: / to prevent them from crawling your content for AI training purposes.

Question 4

Where should I place my robots.txt file?

Accepted Answer

The robots.txt file must be placed in the root directory of your website so it is accessible at https://yourdomain.com/robots.txt. It must be served as a plain text file (text/plain content type). Each subdomain requires its own separate robots.txt file.

Question 5

Does robots.txt affect SEO?

Accepted Answer

Yes, robots.txt directly affects SEO. Blocking important pages can prevent them from being indexed by search engines, removing them from search results. Conversely, a well-configured robots.txt helps search engines focus their crawl budget on your most valuable pages, improving overall indexation and potentially boosting rankings.

User-Agent Groups

Free Robots.txt Generator — Create & Make Robots.txt Online

Frequently Asked Questions