What is robots.txt?
Robots.txt is a plain text file placed at the root of a website that instructs search engine crawlers which parts of the site they are allowed or disallowed to access.
How robots.txt works
Robots.txt lives at the root of a domain and uses a simple set of directives to guide crawler behavior. Well-behaved bots request this file before crawling and follow its User-agent and Disallow rules to decide which paths to fetch.
The file controls crawling, not indexing. A URL blocked in robots.txt can still appear in search results if other pages link to it, because the block only stops the crawler from reading the page's content.
Common directives
The core directives are User-agent, which names the crawler a rule applies to, and Allow and Disallow, which grant or restrict access to paths. A wildcard User-agent value applies rules to all crawlers at once.
Many sites also include a Sitemap directive pointing to their XML sitemap, helping crawlers discover the full list of URLs a site wants indexed. This is a widely supported convention even though it is not part of the original standard.
Robots.txt best practices
Use robots.txt to keep crawlers away from low-value or private sections like internal search results, staging areas, or infinite parameter combinations, preserving crawl budget for important pages. Test rules carefully, since a stray Disallow can block an entire site.
To keep a page out of search results, use a noindex meta directive rather than robots.txt, because a disallowed page cannot be read and therefore cannot have its noindex seen. Robots.txt and indexing controls solve different problems.
Frequently asked questions
Does robots.txt prevent a page from being indexed?
No, robots.txt only controls crawling; a blocked page can still be indexed if linked from elsewhere, so use a noindex meta tag to keep a page out of search results.
Where should the robots.txt file be located?
The robots.txt file must sit at the root of the domain, for example at yoursite.com/robots.txt, or crawlers will not find or apply its rules.
Keep reading
Want this machine pointed at your site?
This page is one of a swarm we generated for ourselves. We build the same for you.
Get Started - $10,000