Robots.txt Definition & Meaning

What Is Robots.txt?

Robots.txt is a file on a web server that tells search engine crawlers which URLs the crawler can access on the site.

The robots.txt file contains a set of instructions that allow or prohibit bots from spending crawl budgets on specific documents. Adding a robot.txt to a website will tell a bot if it can conduct a crawl or not.

All major search engines recognize the instructions provided in a robots.txt file. However, for the robots.txt rules to be effective, the file must be stored on the website’s server, and a website can have only one such file.

Websites that don’t have the robots.txt file on their servers are crawlable by all bots without any restrictions, so having a robots.txt is not a prerequisite for being indexed on a search engine.

Visit the SEO Glossary

Go to the SEO Glossary to find more terms and definitions that relate to the field of search engine optimization.