What Is Robots.txt?
Robots.txt is a file on a web server that tells search engine crawlers which URLs the crawler can access on the site.
The robots.txt file contains a set of instructions that allow or prohibit bots from spending crawl budgets on specific documents. Adding a robot.txt to a website will tell a bot if it can conduct a crawl or not.
All major search engines recognize the instructions provided in a robots.txt file. However, for the robots.txt rules to be effective, the file must be stored on the website’s server, and a website can have only one such file.
Websites that don’t have the robots.txt file on their servers are crawlable by all bots without any restrictions, so having a robots.txt is not a prerequisite for being indexed on a search engine.
Visit the SEO Glossary
Go to the SEO Glossary to find more terms and definitions that relate to the field of search engine optimization.
The Editorial Staff at SEO Chatter is a team of search engine optimization and digital marketing experts led by Stephen Hockman with more than 15 years of experience in search engine marketing. We publish guides on the fundamentals of SEO for beginner marketers.
