Search Engine Crawler Definition & Meaning

What Is A Search Engine Crawler?

A search engine crawler is an Internet bot that systematically browses the World Wide Web in order to create an index of data. Also known as a web crawler or search engine spider.

Search engine crawlers analyze a web page’s content so they can store that information in the search index to retrieve it when necessary. Search engine crawlers also crawl hyperlinks to discover new documents on the web.

In addition to conducting an initial site crawl that indexes pages, these Internet bots also revisit pages occasionally to check for updates. Search engine crawlers are not programmed to crawl every page on the Internet; however, the number of links leading to a page, the amount of traffic a page gets, how often the content changes and countless other factors determine the crawl frequency and budget for each website.

Learn more about the SEO Chatter mentorship

Website owners have the right to deny crawler access to specific pages and directories by setting up rules in the domain’s robot.txt file.

Visit the SEO Glossary

Go to the SEO Glossary to find more terms and definitions that relate to the field of search engine optimization.