Web Crawler Definition & Meaning

What Is a Web Crawler?

A web crawler is an Internet bot that systematically browses the World Wide Web in order to create an index of data. Web crawlers are also known as search engine crawlers and search engine spiders.

Web crawlers automatically visit publicly accessible web pages and follow the hyperlinks on those pages to discover new Internet URLs. Web crawlers also store information found on these web pages in a database to build the search engine index.

Web crawlers are important for finding new and updated content on the World Wide Web and indexing that content so it can be retrieved by search engines for relevant queries.

Visit the SEO Glossary

Go to the SEO Glossary to find more terms and definitions that relate to the field of search engine optimization.