This guide explains the AhrefsBot.
Below, you’ll find out what is AhrefsBot, how it works, and ways to control this web crawler on your website using directives the bot obeys.
What Is AhrefsBot?
AhrefsBot is a web crawler that compiles and indexes the link database for the Ahrefs digital marketing toolset. The AhrefsBot crawls the web to fill the link database with new links and checks the status of existing links to provide up-to-the-minute data for Ahrefs users.
There are currently more than 12 trillion links in the database that AhrefsBot has crawled on the Internet. This link data is used by digital marketers and search engine optimization (SEO) specialists to plan, execute, and monitor their online marketing campaigns.
The AhrefsBot is considered a good bot used for marketing purposes that obeys robots.txt rules and Crawl-Delay directives with User-Agent String: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/).
How Does AhrefsBot Work?
AhrefsBot works by automatically visiting publicly accessible web pages and following links on those pages. The process of crawling from link to link enables AhrefsBot to find new URLs and dead links on the Internet to keep its database fresh with link data.
The AhrefsBot crawls more than 6 billion web pages every 24 hours and updates the link index every 15-30 minutes. It uses an algorithmic process to determine the crawl budget for each website. Based on the crawl rate limit and demand it assigns to the website, the AhrefsBot will crawl a different number of web pages during each visit to fetch the link data.
AhrefsBot is also programmed so it does not crawl a website too fast to avoid overloading it, which can lead to timeouts and server errors. It also does not collect or store any information about the websites it analyzes. Neither does AhrefsBot trigger ad views or show up as visitor traffic in Google Analytics.
According to the Imperva Incapsula Bot Traffic Report, AhrefsBot is one of the most active web spiders used by commercial enterprises to crawl websites and retrieve information for digital marketing purposes. AhrefsBot works continuously to give online marketers better insight into the indexing and ranking algorithms of search engines like Google, Yahoo, and Bing so they can better optimize their websites and SEO campaigns.
Controlling AhrefsBot On Your Website
AhrefsBot can be controlled by your website’s robots.txt file to change the frequency of the crawler visiting your website and to block it entirely from crawling links on your domain.
Changing AhrefsBot Crawl Frequency
AhrefsBot crawl frequency can be changed by specifying the minimum acceptable delay between two consecutive requests in the robots.txt file using this markup:
User-agent: AhrefsBot
Crawl-Delay: [value]Crawl-Delay value is time in seconds. For example, Crawl-Delay: 5.
Blocking AhrefsBot from Your Website
AhrefsBot can be blocked from crawling your website by adding the following markup to the robots.txt file:
User-agent: AhrefsBot
Disallow: /AhrefsBot always respects the Disallow directive that instructs the spider not to crawl the website and will prevent AhrefsBot from storing link data about the website in its database; making it unavailable to Ahrefs users. However, AhrefsBot does need time to pick up the Disallow directive if this is a newly added change in your robots.txt file. Once discovered, AhfefsBot will honor the Disallow directive during the next scheduled crawl.
You’ll also want to disconnect Google Analytics and Search Console from your Ahrefs account if you’ve set those up. Otherwise, Ahrefs can still access your private website data for reporting purposes.
AhrefsBot IP List
AhrefsBot uses IP ranges and individual IPs that can be Whitelisted or Blacklisted for website crawling access which you’ll find below. If you need help Blacklisting these IP addresses for your website, then check out this related guide on how to block AhrefsBot with sample code you can copy and paste into your website’s root .htaccess file.
AhrefsBot IP Ranges
54.36.148.0/24
54.36.149.0/24
195.154.122.0/24
195.154.123.0/24
195.154.126.0/24
195.154.127.0/24
51.222.253.0/26AhrefsBot Individual IP Addresses
168.119.64.245
168.119.64.246
168.119.64.247
168.119.64.248
168.119.64.249
168.119.64.250
168.119.64.251
168.119.64.252
168.119.64.253
168.119.64.254
168.119.65.107
168.119.65.108
168.119.65.109
168.119.65.110
168.119.65.111
168.119.65.112
168.119.65.113
168.119.65.114
168.119.65.115
168.119.65.116
168.119.65.117
168.119.65.118
168.119.65.119
168.119.65.120
168.119.65.121
168.119.65.122
168.119.65.123
168.119.65.124
168.119.65.125
168.119.65.126
168.119.65.43
168.119.65.44
168.119.65.45
168.119.65.46
168.119.65.47
168.119.65.48
168.119.65.49
168.119.65.50
168.119.65.51
168.119.65.52
168.119.65.53
168.119.65.54
168.119.65.55
168.119.65.56
168.119.65.57
168.119.65.58
168.119.65.59
168.119.65.60
168.119.65.61
168.119.65.62
168.119.68.117
168.119.68.118
168.119.68.119
168.119.68.120
168.119.68.121
168.119.68.122
168.119.68.123
168.119.68.124
168.119.68.125
168.119.68.126
168.119.68.171
168.119.68.172
168.119.68.173
168.119.68.174
168.119.68.175
168.119.68.176
168.119.68.177
168.119.68.178
168.119.68.179
168.119.68.180
168.119.68.181
168.119.68.182
168.119.68.183
168.119.68.184
168.119.68.185
168.119.68.186
168.119.68.187
168.119.68.188
168.119.68.189
168.119.68.190
168.119.68.235
168.119.68.236
168.119.68.237
168.119.68.238
168.119.68.239
168.119.68.240
168.119.68.241
168.119.68.242
168.119.68.243
168.119.68.244
168.119.68.245
168.119.68.246
168.119.68.247
168.119.68.248
168.119.68.249
168.119.68.250
168.119.68.251
168.119.68.252
168.119.68.253
168.119.68.254AhrefsBot IP Addresses for Cloudflare
If you’re using Cloudflare, the AhrefsBot may be blocked by the Cloudflare firewall. You can try using the IP ranges above or the individual IP addresses below to lift the restriction by adding them to the firewall Whitelist.
51.222.253.0
51.222.253.1
51.222.253.2
51.222.253.3
51.222.253.4
51.222.253.5
51.222.253.6
51.222.253.7
51.222.253.8
51.222.253.9
51.222.253.10
51.222.253.11
51.222.253.12
51.222.253.13
51.222.253.14
51.222.253.15
51.222.253.16
51.222.253.17
51.222.253.18
51.222.253.19
51.222.253.20
51.222.253.21
51.222.253.22
51.222.253.23
51.222.253.24
51.222.253.25
51.222.253.26
51.222.253.27
51.222.253.28
51.222.253.29
51.222.253.30
51.222.253.31
51.222.253.32
51.222.253.33
51.222.253.34
51.222.253.35
51.222.253.36
51.222.253.37
51.222.253.38
51.222.253.39
51.222.253.40
51.222.253.41
51.222.253.42
51.222.253.43
51.222.253.44
51.222.253.45
51.222.253.46
51.222.253.47
51.222.253.48
51.222.253.49
51.222.253.50
51.222.253.51
51.222.253.52
51.222.253.53
51.222.253.54
51.222.253.55
51.222.253.56
51.222.253.57
51.222.253.58
51.222.253.59
51.222.253.60
51.222.253.61
51.222.253.62
51.222.253.63AhrefsBot Summary
I hope you enjoyed this guide on AhrefsBot.
As you discovered, AhrefsBot is a web crawler that compiles and indexes the link database for the Ahrefs digital marketing toolset. The AhrefsBot works continually by crawling the web to fill its link database with new links and checking the status of existing links to find dead URLs. This process provides up-to-the-minute data for Ahrefs users. You change the frequency of AhrefsBot crawling your website and prevent it from accessing your site through the robots.txt file or IP addresses.
