About the URLinspectorBot
Version : 1.0
Bot Type : Good (well behaved, careful about traffic, identifies itself, has an official moniker)
Category : Site monitoring
Crawling for : Sitemaps, URL Status, Page Titles, Hyperlinks
Obeys Robots.txt : Yes
Obeys Crawl Delay : Not yet (planned)
Robots test tool : Not yet (planned)
User-Agent String : Mozilla/5.0 (compatible; URLinspectorBot/1.0; +https://www.urlinspector.com/bot/)
Reverse DNS suffix: not used yet, will be published here when fixed
IP address range : dynamic, will be published here when fixed
What is URLinspectorBot?
URLinspectorBot is a web crawler that powers the database of web pages and hyperlinks for URLinspector and LinkResearchTools.
This bot crawls web to fill our database with data about websites of our users and new links and checks the status of previously found links to provide the most comprehensive and accurate data to our users.
Link and page status data collected by URLinspectorBot from the web is used by thousands of users of our software to improve their websites.
It is a tool that you can also use to monitor the health of your website, for free currently.
What is URLinspectorBot doing on your website?
URLinspectorBot is crawling your website analyzing links and adding them to our database. It will periodically re-crawl your website to check the current status of previously found links.
URLinspectorBot does not trigger ads on your website (if any) and won’t add numbers to your Google Analytics traffic.
Does URLinspectorBot respect robots.txt file?
Yes. Absolutely.
We strictly respect robots.txt, both disallow and allow rules.
We use the original Google robots.txt library to parse robots.txt files. It is the same library that Googlebot uses to parse robots.txt files.
How to control URLinspectorBot on your website
URLinspectorBot strictly follows the robots.txt file on your website. So you can fully control it on your website if you need.
If for some reason you want to prevent URLinspectorBot from visiting your site, put the two following lines into the robots.txt file on your server:
User-agent: URLinspectorBot
Disallow: /
Please note that URLinspectorBot may need some time to pick the changes in your robots.txt file. This will be made prior to each next scheduled crawl.
Please also note that if your robots.txt contains errors and URLinspectorBot won’t be able to recognize your commands it will continue crawling your website the way it did before. Also a missing or empty robots.txt file will not prevent URLinspectorBot from crawling your website.
You can read more about robots.txt and the Robots exclusion standard
- About Robots exclusion standard at Wikipedia
- Introduction to robots.txt by Google
- How Google (and URLinspector) interprets the robots.txt specification
If you think that URLinspectorBot is somehow misbehaving on your website or if you have any questions about it, please don’t hesitate to contact our support team [email protected].