IP2Location Expands Search Crawler List with PetalBot, Moz dotbot and MJ12Bot

KUALA LUMPUR, Malaysia - Aug. 27, 2026 - PRLog -- IP2Location today announced the addition of three new search crawlers: PetalBot, Moz dotbot, and MJ12Bot to its Search Crawler List, providing website administrators with more options for identifying and managing legitimate search and web data crawlers.

The updated list is available through the free Search Crawler List from IP2Location, which helps webmasters identify crawler IP addresses and generate firewall rules for allowing or restricting crawler traffic.

Search engine crawlers, also known as spiders or web robots, automatically visit websites to discover and analyze content. Allowing legitimate crawlers to access a website can help ensure that its content is properly discovered, indexed, analyzed, and made available through relevant search and web services.

Three New Crawlers Added

The latest update adds:

• PetalBot — A web crawler associated with Huawei's Petal Search ecosystem, used to discover and process web content for search-related services.
• Moz dotbot — The web crawler operated by Moz for crawling and analyzing websites as part of its SEO and link research services.
• MJ12Bot — The long-running crawler operated by Majestic, used to collect web data for its search engine optimization and link intelligence products. Majestic says MJ12Bot has been operating for more than 20 years and continues to be developed as part of its crawling infrastructure.

The addition of these crawlers expands the range of recognized search and web intelligence bots available through the IP2Location tool.

Simplifying Crawler Management

Managing crawler traffic can be challenging for website owners. Blocking legitimate search crawlers may prevent websites from being properly discovered, while blindly allowing automated traffic can create unnecessary server load or security concerns.

IP2Location's Search Crawler List provides a convenient way to obtain crawler IP addresses and generate firewall-compatible rules. Supported formats include Apache .htaccess, Linux iptables, .NET Web.config, and Cisco ACL, allowing administrators to integrate the lists into existing firewall configurations.

The list can also complement crawler detection using IP2Location's IP geolocation and usage-type data. The IP2Location database identifies Search Engine Spider (SES) traffic, allowing developers and website operators to distinguish search crawlers from other types of automated traffic.

Learn more and generate a Search Crawler whitelist:
https://www.ip2location.com/free/robot-whitelist/

About IP2Location

IP2Location is a provider of IP geolocation and IP intelligence solutions that help businesses identify the geographic location, network information, proxy usage, and other characteristics associated with IP addresses. Its products include IP geolocation databases, IP2Proxy databases, APIs, and developer tools for applications including cybersecurity, fraud prevention, analytics, advertising, and website personalization.