XMLSitemapCreator

XMLSitemapCreatorBot

XMLSitemapCreatorBot fetches pages when a person starts a crawl or uses “fetch by URL” on XMLSitemapCreator.com, usually to build or check a sitemap for their own site. It does not crawl on its own schedule and keeps nothing it fetches.

User agent

XMLSitemapCreatorBot/1.0 (+https://xmlsitemapcreator.com/bot)

Behaviour

  • Reads /robots.txt first and obeys rules for XMLSitemapCreatorBot or * (the user can turn this off only with a warning that it is for their own site).
  • At most 3 requests in flight per crawl, with at least 300 ms between requests, and at most 1,000 pages per crawl.
  • GET requests only; follows up to 5 redirects; 12 second timeout; reads at most 3 MB of HTML per page. Does not run JavaScript, submit forms or log in.
  • Only public http(s) URLs on default ports; private and internal addresses are refused.

Blocking it

User-agent: XMLSitemapCreatorBot
Disallow: /

Abuse reports: abuse@xmlsitemapcreator.com — include the time, your domain and the requested URLs.