What a good sitemap contains — and what it leaves out
A sitemap is a list of the URLs you want search engines to index. The useful version lists only those: pages that return 200, are not marked noindex, and are their own canonical. Redirects, 404s, parameter duplicates and printer-friendly copies waste crawl attention and make Search Console report “Submitted URL not selected as canonical”.
The crawler here records why each URL was left out — redirect, HTTP status, robots.txt rule, noindex in the page or the X-Robots-Tag header, or a canonical pointing elsewhere — so you can review the decisions instead of trusting a black box. You can still include any URL by ticking it.
Honest lastmod, no padding
Google uses <lastmod> only when it is consistently accurate, and ignores <changefreq> and <priority> entirely. So by default this creator writes lastmod only when a real date is known — the page’s Last-Modified header, your CSV column, or the file’s modification time — and leaves the other two out. Switch them on if Bing or a legacy CMS expects them.
Stamping every URL with today’s date is the most common sitemap mistake: it tells search engines your dates mean nothing. The validator warns when every URL shares one date.
Four ways in
Crawl follows links from your home page, respects robots.txt, and can seed from your existing sitemap so orphan pages are not lost (up to 1,000 pages per run). Paste / upload accepts plain lists, CSV exports from Search Console or Screaming Frog, existing sitemaps (.xml or .xml.gz) and raw HTML. Static site folder turns an out/, dist/ or _site/ folder into URLs without uploading anything. The console crawler runs in your own browser for staging sites, sites behind a login, or JavaScript-rendered pages.
After that, everything — cleaning, filtering, editing, splitting, gzip and ZIP — happens in your browser. Sites over 50,000 URLs are split into numbered files with a sitemap_index.xml.
Questions
Is the XML sitemap creator free?
Yes. No account, no URL cap on lists and files (the protocol limit of 50,000 URLs per file is handled by splitting), and up to 1,000 pages per crawl run.
Why are some of my pages excluded?
Each excluded row shows the reason: it redirected, returned an error status, is blocked by robots.txt, is marked noindex, or declares a different canonical URL. Those pages should not be in a sitemap. Tick a row to include it anyway.
My site is built with React/Vue and the crawl finds only a few pages. What now?
Our crawler reads the HTML your server sends and does not run JavaScript. Use the console crawler tab: it runs in your own browser on your site, or upload your build folder, or paste a route list.
Do I still need to submit the sitemap to Google?
Upload sitemap.xml to your site root, add a “Sitemap:” line to robots.txt, and submit it once in Google Search Console and Bing Webmaster Tools. Google retired its sitemap ping endpoint in 2023, so there is nothing to “ping”.
Should I set changefreq and priority?
Not for Google — it ignores both. They are off by default and available for Bing, Yandex or tools that expect them. Accurate lastmod values matter far more.