About XMLSitemapCreator
XMLSitemapCreator makes protocol-correct XML sitemaps from a crawl, a URL list, an exported file or a static-site folder, and tells the truth about what search engines do with them: only indexable URLs, real lastmod dates, and no changefreq/priority padding unless you ask.
What runs where
- In your browser: list import, folder reading, normalisation, filtering, the URL table, XML generation, splitting, validation, gzip and ZIP. The sitemap engine (parser, builder, validator, robots.txt matcher, HTML link extractor) was written for this site and has no third-party XML dependency.
- On our edge function: crawl mode and “fetch by URL” call
/api/fetch, which fetches one public URL and returns its status, headers and extracted links (or the text of a robots.txt / sitemap). See the crawler page.
Limits
- Crawl: up to 1,000 pages and depth 10 per run, 3 parallel requests with at least 300 ms between dispatches, 12 s timeout and 3 MB per page. HTML is read as served; JavaScript is not executed.
- Sitemaps: 50,000 URLs and 50 MB per file (protocol limits), split automatically; up to 1,000 images per URL and 1,000 news articles per file.
- Validator: up to 2,000 findings are listed; files up to the 50 MB limit.
Feature status
IDs refer to the product requirements.
| ID | Capability | Status | Notes |
|---|---|---|---|
| XSC-001 | Crawl a public site (R1) | Done | Same-origin BFS via /api/fetch; ≤1,000 pages, depth ≤10, 3 concurrent with ≥300 ms spacing; pause/resume/stop keeps results. |
| XSC-002 | robots.txt respect (R1) | Done | Fetched first; longest-match + wildcard rules; blocked URLs listed with the matching rule. Can be switched off with a warning. |
| XSC-003 | Indexability filtering (R1) | Done | Non-200, redirects, noindex (meta and X-Robots-Tag), non-self canonical and non-HTML excluded with reasons; rel=nofollow links not followed. |
| XSC-004 | Paste / upload URL lists (R1) | Done | TXT, CSV/TSV (URL column + lastmod/changefreq/priority/title auto-detected), XML sitemap or index (.gz), HTML with base URL, fetch existing sitemap by URL. |
| XSC-005 | Local folder (static build) (R1) | Done | webkitdirectory picker; index.html → directory URL; optional .html stripping; mtime → lastmod; noindex/canonical read locally. |
| XSC-006 | Console crawler (R1) | Done | Copyable same-origin script; copies the URL list to the clipboard. |
| XSC-007 | Normalisation (R1) | Done | https, www, trailing slash, fragments, tracking params, query strip/sort, default ports; duplicates merged and counted. |
| XSC-008 | Filters (R1) | Done | Include/exclude by substring, * wildcard or /regex/; same host; pages only. |
| XSC-009 | URL table (R1) | Done | Include toggles, search, bulk include/exclude/lastmod, per-row lastmod editing, path tree view, 100 rows per page. |
| XSC-010 | Honest optional fields (R1) | Done | lastmod only when known by default; changefreq/priority off by default; priority-by-depth option. |
| XSC-011 | Protocol-correct XML (R1) | Done | Escaping, namespaces only when used; every output file is re-validated before download. |
| XSC-012 | Auto split + index (R1) | Done | By URL count and byte size; sitemap-N.xml + sitemap_index.xml with absolute locations and newest child lastmod. |
| XSC-013 | Export (R1) | Done | Copy, .xml, .xml.gz (CompressionStream), ZIP of all files (plain or gzipped), robots.txt line, submission steps. |
| XSC-014 | Validator (R1) | Done | Paste/upload/fetch; urlset and index; line-numbered errors/warnings/info; stats. |
| XSC-015 | Image sitemap (R1) | Done | Table builder + crawl image collection; ≤1,000 per URL. |
| XSC-016 | Video sitemap (R1) | Done | Required fields enforced per row; duration formats; dates, rating, flags, tags. |
| XSC-017 | News sitemap (R1) | Done | Defaults + per-row overrides; 1,000 per file; two-day freshness warning in validator. |
| XSC-018 | Hreflang sitemap (R1) | Done | Grid input; full reciprocal alternates incl. self and x-default; code validation. |
| XSC-019 | Sitemap index generator (R1) | Done | Dates normalised; duplicates and cross-host flagged. |
| XSC-020 | Split / merge (R1) | Done | Multi-file input, dedupe with newest lastmod, re-split, extensions preserved. |
| XSC-021 | URL extractor (R1) | Done | TXT, CSV, JSON; index children fetched (first 50). |
| XSC-022 | robots.txt generator + tester (R1) | Done | Presets incl. AI crawlers; tester shows the deciding rule and line. |
| XSC-023 | HTML sitemap generator (R1) | Done | Full page or nav fragment; titles or path-derived labels. |
| XSC-024 | Offline/PWA (R1) | Partial | Service worker caches pages and assets; local tools work offline after a visit. Crawl and fetch-by-URL need the network. |
| XSC-025 | Diff two sitemaps (R2) | Not done | Roadmap. |
| XSC-026 | Broken-link status sweep (R2) | Not done | Roadmap. The crawler already reports status for crawled URLs. |
| XSC-027 | IndexNow helper (R2) | Not done | Roadmap. |
Example data
All example URLs use example.com / example domains and are synthetic.