CrawlwiseBot

Somebody asked Crawlwise for an SEO audit of your site. In almost every case that is you or a colleague. The crawler reads your pages the way a search engine does, so the audit can report on titles, headings, meta descriptions, structured data and content depth.

We do not collect email addresses, we do not resell what we crawl, and we do not store the text of your pages — only the measurements taken from them.

• It reads robots.txt before requesting any page, and obeys it.
• It makes at most one request per second to a single site, and never more than a few dozen pages in one audit.
• It honours Crawl-delay when you set one.
• If you answer with 429 or 503, it waits as long as your Retry-After header asks, then stops.
• It requests HTML only, and never submits forms or logs in.

Add a delay in your robots.txt. We take the slower of your value and our own.

User-agent: CrawlwiseBot
Crawl-delay: 10

Two lines, and we will stop visiting. No account or request needed.

User-agent: CrawlwiseBot
Disallow: /

To keep us out of one area only, list the paths instead of /. We apply the longest matching rule, so an Allow line can carve an exception out of a broader Disallow.

If CrawlwiseBot hit your site harder than described above, or kept visiting after you blocked it, tell us and we will stop it and fix the cause. Include your domain and roughly when it happened — we keep a record of every URL we request, so we can tell you exactly what we did.

bot@crawlwise.app