Council Site Check

Our crawler

Our crawler identifies itself as:

councilsitecheck-bot/1.0 (+https://councilsitecheck.co.uk/bot/)

What it does

  • Reads public pages of parish and town council websites in England to see which required documents are linked: up to 20 pages for a check someone runs on this site, and up to 25 for our scheduled checks.
  • Makes at most one request per second to a site, and honours a longer Crawl-delay in robots.txt.
  • Reads HTML pages only. It doesn’t download PDFs or other documents; it judges them by their links, titles and file names.
  • Never logs in, fills in forms or follows search, calendar or login links.

How often

Whenever someone runs a free check on this site; for scheduled checks, roughly once a month per council website at most, and monthly for councils that pay for monitoring.

How to block it

It follows robots.txt. To block only our crawler, add:

User-agent: councilsitecheck-bot
Disallow: /

Or ask us to stop checking your website on the opt-out page; that also stops our emails. Problems? Email [email protected].