Our crawler
Our crawler identifies itself as:
councilsitecheck-bot/1.0 (+https://councilsitecheck.co.uk/bot/)
What it does
- Reads public pages of parish and town council websites in England to see which required documents are linked: up to 20 pages for a check someone runs on this site, and up to 25 for our scheduled checks.
- Makes at most one request per second to a site, and honours a longer
Crawl-delayin robots.txt. - Reads HTML pages only. It doesn’t download PDFs or other documents; it judges them by their links, titles and file names.
- Never logs in, fills in forms or follows search, calendar or login links.
How often
Whenever someone runs a free check on this site; for scheduled checks, roughly once a month per council website at most, and monthly for councils that pay for monitoring.
How to block it
It follows robots.txt. To block only our crawler, add:
User-agent: councilsitecheck-bot Disallow: /
Or ask us to stop checking your website on the opt-out page; that also stops our emails. Problems? Email [email protected].