AI Crawler Blocklist

Crawler policies with safe starting choices

• Monitor mode by default; explicitly enable blocking after reviewing activity.
• Training-only, all-category and empty presets, plus individual crawler selections.
• Separate training, AI search, user fetching and social-preview controls.
• Dedicated IP feeds for supported crawlers and clearly identified user-agent fallback.
• Shared Anthropic ranges preserve role separation unless provider-wide blocking is explicitly selected.

Maintained sources and search protection

• Background refreshes for GPTBot, Anthropic, Google, Bing, OpenAI Search and ChatGPT-User source feeds.
• Google/Bing protection through fresh official crawler ranges and cached forward-confirmed reverse DNS.
• No exemption based solely on a Googlebot or Bingbot user-agent claim.
• Daily refresh scheduling with jitter, plus manual refresh requests.
• Administrator review of large source changes, with a scope preview bound to the exact update.
• Last validated ranges retained after errors; stale automatic rules monitor instead of blocking.

Manual rules and clear decisions

• Add-on-owned IP and CIDR allow/deny rules without modifying XF’s global IP bans.
• One-click exact-IP deny rules from request samples.
• Optional ASN deny rules with prefix and address-coverage review before saving.
• Verified search-crawler protection before deny rules; explicit allow rules take priority.
• Offline diagnostics showing the decision, reason, evidence and snapshot reference.
• Restore the previous policy snapshot or use Off and emergency bypass controls.

Activity and controlled data collection

• Atomic daily matching-request totals retained for 90 days.
• Recent samples retained for seven days, limited to 100 per provider/hour and 10,000 overall.
• Samples contain time, IP, bounded user agent, provider/rule, outcome, reason and evidence.
• No request URLs, query strings, cookies, bodies or credentials in samples.
• Administrator permission and CSRF protection for management actions.
• No Redis requirement; the default database path supports counting and sampling.

Before installation

The declared requirements are XenForo 2.2.0+, PHP 7.2+ and the PHP cURL extension. Your server needs outbound HTTPS access to the supported feed providers and dns.google for background DNS verification. XenForo scheduled jobs must be able to run regularly.

This presentation describes a local 1.0.0 Alpha 1 build. Use a verified release archive when one is available; a distributable archive has not been produced in this presentation task.

1. Install and grant administrator access

Install the verified archive through Add-ons → Install/upgrade from archive. For manual installation, upload the archive’s upload contents into the forum root and install DC/AICrawlerBlocklist from the Add-ons page.

Grant Manage AI crawler blocklist to the administrators who will manage protection. Open Setup → AI crawler blocklist.

2. Let the first sources refresh

The add-on starts in Monitor mode and queues its initial source refresh. Check source health on Overview or Maintenance. Keep Monitor active until both Google and Bing protection feeds are fresh and nonempty.

Arrange reliable XenForo job processing with your host. A typical command, run from the forum directory, is:

php cmd.php xf:run-jobs --max-execution-time=55

3. Choose a crawler policy

Use the training-only preset as a starting point. Keep AI search, user-requested fetching and social previews available unless you intend to restrict them. Provider-wide and ASN rules can include other services on the same infrastructure, so review their scope carefully.

4. Review activity and test decisions

Inspect matching requests and their evidence. Use Diagnostics with an example IP and user agent to understand the current policy. Add an allow rule for known legitimate traffic when needed. Network rules fetch or show their scope before you confirm them.

5. Enable enforcement

Select Block on Overview, review the confirmation, and continue. Matching requests receive a short HTTP 403 response. Automatic enforcement falls back to monitoring when the required source protection is unavailable or affected ranges become stale.

6. Maintain and recover

Review staged source changes under Maintenance. Keep cron and job processing functioning. Return to Monitor or Off if you need to inspect a policy, or restore the previous snapshot. To bypass protection immediately without ACP access, add this to src/config.php:

$config['dcAICrawlerBlocklist']['disabled'] = true;

Remove the line when normal protection should resume. Disabling the add-on preserves settings; uninstalling removes its owned configuration and activity data.