Intermediate
AI Crawler Blocklist
Choose a crawler policy in AI Crawler Blocklist
Pick a crawler policy preset, review Monitor mode activity and switch to Block when you are ready.
Updated 1 min read By the D.C Style team
On this page
How Monitor and Block modes differ
- New installs use Monitor mode with training crawlers selected.
- Block mode returns a short HTTP 403 response.
Pick a preset
- Presets: training-only, all-category and empty.
- Training-only is the suggested starting point.
- Individual crawler selections are also available.
Understand the four controls
- Separate controls for training, AI search, user fetching and social previews.
- AI search, user fetching and social previews are unselected by default.
How crawlers are identified
- Dedicated IP feeds can identify a selected crawler even with a different user agent.
- Shared provider ranges need the role's user agent unless provider-wide blocking is enabled.
- Shared Anthropic ranges keep role separation unless provider-wide blocking is selected.
- Provider-wide and ASN rules can affect other services on the same infrastructure.
Review activity before blocking
- Check daily totals and recent request samples for matched provider or rule, outcome and evidence.
- Then enable blocking explicitly.