Methodology

AI Bot Policy Auditor: workflow triage, not proof.

Fetch and audit robots.txt, llms.txt, and agents.json coverage for explicit AI crawler and agent-discovery policy.

Open free tool Trust model

What this catches

  • Missing robots.txt
  • Missing llms.txt
  • Missing agents.json
  • Unclear named AI bot policy
  • Crawler/discovery contradictions

What this misses

  • Crawler compliance
  • Private CDN rules
  • Guaranteed indexing

Method

Backend authenticated audit fetches public site files, applies quota/billing, and returns structured findings. Raw fetched files are not stored.

Bot policy audits read visible public files only and cannot verify crawler compliance.