Bot policy
If a request from us reached your server, it identified itself with a User-Agent pointing to this page. This is what our systems are, what they do, how often they call, and how to make them stop.
Who we are
Desvela runs deterministic watchers: lightweight, low-frequency checks that detect changes and deliver signed webhook events. Two identities call from our infrastructure.
| User-Agent | What it does | How often |
|---|---|---|
DesvelaBot/0.1 |
Reads four public files to index what a domain publishes for AI agents: /.well-known/ai-catalog.json, /llms.txt, /agents.md and the AI-bot directives in /robots.txt. Nothing else is fetched, and no page content is stored. This is what powers the census, the free preflight(domain) tool and Registry Watch. |
Full pass monthly, plus a weekly re-check of domains that publish something. Around 4 requests per second across the whole internet, not per site: your server sees four requests and then nothing for weeks. |
Desvela-BrandWatch/0.1 |
Delivers webhooks to endpoints our customers registered themselves. It does not crawl. | Only when an event fires for that customer. |
Both send (+https://desvela.dev/bot) in the User-Agent so this page is one click away from your logs.
We also look up _index._agents.<your-domain> in DNS, part of the ARD discovery spec. That query goes to a public resolver, never to your server, so it costs you nothing and will not appear in your logs.
How we behave
- Low frequency by design. Four files per domain per pass, no bulk crawling, no rendering, no following links into your site.
- robots.txt respected by default. We read it first, and a disallow stops the other three requests before they happen.
- No disguises. We never rotate identities or spoof browsers to defeat blocks. If you block us, we tell our customer instead of cheating, and we stop charging them for checks we cannot run.
- Webhook deliveries only go to URLs explicitly registered by the receiving customer, are HMAC-signed, and refuse redirects.
- We publish what we ask others to publish. desvela.dev serves its own ai-catalog.json, llms.txt and agents.md. If that seems like the least a crawler of the agent-readable web could do, we agree.
How to opt out
- Add this to your robots.txt, honored on the next pass:
User-agent: DesvelaBot Disallow: /
- Or email hello@desvela.ai with your domain and we blocklist it manually within 48h.
Opting out removes you from the index and from the census counts. It does not stop webhook deliveries you asked for, which you cancel from your own watch.