Bot policy
Our crawlers and our products' webhooks identify themselves with a User-Agent pointing to this page. The one test in our products that sends User-Agents that are not ours is described below, with why. This is what our systems are, what they do, how often they call, and how to make them stop.
Who we are
Desvela runs deterministic watchers: lightweight, low-frequency checks that detect changes and deliver signed webhook events. Our crawlers and products use four identities, plus a fifth for the watcher that keeps track of the standards we build on.
| User-Agent | What it does | How often |
|---|---|---|
DesvelaBot/0.1 |
Reads a handful of public files to index what a domain publishes for AI agents: the AI-bot directives in /robots.txt, /.well-known/ard.json, /.well-known/ai-catalog.json, /.well-known/agent-card.json, /.well-known/mcp.json, /llms.txt and /agents.md. For a domain that publishes one of them, it also asks for two paths that do not exist, to tell real files from servers that answer 200 to anything, and reads the MCP server cards its catalog declares. Nothing else is fetched, and no page content is stored. This is what powers the census, the free preflight(domain) tool and Registry Watch. |
Full pass monthly, plus a weekly re-check of domains that publish something. Around 4 requests per second across the whole internet, not per site: your server sees about seven requests and then nothing for weeks. |
DesvelaCiteCheck/0.1 |
Reads a page that an AI engine (ChatGPT, Perplexity, Claude or Gemini) cited or returned as a source when answering questions about an AI Brand Watch customer's brand, to tell that customer whether the page names them and whether the cited site still exists. Only that URL is fetched, plus any redirect it answers with inside the same site. No links are followed, and a redirect to a different site is noted, not followed. We keep the URL, whether the brand is named, the status code, the size and where a dead source redirects, never the page content. It is off unless the customer turns it on. | Only when a customer's run includes it, usually weekly. Per run: one request per cited page plus its same-site redirects, one robots.txt per origin (http and https, www and the bare domain count apart) plus its redirects, and never more than two cited pages at a time from the same site. |
DesvelaStandardsWatch/1.0 |
Tells us when something changes in the agent-web standards we build on. It reads a fixed, hand-picked list of about thirty public pages and feeds: spec repositories, standards drafts, AI crawler documentation and changelogs. Only those exact URLs are fetched and no links are followed; for the one directory on the list, it walks that directory's own pages, 1.5 seconds apart. It is a reader of a fixed list, not a crawler, so like a feed reader it does not read robots.txt. If your page is on the list and you want it off, email hello@desvela.ai. | Once a day: one request per URL, retried at most twice if it fails. |
Desvela-BrandWatch/0.1 |
Delivers AI Brand Watch webhooks to endpoints our customers registered themselves. It does not crawl. | Only when an event fires for that customer. |
Desvela-Registry/0.1 |
Delivers Registry Watch webhooks to endpoints the subscriber registered. It does not crawl: the files are read by DesvelaBot. | Only when a watched domain changes. |
All five send (+https://desvela.dev/bot) in the User-Agent so this page is one click away from your logs. DesvelaCiteCheck uses the usual crawler form, Mozilla/5.0 (compatible; DesvelaCiteCheck/0.1; +https://desvela.dev/bot). It runs inside AI Brand Watch on the Apify platform, so its requests come from Apify's servers, not from ours.
We also look up _index._agents.<your-domain> in DNS, part of the ARD discovery spec. That query goes to a public resolver, never to your server, so it costs you nothing and will not appear in your logs.
How we behave
- Low frequency by design. DesvelaBot reads a handful of files per domain per pass. DesvelaCiteCheck reads only pages an AI engine already pointed to, at most two at a time per site within a run. No bulk crawling, no rendering, no following links into your site.
- robots.txt respected by default. Our crawlers read it first, and a disallow stops the request before it happens. DesvelaCiteCheck also obeys the rules you wrote for DesvelaBot: they can keep it out, never let it into a path that
User-agent: *closes. It follows redirects of robots.txt itself. If your robots.txt answers with a server error or a 429, or its redirects lead nowhere, it does not fetch the page at all; any other 4xx (404, 403…) counts as having no robots.txt. - No disguises. Our crawlers never rotate identities or spoof browsers to defeat blocks. If you block us, we tell our customer instead of cheating, and we stop charging them for checks we cannot run. The crawler block test below sends other User-Agents to measure blocks, never to get past them.
- Webhook deliveries only go to the URL registered by whoever set up the run or the watch, are HMAC-signed, and refuse redirects. We do not verify that the URL belongs to them: if deliveries reach you that you did not ask for, answer them with an error status (Registry Watch switches a watch off after five failed passes) or email hello@desvela.ai.
- We publish what we ask others to publish. desvela.dev serves its own ai-catalog.json, llms.txt and agents.md. If that seems like the least a crawler of the agent-readable web could do, we agree.
The crawler block test
crawler_block_test(domain), on our registry as an MCP tool and as GET /crawler-block, answers a question robots.txt cannot: does the server of this domain actually let AI crawlers in? When someone asks about a domain, it requests that domain's homepage six times, one after another, each following up to three redirects within the same site: once with a regular browser User-Agent, as the control, and once each with the published User-Agents of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot. Then it compares each response's status and size with the browser's. These requests do not carry a pointer to this page, because they have to look like the crawlers they test. That is the only place our products send User-Agents that are not ours.
- It measures, it does not get past anything. It reads only the homepage, stores no content, and returns one verdict per crawler. If the site refuses the browser control too, or redirects it to another site, it reports no verdict at all.
- It does not read robots.txt, because what it measures is what your server answers regardless of what robots.txt says. To keep it off your domain, use the email opt-out below: a domain on that list gets none of these requests.
- For the census, we have also run it by hand over the top 1,000 domains, so we can publish how many block AI crawlers in practice.
How to opt out
- Add this to your robots.txt, honored on the next pass. It stops DesvelaCiteCheck too:
User-agent: DesvelaBot Disallow: /
- To stop only the cited-page checks and stay in the census:
User-agent: DesvelaCiteCheck Disallow: /
A group that names DesvelaCiteCheck replaces both the*rules and the DesvelaBot rules for it, so write there everything you want it to obey. That is also how you let it into a path your*group closes. - Or email hello@desvela.ai with your domain and we blocklist it manually within 48h, from the census and from the crawler block test. For DesvelaCiteCheck, robots.txt is the opt-out: it is read on every run, so it takes effect on the next one.
Opting out of DesvelaBot removes you from the index and from the census counts. If you block DesvelaCiteCheck in robots.txt or answer it with an error status, Brand Watch customers see your page reported as blocked or unreadable, not as a page that does not mention them. Neither stops webhook deliveries you asked for, which you cancel from your own watch.