We crawl the agent-readable web. Here is exactly how, and how to stop us.
Documentation · Bot policy

Bot policy

Our crawlers and our products' webhooks identify themselves with a User-Agent pointing to this page. The one test in our products that sends User-Agents that are not ours is described below, with why. This is what our systems are, what they do, how often they call, and how to make them stop.

Who we areHow we behaveThe crawler block testHow to opt out

Who we are

Desvela runs deterministic watchers: lightweight, low-frequency checks that detect changes and deliver signed webhook events. Our crawlers and products use four identities, plus a fifth for the watcher that keeps track of the standards we build on.

User-AgentWhat it doesHow often
DesvelaBot/0.1 Reads a handful of public files to index what a domain publishes for AI agents: the AI-bot directives in /robots.txt, /.well-known/ard.json, /.well-known/ai-catalog.json, /.well-known/agent-card.json, /.well-known/mcp.json, /llms.txt and /agents.md. For a domain that publishes one of them, it also asks for two paths that do not exist, to tell real files from servers that answer 200 to anything, and reads the MCP server cards its catalog declares. Nothing else is fetched, and no page content is stored. This is what powers the census, the free preflight(domain) tool and Registry Watch. Full pass monthly, plus a weekly re-check of domains that publish something. Around 4 requests per second across the whole internet, not per site: your server sees about seven requests and then nothing for weeks.
DesvelaCiteCheck/0.1 Reads a page that an AI engine (ChatGPT, Perplexity, Claude or Gemini) cited or returned as a source when answering questions about an AI Brand Watch customer's brand, to tell that customer whether the page names them and whether the cited site still exists. Only that URL is fetched, plus any redirect it answers with inside the same site. No links are followed, and a redirect to a different site is noted, not followed. We keep the URL, whether the brand is named, the status code, the size and where a dead source redirects, never the page content. It is off unless the customer turns it on. Only when a customer's run includes it, usually weekly. Per run: one request per cited page plus its same-site redirects, one robots.txt per origin (http and https, www and the bare domain count apart) plus its redirects, and never more than two cited pages at a time from the same site.
DesvelaStandardsWatch/1.0 Tells us when something changes in the agent-web standards we build on. It reads a fixed, hand-picked list of about thirty public pages and feeds: spec repositories, standards drafts, AI crawler documentation and changelogs. Only those exact URLs are fetched and no links are followed; for the one directory on the list, it walks that directory's own pages, 1.5 seconds apart. It is a reader of a fixed list, not a crawler, so like a feed reader it does not read robots.txt. If your page is on the list and you want it off, email hello@desvela.ai. Once a day: one request per URL, retried at most twice if it fails.
Desvela-BrandWatch/0.1 Delivers AI Brand Watch webhooks to endpoints our customers registered themselves. It does not crawl. Only when an event fires for that customer.
Desvela-Registry/0.1 Delivers Registry Watch webhooks to endpoints the subscriber registered. It does not crawl: the files are read by DesvelaBot. Only when a watched domain changes.

All five send (+https://desvela.dev/bot) in the User-Agent so this page is one click away from your logs. DesvelaCiteCheck uses the usual crawler form, Mozilla/5.0 (compatible; DesvelaCiteCheck/0.1; +https://desvela.dev/bot). It runs inside AI Brand Watch on the Apify platform, so its requests come from Apify's servers, not from ours.

We also look up _index._agents.<your-domain> in DNS, part of the ARD discovery spec. That query goes to a public resolver, never to your server, so it costs you nothing and will not appear in your logs.

How we behave

The crawler block test

crawler_block_test(domain), on our registry as an MCP tool and as GET /crawler-block, answers a question robots.txt cannot: does the server of this domain actually let AI crawlers in? When someone asks about a domain, it requests that domain's homepage six times, one after another, each following up to three redirects within the same site: once with a regular browser User-Agent, as the control, and once each with the published User-Agents of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot. Then it compares each response's status and size with the browser's. These requests do not carry a pointer to this page, because they have to look like the crawlers they test. That is the only place our products send User-Agents that are not ours.

How to opt out

Opting out of DesvelaBot removes you from the index and from the census counts. If you block DesvelaCiteCheck in robots.txt or answer it with an error status, Brand Watch customers see your page reported as blocked or unreadable, not as a page that does not mention them. Neither stops webhook deliveries you asked for, which you cancel from your own watch.