geo bots

Invocation#

echo-cli geo bots list
echo-cli geo bots declare --token <user-agent token> --kind <kind> [--ranges <cidr,cidr…> --ranges-source <url>] --by <name>
echo-cli geo bots remove <bot-id>
echo-cli geo bots refresh --bot <bot-id> --by <name>
echo-cli geo bots report --surface <slug> --since <iso-time> --depth <segments> [--kind <kind>]

Inputs and output#

list writes the declared crawlers by token: id, token, kind, ip_ranges, ranges_source, ranges_refreshed_at and ranges_refreshed_by (null until the first refresh), created_by and created_at. declare and remove write the crawler. refresh reads the crawler's ranges again from its ranges_source (POST /api/geo/bots/<id>/ranges), takes every CIDR range written on that page whatever shape it holds them in — a JSON list of prefixes or plain lines — and writes { bot, before, added, dropped }: the crawler with its new ranges and refresh time, the ranges it had, and those the page added and dropped. report writes host, since, last_delivery (received_at, entries, requests and hits of the drain's newest batch), sections (per crawler and section, most hits first: token, kind, section, hits, verified, unverified, unknown, outside_2xx and with_query) and coverage (per crawler: token, pages, the surface's published posts; crawled, how many of them it requested; uncrawled, the paths it did not).

State effect#

The hosting platform's log drain posts its batches to Echo Web's /api/drains/logs, as JSON or newline-delimited JSON, signed in x-vercel-signature as the hex HMAC-SHA1 of the body with ECHO_LOG_DRAIN_SECRET, the drain's signature secret (https://vercel.com/docs/drains/security). Every entry that records a request whose user agent carries a declared crawler's token (case-blind) is stored in crawl_hits once per log id, with its host, path without the query, whether it had one, status, client address, and verified: true when the address is in the crawler's declared ranges, false when ranges are declared and it is not, null when none are. Every batch is recorded in log_drain_deliveries. A section is the path's first --depth segments, / for 0; outside_2xx counts hits answered with a status outside 200–299, the crawl spent on what cannot be indexed; with_query counts hits whose address had a query. Every verb calls Echo Web's authenticated /api/geo/bots with ECHO_INTERNAL_BEARER at ECHO_WEB_BASE_URL; the Crawler logs section of the GEO screen and of Echo Desktop's GEO screen call the same operation, and echo-mcp reads crawlers and reports as echo_crawl_report. Removing a crawler removes its hits.

Refusals#

• A wrong invocation is refused before anything is sent: "geo bots takes <every verb's usage, separated by semicolons>; got <word>", "geo bots <verb> does not take <flag>; it takes …", "<flag> needs a value", "geo bots <verb> needs <flag>" and "geo bots remove needs a declared crawler id, as geo bots list prints it".
• Echo Web answers a bad request with "token is required: …", "kind is required: …", "ip_ranges holds <range>, which is not an address/prefix range such as the crawler's operator publishes", "ranges_source is required: the address the crawler's operator publishes its ranges at, so they can be checked again", "surface is required: …", "since is required: the ISO 8601 instant to count hits from; got <value>" or "depth is required: how many leading path segments name a section, 0 for the whole site; got <value>"; not found with "no log drain has delivered to Echo: connect the hosting platform's log drain to /api/drains/logs with ECHO_LOG_DRAIN_SECRET as its signature secret", "no blog surface is named <slug>" or "no declared crawler has id <id>; echo-cli geo bots list shows them"; and a conflict with "<token> is already declared as <id>; remove it first". The command prints Echo Web's sentence and exits 1.
• The drain endpoint refuses a batch with 403: "x-vercel-signature is required: the drain signs every batch with its secret" or "x-vercel-signature does not match the body signed with ECHO_LOG_DRAIN_SECRET; the drain's secret and Echo's differ"; and a body that is neither form with "the batch is neither a JSON array nor newline-delimited JSON: <cause>" as a bad request. Without ECHO_LOG_DRAIN_SECRET no batch is read.
• refresh is refused before the page is read with "refreshed_by is required: …" for an internal caller that names no one, "<token> declares no ranges_source, so there is no address to read its ranges from; declare it again with its ranges and their address", "<token>'s ranges_source <value> is not an address, so its ranges cannot be read again" or "<token>'s ranges_source <url> is not an HTTPS address, so what it answers cannot be trusted as the crawler's operator's" as a bad request; a page that answers outside 2xx fails with "<url> answered HTTP <status> when <token>'s ranges were read: <body>", one that cannot be reached with "crawler-ranges read published ranges could not be reached: <cause>" (both written to the integration failure log, echo-cli diagnostics failures), and a page with no range is refused with "<url> holds no CIDR range, so <token> keeps the <n> it had". A refused refresh leaves the crawler's ranges as they were.