seo sites and seo crawl diff

Invocation#

echo-cli seo sites add --origin <https origin> --pages <n> --every-days <n> --notify-user <user-id> --by <who> | list | remove --origin <https origin> | run; echo-cli seo crawl diff --crawl <crawl-id> --against <earlier crawl-id>

Inputs and output#

add and remove write the site: origin, pages, every_days, notify_user, declared_by, declared_at, attempted_at (when the schedule last tried it, null before) and error (what stopped that attempt's crawl or comparison, null when it ran); list writes { sites } by origin. run writes { runs }: per due site, origin, crawl_id, against_id (the previous crawl compared with, null for a first crawl), findings and alert_id (the pending action raised, null without findings), or origin and error; it exits 1 when any site failed. diff writes crawl, against, changed (path, field — title or description —, before, after and emptied, true when a value that was there is now empty or gone), lost (path, before_status, after_status, after_error: a page the earlier crawl answered 2xx that the later one did not, or did not reach) and duplicated_titles (title, paths: a title the later crawl shares across several paths that the earlier one did not).

State effect#

seo sites calls Echo Web's authenticated /api/seo/sites, diff /api/seo/crawls/<id>?report=diff, with ECHO_INTERNAL_BEARER at ECHO_WEB_BASE_URL; the Scheduled site audits section of the GEO screen and of Echo Desktop's GEO screen call the same operations (each site with its last attempt, a failed one in red with its error), and echo-mcp runs them as echo_seo_sites, echo_seo_site_add, echo_seo_site_remove, echo_seo_sites_run and echo_site_crawl_diff. Sites are kept in seo_sites. /api/cron/seo-crawls runs daily: every site whose newest crawl finished at least every_days days ago, or that has none, is crawled as seo crawl run does with its pages; the new crawl is compared with the one before it, and when the comparison finds anything, one pending action of type seo_regression is raised for notify_user, its metadata holding every finding and both crawl ids. Each attempt is kept on the site's row (attempted_at and error), so a failure the daily job met is read from seo sites list; a site that fails is tried again the next day and the others still run. Pages are compared by the path and query of their final address; a path counts only where the crawl answered it 2xx.

Refusals#

• A wrong invocation is refused before anything is sent: "seo sites takes <every verb's usage, separated by semicolons>; got <word>", "seo sites <verb> does not take <flag>; it takes …", "<flag> needs a value", "seo sites <verb> needs <flag>" and "seo crawl diff needs --against".
• Echo Web answers a bad request with "origin is required: the https origin of the site to audit; got <value>", "notify_user is required: the id of the user whose pending actions receive the alerts; got <value>", "declared_by is required: who declares the site", "pages must be a positive whole number: how many pages each crawl fetches at most; got <value>", "every_days must be a positive whole number: how many days pass between crawls; got <value>", "against is required: the earlier crawl of the same site to compare with, as echo-cli seo crawl list prints it" or "crawl <id> is of <origin> and crawl <id> of <origin>; compare two crawls of one site"; not found with "<origin> is not audited on a schedule; echo-cli seo sites list lists them" or "no crawl has id <id>; echo-cli seo crawl list lists them"; and fails naming the cause with "the site <origin> could not be stored: <cause>". A run's failed site carries the sentence that stopped it, such as "the SEO alert for <origin> could not be raised: <cause>". The command prints Echo Web's sentence and exits 1.