seo crawl

Invocation#

echo-cli seo crawl run --origin <https origin> --pages <n> | list | links --crawl <crawl-id> | redirects --crawl <crawl-id> --format <vercel|netlify|nginx> | orphans --crawl <crawl-id> [--app <web source> --days <n>]

Inputs and output#

run writes the stored crawl: id, origin, page_limit, started_at, finished_at and pages, how many it fetched. list writes { crawls }, newest first. links writes crawl, pages, broken (every crawled page that ended in an error or a status of 400 or above, with url, status, error and linked_from, the crawled pages linking to it), redirects (every page reached through redirects, with its hops, each address and status, final_url, status and error) and redirect_map (each redirecting address mapped to the final address that answered 2xx, to point links and redirects at in one step). orphans writes crawl, pages, sitemap (every page the sitemaps list that no crawled page links to, by its address or the one it redirects to) and visited: with --app and --days, the paths the source's web events were seen on in those days that no crawled page links to (path, events), most visited first; null without them.

State effect#

The commands call Echo Web's authenticated /api/seo/crawls with ECHO_INTERNAL_BEARER at ECHO_WEB_BASE_URL; the Site crawls section of the GEO screen and of Echo Desktop's GEO screen call the same operations, and echo-mcp runs them as echo_site_crawl, echo_site_crawls, echo_site_crawl_links and echo_site_crawl_orphans. A crawl reads the site's robots.txt sitemaps as seo deploy-check does, then fetches every sitemap page and every same-origin page the crawled pages link to (<a href>, without the fragment), until --pages pages are fetched. Redirects are followed hop by hop and recorded; a chain stops at a loop or once it is as long as --pages. An HTML page keeps its title, canonical, links, hreflang alternates and images. The crawl is stored in seo_crawls and its pages in seo_crawl_pages; the reports read the stored pages and fetch nothing.

Refusals#

• A wrong invocation is refused before anything is sent: "seo crawl takes <usages>; got <verb>", "seo crawl <verb> does not take <flag>; it takes <flags>", "<flag> needs a value" and "seo crawl <verb> needs <flag>".
• Echo Web answers a bad request with "pages must be a positive whole number: how many pages the crawl fetches at most; got <value>", "origin is required: the https origin of the site to check, such as https://example.com; got <value>", "app_id and days go together: the web source whose visited paths to check and how many days back" or "days must be a positive whole number: how many days of visits to check; got <value>"; not found with "<origin>/robots.txt declares no sitemap and none was named; name one with sitemap=<https url>" or "no crawl has id <id>; echo-cli seo crawl list lists them"; and fails naming the cause with "the crawl of <origin> could not be stored: <cause>" or "the <n> pages of crawl <id> could not be stored: <cause>". The command prints Echo Web's sentence and exits 1.
• redirects writes crawl_id, format, rules (how many) and file: the crawl's redirect map as the file the host reads — vercel, the redirects list of vercel.json ({ source, destination, permanent: true }); netlify, _redirects lines (<source> <destination> 301!); nginx, location blocks returning 301 — each source the path and query of the redirecting address, each destination its path when on the crawl's origin and its whole address otherwise. The Site crawls section of the GEO screen offers the file to download and Echo Desktop's to save, and echo-mcp writes it as echo_site_crawl_redirects. A format Echo does not write is refused with "format must be one of vercel, netlify, nginx: the host whose redirect file to write; got <value>".