Invocation#

bash

spis crawl-docs --host <target> (--site <NN-slug> | --all) [--exclude <slug>]... [--workers <n>] [--refresh]

Submit a pinned Stado job that crawls the checked-in documentation inventories into Spis's resumable full-text corpus.

Inputs and options#

--host <target> is required in coordinator mode.
Choose exactly the intended inventory scope with --site <NN-slug> or --all; --exclude <slug> affects the --all selection.
--workers <n> defaults to 64. Pages are requested without a pause between them; a refusal from the origin is reported as that page's diagnostic.
--worker and --artifact-uri are worker-facing options; public callers normally do not pass them.

Output and state effects#

The worker appends fetched pages to $HOME/.spis/docs-corpus/<slug>/pages.jsonl.gz, atomically updates per-site state.json, writes documentation-site-examples/content-structure/crawl-run.json, prints a JSON site summary, and may archive the corpus to the Spis Stado namespace. A page's text is its <main> element when the page marks one, otherwise the whole page; scripts, styles, templates, no-script fallbacks and SVG are left out because a browser renders them as no text. A page with no text is recorded as no_text; any other page as ok. --refresh is accepted but currently has no behavioral effect.

Refusals#

These messages are emitted by the current Rust implementation:

--host is required; documentation crawls execute as pinned Stado jobs
pass --site <NN-slug> or --all
An unknown inventory is refused as unknown site: SLUG; an unknown worker option is unknown argument: VALUE.
--host cannot be used with --worker
Publishing outside the owned namespace is refused as documentation artifact URI is outside the Spis crawl namespace.