Overview#

Spis has three related lanes: interface evidence, README research, and documentation full text.

text

catalog/record scaffold
        |
        +--> spis crawl start --> Stado jobs --> Appium / Cua / Weles / PTY / HTTP
        |         |
        |         +--> spis crawl status/resume
        |         +--> canonical artifact download and idempotent import
        |
        +--> verify-reference-evidence --apply
        +--> generate-example-catalogs
        +--> generated catalog-stats.json --> public docs

sync-readme-examples --> analyze-readme-examples --> guidelines
crawl-docs --> docs-corpus {status,search,show}

1. Select and scaffold#

catalog-type creates or edits a *-examples family. reference-record add introduces one selected product with source, category, rationale, and a measured overview image. New records are partial and list what has not been observed.

2. Acquire authentic evidence#

The surface chooses the crawler:

crawl-mobile: installed iOS/Android app through Appium.
crawl-desktop: installed native app through Cua Driver.
crawl-web: browser product through Weles.
crawl-tui: installed terminal application inside a real PTY.
crawl-cli: installed command-line application inside a real PTY.
crawl-docs: documentation inventory and full text.
sync-readme-examples: exact upstream README blobs.

spis crawl start preflights the complete selected batch before it submits the first job, records the exact argv and source revision, and selects hosts from the Stado registry. status refreshes by Stado job ID and automatically imports terminal successful artifacts; resume retains the original arguments. No crawler opens a browser, simulator, TUI, CLI, or desktop application on the coordinator workstation. TUI and CLI crawlers use isolated homes and fixtures. Login values are injected from Skarbiec through Stado and referenced in artifacts only by fixture name.

crawl-web covers web apps, dashboards, onboarding and authentication, app-store listings, design systems, reports, pricing pages, and landing pages. Each catalog has its own mandatory coverage contract while Weles remains the shared browser execution boundary. The worker waits for every Weles action and retains sanitized results, receipts, and artifact pointers instead of treating queue acceptance as a completed crawl.

Owner-published material remains valid third-party evidence where the contract calls for it. collect-example-images acquires qualifying interface imagery; capture-widths and audit-reference-accessibility submit Weles work through Stado rather than direct SSH.

Acquisition method and provenance class must agree. Retain bytes under the record, not merely a remote URL.

Marketing-family selection is strict. Pricing accepts only attributable pricing records whose exact URL identifies a pricing or plans surface and whose Weles evidence shows a visible price/plan comparison. Landing accepts only attributable landing records for the exact destination URL. Empty families and category/URL mismatches refuse the whole batch before any submit.

3. Analyze — spis analyze-example-structures#

analyze-example-structures [catalog ...] deterministically writes image layout regions, separators, density, and confidence into sources.json. It writes structure-analysis-failures.json while unresolved entries remain.

4. Accessibility — spis audit-reference-accessibility#

audit-reference-accessibility plans, submits through Stado, reads terminal status, retrieves, verifies and merges Weles axe results. Operational plans, staging files and the execution index remain in the active corpus's ignored .build/accessibility-audit/ directory. --dry-run validates and writes only a plan; it is not a completed audit.

5. Verify — spis verify-reference-evidence#

verify-reference-evidence re-probes retained media, hashes, dimensions, durations, provenance, state matches, journey/interactions, motion analysis, and accessibility. Without --apply it is a report only. With --apply it rewrites records and matching reference-index statuses to what the bytes prove.

6. Regenerate catalogs — spis generate-example-catalogs#

generate-example-catalogs --check validates without writing. Without --check, the same validation precedes rendering every catalog README.md plus example-catalogs.json and example-catalogs.md. A partial record is valid only when its gaps are explicit and consistent.

7. Monitor drift#

check-upstream-drift always verifies local media. Unless --skip-network is used, it also checks README blob SHAs and recorded URLs, distinguishing reachable, gone, guarded and unresolved sources. --strict exits 1 if drift exists. --write-report retains operational observations in the active corpus's ignored .build/upstream-drift.json, not a publishable corpus-root file.

README research lane#

spis sync-readme-examples --host <host> refreshes the fifty verbatim upstream README snapshots and their blob SHAs through Stado.
spis analyze-readme-examples regenerates the structural analysis (readme-examples/analysis.json) behind the README guidance.
spis guidelines <catalog> drafts counted observations for human review; interpretation remains outside Spis.

Documentation full-text lane#

crawl-docs resolves sitemap/override inventories, obeys robots rules, and writes resumable local archives. docs-corpus exposes read-only JSON status, search, and exact-page lookup.