SiteForge Docs

siteforge is tracked from the Tknott95 crates.io catalog because it is above the 64-download threshold. Current tracked version is 0.1.19, with 317 downloads across 20 versions.

crates.io: https://crates.io/crates/siteforge

Docs index: https://docs.rs/crate/siteforge/latest

API docs: https://docs.rs/siteforge/latest/siteforge/

Cargo source: https://docs.rs/crate/siteforge/latest/source/

Keep public site links pointed to crates.io/docs.rs while backend-published formal docs are prepared.

crates.io categories: command-line-utilities, web-programming.

crates.io keywords: ai, archive, crawler, markdown, tui.

docs.rs package facts: latest tracked release is 0.1.19; crates.io source package is about 108.3 kB.

docs.rs dependencies include aisling, anyhow, async-trait, blake3, chrono, clap, crossterm, directories, flate2, futures-util, globset, html-escape, imagesize, regex, reqwest, robots_txt, rusqlite, scraper, scrin, serde, serde_json, serde_yaml, shell-words, tar, thiserror, tokio, tracing, url, uuid, walkdir, and zip.

siteforge is a Rust CLI/TUI tool for archiving websites into local, AI-readable knowledge archives.

It stores raw snapshots, structured metadata, cleaned Markdown, JSON, JSONL semantic chunks, searchable SQLite metadata, token-budgeted AI context packs, AGENTS.md, and agent-index.json entry points.

The TUI is built with Scrin, with terminal loaders and status effects backed by Aisling through Scrin LoaderPlayer and direct LoaderProgress data.

Archive examples:

siteforge archive https://example.com/article
siteforge archive --input urls.txt -j 8
siteforge archive https://example.com/docs --full-site --max-depth 3 --max-pages 200 -j 4.

Authenticated/JS examples:

siteforge archive https://docs.example.test/private --header "Authorization: Bearer <token>" --cookie "session=..."
siteforge archive https://app.example.test/docs --render-js --browser-command chromium --browser-profile ~/.config/chromium-siteforge.

Operations:

siteforge tui
siteforge list
siteforge inspect <archive_id>
siteforge diagnostics --archive-id <archive_id>
siteforge search "query"
siteforge export <archive_id> --format agent-tar
siteforge pack <archive_id> --for-ai --max-tokens 120000
siteforge verify <archive_id>
siteforge resume <archive_id>
siteforge config.

• Polite async HTTP fetching with retries, redirects, compression, timeouts, per-origin pacing, delay, robots crawl-delay, and robots request-rate checks.
• URL normalization, same-domain/path-prefix scope, include/exclude glob filters, persistent frontier, sitemap discovery, gzip XML sitemaps, optional JS-rendered DOM capture, and resume support.
• Raw HTML/assets with BLAKE3 hashes, checksum manifests, total archive size caps, HTML extraction, readable Markdown/basic Markdown/JSON/text, semantic chunks, AI packs, and SQLite FTS5 search with LIKE fallback.
• Agent handoff export packages AGENTS.md, agent-index.json, readable files, chunks, packs, raw pages, rendered DOM captures, and raw assets while skipping transient database/export side files.

TUI uses Scrin Terminal, EventRouter, layouts/widgets, Markdown rendering, command palette, overlays/popups/toasts, help/details popups, quit modal, status toasts, archive browser, page preview, asset/OCR status, search, config, and bounded page/asset/job/log windows.

Limitations: crawling is conservative by default; JS content needs --render-js and a browser/profile; WARC export is WARC-like JSONL metadata, not a full WARC writer; OCR is optional/local; page/asset bodies stream up to configured byte caps; Ctrl+C checkpoints active frontier rows so crawls can resume quickly.

Security note: only pass --header or --cookie for content you are authorized to archive. SiteForge does not bypass authentication, paywalls, CAPTCHAs, DRM, or access controls.