reveal engine · verbatim markdown · ai optional
Extract everything a page knows.
Even what it hides.
crawlDNA opens every page in a real browser and clicks the tabs, accordions and “load more” buttons that static crawlers never touch — then hands you clean, word-for-word Markdown. Turn on AI and it follows your goal and keeps only what matters.
the difference
Most of a page is hidden behind a click
Tabs, accordions, “load more”, wizards, cookie walls. A static fetch stops at the first paint. crawlDNA drives the page like a human until nothing new appears — then measures what’s still hidden, so completeness is a number, not a hope.
Every state is captured whole and de-duplicated — the npm tab and the yarn tab both land in the output, byte-identical repeats don’t. When the loop ends, an audit reports how much text, if any, is still out of reach.
pipeline
One page, end to end
The same deterministic path every time. AI, when on, only decides which controls to click and what to keep — the data is always read straight from the DOM.
architecture
Crawl once. Reshape as often as you like.
Crawl
Visits the site page by page, opens the hidden content, extracts verbatim Markdown and
saves each page the moment it’s captured. A crash loses nothing — resume
picks up where it stopped; --incremental re-crawls only what changed.
Reshape
A chat over the saved files — tables, summaries, filtered subsets, other formats. Originals stay untouched, and every value produced (numbers, URLs, code) is checked against the sources; what doesn’t match is flagged, not served.
one dial
How much AI?
Behaviour is never inferred from your task text — it’s an explicit choice. Three settings, one question: how much should the model decide?
None
Zero calls to any model
- Reveal engine still clicks everything
- Keeps every page, whole
- Guaranteed zero tokens
Scope
AI on · take the whole site
- Every reachable page, pages whole
- AI arbitrates only ambiguous clicks
- Measured payload overrides a wrong “no”
Goal
AI on · only what the task asks
- Follows links that serve the task
- Keeps relevant pages and sections
- Optional embeddings rank by meaning
non-negotiable
The same promises, in every mode
Verbatim, always
The extracted content is the page’s own words. No model summarizes or rewrites it — in any mode.
Completeness measured
Every page ends with an audit of how much text stayed hidden. If something’s missing, you know — you don’t find out later.
Anti-bot declared
A block (CAPTCHA, interstitial) is recognized, retried once on the site’s terms, then skipped and reported. Never bypassed, never faked as content.
No per-site rules
No special config for individual sites — only universal signals: structure, geometry, the measured mass of content. It works the same everywhere.
under the hood
Small, sharp, no build step
Node.js — pure ESM
Zero build, minimal deps. CLI, local Web UI and library share one API.
Playwright / Chromium
A real browser, with a pooled set of reused contexts so assets aren’t re-paid per page.
Turndown + heuristics
Deterministic HTML→Markdown. Heuristics beat neural models on main content (SIGIR 2023).
Ollama or OpenAI-compatible
One provider layer for chat and embeddings — local model or cloud, same interface.
64-bit SimHash
Content fingerprints collapse mirrors and variants — only alongside URL-shape proof, never on similarity alone.
JSONL journal · resume · incremental
Every page hits disk the moment it’s captured. Crash or Ctrl-C: resume finishes the run. --incremental re-crawls only what changed since last time.
Point it at a site. Get clean Markdown.
Open-source, CLI- and library-first, with an optional local Web UI. Start with zero tokens.