← Lab/Entry 04Still runningDraft copy

Data Monster

A company-intelligence engine on one laptop, where every fact has to show its evidence.

Notebook page

The original question: Can I build my own company dataset, at useful scale, where nothing is recorded unless the source actually says it?
Hypothesis: Polite crawling plus small local models is enough, as long as every extracted fact is checked against the page it came from.
What was built or tested: A crawler that snapshots company websites, a resumable job queue, and an extraction step where a local model proposes facts and a separate validator decides whether the evidence supports them.
Technical approach: Everything runs locally with open models and no paid data providers. Respect robots.txt, stop when a site says no, and keep unknown things unknown.
Outcomes and discoveries: The pipeline works and has processed thousands of company sites. Getting a model to say only what the page supports was harder than the crawling.

Covers: Web extraction, source quality, enrichment, structured datasets, AI-assisted research
Feeds: Prospect This and GTM for One
Runs on: One laptop, local models, no paid data providers
Status: Crawling works. Extraction is paused on a model issue.

The idea

Two of my products, Prospect This and GTM for One, depend on knowing things about companies. Buying that data is expensive, and you cannot see where it came from.

Data Monster is my attempt to build it myself, with one rule written at the top of the plan: unknown information stays unknown.

What exists

A polite crawler. It reads a handful of pages per company, obeys robots.txt, spaces out its requests, and stores a snapshot of every page so nothing has to be fetched twice.
A job queue that survives. Runs can pause, crash and resume without losing work.
Discovery. Seed lists of software companies built from public product directories.
Extraction with a second opinion. A local model proposes facts. A separate, model-independent check then asks whether the quoted evidence really supports each one. No support, no fact.

Not built yet: an API, search, and a proper interface.

What I have found so far

Quoting evidence is not the same as being right. A model happily cited "located in 15 countries" as proof of a headquarters city. The validator had to get stricter about what counts as support.
My own benchmark flattered me. A large share of company profiles contained nothing but the company's name, and the benchmark counted a name as a fact. The fix was in the prompt, but I only found it by reading the output.
Small models vary wildly. Similar-sized local models ranged from useful to unusable on the same task.
Your foundation can move under you. A local model update changed its behaviour enough to break the extraction format. That is why extraction is paused right now.
Boring things stop long runs. One overnight run froze because the laptop went to sleep.
When a site says no, stop. One directory began refusing requests partway through. The crawler stopped, and I did not try to get around it.

Where it stands

Crawling and storage work and are in use. Extraction is paused until the model is re-tested or the format is adjusted. This is an active experiment, not a product.

Content OS →