Open-Web Signal Extraction Is An Ingestion Problem

Call the pattern by its real name: an open-web signal radar is an ingestion system with a language model bolted to the far end. The pitch sounds like a model problem — point a model at the internet and have it report what matters — and the architecture that answers the pitch is shaped nothing like it. Almost everything that decides whether the answer is any good happens before the model sees a single token, in code that has nothing to do with intelligence.

The open web is not one source. It is a crowd of them, each with its own schema, its own authentication ritual, its own rate limits, and its own way of going quietly wrong. A demo runs against a handful of clean feeds; the thing a reader actually wants — what changed this week about the one topic that matters, without the same story arriving five times — runs against the mess.

GreatInsights is the radar ML LABS built on that premise, in the internal portfolio at Escape Velocity Labs, a sister practice under the same owner. It reads across RSS, Hacker News, Reddit, X, LinkedIn, arXiv, Google News, newsletters, and the open web, and returns the specific insight a person or an agent asked for. Its shape is the argument: nine source adapters, normalization at the edge, then a relevance-classification and dedup pipeline standing between all of that and the model.

One Adapter Per Source

There are nine adapters because there are nine sources. Not a shared client with per-source configuration — a discrete adapter for each, and each one owns its own authentication, its own backoff, and its own pagination. That looks like duplication on a whiteboard and it is the opposite in production: when a source tightens its rate limits or renames a field, the blast radius is one file, and nothing downstream ever learns that anything changed.

• Auth, backoff, and pagination live inside the adapter that needs them, so one source tightening its limits cannot throttle the other eight.
• Normalization happens at the edge — every adapter emits the same internal record shape, and no stage after it knows or cares whether an item arrived as a feed entry, an API payload, or a scraped page.
• A dead source degrades to a logged gap rather than a failed run, because graceful degradation under partial failure (Google, 2016) is the difference between partial coverage and a stalled radar.

graph TD
    A["Nine open-web sources"] --> B["One adapter each: auth, backoff, pagination"]
    B --> C["Normalization at the edge"]
    C --> D["Dedup"]
    D --> E["Relevance classification"]
    E --> F["Model: the last stage, not the system"]
    style A fill:#1a1a2e,stroke:#0f3460,color:#fff
    style B fill:#1a1a2e,stroke:#16c79a,color:#fff
    style C fill:#1a1a2e,stroke:#16c79a,color:#fff
    style D fill:#1a1a2e,stroke:#16c79a,color:#fff
    style E fill:#1a1a2e,stroke:#16c79a,color:#fff
    style F fill:#1a1a2e,stroke:#ffd700,color:#fff

The same instinct runs the per-vendor adapters in the multi-vendor ECG ingestion pipeline ML LABS built for HeartSciences, where each device manufacturer's dialect is quarantined behind its own parser and the platform behind them sees one clean record. Different industry, identical structural bet: isolate every external party's chaos at the boundary, and let the interior of the system be boring. The finding that glue code and data plumbing dominate real ML systems (NeurIPS, 2015) predates this generation of models by a decade and it still describes the bill of materials exactly.

Dedup Is The Product

An information radar that shows the same event nine times is worse than no radar, because it costs the reader attention and returns nothing for it. Deduplication is not a cleanup step bolted on at the end of the pipeline; it is a substantial part of what the reader is paying for, and it belongs before the model rather than inside the prompt. Exact matching on URL or headline catches almost nothing — the open web republishes, syndicates, rewrites, and reframes the same underlying event under different bylines within hours.

The merge threshold is where the real judgment sits, and it is asymmetric in a way that decides the design. Merge too little and duplicates leak through, which is annoying and visible. Merge too much and genuinely distinct items collapse into one, which is invisible — the reader never learns what was dropped, and the radar's failure mode becomes a confident, clean, incomplete answer. The conservative direction is not the timid choice; it is the only choice whose failures the reader can see.

The failure a reader forgives is a duplicate. The failure that loses them is the item that was quietly merged away.

Relevance Is A Pipeline Stage

Relevance is classified in its own stage, on the normalized record, before anything reaches the model that will write the answer. That separation is the whole point. A judgment that lives inside the generation prompt cannot be measured, cannot be cached, cannot be swapped, and cannot be improved without touching the thing that writes prose — so the pipeline pulls it out and makes it a component with inputs, outputs, and an error rate of its own.

Keeping the judgment upstream is also context economics. Retrieval-augmented generation puts retrieval before generation (Lewis et al., 2020) for the same structural reason a radar does: the model is the last stage, and its answer can only be as good as the candidate set handed to it. Widening that set does not help — long contexts bury information in the middle (Liu et al., 2023), so dumping the firehose into a large context window degrades the answer while raising the bill. Reaching for the simplest composition that works before adding model autonomy (Anthropic, 2024) is the same discipline stated from the agent side.

When The Firehose Needs An Owner

There is a boundary where this stops being a weekend integration and becomes a system with an owner. It arrives when three pressures collide: the number of sources, the volume per source, and the freshness the answer implicitly promises. Past that point the pipeline does not fail loudly — it degrades. Sources rate-limit, the cache ages, the relevance classifier drifts as the topic moves under it, and the output still renders, still reads as reasonable, and is quietly stale.

That is the hardest class of failure to catch from inside, because nothing alarms. The radar returns an answer, the answer is fluent, and the only evidence that the ingestion layer has gone half-blind is a number nobody is looking at: coverage per source, merge rate, the age of the freshest item behind the answer. Instrumentation is the difference between a system that tells you it is degrading and a system that lets you find out from a customer.

First Steps

1. Inventory your sources and wrap each behind its own adapter that normalizes to a single internal record shape, so a source changing its schema never ripples past its own boundary.
2. Make dedup a pipeline stage with a deliberately conservative merge threshold, and log every merge — an unlogged merge is a silent omission you cannot audit later.
3. Pull the relevance judgment out of the generation prompt and give it its own stage, so it can be measured, cached, and replaced without touching the model that writes the answer.

Instrument The Data Path First

Instrument the ingestion layer before touching the model at all. Coverage per source, merge rate, classification agreement, cache hit ratio, and the age of the freshest item behind each answer are the numbers that say where quality is actually leaking — and they point at the model far less than the pitch deck implies. Alerting on symptoms someone named in advance (Google, 2016) is what turns a degrading radar into a paged incident instead of a slow, unexamined decline. Build those metrics as first-class and the model becomes a swappable component rather than a load-bearing mystery.

The general form of this problem is the one every path from an AI pilot to production has to walk, and the specific form is upstream of it: the radar is only as trustworthy as the data path feeding it, which is the same reason a build-ready data foundation is where AI budgets are won or lost.

What a production workflow build does about this is concrete rather than advisory: one contained workflow — the ingestion path, the dedup stage, the relevance stage, the instrumentation — taken to production against acceptance targets written into the SOW before work starts, built and run in our environment until it hits them, then migrated to yours. It carries its first 30 days of operation, run by the person who built it, which is when a data path's real failure modes show up. The model at the end stays interchangeable, exactly as it should be. The pipeline in front of it is the asset.

References

1. Sculley, D., et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015.
2. Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv, 2020.
3. Liu, N. F., et al. Lost in the Middle: How Language Models Use Long Contexts. arXiv, 2023.
4. Anthropic. Building Effective Agents. Anthropic Engineering, 2024.
5. Google. Monitoring Distributed Systems. Site Reliability Engineering, 2016.