The record said Nike FuelBand, 2014, product launch. All four required fields populated, all four written in the same competent register as every other record in the table. Nothing about it looked wrong. Nike FuelBand happened — the campaign is real, and the description was factually-correct-sounding, which is the only claim I can actually make about it, because nobody ever checked it against a source. There wasn’t one.
The page the model had been handed was a 404.
Same story for Channel 4’s “Meet the Superhumans” and Huawei’s StorySign. Three records where the source page returned nothing, and the model wrote a confident, correct-sounding description anyway — out of its own training knowledge, without flagging that it had nothing to read. The notes I kept at the time call these “indistinguishable from real extraction by content review alone,” and that phrasing is doing a lot of work. You could not tell by reading. Not by reading harder, not by reading with more care. The output carried no signal about where it came from.
This was inside a small tool I was building — a campaign-precedent checker: ingest a corpus of ad campaigns, have a model extract structured fields from each page, embed them, and later ask “has this idea been done before?” The tool isn’t the point. The point is the shape of the failure, which belongs to any pipeline that fetches something, hands it to a model, and stores what comes back as fact. The model’s job was described to it as extraction. What it actually did, some of the time, was generation. Downstream, those are the same bytes.
Length was standing in for content
The ingest step validated pages by one rule: raw_text had to be at least 100 characters. That’s the kind of check you write in thirty seconds and never revisit, because it’s obviously right — an empty page has no content, a real page has lots.
Ads of the World’s 404 template is well over 100 characters. So is a Cloudflare bot-management interstitial. So is a geo-block notice. Every one of those sailed through, landed in the database marked ingested, and was passed to the model as though it were a campaign page.
The bar wasn’t measuring what I thought it was measuring. It was measuring bytes, and I had let bytes stand in for content. Which is the first version of the same mistake the model made later: a plausible-looking proxy accepted in place of the thing itself.
The audit missed a lot of it
I did a manual pass. Spot-check plus a systematic read-through, and it turned up 9 contaminated records — two wrong-campaign matches where the source URL slug had collided with an unrelated ad, and a set of dead pages. At the time I considered that sufficient. The pass felt thorough because I had read every record.
Then an independent review pass — a different agent, running a different check — found 5 more. Not by reading the enriched output more carefully, which was the thing I had done. By cross-referencing the raw text against 404 and dead-page signatures, a check my audit had simply never run. Two of those 5 were the Nike FuelBand / Meet the Superhumans cases. My audit could not have found them the way I was looking, because I was looking at the output and the output was clean.
The final excluded list came to 13 records. [needs check] — 9 plus 5 is 14, and my own notes say both “9 identified,” “5 additional,” and “final count: 13 excluded,” with 60 processed minus 13 leaving 47 clean. I have not reconciled that. It’s possible one record was double-counted across the two passes; I’m not going to guess which, and I’m not going to quietly pick the number that makes the paragraph read better. The whole argument of this piece is that a confident-sounding description generated to fill a gap is worse than an admitted gap.
What I wrote in the plan doc when the second pass landed was: the independent review caught real problems the manual audit missed, and that’s the review process working correctly, not something to downplay. I still think that’s right. It’s also a note you only write when you’ve just been shown you were wrong about being done.
The filter that structurally could not work
The fix I shipped first was a hedge-language filter. Some records had come back with the model flagging its own uncertainty in the field text — “does not provide enough information,” that kind of thing — and the required-field validation only tested for null, undefined, and empty string. A field full of a polite refusal counted as populated. So I added a phrase list and rejected any required field containing one:
"does not provide enough information"
"does not describe"
"appears to be a 404"
"cannot determine"
"unable to determine"
"no information available"That’s a real fix for a real class. It also cannot possibly touch the class this article is about, and I wrote that down at the time rather than letting it read as solved: it does not catch the harder confident-hallucination pattern — out of scope, flagged as a known gap, not solved here.
The reason is structural, and it’s the part worth carrying out of here. A hedge filter inspects what the model said. It works by catching the model admitting it doesn’t know. A model hallucinating with a straight face is, by definition, not admitting anything. There is no phrase list that catches confidence, because confident wrong text and confident right text are the same text. Every check that reads the output is, at best, a check on the model’s willingness to hedge — which is a property of the model’s manners, not of whether the data exists.
Later I added "n/a" to that same list, after a record came back with all four required fields literally "N/A" because the source geo-blocks non-US traffic. Safe to match on, since the literal slash doesn’t show up in ordinary prose. Still an output check. Still the same limitation.
Checking the input instead
Two days later I shipped a different kind of check: a deterministic scan of raw_text for block and dead-page signatures, running before the model is called at all.
// unconditional — can't appear in legitimate campaign prose
"you have been blocked"
"cloudflare ray id"
"please enable cookies"
"error 404"
"page you were looking for"
// length-gated — legitimate in a long case study,
// disqualifying on a short page (< 1500 chars)
"is not available to customers"
"outside of the united states"No model in the loop, no judgment, no confidence score. Either the string is in the fetched text or it isn’t. Run against every already-enriched record it produced zero false positives, and it retroactively caught two of the excluded 404s — both of their pages contained “page you were looking for.”
That’s the shape of the answer, and it generalizes past this tool: if the model can generate plausible content when the input is empty, then the only place a reliable signal exists is the input. Validate the retrieved artifact, before extraction, with something that can’t be talked out of its answer. Everything downstream of the model call is inspecting a witness who doesn’t know they’re lying.
It kept finding more
A later corpus round — 21 new candidates, a batch of Super Bowl spots — turned up 3 more confident hallucinations that the hedge filter had let through. One was a bot-management challenge page from which the model produced a factually-plausible campaign description. One was another bot-management page from the same source, where it invented a detailed pedestrian-safety storyline rather than refuse. One was a page where the fetch had captured a corporate site’s navigation menu instead of any campaign content, and the model produced vague-but-plausible filler.
Worth noting that 3 of the 5 candidates from that same source fetched real content fine in the same round. The block is probabilistic per request, not a wall you can route around by blacklisting a domain. Some runs get the page, some get the challenge, and the pipeline can’t tell which kind of run it just had unless it checks.
The block-signature scan was the response to that round, and it catches two of the three — the two bot-management pages. The navigation-menu one it does not catch, and won’t. [needs check] — my notes also contain a line claiming the check catches all three, confirmed via grep, which contradicts the same notes listing the navigation-menu record among the explicitly-unfixed cases. Two of three is the better-supported reading and it’s the one I’ve kept, but the source disagrees with itself.
What’s still broken
Two classes remain open, and I decided not to fix them.
Nav-chrome and page shells. The fetch succeeds, returns real text, and the text is a navigation menu or a page skeleton rather than campaign content. There’s no signature to grep for — it’s genuine site copy. The obvious approach is a nav-link-density heuristic, and I rejected it as out of scope on false-positive risk: I’d rather manually audit a round than silently drop legitimate records to a threshold I tuned by feel.
Thin-but-real pages. A live page with genuinely little on it — production credits and nothing else. One of these produced speculative filler (“likely leveraged…”) from a page that was completely real. There is no mechanical way to distinguish “a real page with little content” from “garbage,” because the difference isn’t in the artifact, it’s in whether a human thinks what’s there is enough.
[needs check] on one of these: my notes classify the Huawei StorySign record two ways — once as a 404 confident-hallucination case, and once, later, as nav-chrome/shell content in the still-unfixed class. Both appear. I haven’t resolved which is right, and both readings are consistent with the argument, so I’ve left the contradiction standing rather than pick.
The consequence is that the manual post-round audit didn’t go away. Every new corpus round still gets a raw-text signature grep, a brand and year cross-reference, and a hedge-language scan before its records are trusted. The block-signature check reduced the need for that step. It did not eliminate it, and the notes say so in those words.
The number I didn’t get to move
One more thing in this same family, because it’s the same temptation wearing different clothes.
The original definition of done for the ingest milestone was a locked number: at least 80 campaigns with real text in the database. I got 60 out of 95. Twenty-eight of the 35 failures were a single domain, a 100% failure rate, because its bot management blocks automated traffic from every path I had — Worker-origin fetch and harness-origin fetch alike. Not a quota problem. A wall.
The easy move is to quietly decide 80 was aspirational. What I did instead was write the reframe down in full — the pipeline ingests everything reachable through it, the grading fixtures are present and clean, and the shortfall is a confirmed external constraint — tag it ⚠️ pending sign-off, and refuse to mark the milestone done until it’s explicitly acked.
I’ll be honest about where that stands: the gate is still open. The reframe is documented and not ratified, and subsequent work proceeded in parallel without blocking on it. So it’s a process that stopped me from silently loosening a locked number, which is the part that matters, but it hasn’t yet produced a decision. A reframe pending sign-off is not the same as a reframe approved, and writing it up as though it were would be, again, the exact failure this whole piece is about.
The one-line version
A model asked to extract will generate when there’s nothing to extract, and generated text is indistinguishable from extracted text at the point where you store it. So the check has to happen upstream of the model, on the retrieved artifact, deterministically — and the checks that feel most natural to write (validate the output, filter the hedges, read it carefully yourself) are all downstream of the moment the information was lost.
That’s the reusable part. The rest is me finding out, twice, that I wasn’t done auditing.