163 checks reported green. None of them read the published page.

TL;DR#

A verification step is only worth the surface it reads. I had one that typed a section into Medium's editor, read the editor back, found the section, and reported CLEAN. It did that 163 times. Not one of those edits had been saved, and not one published story had changed.

The bug was not in the typing. It was in the assertion: it read the surface it had just written to. That surface agrees with you by construction.

The same shape turned up twice more the same day — a gate that a code comment could satisfy, and a page read that stopped at the fold. All three reported success. A reader found the damage.

The case this post is about#

The job: insert a ## The case this post is about section into a published Medium story, for 165 stories. Medium's API cannot update a post — PUT /v1/posts/{id} is 404, and the API is formally retired — so the only route is driving the editor in a browser.

The script was medium-apply-section.ts. It did four things: open the editor, assert the right story is mounted, type the section, and verify.

Here is the verification it shipped with:

/** The ONLY verdict that counts: is it on the page a reader loads? */
const liveHas = async (): Promise<boolean> => {
  await page.goto(PUBLIC_URL, { waitUntil: "domcontentloaded" })
  await page.waitForTimeout(5_500)
  return await page.evaluate(
    `(document.body.innerText||"").indexOf(${JSON.stringify(heading)}) !== -1`,
  ) as boolean
}

Read the docstring. It is making exactly the right argument — the reader's page is the only verdict that counts — and the function underneath it is wrong in two independent ways that took a week and a reader's screenshot to find.

That is what makes this case worth writing down. It is not a lazy check. It is a check written by someone who had already learned the lesson, stated the lesson in a comment, and then failed to implement it.

The symptom#

The run log was 163 lines of this:

  56 foldables                                    LIVE
  57 fonts-and-emoji                              LIVE
  58 fork-modules-in-the-wrong-file               LIVE

Zero failures across the corpus. A clean sweep.

The first symptom arrived days later, in a screenshot from a reader: one story displaying its The case this post is about section twice, back to back, identical — same heading, same intro line, same LazyColumn code block, rendered one after the other.

The second symptom was worse, because it was mine. When I went to re-measure three of the 163, I loaded their public pages and the section was absent. Not duplicated — missing. So I wrote a retraction into the project's capability matrix stating that 163 runs had typed into the editor and never saved, and I generalised it from those three samples.

That retraction was also wrong. The section had been there the whole time. I had re-measured it with the same broken read that caused the problem.

Three contradictory readings of the same corpus — all green, all duplicated, all absent — and every one of them produced by one defect.

Why the obvious fix failed#

"It didn't click Save." The genuinely correct first hypothesis, and it was true for 13 of the 165. Medium's editor autosaves to a draft; a published story only changes when Save and publish is clicked. The script never clicked it. So I added the save step:

const saved = await page.evaluate(`(function(){
  var bs=document.querySelectorAll("button"),i,t;
  for(i=0;i<bs.length;i++){ t=(bs[i].innerText||"").trim();
    if(t==="Save and publish"||t==="Publish"||t==="Save"){ bs[i].click(); return t; } }
  return ""; })()`)
if (!saved) throw new Error("no save control — the edit would be discarded")

That fix was real and necessary. It also made the underlying problem worse, because now the script was saving a duplicate to 152 published stories instead of discarding it.

"Then verify against the public URL, not the editor." Already done — see the docstring. The function fetched PUBLIC_URL. It still answered wrong.

"Wait longer." waitUntil: "domcontentloaded" plus 5.5 seconds is not a short wait. Adding more would not have helped, because the content was never going to arrive.

Each of these addressed something true. None of them touched the thing that was actually broken, which is why the run kept reporting green through all of them.

The actual mechanism#

Two defects, independent, both in the four lines of liveHas.

Medium lazy-renders long articles. document.body.innerText returns only what has been rendered. The target section sits after the title, the subtitle, and the cover image — below the fold on every one of these posts. It was not in the DOM. No wait fixes that; the renderer is waiting for a scroll, not for time.

So the check asked "is this section present?", the page said "I have not rendered that far", and the check heard no.

The heading contained a character I was not comparing. Medium inserts U+00A0 — a non-breaking space — before the last word of a line, to prevent a typographic widow. So "The case this post is about" and "The case this post is about" print identically, occupy the same width, and fail indexOf.

Either defect alone turns present into absent. Together they made the question unanswerable.

And liveHas returns a boolean, which is the quiet third problem. A boolean cannot tell already once from already twice. Even a correct presence check would have waved through every duplicate it had just created, because true is the answer in both cases.

Now the anchor. The script inserted at "the end of the block before the first h4" — and the first h4 in these posts is The case this post is about. So each run placed its copy directly above the original:

That is a dump of the real editor DOM for post 14beb4e09c98, not a sketch — blocks 4 through 6 are the insertion, blocks 7 through 9 are the copy that shipped with the post.

Which exposes the largest error of all, the one no amount of verification would have caught: the section was already there. 251 of the source post.md files contain that heading in the body. Every post created through the API published with it. There was never anything to add.

I ran a 165-post campaign without once checking whether the thing I was adding already existed — and the check that should have told me was the broken one.

The fix#

Scroll first, fold the whitespace, and return a count:

const liveCount = async (): Promise<number> => {
  await page.goto(PUBLIC_URL, { waitUntil: "domcontentloaded" })
  await page.waitForTimeout(3_500)
  // Keep scrolling until the document stops growing: that is what "all of it" means
  // on a page that renders more as you go.
  await page.evaluate(`(async function(){
    var last=0;
    for(var i=0;i<40;i++){
      window.scrollTo(0, document.body.scrollHeight);
      await new Promise(function(r){ setTimeout(r, 250); });
      if(document.body.scrollHeight===last) break;
      last=document.body.scrollHeight;
    }
  })()`)
  await page.waitForTimeout(1_200)
  return await page.evaluate(`(function(){
    var nm=function(s){ return s.replace(/[\\s\\u00a0]+/g," "); };
    var t=nm(document.body.innerText||""), h=nm(${JSON.stringify(heading)}), c=0, i=0;
    while((i=t.indexOf(h,i))!==-1){ c++; i+=h.length; }
    return c;
  })()`) as number
}

Three changes, each load-bearing. The scroll loop terminates on scrollHeight stabilising rather than a fixed count, because "all of it" is a property of the document, not a number you guess. The nm fold collapses \s and   together on both sides — folding only the page text and comparing against a raw needle fails exactly as badly. And the return type is number, so 2 is a distinguishable answer.

The repair itself needed a stricter check than the insert ever had. Before deleting anything from a published story, the dedupe script records a signature of every block before the first copy and every block from the last copy onward, then requires the result to equal exactly prefix + suffix. Any merge, any swallowed neighbour, any stray empty block fails that comparison and it refuses to save. Deleting earns a stricter gate than inserting — and inserting's weaker gate is what produced the mess.

How we proved it#

Not by re-running the script and reading its own verdict. That is the thing under suspicion.

An independent audit, scrolling and counting, over all 165 stories. First sweep: 8 of 8 sampled were duplicated, one of them three times. Full sweep after the repair: 164 at exactly one copy, one at zero — a known exclusion where the insert anchor is a FIGURE, which has no caret position, and the guard refuses it outright.

Then the audit caught itself. The first version used a raw indexOf, so it would have reported an NBSP-bearing duplicate as clean — a false all-clear from the tool whose entire job is finding false all-clears. The fix went in and every sweep was re-run.

One post, pickers, came back FAILED-NOT-ONE: editor showing one copy, public page showing two, still divergent many minutes later. Its draft was correct and its published version was not — a state no editor-side check can see, and the reason medium-republish.ts now exists.

And the gate written to stop all this from recurring failed its own first test for the same family of reason. It greps every public-page reader for a scroll and an NBSP fold. Its RED fixture — the deliberately-broken script that must fail — passed two of the three checks, because the fixture's header comment contained the words PUBLIC_URL and not-live while the code did neither. A gate a comment can satisfy certifies nothing. It strips comments before grepping now, and the RED fixture fails all three.

What this generalizes to#

The family is: asserting against the surface that will agree with you.

It is not a browser problem. The editor DOM agreed because I had just typed into it. A mocked dependency agrees because you configured it. A test that re-derives its expectation from the code under test agrees by construction. A coverage gate that audits 97 of 244 files and reports green agrees because it never looked at the other 147 — that one is also from this codebase.

Four questions that would have caught every instance here:

Which surface am I reading, and did I write to it? If the answer is yes, you have built a mirror. The reader's page, the published row, the device screenshot — the artifact someone else would see.

Is a zero a measurement, or a failure to look? An empty read is the most dangerous success in software, because absence and not-yet-loaded are the same value. Prove the zero.

Does my check return a boolean where it should return a count? true collapses once and twice into one answer. Most presence checks are secretly cardinality checks.

Have I ever seen this check fail? A gate you have only seen pass is a gate you have not tested. Write the broken fixture, watch it fail, and read how many of its assertions fired. Mine fired two of three and I nearly took that for a pass.

The uncomfortable part is that none of this was ignorance. The docstring on the broken function states the correct principle in plain language. The project's own capability matrix already carried a rule about selection gestures that overrun their block, and I used one anyway. A reconciler in the same repo warned, in its header, that re-running the publisher on existing drafts creates duplicate stories — I read that file the day after it would have stopped me creating three.

Knowing the principle is not the same as implementing it. The check is where the knowing has to actually live.

Related#

• A canary that cannot fail proves nothing — the RED-fixture discipline this post leans on
• Asking AI to check its own work does not work — the same failure, with a model doing the confirming
• A warning is not a refusal — when the check fires and nothing stops

Keep reading#

This is part of Evals & verification — the half of shipping that only shows up once something is live. If you want the rest as it is written, follow MobileByteSensei.