The Three Loss Gates Before Clean Markdown

A crawler returns 200, renders the page, and emits well-formed Markdown. The output can still be wrong before a field extractor sees it. A client option may never reach the server. Overlay cleanup may delete the document body. A relevance filter may remove whitespace that gives code meaning. A heuristic intended to drop large JSON blobs may instead abort conversion on an unusual line.

These are different failures, but they share an operational boundary: “clean content” is not one feature. It is a lossy transformation pipeline.

The practical thesis is: clean Markdown should be accepted only when the system can prove the requested transformation ran, protected document structure survived, and every lossy stage stayed within declared content-class budgets. A successful navigation and a non-empty Markdown string prove none of those claims.

This is defensive data-quality guidance for public or properly authorized content. Cleaning must not hide login pages, challenges, payment boundaries, denials, rate limits, or publisher restrictions. Classify those outcomes before transformation and stop or schedule according to source policy.

Fresh evidence from three loss gates

Current changes in three crawler projects expose the pipeline at different stages:

Earlier Better Fetch posts already cover representation provenance, Unicode interpretation, accessibility planes, and packaged-document structure. The repeated article would say that extraction is lossy and operators should log converter versions. The new surplus here is a stage-level acceptance protocol: configuration admission, DOM mutation, content pruning, and text serialization need independent receipts and invariants because they fail in opposite directions.

Gate one: prove the effective transformation plan

A job specification is only intent. Between application code and the cleaning engine sit typed SDK models, default handling, naming conversion, JSON serialization, API validation, feature flags, and version skew.

The Firecrawl report is an example of false admission: application code can appear to request clean-only content while an SDK model ignores the field. The fetch still succeeds, but the returned representation is not the one the caller thinks it purchased or tested. This can preserve navigation, cookie banners, repeated menus, or unrelated page text, consuming model context and changing downstream classification without raising a transport error.

Mint a transformation-plan receipt before fetching:

• caller and SDK versions, server or worker version when known, and schema version;
• requested options plus a canonical hash of the request model;
• serialized wire keys and a hash of the actual outbound payload;
• defaults applied by the client and defaults known to be server-side;
• rejected, ignored, unsupported, or unknown options;
• effective transform ID returned by the service, when the service supports one;
• final plan verdict: acknowledged, client_only, server_unknown, rejected, or version_skew.

Request models should reject unknown fields by default. If compatibility requires permissive handling, emit a warning and mark the plan unacknowledged. Unit tests should inspect serialized payloads, not merely prove that a constructor accepts an option.

Gate two: protect structure before scoring relevance

Cleanup usually mutates a document before Markdown conversion. It may close or remove overlays, discard hidden regions, select a main-content subtree, remove navigation, and score nodes for relevance. Each operation can delete more than intended.

The Crawl4AI overlay report demonstrates a structural false positive: a broad class-substring rule selected body, so a valid cleanup action emptied the page. The code-block report is subtler. Whitespace-only syntax-highlighting spans have little text density and can score as noise, yet their spaces and newlines distinguish FROM golang from FROMgolang and preserve indentation-sensitive examples.

Do not apply one loss budget to every node. Define protected classes and invariants:

• document roots must exist after DOM cleanup;
• the selected main region must still descend from the current document generation;
• headings retain order and non-empty text;
• tables retain header-to-cell relationships required by the data product;
• pre and code retain whitespace, line count, and fence boundaries;
• links retain destination association even when navigation chrome is removed;
• source-specific required regions and harmless sentinel markers survive;
• policy pages remain recognizable rather than being cleaned into ordinary-looking emptiness.

Record counts before and after each destructive stage: elements, visible text characters, headings, lists, tables, code blocks, links, images with alternatives, and required-region markers. Also record removed counts by rule ID and content class. A single before/after character ratio cannot distinguish desirable menu removal from a missing price table.

Gate three: make conversion locally recoverable

After DOM selection, Markdown preprocessors still make semantic decisions. They collapse blank lines, rewrite links, serialize tables, normalize text, and may remove script-shaped or JSON-shaped lines. Input remains page-controlled and can be malformed, huge, deeply nested, or merely unusual.

The Browser Use proposal shows why a narrow parser heuristic needs a narrow failure domain. If a line cannot be parsed as the JSON blob a filter wants to remove, the conservative result is normally to keep that line with a warning. Letting one recursion exception abort the entire page turns an optional noise filter into a total extraction dependency.

Every heuristic should declare one of four outcomes for each candidate: removed_by_rule, retained, retained_unparseable, or stage_failed. Do not convert stage_failed into empty Markdown, and do not silently fall back from clean to raw output while labelling it clean. If fallback is allowed, publish an explicit representation class such as raw_fallback and rerun the downstream quality contract for that class.

Useful conversion evidence includes stage name and version, input and output digests, duration, warnings by class, retained-unparseable count, removed-byte and removed-line counts, truncation reason, and final representation verdict. Keep sensitive page text out of broad logs; bounded hashes, counts, rule IDs, and restricted artifact references are usually enough.

A conservation test for cleaning pipelines

Build owned fixtures that prove both removal and preservation:

1. Send a valid option, its explicit false value, an unknown option, and a misspelling. Assert the outbound payload and effective-plan verdict for each.
2. Put popup in the body class while rendering a real removable overlay. Require the body and main marker to survive and the overlay marker to disappear.
3. Use a fixed-position, scroll-locked body plus a modal. Verify structural roots are protected without preserving the unwanted modal.
4. Include syntax-highlighted code where spaces live in child spans, indentation changes meaning, and a low-text-density block is still required. Compare normalized token boundaries and line structure, not only presence of a code fence.
5. Add a genuinely removable large JSON blob, malformed JSON, and a deeply nested bracket line. Require bounded processing and a declared per-line outcome.
6. Include headings, a table, a list, links, alternative text, and source-specific required fields. Assert per-class conservation budgets after every stage.
7. Return a login, challenge, 403, 429, and payment boundary. Require a policy verdict before cleanup and prohibit publication as ordinary clean content.
8. Run the matrix against the exact SDK, server, crawler, browser, and filter versions in production. An open pull request or a merge to a development branch is evidence to test, not a deployment guarantee.

The key metric is not maximum retention. Cleaning is supposed to remove material. Measure explained loss: every removed unit belongs to a versioned rule, every protected class meets its invariant, and every fallback changes the representation label.

The operator decision rule

When clean output becomes empty, noisy, or semantically damaged, find the first unsupported gate. If the serialized request did not contain the option, classify configuration admission failure. If document roots or required regions disappeared, reject the structural transform. If protected content classes breached their budgets, quarantine the representation even when Markdown parses. If a heuristic failed without a local outcome, fail or explicitly fall back; never publish an empty success. If the source returned an access-policy boundary, preserve it and stop rather than tuning cleaners to make it disappear.

Clean Markdown is useful because it compresses a complicated page into a representation that extractors and agents can handle. That compression is trustworthy only when the pipeline can explain what it was asked to do, what it removed, what it protected, and where uncertainty remained.