Packaged Web Documents Need Extraction Provenance

A web-data pipeline can be correct for HTML pages and still fail when the same source publishes the important facts inside a packaged document. Product manuals, policy documents, regulatory filings, research reports, catalogs, training material, and help-center exports often arrive as PDFs, EPUBs, ZIP-like bundles, or HTML generated by a reading system. The crawler records a successful download. The extractor returns text. The downstream dataset looks populated. Then an operator discovers that navigation order, alt text, table headers, language metadata, or a hidden section was lost on the way from document to data.

Two current W3C updates make this a good week to tighten that contract. On July 21, 2026, W3C announced that it invites implementations of EPUB 3.4, EPUB Reading Systems 3.4, and EPUB Accessibility 1.2. On July 23, W3C published WCAG Evaluation Methodology 2.0 as a Group Note. Neither announcement is a scraping tutorial. Together, they point at a production retrieval problem: when documents are web-shaped but not ordinary pages, extraction quality depends on the reading model and the sample design, not only on bytes, selectors, and OCR.

The thesis is: document extraction needs provenance for the reading system, accessibility surfaces, and sampling scope, because packaged web documents can produce plausible text while losing the structure that made the original content meaningful.

This is not a guide to access paywalled books, bypass document controls, or defeat publisher restrictions. The safe frame is permitted retrieval and defensive data quality.

The repeated angle to avoid

Recent Better Fetch posts have already covered auth boundaries, agent control points, accessibility sampling for rendered pages, resolver view, WAF change windows, Unicode matching, DNSSEC incidents, session evidence, browser-family variance, and runtime-specific fetching. The repeated angle would be another broad claim that crawlers should log more state.

The surplus here is narrower: packaged documents are not just files attached to pages. EPUB in particular is made from web technologies, reading-system behavior, metadata, navigation documents, resources, and accessibility expectations. A crawler that treats it as one blob of text throws away the same kind of state that browser-grade fetching was built to preserve for HTML.

Source map

Fresh primary evidence from the last seven days:

Older background context includes the general EPUB model of packaged web content and the long-running accessibility principle that structure, navigation, labels, and alternatives are part of meaning. The production implication is not to turn every crawler into a conformance test suite. It is to stop accepting unproven text dumps as equivalent to source documents.

Why packaged documents create quiet data drift

Document extraction failures are often quieter than page-rendering failures. A broken browser run may show a blank screenshot, an obvious interstitial, or a missing API call. A broken document run often produces thousands of words, which makes it look successful.

Common failure modes include:

• reading chapters in file-system order rather than publication order;
• dropping navigation labels, landmarks, headings, or table-of-contents hierarchy;
• flattening tables so row and column headers no longer attach to values;
• losing footnote references, figure captions, alt text, glossary entries, or sidebars;
• extracting hidden fallback content while missing the intended visible resource;
• ignoring language and direction metadata inside sections;
• mixing multiple renditions or supplemental files into one text stream;
• treating OCR output as equivalent to structured document text;
• caching a converted representation after the source package changed.

Those are not cosmetic issues. They can change facts. A safety notice separated from its product section, a price table detached from its currency header, a policy exception moved away from the clause it qualifies, or a right-to-left label stored without direction context can all become bad data.

The mechanism is that packaged documents have internal relationships. The fetch target may be one URL, but the representation is a graph: manifest, spine, navigation, content documents, media, styles, metadata, accessibility declarations, and sometimes scripts or fallbacks. If the extractor records only source_url, download_status, mime_type, and text_length, incident review cannot tell whether the right reading path was used.

Add document provenance to fetch records

For any pipeline that extracts from EPUB, PDF, office documents, archives, or generated document viewers, add a document-provenance block beside the normal fetch evidence.

Do not store protected content in broad operational logs. Hashes, counts, converter metadata, and sampled structural assertions are often enough for fleet observability. Store full artifacts only where retention, permission, and access controls are clear.

A representative sampling plan

WCAG-EM is useful to crawl operators because it starts with scope and representative selection. Document repositories have the same problem: a few happy-path samples do not prove the whole corpus.

Build a document QA set that includes:

1. A structured born-digital document. It should contain headings, navigation, tables, captions, links, metadata, and multiple sections. Assert reading order and table header attachment.
2. A scanned or partially scanned document. Verify that OCR fallback is labeled as OCR, not silently treated as structured text.
3. A multilingual document. Include section-level language changes and at least one right-to-left passage if your corpus supports it.
4. A package with multiple resources. Confirm that images, styles, navigation files, and content resources are accounted for even when only text is emitted downstream.
5. A changed-document fixture. Modify one internal resource without changing the landing page. Confirm that package digesting and cache invalidation catch it.
6. A known-bad fixture. Include a document with missing navigation or broken references and assert that the pipeline reports warnings rather than publishing with normal confidence.

The sample should be refreshed when source mix changes. A crawler processing ecommerce manuals, government PDFs, and publisher EPUBs is not one document workload. It is several workloads sharing a queue.

Diagnostics before parser changes

When document-derived records drift, use a triage flow that preserves the original representation:

1. Confirm the artifact identity: URL, redirect chain, content digest, media type, size, and last-modified or ETag evidence.
2. Re-run conversion with the same converter version and a current converter version. If outputs differ, classify it as a conversion change before changing extraction rules.
3. Compare reading order against the document's own navigation or structure source. If file order and reading order disagree, fix ordering before tuning field extraction.
4. Sample high-risk structures: tables, captions, links, warnings, footnotes, alt text, and section headings near extracted fields.
5. Check language, direction, and Unicode normalization for fields used in joins, deduplication, policy classification, or agent summaries.
6. Suppress publishing when the extracted text is complete enough to look plausible but missing the structure required to interpret it safely.

That last point matters. The worst document extraction incident is a confident, well-formed, wrong output.

Operational decision rule

Treat document conversion as a fetch stage, not a warehouse cleanup stage. If the data product depends on document facts, the run is not complete when the file downloads. It is complete when the pipeline can explain which artifact was fetched, which reading model interpreted it, which structures survived conversion, which samples passed, and which warnings affected confidence.

That gives operators a safer response to drift. Instead of guessing whether the source changed or the parser broke, they can ask whether the document package changed, the converter changed, the reading order changed, the accessibility surface changed, or the sampling plan was too narrow. Those are different incidents with different fixes.

The durable lesson from this week's W3C updates is that web data is not only pages and APIs. Some of the most valuable source material is packaged, structured, and accessibility-sensitive. Browser-grade fetching should carry that provenance all the way through extraction so a useful-looking text dump does not become a silent data-quality failure.