Before the Agent Sees the Screenshot

A browser worker captures the right page at the right viewport and returns a valid image. The agent can still receive different evidence. A screencast may miss the only frame a static page emits. An MCP server may silently resize a screenshot before placing it in a tool response. Switching PNG to JPEG may shrink transfer bytes without reducing image-token cost. The final answer can look like a model failure even though the model never saw the artifact the fetch operator thought it saw.

The practical thesis is: visual browser evidence needs an end-to-end envelope that proves when the first frame became observable, preserves original pixel geometry, records every crop or resize, and distinguishes encoded byte cost from model-input cost before an agent-derived field can be accepted. A screenshot pathname, MIME type, or successful tool call is not that envelope.

This is reliability guidance for public or properly authorized pages. Screenshots can contain credentials, personal data, private account state, and publisher-restricted content. Capture scope, retention, redaction, and access decisions remain part of the fetch policy; changing image settings must never become a way to inspect content beyond an approved boundary.

Fresh evidence across the visual path

Current changes expose three separate boundaries between browser pixels and an agent:

Earlier Better Fetch posts already cover browser-version canaries, composite agent snapshots, artifact finalization, and screenshot hashes as supporting evidence. The repeated article would say to pin browser versions and save screenshots. The new information surplus is the visual delivery chain: capture admission, original geometry, transformation lineage, and consumer admission can each change what a visual claim means even when the browser and page are unchanged.

Four receipts hidden inside one image

Treat a visual tool result as the join of four receipts.

1. Capture admission

A command acknowledgement proves that a browser accepted startScreencast; it does not prove that the collector observed a frame. Puppeteer's fresh race is a classic lost-event problem: if the producer can emit during command settlement, subscribing afterward creates a gap. A dynamic page may later emit another frame and hide the defect. A static fixture may not, so startup waits forever.

Establish listeners before enabling the producer, then require a first-frame identifier and timestamp. If start fails, remove listeners and settle the capture as failed. Do not infer readiness from an output file created early or from a command response alone.

For one-shot screenshots, the analogous admission questions are which document generation was current, whether the intended viewport had settled, whether fonts and required images were ready, and whether a navigation or overlay transition occurred during capture. Generic network idleness is not a visual commit point.

2. Original geometry

Record width, height, device-pixel ratio, viewport dimensions, full-page versus viewport mode, scroll position, clip rectangle, and page/document generation at the browser boundary. These fields explain what pixels could have existed before another component touched them.

Geometry is semantic for web-data work. Downscaling can erase small prices, superscripts, decimal separators, chart labels, disclosure text, or OCR landmarks while preserving a plausible overview. A full-page capture can contain more pixels but make each region smaller after a model-facing size cap. A high device-pixel ratio can improve source detail while increasing transport and consumer cost.

The right dimensions therefore come from the evidence contract, not a universal maximum. A layout-diff job, OCR pass, broad navigation agent, and human incident review may require different derivatives from the same original.

3. Transformation lineage

PNG, JPEG, and WebP are encodings. Resize, crop, color conversion, quality reduction, and annotation are transformations. Keep those concepts separate.

Changing format can reduce network transfer, object-storage usage, and serialization overhead. It may also introduce lossy artifacts. It does not necessarily reduce a vision model's input charge when the provider meters by pixel dimensions, patches, or tiles. The current Chrome DevTools MCP proposal makes that operational distinction explicit and links to the Claude vision guidance, OpenAI image-input guidance, and Gemini image-token guidance.

Every derivative should point to its parent and declare the operation: encoded_only, resized, cropped, redacted, annotated, or a combination. Never label a resized image as the original screenshot merely because its MIME type and filename stayed the same.

4. Consumer admission

The MCP response is another boundary. Record which image derivative entered the tool response, its dimensions and digest, and which client or model adapter received it. The adapter may resize again, reject a large image, tile it, compress it, or omit it under a context policy.

Do not assume the model can report these transformations. The Playwright issue notes that the model-facing workflow did not expose the delivered dimensions. Make delivery metadata available to the orchestrator and incident reviewer, then bind agent-derived fields to the delivered derivative ID rather than to the original capture ID.

Issue a visual evidence envelope

A compact envelope should include:

• run, task, session segment, browser context, target, frame, and document-generation IDs;
• automation and browser versions, operating system, graphics mode, viewport, device-pixel ratio, and color profile when material;
• capture operation ID, requested mode, clip or scroll coordinates, readiness marker, dispatch time, command acknowledgement, and first-frame or screenshot-completion time;
• original width, height, format, byte length, digest, and restricted artifact reference;
• derivative ID and parent ID, transform sequence, resize algorithm, output dimensions, quality, redaction policy, and transformer version;
• tool response ID, delivered derivative, delivered dimensions, encoded bytes, client adapter, model class, and any known model-side image policy;
• OCR or vision result ID, required-region checks, confidence or disagreement indicators, and provenance for each accepted field;
• final verdict: original_preserved, derived_declared, capture_unsettled, geometry_unknown, consumer_transformed, required_detail_lost, policy_stop, or unknown.

Keep raw visual artifacts behind stricter controls than ordinary metrics. Broad telemetry usually needs dimensions, hashes, classifications, timings, and restricted references—not the image itself or a full authenticated URL.

An owned-fixture visual delivery test

Build one deterministic page with a static region, a delayed region, small text, a decimal price, thin chart labels, high-contrast and low-contrast controls, an overlay, and markers near every viewport edge. Then test the complete deployed path:

1. Start a screencast on the already-static page. Require the first frame to be observed without waiting for a later mutation, and verify listener cleanup when start is rejected.
2. Capture viewport, clipped, high-device-pixel-ratio, and full-page variants. Assert original dimensions and document generation before examining content.
3. Pass an image above every configured side and pixel threshold through the MCP server and client adapter. Compare digests and dimensions at browser output, tool response, and model input.
4. Encode the same original as PNG, JPEG, and WebP. Measure bytes, transfer time, storage cost, delivered dimensions, and provider-reported usage separately. Do not infer one from another.
5. Create declared downscaled derivatives. Run OCR and required-region checks at each size to find the smallest acceptable derivative for that task rather than choosing a fleet-wide limit.
6. Trigger a navigation, overlay transition, and font completion around capture. Require the envelope to identify the generation and readiness marker that supplied the pixels.
7. Remove one lineage link or delivered-dimension field. Confirm that downstream visual claims become unknown rather than inheriting confidence from the original file.
8. Return a login, challenge, denial, payment boundary, or private fixture. Require policy classification before image delivery and prohibit publication as ordinary page evidence.

Run this matrix after browser, automation library, MCP server, image codec, client adapter, model, or image-policy changes. An upstream merge is not enough; the exact released chain must preserve the contract.

The operator decision rule

When an agent misses visible detail or visual automation hangs, locate the first unsupported boundary. No observed first frame means capture never became ready. Correct browser geometry plus different delivered geometry means an undeclared downstream transform. Equal dimensions but a changed digest points to encoding, quality, rendering, or content drift. Smaller bytes with unchanged dimensions is a transfer/storage optimization, not proof of lower model cost. If the delivered artifact cannot be identified, quarantine every field that depends on it.

The browser does not hand pixels directly to an agent. It hands them through event timing, screenshot code, image codecs, tool responses, client adapters, and model-specific preprocessing. Preserve that chain, and visual failures become explainable differences in capture, geometry, transformation, or delivery instead of vague claims that the model could not see the page.