The Button Said Next. The Crawl Changed the Wrong State.

A crawler sees a button labelled 15, classifies it as page 15, clicks it, and waits for more records. The control was actually a datepicker day. On another page, Preview becomes prev because a detector matched a substring. On page one, a native disabled Previous button is reported as enabled, so the agent spends a step clicking something that cannot advance.

These are not only element-detection mistakes. Each one can corrupt crawl state: the agent may change a date, rating, size, quantity, wizard step, or filter while believing it expanded the result set. The page can remain healthy and the click can succeed, leaving the dataset incomplete or drawn from the wrong application state.

Fresh Browser Use reports make the mechanism concrete. An issue opened August 22 describes pagination detection matching words inside unrelated labels and CSS classes, while also treating short numeric clickable text as page numbers. An open fix from August 22 narrows matching to complete tokens in user-facing labels, excludes CSS classes, and requires an explicit button or link role for numeric candidates. A separate open pull request from August 22 fixes native disabled-state handling: Chrome DevTools Protocol represents an HTML boolean disabled attribute by its presence, commonly with an empty string value, not necessarily the string true.

The thesis: pagination is a state-transition hypothesis, not a DOM classification fact; a production crawler should admit candidates from several weak signals, then prove that one approved action advanced the intended collection before it expands the crawl frontier or accepts new records.

This is guidance for public or properly authorized collection. A login, challenge, payment boundary, denial, publisher restriction, or rate limit remains a stop or scheduling signal. Pagination evidence must not become a reason to change identity, route, or session after access is limited.

The repeated angle to avoid

Recent Better Fetch posts already cover composite agent snapshots, target-generation receipts, frame completeness, API discovery, session evidence, and rate-aware pagination. The repeated article would say that labels can be ambiguous, browser state belongs in logs, or structured APIs are preferable to clicking.

The new surplus is a frontier-admission contract. Candidate detection, actionability, transition identity, collection progress, and loop prevention are independent questions. Better token matching improves the first question; it cannot prove the last four.

Source map

Fresh primary evidence from the last seven days:

Older background is the distinction between native HTML boolean attributes, ARIA states, DOM roles, link relations, and application-specific list state. None alone proves that a control advanced the collection the data product intended to crawl.

Five claims hidden inside one Next button

A pagination action should separate these claims:

1. Candidate. The control has evidence consistent with pagination: a complete visible or accessible label, rel=next, a recognized navigation region, a numeric sibling sequence, an observed list API cursor, or a source-specific contract.
2. Eligible. The control is visible or otherwise valid for the approved interaction path, belongs to the expected document generation and list region, and is not natively or semantically disabled.
3. Action dispatched. One click, navigation, scroll, or approved API request was sent to the exact candidate generation.
4. Intended transition observed. The expected list, cursor, URL, request, or item-key set changed—not merely some page state.
5. Frontier advanced. The resulting collection state is new, policy-valid, non-cyclic, and eligible to contribute records or another continuation.

The August 22 reports concern claims one and two. Preview matching prev is a candidate false positive. A native disabled attribute represented as an empty value is an eligibility error. Even perfect fixes at those layers cannot show that a click loaded page two rather than applying a filter, reopening cached page one, or firing an analytics-only handler.

Roles also need the right weight. Requiring an explicit button or link role can reduce numeric false positives, but many valid sites omit or misuse roles. Treating the role as absolute trades false positives for false negatives. For deterministic sources, a reviewed selector or discovered cursor contract can outweigh generic semantics. For unknown pages, ambiguous candidates should require more contextual evidence rather than an agent guess.

Issue a collection-transition receipt

Before acting, snapshot the collection state and mint an attempt ID. Afterward, join the evidence into one receipt:

• run, session segment, browser context, target, document generation, and pagination_attempt_id;
• collection identity: template, list container, query and filter digest, locale, region, consent class, and authorization class;
• candidate reference and evidence: text, accessible name, title, role, native link relation, ancestor region, numeric-sibling pattern, and source-specific rule;
• eligibility evidence: native disabled presence, explicit aria-disabled, computed visibility, actionability, occlusion, and generation match;
• baseline URL, history entry, cursor or offset, item count, stable item-key digest, and relevant response identifiers;
• action type, dispatch time, exact target reference, and action result;
• post-action document generation, URL, cursor, request set, item count, item-key delta, and unrelated state mutations;
• policy outcome and rate-limit evidence;
• final verdict: advanced, advanced_with_overlap, no_op, unrelated_mutation, cycle, disabled, policy_stop, or unknown.

Do not broadly log full authenticated URLs, cookies, private item content, or raw page snapshots. Normalized query shapes, counts, bounded hashes, state classes, and restricted artifact references are usually enough.

A new URL alone is weak progress evidence: tracking parameters can change while the list stays fixed. A larger item count is also weak: virtualized lists can replace rows, and duplicate append operations can inflate the DOM. Prefer stable entity keys from an approved API or extraction contract. When those do not exist, combine normalized field hashes with cursor, request, and visible-container evidence.

Keep heuristics away from the frontier

Use generic detection to propose candidates, not to commit crawl work. A safe sequence is:

1. Prefer an approved structured cursor or explicit next link when it explains the rendered collection.
2. Otherwise score controls using user-facing labels, navigation context, sibling structure, actionability, and a source-specific contract. CSS class substrings are hints at most.
3. Reject disabled, ambiguous, detached, stale-generation, or policy-bound candidates before action.
4. Capture the baseline receipt, then dispatch one bounded action.
5. Wait for a source-specific transition signal—not generic network idleness or a fixed sleep.
6. Compare item keys and collection state. Admit only a proved new state to the frontier.
7. Stop on no progress, a repeated state digest, a repeated cursor, a policy outcome, or the configured page and item budget.

This structure contains both kinds of classifier error. A false positive cannot silently expand the frontier because an unrelated mutation fails the transition check. A false negative reduces coverage but becomes observable through expected-count checks, missing continuation signals, or canary comparisons instead of causing an arbitrary click.

An owned-fixture pagination test

Build a fixture matrix containing:

1. real Previous, Next, First, Last, and numbered controls;
2. Preview, Next steps, Last name, CSS classes containing similar substrings, a datepicker, ratings, sizes, quantities, and cart badges;
3. native disabled attributes with empty serialized values, explicit aria-disabled states, class-only disabled styling, and visually occluded controls;
4. link navigation, client-side list replacement, append-only infinite scroll, cursor-backed API loading, and virtualization that reuses DOM rows;
5. a no-op handler, an unrelated filter mutation, duplicated page content, a cursor loop, and a URL change without item change;
6. a re-render between detection and action so the candidate's document generation becomes stale;
7. challenge, login, denial, and 429 fixture outcomes that must stop or schedule the run rather than select another path.

Measure candidate precision and recall separately from transition accuracy and frontier accuracy. A detector can score well while the crawler still loops or admits duplicates. Run the matrix against the exact browser-agent and automation builds in production; today's pull requests are evidence to test, not proof that a released package contains the behavior.

The operator decision rule

When pagination coverage drops or a crawl mutates the wrong widget, locate the first unsupported claim. If candidate evidence was weak, tighten context and source rules. If eligibility was wrong, repair native, ARIA, visibility, and generation handling. If the intended control was clicked but no collection transition occurred, stop rather than clicking harder. If the list changed but item keys or cursor repeat, classify a cycle and close that frontier branch. If a policy outcome appears, preserve it and end or reschedule according to source policy.

A label can suggest where more records live. Only a verified collection transition can prove the crawl reached them. Keep that distinction at the frontier, and pagination failures become bounded, diagnosable gaps instead of silent changes to the dataset.