Treat 429 as Flow Control, Not a Proxy Cue

A 429 Too Many Requests response is not automatically evidence that your scraper needs a new proxy, a stealthier browser, or a bigger retry loop. It is often the server telling you that the request rate, quota scope, or queue shape is wrong.

The practical move is to treat 429 as flow-control feedback. Capture the limit signal, slow the right worker, and only escalate to browser or proxy changes when the evidence points to identity or bot enforcement.

429 has a specific job

RFC 6585 defines 429 Too Many Requests for rate limiting: the user has sent too many requests in a given amount of time. The same section says a response should include details explaining the condition and may include a Retry-After header that tells the client how long to wait before making another request.

MDN's 429 reference gives the same operational summary: 429 indicates rate limiting, and a Retry-After header may be present. Cloudflare's 429 support documentation is more explicit about production systems: servers use 429 to prevent excessive API requests, manage traffic spikes, and temporarily block excessive requests.

That does not mean every 429 is friendly or perfectly documented. It means the first response should not be randomization. The first response should be measurement.

Retry-After is a contract surface

RFC 9110 defines Retry-After as a response header that tells a user agent how long it ought to wait before a follow-up request. The value can be either an HTTP date or a number of seconds.

For fetch infrastructure, that header should become a scheduling input, not a log line you ignore. A production client should parse it, persist it with the request outcome, and apply it to the right queue.

The right queue matters. If one product-listing URL returns 429, that may not mean every job for the domain should stop. If the limit is account-wide, every worker using that credential should slow down. If the limit is per IP, rotating too quickly can make the job noisier. If the limit is per endpoint, a discovery pass might still be allowed while pagination waits.

Do not flatten 429 into blocked

Many scraping stacks collapse all non-200 responses into a single retry path:

status != 200 -> retry with a new proxy

That throws away the most useful part of the signal. A better fetch record separates rate limiting from bot walls, policy blocks, and transient failures:

url
final_url
status_code
retry_after_seconds
rate_limit_scope_guess
credential_or_session_key
country
browser_context_id
request_family
queue_name
attempt_number
next_allowed_at
replay_mode

The rate_limit_scope_guess field will be imperfect at first. That is fine. Start with values like endpoint, host, credential, ip, session, and unknown. The point is to force the retry layer to ask what is being limited before it changes the wrong variable.

Browser-grade fetching still needs pacing

A real browser can solve a rendering problem. It can execute JavaScript, receive cookies, expose XHR traffic, and keep a sticky session. It does not make an unlimited request budget appear.

This is especially important when API discovery works. Once you find the JSON endpoint behind a page, it is tempting to fan out pagination as fast as your workers can run. That is exactly when 429 should change the plan. The clean endpoint is still a product resource with capacity, abuse, or commercial limits.

Use browser-grade fetching to discover and validate the data path. Then run extraction with a queue that understands per-host and per-endpoint pacing. If the target gives you Retry-After, let that value override your default backoff for that scope.

Separate rate limits from bot enforcement

A 429 can coexist with bot management, but it is not the same diagnosis as a challenge page, a JavaScript detection failure, or a 403 with WAF evidence.

A useful retry classifier might look like this:

This split prevents two bad behaviors. First, it stops you from rotating proxies against a normal quota response. Second, it stops you from pretending a bot challenge is only a rate-limit problem.

A simple pacing loop

A practical scheduler only needs a few rules to improve reliability:

1. Parse Retry-After as seconds or an HTTP date.
2. Store next_allowed_at for the narrowest safe scope you can infer.
3. Pause queued requests for that scope instead of retrying immediately.
4. Add jitter when no explicit delay is provided.
5. Lower concurrency after repeated 429s, even if individual retries eventually succeed.
6. Keep browser sessions sticky, but do not use stickiness as a substitute for pacing.

The builder takeaway is simple: 429 is not a mystery bucket. It is a scheduling signal until proven otherwise.

If your fetch pipeline records the status, retry window, scope guess, session, region, and replay mode, you can make a precise decision: wait, slow one queue, change credentials, keep the request inside the browser context, or stop. That is cheaper and safer than treating every 429 as a proxy problem.