Proxy errors 407 and 429 need different fixes
Diagnose the response, respect Retry-After and keep retries bounded.
A collector that retries every failure makes a small problem expensive. A 407 needs an authentication check. A 429 needs a rate-limit response. Neither becomes easier to diagnose when hundreds of workers repeat the same request.
Give each failure a name before giving it another attempt. Record which request failed, which component returned the response and what the scheduler did next. That turns an error spike into something a person can investigate.
Route failures to different actions. Pause repeated authentication failures for configuration repair. Schedule rate-limited work after the indicated delay. Put a limit on attempts and on total elapsed time so the queue can reach a clear outcome.
Read the response before changing the connection
Start with the status code, response headers and a small, redacted description of the response body. If your client exposes a separate CONNECT response, keep that separate from the destination's response. A dashboard total labelled “failed requests” is too broad to explain either one.
| Observed result | Meaning or next check | Suggested action |
|---|---|---|
| 407 | The proxy requires valid authentication. | Inspect authentication configuration and the proxy challenge. |
| 429 | The responding service is rate limiting requests. | Inspect Retry-After and pause the relevant queue. |
| No HTTP response | Inspect the client's DNS, connection, TLS or timeout error. | Diagnose that failure separately before choosing a retry policy. |
| 200 with missing fields | Transport succeeded, but the record may be unusable. | Validate the page and extraction result before accepting it. |
Keep a distinction between evidence and diagnosis. “Received 429 from this endpoint” is an observation. “The destination limits every IP to ten requests” is a hypothesis unless the service documents that rule or you have reliable evidence for it.
Fix 407 at the authentication boundary
A 407 Proxy Authentication Required response means the request lacks acceptable proxy credentials. The response includes a Proxy-Authenticate challenge; a client can answer with Proxy-Authorization. That exchange concerns the proxy rather than a website login.
Inspect the configuration that actually reached the worker. Check whether a secret was loaded, whether the username contains the intended routing options, and whether the client is using the correct endpoint and authentication method. If your setup uses an IP allowlist, check the worker's outbound address against the intended configuration.
Do not print the password or authorization header to prove that a value exists. Log the configuration revision and a boolean such as proxy_credentials_loaded. Keep any detailed diagnostic capture in restricted storage with credentials removed.
An initial challenge can be part of an authentication exchange. A repeated final 407 after the client has attempted authentication deserves a different response: stop the affected worker group and repair the configuration. A longer retry interval will not correct a mistyped secret.
Treat 429 as a scheduling signal
429 Too Many Requests reports rate limiting and may include Retry-After. The standard does not prescribe how requests are counted or how the caller is identified. A quota might be shared across more than one connection, so changing an IP is not a reliable diagnosis or remedy.
First establish which service returned the response. In a layered collection system, the limit could apply to your account, an intermediary or the destination. Check the service's response details and request identifiers without assuming that every 429 originated at the website you wanted to collect.
Then coordinate the response across workers. If ten workers use one account-level quota, putting one worker to sleep leaves the other nine generating traffic. Pause the queue that shares the suspected limit, preserve its pending work and resume with a lower configured rate.
That queue scope is an engineering choice, not a fact encoded in the status code. If the scope is unknown, start conservatively and use documentation or support evidence to narrow it. Keep request spacing and maximum concurrency as separate settings: a small number of fast workers can still send a burst.
Interpret both forms of Retry-After
Retry-After can contain a non-negative integer number of seconds or an HTTP date. Treat a valid value as the earliest time for another attempt. A date requires comparing it with the current time; account for clock accuracy when scheduling.
Retry-After: 120In this illustrative response, defer the next attempt for at least 120 seconds after receipt. Do not reduce that delay to fit a shorter local backoff cap. If the wait exceeds the job's deadline, end the job as deferred or incomplete and make that state visible.
If the header is missing or invalid, choose a documented local fallback. An exponential delay with a small random addition can help spread retries from your own workers, but its parameters are a workload decision. It cannot reveal a server's undisclosed quota.
For example, a low-volume collection job might allow three total attempts and schedule its second attempt after a locally chosen delay. If a subsequent response asks for a longer wait, honor that wait or defer the job. Those numbers are example policy settings, not a recommended request rate for any website.
Put retries inside a job budget
A useful policy has an attempt limit, a deadline and a clear list of retryable outcomes. Count the first request as an attempt. Record whether the job finished, exhausted its budget, or stopped because a configuration problem needs attention.
Also check whether repeating the operation is safe. HTTP distinguishes idempotent methods from operations that may have additional effects when repeated. Do not automatically replay a transaction merely because a response was lost; establish the application's retry contract first.
For a read-oriented collector, preserve a stable work-item identifier and deduplicate accepted results. A worker crash between saving a record and acknowledging the queue should not turn one observation into two apparent price changes.
Keep the outcome states explicit. A pending retry is not an empty result. An exhausted job is not a zero count. A response that arrived after the deadline should not silently replace a newer accepted observation.
Measure whether the policy actually recovers work
Review failures by category, retry attempts per work item and the number of records accepted after retry. Look at queue age as well as request success. A queue can appear healthy while its oldest items remain stuck behind repeated authentication failures.
Use a small incident record: time, redacted endpoint identity, configuration revision, status, attempt number, next eligible attempt and final outcome. Keep full URLs out of ordinary logs when query strings can contain private information.
Then inspect a few recovered records. If retries mostly return consent pages or missing fields, transport recovery is hiding an extraction problem. Our guide to cost per usable page explains why accepted records belong in the denominator. For client configuration, use the Crawlhub integration guides.
