Repeat crawls without downloading every page again

A scheduled crawl should check what changed, without treating every URL as a new discovery. Conditional requests can avoid transferring an unchanged response body when the server supports validation. They still make a request, and they do not remove the need for rate limits or a reliable stored copy.

Keep a usable baseline

Start with an ordinary successful response. Save the body, its retrieval time and any validator returned with it. Keep the parsed records too, but do not confuse a parsed price with a reusable page representation. A later validation response may depend on the body you retained.

Define what makes two observations comparable. For a retail crawl, the same address can represent different markets, currencies or signed-in states. Store the language and other representation-selecting request settings with the baseline. Avoid mixing authenticated data into a public-page collection.

Use the validator the server supplies

An ETag is an opaque validator for a representation. Preserve it exactly, including quotes and any weak prefix. A later GET can send that value in If-None-Match. If the selected representation has not changed under the applicable comparison rules, a supporting server can return 304 instead of sending its body again.

Last-Modified can also support validation through If-Modified-Since. It has timestamp precision limits and may not be present. When both conditions are supplied, If-None-Match takes precedence. Do not manufacture either validator from your own collection timestamp.

GET /catalog/item-42 HTTP/1.1
Host: example.com
If-None-Match: "catalog-version-7"

This is an illustrative request, not a live endpoint or a Crawlhub connection format. Your client must retain the earlier response body and associate the validator with the correct request variant.

Handle 200 and 304 differently

A 200 response supplies a representation to inspect. Validate its content before replacing a useful baseline: an HTML error page can arrive with a successful status. A 304 has no response content. Reuse the appropriate stored representation and update its metadata as required; do not pass an empty response body to your product parser.

If the stored body has gone missing, a 304 cannot recreate it. Recover with an unconditional retrieval when appropriate. Keep that recovery bounded so a storage bug does not trigger a second request for every item in a large job.

Respect cache policy and variants

Vary identifies request fields that influence cache selection. Preserve the relevant values rather than sharing one cached body across incompatible requests. Also read Cache-Control: no-cache requires validation before reuse, while no-store tells a cache not to store the response. They have different meanings.

A changed proxy route does not prove that a page changed. An unchanged route does not prove that a price stayed the same. Track the intended market and the observed response separately. Validation is useful only when the current request and the saved representation belong together.

Measure useful work, not just smaller responses

For a pilot, count full responses, successful validations, missing-baseline recoveries and accepted changed records. Keep request volume and transferred bytes as separate measures. A smaller response is not evidence of lower server-side work or a guaranteed reduction in your bill.

Test a changed item, an unchanged item and a different language variant before widening the schedule. Include a missing-body recovery and a response without validators. If the server ignores conditional headers and returns 200, process that response normally instead of assuming your client has failed.

Use the integration guides to configure the client connection, then add caching behavior in the collector that owns the stored responses. Keep your failure and retry policy separate from change detection.