Catch silent scraper breakage with record contracts

A collector can return a successful HTTP response while quietly dropping the price, extracting the wrong identifier or repeating the first item on the page. Monitoring only request success leaves those failures hidden until somebody tries to use the export. A record contract gives you a separate decision: is this observation acceptable for the job it was collected to do?

Define the observation you need

Start with a concrete use case. A product-price observation might need an item identifier, the intended market, an observation timestamp, a currency and a price. A product description dataset might allow the price to be absent. Do not make one universal validator serve every collection task.

Write down what missing means for each field. A missing key, a null value, an empty string and a zero are different inputs. If a parser cannot find a price, inserting zero creates a misleading observation. Preserve the reason for rejection instead of disguising extraction failure as valid business data.

Check structure before meaning

A schema can enforce an object's required properties and property types. Defining a property does not automatically make it required: declare required fields explicitly. A present null value is different from an absent property, and it needs a type that permits null if you intend to accept it.

Additional fields are allowed by default in a typical object schema unless you constrain them. Decide whether an unfamiliar field should fail validation or be retained for review. Rejecting all additions can catch unexpected parser output, but it can also make a harmless extension break an older consumer. Version the contract and choose that tradeoff deliberately.

Structural validation cannot establish that a number came from the right product or that the requested currency matches the page. Those checks need context. Keep the item key, market and relevant extraction evidence associated with the record so a reviewer can investigate a suspicious value.

Keep missing-field outcomes explicit

This small Python helper demonstrates four distinct outcomes. It is an illustrative classification step, not a complete record validator.

def field_state(record, key):
    if key not in record:
        return "missing"
    value = record[key]
    if value is None:
        return "null"
    if isinstance(value, str) and not value.strip():
        return "blank"
    return "present"

A numeric zero returns present. Whether zero is acceptable belongs in the field's business rule. Likewise, a non-empty string is not automatically a valid price, timestamp or identifier. Add explicit validation for the representation your downstream system expects.

Watch the distribution of outcomes

Record accepted rows, rejected rows and missing-field counts separately from transport failures. If price completeness falls while request success stays steady, investigate extraction before replacing the network route. A changed page template, a consent screen and a genuinely unavailable offer can require different responses.

Compare like with like. Split checks by page type and market when they have different expected fields. A collector that suddenly discovers more category pages may lower aggregate price completeness even if every product-page parser still works. A single global percentage can hide that distinction.

Choose alert thresholds from your own observed workload and tolerance for incomplete data. Do not copy an arbitrary percentage into production and call it a quality guarantee. For a new collector, manually inspect a small sample of accepted and rejected records before trusting its baseline.

Test changes before replacing good output

Keep fixtures for a normal item, an unavailable item, a missing field and an unexpected page layout. Run them after changes to selectors, parsing logic or the record contract. Include an example that has valid structure but the wrong item identifier to exercise the contextual checks.

Publish accepted output and rejected output separately. Include rejection reasons and a batch manifest that explains which observations are still pending. Avoid exposing private page content or credentials in debugging artifacts. Your export format and recovery process should preserve those decisions rather than silently discard failed rows.

A working client connection gets you a response. The record contract decides whether the response produced something worth keeping.