Build JSONL exports you can validate and resume

A collection job becomes easier to operate when its output can be inspected one record at a time. JSONL is a useful export format for that job. It does not validate your business fields, prevent duplicate records or decide when a batch is complete. Those are separate decisions your pipeline must make.

Use one complete JSON value per line

JSONL uses UTF-8 and one valid JSON value on each line. Objects are convenient for records, though the format also permits other JSON values. A blank line is not a valid record. End each record with a newline so later records can be appended without merging two values onto the same line.

Use compact serialization for each record. A description containing a newline should contain an escaped newline inside its JSON string, not a physical line break that splits the record. Pretty-printing each object over several lines defeats this framing.

{"record_id":"item-42:2026-11-10","price":"19.90","currency":"EUR"}
{"record_id":"item-43:2026-11-10","price":null,"currency":"EUR"}

These two rows are illustrative fixture data. A missing price is represented explicitly; it is not silently converted into a zero-price offer.

Write a record contract first

Choose required fields, types and allowed nulls before the first batch. For a price observation, specify the item identifier, market, currency, observation time and price representation. Store a schema version with the batch. A date in a sample identifier is not a substitute for a proper timestamp field in production.

Use a decimal string or an agreed integer minor-unit representation when your downstream system needs exact monetary values. State that choice in the contract and validate it. A JSON number alone does not tell a consumer which currency or unit you meant.

Serialize and reject invalid values

A serializer handles quoting and escaping more reliably than string concatenation. In Python, disable non-finite numeric output when you want strict JSON compatibility. The default encoder can otherwise emit special numeric values that strict JSON consumers reject.

import json

def encode_record(record):
    return json.dumps(
        record, ensure_ascii=False, allow_nan=False,
        separators=(",", ":")
    ) + "\n"

This helper only serializes a record. Validate its required fields before calling it. Quarantine rejected records with an error reason and a safe input identifier, rather than inserting malformed lines into the accepted export.

Checkpoint acknowledged work

Define a checkpoint around what the storage destination has acknowledged, not what the scraper merely attempted. A local write returning is not a universal durability guarantee. Choose the flush, persistence or transaction behavior your destination needs, and document what an interrupted write can leave behind.

After interruption, check the last complete record boundary before appending. Do not automatically discard the whole batch because the final line is truncated. Retain a separate record of pending items and retry them through an idempotent sink or a deliberate deduplication stage.

A deterministic record identifier helps identify a replay, but it does not make an append-only file deduplicate itself. Decide whether repeated observations are legitimate history or duplicate attempts. For changing prices, an item identifier alone is usually insufficient to distinguish those cases.

Verify the batch before releasing it

Parse every accepted line, validate the schema and reconcile accepted, rejected and pending counts. Keep the input manifest so that a downstream user can see whether the export covers the intended collection window. Include a checksum of the completed export in a separate batch manifest.

Run a small recovery exercise: interrupt the writer, leave a partial last line and replay an already accepted item. Your acceptance criteria should say what survives and how duplicates are handled. Compression and upload completion belong in that exercise too.

Configure collection through a client integration, but keep export validation independent of network success. An HTTP response is only the beginning of a usable record.