What 100,000 usable pages really cost

A scraper can make every request successfully and still cost too much to run. It can also look cheap on a per-request basis while producing records that your team cannot use. Budget for the accepted output at the end of the pipeline.

For a job that needs 100,000 usable pages, the relevant questions are how much traffic each attempt carries, how often collection succeeds, how much of that output passes validation, and what the team spends maintaining the workflow. The unit price is one input to that calculation.

Define “usable” before estimating volume

A usable page is one that produces the record your application requires. For a product-price job, that might mean a stable product ID, amount, currency, stock status, requested market, retrieval time and source URL. A page that loads without a price does not meet that contract.

Write the acceptance rules before the first benchmark. Decide how you handle duplicate products, empty optional fields, regional redirects and stale data. Count a deduplicated, valid output once, even if several requests contributed to it.

This distinction makes the budget explainable. An increase in requests may reflect a wider catalog, more retries or a parser regression. Those are different problems with different fixes. A single “success rate” hides them.

Three common ways the bill grows

A successful-response charge is not automatically an accepted-record charge. If an API returns the HTML you requested but your parser finds no price, that may still be a billable response. Read the billing definition and the extraction contract together.

Monthly plans also have a floor. Unused included traffic, request allowances, overages and fixed IP allocations can matter more than an attractive marginal rate. Compare the invoice for the intended workload, including any minimum commitment.

Measure traffic at the boundary you pay for

A browser can load scripts, images, fonts and third-party resources beyond the HTML your parser needs. Playwright documents request interception and resource blocking, which can help you test what is necessary for the page to render.

Build a comparison around the same permitted target list and acceptance rules. Record the billed traffic for an HTTP client, a browser with its normal resource loads, and a browser with selected resources blocked. Keep the collection period and session policy explicit for each run.

A lighter response is useful only if it still produces the required fields. Check the extracted output after each change and record all retries. The worked calculation below uses editable traffic assumptions; replace them with measurements from your own collection before estimating a real bill.

Measure your own workload using the provider’s usage counters or a byte-counting boundary that you can reconcile with them. Browser resource timing can help explain page resources, but its transferSize has caching and cross-origin reporting rules; it is not a proxy invoice.

Turn acceptance rates into required attempts

Keep two rates separate. Let transport success be the share of attempts that return the intended page response. Let record validity be the share of those responses that produce an accepted record after extraction and deduplication.

The calculator uses decimal units: one KB is 1,000 bytes and one GB is 1,000,000,000 bytes. It rounds the required event count up to a whole request. Confirm whether your provider uses the same units.

This is an average planning model, not a promise of job completion. It assumes the rates remain representative as you scale. In a real run, some targets can become persistently unavailable and repeated attempts can be correlated. A hard attempt limit can prevent an expensive loop even when it means finishing with fewer records.

Work through 100,000 accepted records

The editable defaults below are illustrative assumptions, not measured performance or a price quote. Use them to see how the math behaves, then replace them with your own billed traffic, acceptance rates and contract prices.

The browser and HTTP examples share acceptance rates only to make their traffic assumptions easy to compare. That does not mean those approaches perform equally on your targets. If a browser is required to retrieve the fields at all, a cheaper HTTP estimate is irrelevant until that implementation can produce accepted records.

Add the costs the request bill misses

Variable collection charges are the starting point. Add browser or worker compute, storage, monitoring and engineering time. If your team spends four hours a month maintaining a collector and you assign that work an internal rate of $50 per hour, that is another $200. This is an arithmetic example; use your actual accounting rate.

Allocate engineering costs to the time period you are comparing. An eight-hour build can be amortized across several months if the workflow is expected to remain useful. A recurring four-hour maintenance burden belongs in each month. Keep one-time and recurring costs on separate lines.

For AI workflows, also count model input and output usage. Fetching ten full pages when two short, verified sources would answer the question changes a different bill. The search-first research workflow shows how to bound those page reads.

Run a representative test before buying volume

Choose a small set of permitted pages that reflects the real workload: static and rendered pages if both matter, multiple layouts, target regions and the fields your application requires. Keep the same acceptance rules across implementations.

1. Set request and retry limits before the run. Respect applicable access rules and rate limits.
2. Start and finish with recorded usage counters. Separate warm and fresh sessions if both occur in production.
3. Count all attempts and charged events, including failures.
4. Validate records independently of the transport outcome.
5. Record the wall-clock duration, compute used and manual repair time.
6. Repeat on another representative batch before extrapolating a monthly budget.

Reducing resources can also change correctness. Blocking a script that supplies the product price may save traffic while silently removing the value you need. Inspect the output after every optimization.

Choose a collection method with an explicit budget

Use the smallest reliable collection path for each target. A direct HTTP client can be economical when the needed data is present in the response. A browser can be justified when interaction or rendering is required. A managed API can reduce infrastructure work when its output contract fits the job.

Set limits on total attempts, monthly spend and the share of invalid records you are willing to accept. Review cost per accepted record alongside field completeness and freshness. A falling unit cost is only an improvement if the output still meets the contract.

Compare the billing basis for residential traffic and the Web Scraping API against your measured workload. Use the published plan’s actual allowance and commitment when you turn a variable estimate into a purchasing decision.