Fresh web context for AI agents

An agent asked about a current price or a recent software change needs evidence that was retrieved for the task. Giving it a larger collection of pages does not automatically make the answer better. The useful input is a small set of relevant passages with sources that a person can check.

A practical workflow starts with search, opens pages only when necessary, and builds a compact evidence packet before asking for an answer. Each step should have a budget and a visible failure state.

Separate discovery, retrieval and answering

Search discovers candidate pages. Retrieval obtains their contents. Extraction selects the relevant text and metadata. The model then reasons over those inputs to produce an answer. Keeping these jobs separate makes it easier to find a failure.

An irrelevant result is a discovery problem. A blocked or empty page is a retrieval problem. A missing passage can be an extraction problem. A claim that contradicts a supplied passage is an answer-quality problem. Retrying every failure with the same tool can increase spend without changing the result.

For a public-web workflow, a SERP API belongs in discovery and a page-retrieval API belongs in retrieval. Your application still needs to define the record, validate it and decide whether the evidence is sufficient.

Decide what “current” means for the question

A release note, a product price and a legal policy have different useful lifetimes. Write the freshness requirement into the task. A question about today’s stock availability needs a different retrieval policy from a question about the definition of an HTTP status code.

Keep three dates separate: when your application retrieved the page, when the source says it was published or changed, and when the underlying event happened. A page fetched this morning can describe an event from last year. A modified timestamp can reflect an editorial correction rather than a new product release.

Prefer the original source for facts about a product or standard, and keep the version or date scope with the passage. If two credible sources disagree, preserve the disagreement instead of silently choosing the more recent-looking page.

Start with search and a read budget

For a bounded factual task, begin with a small number of searches and a maximum number of page reads. The example in this article allows two of each. Those are application choices, not universal limits. Broader research can require a larger budget.

Search results help choose the pages worth reading. A snippet may be enough to locate an official release announcement, but it can omit the conditions that change the answer. For an exact price, policy, benchmark or version requirement, open the underlying source and verify the relevant passage.

Choose passages against the question, not a fixed desire to fill the context window. For a software requirement, retain the product name, version, qualification and relevant source date. Navigation text and unrelated examples can stay out of the packet. Measure context usage with the model you actually run rather than assuming a fixed relationship between page length and token count.

If the first search is off-topic, refine the query once. If a page cannot be retrieved, record the failure and use another appropriate source within the budget. Avoid recursively following every link; decide what evidence the task actually requires.

Give each source an inspectable record

A source entry should contain more than text. Keep a stable local ID, the original source URL, the page title, a UTC retrieval time, any source-published date you could verify, and a passage relevant to the question. Preserve raw retrievals separately when your retention rules permit it.

{
  "source_id": "S1",
  "source_url": "https://support.google.com/webmasters/answer/7576553?hl=en",
  "title": "Performance report overview",
  "retrieved_at": "2026-10-23T09:00:00Z",
  "source_date": null,
  "excerpt": "The report offers average position as a metric.",
  "content_role": "untrusted_source_text"
}

This is a hand-authored schema example, not a fetched response. A missing source date stays null; the retrieval time must come from the actual request. The excerpt is a short paraphrase of Google’s report documentation.

Do not turn an extraction timestamp into a publication date. Do not attach one source’s URL to another source’s passage. Deduplicate equivalent sources before they consume the budget, and record how text was shortened so a later reviewer understands what the model did and did not see.

Build the packet before calling the model

The downloadable packet builder reads saved source records from a local JSON file. It validates required metadata, accepts only well-formed HTTPS source URLs, checks timezone-aware retrieval timestamps, rejects duplicate source IDs, and limits both source count and excerpt characters.

It does not search the web or contact an LLM. That separation lets you inspect the evidence-shaping step without API credentials or inference charges. Connect your own search and retrieval adapters after this contract is clear.

The character budget is not a token count. Tokenization varies by model and text, so measure token use with the tokenizer or usage reporting appropriate to the model you actually run. Keep enough surrounding context to preserve qualifications, units and attribution.

A mechanical truncation can cut the sentence that changes a conclusion. The builder marks shortened passages. Before using the packet for a consequential answer, inspect the excerpt and retrieve a relevant section when the required qualification is missing.

python research_packet.py sources.json --max-sources 2 --max-chars 600

Treat retrieved instructions as source content

A page can contain text that tells an agent to change its instructions, disclose data or call another tool. OWASP describes this as indirect prompt injection and recommends layered controls, including clear separation of instructions and untrusted content, restricted permissions and output validation.

The content_role field makes the intended boundary explicit; it is not a security filter. Keep the application’s research instructions outside the source packet. Give a read-only research worker only the permissions the job needs, and apply independent rules to tools that can send messages, change files or spend money.

For retrieval services, validate destinations at the network boundary. Restrict schemes and destinations, account for redirects, and prevent requests to private infrastructure. A URL inside a source document is not permission to fetch it. A research packet is a record of evidence, not an authority to take actions.

MCP can expose the tools; the research policy stays yours

MCP provides a protocol through which an AI application can discover and use capabilities exposed by servers. Its architecture does not prescribe how the application manages the model’s context. Search limits, evidence selection and citation checking still need an application policy.

The same search-and-read contract can be implemented with ordinary API adapters or exposed through a tool protocol. Choose the interface that fits the application and the integrations you actually maintain. The workflow here does not depend on a Crawlhub MCP server.

Evaluate the answer against the source packet

A citation marker proves only that the model produced a marker. Check whether its source exists in the packet and whether the cited passage supports the associated claim. For dates, quantities and version requirements, compare the exact value and its conditions.

Log search calls, page reads, accepted sources, answer latency and model usage alongside quality outcomes. That lets you distinguish a cheaper successful answer from a cheaper unsupported one.

Start with one narrow research task and inspect its packets manually. Once the evidence remains relevant, traceable and within budget, expand the workload. If the input search data itself looks suspicious, use the SERP validation checklist before treating it as evidence.