Agent Search Is More Than a Search API: An Evaluation Framework

Do not judge an agent-search system by counting its search engines. Judge whether it turns queries, routing, failures, evidence, and cost into a contract an agent can inspect.

Conventional web search is designed for a person. It returns links, and the person rewrites the query, opens pages, ignores ads, and decides whether the evidence is credible.

Agent search sits inside a loop. An agent may search repeatedly, revise a plan, call another tool, and pass the result into a later decision. If the search layer turns an upstream failure into “no results,” or returns pages of unattributed summaries, the error keeps moving through the task.

Agent search is therefore not just a search API wrapped in MCP. A more useful definition is: an interface that converts retrieval work into executable, bounded, traceable evidence.

How is agent search different from web search?

A complete agent-search loop has at least seven steps:

1. Rewrite a task goal as one or more queries.
2. Choose language, sources, and time constraints.
3. Call one or more retrieval sources within a budget.
4. Separate useful results, weak results, and upstream failures.
5. Decide whether the evidence is sufficient or another retrieval path is justified.
6. Pack sources, passages, and limitations into context the model can use.
7. Preserve an execution record so the caller knows what actually happened.

A search API usually focuses on step three. Most of the product value in agent search appears in the decisions and boundaries around it.

Seven dimensions for evaluating agent search

1. Is the query contract explicit?

Start with the input. Does the tool accept only a query string, or can it express language, domain, result count, time range, and evidence-size constraints?

More importantly, what happens when the system cannot enforce a filter? A multi-source router that cannot apply “last 24 hours” consistently should return an unsupported-filter error. It should not silently ignore the constraint and present old results as fresh evidence.

2. Does multi-source retrieval add independent evidence?

“Sixteen engines” is not a quality claim. Several adapters may depend on the same underlying index, and ten results may all repeat one article.

Inspect whether the response tells you:

• which sources were actually called;
• how many independent provider families contributed;
• whether deduplication works on URLs, titles, or meaning;
• why the router stopped searching.

The purpose of multi-source retrieval is not to put more links in the context window. It is to add independent, verifiable evidence with a bounded number of calls.

3. Is paid escalation controlled by policy?

Adding an API key should not automatically authorize paid traffic. A controllable agent-search layer separates free-only, free-first, quality-triggered escalation, and paid-first behavior.

That separation lets the agent or operator make an explicit trade-off between task value, quality threshold, and spend. It prevents cost from hiding inside retries and fallback behavior.

4. Are failures visible?

This is easy to overlook and critical for agent judgment.

The response should distinguish at least these cases:

• the query genuinely produced no matching result;
• a source timed out or rate-limited the request;
• credentials or permissions failed;
• a requested filter is unsupported;
• extraction succeeded, but the content does not support an answer.

If every case becomes an empty array, the agent cannot decide whether to rewrite the query, change providers, wait, or stop answering.

5. Does it return an evidence packet or an opaque answer?

For an agent that must continue reasoning, longer output is not automatically better. A useful evidence packet retains titles, URLs, relevant passages, source information, failure information, and execution limits.

Synthesis may help the reader, but it should not erase provenance. Full page text should not enter context by default either. Result count, full-result count, passage length, and total evidence size need separate controls.

6. Are budgets and stopping decisions observable?

Limiting the final response to ten results is not enough. Retrieval also needs boundaries for adapter attempts, elapsed time, admitted candidates, and evidence size.

Ask whether the router stops when evidence passes a quality gate or always fans out to every source. If one batch is insufficient, why does it escalate? Are those decisions present in response metadata?

7. Can a quality claim be reproduced?

Search results change with time, region, and network conditions. Any claim about higher quality, lower latency, or fewer tokens should identify:

• version, commit, and run date;
• query set and language mix;
• allowed sources;
• network environment, failures, and retry policy;
• tokenizer and aggregation method;
• retained raw evidence;
• what the experiment does not prove.

A frozen fixture can verify formatting and regression behavior. It cannot prove live search quality. A live run can describe one environment, but it is not a long-term SLA.

A practical agent-search checklist

If a product can answer these questions only with “we connect many engines,” it is still a provider aggregator, not a complete agent-search system.

Applying the framework to Agent Search MCP

I maintain Agent Search MCP, an open-source implementation built around these questions. It is not the only answer to the framework, but it is a useful inspectable sample.

As of August 7, 2026, the main-branch capability matrix lists 16 adapters: nine zero-key sources and seven optional API providers. The stable public distribution remains 3.2.0. Keep those facts separate: the main branch can move ahead, so an evaluation should pin the version it actually installs instead of assigning the latest README state to an older package.

The project exposes several boundaries that can be checked in source:

• free_only, free_first, quality_escalation, and paid_first separate free retrieval from permission to spend;
• default request budgets cap adapter attempts, elapsed time, and admitted raw results;
• EVIDENCE_BUDGET_CHARS separately caps query-relevant evidence across the response;
• meta.execution exposes routing decisions, while partialFailures preserves upstream failures;
• time_range returns UNSUPPORTED_FILTER because the general providers cannot enforce one shared recency contract;
• search tools are declared read-only and idempotent, with an exact-result cache and optional local persistence.

The repository’s frozen bilingual fixture reports 2311.0 average tokens for Normal output, 1655.8 for Compact, a 28.4% relative reduction, and 1607.5 for Compact+, a 30.4% relative reduction. This verifies formatting and evidence-packet behavior on fixed results. It does not prove higher live-search quality, the same savings for every query, or permanent upstream availability.

The search-quality pipeline also separates fixture regression, live capture, blinded review, and a publishable claim. Its current method requires at least 30 distinct queries, complete traces, two independent model-family reviews, and third-family adjudication before a public quality conclusion can become eligible. “The evaluation pipeline exists” is not the same as “quality has been proven.”

That is the part of agent search I believe is worth promoting: not an exaggerated promise to replace every search API for free, but a search path whose routing, spend, failures, and evidence limits remain visible.

How can you evaluate an agent-search tool in ten minutes?

Do not start with a leaderboard. Start with four tasks:

1. A common factual query: verify that results retain openable sources and that execution metadata explains the stop decision.
2. A Chinese-source query: check whether the system retrieves Chinese sources directly instead of translating an English result set.
3. A recency-constrained query: request a filter the product cannot guarantee and confirm that it rejects the constraint instead of pretending to support it.
4. A failure-and-budget query: restrict sources or evidence size, then inspect whether partial failures remain visible and the remaining output is usable.

Only then freeze a query set, record version, time, sources, and raw traces, and compare nDCG, Precision, Success, citation support, latency, and failure disclosure. One or two answers that look good are not an evaluation.

To inspect the implementation used in this article, start the default MCP path:

$ pnpm dlx -y agent-search-mcp

Then review the Agent Search MCP source, capability matrix, and benchmark method on GitHub. If your narrower question is whether it can serve as a free-first Tavily alternative, continue with the Tavily-alternative boundaries and install guide.

When should you choose a hosted search API instead?

A hosted API is usually a better fit when you need an explicit SLA, vendor support, a proprietary index, stable semantic retrieval, or no responsibility for runtimes, caches, logs, and upstream limits.

Self-hosted agent search offers control and inspectability, but you own deployment and failure handling. The choice does not have to be binary. A good router can begin with zero-key sources, then use a clearly authorized commercial provider when the free evidence misses a quality threshold.

Conclusion

The core of agent search is not “find more.” It is letting the agent know what was searched, why the router stopped, what failed, where the evidence came from, and how much budget the search consumed.

If you are adding web search to Claude Code, Codex, Cursor, or your own agent, evaluate the interface with the seven dimensions above before choosing a product. To begin with a runnable open-source sample, open the Agent Search MCP product page or inspect the implementation on GitHub.