Five turns reveal what one tool call cannot.
Bright MCP is built around a simple boundary: the model makes decisions where judgment helps, while the MCP owns retries, polling, batching, and result shaping. The test measures whether that boundary still pays off across a complete conversation.
The test gave Bright MCP and the official Bright Data MCP the same five-turn tasks across Current Search, Known Pages, and Marketplace retrieval. Both sides used the same model and Bright Data account. A stronger model then judged the answers without knowing which MCP produced them.
Across fifteen matched conversations, Bright completed 14 workflows to the official MCP's 13, scored 7.59/10 in blind answer quality to 6.43, and won 9–4 in pairwise preference, with two ties.
Why test the conversation instead of the tool?
A single tool call mostly tells us whether a server can return data. Real agent work continues after that. The model has to choose a source, preserve useful evidence, recover when something breaks, and turn the result into an answer several turns later.
This is where the two MCPs make different bets. The official MCP is a broad Bright Data toolbox. It gives a knowledgeable caller direct product controls and can take a very short route when one mature tool already matches the task.
Bright MCP exposes a smaller set of outcomes: search, discover, read, extract, research, find a dataset, and run it. The model still chooses what evidence it needs and which source fits. Deterministic mechanics such as retries, polling, pagination, batching, and partial recovery remain below the tool boundary.
The model should spend context on judgment, not on replaying provider workflows.
That narrower surface matters only if it still finishes the job. The benchmark was designed to test exactly that.
The quality difference appeared across turns.
Bright led every aggregate quality dimension: task fulfillment, evidence grounding, information density, source quality, and actionability. Its clearest advantage came in Marketplace, where the agent had to discover a typed dataset before running it and then carry the result into the final answer.
The blind judge preferred Bright nine times and the official MCP four times. Marketplace finished 5–0 for Bright and Search finished 3–2. The result was not uniform, which is useful: Known Pages favored the official MCP 2–1, with two ties.
That Known Pages result fits the architecture. When the URL is already known and the job is simply to read it, the official MCP's direct scrape-and-clean path is already excellent. Bright's extra workflow structure adds less value there.
Neither architecture won every operational metric.
Successful Search was effectively tied, with the official MCP slightly faster. Known Pages tied. Bright was faster on Marketplace despite spending an extra call on dataset discovery.
The official MCP benefits when its broad surface lets the agent jump directly to the right provider operation. Bright accepts more internal machinery when that machinery can improve recovery, preserve typed transitions, or reduce what the model has to reason through.
Context shows the same architectural split. After Bright normalized readable pages and taught the agent to stop when search summaries were sufficient, its targeted Search rerun used 39% fewer tokens than its earlier baseline. Marketplace used 28,455 tokens per successful run versus 166,517 for the official MCP.
Known Pages was nearly even. Again, a simple direct reading task leaves less workflow overhead for either MCP to remove.
Tool-call count by itself is not the goal. Bright used fewer calls on Known Pages because it accepts several URLs in one typed batch. The official MCP used fewer calls on Marketplace because its larger, dataset-specific surface can jump straight to the operation. Bright deliberately spends one call discovering the catalog first so the model does not have to guess a provider tool.
What this changes about the MCP surface
The benchmark does not say every MCP should have fewer tools. It says the public surface should match the decisions an agent is actually qualified to make.
Choose the official MCP when broad Bright Data coverage and direct provider control matter most. Choose Bright MCP when a production agent needs a stable, caller-scoped path through several connected decisions.
Bright MCP does not try to rebuild Bright Data's control plane or hide every product behind one giant tool. Its direction is narrower: keep judgment visible, keep mechanics internal, and make sure the smaller surface still completes the workflow.
How the benchmark ran +
MCPJam HostRunner gave both MCPs the same five-turn prompts, Bright Data credential, and Claude Haiku 4.5 agent. Each MCP could use its own valid tool path because the surface design was part of the comparison.
Fifteen matched conversations covered Current Search, Known Pages, and Marketplace retrieval. Claude Sonnet 5 judged anonymized answers against their tool evidence for fulfillment, grounding, density, source quality, and actionability.
Read the evaluation design and results →