- ToolBench, built by Arcade.dev, is a public benchmark that grades MCP servers on a scale from A+ to F based on definition quality, protocol compliance, security, and supportability.
- The benchmark evaluates how well enterprise applications expose their functionality to AI agents through the Model Context Protocol (MCP), the standard created by Anthropic and now managed by the Linux Foundation.
- Definition quality, which measures how clearly tools are named, described, and structured for AI consumption, accounts for 50 percent of the score for local MCP servers.
- Slack, Salesforce, Notion, and Google Workspace are among the most widely adopted MCP integrations, though individual quality grades vary significantly.
What Happened
Arcade.dev launched ToolBench, a public benchmark that scores and grades MCP servers, the standardized interfaces that let AI agents interact with enterprise software. The platform assigns letter grades from A+ to F based on weighted evaluation criteria and lets developers compare servers side by side before integrating them into their agent workflows.
Alex Salazar, CEO and co-founder of Arcade.dev, previously co-founded Stormpath, the first authentication API for developers, which was acquired by Okta. He co-founded Arcade with Sam Partee in 2024 to build what the company describes as “the MCP runtime for teams deploying multi-user AI agents with secure authorization, high-accuracy tools, and centralized governance.” Arcade.dev was selected for Wing Venture Capital’s 2026 Enterprise Tech 30 list, which tracks the most promising private enterprise technology companies.
Why It Matters
The Model Context Protocol, originally created by Anthropic in November 2024 and donated to the Linux Foundation’s Agentic AI Foundation in December 2025, has become the standard way large language models connect to external tools and data sources. Claude alone now has more than 75 MCP connectors. But not all MCP implementations are equal. A poorly described tool confuses AI agents. A server without proper authentication creates security risks in enterprise environments.
According to Arcade.dev’s 2026 State of AI Agents report, 46 percent of respondents cited integration with existing systems as their primary challenge when deploying AI agents. Another 57 percent reported using multi-step workflows that chain actions across multiple enterprise tools. ToolBench addresses these problems by giving developers an objective way to evaluate which enterprise integrations will actually work reliably with their agents before committing to an implementation.
Technical Details
ToolBench uses two distinct scoring models depending on server architecture. Local MCP servers, which have public source code on GitHub, are scored on definition quality (50 percent), protocol compliance (20 percent), and supportability (30 percent). Remote MCP servers, which run as hosted endpoints without public source code, are scored on protocol compliance (40 percent), security checks (30 percent), and supportability (30 percent).
Definition quality evaluates tool naming clarity, description comprehensiveness, parameter schemas, and composability. These factors directly determine whether an AI agent can understand and correctly invoke a tool without human intervention. Protocol compliance measures adherence to the MCP specification, with HTTP servers eligible for full marks and STDIO-only servers capped at 50 points due to limited remote access capabilities.
Security checks for remote servers cover OAuth 2.0 implementation, PKCE (Proof Key for Code Exchange), transport security, and authentication flow correctness. Supportability assesses maintenance health through GitHub activity metrics, licensing terms, documentation completeness, and whether the server has commercial backing or dedicated maintainers.
The grading scale runs from A+ (90-100) through A (80-89), B (70-79), C (60-69), D (50-59), to F (below 50). A weighted average of all applicable dimensions determines the final letter grade.
Who’s Affected
Enterprise software vendors now face public accountability for the quality of their AI agent interfaces. Companies like Slack, Salesforce, Notion, HubSpot, and Google, which already have widely used MCP servers, can use ToolBench scores as competitive differentiators or identify specific areas for improvement in their implementations.
AI agent developers benefit from a standardized way to choose between competing MCP integrations. Teams building multi-step agent workflows that span multiple tools need reliable interoperability between servers, and ToolBench provides a way to predict which combinations will work before writing integration code.
What’s Next
ToolBench includes a submission page for new MCP servers and an improvement guide for developers looking to raise their scores. The benchmark’s long-term impact will depend on adoption: if enterprise vendors start optimizing their MCP implementations to achieve higher grades, the overall quality of AI agent integrations should improve across the ecosystem. The main limitation is that ToolBench evaluates server quality at the protocol and documentation level, not real-world agent performance, which also depends on factors like latency, uptime, rate limits, and data freshness that the benchmark does not yet measure.
