Repository navigation
Client example proposal: host-side bounded retrieval over large tools/list catalogs #3617
Closed as not planned
Description
The current SDK docs make an important boundary explicit: servers should list tools normally and filtering is the host's job. That matches a validation experiment I am running with SchemaRouter, a typed capability/field retrieval layer.
I would like to contribute only a client-side example/benchmark, not a protocol or server change:
- fetch a deliberately large
tools/listcatalog with the normal Python SDK client; - build/update a typed local capability index from those tool schemas;
- for each query retrieve a bounded relevant subset for host/model exposure;
- keep MCP sessions, notifications, tool calls, and server semantics unchanged;
- when
ToolsListChangedarrives, refetch and refresh the index rather than assuming the delta.
The comparison would report required-tool recall, exposed schema/context size, unsupported-query rejection, and refresh behavior after tool-list changes.
Would this be appropriate as a client example/story in this repository, or should host-side tool selection examples live outside the SDK? I would wait for maintainer guidance before opening code.
Activity
Ben-Home commentedon Oct 2, 2026
This is a well-scoped proposal and the answer to your maintainer question is probably "client example, yes", but there is one design decision worth settling before you write code, because it determines whether the benchmark is meaningful.
The stated boundary (servers list tools normally, filtering is the host's job) means your example should not touch the server at all. That is the right instinct. The thing to be careful about is the incrementality: you plan to refetch and rebuild the index on
ToolsListChanged. For a large catalog, a full rebuild on every notification is the wrong default and will make your benchmark look worse than the approach deserves. Worth handling two cases explicitly: incremental upsert/delete when the notification carries enough information, and a full refetch as the fallback when it does not. Reporting which path a given refresh took is itself a useful benchmark dimension.Second, a bounded subset needs a stated selection rule you can defend comparing against, or the recall numbers will not mean much. The natural baselines are (a) expose everything and (b) retrieve top-k with a fixed k. Reporting required-tool recall against at least those two gives the result context; a single absolute recall figure without a baseline is hard to read.
Third,
unsupported-query rejectionis the metric that will differentiate the design, more than recall. A retrieval layer that silently returns a plausible-but-wrong subset is worse than one that says "no tool matches," so make sure that metric is framed as a correctness property and not just a rate.The refetch-on-change behavior also interacts with sessions: if an index update lands mid-session, decide whether in-flight tool calls see the old or new index and state that explicitly. Silent mid-session swaps are a common source of hard-to-trace failures.
We built the same shape of thing for a large connector catalog behind a single MCP surface; the two decisions that mattered were incremental index updates and an explicit "no match" path. Notes at https://corpusiq.io.
maxisbey commentedon Oct 6, 2026
Thanks for asking before writing any code. This one is better kept outside the SDK, so I'm going to close it as not planned.
We'd like the examples in this repo to run on the SDK alone, with no third-party package, and to show how to use the SDK rather than how a host picks tools. As you say, tool selection is the host's job, so a benchmark for it fits better in your own repository.
AI Disclaimer
Generated by Claude Code