If you've plugged a few MCP servers into an agent lately, you might have noticed that tools are not exactly free. Tool definitions often get loaded into context, and that can add up quickly. Anthropic shared an example of a five-server setup where 58 tools took around 55K tokens before the conversation even started, and mentioned seeing tool definitions reach 134K tokens before optimization. Yikes.
Tokens are one thing, but it also seems like agents can get worse at picking the right tool as the list grows. The same post mentions wrong tool selection and incorrect parameters as the most common failures they see, especially when tools have similar names. Some research seems to point the same way: in their experiments, giving a model fewer, more relevant tools helped it pick the right one a lot more often.
Some of the proposed fixes seem to be heading a particular way: let agents discover capabilities on demand, and let them write code against a typed API instead of picking from a giant list of tools. I don't know about you, but that sounds a lot like something we already have 👀. A GraphQL schema is a typed, discoverable API surface, and a single operation can do the work of a handful of tool calls.
That doesn't make GraphQL free for agents either (good luck fitting a large schema in context), and as I mentioned in my last post, it's not a panacea. But I do think it's a pretty great foundation if we're deliberate about how agents build their operations. In this post, I want to go over the different ways agents can do that, from the cheapest to the most dynamic.
From Use Case to Operation
Approaches for agents to build GraphQL operations, from cheapest and least dynamic to most expensive and most dynamic: skills / local operations, persisted operations search / modification, operation build against schema, schema modification workflow, ad-hoc optimized endpoint end-to-end generation
What I'm proposing is some sort of layered approach to go from use case (what the agent wants to do / retrieve) to operation (a valid GraphQL operation against a schema). Similar to how we approach most things with LLMs, it tries the cheapest and fastest approaches first when the use case fits, only turning to more expensive or slower approaches when the use case is more dynamic in nature or requires "creativity" for the API call itself. We'll cover each approach in order.
Skills / Local Operations
Aside from actual scripts running GraphQL (let's call those the base of our layered approach), a relatively small set of local operations is most likely the cheapest and fastest way to retrieve GraphQL operations for an agent. A good description of the operation, along with its variable definitions and persisted query ID (rather than inlining large documents), seems like a great, compact way to represent use cases.
I see this approach as being primarily useful for simpler, repeatable workflows. However, it runs into limitations similar to those of other local memories and skills:
- They can be annoying to distribute/share with teammates.
- They can get out of date. Either the schema changes underneath them (deprecations, removed fields), or worse, the description drifts away from what the operation actually does. The agent trusts the description.
- They only cover the use cases someone thought of ahead of time. When nothing quite fits, the agent needs to realize it and move up a level, rather than force-fitting an operation that almost does the job.
And of course, the more of them we have, the more we're back to the problem from the intro: a long list of similar-looking options for the agent to pick from, eating up context. That's a good hint it's time to look at the next level.
Persisted Operation Index
Once we hit too many of these problems, there's an additional layer I seldom see used before having an LLM generate an operation against the schema: we can have it look for existing work rather than spend too much time building from scratch.
Many of us already leverage persisted operations in production. In fact, you may already have a large corpus of examples ready to be searched and used. A few things can make this work even better:
- We need to capture more metadata on persist. Let's not forget that GraphQL operations can have descriptions too!
"""
Fetches the active user profile along with their most recent posts
"""
query GetUserProfile($id: ID!) {
user(id: $id) {
id
name
posts(first: 5) {
title
}
}
}
- We need to store / index them a bit differently, so we can fetch the top k operations matching our use case (hybrid BM25 / embeddings?).
Partial Matches
Persisted queries still lock the agent into a particular query shape. This can be problematic in two ways:
- The query almost matches the use case but is underfetching.
- The query matches the use case but is overfetching.
The overfetching case is the easier one. An LLM can prune a query without needing complex tool calls. The underfetching case is harder and starts resembling what would be required to build an operation from the schema. Still, I'd be willing to bet we can do slightly better than generating an operation from scratch. The reason is that we start from valid paths, which in theory allows us to find the schema paths or members we're interested in with fewer tool calls / less compute.
The Gift that Keeps on Giving
There's something else quite powerful about this approach, which I covered in my last post too. This index is an ever-growing corpus of use cases. Operations generated from other layers can end up here, and the persist step serves as a great checkpoint for automated review, linting, human oversight, or even training some sort of model / refining the index.
Agentic GraphQL Operation Building
This layer is the one we generally think of when trying to solve this problem today (at least for cases where the schema is too large to fit into context windows, which is what I'm assuming throughout this post). We give the LLM the necessary tools so it can find what it needs and build a valid operation. There is a ton to be said about this layer, but let's try to cover the main points.
Assuming we can't load the schema into context, we need to find the relevant parts of the schema first, and then ask the agent to construct an operation against it. The fact that we can validate queries against the schema (including any custom lints / hints) is of great use to the agent.
There are various strategies for finding the relevant schema members, like walking the schema from the root types, finding relevant types/fields and backtracking, splitting the schema into various sub-domains, etc. It might be best to think of how you'd attempt to construct a query given enough time. Overall, to make a good set of tools for operation construction, I think we need to take care of a few key points. They end up sounding similar to the tools you would use to build an operation:
- Translating the use case into a schema search
- Searching for (and ranking) candidate schema members (usually types and fields)
- Finding paths from the root types to those candidates
- Transmitting paths and candidates to the agent in a token-efficient way (GraphQL SDL is pretty verbose)
- Providing ways for the agent to augment and/or combine paths (tool calls to get type info or merge trees)
- Constructing and validating the final operation (looping if necessary)
I find a workflow of this kind avoids many pitfalls seen in other approaches. For example, an approach walking from roots could fail to find relevant paths early on, needing to explore a lot of the schema (more tool calls, more tokens). Transmitting paths also avoids sending large parts of the schema that happen to come along with our "matches".
While writing this post and reviewing the offerings out there, I realized Apollo seems to use a very similar approach. Their approach pretty much covers all our goals. They also share some nice heuristics for the search and ranking step (types/fields closer to the roots score higher!). Highly recommended read if you are building something like this yourself.
There are various other heuristics I can think of that could greatly improve query generation, like:
- Using schema-usage statistics as a way to score how commonly used types/fields are
- Using performance data
- Indexing documentation and examples, not only the schema itself
Overall, this is the most complex but also the most powerful layer. Of course, this one can feed into other layers. After constructing an operation, agents can persist it along with metadata, making similar use cases faster and cheaper the next time.
LLM-Driven Schema Evolution
When an LLM can't find what it needs from an existing schema, we potentially get a great piece of feedback. This can be used to drive schema evolution. Of course, the devil's in the details. How do we avoid getting spammed with low-quality schema changes? Who actually implements, reviews, and deploys the changes? Something like a Schema Update Proposal workflow (many platforms have adopted this idea) would be a great place to do so.
Moving Up a Layer
Putting all of these layers together, we end up with something like this:
Decision flow for fulfilling an agent's data need: if a local script, operation or skill exists, use skills / local operations; otherwise, if a persisted operation fits or nearly fits, use persisted operations search / modification; otherwise, if the current schema can serve it efficiently, build an operation against the schema; if not, go through the schema modification workflow
It looks simple on paper, but that decision tree is one of the hard parts. For example, how does the LLM decide the schema search didn't give results that are good enough? How does it decide whether existing operations are "too far" from a good operation? Each layer probably needs some kind of fit evaluation before the agent commits to it. It could compare what the use case needs with what the operation actually selects, or ask a cheap model for a yes or no. Also, misses could get expensive. We'd need to be careful that our initial layers actually get good "cache hits".
Conclusion
Of course, as I mentioned in my last post, there's always the possibility that agents don't even need a schema and can just write their own, highly optimized client and server ad-hoc API implementations. I don't think we're quite there yet, but it's a possibility. Until then, I think a layered approach makes a ton of sense from an accuracy, performance, and cost perspective.
We've only scratched the surface of each of these "layers". I'm currently playing around with implementing something like this myself, and I'd love to actually benchmark the different layers against each other (tokens, latency, tool calls, and of course, whether the agent got it right). I'm curious to see if the numbers back up my intuition here, or if I'm completely wrong 😅. Expect a follow-up post on that!
Thanks for reading!