Just-In-Time Tool Discovery: Escape the AI Context Window Trap

Enterprise agents do not usually run out of context because a user asks an unusually complicated question. They run out because the runtime front-loads everything the agent might ever need: dozens of connector descriptions, lengthy JSON schemas, system guidance, and conversation history. The result is a context window crowded with possibilities before the actual work begins.

Just-in-time tool discovery changes the sequencing. Instead of attaching every integration to every model call, the agent first searches a compact tool catalog, inspects only the matching tool’s exact schema, and executes it through the governed runtime. Koios implements this approach as a three-step flow—list-tools, inspect-tool, and execute-tool—within its Enterprise AI Harness.

Why the context window trap is an architectural problem

A tool-enabled agent needs enough information to choose and call the right capability. A naive architecture solves this by placing every tool schema in the prompt on every turn. For a company with many connected systems, the aggregate is substantial: each schema includes parameter names, types, requirements, and usage constraints. Those details are useful only when the agent actually needs that tool.

Imagine a task to look up a customer’s open case. If the runtime sends schemas for CRM updates, file search, billing, ticket creation, calendar scheduling, and dozens of unrelated operations, most of the input is irrelevant. The model pays the context and processing cost regardless of whether it calls those tools. On multi-step workflows, repeated payloads compound.

Large context capacity does not make irrelevant context free. A bloated prompt increases input-token consumption, can add latency, and makes it harder to keep instructions and the current task salient. The operational objective is not to maximize how much the model can see; it is to provide the right information at the moment it is needed.

How three-step Just-In-Time Tool Discovery works

1. Discover candidates with list-tools

The agent searches a compact catalog using the task’s intent and domain vocabulary. For example, a request to publish an article can identify WordPress content tools; a request to notify a stakeholder can find an email sender. Catalog matching narrows the search without injecting every connector schema into the model context.

2. Inspect the exact schema only when needed

After choosing a candidate, the agent calls inspect-tool to obtain its validated input schema and any relevant usage constraints. Required fields, types, allowed values, and defaults are now available for that specific operation. This reduces parameter guessing and avoids carrying unrelated definitions through the rest of the workflow.

3. Execute through the governed runtime

The agent supplies an input object to execute-tool. Koios routes tool execution through the appropriate platform, organization, or workspace layer. The agent can use a connected tool without embedding the connector’s entire specification in every prior turn. Tool calls remain explicit steps rather than unrestricted model-generated side effects.

Task: publish a new company article

1. list-tools(search="WordPress create post")
   → org:wp_create_post
2. inspect-tool(tool="org:wp_create_post")
   → required title; optional content, status, slug, categories...
3. execute-tool(tool="org:wp_create_post", input={...})
   → post ID, status, and link

This pattern is lazy evaluation for tools: discover first, load detail second, act third. The distinction is important. Discovery is not execution, and inspecting a schema is not permission to take an action. The explicit sequence gives the runtime and operator clear points to apply policy and validation.

Why tool discovery improves more than token economics

More task-relevant context

When the prompt carries fewer unused definitions, task instructions, user-provided facts, and relevant tool results occupy a larger share of the available context. This does not eliminate model error, but it reduces a predictable source of noise and context pressure.

Lower repeated overhead

In workflows that take several model turns, repeating a catalogue of large schemas can create substantial avoidable input. Retrieving a compact match and inspecting the needed operation only as the workflow reaches that step can cut this overhead. Actual savings depend on the number and size of schemas, model, caching behavior, and number of calls; teams should measure their own baseline rather than assume a universal percentage.

Better schema accuracy

On-demand inspection gives the model the actual contract for the chosen tool at call time. That is safer than relying on memory or an incomplete description: the precise field names and accepted values can be passed directly into a proposed call.

Clearer operational boundaries

A three-step process creates an observable boundary between finding a capability, reading its contract, and invoking it. Teams can reason about tool access and execution separately, and can align review, authorization, and audit practices to the action being performed.

Design principle: Tool availability should be discoverable; tool execution should be deliberate.

JIT discovery in an Enterprise AI Harness

Tool discovery is most effective as part of a broader execution architecture, not as a standalone prompt trick. Koios positions the Enterprise AI Harness as the layer that channels model reasoning into controlled operations. In that model, Just-In-Time Tool Discovery manages when schemas enter context, while the platform’s execution boundaries manage how a selected integration is called.

User request
   ↓
Agent identifies required capability
   ↓
list-tools: find candidate tools
   ↓
inspect-tool: retrieve exact schema
   ↓
Policy-aware execution via execute-tool
   ↓
Result returned to agent for next step

The three calls are not a claim that every tool action is automatically safe. A production deployment still needs identity and authorization controls, tenant boundaries, input validation, rate limits, safe handling of secrets, and appropriate audit trails. A well-designed harness makes these controls part of the execution path rather than asking a prompt to substitute for them.

Implementation checklist for AI platform teams

1. Keep catalog descriptions concise. Include enough vocabulary for reliable discovery, but reserve detailed parameter contracts for inspection.
2. Make schemas authoritative. Return exact types, required fields, allowed enums, and constraints from the tool definition rather than duplicating them in prose.
3. Separate read and write actions. Treat a lookup differently from an operation that publishes, sends, updates, or deletes.
4. Scope the available catalog. Discovery should reflect what the current organization and user are allowed to access.
5. Measure token and outcome metrics. Record tool-schema tokens per model call, task completion rate, invalid-argument errors, latency, and retry frequency.
6. Test ambiguous requests. Check that synonyms and natural-language intents find the right capability without causing premature execution.
7. Plan for failures. Handle unavailable tools, schema changes, authorization denials, timeouts, and third-party rate limits explicitly.

What to measure in a pilot

Compare a representative workflow using all schemas upfront with one using JIT discovery. Track the average input tokens attributable to tool definitions, total task tokens, schema-inspection calls, successful tool calls, malformed arguments, end-to-end latency, and completion quality. Separate model caching discounts from actual reductions in prompt size so that cost attribution is credible.

Also examine operational quality: Did the right tool appear in discovery results? Did the inspected schema match the executed operation? Were permissions enforced at execution time? Could an operator determine why a call was made? These questions establish whether a smaller prompt also produces a more governable system.

Common mistakes to avoid

• Making discovery descriptions too vague: a lean catalog is useful only if it can retrieve the relevant tool.
• Skipping schema inspection: the agent may hallucinate a parameter contract or rely on stale instructions.
• Treating discovery as authorization: a tool being discoverable must not bypass access checks when executing it.
• Optimizing tokens at the expense of observability: capture enough structured trace data to debug failures without indiscriminately logging sensitive content.
• Assuming a fixed savings percentage: benchmark real prompts and usage patterns before projecting ROI.

Conclusion: give agents capability on demand

Enterprise agents need access to many systems, but a model does not need every system’s full contract in every turn. Just-In-Time Tool Discovery replaces upfront schema loading with a deliberate progression: search the catalog, inspect the selected schema, and execute through the platform’s governed path.

For AI architects, this is a practical way to reduce avoidable context overhead while improving the clarity of tool calls. Koios applies the pattern as part of its Enterprise AI Harness: discover the capability when a task needs it, then expose the exact contract and invoke it through the appropriate execution layer.

Further reading: Model Context Protocol: Tools · OpenAI: Function calling · AI Agent Observability: How to Know Your Automation Is Working