What exactly is AgentMesh Gateway?

1. The idea in one sentence

AgentMesh sits between an AI client and one or more AI providers. The client sends a request to AgentMesh; AgentMesh decides which provider can safely and meaningfully handle it, then forwards the request through the correct protocol adapter.

2. What problem does it solve?

Modern AI systems often need to work with several local or remote model providers. Those providers can differ in API format, supported models, tool calling, reasoning features, price, latency, reliability and quotas. If a client is tightly coupled to one vendor, switching becomes difficult. If a router blindly chooses the cheapest or fastest provider, it can select a provider that cannot preserve the request semantics.

AgentMesh separates these concerns. A client can use one gateway endpoint while provider-specific networking remains behind adapters. Routing is performed only after incompatibilities have been removed.

Typical use cases

• Give a coding agent one stable local endpoint while changing upstream providers independently.
• Prefer local models for ordinary text but reserve native Responses providers for semantics that cannot be translated safely.
• Fail over to a second provider when the first one fails before a streaming response is committed.
• Measure actual reported token usage and recent latency without confusing those measurements with hand-written routing hints.
• Replay routing policies offline for research without spending money on live API calls.

3. What happens to a request?

4. What are the main parts?

Protocol adapters

They understand client-facing request and response formats. The current project contains OpenAI Chat Completions-shaped, OpenAI Responses-shaped and Anthropic Messages-shaped ingress logic.

Protocol-neutral domain model

Core objects such as normalized messages, requests, responses and stream chunks allow the routing/service layers to avoid vendor-specific HTTP details.

Provider adapters

They own outbound network behavior. v0.3.0 includes a generic OpenAI-compatible adapter, an Anthropic-compatible adapter and a native Responses-compatible adapter.

Provider registry

The registry turns provider configuration into concrete adapter instances and provides the router with provider specifications.

Router

The router computes the feasible set and orders candidates using the selected production policy. Feasibility is a hard boundary, not just another score.

Runtime state store

Tracks failures, successful latencies, EWMA, recent latency samples, p50/p95 diagnostics, observed tokens/cost, circuit breaker state and local quota usage.

Gateway service

Coordinates provider attempts, fallback and the rule that streaming may switch providers only before the first output has been committed to the client.

Offline simulation layer

Replays deterministic traces with fresh policy state and no provider network calls. This is where adaptive research policies live in v0.3.0.

5. What interfaces does it expose?

The Responses surface is deliberately described as partial rather than “fully OpenAI compatible.” Text, custom function loops, native reasoning preservation and recognized native tool preservation are covered in tested scenarios, but the full upstream Responses platform is much larger.

The admin provider endpoint can expose effective capabilities and runtime evidence. When a gateway token is configured, /v1/* and /admin/providers require bearer authentication; health/readiness probes remain public for orchestration.

6. How does routing work?

AgentMesh separates eligibility from preference.

Hard eligibility gates

• Does the provider support the requested model?
• Is its circuit currently available?
• Can the request semantics be translated without silent loss?
• Does the provider declare the capabilities required by the request?
• Has a configured local request quota been exhausted?

Production ranking policies

• ordered: configured order/weight determines preference.
• latency: favors lower live latency evidence.
• cost: favors configured cost hint.
• quality: favors configured quality hint.
• balanced: combines normalized latency, cost and quality terms.

In v0.3.0, observed USD cost does not silently replace cost_hint, and p50/p95 do not silently replace the existing EWMA routing signal. Measurement and production policy are intentionally separate.

7. What can a provider describe?

A provider specification can define its name, adapter type, base URL, allowed models, secret environment variable, ordering weight, cost/latency/quality hints, explicit capabilities, token prices and an optional local request quota.

Capabilities currently cover text, tools, reasoning and native_responses_tools. If capabilities are explicitly supplied, they are authoritative. If omitted, adapter-derived defaults preserve backward compatibility.

Secrets are referenced by environment-variable name rather than embedded in repository configuration.

8. What happens when a provider fails?

Provider/network errors are normalized into a common error model. Network failures plus selected HTTP conditions such as timeout, conflict, too-early, rate limiting and server errors can be considered retryable. Most other 4xx errors are terminal.

The runtime tracks consecutive failures and can temporarily open a provider circuit. A later successful request resets the relevant failure state.

For streaming, there is a strict safety boundary: fallback is allowed only before the first chunk is committed. Once the client has begun receiving one provider's stream, AgentMesh does not silently splice a different provider into the same response.

9. What evidence does it track?

Latency

Successful attempts update an EWMA and a bounded recent-success window. Deterministic nearest-rank p50/p95 values are available for diagnostics and research.

Usage and cost

If an upstream provider reports exact input/output token counts and the provider configuration includes both input/output prices, AgentMesh calculates observed cost. Missing usage or missing price remains missing; it is not converted to zero and it is not estimated from prompt length.

Failures and circuits

Successes, failures, consecutive failures and the last safe error summary are tracked without exposing provider secrets.

Local quota

An optional fixed local request-attempt window counts outbound attempts, including failed attempts. When exhausted, the provider becomes ineligible until reset. This is a local control model, not a claim to reproduce a vendor's private RPM/TPM counters.

10. What does the research simulator do?

The simulator reads provider configuration and a JSONL trace of counterfactual outcomes. It never contacts provider URLs. Each policy is replayed from fresh deterministic state so comparisons are not contaminated by another policy's history.

It can export JSON or CSV, making it suitable for statistical analysis, plotting and reproducible experiments.

Static baselines

ordered, latency, cost, quality and balanced.

Adaptive research baselines

adaptive_balanced updates a constrained multi-objective score using evolving evidence. constrained_ucb is a contextual UCB-style experimental baseline with chosen-provider feedback and deterministic tie-breaking.

11. What can a researcher study with it?

• Cost-constrained latency minimization.
• Quality-constrained cost minimization.
• Quota-aware routing under bursty workloads.
• How capability constraints change the apparent “best” provider.
• Static policy versus contextual/adaptive policy behavior.
• The exploration–exploitation tradeoff under selected-feedback conditions.
• How circuit failures alter feasible provider sets over time.
• Sensitivity to price assumptions, quality profiles and latency distributions.
• Robustness when some measurements are missing instead of incorrectly treated as zero.

Quality profiles require explicit benchmark provenance fields such as benchmark ID/version, source, metric and sample count. That structure helps reproducibility, but it does not automatically certify that an external benchmark is scientifically good.

12. Security model

• No real API keys are committed.
• Provider keys are read from configured environment variables.
• Optional bearer authentication protects gateway and admin paths.
• Request IDs support traceability without requiring external telemetry.
• Provider error bodies are normalized to avoid leaking credentials or unsafe upstream details.
• No external telemetry is sent by default.

For non-loopback deployments, the project documentation recommends using the gateway token plus normal network controls such as firewalls, reverse proxies and TLS termination.

13. Reproducibility and research integrity

The default test and simulation paths need no paid API key. CI runs on Python 3.11, 3.12 and 3.13 with Ruff and pytest. Architectural decisions are recorded as ADRs. The repository also contains provenance rules that prohibit copying another gateway's source, tests, documentation, assets or internal structure.

Version v0.3.0 is archived on Zenodo with DOI 10.5281/zenodo.22069468, and the repository includes CITATION.cff.

14. What could be added next?

15. What does v0.3.0 not claim?

• It is not fully compatible with every OpenAI Responses feature.
• It is not a complete Codex, Claude Code, Cline or OpenCode implementation.
• It does not translate every native reasoning or built-in tool semantic across vendors.
• It does not yet normalize image/audio requests across protocols.
• It does not route by context-window capacity yet.
• Its local quota window is not the same thing as vendor billing or private rate-limit state.
• Observed cost is incomplete when upstream usage is unavailable.
• Adaptive/UCB policies are not active in production.
• No empirical superiority claim is made from the synthetic example traces.