Envoy AI Gateway

Envoy AI Gateway is an Apache 2.0 project from the Envoy Proxy community that extends Envoy with LLM-aware traffic management. It builds on Envoy Gateway and the Kubernetes Gateway API to add AI routing, token-based metrics, JWT authentication, and MCP OAuth 2.1 support on top of Envoy's filter chain.

What it does

The gateway routes LLM traffic across OpenAI, Anthropic, AWS Bedrock, Azure, Vertex, and OpenAI-compatible upstreams, using CEL expressions for dynamic routing decisions based on request content, headers, or upstream health. Token-based usage metrics are emitted to Prometheus and OpenTelemetry with per-backend cost attribution.

JWT authentication runs through Envoy's built-in JWT filter with automatic JWKS refresh against any OIDC provider's discovery endpoint. MCP support covers the OAuth 2.1 authorization server flow and CEL-driven policy decisions. Request transformation uses Envoy's filter chain — request and response payloads can be rewritten, injected, or blocked inline.

Licensing

Apache 2.0 under the Envoy Proxy community governance, CNCF-adjacent.

Deployment

Kubernetes-native. Installed via Helm or raw manifests; the Envoy Gateway controller reconciles Gateway API resources into Envoy configuration. A standalone non-Kubernetes mode is available but earlier in stabilization than the Kubernetes path.

Limitations

• Kubernetes Gateway API is the primary deployment target; standalone Envoy deployments are supported but less documented.
• No built-in virtual-key wallet — token metrics are captured but budget enforcement requires wiring the gateway to an external rate-limit service.
• Configuration is YAML manifests via Gateway API resources, without an admin UI; this fits GitOps workflows and is heavier for interactive management.
• MCP support is newer than the LLM routing layer and is still stabilizing around transport and authorization edge cases.
• Pre-1.0 release series; expect configuration API changes between minor versions.