TensorZero

TensorZero is a Rust-based LLM gateway from TensorZero Inc. with integrated observability, experimentation, and fine-tuning workflows. Its architectural premise is that every inference is structured data — captured into ClickHouse with typed schemas — that can drive A/B testing, evaluation, dataset curation, and automated optimization over time.

What it does

TensorZero routes requests to LLM providers — OpenAI, Anthropic, Gemini, Azure, AWS Bedrock, Mistral, Together, Fireworks, OpenAI-compatible endpoints, and self-hosted inference servers — through an OpenAI-compatible API and native Python and TypeScript SDKs.

Each inference is captured with a typed schema for inputs, outputs, and metrics, stored in ClickHouse for high-volume retention. The experimentation framework runs A/B tests across prompts, models, and routing policies with statistical analysis of the results. Feedback collection connects user or system signals back to specific inferences for evaluation and fine-tuning inputs. An automated prompt optimization loop uses captured data to propose prompt or model changes.

Licensing

Apache 2.0. TensorZero Inc. offers commercial hosted services; the self-hosted product is functionally complete.

Deployment

Single Rust binary or Docker container plus ClickHouse for analytics storage.

Limitations

• No built-in virtual-key wallet with hard budget enforcement — governance focus is on experimentation and data capture rather than multi-tenant quotas.
• No MCP gateway — LLM inference only.
• ClickHouse dependency adds operational weight compared to Postgres-only gateways.
• Experimentation and fine-tuning tooling assumes ML-engineering familiarity — typed schemas, statistical tests, and optimization loops are more natural for ML-aware teams than for general application developers.
• Ecosystem integrations and community examples are smaller than LiteLLM's or Portkey's.