TensorZero

Company

TensorZero is a New York-based startup building an open-source, high-performance gateway and data plane for production LLM applications. The team focuses on observability, experimentation, and evaluation for teams running LLMs at scale.

Product — TensorZero

TensorZero (github.com/tensorzero/tensorzero, Apache 2.0) ships a single Rust gateway plus a ClickHouse-backed data store. It unifies inference across OpenAI, Anthropic, Google, AWS Bedrock, Azure, Mistral, Together, Fireworks, and self-hosted models with built-in structured outputs, tool use, caching, retries, and fallbacks. Every call is logged with inputs, outputs, latency, tokens, and tool traces for offline analysis.

Differentiator

Most gateways (LiteLLM, Portkey, Helicone) are observability wrappers. TensorZero's pitch is a data-flywheel: every production inference becomes training data that can be used to fine-tune open models, run A/B experiments, and automatically optimize prompts / parameters. The gateway ships as a single static binary with sub-millisecond overhead at 10K+ req/s.

Positioning

TensorZero competes with LiteLLM / Portkey / Helicone / LangSmith on the LLM gateway + observability axis, with a stronger bent toward experimentation and eval (similar in spirit to Braintrust and LangSmith, but OSS and self-hostable).