LLM GATEWAY
PROJECT OVERVIEW LLM Gateway is a production-grade API gateway designed to manage, secure, and route large language model (LLM) traffic. It acts as a single, intelligent entry point for applications communicating with multiple AI providers, such as OpenAI and Anthropic. The project abstracts away the complexities of multi-provider LLM integrations, ensuring high availability, cost control, and seamless failover.
CORE ARCHITECTURE The system is built on a high-performance, three-tier architecture designed for reliability and speed: - Envoy Proxy: Serves as the primary data plane and traffic router. It handles public API endpoints, enforces rate limits (e.g., 100 requests per minute via a token bucket), and executes internal redirects for fallback scenarios. - Go External Processor Sidecar: A custom gRPC service written in Go. It acts as the control plane, inspecting request and response metadata, validating API keys, making dynamic routing decisions based on requested models, and performing on-the-fly payload translations. - Local Testing Environment: A custom mock server that simulates OpenAI and Anthropic endpoints, allowing for rigorous local testing and fault injection without incurring real-world API costs.
KEY FEATURES - Intelligent Model Routing: Automatically routes requests to the appropriate provider based on the requested model name (e.g., routing GPT-4 models to OpenAI and Claude models to Anthropic) and endpoint paths. - Transparent Failover: Detects 5xx server errors from primary AI providers and seamlessly redirects traffic to a fallback provider using Envoy internal redirects. This mechanism operates transparently to the client, prevents infinite retry loops, and minimizes application downtime. - Bidirectional Schema Translation: Dynamically converts request payloads between OpenAI and Anthropic formats. It automatically translates system prompts, token limits, and message arrays, allowing client applications to use a single, unified API format regardless of the backend provider. - Enterprise-Grade Security: Clients authenticate using virtual gateway keys, while actual provider API keys remain securely isolated on the server and are injected dynamically into upstream requests by the sidecar. - Financial Controls and Budgeting: Tracks real-time token usage and calculates request costs based on model-specific pricing maps. The gateway enforces virtual key spend budgets, automatically blocking traffic with custom 429 Too Many Requests errors if a client exceeds their allocated spend limit. - Comprehensive Observability: Outputs highly structured JSON logs for every routing decision, capturing critical metrics such as request IDs, chosen providers, fallback activation statuses, and end-to-end latencies.
TESTING AND RELIABILITY To guarantee production readiness, the gateway is validated by a robust CI/CD pipeline and an extensive suite of automated tests. This includes comprehensive Go unit tests and dozens of Playwright end-to-end tests that cover standard routing, malformed inputs, upstream failures, and heavy workload patterns. The local testing stack supports custom HTTP headers to purposefully inject connection drops, latency spikes, and corrupt bodies, ensuring the system fails gracefully under pressure.
TECHNICAL STACK - Languages: Go (Golang), JavaScript/TypeScript - Infrastructure: Envoy Proxy, Docker, Docker Compose - Protocols: gRPC, HTTP/REST - Testing: Playwright, Node.js