Tardi A robust, concurrent testing framework engineered specifically for evaluating agentic workflows, autonomous LLM scripts, and non-deterministic applications. Tardi bridges the gap between traditional software testing and AI evaluation by introducing a multi-tiered assertion pipeline and an LLM-as-a-judge architecture.
The Architecture Evaluating AI agents presents a unique challenge: traditional testing frameworks cannot reliably evaluate non-deterministic natural language outputs, while raw "LLM-as-a-judge" evaluation pipelines are prohibitively expensive and prone to hallucination at scale. Tardi solves this by implementing a strict, cost-saving evaluation gauntlet.
Rather than indiscriminately sending every agent execution trace to an evaluation model, Tardi enforces a series of deterministic constraints first. The pipeline evaluates execution through sequential checks: process stability (preventing crashes), temporal bounds (timeout prevention), regex pattern matching, and JSON schema validation. If an agent crashes, hangs in an infinite loop, or returns malformed outputs, Tardi fails the test immediately. The expensive LLM judge is only invoked for semantic validation once all deterministic criteria are met, drastically accelerating the feedback loop and minimizing API costs.
Key Infrastructure Features - Tiered Evaluation Engine: Catch process crashes, timeouts, and schema mismatches deterministically before triggering semantic validation. - Concurrency and Rate-Limiting: Executes test suites in parallel with intelligent chunking to maximize throughput without exceeding provider API rate limits. - Provider Agnostic Abstraction: Built on the standard Vercel AI SDK, allowing for hot-swappable evaluation models across OpenAI, Google, Anthropic, and local endpoints. - Interactive REPL and CLI: Features a zero-friction CLI environment for rapid test synthesis, execution, and debugging, natively powered by an onboard NLP intent parser. - Secure Credential Management: Bypasses plain-text environment variables by safely storing provider API keys directly in the native OS keychain during local development.
Technologies TypeScript, Vercel AI SDK, Node.js, CLI Instrumentation, OS Keychain Integration
