What it does
Runs automated evaluations across multiple LLMs using text, RAG and voice inputs, scoring them on faithfulness, relevance and hallucination (plus latency and cost), to pick the best model for a job.
- Five modes:
text,rag,voice-stt,voice-tts,voice-e2e - Providers: OpenAI, Anthropic, MLX local, and custom OpenAI-compatible endpoints via a YAML model registry
- Multi-model comparison/ranking and baseline regression detection
- FastAPI endpoints:
POST /api/evals/run,GET /api/evals/compare,GET /api/regressions,GET /api/models
How it works
Dataset sample → Retrieve context (RAG) → Model provider → Metric suite → OTel trace span → PostgreSQL
- EvalRunner orchestrates runs; for each sample it optionally retrieves context, generates a response, and computes every metric.
- Text metrics: F1 faithfulness, token-overlap relevance, sentence-level NLI hallucination, latency, cost. Voice metrics: WER/CER, MOS/naturalness, voice latency.
- RAG pipeline: chunker → embeddings → cosine-similarity retriever → context builder, backed by ChromaDB.
- Tracing:
@trace_evalOTel spans exported via Kafka to a log pipeline; results stored in PostgreSQL.
Tech stack
Python 3.11+FastAPIMLXChromaDBPostgreSQLOpenTelemetryKafkaDocker ComposeWhisper