Tracely-ai

View on GitHub

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

★ 1.4K153 forksPythonMITJwuthri

Trace-native CI/CD for AI agents: ingests OTLP traces, grades runs with LLM judges, clusters failures, freezes bad runs into hermetic replayable regression cases, and blocks the PR that would ship them again at $0 replay cost. Self-hosted stack with MCP endpoint and alerts.

Use Cases

Ingest agent traces via OTLP and grade every run automaticallyCluster production failures into deduplicated issuesFreeze failing traces into hermetic regression casesReplay recorded fixtures in CI with no model spendBlock pull requests that reintroduce fixed failuresRun multi-turn and adversarial red-team scenarios against an agentCalibrate LLM-judge verdicts against human labelsAlert to Slack, email or webhooks on gate failuresDrive the workspace from a coding agent over MCPAnalyze trends in failure rate, latency and token cost

Built With

Language: Python
Frameworks: FastAPI · Next.js · ClickHouse · PostgreSQL · Redis · MinIO · OpenTelemetry · Docker Compose · LangGraph · OpenAI Agents SDK · MCP · Alembic

Tracely-ai

Use Cases

Built With

Tags