LLM and agent quality tool directory
Updated 2026-10-12
Tracing & production observability
Trace LLM calls and inspect prompts, outputs and runtime signals when an application behaves unexpectedly.
- Langfuse provides tracing, prompt management, evaluations and analytics dashboards for LLM applications.
Inspect agent execution and monitor operational signals across LLM applications built with different frameworks.
- LangSmith monitoring dashboards support token usage, latency, error rates, cost breakdowns and feedback scores.
Explore agent traces, annotate failures and create experiments in a locally run or self-hosted observability project.
- Phoenix supports tracing, output evaluation and experiments, and can run locally or be self-hosted.
Evaluation & regression tests
Build evaluation datasets and compare changes to prompts or models, with production traces available for investigation.
- Braintrust experiments can score outputs using LLMs, code or humans and compare prompts and models against datasets.
Red teaming & security testing
Exercise prompts, models and RAG pipelines with repeatable evaluations, then test attack scenarios against applications.
- Promptfoo offers prompt and RAG evaluations alongside red teaming that creates application-specific attacks.
Retrieval & RAG quality
Trace AI agents and examine retrieval or tool-use behavior through task-specific evaluation feedback.
- TruLens includes evaluation criteria for tool selection and plan adherence, plus RAG groundedness, context relevance and answer relevance.