Devtoolwatch BoardBest LLM evals and observability tools, ranked by real usage
20 LLM evals and observability tools ranked by new GitHub stars and npm and PyPI download growth, the week of Oct 25–31, 2026. Updated every Monday.
| Rank | Tool | Stars | Downloads/wk | vs 4-wk avg |
|---|---|---|---|---|
| #1 | SkillSpectorSkillSpector is NVIDIA's security scanner that checks AI agent skills for vulnerabilities, malicious patterns, and supply-chain risks… | 23.7k | – | – |
| #2 | JudgevalJudgeval is an open-source Python SDK for tracing and evaluating LLM-powered agents with prompt-based judges and online monitoring. | 1.1k | 80.2k | +81% |
| #3 | A.I.G (AI-Infra-Guard)A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak… | 7.2k | – | – |
| #4 | LangfuseLangfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. | 36.8k | 9.8M | +9% |
| #5 | OpikOpik is an open-source platform from Comet for tracing, evaluating, and monitoring LLM applications, RAG systems, and AI agents. | 23.1k | 709.1k | +33% |
| #6 | promptfoopromptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, including prompts, agents, and RAG pipelines. | 26.7k | 1.2M | +16% |
| #7 | garakgarak is a command-line LLM vulnerability scanner from NVIDIA that probes models for jailbreaks, prompt injection, data leakage,… | 9.8k | 12.9k | +16% |
| #8 | MLflowMLflow is an open source AI engineering platform for tracing, evaluating, monitoring, and managing agents, LLM applications, and ML models. | 28.6k | 5.2M | +11% |
| #9 | DeepEvalDeepEval is an open-source Python framework for unit testing and evaluating LLM apps, agents, chatbots, voice agents, and RAG pipelines. | 19.2k | 769.5k | +4% |
| #10 | Arize PhoenixArize Phoenix is an open-source, self-hosted AI observability platform for tracing, evaluating, and troubleshooting LLM applications and… | 12.1k | 158.1k | +2% |
| #11 | LogfireLogfire is an observability platform from the Pydantic team for LLM apps, agents, and Python services, built on OpenTelemetry. | 4.5k | 4.6M | +16% |
| #12 | OpenLITOpenLIT is an open-source, self-hosted platform for tracing, evaluating, guarding, and improving AI agents and coding agents on… | 2.9k | 88.1k | +11% |
| #13 | LangSmith SDKLangSmith SDK is the Python and JavaScript client library for tracing, debugging, evaluating, and monitoring LLM applications on the… | 1.1k | 32.1M | +11% |
| #14 | GuardrailsAdding guardrails to large language models. | 7.6k | 33.3k | +32% |
| #15 | BraintrustBraintrust is a platform for evaluating and shipping AI products, with JavaScript/TypeScript SDKs for logging, tracing, evals, and a CLI. | 29 | 4.2M | +10% |
| #16 | Inspect AIInspect AI is an open-source Python framework from the UK AI Security Institute for writing and running large language model evaluations. | 3.1k | 727.6k | −13% |
| #17 | EvidentlyEvidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline.… | 8k | 196.7k | +5% |
| #18 | NeMo GuardrailsNeMo Guardrails is an open-source Python toolkit for adding programmable guardrails to LLM-based conversational applications. | 7.4k | 109.3k | −18% |
| #19 | RagasRagas is a Python evaluation framework for RAG and LLM applications, with LLM-based and traditional metrics plus test data generation. | 16.2k | 273k | −13% |
| #20 | LaminarLaminar - open-source observability platform purpose-built for AI agents. YC S24. | 3.5k | 892.3k | −37% |
FAQ
What is the fastest-growing of the LLM evals and observability tools right now?
SkillSpector ranked #1 among LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, with a momentum score of 94 out of 100.
Which of the LLM evals and observability tools has the most GitHub stars?
Of the LLM evals and observability tools ranked in the week of Oct 25–31, 2026, Langfuse has the most GitHub stars: 36.8k.
Which of the LLM evals and observability tools is downloaded the most?
LangSmith SDK's main package had the most downloads of the ranked LLM evals and observability tools: 32.1M in the week of Oct 25–31, 2026.
How does Devtoolwatch rank LLM evals and observability tools?
Every Monday the Board averages four percentile ranks: new GitHub stars from daily snapshots, new stars for the repo's size, added npm and PyPI downloads, and download growth beyond the market. Star spikes without forks, downloads or Hacker News to back them are held back.
Rankings come from Devtoolwatch's daily GitHub snapshots and the npm and PyPI download APIs. How the ranking works.