Devtoolwatch BoardJudgeval review
Judgeval is an open-source Python SDK for tracing and evaluating LLM-powered agents with prompt-based judges and online monitoring.
TL;DR
Judgeval is an open-source Python SDK from Judgment Labs for agent improvement. It provides OpenTelemetry-based tracing and agent judge evaluation for LLM applications. Developers can use it to detect failures, triage root causes, and validate fixes against real production cases before shipping.
- 1.1kGitHub stars
- +2new stars, week of Oct 25
- 80.2kdownloads a week, +81% vs 4-week avg
- #2in Evals and observability this week
Judgeval at a glance
- Category
- Evals and observability
- Maintained by
- Judgment Labs (organization, @JudgmentLabs on GitHub)
- License
- Apache 2.0
- Language
- Python
- Repository
- github.com/JudgmentLabs/judgeval
- Website
- judgmentlabs.ai
- PyPI package
- judgeval 1.5.0
- Latest release
- v1.5.0 on GitHub, Oct 25, 2026
- First published
- Oct 25, 2024
What Judgeval does
- Judgeval traces any function with @Tracer.observe(), capturing inputs, outputs, and LLM token usage on top of OpenTelemetry.
- Judgeval agent judges are prompt-based scorers that apply binary or classification labels to traces at scale.
- Judgeval runs judges on live production traffic or replays them on historical traces to validate fixes.
- Judgeval scores live production traffic server-side with no latency impact and can send Slack alerts on regressions.
- Judgeval auto-instruments OpenAI, Anthropic, Google GenAI, and Together AI, with framework support for LangGraph and Claude Agent SDK.
- Judgeval offers read-only SQL queries against Judgment's virtual schema through Judgeval.sql() and schema discovery.
- Judgeval connects to MCP-compatible AI tools and a CLI to query traces, invoke judges, and review labels.
How it works
Judgeval is installed as a Python package. You set a Judgment API key and organization ID, call Tracer.init with a project name, wrap your LLM client, and decorate functions with @Tracer.observe(). Traces go to the Judgment platform, where judges label them. A CLI, an MCP server, and an HTTP SQL API expose the same trace and label data.
How to install and start using Judgeval
With pip
pip install judgevalRequirements
- Python 3.10 or later.
- A Judgment API key and organization ID.
First steps
- Install the SDK with pip install judgeval.
- Export JUDGMENT_API_KEY and JUDGMENT_ORG_ID as environment variables.
- Call Tracer.init(project_name="my-project") and wrap your client, for example wrap(OpenAI()).
- Decorate tool and agent functions with @Tracer.observe(span_type=...) and run your agent.
Who Judgeval is for
- Python developers running LLM agents in production who need tracing and failure detection.
- Teams that want to test agent fixes against real production traces before shipping.
- Teams already using OpenTelemetry-compatible observability stacks.
Limitations to know
- Judgeval requires Judgment credentials, so tracing and judging depend on the Judgment Labs platform.
- Judgeval SQL is read-only and limits results to 1,000 rows and 5 MiB; writes and multiple statements are unsupported.
- Judgeval SQL text cannot exceed 50,000 characters, and a public query rate limit applies.
- Judgeval is a Python SDK; the sources list no other language SDKs.
Is Judgeval growing? Usage data
In the week of Oct 25–31, 2026, Judgeval ranked #43 on the Devtoolwatch Board and #2 in Evals and observability, with a momentum score of 78 out of 100. Its main package had 80.2k downloads that week, +81% against its own 4-week average.
Show the weekly numbers
| Week of | New stars |
|---|---|
| Aug 9, 2026 | 1 |
| Aug 16, 2026 | 1 |
| Aug 23, 2026 | 0 |
| Aug 30, 2026 | 2 |
| Sep 6, 2026 | 0 |
| Sep 13, 2026 | 3 |
| Sep 20, 2026 | 1 |
| Sep 27, 2026 | 1 |
| Oct 4, 2026 | 1 |
| Oct 11, 2026 | 5 |
| Oct 18, 2026 | 4 |
| Oct 25, 2026 | 2 |
The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.
Judgeval alternatives
Other LLM evals and observability tools on the Board in the week of Oct 25–31, 2026, by momentum:
| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | SkillSpectorSkillSpector is NVIDIA's security scanner that checks AI agent skills for vulnerabilities, malicious patterns, and supply-chain risks… | 23.7k | – |
| #3 | A.I.G (AI-Infra-Guard)A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak… | 7.2k | – |
| #4 | LangfuseLangfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. | 36.8k | 9.8M |
| #5 | OpikOpik is an open-source platform from Comet for tracing, evaluating, and monitoring LLM applications, RAG systems, and AI agents. | 23.1k | 709.1k |
| #6 | promptfoopromptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, including prompts, agents, and RAG pipelines. | 26.7k | 1.2M |
| #7 | garakgarak is a command-line LLM vulnerability scanner from NVIDIA that probes models for jailbreaks, prompt injection, data leakage,… | 9.8k | 12.9k |
Recent Judgeval releases
Judgeval FAQ
Is Judgeval free and open source?
Yes. Judgeval's source code is on GitHub (JudgmentLabs/judgeval) under the Apache 2.0 license, so you can use it for free.
How do I install Judgeval?
Run pip install judgeval. The judgeval package needs Python 3.10 or newer.
Is Judgeval still maintained?
Yes. The JudgmentLabs/judgeval repository was last updated on Oct 25, 2026, and the latest release, v1.5.0 on GitHub, came out on Oct 25, 2026.
How popular is Judgeval?
Judgeval has 1.1k GitHub stars as of Nov 1, 2026. It gained 2 new stars in the week of Oct 25–31, 2026, against a 12-week average of 2 a week. Its main package had 80.2k downloads that week, +81% against its own 4-week average. That made it #2 among LLM evals and observability tools on the Devtoolwatch Board.
What are the best alternatives to Judgeval?
Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5), promptfoo (#6). The alternatives table above has their numbers.
What does Judgeval do?
Judgeval traces LLM-powered applications and evaluates those traces with agent judges. Teams use it to detect failures, understand root causes, and validate fixes against production cases before shipping.
Which LLM providers and frameworks does Judgeval support?
Judgeval auto-instruments OpenAI, Anthropic, Google GenAI, and Together AI. It also supports the LangGraph and Claude Agent SDK frameworks.
Does Judgeval's online monitoring add latency?
No. Judgeval scores live production traffic server-side with no latency impact, according to the README. Labels from judges surface as structured signals, and you can configure Slack alerts.
Is Judgeval compatible with OpenTelemetry?
Yes. Judgeval's tracing is built on OpenTelemetry for compatibility with existing observability stacks.
Can I query Judgeval trace data with SQL?
Yes. Judgeval offers read-only SQL through Judgeval.sql() or a POST endpoint, against a virtual schema. Results are limited to 1,000 rows and 5 MiB. Writes are rejected.
Can I use Judgeval from an AI assistant or IDE?
Yes. Judgeval has an MCP server that connects Judgment to MCP-compatible tools. From there you can query traces, invoke judges, browse labels, and surface failures.
Sources
Judgeval README badge
Maintainers can show the tool's weekly Board rank in their README:
[](https://index.devtoolwatch.com/tools/judgmentlabs-judgeval/)How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].