# Judgeval review

> Judgeval is an open-source Python SDK for tracing and evaluating LLM-powered agents with prompt-based judges and online monitoring.

Devtoolwatch Board, the week of Oct 25–31, 2026. Canonical page: https://index.devtoolwatch.com/tools/judgmentlabs-judgeval/

## TL;DR

Judgeval is an open-source Python SDK from Judgment Labs for agent improvement. It provides OpenTelemetry-based tracing and agent judge evaluation for LLM applications. Developers can use it to detect failures, triage root causes, and validate fixes against real production cases before shipping.

## Judgeval at a glance

- **Category:** [Evals and observability](https://index.devtoolwatch.com/categories/evals-and-observability/)
- **Maintained by:** Judgment Labs (organization, [@JudgmentLabs](https://github.com/JudgmentLabs) on GitHub)
- **License:** Apache 2.0
- **Language:** Python
- **Repository:** [github.com/JudgmentLabs/judgeval](https://github.com/JudgmentLabs/judgeval)
- **Website:** [judgmentlabs.ai](https://judgmentlabs.ai/)
- **PyPI package:** [judgeval](https://pypi.org/project/judgeval/) 1.5.0
- **Latest release:** [v1.5.0](https://github.com/JudgmentLabs/judgeval/releases/tag/v1.5.0) on GitHub, Oct 25, 2026
- **First published:** Oct 25, 2024

## What Judgeval does

- Judgeval traces any function with @Tracer.observe(), capturing inputs, outputs, and LLM token usage on top of OpenTelemetry.
- Judgeval agent judges are prompt-based scorers that apply binary or classification labels to traces at scale.
- Judgeval runs judges on live production traffic or replays them on historical traces to validate fixes.
- Judgeval scores live production traffic server-side with no latency impact and can send Slack alerts on regressions.
- Judgeval auto-instruments OpenAI, Anthropic, Google GenAI, and Together AI, with framework support for LangGraph and Claude Agent SDK.
- Judgeval offers read-only SQL queries against Judgment's virtual schema through Judgeval.sql() and schema discovery.
- Judgeval connects to MCP-compatible AI tools and a CLI to query traces, invoke judges, and review labels.

### How it works

Judgeval is installed as a Python package. You set a Judgment API key and organization ID, call Tracer.init with a project name, wrap your LLM client, and decorate functions with @Tracer.observe(). Traces go to the Judgment platform, where judges label them. A CLI, an MCP server, and an HTTP SQL API expose the same trace and label data.

## How to install Judgeval

**With pip**

```
pip install judgeval
```

### Requirements

- Python 3.10 or later.
- A Judgment API key and organization ID.

### First steps

1. Install the SDK with pip install judgeval.
2. Export JUDGMENT_API_KEY and JUDGMENT_ORG_ID as environment variables.
3. Call Tracer.init(project_name="my-project") and wrap your client, for example wrap(OpenAI()).
4. Decorate tool and agent functions with @Tracer.observe(span_type=...) and run your agent.

## Who Judgeval is for

- Python developers running LLM agents in production who need tracing and failure detection.
- Teams that want to test agent fixes against real production traces before shipping.
- Teams already using OpenTelemetry-compatible observability stacks.

### Limitations

- Judgeval requires Judgment credentials, so tracing and judging depend on the Judgment Labs platform.
- Judgeval SQL is read-only and limits results to 1,000 rows and 5 MiB; writes and multiple statements are unsupported.
- Judgeval SQL text cannot exceed 50,000 characters, and a public query rate limit applies.
- Judgeval is a Python SDK; the sources list no other language SDKs.

## Usage data (the week of Oct 25–31, 2026)

- Board rank: #43 overall, #2 in Evals and observability, momentum score 78
- Downloads that week: 80.2k (+81% vs 4-week average)
- New GitHub stars per week (last 12 weeks): Aug 9: 1; Aug 16: 1; Aug 23: 0; Aug 30: 2; Sep 6: 0; Sep 13: 3; Sep 20: 1; Sep 27: 1; Oct 4: 1; Oct 11: 5; Oct 18: 4; Oct 25: 2

## Judgeval alternatives

| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | [SkillSpector](https://index.devtoolwatch.com/tools/nvidia-skillspector/) | 23.7k | – |
| #3 | [A.I.G (AI-Infra-Guard)](https://index.devtoolwatch.com/tools/tencent-ai-infra-guard/) | 7.2k | – |
| #4 | [Langfuse](https://index.devtoolwatch.com/tools/langfuse/) | 36.8k | 9.8M |
| #5 | [Opik](https://index.devtoolwatch.com/tools/opik/) | 23.1k | 709.1k |
| #6 | [promptfoo](https://index.devtoolwatch.com/tools/promptfoo/) | 26.7k | 1.2M |
| #7 | [garak](https://index.devtoolwatch.com/tools/garak/) | 9.8k | 12.9k |

## Judgeval FAQ

### Is Judgeval free and open source?

Yes. Judgeval's source code is on GitHub (JudgmentLabs/judgeval) under the Apache 2.0 license, so you can use it for free.

### How do I install Judgeval?

Run `pip install judgeval`. The judgeval package needs Python 3.10 or newer.

### Is Judgeval still maintained?

Yes. The JudgmentLabs/judgeval repository was last updated on Oct 25, 2026, and the latest release, v1.5.0 on GitHub, came out on Oct 25, 2026.

### How popular is Judgeval?

Judgeval has 1.1k GitHub stars as of Nov 1, 2026. It gained 2 new stars in the week of Oct 25–31, 2026, against a 12-week average of 2 a week. Its main package had 80.2k downloads that week, +81% against its own 4-week average. That made it #2 among LLM evals and observability tools on the Devtoolwatch Board.

### What are the best alternatives to Judgeval?

Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5), promptfoo (#6). The alternatives table above has their numbers.

### What does Judgeval do?

Judgeval traces LLM-powered applications and evaluates those traces with agent judges. Teams use it to detect failures, understand root causes, and validate fixes against production cases before shipping.

### Which LLM providers and frameworks does Judgeval support?

Judgeval auto-instruments OpenAI, Anthropic, Google GenAI, and Together AI. It also supports the LangGraph and Claude Agent SDK frameworks.

### Does Judgeval's online monitoring add latency?

No. Judgeval scores live production traffic server-side with no latency impact, according to the README. Labels from judges surface as structured signals, and you can configure Slack alerts.

### Is Judgeval compatible with OpenTelemetry?

Yes. Judgeval's tracing is built on OpenTelemetry for compatibility with existing observability stacks.

### Can I query Judgeval trace data with SQL?

Yes. Judgeval offers read-only SQL through Judgeval.sql() or a POST endpoint, against a virtual schema. Results are limited to 1,000 rows and 5 MiB. Writes are rejected.

### Can I use Judgeval from an AI assistant or IDE?

Yes. Judgeval has an MCP server that connects Judgment to MCP-compatible tools. From there you can query traces, invoke judges, browse labels, and surface failures.

## Sources

1. [JudgmentLabs/judgeval on GitHub](https://github.com/JudgmentLabs/judgeval)
2. [Judgeval README](https://github.com/JudgmentLabs/judgeval/blob/main/README.md)
3. [judgeval on PyPI](https://pypi.org/project/judgeval/)
4. [Judgeval website](https://judgmentlabs.ai/)
5. [Release v1.5.0 (Oct 25, 2026)](https://github.com/JudgmentLabs/judgeval/releases/tag/v1.5.0)
6. [Release v1.4.0 (Oct 21, 2026)](https://github.com/JudgmentLabs/judgeval/releases/tag/v1.4.0)
7. [Release v1.3.3 (Oct 1, 2026)](https://github.com/JudgmentLabs/judgeval/releases/tag/v1.3.3)
8. [Devtoolwatch Board, the week of Oct 25–31, 2026](https://index.devtoolwatch.com/)