# Ragas review

> Ragas is a Python evaluation framework for RAG and LLM applications, with LLM-based and traditional metrics plus test data generation.

Devtoolwatch Board, the week of Oct 25–31, 2026. Canonical page: https://index.devtoolwatch.com/tools/ragas/

## TL;DR

Ragas is a Python library for evaluating RAG systems, agents and other LLM applications. Ragas is for developers who want metrics in place of subjective assessments. Ragas combines LLM-based metrics with traditional non-LLM metrics, generates test datasets when you have none, and integrates with frameworks like LangChain and observability tools.

## Ragas at a glance

- **Category:** [Evals and observability](https://index.devtoolwatch.com/categories/evals-and-observability/)
- **Maintained by:** Vibrant Labs (organization, [@vibrantlabsai](https://github.com/vibrantlabsai) on GitHub)
- **License:** Apache 2.0
- **Language:** Python, Jupyter Notebook, Makefile
- **Repository:** [github.com/vibrantlabsai/ragas](https://github.com/vibrantlabsai/ragas)
- **Website:** [docs.ragas.io](https://docs.ragas.io)
- **PyPI package:** [ragas](https://pypi.org/project/ragas/) 0.4.3
- **Latest release:** [v0.4.3](https://github.com/vibrantlabsai/ragas/releases/tag/v0.4.3) on GitHub, Jan 13, 2026
- **First published:** May 8, 2023

## What Ragas does

- Ragas scores LLM application output with both LLM-based metrics and traditional metrics such as BLEU, ROUGE, CHRF and exact match.
- Ragas includes RAG metrics such as context precision, context recall, faithfulness, response relevancy and noise sensitivity.
- Ragas evaluates agents and tool use with metrics such as tool call accuracy, tool call F1, topic adherence and agent goal accuracy.
- Ragas generates test datasets for RAG and for agent or tool-use cases, including single-hop and multi-hop queries.
- Ragas lets you define custom metrics, such as a DiscreteMetric that grades any aspect of a response with an LLM.
- Ragas includes SQL metrics, including SQL query equivalence and an execution-based Datacompy score.
- Ragas supports prompt optimization, including a DSPyOptimizer with MIPROv2 added in v0.4.3.

### How it works

Ragas runs as a Python library and command-line tool. You create an LLM with llm_factory and an OpenAI client, then score outputs with metrics. The ragas quickstart command scaffolds example projects. Ragas integrates with frameworks such as LangChain, LangGraph, LlamaIndex and Haystack. It also works with observability tools like Arize and LangSmith, and with providers such as Amazon Bedrock and Google Gemini.

## How to install Ragas

**With pip**

```
pip install ragas
```

**From source**

```
pip install git+https://github.com/vibrantlabsai/ragas
```

### Requirements

- Python 3.9 or later
- An LLM provider key, such as OPENAI_API_KEY for the README example

### First steps

1. List the available project templates by running ragas quickstart.
2. Create a RAG evaluation project with ragas quickstart rag_eval -o ./my-project.
3. Set the OPENAI_API_KEY environment variable before running the README example.
4. Define a metric such as DiscreteMetric and call metric.ascore with your LLM and response.

## Who Ragas is for

- Developers evaluating retrieval-augmented generation pipelines
- Teams that need synthetic test datasets before collecting production data
- Developers measuring agent tool-call accuracy and goal completion
- Python projects built on LangChain, LlamaIndex or Haystack

### Limitations

- Ragas quickstart offers only the rag_eval template for now; agent, benchmark, prompt and workflow templates are listed as coming soon.
- Ragas LLM-based metrics require calls to an LLM provider, which needs an API key.
- Ragas collects anonymized usage data by default; set RAGAS_DO_NOT_TRACK to true to opt out.
- Ragas has had API changes across versions, with migration guides from v0.1 to v0.2 and from v0.3 to v0.4.

## Usage data (the week of Oct 25–31, 2026)

- Board rank: #360 overall, #19 in Evals and observability, momentum score 18
- Downloads that week: 273k (−13% vs 4-week average)
- New GitHub stars per week (last 12 weeks): Aug 9: 130; Aug 16: 107; Aug 23: 120; Aug 30: 90; Sep 6: 93; Sep 13: 71; Sep 20: 75; Sep 27: 63; Oct 4: 79; Oct 11: 70; Oct 18: 106; Oct 25: 60

## Ragas alternatives

| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | [SkillSpector](https://index.devtoolwatch.com/tools/nvidia-skillspector/) | 23.7k | – |
| #2 | [Judgeval](https://index.devtoolwatch.com/tools/judgmentlabs-judgeval/) | 1.1k | 80.2k |
| #3 | [A.I.G (AI-Infra-Guard)](https://index.devtoolwatch.com/tools/tencent-ai-infra-guard/) | 7.2k | – |
| #4 | [Langfuse](https://index.devtoolwatch.com/tools/langfuse/) | 36.8k | 9.8M |
| #5 | [Opik](https://index.devtoolwatch.com/tools/opik/) | 23.1k | 709.1k |
| #6 | [promptfoo](https://index.devtoolwatch.com/tools/promptfoo/) | 26.7k | 1.2M |

## Ragas FAQ

### Is Ragas free and open source?

Yes. Ragas's source code is on GitHub (vibrantlabsai/ragas) under the Apache 2.0 license, so you can use it for free.

### How do I install Ragas?

Run `pip install ragas`. The ragas package needs Python 3.9 or newer. The install section above has the other options from the project's README.

### Is Ragas still maintained?

Activity has slowed: the vibrantlabsai/ragas repository was last updated on Feb 24, 2026, and the latest release, v0.4.3 on GitHub, came out on Jan 13, 2026.

### How popular is Ragas?

Ragas has 16.2k GitHub stars as of Nov 1, 2026. It gained 60 new stars in the week of Oct 25–31, 2026, against a 12-week average of 89 a week. Its main package had 273k downloads that week, −13% against its own 4-week average. That made it #19 among LLM evals and observability tools on the Devtoolwatch Board.

### Ragas vs DeepEval: how do they compare?

Ragas and DeepEval are both tracked by Devtoolwatch as LLM evals and observability tools. Ragas has 16.2k GitHub stars and DeepEval has 19.2k. In the week of Oct 25–31, 2026, Ragas's main package had 273k downloads and DeepEval's had 769.5k. DeepEval is an open-source Python framework for unit testing and evaluating LLM apps, agents, chatbots, voice agents, and RAG pipelines. [DeepEval review](https://index.devtoolwatch.com/tools/deepeval/)

### Ragas vs Langfuse: how do they compare?

Ragas and Langfuse are both tracked by Devtoolwatch as LLM evals and observability tools. Ragas has 16.2k GitHub stars and Langfuse has 36.8k. In the week of Oct 25–31, 2026, Ragas's main package had 273k downloads and Langfuse's had 9.8M. Langfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. [Langfuse review](https://index.devtoolwatch.com/tools/langfuse/)

### What are the best alternatives to Ragas?

Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), Judgeval (#2), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5). The alternatives table above has their numbers.

### Does Ragas use an LLM as a judge?

Yes, for many metrics. Ragas uses LLM-based metrics, such as Aspect Critic and DiscreteMetric, where an LLM grades a response. Ragas also offers traditional non-LLM metrics like BLEU, ROUGE and exact match, and documents how to align an LLM as a judge.

### How do I use Ragas for RAG evaluation?

Run ragas quickstart rag_eval to create a RAG evaluation project. Then apply RAG metrics such as faithfulness, context precision, context recall and response relevancy. Ragas can also generate a RAG test set if you lack one.

### Which LLM providers does Ragas work with?

Ragas documents integrations with OpenAI, Amazon Bedrock, Google Gemini, OCI Gen AI and Vertex AI models. Version 0.4.0 moved to instructor.from_provider for broader provider support, and v0.4.1 added Anthropic and Gemini clients.

### Can Ragas evaluate AI agents?

Yes. Ragas provides agent and tool-use metrics, including tool call accuracy, tool call F1, topic adherence and agent goal accuracy. Ragas can also generate test sets for agent use cases.

### Does Ragas collect telemetry?

Yes. Ragas collects minimal, anonymized usage data with no personal or company-identifying information. To opt out, set the RAGAS_DO_NOT_TRACK environment variable to true.

## Sources

1. [vibrantlabsai/ragas on GitHub](https://github.com/vibrantlabsai/ragas)
2. [Ragas README](https://github.com/vibrantlabsai/ragas/blob/main/README.md)
3. [ragas on PyPI](https://pypi.org/project/ragas/)
4. [Ragas website](https://docs.ragas.io)
5. [Release v0.4.3 (Jan 13, 2026)](https://github.com/vibrantlabsai/ragas/releases/tag/v0.4.3)
6. [Release v0.4.2 (Dec 23, 2025)](https://github.com/vibrantlabsai/ragas/releases/tag/v0.4.2)
7. [Release v0.4.1 (Dec 10, 2025)](https://github.com/vibrantlabsai/ragas/releases/tag/v0.4.1)
8. [Devtoolwatch Board, the week of Oct 25–31, 2026](https://index.devtoolwatch.com/)