Devtoolwatch BoardRagas review
Ragas is a Python evaluation framework for RAG and LLM applications, with LLM-based and traditional metrics plus test data generation.
TL;DR
Ragas is a Python library for evaluating RAG systems, agents and other LLM applications. Ragas is for developers who want metrics in place of subjective assessments. Ragas combines LLM-based metrics with traditional non-LLM metrics, generates test datasets when you have none, and integrates with frameworks like LangChain and observability tools.
- 16.2kGitHub stars
- +60new stars, week of Oct 25
- 273kdownloads a week, −13% vs 4-week avg
- #19in Evals and observability this week
Ragas at a glance
- Category
- Evals and observability
- Maintained by
- Vibrant Labs (organization, @vibrantlabsai on GitHub)
- License
- Apache 2.0
- Language
- Python, Jupyter Notebook, Makefile
- Repository
- github.com/vibrantlabsai/ragas
- Website
- docs.ragas.io
- PyPI package
- ragas 0.4.3
- Latest release
- v0.4.3 on GitHub, Jan 13, 2026
- First published
- May 8, 2023
What Ragas does
- Ragas scores LLM application output with both LLM-based metrics and traditional metrics such as BLEU, ROUGE, CHRF and exact match.
- Ragas includes RAG metrics such as context precision, context recall, faithfulness, response relevancy and noise sensitivity.
- Ragas evaluates agents and tool use with metrics such as tool call accuracy, tool call F1, topic adherence and agent goal accuracy.
- Ragas generates test datasets for RAG and for agent or tool-use cases, including single-hop and multi-hop queries.
- Ragas lets you define custom metrics, such as a DiscreteMetric that grades any aspect of a response with an LLM.
- Ragas includes SQL metrics, including SQL query equivalence and an execution-based Datacompy score.
- Ragas supports prompt optimization, including a DSPyOptimizer with MIPROv2 added in v0.4.3.
How it works
Ragas runs as a Python library and command-line tool. You create an LLM with llm_factory and an OpenAI client, then score outputs with metrics. The ragas quickstart command scaffolds example projects. Ragas integrates with frameworks such as LangChain, LangGraph, LlamaIndex and Haystack. It also works with observability tools like Arize and LangSmith, and with providers such as Amazon Bedrock and Google Gemini.
How to install and start using Ragas
With pip
pip install ragasFrom source
pip install git+https://github.com/vibrantlabsai/ragasRequirements
- Python 3.9 or later
- An LLM provider key, such as OPENAI_API_KEY for the README example
First steps
- List the available project templates by running ragas quickstart.
- Create a RAG evaluation project with ragas quickstart rag_eval -o ./my-project.
- Set the OPENAI_API_KEY environment variable before running the README example.
- Define a metric such as DiscreteMetric and call metric.ascore with your LLM and response.
Who Ragas is for
- Developers evaluating retrieval-augmented generation pipelines
- Teams that need synthetic test datasets before collecting production data
- Developers measuring agent tool-call accuracy and goal completion
- Python projects built on LangChain, LlamaIndex or Haystack
Limitations to know
- Ragas quickstart offers only the rag_eval template for now; agent, benchmark, prompt and workflow templates are listed as coming soon.
- Ragas LLM-based metrics require calls to an LLM provider, which needs an API key.
- Ragas collects anonymized usage data by default; set RAGAS_DO_NOT_TRACK to true to opt out.
- Ragas has had API changes across versions, with migration guides from v0.1 to v0.2 and from v0.3 to v0.4.
Is Ragas growing? Usage data
In the week of Oct 25–31, 2026, Ragas ranked #360 on the Devtoolwatch Board and #19 in Evals and observability, with a momentum score of 18 out of 100. Its main package had 273k downloads that week, −13% against its own 4-week average.
Show the weekly numbers
| Week of | New stars |
|---|---|
| Aug 9, 2026 | 130 |
| Aug 16, 2026 | 107 |
| Aug 23, 2026 | 120 |
| Aug 30, 2026 | 90 |
| Sep 6, 2026 | 93 |
| Sep 13, 2026 | 71 |
| Sep 20, 2026 | 75 |
| Sep 27, 2026 | 63 |
| Oct 4, 2026 | 79 |
| Oct 11, 2026 | 70 |
| Oct 18, 2026 | 106 |
| Oct 25, 2026 | 60 |
The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.
Ragas alternatives
Other LLM evals and observability tools on the Board in the week of Oct 25–31, 2026, by momentum:
| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | SkillSpectorSkillSpector is NVIDIA's security scanner that checks AI agent skills for vulnerabilities, malicious patterns, and supply-chain risks… | 23.7k | – |
| #2 | JudgevalJudgeval is an open-source Python SDK for tracing and evaluating LLM-powered agents with prompt-based judges and online monitoring. | 1.1k | 80.2k |
| #3 | A.I.G (AI-Infra-Guard)A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak… | 7.2k | – |
| #4 | LangfuseLangfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. | 36.8k | 9.8M |
| #5 | OpikOpik is an open-source platform from Comet for tracing, evaluating, and monitoring LLM applications, RAG systems, and AI agents. | 23.1k | 709.1k |
| #6 | promptfoopromptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, including prompts, agents, and RAG pipelines. | 26.7k | 1.2M |
Recent Ragas releases
Ragas FAQ
Is Ragas free and open source?
Yes. Ragas's source code is on GitHub (vibrantlabsai/ragas) under the Apache 2.0 license, so you can use it for free.
How do I install Ragas?
Run pip install ragas. The ragas package needs Python 3.9 or newer. The install section above has the other options from the project's README.
Is Ragas still maintained?
Activity has slowed: the vibrantlabsai/ragas repository was last updated on Feb 24, 2026, and the latest release, v0.4.3 on GitHub, came out on Jan 13, 2026.
How popular is Ragas?
Ragas has 16.2k GitHub stars as of Nov 1, 2026. It gained 60 new stars in the week of Oct 25–31, 2026, against a 12-week average of 89 a week. Its main package had 273k downloads that week, −13% against its own 4-week average. That made it #19 among LLM evals and observability tools on the Devtoolwatch Board.
Ragas vs DeepEval: how do they compare?
Ragas and DeepEval are both tracked by Devtoolwatch as LLM evals and observability tools. Ragas has 16.2k GitHub stars and DeepEval has 19.2k. In the week of Oct 25–31, 2026, Ragas's main package had 273k downloads and DeepEval's had 769.5k. DeepEval is an open-source Python framework for unit testing and evaluating LLM apps, agents, chatbots, voice agents, and RAG pipelines. DeepEval review.
Ragas vs Langfuse: how do they compare?
Ragas and Langfuse are both tracked by Devtoolwatch as LLM evals and observability tools. Ragas has 16.2k GitHub stars and Langfuse has 36.8k. In the week of Oct 25–31, 2026, Ragas's main package had 273k downloads and Langfuse's had 9.8M. Langfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. Langfuse review.
What are the best alternatives to Ragas?
Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), Judgeval (#2), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5). The alternatives table above has their numbers.
Does Ragas use an LLM as a judge?
Yes, for many metrics. Ragas uses LLM-based metrics, such as Aspect Critic and DiscreteMetric, where an LLM grades a response. Ragas also offers traditional non-LLM metrics like BLEU, ROUGE and exact match, and documents how to align an LLM as a judge.
How do I use Ragas for RAG evaluation?
Run ragas quickstart rag_eval to create a RAG evaluation project. Then apply RAG metrics such as faithfulness, context precision, context recall and response relevancy. Ragas can also generate a RAG test set if you lack one.
Which LLM providers does Ragas work with?
Ragas documents integrations with OpenAI, Amazon Bedrock, Google Gemini, OCI Gen AI and Vertex AI models. Version 0.4.0 moved to instructor.from_provider for broader provider support, and v0.4.1 added Anthropic and Gemini clients.
Can Ragas evaluate AI agents?
Yes. Ragas provides agent and tool-use metrics, including tool call accuracy, tool call F1, topic adherence and agent goal accuracy. Ragas can also generate test sets for agent use cases.
Does Ragas collect telemetry?
Yes. Ragas collects minimal, anonymized usage data with no personal or company-identifying information. To opt out, set the RAGAS_DO_NOT_TRACK environment variable to true.
Sources
Ragas README badge
Maintainers can show the tool's weekly Board rank in their README:
[](https://index.devtoolwatch.com/tools/ragas/)How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].