Devtoolwatch BoardMLflow review
MLflow is an open source AI engineering platform for tracing, evaluating, monitoring, and managing agents, LLM applications, and ML models.
TL;DR
MLflow is an open source platform for debugging, evaluating, monitoring, and optimizing agents, LLM applications, and ML models. It is for teams that ship AI to production. MLflow combines OpenTelemetry-based tracing, LLM judges, prompt management, an AI Gateway, and classic ML experiment tracking and model registry in one vendor-neutral tool.
- 28.6kGitHub stars
- +103new stars, week of Oct 25
- 5.2Mdownloads a week, +11% vs 4-week avg
- #8in Evals and observability this week
MLflow at a glance
- Category
- Evals and observability
- Maintained by
- MLflow (organization, @mlflow on GitHub)
- License
- Apache 2.0
- Language
- Python, TypeScript, JavaScript
- Repository
- github.com/mlflow/mlflow
- Website
- mlflow.org
- PyPI package
- mlflow 3.17.2
- Latest release
- mlflow 3.17.2 on PyPI, Oct 21, 2026
- First published
- Jun 5, 2018
What MLflow does
- MLflow captures complete traces of LLM applications and agents, built on OpenTelemetry, with one-line automatic tracing for more than 60 frameworks.
- MLflow runs systematic evaluations using 50+ built-in metrics and LLM judges, or custom ones, to catch regressions before production.
- MLflow versions, tests, and deploys prompts with lineage tracking, and can automatically optimize prompts.
- MLflow's AI Gateway routes requests across LLM providers through an OpenAI-compatible interface with rate limits, fallbacks, guardrails, and traffic splitting.
- MLflow tracks experiments, parameters, and metrics, and manages model lifecycles in a Model Registry.
- MLflow deploys models to batch and real-time scoring on Docker, Kubernetes, Azure ML, AWS SageMaker, and more.
- MLflow Agent Server hosts agents on FastAPI with request validation, streaming support, and built-in tracing.
How it works
MLflow runs as a tracking server, for example started with uvx, that your code logs to through the Python SDK or autologging. Traces and metrics appear in the MLflow UI at localhost:5000. MLflow supports Python, TypeScript/JavaScript, and Java, integrates with OpenTelemetry and MCP, and runs locally, on-premises, on cloud platforms, or on managed services like Databricks and SageMaker.
How to install and start using MLflow
Setup wizard
curl -LsSf https://mlflow.org/wizard/setup.sh | shStart server with uvx
uvx mlflow serverRequirements
- Python 3.10 or later for the mlflow package.
- The setup wizard requires a Git repository and Claude Code, Codex, or OpenCode installed.
First steps
- Start the MLflow server with uvx mlflow server.
- In Python, call mlflow.set_tracking_uri("http://localhost:5000") and mlflow.openai.autolog() to enable logging.
- Run your OpenAI client code as usual so calls are traced.
- Explore traces and metrics in the MLflow UI at http://localhost:5000.
Who MLflow is for
- Teams debugging and monitoring LLM applications and agents in production.
- Developers evaluating LLM output quality with built-in or custom LLM judges.
- ML teams tracking experiments and managing models through a registry.
- Organizations wanting a self-hostable, vendor-neutral platform across cloud providers.
Limitations to know
- Some features, such as review queues and Unity Catalog traces, are specific to Databricks.
Is MLflow growing? Usage data
In the week of Oct 25–31, 2026, MLflow ranked #165 on the Devtoolwatch Board and #8 in Evals and observability, with a momentum score of 49 out of 100. Its main package had 5.2M downloads that week, +11% against its own 4-week average.
Show the weekly numbers
| Week of | New stars |
|---|---|
| Aug 9, 2026 | 110 |
| Aug 16, 2026 | 98 |
| Aug 23, 2026 | 122 |
| Aug 30, 2026 | 94 |
| Sep 6, 2026 | 118 |
| Sep 13, 2026 | 119 |
| Sep 20, 2026 | 100 |
| Sep 27, 2026 | 110 |
| Oct 4, 2026 | 102 |
| Oct 11, 2026 | 101 |
| Oct 18, 2026 | 98 |
| Oct 25, 2026 | 103 |
On Hacker News
- Show HN: MLflow – A platform for the complete machine learning, 5 points and 0 comments (Jun 8, 2018)
The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.
MLflow alternatives
Other LLM evals and observability tools on the Board in the week of Oct 25–31, 2026, by momentum:
| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | SkillSpectorSkillSpector is NVIDIA's security scanner that checks AI agent skills for vulnerabilities, malicious patterns, and supply-chain risks… | 23.7k | – |
| #2 | JudgevalJudgeval is an open-source Python SDK for tracing and evaluating LLM-powered agents with prompt-based judges and online monitoring. | 1.1k | 80.2k |
| #3 | A.I.G (AI-Infra-Guard)A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak… | 7.2k | – |
| #4 | LangfuseLangfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. | 36.8k | 9.8M |
| #5 | OpikOpik is an open-source platform from Comet for tracing, evaluating, and monitoring LLM applications, RAG systems, and AI agents. | 23.1k | 709.1k |
| #6 | promptfoopromptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, including prompts, agents, and RAG pipelines. | 26.7k | 1.2M |
Recent MLflow releases
- model-catalog/latest, Apr 6, 2026
- v3.17.2, Oct 21, 2026
- v3.17.1, Oct 13, 2026
- v3.17.0, Oct 7, 2026
- v2.11.5, Sep 23, 2026
MLflow FAQ
Is MLflow free and open source?
Yes. MLflow's source code is on GitHub (mlflow/mlflow) under the Apache 2.0 license, so you can use it for free.
How do I install MLflow?
Run curl -LsSf https://mlflow.org/wizard/setup.sh | sh. The mlflow package needs Python 3.10 or newer. The install section above has the other options from the project's README.
Is MLflow still maintained?
Yes. The mlflow/mlflow repository was last updated on Oct 25, 2026, and the latest release, mlflow 3.17.2 on PyPI, came out on Oct 21, 2026.
How popular is MLflow?
MLflow has 28.6k GitHub stars as of Nov 1, 2026. It gained 103 new stars in the week of Oct 25–31, 2026, against a 12-week average of 106 a week. Its main package had 5.2M downloads that week, +11% against its own 4-week average. That made it #8 among LLM evals and observability tools on the Devtoolwatch Board.
MLflow vs Langfuse: how do they compare?
MLflow and Langfuse are both tracked by Devtoolwatch as LLM evals and observability tools. MLflow has 28.6k GitHub stars and Langfuse has 36.8k. In the week of Oct 25–31, 2026, MLflow's main package had 5.2M downloads and Langfuse's had 9.8M. Langfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. Langfuse review.
What are the best alternatives to MLflow?
Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), Judgeval (#2), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5). The alternatives table above has their numbers.
What is MLflow?
MLflow is an open source AI engineering platform for agents, LLMs, and ML models. It covers tracing, evaluation, prompt management, an AI Gateway, experiment tracking, model registry, and deployment.
Is MLflow a Python library?
MLflow is distributed as a Python package on PyPI requiring Python 3.10 or later. MLflow also supports TypeScript/JavaScript and Java, and runs as a server with a web UI.
Is MLflow a model registry?
MLflow includes a Model Registry for collaboratively managing the full lifecycle of ML models, alongside experiment tracking, evaluation, and deployment tools.
How do I use MLflow locally?
MLflow runs locally by starting a server with uvx mlflow server, pointing your code at http://localhost:5000 with mlflow.set_tracking_uri, and enabling autologging. View traces in the UI at that address.
How do I use MLflow in Databricks?
MLflow can be hosted on Databricks, and the setup wizard can connect to a Databricks workspace. Recent releases add Databricks-specific features such as review queues.
Can MLflow be used with AWS SageMaker?
MLflow lists Amazon SageMaker as a hosting option and can deploy models to AWS SageMaker for scoring.
Sources
MLflow README badge
Maintainers can show the tool's weekly Board rank in their README:
[](https://index.devtoolwatch.com/tools/mlflow/)How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].