Devtoolwatch BoardOpenLLMetry review
OpenLLMetry is an open-source set of OpenTelemetry extensions that adds observability to LLM applications, with instrumentations for LLM providers, vector DBs and frameworks.
TL;DR
OpenLLMetry is a set of extensions built on OpenTelemetry that traces LLM applications. It is for Python developers who want LLM observability inside the stack they already use. Because OpenLLMetry outputs standard OpenTelemetry data, you can send traces to Datadog, Honeycomb, Grafana and other destinations.
- 7.5kGitHub stars
- +12new stars, week of Oct 25
- –not ranked this week
OpenLLMetry at a glance
- Category
- Evals and observability
- Maintained by
- traceloop from ServiceNow (organization, @traceloop on GitHub)
- License
- Apache 2.0
- Language
- Python, Shell
- Repository
- github.com/traceloop/openllmetry
- Website
- www.traceloop.com/openllmetry
- PyPI package
- traceloop-sdk 0.62.4
- Latest release
- 0.62.4 on GitHub, Sep 29, 2026
- First published
- Sep 2, 2023
What OpenLLMetry does
- OpenLLMetry instruments calls to LLM providers such as OpenAI, Anthropic, Bedrock, Cohere, Gemini, Mistral AI and Ollama.
- OpenLLMetry instruments vector databases including Chroma, Pinecone, Qdrant, Weaviate, Milvus, LanceDB and Marqo.
- OpenLLMetry supports frameworks such as LangChain, LangGraph, LlamaIndex, CrewAI, Haystack, LiteLLM and OpenAI Agents.
- OpenLLMetry instruments the MCP protocol, including client session spans.
- OpenLLMetry can also instrument everything OpenTelemetry already covers, such as databases and API calls.
- OpenLLMetry exports traces to tested destinations including Traceloop, Datadog, Honeycomb, New Relic, Splunk, Sentry and the OpenTelemetry Collector.
- OpenLLMetry provides decorators such as @workflow and @agent to name and group traced operations.
How it works
OpenLLMetry runs inside your Python application. You install the Traceloop SDK and call Traceloop.init(), which instruments supported libraries and emits standard OpenTelemetry data. You can route that data to your existing observability backend. If you already use OpenTelemetry, you can add individual OpenLLMetry instrumentations directly without the SDK.
How to install and start using OpenLLMetry
With pip
pip install traceloop-sdkRequirements
- Python 3.10 or later, below version 4.
First steps
- Install the SDK with pip install traceloop-sdk.
- Import Traceloop from traceloop.sdk and call Traceloop.init() in your code to start tracing.
- When running locally, use Traceloop.init(disable_batch=True) to see traces immediately.
- Follow the integration docs to connect your chosen observability destination.
Who OpenLLMetry is for
- Teams that want LLM traces in an existing OpenTelemetry-based observability stack.
- Python developers building on LLM provider APIs, vector databases or agent frameworks.
- Projects already instrumented with OpenTelemetry that need LLM-specific instrumentations added.
Limitations to know
- This repository covers Python. JavaScript and TypeScript use the separate OpenLLMetry-JS project.
- OpenLLMetry does not store or display traces itself. You need a destination backend to view them.
- Older SDK versions collected anonymous telemetry. Telemetry is removed from v0.49.2 onward, so upgrade to that version or later.
Is OpenLLMetry growing? Usage data
OpenLLMetry was not ranked on the Devtoolwatch Board in the week of Oct 25–31, 2026: it needs 25 or more new stars or 1,000 or more added downloads in a week to rank.
Show the weekly numbers
| Week of | New stars |
|---|---|
| Aug 9, 2026 | 15 |
| Aug 16, 2026 | 13 |
| Aug 23, 2026 | 23 |
| Aug 30, 2026 | 7 |
| Sep 6, 2026 | 18 |
| Sep 13, 2026 | 19 |
| Sep 20, 2026 | 18 |
| Sep 27, 2026 | 18 |
| Oct 4, 2026 | 19 |
| Oct 11, 2026 | 22 |
| Oct 18, 2026 | 8 |
| Oct 25, 2026 | 12 |
On Hacker News
- Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs, 154 points and 31 comments (Oct 11, 2023)
- Show HN: You don't need to adopt new tools for LLM observability, 102 points and 23 comments (Feb 14, 2024)
The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.
OpenLLMetry alternatives
Other LLM evals and observability tools on the Board in the week of Oct 25–31, 2026, by momentum:
| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | SkillSpectorSkillSpector is NVIDIA's security scanner that checks AI agent skills for vulnerabilities, malicious patterns, and supply-chain risks… | 23.7k | – |
| #2 | JudgevalJudgeval is an open-source Python SDK for tracing and evaluating LLM-powered agents with prompt-based judges and online monitoring. | 1.1k | 80.2k |
| #3 | A.I.G (AI-Infra-Guard)A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak… | 7.2k | – |
| #4 | LangfuseLangfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. | 36.8k | 9.8M |
| #5 | OpikOpik is an open-source platform from Comet for tracing, evaluating, and monitoring LLM applications, RAG systems, and AI agents. | 23.1k | 709.1k |
| #6 | promptfoopromptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, including prompts, agents, and RAG pipelines. | 26.7k | 1.2M |
Recent OpenLLMetry releases
OpenLLMetry FAQ
Is OpenLLMetry free and open source?
Yes. OpenLLMetry's source code is on GitHub (traceloop/openllmetry) under the Apache 2.0 license, so you can use it for free.
How do I install OpenLLMetry?
Run pip install traceloop-sdk. The traceloop-sdk package needs Python 3.10 or newer, below 4.
Is OpenLLMetry still maintained?
Yes. The traceloop/openllmetry repository was last updated on Oct 19, 2026, and the latest release, 0.62.4 on GitHub, came out on Sep 29, 2026.
How popular is OpenLLMetry?
OpenLLMetry has 7.5k GitHub stars as of Nov 1, 2026. It gained 12 new stars in the week of Oct 25–31, 2026, against a 12-week average of 16 a week.
What are the best alternatives to OpenLLMetry?
Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), Judgeval (#2), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5). The alternatives table above has their numbers.
Does OpenLLMetry work with my existing observability tool?
OpenLLMetry likely works with your current tool, because it outputs standard OpenTelemetry data. The README lists tested destinations including Datadog, Honeycomb, Grafana, New Relic, Splunk, Dynatrace, Sentry, SigNoz and the OpenTelemetry Collector.
Can I use OpenLLMetry without the Traceloop SDK?
Yes, OpenLLMetry works without the SDK. If you already have OpenTelemetry instrumented, you can add individual OpenLLMetry instrumentations directly. No telemetry is collected when you use the instrumentations without the SDK.
Does OpenLLMetry collect telemetry?
No, current versions of OpenLLMetry do not collect telemetry. The project no longer logs or collects telemetry in the SDK or the instrumentations from v0.49.2 onward.
How is OpenLLMetry related to OpenTelemetry semantic conventions?
OpenLLMetry's semantic conventions are now part of OpenTelemetry, according to the README. Recent releases also added GenAI semantic convention compliance for OpenAI Agents.
Is there a JavaScript version of OpenLLMetry?
Yes, there is a JavaScript version. JS and TypeScript users should use OpenLLMetry-JS, a separate repository, with the @traceloop/node-server-sdk package.
Sources
- traceloop/openllmetry on GitHub
- OpenLLMetry README
- traceloop-sdk on PyPI
- OpenLLMetry website
- Release 0.62.4 (Sep 29, 2026)
- Release 0.62.3 (Aug 10, 2026)
- Release 0.62.2 (Aug 9, 2026)
- Hacker News: Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs
- Hacker News: Show HN: You don't need to adopt new tools for LLM observability
- Devtoolwatch Board, the week of Oct 25–31, 2026
OpenLLMetry README badge
Maintainers can show the tool's weekly Board rank in their README:
[](https://index.devtoolwatch.com/tools/openllmetry/)How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].