Python Auto-instrumentation with OpenTelemetry

Photo by Amy-Leigh Barnard on Unsplash

We’ve all been there. You want to get started with distributed tracing, but don’t have the time to revisit the codebase of dozens, if not hundreds of dozens of services in your system. Don’t worry, OpenTelemetry’s got you covered. Thanks to a great community effort from members of many organizations, the OpenTelemetry project has been able to quickly ramp up its ability to auto-instrument code in a variety of languages for many widely used third-party libraries.

Why auto-instrument?

For many organizations’ engineering teams, the biggest barrier to using distributed tracing, and therefore making the lives of operators 10,000 times better at least, is the time it takes to instrument a system. Auto-instrumentation helps alleviate this burden by hooking directly into existing code. With auto-instrumentation, engineers can

• enable instrumentation with few, if any, changes to their code
• gain visibility into what libraries are doing without having to understand all the details first
• expect consistent instrumentation across applications leveraging those libraries

Instrumentation is the first step in developing observability in your applications, and gives developers the superpower to ask meaningful questions about their code.

How do I use it?

I could go on and on about how auto-instrumentation makes life better and easier, but what better way to demonstrate it than by trying it out. The following example will walk through instrumenting a Python application with OpenTelemetry. If you already have an application that uses any of the supported libraries, feel free to skip this step and go straight to the section on configuring the exporter.

Otherwise, we’ll create a small application that receives web requests and makes a request upstream.

Requirements

• Python 3.4+ is required to use OpenTelemetry
• Docker is required if you want to run Jaeger locally

The example

First, we’ll install the Python packages that our application will use:

Save the code below in a new file, server.py. You can find all the code for this example in the lightstep/opentelemetry-examples repo.

NOTE: This is potentially the worst proxy server ever

Configuring the exporter

We’ll run a Jaeger backend locally to collect and show tracing information. If you already have Jaeger running somewhere in your environment, you can skip the following step:

docker run --name jaeger -d -p 6831:6831/udp -p 16686:16686 jaegertracing/all-in-one:latest

To allow our application to externalize its telemetry, we need to configure OpenTelemetry to use an exporter.

Insert the following code near the top of the server.py file we created earlier.

Next, we’ll install the auto-instrumentation packages and the Jaeger exporter packages viapip.

pip3 install opentelemetry-instrumentation opentelemetry-exporter-jaeger opentelemetry-instrumentation-flask

We’re now ready to run our application! Open a terminal and run the opentelemetry-instrument executable with our application as an argument.

NOTE: we’re exporting an environment variable to set the OpenTelemetry TracerProvider to the opentelemetry-sdk’s TracerProvider. Configuring this before starting the auto-instrumentation ensures that all instrumentation will use the same TracerProvider.

export OTEL_PYTHON_TRACER_PROVIDER=sdk_tracer_provider
opentelemetry-instrument ./server.py

In another terminal, we’ll make a few requests to our app. One of these requests is going through a proxy that introduces a delay in the response.

curl -s "localhost:5000/?url=https://en.wikipedia.org/wiki/Mars"  > /dev/null
curl -s "localhost:5000/?url=http://slowwly.robertomurray.co.uk/delay/3000/url/http://www.google.com" > /dev/null
curl -s "localhost:5000/cache"  > /dev/null

Looking at the traces

Here comes the exciting part. Let’s go and search for traces in our Jaeger interface, available at http://localhost:16686/. Right away, we can see traces for all the requests that we made to the app and we wrote ZERO instrumentation code.

A valuable piece of information available through the auto-instrumentation are tags attached to the spans for each library. In the case below, it makes identifying the root cause of a slow response in the application trivial.

Limits of auto-instrumentation

Auto-instrumentation is tightly coupled with the libraries it instruments. If an application is reliant on libraries that are not yet supported by auto-instrumentation, no additional insights will be gained. Thankfully, there are a lot of people in the OpenTelemetry project working to increase the number of supported libraries every day. Take some time to read through the registry to see supported languages and frameworks. Got a framework you’re using, but don’t see it in the registry? Create an issue in the project and let’s collaborate on it! Follow the project at OpenTelemetry.io to stay up-to-date on the latest news.

What’s next? Start tracing! Auto-instrumentation may never be quite as thorough as manual instrumentation, but it’s a great starting point in your observability journey.

2020–08–28: Updated to reflect changes in opentelemetry-python 0.12b0

2020–06–11: Updated to reflect changes in opentelemetry-python 0.9b0

Originally published at https://lightstep.com.