Using OpenTelemetry instrumentation
Install tomodachi
using the opentelemetry
extras to enable instrumentation for OpenTelemetry. In addition, install with the opentelemetry-exporter-prometheus
extras to use Prometheus exporter metrics.
pip install tomodachi[opentelemetry]
pip install tomodachi[opentelemetry,opentelemetry-exporter-prometheus]
When added as a Poetry dependency the opentelemetry extras can be enabled by adding tomodachi = {extras = ["opentelemetry"]}
to the pyproject.toml file, and when added to a requiements.txt file the opentelemetry extras can be enabled by adding tomodachi[opentelemetry]
to the file.
Auto instrumentation: tomodachi --opentelemetry-instrument
tomodachi --opentelemetry-instrument
Passing the --opentelemetry-instrument
argument to tomodachi run
will automatically instrument the service with the appropriate exporters and configuration according to the set set OTEL_*
environment variables..
If tomodachi
is installed in the environment, using tomodachi --opentelemetry-instrument service.py
is mostly equivalent to running opentelemetry-instrument tomodachi run service.py
and will load distros, configurators and instrumentors automatically in the same way as the opentelemetry-instrument
CLI would do.
OTEL_LOGS_EXPORTER=console \
OTEL_TRACES_EXPORTER=console \
OTEL_METRICS_EXPORTER=console \
OTEL_SERVICE_NAME=example-service \
tomodachi --opentelemetry-instrument run service/app.py
Auto instrumention by setting env variable value: TOMODACHI_OPENTELEMETRY_INSTRUMENT=1
TOMODACHI_OPENTELEMETRY_INSTRUMENT=1
The environment variable TOMODACHI_OPENTELEMETRY_INSTRUMENT
if set will also enable auto instrumentation in the same way.
OTEL_LOGS_EXPORTER=console \
OTEL_TRACES_EXPORTER=console \
OTEL_METRICS_EXPORTER=console \
OTEL_SERVICE_NAME=example-service \
TOMODACHI_OPENTELEMETRY_INSTRUMENT=1 \
tomodachi run service/app.py
Auto instrumentation using the opentelemetry-instrument
CLI
opentelemetry-instrument
CLIAuto instrumentation using the opentelemetry-instrument
CLI can be achieved by starting services using opentelemetry-instrument [otel-options] tomodachi run [options] <service.py ...>
.
# either define the OTEL_* environment variables to specify instrumentation specification
OTEL_LOGS_EXPORTER=console \
OTEL_TRACES_EXPORTER=console \
OTEL_METRICS_EXPORTER=console \
OTEL_SERVICE_NAME=example-service \
opentelemetry-instrument tomodachi run service/app.py
# or use the arguments passed to the opentelemetry-instrument command
opentelemetry-instrument \
--logs_exporter console \
--traces_exporter console \
--metrics_exporter console \
--service_name example-service \
tomodachi run service/app.py
Manual instrumentation
Auto instrumentation using either tomodachi --opentelemetry-instrument
, setting the TOMODACHI_OPENTELEMETRY_INSTRUMENT=1
env value or using the opentelemetry-instrument
CLI are the recommended ways of instrumenting services, as they will automatically instrument the service (and libs with instrumentors installed) with the appropriate exporters and configuration.
However, instrumentation can also be enabled by importing the TomodachiInstrumentor
instrumentation class and calling its' instrument
function before defining the service class.
import tomodachi
from tomodachi.opentelemetry import TomodachiInstrumentor
TomodachiInstrumentor().instrument()
class Service(tomodachi.Service):
name = "example-service"
@tomodachi.http(GET, r"/example")
async def example(self, request):
return 200, "hello world"
Starting such a service with the appropriate OTEL_*
environment variables would properly instrument traces, logs and metrics for the service without the need to use the opentelemetry-instrument
CLI.
OTEL_LOGS_EXPORTER=console \
OTEL_TRACES_EXPORTER=console \
OTEL_METRICS_EXPORTER=console \
OTEL_SERVICE_NAME=example-service \
tomodachi run service/app.py
Service name dynamically set if missing OTEL_SERVICE_NAME
value
OTEL_SERVICE_NAME
valueIf the OTEL_SERVICE_NAME
environment variable value (or --service_name
argument to opentelemetry-instrument
) is not set, the resource' service.name
will instead be set to the name
attribute of the service class.
In case the service class uses the default generic names ("service"
or "app"
), the resource' service.name
will instead be set to the default as specified in https://github.com/open-telemetry/semantic-conventions/tree/main/docs/resource#service.
In the rare case where there's multiple tomodachi
services started within the same Python process, it should be noted that OTEL traces, metrics and logging will primarily use the OTEL_SERVICE_NAME
, and if it's missing then use the name from the first instrumented service class. The same goes for the service.instance.id
resource attribute, which will be set to the first instrumented service class' uuid
value (which in most cases is automatically assigned on service start). Multi-service execution won't accurately distinguish the service name of tracers, meters and loggers. The recommended solution if this is an issue, is to split the services into separate processes instead.
Exclude lists to exclude certain URLs from traces and metrics
To exclude certain URLs from traces and metrics, set the environment variable OTEL_PYTHON_TOMODACHI_EXCLUDED_URLS
(or OTEL_PYTHON_EXCLUDED_URLS
to cover all instrumentations) to a string of comma delimited regexes that match the URLs.
Regexes from the OTEL_PYTHON_AIOHTTP_EXCLUDED_URLS
environment variable will also be excluded.
For example,
export OTEL_PYTHON_TOMODACHI_EXCLUDED_URLS="client/.*/info,healthcheck"
will exclude requests such as https://site/client/123/info
and https://site/xyz/healthcheck
.
You can also pass comma delimited regexes directly to the instrument
method:
TomodachiInstrumentor().instrument(excluded_urls="client/.*/info,healthcheck")
Prometheus meter provider (experimental)
The tomodachi.opentelemetry
module also provides a Prometheus meter provider that can be used to export metrics to Prometheus. Run opentelemetry-instrument
with the --meter_provider tomodachi_prometheus
argument (or set OTEL_PYTHON_METER_PROVIDER=tomodachi_prometheus
environment value) to enable the Prometheus meter provider.
Environment variables to configure Prometheus meter provider
OTEL_PYTHON_TOMODACHI_PROMETHEUS_ADDRESS
specifies the host address the Prometheus export server should listen on. (default:"localhost"
)OTEL_PYTHON_TOMODACHI_PROMETHEUS_PORT
specifies the port the Prometheus export server should listen on. (default:9464
)OTEL_PYTHON_TOMODACHI_PROMETHEUS_INCLUDE_SCOPE_INFO
specifies whether to include scope information asotel_scope_info
value. (default:true
)OTEL_PYTHON_TOMODACHI_PROMETHEUS_INCLUDE_TARGET_INFO
specifies whether to include resource attributes astarget_info
value. (default:true
)OTEL_PYTHON_TOMODACHI_PROMETHEUS_EXEMPLARS_ENABLED
specifies whether exemplars (experimental) should be collected and used in Prometheus export. (default:false
)OTEL_PYTHON_TOMODACHI_PROMETHEUS_NAMESPACE_PREFIX
specifies the namespace prefix for Prometheus metrics. A final underscore is automatically added if prefix is used. (default:""
)
Dependency requirement for Prometheus meter provider
The tomodachi_prometheus
meter provider requires that the opentelemetry-exporter-prometheus
and prometheus_client
packages are installed.
Use tomodachi
extras opentelemetry-exporter-prometheus
to automatically include a compatible version of the exporter.
The tomodachi_prometheus
meter provider requires that the opentelemetry-exporter-prometheus
package is installed. Use tomodachi
extras opentelemetry-exporter-prometheus
to automatically include a compatible version of the exporter.
OpenMetrics output from Prometheus with exemplars enabled
With exemplars enabled (env variable: OTEL_PYTHON_TOMODACHI_PROMETHEUS_EXEMPLARS_ENABLED=true
), make sure to call the Prometheus client with the accept header application/openmetrics-text
to ensure exemplars are included in the response.
curl http://localhost:9464/metrics -H "Accept: application/openmetrics-text"
💡 Note that if the accept header application/openmetrics-text
is missing from the request, exemplars will be excluded from the response.
Example: starting a service with instrumentation
This example will start and instrument a service with OTLP exported traces sent to the endpoint otelcol:4317
and metrics that can be scraped by Prometheus from port 9464
.
All metrics except for target_info
and otel_scope_inf
will be prefixed with "tomodachi_"
.
Additionally exemplars will be added to the Prometheus collected metrics that includes sample exemplars with trace_id and span_id labels.
TOMODACHI_OPENTELEMETRY_INSTRUMENT=1 \
OTEL_TRACES_EXPORTER=otlp \
OTEL_EXPORTER_OTLP_ENDPOINT=otelcol:4317 \
OTEL_PYTHON_METER_PROVIDER=tomodachi_prometheus \
OTEL_PYTHON_TOMODACHI_PROMETHEUS_EXEMPLARS_ENABLED=true \
OTEL_PYTHON_TOMODACHI_PROMETHEUS_ADDRESS=0.0.0.0 \
OTEL_PYTHON_TOMODACHI_PROMETHEUS_PORT=9464 \
OTEL_PYTHON_TOMODACHI_PROMETHEUS_NAMESPACE_PREFIX=tomodachi \
tomodachi run service/app.py
Updated 8 months ago