Working with Heroku Telemetry Drains
Last updated December 03, 2024
Table of Contents
Telemetry and observability are crucial for maintaining the health and performance of applications. Heroku’s cloud-native telemetry services make it easy to collect trace, metrics and logs from your application and forward them to archival, observability, and alerting services offered by Heroku’s telemetry add-on partners.
Heroku Telemetry Drains give you ultimate control over where to send your telemetry. You can configure multiple drains and set them up at the app or space level. This article describes how to configure them on our platform.
Add a Telemetry Drain
To add a telemetry drain to an app or space, use the heroku telemetry:add command.
For example, to add a telemetry drain to an application:
heroku telemetry:add --app myapp --signals traces,metrics --endpoint https://my-endpoint.com --transport http '{"x-sample-header":"sample-value"}'
To add a telemetry drain to the entire space:
heroku telemetry:add --space myspace --signals logs --endpoint https://my-endpoint.com --transport grpc '{"x-sample-header":"sample-value"}'
Signals
You can specify which signals you want to send to the drain by passing a comma-separated list of values. The available options are: logs, metrics, and traces. If you don’t pass a signals list, it defaults to all signals. You can mix and match the signals as you choose between telemetry drains.
# Send all signals, the default when --signals is not set
heroku telemetry:add --app myapp --signals traces,metrics,logs --endpoint https://my-endpoint.com --transport http '{"x-sample-header":"sample-value"}'
# Send only traces and metrics
heroku telemetry:add --app myapp --signals traces,metrics --endpoint https://my-endpoint.com --transport http '{"x-sample-header":"sample-value"}'
# Send only traces
heroku telemetry:add --app myapp --signals traces --endpoint https://my-endpoint.com --transport http '{"x-sample-header":"sample-value"}'
Transport Protocol Type
When configuring the transport protocol for a drain, it defines the transport protocol used to transport OTLP signals from the Heroku OpenTelemetry Collectors to the appropriate observability provider. The options are grpc and http.
Conversely, within your application the OpenTelemetry SDK is used to emit application-specific OTLP telemetry signals to the Heroku OpenTelemetry Collector. The Heroku Collector is capable of receiving telemetry from your application dynos over either transport, grpc or http.
It’s important to understand the difference between the two scenarios. For example, it’s possible for the Heroku OpenTelemetry Collectors to collect telemetry data from a Ruby application over HTTP transport, while having a drain associated to the application which exports data to an observability provider over gRPC. See OpenTelemetry Concepts and Heroku for a diagram and more info about the transport protocol types.
Headers
We expect the headers to be provided as a JSON object of key/value pairs. For example, your observability vendor may require you to authenticate with their endpoints. You can do so by specifying the Authorization header.
heroku telemetry:add --app myapp --endpoint https://my-endpoint.com --transport http '{"Authorization":"Bearer ABC123abcABC123abcABC123abcABC123"}'
It’s possible to specify more than one header, which is helpful if your chosen observability vendor requires both an API key and a dataset to store the telemetry data within.
heroku telemetry:add --app myapp --endpoint https://my-endpoint.com --transport http '{"x-api-key": "API_KEY", "x-dataset": "MY_DATASET"}'
Note that there are some providers that require no headers because the authentication or API token is embedded within the path of the endpoint URL. As a result the command follows the form:
heroku telemetry:add --app myapp --endpoint https://api.example-drain-example.io/somereallylongkeywhichincludesthedataset --transport http '{}'
Get a Drain’s ID
Pass the name of the app or space to the heroku telemetry to get a list of your drains. For example:
heroku telemetry -a example-app
The output lists all of the app’s (or space’s) drains, along with their drain IDs. Use this ID for other heroku telemetry commands.
View Info for a Specific Drain
To see detailed info about a specific drain, pass in the drain’s ID to heroku telemetry:info. For example:
heroku telemetry:info 1d093893-d620-4d43-8b6b-2c0855885d13
Updating a Telemetry Drain
You can update the details of a telemetry drain using the heroku telemetry:update command. It is possible to change the signals, update the endpoint, change the headers and even switch transports.
Any attributes of the telemetry drain that you don’t provide on the update command is left intact. For example, you can update an API key header without needing to re-specify the signals, endpoint or transport used.
The only exception is when you want to update the signals or the headers. In this case, the updated signals and headers fully replace the existing details.
For example, the following command updates the telemetry drain to emit only logs and metrics, regardless of what was previously set:
heroku telemetry:update 1d093893-d620-4d43-8b6b-2c0855885d13 --signals logs,metrics
Remove a Telemetry Drain
To remove a specific drain, pass in the drain’s ID to heroku telemetry:remove. For example:
heroku telemetry:remove 1d093893-d620-4d43-8b6b-2c0855885d13
To remove all drains for an app or a space, pass in the app or space name instead. For example:
heroku telemetry:remove example-space
Using Multiple Telemetry Drains
Telemetry drains are additive. For a Fir space that contains 2 applications, you can configure telemetry drain for that space that applies to both applications. You can also configure each application with its own independent telemetry drain.