Last updated December 03, 2024

Heroku’s logging infrastructure makes it easy to collect logs from your applications and forward them to archival, search, and alerting services. In the Cedar generation of Heroku, you can forward them to logging add-on providers. On Fir, you can forward logs, along with other OpenTelemetry signals to telemetry drains. You can also add your own log drain and take full control of how your application logs are processed. This article covers how to configure log drains on Heroku.

Heroku supports three types of log drains:

• Syslog drains (only on Cedar)
• HTTPS drains (only on Cedar)
• Telemetry drains (only on Fir)

Syslog Drains

Syslog drains allow you to forward your Heroku logs to an external syslog server for long-term archiving. Heroku supports both secure (TLS) and insecure syslog drains. You must configure the logging service (or your server) to be able to receive syslog packets from Heroku, and then add its syslog URL (which contains the host and port) as a syslog drain.

Drain log messages are formatted according to RFC5424. They are delivered over TCP, as described in RFC6587, using the octet counting framing method.

TLS Syslog

$ heroku drains:add syslog+tls://logs.example.com:12345 -a myapp

Plain-Text Syslog

$ heroku drains:add syslog://logs.example.com -a myapp

Drain Tokens

When you add a drain to an application, we assign it a unique drain token that looks similar to the following:

d.9173ea1f-6f14-4976-9cf0-7cd0dafdcdbc

The drain token is written into all log messages that originate from the corresponding drain. For example:

2013-01-01T01:01:01.000000+00:00 d.9173ea1f-6f14-4976-9cf0-7cd0dafdcdbc app[web.1] Your message here.

You can obtain the drain tokens for your app’s log drains by running heroku drains --json:

$ heroku drains --json
[
  {
    "addon": null,
    "created_at": "2018-12-04T00:59:46Z",
    "id": "906262a4-e151-45d2-b35a-a2dc0ea9e688",
    "token": "d.f14da5dc-106b-468d-b1bd-bed0ed9fa1e7",
    "updated_at": "2018-12-04T00:59:47Z",
    "url": "syslog://logs.example.com"
  },
  {
    "addon": {
      "id": "130dfc11-ca20-48ea-8a29-c0da56ea2c4f",
      "name": "papertrail-curly-14926"
    },
    "created_at": "2018-10-26T21:44:01Z",
    "id": "b8c04a36-c914-4ec4-a5bd-eaa8c965d0dd",
    "token": "d.0b55c011-f49b-4428-ac2f-a5935fcffd50",
    "updated_at": "2018-10-26T21:44:02Z",
    "url": "syslog+tls://logs7.papertrailapp.com:41942"
  }
]

The example output above shows that an app has two log drains. The user added the first drain listed directly (its addon field is null), and Papertrail added the . Each drain’s token is available in its token field.

Use the drain token to separate log messages sent by different drains to the same drain URL. Drains to the same URL on different apps have different drain tokens. The drain token remains the same for the lifetime of the drain, but changes if you delete and re-add the same drain.

HTTPS Drains

Log drains also support messaging via HTTPS. This support makes it easy to write your own log processing logic and run it on a web service, such as another Heroku app.

You must configure your web service so that Heroku can make POST requests to it. Then, you add an HTTPS drain like so:

$ heroku drains:add https://user:pass@mylogdrain-1234567890ab.herokuapp.com/logs -a myapp

With HTTPS drains, Logplex buffers log messages and submits batches of them to an HTTPS endpoint via a POST request. The POST body contains syslog-formatted messages, framed using the Syslog TCP protocol octet counting framing method. These batches are posted with a Content-Type header of application/logplex-1.

A Logplex POST body resembles the following:

83 <40>1 2012-11-30T06:45:29+00:00 host app web.3 - State changed from starting to up
119 <40>1 2012-11-30T06:45:26+00:00 host app web.3 - Starting process with command `bundle exec rackup config.ru -p 24405`

The following request headers are sent with every Logplex POST request:

• Logplex-Msg-Count: The number of messages encoded in the body of the request. For the example body above, this value is 2. Use this field to ensure that you’ve parsed the body correctly.
• Logplex-Frame-Id: The unique identifier for this request. If this request is retried for some reason, like non-2xx response code or network connection failure, this identifier enables you to spot duplicate requests.
• Logplex-Drain-Token: The drain token for the log drain sending the request.
• User-Agent: This field describes the version of Logplex. It changes with Logplex releases (e.g., Logplex/v72).
• Content-Type: This field describes the encoding of the body for the request. Currently, this value is always application/logplex-1.

HTTPS Drain Caveats

• Logplex doesn’t support chunked transfer encoding of responses. Consequently, set the Content-Length header of your response to 0, and do not return a body.
• The application/logplex-1 content type doesn’t conform to RFC 5424. It omits STRUCTURED-DATA but doesn’t replace it with a NILVALUE.

Telemetry Drains

Telemetry drains provide apps and spaces the ability to export telemetry signals (traces, metrics and logs) to a provider that supports the OpenTelemetry OTLP protocol. You can manage OTLP drains via the heroku telemetry set of commands.

Our tooling batches telemetry signals before exporting them. See Heroku Telemetry Drains for more information.

Security Considerations

HTTPS drains support transport-level encryption using the HTTPS protocol, and authentication using HTTP Basic Authentication. Logplex currently validates the server certificate on all TLS and HTTP connections. You can opt-out of this functionality by switching to insecure mode for your TLS Syslog drain by following the instructions in the Debugging drains section below.

Syslog drains support transport-level encryption, including certificate and hostname verification during TLS handshake. However, Logplex does not support authentication for syslog drains. Therefore, it is possible for third parties to send log messages to your server if they know its address. As a security measure, you can filter out messages with unknown (or missing) drain tokens.

TLS Syslog drains fully support modern TLS, such as TLS 1.2 and TLS 1.3.

OTLP drains are transported over TLS and Heroku Runtime services are managed internally.

Debugging Drains

When using TLS with Syslog drains, it’s possible that either the certificate or hostname verification process will fail, and your drain will stop receiving logs. This can happen for a variety of reasons, including:

• Using an expired certificate
• Using a self-signed certificate
• Certificate hostname mismatches

Errors such as these appear in the log buffer displayed by the heroku logs command. You might need to stream your app’s logs for a period of time using heroku logs -t in order to capture an error. Removing and re-adding a misbehaving drain while streaming the logs can also help track down an issue.

You can also switch to insecure mode for your TLS Syslog drain. This works similarly to curl --insecure. Transport-level encryption will remain active, however all verification checks during the TLS handshake will be disabled. Logplex uses the #insecure URI fragment to enable insecure mode:

$ heroku drains:add syslog+tls://logs.example.com:12345/#insecure -a myapp

Removing Drains

Removing Syslog or HTTP Drains

You can remove a log drain from your app with the heroku drains:remove command:

$ heroku drains:remove syslog+tls://logs.example.com:12345/#insecure -a myapp

If a log drain was created by an add-on, you can’t manually remove it. It gets removed when you detach or destroy the add-on.

Removing Telemetry Drains

You can remove telemetry drains from your app or space with the heroku telemetry:remove command.