Last updated October 18, 2022

Heroku’s Logplex logging service makes it easy to collect logs from your applications and forward them to archival, search, and alerting services offered by Heroku’s logging add-on providers. You can also add your own log drain and take full control of how your application logs are processed.

Heroku supports two types of log drains:

• Syslog drains
• HTTPS drains

Syslog drains

Syslog drains allow you to forward your Heroku logs to an external Syslog server for long-term archiving. Heroku supports both secure (TLS) and insecure Syslog drains. You must configure the logging service (or your server) to be able to receive Syslog packets from Heroku, and then add its Syslog URL (which contains the host and port) as a Syslog drain.

Drain log messages are formatted according to RFC5424. They are delivered over TCP, as described in RFC6587, using the octet counting framing method.

TLS Syslog

$ heroku drains:add syslog+tls://logs.example.com:12345 -a myapp

Plain text Syslog

$ heroku drains:add syslog://logs.example.com -a myapp

Drain tokens

When you add a drain to an application, it is assigned a unique drain token that looks similar to the following:

d.9173ea1f-6f14-4976-9cf0-7cd0dafdcdbc

The drain token is written into all log messages that originate from the corresponding drain. For example:

2013-01-01T01:01:01.000000+00:00 d.9173ea1f-6f14-4976-9cf0-7cd0dafdcdbc app[web.1] Your message here.

You can obtain the drain tokens for your app’s log drains by running heroku drains --json:

$ heroku drains --json
[
  {
    "addon": null,
    "created_at": "2018-12-04T00:59:46Z",
    "id": "906262a4-e151-45d2-b35a-a2dc0ea9e688",
    "token": "d.f14da5dc-106b-468d-b1bd-bed0ed9fa1e7",
    "updated_at": "2018-12-04T00:59:47Z",
    "url": "syslog://logs.example.com"
  },
  {
    "addon": {
      "id": "130dfc11-ca20-48ea-8a29-c0da56ea2c4f",
      "name": "papertrail-curly-14926"
    },
    "created_at": "2018-10-26T21:44:01Z",
    "id": "b8c04a36-c914-4ec4-a5bd-eaa8c965d0dd",
    "token": "d.0b55c011-f49b-4428-ac2f-a5935fcffd50",
    "updated_at": "2018-10-26T21:44:02Z",
    "url": "syslog+tls://logs7.papertrailapp.com:41942"
  }
]

The example output above shows that an app has two log drains. The first drain listed was added directly by the user (so its addon field is null), and the second was added by the Papertrail add-on. Each drain’s token is available in its token field.

Use the drain token to separate log messages sent by different drains to the same drain URL. Drains to the same URL on different apps have different drain tokens. The drain token remains the same for the lifetime of the drain, but will change if you delete and re-add the same drain.

HTTPS drains

Log drains also support messaging via HTTPS. This makes it easy to write your own log-processing logic and run it on a web service (such as another Heroku app).

You must configure your web service such that Heroku can make POST requests to it. Then, you add an HTTPS drain like so:

$ heroku drains:add https://user:pass@mylogdrain.herokuapp.com/logs -a myapp

With HTTPS drains, Logplex buffers log messages and submits batches of them to an HTTPS endpoint via a POST request. The POST body contains Syslog-formatted messages, framed using the Syslog TCP protocol octet counting framing method. These batches are posted with a Content-Type header of application/logplex-1.

A Logplex POST body resembles the following:

83 <40>1 2012-11-30T06:45:29+00:00 host app web.3 - State changed from starting to up
119 <40>1 2012-11-30T06:45:26+00:00 host app web.3 - Starting process with command `bundle exec rackup config.ru -p 24405`

The following request headers are sent with every Logplex POST request:

• Logplex-Msg-Count: The number of messages encoded in the body of the request. For the example body above, this value would be 2. Use this field to ensure that you’ve parsed the body correctly.
• Logplex-Frame-Id: The unique identifier for this request. If this request is retried for some reason (non-2xx response code, network connection failure, etc.) this identifier enables you to spot duplicate requests.
• Logplex-Drain-Token: The drain token for the log drain sending the request.
• User-Agent: This describes the version of Logplex. It changes with Logplex releases (e.g., Logplex/v72).
• Content-Type: This field describes the encoding of the body for the request. Currently, this value is always application/logplex-1.

HTTPS drain caveats

• Logplex does not support chunked transfer encoding of responses. Consequently, set the Content-Length header of your response to 0, and do not return a body.
• The application/logplex-1 content type does not conform to RFC 5424. It omits STRUCTURED-DATA but does not replace it with a NILVALUE.

Security considerations

HTTPS drains support transport-level encryption using the HTTPS protocol, and authentication using HTTP Basic Authentication. Logplex currently validates the server certificate on all TLS and HTTP connections. You can opt-out of this functionality by switching to insecure mode for your TLS Syslog drain by following the instructions in the Debugging drains section below.

Syslog drains support transport-level encryption, including certificate and hostname verification during TLS handshake. However, Logplex does not support authentication for syslog drains. Therefore, it is possible for third parties to send log messages to your server if they know its address. As a security measure, you can filter out messages with unknown (or missing) drain tokens.

TLS Syslog drains fully support modern TLS, such as TLS 1.2 and TLS 1.3.

Debugging drains

When using TLS with Syslog drains, it’s possible that either the certificate or hostname verification process will fail, and your drain will stop receiving logs. This can happen for a variety of reasons, including:

• Using an expired certificate
• Using a self-signed certificate
• Certificate hostname mismatches

Errors such as these appear in the log buffer displayed by the heroku logs command. You might need to stream your app’s logs for a period of time using heroku logs -t in order to capture an error. Removing and re-adding a misbehaving drain while streaming the logs can also help track down an issue.

You can also switch to insecure mode for your TLS Syslog drain. This works similarly to curl --insecure. Transport-level encryption will remain active, however all verification checks during the TLS handshake will be disabled. Logplex uses the #insecure URI fragment to enable insecure mode:

$ heroku drains:add syslog+tls://logs.example.com:12345/#insecure -a myapp

Removing drains

You can remove a log drain from your app with the heroku drains:remove command:

$ heroku drains:remove syslog+tls://logs.example.com:12345/#insecure -a myapp

If a log drain has been created by an add-on, you cannot manually remove it. It will be removed when you detach or destroy the add-on.