Last updated 16 February 2017
Table of Contents
Heroku’s Logplex logging service makes it seamless to collect logs from your applications and forward them to archival, search and alerting services offered by one of several logging add-on providers. In addition, you can also use your own log drain and take full control of how your application logs are processed.
Heroku supports two types of log drains:
- Syslog drains
- HTTPS drains
Syslog drains allow you to forward your Heroku logs to an external syslog server for long-term archiving. Heroku supports both secure (TLS) and insecure syslog drains. You must configure the service or your server to be able to receive syslog packets from Heroku, and then add its syslog URL (which contains the host and port) as a syslog drain:
$ heroku drains:add syslog+tls://logs.example.com:12345 -a myapp
Plain Text Syslog
$ heroku drains:add syslog://logs.example.com -a myapp
Heroku uses local0, local3 and local7 for its syslog facilities. This is subject to change at any time.
When you add a drain to an application, it is assigned a unique drain ID that will look something like
d.9173ea1f-6f14-4976-9cf0-7cd0dafdcdbc. You can discover the drain IDs for your application by running
$ heroku drains syslog://logs.example.com (d.9173ea1f-6f14-4976-9cf0-7cd0dafdcdbc)
This identifier will be written into all log messages from that drain. For example:
2013-01-01T01:01:01.000000+00:00 d.9173ea1f-6f14-4976-9cf0-7cd0dafdcdbc app[web.1] Your message here.
The ID can be used to separate log messages sent by different drains to the same drain URL. Drains to the same URL on different apps will have different IDs. The drain ID will remain the same for the lifetime of the drain, but will change if you delete and re-add the same drain.
While syslog drains make it easy to drain logs to a standard off-the-shelf syslog based server of your choice, Heroku also supports the HTTPS protocol for sending logs to a drain service so that you can more easily write your own log processing logic and run it on a service like Heroku itself.
Like with a syslog drain, you must configure the service such that Heroku can make requests to it (for example by deploying it as a Heroku app). Then, a HTTPS drain is added in a similar manner:
$ heroku drains:add https://user:email@example.com/logs -a myapp
With HTTPS drains, Logplex buffers log messages and submits batches of messages to an HTTPS endpoint via a
POST request. The
POST body contains syslog formatted messages, framed using the syslog TCP protocol octet counting framing method. These batches are posted with a “Content-Type” header of “application/logplex-1”.
83 <40>1 2012-11-30T06:45:29+00:00 host app web.3 - State changed from starting to up 119 <40>1 2012-11-30T06:45:26+00:00 host app web.3 - Starting process with command `bundle exec rackup config.ru -p 24405`
The following request headers are also sent:
- Logplex-Msg-Count: The number of messages encoded in the body of this request. 2 in the example above. You can use this field as a sanity check to detect if you have not parsed the body correctly.
- Logplex-Frame-Id: The unique identifier for this request. If this request is retried for some reason (non-2xx response code, network connection failure, etc.) this identifier will allow you to spot duplicate requests.
Logplex-Drain-Token: This is the unique identifier for the Logplex drain. It will be the same identifier you see if you run
heroku drainsfor your app.
- User-Agent: This describes the version of Logplex, and is changed with Logplex releases. (e.g. Logplex/v72)
- Content-Type: This field describes the encoding of the body for the request. “application/logplex-1” is the only value at this time.
HTTPS drain caveats
- Logplex does not support chunked transfer encoding of responses, so make sure to not return a body in the response and to set the “Content-Length” header to “0”.
- “application/logplex-1” does not conform to RFC5424. It leaves out
STRUCTURED-DATAbut does not replace it with a
HTTPS drains support transport level encryption using the HTTPS protocol and authentication using HTTP Basic Authentication. However, Logplex does not currently validate the server certificate during TLS handshake.
Syslog drains support transport level encryption including certificate and hostname verification during TLS handshake. However, Logplex does not support authentication for syslog drains. Therefore, it is possible for third parties to send log messages to your server if they know its address. As a security measure, you can filter out messages with unknown (or missing) drain IDs.
TLS Syslog drains fully support modern TLS such as, TLS 1.2 and TLS 1.1. Older TLS (v1.0) and all versions of SSL (v1,2,3) are not supported. Modern Galois/Counter Mode (GCM) based ciphers are preferred, however older Cipher Block Chaining (CBC) is still supported. However, Logplex utilizes only 128 bit AES variants of these ciphers.
With the introduction of TLS Syslog drains it is possible that either the certificate or hostname verifcation process will fail and your drain will stop receiving logs. This can happen for a variety of reasons including:
- use of an expired certificate
- use of a self-signed certificate
- certificate hostname mismatches
Errors, such as the ones listed above will appear in the buffer displayed by the
heroku logs command. It may be necessary to stream the logs for a period of time using
heroku logs -t in order to capture an error. Removing and re-adding the drain while streaming the logs can also help track down an issue.
Furthermore, you can switch to insecure mode for your TLS Syslog drain. This works much the same way as
curl --insecure. Transport level encryption will remain active, however all verification checks during the TLS handshake will be disabled. Logplex uses the URI fragment to enable insecure mode:
$ heroku drains:add syslog+tls://logs.example.com:12345/insecure -a myapp
If you do not wish to have a drain anymore, you can remove it easily by running:
$ heroku drains:remove syslog+tls://logs.example.com:12345/insecure -a myapp