Private Space Logging

Last updated November 14, 2022

Private Space Logging is available only for Shield Private Spaces.

Private Space Logging is a feature that enables you to configure log capture at the space level instead of the app level. When Private Space Logging is enabled, all log events from applications, Heroku Postgres databases and Heroku system services in the space are forwarded to a single log capture destination. Heroku API logs related to space operations are also sent to this same destination. Only users with admin permissions may set and change the space logging configuration. It is not possible to set a different log destination for apps in the space once Private Space Logging is enabled.

There are several benefits of managing logging at the space level:

  • Developers don’t have to deal with log configuration
  • Consistent log configuration is ensured for all apps in the space. One app cannot accidentally forget to capture logs or send logs to its own destination
  • Auditors can easily audit that logging is configured correctly for the space and auditors can be assured that log configuration will remain correct even as developers add and remove apps in the space.

Additionally, Private Space Logging allows you to better control the locality of log data.

Enable Private Space Logging

Private Space Logging is enabled when you create a space. You enable it by specifying a Log Drain URL when creating the space:

$ heroku spaces:create acme-space --shield --team my-team --log-drain-url https://somename:somesecret@loghost.example.com/logpath

If you do not supply a log drain when creating a Shield Space then that Shield Private Space will not have Private Space Logging enabled and will not benefit from the compliance and data-residency related enhancements associated with this feature.

You cannot turn on Private Space Logging after a space has been created. You can, however, change the Log Drain URL at a later point if the space was created with Private Space Logging enabled.

Check Logging Configuration

You can use the drains:get command to get the current space log config. If Private Space Logging was not configured for this space when it was created, the command will return a warning This Private Space does not support direct logging.

$ heroku drains:get --space acme-space
https://loghost.example.com/logpath (<id>)

Change Space Logging Configuration

You can update a space’s log destination at any time with the following command:

heroku drains:set <log-drain-url> --space <space>

Updating a space’s drain URL requires admin permissions for the associated Heroku account.

Logging format

Requests

Logs are sent as HTTPS POST requests.

Each request body can contain up to 500 log lines. A request is sent from Private Space Logging to the log drain once the log batch is equal to 500 log lines or 250 ms has elapsed, whichever comes first.

The maximum length of a single log line is 10k bytes, with longer lines split into multiple lines.

While unlikely, this means that the maximum Private Space Logging request could be as large as 5000 KB. If you intend on using a 3rd-party logging provider, you should check their logging request size limits.

Request Body

The body of a request is a series of syslog formatted messages based on RFC6587. Here is an example of a request:

POST /logs HTTP/1.1
Host: example.com
Content-Type: application/logplex-1
Logplex-Msg-Count: 10
User-Agent: log-shuttle/x (y; z; w; v)
Content-Length: 2660

266 <190>1 2017-03-15T16:10:27.008803+00:00 11844141-29d5-4425-9fd8-894759ae0dc7 heroku web.1 - - source=web.1 dyno=heroku.11844141-29d5-4425-9fd8-894759ae0dc7.91bd5d7c-b12c-47a0-9d67-daef0e4aca96 sample#load_avg_1m=0.00 sample#load_avg_5m=0.00 sample#load_avg_15m=0.00
391 <190>1 2017-03-15T16:10:27.009270+00:00 11844141-29d5-4425-9fd8-894759ae0dc7 heroku web.1 - - source=web.1 dyno=heroku.11844141-29d5-4425-9fd8-894759ae0dc7.91bd5d7c-b12c-47a0-9d67-daef0e4aca96 sample#memory_total=8.15MB sample#memory_rss=3.95MB sample#memory_cache=2.66MB sample#memory_swap=0.00MB sample#memory_pgpgin=2603pages sample#memory_pgpgout=1423pages sample#memory_quota=2560.00MB
266 <190>1 2017-03-15T16:10:57.008773+00:00 11844141-29d5-4425-9fd8-894759ae0dc7 heroku web.1 - - source=web.1 dyno=heroku.11844141-29d5-4425-9fd8-894759ae0dc7.91bd5d7c-b12c-47a0-9d67-daef0e4aca96 sample#load_avg_1m=0.00 sample#load_avg_5m=0.00 sample#load_avg_15m=0.00
391 <190>1 2017-03-15T16:10:57.009215+00:00 11844141-29d5-4425-9fd8-894759ae0dc7 heroku web.1 - - source=web.1 dyno=heroku.11844141-29d5-4425-9fd8-894759ae0dc7.91bd5d7c-b12c-47a0-9d67-daef0e4aca96 sample#memory_total=8.15MB sample#memory_rss=3.95MB sample#memory_cache=2.66MB sample#memory_swap=0.00MB sample#memory_pgpgin=2603pages sample#memory_pgpgout=1423pages sample#memory_quota=2560.00MB
266 <190>1 2017-03-15T16:11:27.008840+00:00 11844141-29d5-4425-9fd8-894759ae0dc7 heroku web.1 - - source=web.1 dyno=heroku.11844141-29d5-4425-9fd8-894759ae0dc7.91bd5d7c-b12c-47a0-9d67-daef0e4aca96 sample#load_avg_1m=0.00 sample#load_avg_5m=0.00 sample#load_avg_15m=0.00
391 <190>1 2017-03-15T16:11:27.009436+00:00 11844141-29d5-4425-9fd8-894759ae0dc7 heroku web.1 - - source=web.1 dyno=heroku.11844141-29d5-4425-9fd8-894759ae0dc7.91bd5d7c-b12c-47a0-9d67-daef0e4aca96 sample#memory_total=8.15MB sample#memory_rss=3.95MB sample#memory_cache=2.66MB sample#memory_swap=0.00MB sample#memory_pgpgin=2603pages sample#memory_pgpgout=1423pages sample#memory_quota=2560.00MB
266 <190>1 2017-03-15T16:11:57.008775+00:00 11844141-29d5-4425-9fd8-894759ae0dc7 heroku web.1 - - source=web.1 dyno=heroku.11844141-29d5-4425-9fd8-894759ae0dc7.91bd5d7c-b12c-47a0-9d67-daef0e4aca96 sample#load_avg_1m=0.00 sample#load_avg_5m=0.00 sample#load_avg_15m=0.00
391 <190>1 2017-03-15T16:11:57.009221+00:00 11844141-29d5-4425-9fd8-894759ae0dc7 heroku web.1 - - source=web.1 dyno=heroku.11844141-29d5-4425-9fd8-894759ae0dc7.91bd5d7c-b12c-47a0-9d67-daef0e4aca96 sample#memory_total=8.15MB sample#memory_rss=3.95MB sample#memory_cache=2.66MB sample#memory_swap=0.00MB sample#memory_pgpgin=2603pages sample#memory_pgpgout=1423pages sample#memory_quota=2560.00MB

Each message starts with a number and a space. The number indicates the length of the rest of the message after the space. The rest of the message is formatted as a syslog message according to RFC5424. It has the following elements:

  • A “PRIVAL” number in angular brackets. This number is the octal sum of the syslog facility number and the syslog severity number.
  • A version number right after the angular brackets.
  • The actual log message contents. It can be one of several types of Heroku logs

It is the same format used by Logplex to receive log messages. Consult RFC5424 for more details on the format including a full ABNF definition.

Find log messages from a specific Heroku app

When log messages are sent via Private Space Logging from the app, the runtime control plane and the Heroku router, each message has the Heroku app UUID as its first element. For example, the message in the sample above came from the application with uuid 11844141-29d5-4425-9fd8-894759ae0dc7. You can find the UUID of your heroku apps using heroku info with the --json option:

$ heroku info -a myapp --json
{
  ...,
  "app": {
    ...,
    "id": "1234abcd-352a-4931-1234-1bb5e6f4afa2",
    ...
  }
}

Data residency

When Private Space Logging is enabled, log data will be sent directly from the source to the configured log destination. It will not be routed through Logplex. This enables better control of data residency. For example, if a Private Space is located in Frankfurt and the log destination is in Frankfurt as well, then the log data will be sent directly to the destination via standard Internet routing. It will not be routed through other Heroku services.

Some Heroku services are not co-located with Private Spaces. Heroku API and Heroku Build are examples of global services. These services will send logs directly from where they are located to the log destination configured for the space. Therefore, data residency only applies to log data sent from applications and services residing inside the space. These services include the runtime control plane, HTTP proxy and any in-space data services such as Heroku Private Postgres, Private Redis and Private Kafka.

Capture logs

You can forward logs to any logging service that supports the HTTPS-based logging format described above. This includes cloud service providers, a local instance of an existing logging platform, or your own log capture service that programmatically decides how to store or route logs.

The log drain destination must be a publicly resolvable and addressable URL. It cannot be a space-internal URL or a private URL in (for example) a peered VPC. Components outside the Heroku Private Space itself (like Heroku Data and Heroku API) will also be sending logs to the drain destination and must be able to resolve the URL

Splunk

Splunk Enterprise can ingest Private Space logs using the HTTP Event Collector and RFC5424 Syslog add-on:

  1. Install the RFC5424 Syslog add-on.

  2. Create a new HTTP Event Collector token. Follow the Splunk documentation and fill out the fields as follows:

    • Name: The name or other unique identifier of your space. It is recommended to create a token per space to differentiate events.
    • Source Type: rfc5424_syslog

  3. Generate a random channel UUID. This is required to for raw event collection by Splunk.

  4. Construct your Log Drain URL with the token and channel created in the steps above:

    https://x:TOKEN@yoursplunkhost.example.com:yourport/services/collector/raw?channel=CHANNEL

    For example:

    https://x:1234ABCD-C66E-4B22-854F-5958C7FA637D@yoursplunkhost.domain.com:yourport/services/collector/raw?channel=12341238-adbd-abcd-9dbe-16629695fb1d

  5. Set the Log Drain URL on your Private Space as explained above.

Sumo Logic

You may configure your Heroku Shield Private Space to send logs to Sumo Logic by following these steps:

  1. Create an app in Heroku that generates logs.
  2. Ensure you have installed the Heroku CLI.
  3. Configure a Sumo Logic Hosted Collector with an HTTP Source.
  4. Point a Heroku log drain to the URL for the HTTP Source you created above.

For more details, please see this article from Sumo Logic.

LogDNA

To configure your Heroku Shield Private Space to send logs to LogDNA, do the following:

  1. Ensure you have installed the Heroku CLI.

  2. Go to www.logdna.com and create an account by clicking the Get Started button.

  3. Click the link in the registration email to verify your email address and set a password for your LogDNA account.

  4. Open the LogDNA webapp and click All Hosts > Add a host.

  5. Navigate to the Heroku section and execute the account-specific commands found under the Installing via Heroku Log Drains section.

You should now be able to see your application’s logs in the LogDNA web app.

Coralogix

To configure your Heroku Shield Private Space to send logs to Coralogix, do the following:

  1. Create an account in Coralogix.

  2. Get the new account private-key and company ID by going to Settings → Send Your Logs. Choose an application-name to be associated with the logs.

  3. Point the Heroku log drain to the following URL: https://(redacted)@api.coralogix.com/logs/heroku/private?appName=

Watch logs stream into Coralogix. For more information, see the Coralogix Tutorials.

Run a custom log capture service

log-iss is an open source implementation of a log capture service. It is written in Go and it allows you to forward logs to a syslog destination of your choice. You can run this application on Heroku or on another infrastructure of your choice. You can fork the code and customize it to forward logs to ElasticSearch, save to the file system store in S3, etc.

Limitations with Private Space Logging

Cannot use Logplex features

When Private Space Logging is enabled, log messages are sent directly to the log destination and are not handled by Logplex. Features provided by Logplex are therefore not available when Private Space Logging is enabled. These include:

  • Logs cannot be forwarded to multiple log drains
  • Logs cannot be viewed in CLI with heroku logs or in the Dashboard log viewer.

Private Space Logging does not support logging to drains only accessible via private connection

If you have an on-premises logging system (such as Splunk) and a VPN connection from your on-premises network to your Private Space, Private Space Logging will not be able to access the on-premises logging system via the private connection. An on-premises logging system must be exposed to the public internet.

Feedback

Log in to submit feedback.