Heroku Private Spaces
Last updated 18 June 2018
Table of Contents
Private Spaces are currently available only in Heroku Enterprise.
Private Spaces are dedicated environments for running dynos and certain types of add-ons within an isolated network. Access to apps in a Private Space can be controlled at the network level. Outbound requests from apps in a Private Space originate from a set of stable IP addresses, which allows you to securely communicate with IP white-listed services on-premise or on other networks. Private Spaces enable you to build and run Heroku apps that meet strict requirements for data protection and change control, as commonly required by corporate and regulatory governance standards.
There are two types of Private Space: the standard Private Space and the Shield Private Space. The Shield Private Space includes additional features for building and running high-compliance applications.
You can deploy to Private Spaces with the familiar
git push heroku master command or take full advantage of Heroku Pipelines. The powerful governance and security features provided by Private Spaces do not change the standard Heroku developer experience.
Private Space management
Only team administrators can create, destroy, or modify Private Spaces. All team members can view details about Private Spaces in the team.
Private Spaces can be created in the following regions:
|oregon||Oregon, United States|
|virginia||Virginia, United States|
To create a Private Space in a given region, simply specify the location when creating the space.
The region cannot be changed after the Private Space is created. All apps created in a Private Space are in the same region as the space.
Not all add-ons are available in all regions. To view which add-ons are available in a region, see Add-on Runtime Availability.
Create a Private Space
Only team administrators can create a new Private Space.
To create a Private Space using Dashboard, click the Spaces tab inside your team and click Create new Private Space. When creating the Private Space, you can optionally choose a region (geographic location) for your apps and data services. When you fill out the form and click Create Private Space, your new space is created within ten minutes.
To create a Private Space using the Heroku CLI, use the
$ heroku spaces:create my-space-name --team my-team-name Creating space my-space-name in team my-team-name... done === my-space-name Team: my-team-name Region: virginia State: allocating
To create a Private Space in a particular region, specify the
--region option with the region’s name from the list of regions. For example:
$ heroku spaces:create my-space-name --team my-team-name --region tokyo Creating space my-space-name in team my-team-name... done === my-space-name Team: my-team-name Region: tokyo State: allocating
Creating a new Private Space can take a few minutes. To track its status, use the
$ heroku spaces:wait my-space-name Waiting for space my-space to allocate... done
View information about a Private Space
All team members can view information about the Private Spaces in an team.
In Dashboard, click the Spaces tab inside your team. The Space info displays the team to which the Space belongs, as well as the current state of the Space. The
allocating state indicates that the Space is being set up and is not yet operational. When it becomes operational, the state will change to
In the Heroku CLI, use the
$ heroku spaces:info --space my-space-name === my-space-name ID: 12345678-abcd-1234-abcd-12345678abcd Team: acme Region: Tokyo, Japan State: allocated Shield: on Outbound IPs: 220.127.116.11, 18.104.22.168, 22.214.171.124, 126.96.36.199 Created at: 2016-10-13T05:36:15Z
Trusted IP ranges
Only team administrators can manage trusted IP ranges for a Private Space.
Each Private Space has a set of trusted IP ranges, with each range represented in CIDR block notation (for example,
192.0.2.0/24). Only clients originating from one of these trusted IP ranges can access web processes running in the Private Space. Use trusted IP ranges to restrict traffic to apps that come from your corporate network or from a CDN service that proxies traffic for your apps.
Trusted IP ranges only restrict access to applications in the Private Space. They do not control from which source IP ranges you can execute CLI commands or connect to Dashboard.
Trusted IP ranges can be managed in Dashboard by opening the Network tab for a Private Space.
List current trusted IP ranges for a Private Space using the CLI:
$ heroku trusted-ips --space acme-prod === Trusted IP Ranges 192.0.2.0/26 192.0.2.64/26
Add a new range using the CLI:
$ heroku trusted-ips:add 192.0.2.128/26 --space acme-prod Added 192.0.2.128/26 to trusted IP ranges on acme-prod ▸ WARNING: It may take a few moments for the changes to take effect.
Trusted IP ranges for data services
This is a beta feature. Email firstname.lastname@example.org to have Trusted IPs for data products. Once your private space is added to the Trusted IPs for data products beta, you will have to remove and re-add any trusted IPs you want to apply to both runtime and data.
By default, trusted IP ranges only apply to web processes running in the space. Data services, like Heroku Postgres, Heroku Redis and Apache Kafka on Heroku, can only be accessed from dynos within the space. You can optionally choose to also allow access from trusted IP addresses to data services in the space. During the beta, this feature can only be controlled by a Heroku operator. Email email@example.com to have it turned on or off. Some caveats do exist for this feature:
0.0.0.0/0will be ignored as a Trusted IP for data products because this CIDR block would expose the database to the wider Internet.
- Existing Trusted IPs will not be applied until a new Trusted IP is added.
- Granular controls do not exist. A Trusted IP will be able to reach both web dynos and data products.
Stable outbound IP addresses
All team members can view the list of stable outbound IP addresses for a Private Space.
All outbound traffic from apps in a Private Space will originate from a small, stable list of IP addresses that are dedicated to the space. This allows you to use IP whitelisting to secure services being accessed by apps in the space. For example, you can configure a web services gateway in your corporate data center to only allow access from the IP addresses assigned to the Private Space. By using this in combination with TLS and application level authentication tokens, you get an extra level of security for protecting resources in your corporate data center.
You can find the stable outbound IP addresses for a Space on the Settings tab for the Space in Dashboard. You can also use the CLI:
$ heroku spaces:info --space acme-prod === acme-prod ID: 12345678-ed4a-4e96-b998-3fcb499439e0 Team: acme Region: virginia State: allocated Outbound IPs: 192.0.2.2, 192.0.2.3, 192.0.2.4, 192.0.2.5 Created at: 2015-08-07T16:16:56Z
Destroy a Private Space
Only team administrators can destroy a Private Space. Private Spaces that contain applications or other resources cannot be destroyed. Apps must be explicitly deleted before destroying a Private Space.
In the Heroku dashboard, click “Space Settings” and choose “Delete Space”.
In the Heroku CLI, use the
$ heroku spaces:destroy --space my-space-name Destroying space my-space-name... done === my-space-name Team: my-team-name State: deleting
Managing apps in Private Spaces
A Private Space represents a trust boundary within which your team can deploy and manage apps that handle sensitive data. Therefore, the ability to create apps in a Private Space is constrained to team administrators and those who have been granted the “app creation” permission for a particular space. Once an app has been created, the administrator can give other members additional access to the app using fine-grained access controls.
Create an app in a Private Space
The only users who can create apps in a Private Space are:
- Team administrators
- Team members that have been granted the “app creation” permission by an administrator. (This can be done in the Heroku Dashboard by clicking on the Space and then clicking Access.)
In the Heroku Dashboard, click on a space and choose Create New App.
In the Heroku CLI use the
$ heroku create my-space-app --space my-space-name Creating my-space-app in space my-space-name... done, stack is cedar-14 http://my-space-app.herokuapp.com/ | https://git.heroku.com/my-space-app.git
After the app is created, you can grant other team members access using one of several app permissions. Only members who are trusted to access any sensitive data protected by the space should be granted more than view permissions. All other members, who may develop on the app but who do not need production access, should only be granted access to staging and development environments.
Deploy an app in a Private Space
Code can be deployed (pushed) to Private Space apps in the same ways as with Common Runtime apps (such as by using
git push heroku master or a GitHub integration).
About delays and downtime during new Private Space app releases
The first time an app is deployed (pushed) to a Private Space, it can take several minutes for the app to become available because of the time needed to set up the underlying dedicated infrastructure (e.g., routing, DNS, and the dynos).
For subsequent releases (code deployments or config var changes) there is a much shorter delay, and users may experience little to no downtime. There may be brief downtime if there is only one dyno for the app. If there are multiple dynos, zero downtime is possible. This is because of a Private Spaces feature called rolling deploys.
Rolling deploys are analogous to Preboot on the Common Runtime. Both enable zero downtime during new releases. Preboot does this by ensuring that new web dynos are started (and receiving traffic) before the existing ones are terminated. Rolling deploys do this by stopping and changing only 25% (rounded up) at a time of the existing dynos in each of dyno types including
web dynos and other worker dynos, while the remaining dynos handle requests and tasks. The effect is that all requests are handled (no user downtime) while the new release is pushed gradually to the entire formation.
To be more precise, because of rolling deploys, there will be zero downtime during a new release for a Private Space app if all of the following are true:
- The app has been deployed at least once and is running.
- The app has at least two dynos and is not currently at capacity (that is, it has enough dynos that it can still service requests with about 25% of its dynos missing).
- Any databases attached to the app are not incurring schema changes and are compatible with both the old and new app release (forwards- and backward-compatible).
Scaling an existing app in a Private Space does not incur downtime (as long as there is enough capacity to handle the existing load). Changing the dyno type does incur a short downtime, under some circumstances, while the new dyno types are created and traffic is switched over.
List apps in a Private Space
All team members can view all apps in a Private Space:
$ heroku apps --space my-space-name === Apps in space my-space-name my-space-app-1 my-space-app-2
View info about an app in a Private Space
All team admins can detailed info about a single app in a Private Space:
$ heroku info --app my-space-app === my-space-app Collaborators: firstname.lastname@example.org Git URL: https://git.heroku.com/my-space-app.git Owner: my-team-name Region: tokyo Space: my-space-name Stack: cedar-14 Web URL: http://my-space-app.herokuapp.com/
Destroy an app in a Private Space
Team admins and members who have manage permission on an app can destroy it:
$ heroku apps:destroy -a my-space-app Destroying my-space-app (including all add-ons)... done
Adding custom domains and SSL
Only team administrators and members who have manage permission on an app can add custom domains and manage SSL.
SSL and custom domains for apps in Spaces are handled similarly to apps in the Common Runtime, with a few important exceptions:
- Apps in Private Spaces must use a custom domain and SSL Certificate to enable HTTPS. The default app domain (e.g. http://example.herokuapp.com) will work with HTTP, but will not work with HTTPS. If your app automatically redirects HTTP requests to your HTTPS URL, you will not be able to access your app using the default app domain. In order to use the default app domain, you will need to disable the redirection for hosts like
- Apps in Private Spaces currently require TLS connections to use the SNI extension. Older clients that do not send the SNI extension will not be able to connect to a Private Space App using https.
- Once you have purchased and configured a custom domain, and purchased an SSL certificate, you can skip the step of running
heroku addons:create ssl:endpoint. (The Heroku SSL endpoint comes for free on apps in Private Spaces.)
Scaling and One-Off Dynos
Scaling your formation follows the same pattern within Private Spaces as with the Common Runtime. For example:
$ heroku ps:scale web=2 worker=1
“One-off” dynos are also possible via
$ heroku run bash Running bash... ▸ WARNING: Warning: Dynos can take up to a few minutes to be provisioned in Private Spaces.
Scaling and one-off dynos require that a new dyno is provisioned and launched within your Space. This process takes a few minutes to complete.
Shield Private Spaces
Shield Private Spaces include additional features for building high-compliance applications. Shield Private Spaces come at an additional cost, and it is recommended that you consult your Solution Engineer or Account Executive to decide if a Shield Private Space is appropriate for your application requirements. Shield Private Spaces differ from standard Private Spaces in the following ways:
- Only dynos of the
shieldtype can run in a Shield Private Space. Shield dynos have an encrypted ephemeral file system and are available in small, medium, and large sizes.
- Shield Private Spaces allow the use of
shieldHeroku Postgres plans. These databases allow storage of certain regulated data classes that cannot be stored in
privateHeroku Postgres plans.
- Shield Private spaces allow the use of the
shieldHeroku Connect plan.
- In a Shield Private Space, all input typed into an interactive
heroku runsession is logged to the app or space log stream. This allows you to meet specific compliance requirements for production access auditing.
- Shield Private Spaces allow you to manage logging at the space level, configuring a single log drain for all applications in the space.
- Shield Private Spaces enforce stricter requirements for TLS termination. TLS 1.0 cannot be used to connect to applications in a Shield Private Space.
To create a Shield Private Space using the Heroku CLI, specify the
$ heroku spaces:create my-shield-space-name --shield --team my-team-name Creating space my-space-name in team my-team-name... done === my-space-name Team: my-team-name Region: virginia State: allocating
To enable Private Space Logging, specify a log drain URL when creating the space:
Private Space Logging cannot be enabled after a Private Space has been created.
$ heroku spaces:create my-shield-space-name --shield --team my-team-name --log-drain-url https://mylogdrain.com Creating space my-space-name in team my-team-name... done === my-space-name Team: my-team-name Region: virginia State: allocating
To create a one-off dyno, you must first add an SSH key to your user account (this only has to be done once):
$ heroku keys:add Generating public/private rsa key pair. ... $ heroku run bash ... ~ $
Add-ons and Private Spaces
If you attempt to provision an add-on for an app in a Heroku Private Space and that add-on is not available in that app’s Private Spaces region, you will see a message saying so.
To view which add-ons are available in a region, go to the Add-on Runtime Availability. Select the Private Spaces tab to view information for Private Spaces regions.
You can also view this information in the Region Availability section of the Elements listing page for each add-on.
Add-ons marked as Available in a Private Space
Many add-ons offer plans that are compatible with Private Space regions, even though they operate outside the Private Space. This means that network traffic between your private dyno and the add-on resource travels across the public Internet. These add-ons are marked as “Available” with a green dot in Add-on Runtime Availability.
You can add these add-ons to apps running in a Private Space as you would normally for our Common Runtime:
$ heroku addons:create newrelic:wayne -a appname Creating newrelic-cubic-72448... done, (free) Adding newrelic-cubic-72448 to appname... done Setting NEW_RELIC_LICENSE_KEY, NEW_RELIC_LOG and restarting appname... done, v80 Use `heroku addons:docs newrelic` to view documentation.
Some add-ons of type “datastore” are compatible with Private Spaces regions. They can be provisioned through the CLI with the
addons:create command, or they can be provisioned in the Dashboard. Either way, you will have to confirm that you understand that this use of a data store not contained within the Private Space network boundary is explicitly intended; that using this add-on plan will send data over the public Internet.
To provision from the CLI you must follow the confirmation prompt:
$ heroku addons:create <addon name> -a appname ! This add-on is not automatically networked with this Private Space. ! To proceed, type "appname" or re-run this command with --confirm appname
For information on Heroku add-on credits and overage payments mechanisms, see the Heroku Enterprise documentation
Add-ons marked as Available and Installable in a Private Space
Some add-ons are installable in Private Spaces. This means that all network traffic between dynos and the add-on resource stays within the private network of the space, and the add-on resource does not have to be exposed on the public Internet. These add-ons are marked as “Available & Installable in Private Space” with a blue dot in Add-on Runtime Availability.
Heroku Postgres and Heroku Redis are examples of add-ons that are created inside Heroku Private Spaces. When you select a Heroku Postgres or Heroku Redis private plan for an app in a Private Space, the resource will be automatically created in the same region as the Heroku Private Space and peered to the Heroku Private Space.
To create a Heroku Postgres instance inside your Heroku Private Space, you must select a
private-0 plan or above. Since this Add-On will be automatically created inside your Space, confirmation is not required.
$ heroku addons:create heroku-postgresql:private-0 -a your-private-app-name
It is possible to provision Heroku Postgres and Heroku Redis plans outside of your Space. Provision by specifying app, plan, and region using the
$ heroku addons:create heroku-postgresql:standard-4 --region=us -a your-private-app-name
You will be asked to confirm the addition of the plan via the CLI.
! This add-on is not automatically networked with this Private Space. ! To proceed, type "appname" or re-run this command with --confirm appname
Migrate apps and data to or from a Private Space
There is no automated feature, at this time, to migrate apps and data to or from a Private Space, but it can be done manually. You should first back up and migrate the database(s), if any. App code should then be re-pushed to the destination. You should also consider differences in the way Private Spaces operate and whether any add-ons the app uses will work or need to be configured differently in the destination. For instance, SSL for apps in Private Spaces works slightly differently.
Default limits for Private Spaces
Private Spaces Runtime are built on a different architecture than the Common Runtime, and Private Spaces has different default limits. To have limits increased for any of your private spaces, please open a support ticket.
The current default limits for newly created spaces are:
- 20 Trusted IP ranges
- 25 apps
- 100 process types (10 dynos per process type)
- 100 dynos for the entire Space