Heroku Private Spaces
Last updated 10 September 2019
Table of Contents
Private Spaces are currently available only in Heroku Enterprise.
Private Spaces are dedicated environments for running dynos and certain types of add-ons within an isolated network. Access to apps in a Private Space can be controlled at the network level. Outbound requests from apps in a Private Space originate from a set of stable IP addresses, which allows you to securely communicate with IP white-listed services on-premise or on other networks. Private Spaces enable you to build and run Heroku apps that meet strict requirements for data protection and change control, as commonly required by corporate and regulatory governance standards.
There are two types of Private Space: the standard Private Space and the Shield Private Space. The Shield Private Space includes additional features for building and running high-compliance applications.
You can deploy to Private Spaces with the familiar
git push heroku master command or take full advantage of Heroku Pipelines. The powerful governance and security features provided by Private Spaces do not change the standard Heroku developer experience.
Private Space management
Only team administrators can create, destroy, or modify Private Spaces. All team members can view details about Private Spaces in the team.
Private Spaces can be created in the following regions:
|oregon||Oregon, United States|
|virginia||Virginia, United States|
To create a Private Space in a given region, simply specify the location when creating the space.
The region cannot be changed after the Private Space is created. All apps created in a Private Space are in the same region as the space.
Not all add-ons are available in all regions. To view which add-ons are available in a region, see Add-on Runtime Availability.
Create a Private Space
Only team administrators can create a new Private Space.
To create a Private Space using Dashboard, click the Spaces tab inside your team and click Create new Private Space. When creating the Private Space, you can optionally choose a region (geographic location) for your apps and data services. When you fill out the form and click Create Private Space, your new space is created within ten minutes.
To create a Private Space using the Heroku CLI, use the
$ heroku spaces:create my-space-name --team my-team-name Creating space my-space-name in team my-team-name... done === my-space-name Team: my-team-name Region: virginia State: allocating
Creating a new Private Space can take a few minutes. To track its status, use the
$ heroku spaces:wait my-space-name Waiting for space my-space to allocate... done
To create a Private Space in a particular region, specify the
--region option with the region’s name from the list of regions. For example:
$ heroku spaces:create my-space-name --team my-team-name --region tokyo Creating space my-space-name in team my-team-name... done === my-space-name Team: my-team-name Region: tokyo State: allocating
Choosing Private Network CIDR Range
If you wish to connect your Private Space to other networks, such as an Amazon VPC or an on-premises network, you may wish to specify the dyno or data CIDR ranges that your Private Space will use to ensure no overlap with your private network.
The data CIDR range is used for Heroku private data add-ons.
To create a space with a custom dyno or data VPC CIDR range, the
--data-cidr flags can be specified via the CLI.
$ heroku spaces:create --cidr 172.16.0.0/16 --data-cidr 10.100.0.0/20 --space my-space-name Creating space my-space-name in team my-team-name... done === my-space-name Team: my-team-name Region: tokyo State: allocating
/16CIDR range is required for the dyno VPC (
/20is the minimum CIDR range size for the data VPC (
- You cannot specify a CIDR range that overlaps with
- After Private Space creation, these dyno and data CIDR values cannot be changed
- Specifying the Private Space CIDR ranges can only be done via the CLI (and not the Dashboard)
View information about a Private Space
All team members can view information about the Private Spaces in a team.
In Dashboard, click the Spaces tab inside your team. The Space info displays the team to which the Space belongs, as well as the current state of the Space. The
allocating state indicates that the Space is being set up and is not yet operational. When it becomes operational, the state will change to
In the Heroku CLI, use the
$ heroku spaces:info --space my-space-name === my-space-name ID: 12345678-abcd-1234-abcd-12345678abcd Team: acme Region: Tokyo, Japan State: allocated Shield: on Outbound IPs: 220.127.116.11, 18.104.22.168, 22.214.171.124, 126.96.36.199 Created at: 2016-10-13T05:36:15Z
Trusted IP ranges
Only team administrators can manage trusted IP ranges for a Private Space.
Each Private Space has a set of trusted IP ranges, with each range represented in CIDR block notation (for example,
192.0.2.0/24). Only clients originating from one of these trusted IP ranges can access web processes running in the Private Space. Use trusted IP ranges to restrict traffic to apps that come from your corporate network or from a CDN service that proxies traffic for your apps.
When a new Private Space is created, it’s configured with a default trusted IP range of
0.0.0.0/0 which will admit traffic from the entire internet.
Trusted IP ranges only restrict access to applications in the Private Space. They do not control from which source IP ranges you can execute CLI commands or connect to Dashboard.
Trusted IP ranges can be managed in Dashboard by opening the Network tab for a Private Space.
List current trusted IP ranges for a Private Space using the CLI:
$ heroku trusted-ips --space acme-prod === Trusted IP Ranges 192.0.2.0/26 192.0.2.64/26
Add a new range using the CLI:
$ heroku trusted-ips:add 192.0.2.128/26 --space acme-prod Added 192.0.2.128/26 to trusted IP ranges on acme-prod ▸ WARNING: It may take a few moments for the changes to take effect.
To open up a Private Space to traffic from the whole Internet (the default for newly created spaces), add the CIDR range
Trusted IP ranges for data services
This is a beta feature. Open a ticket at help.heroku.com to request to have Trusted IPs for data products enabled.
Trusted IP ranges for data services are not available in Shield Private Spaces.
By default, trusted IP ranges only apply to web processes running in the space. Data services, like Heroku Postgres, Heroku Redis and Apache Kafka on Heroku, can only be accessed from dynos within the space. You can optionally choose to also allow access from trusted IP addresses to data services in the space. During the beta, this feature can only be controlled by a Heroku operator. Open a ticket at help.heroku.com to request to have Trusted IPs for data products enabled or disabled. Some caveats do exist for this feature:
0.0.0.0/0will be ignored as a Trusted IP for data products because this CIDR block would expose the database to the wider Internet.
- Granular controls do not exist. A Trusted IP will be able to reach both web dynos and data products.
Stable outbound IP addresses
All team members can view the list of stable outbound IP addresses for a Private Space.
All outbound traffic from apps in a Private Space will originate from a small, stable list of IP addresses that are dedicated to the space. This allows you to use IP whitelisting to secure services being accessed by apps in the space. For example, you can configure a web services gateway in your corporate data center to only allow access from the IP addresses assigned to the Private Space. By using this in combination with TLS and application level authentication tokens, you get an extra level of security for protecting resources in your corporate data center.
You can find the stable outbound IP addresses for a Space on the Settings tab for the Space in Dashboard. You can also use the CLI:
$ heroku spaces:info --space acme-prod === acme-prod ID: 12345678-ed4a-4e96-b998-3fcb499439e0 Team: acme Region: virginia State: allocated Outbound IPs: 192.0.2.2, 192.0.2.3, 192.0.2.4, 192.0.2.5 Created at: 2015-08-07T16:16:56Z
Destroy a Private Space
Only team administrators can destroy a Private Space. Private Spaces that contain applications or other resources cannot be destroyed. Apps must be explicitly deleted before destroying a Private Space.
In the Heroku dashboard, click “Space Settings” and choose “Delete Space”.
In the Heroku CLI, use the
$ heroku spaces:destroy --space my-space-name Destroying space my-space-name... done === my-space-name Team: my-team-name State: deleting
If your app requires high availability, we suggest that you scale up your dyno formation to a minimum of 2 dynos for each process type. Dynos are provisioned via round robin in one of two availability zones (AZ) associated with the Private Space. This helps maintain application availability in the unlikely scenario of an AZ failure.
If an AZ becomes unhealthy or unavailable, Private Spaces will automatically reschedule your dyno on a working AZ. When the unhealthy AZ returns to a healthy state, Private Spaces automatically redistributes your dynos.
Rolling deploys with zero downtime
The first time an app is deployed (pushed) to a Private Space, it can take several minutes for the app to become available because of the time needed to set up the underlying dedicated infrastructure (e.g., routing, DNS, and the dynos).
For subsequent releases (code deployments or config var changes) there is a much shorter delay, and users may experience little to no downtime. There may be brief downtime if there is only one dyno for the app. If there are multiple dynos, zero downtime is possible. This is because of a Private Spaces feature called rolling deploys.
Rolling deploys are analogous to Preboot on the Common Runtime. Both enable zero downtime during new releases. Preboot does this by ensuring that new web dynos are started (and receiving traffic) before the existing ones are terminated. Rolling deploys do this by stopping and changing only 25% at a time of the existing dynos in each of dyno types including
web dynos and other worker dynos, while the remaining dynos handle requests and tasks. The effect is that all requests are handled (no user downtime) while the new release is pushed gradually to the entire formation.
To be more precise, because of rolling deploys, there will be zero downtime during a new release for a Private Space app if all of the following are true:
- The app has been deployed at least once and is running.
- The app has at least two dynos and is not currently at capacity (that is, it has enough dynos that it can still service requests with about 25% of its dynos missing).
- Any databases attached to the app are not incurring schema changes and are compatible with both the old and new app release (forwards- and backward-compatible).
Scaling an existing app in a Private Space does not incur downtime (as long as there is enough capacity to handle the existing load). Changing the dyno type does incur a short downtime, under some circumstances, while the new dyno types are created and traffic is switched over.
Managing apps in Private Spaces
A Private Space represents a trust boundary within which your team can deploy and manage apps that handle sensitive data. Therefore, the ability to create apps in a Private Space is constrained to team administrators and those who have been granted the “app creation” permission for a particular space. Once an app has been created, the administrator can give other members additional access to the app using fine-grained access controls.
Create an app in a Private Space
The only users who can create apps in a Private Space are:
- Team administrators
- Team members that have been granted the “app creation” permission by an administrator. (This can be done in the Heroku Dashboard by clicking on the Space and then clicking Access.)
In the Heroku Dashboard, click on a space and choose Create New App.
In the Heroku CLI use the
$ heroku create my-space-app --space my-space-name Creating my-space-app in space my-space-name... done, stack is heroku-18 http://my-space-app.herokuapp.com/ | https://git.heroku.com/my-space-app.git
After the app is created, you can grant other team members access using one of several app permissions. Only members who are trusted to access any sensitive data protected by the space should be granted more than view permissions. All other members, who may develop on the app but who do not need production access, should only be granted access to staging and development environments.
Deploy an app in a Private Space
Code can be deployed (pushed) to Private Space apps in the same ways as with Common Runtime apps (such as by using
git push heroku master or a GitHub integration).
The build process itself occurs outside the Private Space and will not have access to any resources located inside the Private Space.
Scaling a process
Scaling your dyno formation follows the same pattern within Private Spaces as with the Common Runtime. For example:
$ heroku ps:scale web=2 worker=1
Scaling a formation up will require that a new dyno(s) is provisioned and launched within your Private Space. This process takes a few minutes to complete.
“One-off” dynos are available via
$ heroku run bash Running bash... ▸ WARNING: Warning: Dynos can take up to a few minutes to be provisioned in Private Spaces.
Creating a one-off dyno will require that a new dyno is provisioned and launched within your Private Space. This process takes a few minutes to complete.
List apps in a Private Space
All team members can view all apps in a Private Space:
$ heroku apps --space my-space-name === Apps in space my-space-name my-space-app-1 my-space-app-2
View info about an app in a Private Space
All team admins can detailed info about a single app in a Private Space:
$ heroku info --app my-space-app === my-space-app Collaborators: email@example.com Git URL: https://git.heroku.com/my-space-app.git Owner: my-team-name Region: tokyo Space: my-space-name Stack: heroku-18 Web URL: http://my-space-app.herokuapp.com/
Destroy an app in a Private Space
Team admins and members who have manage permission on an app can destroy it:
$ heroku apps:destroy -a my-space-app Destroying my-space-app (including all add-ons)... done
Adding custom domains and SSL
Only team administrators and members who have manage permission on an app can add custom domains and manage SSL.
SSL and custom domains for apps in Spaces are handled similarly to apps in the Common Runtime, with a few important exceptions:
- Newly created Private Spaces apps get working HTTPS on
<appname>.herokuapp.coma couple minutes after app creation and after apps are renamed. When enabling SSL for custom domains using either a manually uploaded certificate or ACM HTTPS will stop working on
<appname>.herokuapp.comfor Private Space apps. The default app domain (e.g. http://example.herokuapp.com) will always work with standard HTTP, but will not work with HTTPS if SSL is enabled for custom domains on the app.
- Apps in Private Spaces currently require TLS connections to use the SNI extension. Older clients that do not send the SNI extension will not be able to connect to a Private Space App using https.
- Once you have purchased and configured a custom domain, and purchased an SSL certificate, you can skip the step of running
heroku addons:create ssl:endpoint. (The Heroku SSL endpoint comes for free on apps in Private Spaces.)
Shield Private Spaces
Shield Private Spaces include additional features for building high-compliance applications. See the Shield Private Space article for details on features.
Add-ons and Private Spaces
If you attempt to provision an add-on for an app in a Heroku Private Space and that add-on is not available in that app’s Private Spaces region, you will see a message saying so.
To view which add-ons are available in a region, go to the Add-on Runtime Availability. Select the Private Spaces tab to view information for Private Spaces regions.
You can also view this information in the Region Availability section of the Elements listing page for each add-on.
Add-ons marked as Available in a Private Space
Many add-ons offer plans that are compatible with Private Space regions, even though they operate outside the Private Space. This means that network traffic between your private dyno and the add-on resource travels across the public Internet. These add-ons are marked as “Available” with a green dot in Add-on Runtime Availability.
You can add these add-ons to apps running in a Private Space as you would normally for our Common Runtime:
$ heroku addons:create newrelic:wayne -a appname Creating newrelic-cubic-72448... done, (free) Adding newrelic-cubic-72448 to appname... done Setting NEW_RELIC_LICENSE_KEY, NEW_RELIC_LOG and restarting appname... done, v80 Use `heroku addons:docs newrelic` to view documentation.
Some add-ons of type “datastore” are compatible with Private Spaces regions. They can be provisioned through the CLI with the
addons:create command, or they can be provisioned in the Dashboard. Either way, you will have to confirm that you understand that this use of a data store not contained within the Private Space network boundary is explicitly intended; that using this add-on plan will send data over the public Internet.
To provision from the CLI you must follow the confirmation prompt:
$ heroku addons:create <addon name> -a appname ! This add-on is not automatically networked with this Private Space. ! To proceed, type "appname" or re-run this command with --confirm appname
For information on Heroku add-on credits and overage payments mechanisms, see the Heroku Enterprise documentation.
Add-ons marked as Available and Installable in a Private Space
Some add-ons are installable in Private Spaces. This means that all network traffic between dynos and the add-on resource stays within the private network of the space, and the add-on resource does not have to be exposed on the public Internet. These add-ons are marked as “Available & Installable in Private Space” with a blue dot in Add-on Runtime Availability.
Heroku Postgres and Heroku Redis are examples of add-ons that are created inside Heroku Private Spaces. When you select a Heroku Postgres or Heroku Redis private plan for an app in a Private Space, the resource will be automatically created in the same region as the Heroku Private Space and peered to the Heroku Private Space.
To create a Heroku Postgres instance inside your Heroku Private Space, you must select a
private-0 plan or above. Since this Add-On will be automatically created inside your Space, confirmation is not required.
$ heroku addons:create heroku-postgresql:private-0 -a your-private-app-name
It is possible to provision Heroku Postgres and Heroku Redis plans outside of your Space. Provision by specifying app, plan, and region using the
$ heroku addons:create heroku-postgresql:standard-4 --region=us -a your-private-app-name
You will be asked to confirm the addition of the plan via the CLI.
! This add-on is not automatically networked with this Private Space. ! To proceed, type "appname" or re-run this command with --confirm appname
Migrate apps and data to or from a Private Space
There is no automated feature, at this time, to migrate apps and data to or from a Private Space, but it can be done manually. You should first back up and migrate the database(s), if any. App code should then be re-pushed to the destination. You should also consider differences in the way Private Spaces operate and whether any add-ons the app uses will work or need to be configured differently in the destination. For instance, SSL for apps in Private Spaces works slightly differently.
Transferring a Private Space
It’s possible to transfer a Private Space from one Enterprise Team to another. The user making the transfer should be an “admin” in both Enterprise Teams or have “Manage” permissions it the company’s Enterprise Account (currently in Public Beta). To transfer a Space navigate to its “Settings” tab, then find the Enterprise Team you want to move it to in the “Space Ownership” section.
When transferring spaces, only apps that are inside the space will move to the new Enterprise Team. If you have a pipeline that contains apps both inside the space and outside, only the apps inside the space will move to the new Enterprise Team while the pipeline stays in the old Enterprise Team with the apps that were not inside that space.
Default limits for Private Spaces
Private Spaces Runtime are built on a different architecture than the Common Runtime, and Private Spaces has different default limits. To have limits increased for any of your private spaces, please open a support ticket.
The current default limits for newly created Spaces are:
- 20 Trusted IP ranges
- 25 apps
- 100 process types (10 dynos per process type)
- 100 dynos for the entire Space
Concurrent one-off dyno limits
A Private Space app has the following limits for one-off dynos:
- Up to 5 one-off
private-sdynos running concurrently
- Up to 5 one-off
private-mdynos running concurrently
- Up to 5 one-off
private-ldynos running concurrently
- Up to 5 one-off
shield-sdynos running concurrently
- Up to 5 one-off
shield-mdynos running concurrently
- Up to 5 one-off
shield-ldynos running concurrently