Last updated 24 August 2016
Table of Contents
A pipeline is a group of Heroku apps that share the same codebase. Apps in a pipeline are grouped into “review”, “development”, “staging”, and “production” stages representing different deployment steps in a continuous delivery workflow. A code change will typically be deployed first to a pull request, which automatically creates a review app, then merged into master which is automatically deployed to staging for further testing before promotion to production where the new feature will be available to end users of the app. The pipelines overview page helps you visualize this flow, as well as meta information about the status of each stage. For example, you can see if your production app is running different code than staging.
Maintaining multiple environments, such as staging and production, is simple on Heroku. However, maintaining consistent deployments between these environments requires app owners to manually manage the deployment workflow. Issues that can arise from a manual deployment workflow typically include pushing the wrong commit, releasing untested code, or pushing to the incorrect environment.
Deployment with pipelines
Pipelines allow you to define how your code should flow from one environment to the next. For example, you can push code to
staging, have it built into a slug and later promote the
staging slug to
production. Not only will this ensure that the release contains the exact same compiled code, but it will be much faster than re-compiling the slug. Common examples of pipelines include:
A simple staging to production pipeline:
myapp-staging ---> myapp
Or, a team’s more complex pipeline:
myapp-jim-dev --- \ ---> myapp-staging ---> myapp / myapp-kim-dev ---
In pipeline vocabulary, “downstream” refers to the next target app in a pipeline. For example, given a
dev ---> staging ---> production pipeline, staging is downstream of dev, and production is downstream of staging.
Once you’ve defined a pipeline, you and your team no longer have to worry about the next app to deploy to. Instead,
heroku pipelines:promote will copy your app’s build artifact (i.e. slug) to the downstream app(s) as a new release.
Pipelines only manage the application slug. The Git repo, config vars, add-ons and other environmental dependencies are not considered part of a pipeline and must be managed independently. You can use Heroku Fork to quickly clone production apps to be used for dev and staging and Review Apps to create temporary apps for every pull request.
When promoting slugs between apps, Heroku will copy the slug verbatim between apps, without changes. In particular, the slug is not rebuilt with the environment of the target app. If your builds tend to bake in environment from the build-app context, then your slugs will not be portable between apps, and will not work correctly when promoted from the app where they were built. This may be the case, for example, if you’re using Ruby on Rails and the asset pipeline to build assets hosted at a CDN defined by a URL in your build environment. This is because URLs specific to the build-app will be baked into css and js-files, and those will not work or not work correctly when promoted.
Apps in a pipeline no longer show up as individual apps. Instead, you’ll see a single line for the pipeline with a drop-down to see the individual apps if needed.
Pipelines via the CLI
Many Pipelines operations can be performed via the CLI, which requires the Pipelines plugin for the Heroku CLI. To install:
$ heroku plugins:install heroku-pipelines
To create a new pipeline, click the + icon in the top right of your app list and select “Create new pipeline”, or navigate to an app’s deploy tab and create a new pipeline to include that app.
You can also create a pipeline from the CLI. You must start with an app to add to the pipeline, although it doesn’t have to be for a particular stage. If you don’t specify
--stage STAGE, the CLI will guess at the appropriate stage, but also let you override the default. The name of the pipeline will be guessed from the app name as well, but can be overridden either by adding the
NAME on the command line, or entering a different name when prompted.
$ heroku pipelines:create -a example ? Pipeline name: example ? Stage of example: production Creating example pipeline... done Adding example to example pipeline as production... done
If you previously used GitHub integration to connect Heroku apps to GitHub, you may already see pipelines in your app list as we automatically migrated those apps to pipelines.
Adding apps to a pipeline
In the pipeline overview page, click on “+ Add app” to add an existing application to the pipeline. From the CLI:
$ heroku pipelines:add -a example-staging example ? Stage of example-staging: staging Adding example-staging to example pipeline as staging... done
Multiple apps in a pipeline stage
Pipeline stages may have more than one app in them. For example, the production stage might have the main production app and an admin app running the same version of code, but with different configurations.
When a change has been tested sufficiently in a particular stage, the slug can be promoted to the downstream stage using the Promote button. If there are multiple apps in the downstream stage, the slug will be promoted to all apps by default. Don’t worry, you’ll get a chance to see what you’re promoting and where it’s going before you commit.
From the CLI, you can promote by specifying the source app. The target app(s) will be automatically determined by the downstream stage. For example, if there are two apps in production, and you promote from the staging app, both production apps will be released.
$ heroku pipelines:promote -r staging Promoting example-staging to example (production)... done, v23 Promoting example-staging to example-admin (production)... done, v54
It is also possible to promote to a specific app (or set of apps).
$ heroku pipelines:promote -r staging --to my-production-app1,my-production-app2 Starting promotion to apps: my-production-app1,my-production-app2... done Waiting for promotion to complete... done Promotion successful my-production-app1: succeeded my-production-app2: succeeded
Promoting from staging to production is great, but you can further optimize your flow by automatically deploying to staging using GitHub integration. For example whenever the
master branch is updated on GitHub and continuous integration (CI) tests pass, staging can be auto-deployed.
If you’re using GitHub, we highly recommend using pull requests and setting up corresponding review apps.
Many developers start hacking on a project and work with a single app on Heroku. At this point, it’s not really “production”, it’s just the app. But at some point you start showing it to other people, and maybe people start depending on it. Then you realize that pushing bad code to your precious users might not be a good idea, so you spin up a “staging” app. At some point a “development” app pops up for even earlier testing of new ideas, usually when you want someone else to see the change before deciding if you should merge it into
master and put it through the staging and production flow.
When the team grows, developers see contention for that single dev app and start creating individual dev apps for each developer. This is all well and good, but with the advent of review apps, there’s a new, temporary app for every pull request. Want to show your changes to someone else on the team? Just create a pull request and click on the link to the new app. This works beautifully with GitHub Flow. Rather than thinking of the pull request as the final step before merging into
master, think of it as the beginning of a discussion. And as the feature evolves, there’s a running history of the discussion. When the feature is finally merged, the pull request is closed and the review app is automatically deleted. (And with GitHub integration, the new merged code is automatically deployed to staging.)
Many teams discover that individual development apps aren’t necessary anymore so the traditional
dev ---> staging ---> prod pipeline becomes
review ---> staging ---> prod. Heroku Pipelines supports both of these flows, but we highly recommend the latter.
Another popular flow, called git-flow can fit in Heroku Pipelines as well, by mapping review apps to feature branches via pull requests, the
develop branch to a “development” stage app, one or more
release branches to “staging” apps, and
master to “production”. “Hotfix” branches don’t quite fit in, but can be handled by creating temporary apps, perhaps using fork, and placing them in “staging”.
Some of our customers also use “testing”, “qa”, “user-acceptance” and other stages. For now, try fitting those apps into either “development” or “staging” and let us know if it really doesn’t work out.
What else can I do with the pipelines CLI?
A complete list of Pipelines commands with usage details is available in the console:
$ heroku help pipelines
The repository README for the Heroku CLI plugin provides additional usage examples and a feature history.
Can I have pipelines stages other than development, staging and production?
No, only those three stages are currently supported, plus the special stage, review.
Can I run scripts, such as
rake db:migrate when promoting?
No, not at this time. If the code change requires a database migration or any action that requires the particular app’s environment such as uploading static assets to S3 or a CDN, these actions will need to be performed manually after pipeline promotion.
Can I migrate from old to new Pipelines?
No, you currently have to re-configure your new Pipeline in Dashboard
Can I have more than one app in a pipeline stage?
Does pipelines work without GitHub?
Yes. We’ve worked hard to make Pipelines work well with GitHub, but it’s not necessary.
I don’t see a “development” stage, how do I add an development app?
You can add the app to any other stage, and then using the context menu on the app card, move the app to “development”. Alternatively, you can go to the Deploy tab of the app and add it to a pipeline from there, while specifying the “development” stage.
Do Pipelines work with Heroku Organization Accounts?
Yes, Pipelines are fully supported for organization accounts. However, in some cases team members may find they cannot operate on a given app in a pipeline due to the access control, such as that described in Using App Permissions in Heroku Organizations. Users must have adequate permissions to perform the relevant operations on the app.