Heroku CI

Last updated 26 May 2017

Heroku CI is a visual, low-configuration test runner that integrates easily with Heroku Pipelines (and so complements Review apps, existing Heroku apps, and our GitHub integrations). Any Heroku Pipeline is already Heroku CI ready – just turn it on in the Pipeline’s settings tab. Your test scripts will execute in automatically created, disposable environments that have strong parity to the Heroku staging and production environments you’re already using. Workflow support includes test run results and run log access from Heroku or GitHub, automated deployments conditional on CI results, and a fully visual interface built to support Heroku CI’s role in team-based continuous delivery.

Quick start

We recommend that one Team or Enterprise Org own all the apps in a Heroku Pipeline (dev, staging, and production stages).

1. Create or identify a Heroku Team that will own all of the apps in your Pipeline.

2. Set up a Heroku Pipeline that includes only apps owned by your Heroku Team.

3. Connect your GitHub repository. You must have admin access to the repo.

4. Go to the new Pipeline tests page, and turn on automatic test runs

5. Choose your Heroku Team or Enterprise Organization for Heroku CI billing.

6. Optionally, add a test environment definition to your app.json file:

   • Add a test-setup script
   • Add a test script
   • Specify add-ons (and/or buildpacks, if you wish to add any) required for CI test runs.

7. Push to GitHub (to master, or by creating a Pull Request).

8. Find your tests running in the Pipeline.

You will often notice that the GitHub interface displays the CI results next to a Heroku icon.

Test run behavior

After you enable Heroku CI for a Pipeline, it runs tests automatically for every subsequent push to your GitHub repository. This means that all GitHub pull requests are automatically tested, along with any merges to master (typically deployed to Staging).

Test runs are executed inside an ephemeral Heroku app that is provisioned for each test run. The app is automatically destroyed when the run completes.

Configuration using app.json

Heroku CI is configured via an app.json in the root of your repository. Keys defined in the “test” environment will take precedence over those in base configuration.

Keys that can affect CI app environment configuration are the env, add-ons, buildpacks, formation, and scripts keys.

Environment variables (env key)

If your application or add-ons require environment variables during CI runs, you can specify them in app.json.

For example:

{
  "environments": {
    "test": {
      "env": {
        "PHOTON_TORPEDOES": "online"
      }
    }
  }
}

Unlike Heroku Review Apps, environment variables are not inherited from a parent app. For any sensitive or volatile environment variables (like access tokens) that you do not want in your app.json manifest, you can add them to your pipeline’s Heroku CI settings instead. Any environment variables specified in the Config variables will be available to each test run:

Additionally, the following immutable environment variables will be available for each test run (even though they do not appear in app.json):

• HEROKU_TEST_RUN_BRANCH: A string representing the branch of the commit under test
• HEROKU_TEST_RUN_ID: A string uuid representing the unique ID of the test run

Scripts (scripts key)

The scripts section in your app.json manifest identifies test setup tasks and the test command. For typical usage with our officially supported languages, you do not need to specify these. Our buildpacks are able to detect, set up, and run your tests without configuration (please see additional documentation below specific to your language). If you do specify tests here, they will override the tests that would have been autodetected.

The test and test-setup scripts in app.json can be any shell command, including the path to an executable that exists within your application repo (e.g. make tests).

If you are using a less common or custom test framework or language, you’ll need to tell Heroku CI how to set up and run your tests.

For example, if you didn’t have any tests, but you wanted to at least verify the code meets your standards, you could install and run rubocop:

{
  "environments": {
    "test": {
      "scripts": {
        "test-setup": "gem install rubocop",
        "test": "rubocop ."
      }
    }
  }
}

test-setup:

The test-setup script can optionally be used to perform one-time setup tasks, for example:

• Installing test dependencies like code linters, compilers, test runners etc.
• Setting up and seeding databases

test

The test script should be the command you use to execute your tests. It’s likely the same thing you use in your local environment, e.g.: bundle exec rspec, or npm test.

Add-ons (addons key)

CI apps will use your project’s app.json to determine which add-on services to provision using the default plan for temporary deployments (CI and Review apps) specified by each partner. These defaults are designed to be functionally similar to the production add-on plan, with enhanced provisioning and de-provisioning efficiency.

For example, test runs that use Heroku Postgres or Heroku Redis will substitute the Hobby plans for CI test runs.

{
  "environments": {
    "test": {
      "addons":[
        "heroku-redis:hobby-dev"
      ]
    }
  }
}

Release Phase and Heroku CI

Release Phase is ignored during Heroku CI test runs. We suggest placing any scripts that need to run before your tests in your test-setup script.

Browser tests and UAT on Heroku CI

The (headless) Chrome buildpack is prescribed to enable tests that require Chrome (e.g. Selenium and other UAT technologies). Other solutions are also supported.

See documentation for Heroku CI user acceptance testing (UAT) for more detail.

Technical Detail on test run lifecycle

Third-party buildpack maintainers can include explicit support for Heroku CI in their buildpacks (if you need additional support we are here to help). Consult Heroku CI: Technical Detail on Test Run Lifecycle for the sequence of events performed by Heroku CI during a test run. This information can be also be helpful to advanced developers who want to debug certain test run failures.

Currently supported languages

Heroku CI offers language support by the way of our Testpack API, which allow the buildpack to prepare your app for testing, and in some cases, detect what tests to run.

All of our officially supported languages have full Heroku CI support. Some third-party buildpacks also provide support.

In order to add Heroku CI support to a buildpack, the buildpack needs to implement the Testpack API. As an example, you can check out this Pull Request to the Elixir buildpack. For additional help or assistance, reach out to us at heroku-ci-feedback@heroku.com.

Go

The Heroku Go buildpack supports Heroku CI:

• Compiles and installs dependencies with detected tool (e.g. govendor, godep);
• Runs tests with go test ./….

Node

The Heroku Node buildpack supports Heroku CI:

• Compiles your tests with NPM_CONFIG_PRODUCTION=false and NODE_ENV=test (which installs your test and dev dependencies)
• Runs your tests with npm test or yarn test as appropriate

For typical Node apps, this should mean you do not need to define test scripts in app.json (but you may in package.json).

Java

The Heroku Java buildpack supports Heroku CI.

• Compiles test source code into the target directory by executing Maven’s test-compile lifecycle.
• Executes tests by running mvn -B test or mvnw -B test depending if using your own Maven wrapper.

PHP

The Heroku PHP buildpack supports Heroku CI:

• Dependencies from require-dev in composer.json will be installed for test runs.

You’ll need to tell Heroku CI how to run your tests with scripts.test, as they are not auto-detected.

Example app.json:

{
  "environments": {
    "test": {
      "scripts": {
        "test": "phpunit"
      }
    }
  }
}

Python

The Heroku Python buildpack supports Heroku CI:

• All dependencies specified within requirements.txt and optionally requirements-test.txt will be installed

The Python buildpack does not automacally detect your tests. You’ll need to tell Heroku CI what tests to run with scripts.test in your app.json.

Example app.json:

{
  "environments": {
    "test": {
      "scripts": {
        "test": "nose test"
      }
    }
  }
}

Ruby

The Heroku Ruby buildpack supports Heroku CI:

• installs development and test dependencies defined in your Gemfile
• supports both :ruby or :sql database schema when preparing database tests. In addition, will run rake db:migrate.
• Runs your tests with the appropriate Rake command (e.g. bundle exec rspec, bin/rails test, or rake test).

For typical Ruby apps, you should not need to define scripts in app.json.

Example (for a Rails app):

{
  "environments": {
    "test": {
      "addons":[
         "heroku-redis",
         "heroku-postgresql"
      ]
    }
  }
}

Gradle

The Heroku Gradle buildpack supports Heroku CI.

• Runs tests with gradlew test.

Scala

The Heroku Scala buildpack supports Heroku CI.

• Runs tests with sbt test.

Clojure

The Heroku Clojure buildpack supports Heroku CI.

• Runs tests with lein test.

Elixir

The third-party Elixir buildpack supports Heroku CI:

• Compiles and installs dependencies with MIX_ENV=test.
• Runs tests with mix test.

Note that as Elixir is not an officially supported language you must still specify the buildpack in your app.json.

Example app.json:

{
  "buildpacks": [
    {"url": "https://github.com/HashNuke/heroku-buildpack-elixir"}
  ]
}

Debugging

The Heroku Command Line Interface (CLI) includes a ci:debug command which can be used to start a debug test run. This allows for inspection of the Heroku CI environment and execution of tests inside a test dyno. This is handy for solving issues where tests pass locally but not on CI or for debugging test setup problems.

This command should be run from within your code repository. It will build a new test run and execute the test setup phase. Note that the heroku ci:debug command doesn’t require you to push your latest commits to Github, it will create a new test dyno based on your latest local commit.

$ cd my-repository
$ heroku ci:debug
Preparing source... done
Creating test run... done
Running setup and attaching to test dyno...

~ $ npm test # or whatever test command your application uses

To skip the test setup phase (for debugging setup issues), run with the --no-setup flag:

$ heroku ci:debug --no-setup
Preparing source... done
Creating test run... done
Attaching to test dyno...
 ▸    Skipping test setup phase.
 ▸    Run `sprettur setup && for f in .profile.d/*; do source $f; done`
 ▸    to execute a build and configure the environment
~ $

Costs

Each CI-enabled Pipeline is $10/month (prorated to the second).

Additionally, dyno and add-on run time for the duration of the test run is charged (prorated to the second).

This pricing will enable you to try Heroku CI on your Pipeline for some minutes for a few cents.

Cost details are available on our pricing page.

Dynos

Each test runs on a Performance-M dyno by default, and will be billed at that rate prorated to the second. Note that this run time will include both test-setup and test phases. For example, a 5 minute test run would incur $0.03 in dyno costs.

$250/month * 5 minutes / 43200 minutes/month = $0.03

Alternative dyno sizes can be specified with the formation key in the test environment section of your app.json file. Dyno sizes of standard-1x and above are supported. e.g.:

{
  "formation": {
      "web": {
        "size": "hobby"
      },
      "test": {
        "size": "performance-l"
      }
  }
}

Add-ons

When a test run is created, any add-ons listed in app.json’s test environment will be provisioned with the temporary deployments plan specified by the add-on vendor. If the test environment is either absent or omits the addons key, in the base manifest is used. Note that add-on providers are informed that these add-on will be used to support ephemeral CI runs, partners may take measures to allow the add-on to provision and destroy faster, such as omitting long-term logging or regular backups.

Once the test run has completed – reported as failed, errored, or succeeded – the test run’s add-ons will be de-provisioned and destroyed. This means that any paid add-ons will be billed at the plan’s rate, prorated to the second, only for the duration of the test run. Free add-on plans remain free on Heroku CI.

Billing

On the settings page in your Pipelines interface, under “Configure Heroku CI” you can select which party will be billed for CI runs.

If your Pipeline is comprised only of resources owned by a Heroku Teams or Heroku Enterprise Organization (as we recommend), you will have only the option to bill that Team or Organization. If you have Personal apps in your Pipeline (which we do not recommend), you will be offered the option to bill CI runs to your personal account.

Known Issues

Rails Fixtures and Referential Integrity

Rails default fixtures load test data by first disabling referential integrity. Due to restrictions in Heroku Postgres, referential integrity cannot be disabled. To work around this, consider one using another test data scaffolding tool (like FactoryGirl or Fabrication Gem). We are working to provide better support for Rails Fixtures, and hope to resolve this soon.

Xvfb Chrome Buildpack and heroku-16

The Xvfb Google Chrome Buildpack is incompatible with the heroku-16 stack. We suggest either using the Google Chrome Buildpack (without Xvfb), or changing the stack in your app.json to use the cedar-14 stack.

Buildpack shorthand names

Shorthand names for buildpacks is not currently supported. For example, heroku/jvm will not be automatically resolved to the fully qualified name of the buildpack https://github.com/heroku/heroku-buildpack-java. While we work on resolving this, please use the fully qualified name which can be found from the Elements page of each buildpack.

Disabling Heroku CI on a given Pipeline

To disable Heroku CI settings on a given Pipeline, visiting the Settings tab on that Pipeline’s web interface and click the “Disable” button.

Any charges for CI-enablement, or CI runs on that Pipeline will stop immediately. Note that CI is billed per-second.

Feedback

We’re eager to hear about your experience (and help) with Heroku CI, so please send comments and questions to heroku-ci-feedback@heroku.com .