Migrating to the Celadon Cedar Stack

Last Updated: 30 March 2014

bamboo cedar

Table of Contents

The Cedar stack is the default runtime stack on Heroku and is the successor to Bamboo. It includes support for multiple languages, flexible process types, HTTP 1.1, and substantially less code injection. While there is no automation available to migrate to Cedar from a previous stack this article outlines the steps and consideration required when performing a manual migration.

During this process two Heroku applications will exist representing the one conceptual application on each stack. A migration occurs by copying data and configuration from the Bamboo version of the app to the Cedar version and re-routing traffic to the Cedar app.

Major changes to an application, including stack migrations, should always be executed in a staging environment first. It’s highly recommended that you try pushing your app to a temporary Heroku app to test the migration process and the new environment, so that you can make any changes necessary to run on the new stack without affecting your production environment. Once your code runs cleanly, you can push those changes back to your production app and do the migration.

Create git topic branch

Any changes made in preparation for migrating to Cedar should be stored in a separate cedar topic branch for better code organization.

$ git checkout -b cedar
Switched to a new branch 'cedar'

Commits made to this topic branch will stay independent from the production-ready master branch which contains the Bamboo-compatible codebase.

Specify Ruby version

A number of Ruby versions are supported on Cedar. The default version is 2.0.0, but you can specify an alternate version with the ruby declaration in your Gemfile. For instance, to use Ruby 1.8.7:

source "https://rubygems.org"
ruby "1.8.7"
# ...

If you want to use 1.8.7 on Cedar be warned that it is no longer being actively maintained by Ruby core. Because of this Heroku encourages you run Ruby 1.9.2 or above in production.

Cedar does not have Ruby Enterprise Edition (REE) since it has reached the end of life. Ruby 1.9.3 and above does not need the same memory management fixes that REE provided and is generally considered faster than Ruby 1.8.7.

Migrating to Ruby 1.9

Using Ruby 1.9 on Cedar is recommended, though not required. If you are moving from Bamboo and wish to upgrade to Ruby 1.9 you can check that the app runs and works with 1.9.2 before upgrading the production environment to Cedar. This means looking into syntax changes as well as making sure 1.9.2’s encoding issues are resolved.

Applications not already running on the bamboo-mri-1.9.2 stack can migrate to it as the first step. This keeps your application on Bamboo, but moves it to Ruby 1.9.2.

$ heroku stack:migrate bamboo-mri-1.9.2
-----> Preparing to migrate bamboo-app
       bamboo-ree-1.8.7 -> bamboo-mri-1.9.2
...
-----> Migration prepared.
   Run 'git push heroku master' to execute migration.

$ git push heroku master
...
-----> Heroku receiving push
-----> Migrating from bamboo-ree-1.8.7 to bamboo-mri-1.9.2
...
-----> Migration complete, your app is now running on bamboo-mri-1.9.2

Test the application to ensure there are no Ruby 1.9.2 incompatibilities.

Specify dependencies

Use Bundler

If you’re using Bundler to manage your gems, you won’t need to do anything new here. If you’re not, however, you’ll want to migrate to it and declare any gems that you need. Bamboo had pre-installed gems. On Cedar all gems must be declared in the Gemfile.

Choose embedded webserver

Bamboo only allowed Thin as the embedded webserver for Ruby and Rails apps. On Cedar, you can use any embedded webserver, including Webrick, Mongrel, Thin, Goliath, or Unicorn.

For production apps, we recommend updating to use Unicorn.

Postgres

Applications that utilize a Postgres database should specify the pg gem in their Gemfile since it is no longer auto-injected on the Cedar stack.

gem 'pg'

The database credentials will still be configured automatically: at slug compile time, a config/database.yml that parses the DATABASE_URL from the environment will be written into the app.

The database.yml file is really an ERB file that is parsed at runtime to create the expected YAML-formatted file. To parse the file yourself run it through ERB before loading into YAML: YAML.load(ERB.new(File.read('config/database.yml')).result)

New Relic

Bamboo auto-injected the New Relic plugin and auto-created the config/newrelic.yml file. In Cedar, this is no longer the case.

If the application is using New Relic on Bamboo, manually add the newrelic_rpm gem to the application Gemfile.

gem 'newrelic_rpm'

And create the config/newrelic.yml file from New Relic’s template.

$ curl https://gist.github.com/rwdaigle/2253296/raw/newrelic.yml > config/newrelic.yml

Sendgrid

Further documentation and support for other frameworks can be found in the Sendgrid documentation.

Sendgrid add-on configuration is also no longer automatically provisioned on Cedar. Rails applications relying on Sendgrid to deliver emails will need to add a mail initializer at config/initializers/mail.rb.

$ curl https://gist.github.com/rwdaigle/1690653/raw/mail.rb > config/initializers/mail.rb

Commit dependencies

At this point there may be several changes made to the application source to ready it for deployment to the Cedar stack, including modified gem dependencies.

Run bundler to install any newly required gems.

$ bundle install
Fetching source index for https://rubygems.org/
Using rake (0.8.7)
...

Add any new files to git.

$ git add config/newrelic.yml

Commit changes (to the current cedar topic branch).

$ git commit -a -m "Modified application dependencies for Heroku/Cedar compatability"

Create Procfile

The Procfile is new to Cedar and specifies what processes Heroku should run for an application. Apps that don’t use background workers on Bamboo should create a new Procfile at the application root with a single web process type.

web: bundle exec unicorn -p $PORT -E $RACK_ENV -c config/unicorn.rb

worker is only a suggested process type. Non-web process types can have custom labels to more accurately reflect their role.

If the application is using Bamboo’s background worker support then a worker process type should be specified in the Procfile as well.

web: bundle exec unicorn -p $PORT -E $RACK_ENV -c config/unicorn.rb
worker: bundle exec rake jobs:work

The Procfile is not a Heroku-specific manifest. It can be used to orchestrate an application’s dyno formation during local development as well with the Foreman gem.

$ gem install foreman
Fetching: foreman-0.38.0.gem (100%)
Successfully installed foreman-0.38.0

Variables saved in the .env file of a project directory will be added to the environment when run by Foreman. Set the RACK_ENV to development in your environment. Do not commit the .env file to source control it should only be used for local configuration.

$ echo "RACK_ENV=development" >>.env

You can run your app locally using the environment variables in your .env, the settings in your Procfile, and Foreman by running.

$ foreman start
10:01:59 web.1      | started with pid 3484
10:01:59 worker.1   | started with pid 3485

Once the application is running locally with Foreman commit the Procfile to the topic branch.

$ git add Procfile
$ git commit -a -m "Added Procfile to define app's process-formation on Heroku/Cedar"

Create Cedar app

As there is no automated way of migrating a Bamboo app to Cedar, a new app should be created on the Cedar stack. Naming the remote git repository heroku-cedar will differentiate between the Bamboo and Cedar versions of the application.

$ heroku create --remote heroku-cedar cedar-app
Creating cedar-app... done, stack is cedar
http://cedar-app.herokuapp.com/ | git@heroku.com:cedar-app.git
Git remote heroku-cedar added

Deploy

Deploying the modified codebase to the Cedar application will not affect the currently running app on Bamboo. It will merely stage the application and provide the opportunity to verify functionality.

Push the Cedar-compatible source from the cedar topic branch to the master branch of the heroku-cedar remote repository to deploy.

$ git push heroku-cedar cedar:master
Counting objects: 67, done.
...
-----> Heroku receiving push
-----> Rails app detected
...

The application should deploy successfully though it will not likely be functional as no add-ons or configuration variables have been set and there is no backing datastore.

Reprovision add-ons

Transferring add-ons from one application to another is not currently supported. All add-ons from the Bamboo application should be manually provisioned on the Cedar application.

Store a list of all add-ons currently attached to the Bamboo application in an addons.txt file.

$ heroku addons --app bamboo-app > addons.txt
$ more addons.txt
=== bamboo-app Configured Add-ons
memcachier:dev
newrelic:standard
pgbackups:auto-month
heroku-postgresql:dev
heroku-postgresql:ronin

Applications with Heroku Postgres databases (heroku-postgresql:XXX) should not recreate the db add-on in this manner. Remove it from addons.txt as well as the === bamboo-app line.

$ more addons.txt
memcachier:dev
newrelic:standard
pgbackups:auto-month

Though it exhibits platform-specific behavior, the command line utility xargs can also be used to automate this process.

Next, manually re-provision each add-on in the addons.txt file on the Cedar app:

$ heroku addons:add memcachier:dev --app cedar-app
-----> Adding memcachier:dev to cedar-app
$ heroku addons:add newrelic:standard --app cedar-app
...

Please keep in mind that, depending on the tiers of add-on service that exist on the Bamboo app, some add-ons may incur fees.

Copy configuration variables

Similar to addons there is not an automated way to transfer configuration values from one application to another. They will have to be manually copied.

Store the configuration variables from the Bamboo app into a config.txt file.

$ heroku config -s --app bamboo-app > config.txt
$ more config.txt
DATABASE_URL=postgres://u:p@host/db
MEMCACHE_PASSWORD=pass
MEMCACHE_SERVERS=mcX.ec2.northscale.net
MEMCACHE_USERNAME=pp123%40heroku.com
RACK_ENV=production
PATH=vendor/bundle/ruby/1.9.1/bin:/usr/local/bin:/usr/bin:/bin
SESSION_SECRET=secret
HEROKU_POSTGRESQL_RED_URL=postgres://u:p@host/db

Only the configuration variables that have been manually set should be kept. Open the config.txt and remove any add-on or Heroku configuration such as PATH, MEMCACHE_, _DATABASE_URL etc…

Overwriting add-on and database configuration variables can result in unpredictable and destructive side-effects. Please confirm such settings are removed from config.txt.

$ more config.txt
RACK_ENV=production
SESSION_SECRET=secret

Though it exhibits platform-specific behavior, the command line utility xargs can also be used to automate this process.

Manually add these application-specific configuration settings to the Cedar application (you can set more than one config var at a time):

$ heroku config:set --app cedar-app RACK_ENV=production SESSION_SECRET=secret ...
Adding config vars and restarting app... done, v32
  RACK_ENV            => production
  SESSION_SECRET      => secret
...

Use heroku releases:rollback to quickly undo errant configuration changes.

Memcache

If you are using memcache in your app you might be using a gem that requires c-bindings. The libraries to bind to memcache are not present by default on the Cedar stack. To get around this you can use the Dalli gem or memcache-client both of which should work on all versions and implementations of Ruby and require no c-bindings. We recommend the Dalli gem, but it is not compatible with Rails 2.x. To change over you will need to add the new gem to your gemfile

gem "dalli"

If you were using the rails cache store you can configure Dalli to be used here by setting this value in your config/production.rb

config.cache_store = :dalli_store

If you are on Rails 2.x and have to use MemcacheClient first add it to your Gemfile.

gem 'memcache-client', :require => 'memcache'

Then you can set it to be your cache store:

config.cache_store = :mem_cache_store

If you were manually calling memcache clients you may need to replace calls from other libraries to call to Dalli::Client for Dalli, or MemCache for memcache-client.

Make Cedar app live

At this phase of the migration there should be a copy of the application running on Cedar with identical configuration as the Bamboo app. The final step to the migration is to transfer the data from the Bamboo app to the Cedar app and enable the Cedar app to receive live traffic.

To ensure data consistency both applications should be put in maintenance mode for the duration of the transfer to prevent any data modification.

$ heroku maintenance:on --app bamboo-app
Maintenance mode enabled.
$ heroku maintenance:on --app cedar-app
Maintenance mode enabled.

Postgres

Applications that have a Heroku Postgres database provisioned should use the PG Backups add-on to migrate data from the Bamboo application’s database to a newly provisioned database on the Cedar app.

MongoDB, Xeround and other data add-ons

Applications on Bamboo utilizing a datastore other than Heroku’s Postgres should reference their add-on’s documenation for instructions on how to properly migrate data between applications.

Custom domains

Traffic routed from a custom domain will still be pointing to the Bamboo application. If custom domains are enabled they will need to be removed from the Bamboo application before being added to Cedar.

Store the list of custom domains in a domains.txt file and remove them from the Bamboo app.

Clearing the domains from the Bamboo app is necessary for the migration but be aware that no traffic will be routed to the Bamboo app afterwards without re-adding the domains should a rollback be necessary.

$ heroku domains --app bamboo-app | grep -v "Domain Names" > domains.txt
$ more domains.txt
secure.mydomain.com
www.mydomain.com
$ heroku domains:clear --app bamboo-app
Removed all domain names for bamboo-app

Add the custom domains onto the Cedar application.

$ cat domains.txt | xargs -L1 heroku domains:add --app cedar-app

DNS

To route traffic to your new app, adjust the DNS settings so any custom domains resolve to cedar-app. Add the following CNAME record in your DNS provider’s control panel.

Type Name Target
CNAME www cedar-app.herokuapp.com

SSL

SSL on Cedar is provided with the SSL Endpoint add-on. You can use the certificate and private key originally used to provision SSL on Bamboo. If you don’t have access to those files anymore you will need to acquire a new SSL certificate and use it along with the private key to setup the endpoint.

$ heroku addons:add ssl
$ heroku certs:add server.crt server.key
-----> Adding certificate to myapp... done.
       myapp now served by tokyo-2121.herokussl.com.
...

After the endpoint is provisioned and certificate/key added you’ll need to setup a DNS CNAME record to map your secure subdomain to the assigned endpoint host (here tokyo-2121.herokussl.com).

Type Name Target
CNAME secure tokyo-2121.herokussl.com

This mapping will provide SSL for the https://secure.yourdomain.com domain served by the tokyo-2121.herokussl.com endpoint.

Heroku does not support DNS A-records (which are used to map root domains). If you are using a root domain (mydomain.com versus www.mydomain.com) with SSL consider migrating to a secure subdomain instead. Additionally, you may consider non-standard DNS functionality such as the ALIAS record supported by DNSimple.

Turn off maintenance mode

Turn off maintenance mode on the Cedar application to begin accepting traffic once it has been scaled to the appropriate number of dynos (using the Bamboo application’s levels as a good starting point).

$ heroku dynos --app bamboo-app
bamboo-app is running 5 dynos and 2 workers
$ heroku ps:scale web=5 worker=2 --app cedar-app
Scaling web processes... done, now running 5
Scaling worker processes... done, now running 2
$ heroku maintenance:off --app cedar-app
Maintenance mode disabled.

Finalize

Once the Cedar application is live, accepting visitors and confirmed to be operational the modifications made on the cedar topic branch should be merged back into master.

$ git checkout master
$ git merge cedar

The old version of the application can also be decommissioned to reduce cost. Scaling down the old application to 1 dyno and downgrading addons to their free equivalents during a transition period of reasonable length allows for rollbacks to Bamboo should issues be discovered.

Destroy Bamboo app

Once you are comfortable with the efficacy of the newly provisioned Cedar app you can destroy the Bambo app. This is an optional step that removes the necessity to use the --app flag with every Heroku CLI command.

Destroying an application cannot be reversed. Please be certain of this step before executing the command.

$ heroku destroy bamboo-app

Notes

Timezones

It should be noted that the system timezone on Bamboo dynos is Pacific Time (US & Canada) whereas on Cedar dynos it is UTC. Any logic relying on non-normalized system time should be updated to account for this disparity.

File compression

On Bamboo, responses were automatically compressed using gzip. On Cedar no such compression is done on your behalf. To add response compression to your application use middleware such as Rack::Deflater.