Distributing Reads to Followers with the Octopus Gem

Last Updated: 30 January 2015

followers octopus postgres ruby

Table of Contents

Heroku allows you to easily add horizontal scaling to your application by way of adding additional dynos to meet capacity. Similarly, Heroku Postgres allows you to horizontally scale your database by adding read-only followers to your primary database. While these followers are great for analytical purposes, you can also use them as a part of your application for handling read-only queries for your data.

The Octopus gem is a Ruby gem that enables sharding and replication features to ActiveRecord. In this guide we’ll add it into a Ruby on Rails application to distribute read queries to a follower.

Prerequisites

  • Ruby and Rails experience
  • an existing Ruby on Rails application with models set up.
  • A Heroku Postgres production database (Standard tier or above) with a Follower.

Estimated time: 15 minutes

Setup

Source for this article’s reference application is available on GitHub and can be seen running at http://distributed-reader.herokuapp.com

An example repository for this guide can be found here: https://github.com/catsby/distributed-reads

In a normal Octopus setup, you hard code your shard and replication definitions in a file called config/shards.yml. Because Heroku advocates 12 Factor applications however, hardcoding your database information is not recommend. Here we’ll utilize a dynamic shards.yml file, as well as an initializer file, catered to the Heroku environment.

We start by first adding the Ocotpus gem to your Gemfile and running bundle install

# Gemfile
[...]
gem 'ar-octopus', require: 'octopus'
[...]
$ bundle install
Fetching gem metadata from https://rubygems.org/....

Second, we add a tailored config/shards.yml to dynamically populate the appropriate configurations for Octopus to use your follower(s) for read operations. Copy this file into config/ as shards.yml:

The full shards.yml file can be seen here: https://gist.github.com/catsby/6923840

$ curl -L -o config/shards.yml https://gist.github.com/catsby/6923840/raw/0aaf94ccc383951118c43b9b794fc62e427c2e51/shards.yml

Additionally we’ll add an initializer to config/initializers to setup Octopus specifically for Heroku, which adds some convenience methods like Octopus.followers, and setups some additional logging. Copy this file info config/initializers/:

$ curl -L -o config/initializers/octopus.rb https://gist.github.com/catsby/6923632/raw/87b5abba2e22c3acf8ed35d06e0ab9ca1bd9f0d0/octopus.rb

The full octopus.rb file can be seen here: https://gist.github.com/catsby/6923632

Commit those changes in git and test out your setup locally. In development mode Octopus will simulate 2 followers for convenience:

$ foreman start
foreman | starting web on port 5000
web     | Puma starting in single mode...
web     | * Version 2.6.0, codename: Pantsuit Party
web     | * Min threads: 5, max threads: 5
web     | * Environment: development
web     | => 2 databases enabled as read-only slaves
web     |   * FOLLOWER 1
web     |   * FOLLOWER 2

Designating Replicated Models

With this setup, your followers will not be marked as ‘fully replicated’; a fully replicated application will send all writes queries to the primary database, and all reads queries to followers. Depending on your application, this may not be desirable. In this configuration by default you are required to set the appropriate models with replicated_model, or explicitly query them using methods to have queries sent to your followers.

# app/models/person.rb
class Person < ActiveRecord::Base
  replicated_model
end

This will configure the Person class to be replicated, and read queries will be sent to the follower.

You can explicitly use a follower as well:

$ heroku run rails console
Running `rails console` attached to terminal... up, run.2106
=> 1 database enabled as read-only slave
  * PURPLE follower
Loading production environment (Rails 4.0.0)
irb(main):001:0> Octopus.using(:purple_follower) do
irb(main):002:1* Person.first.name
irb(main):003:1> end
=> "Jessica"
irb(main):005:0> Person.using(:purple_follower).pluck(:name)
=> ["Jessica", "Leto", "Paul"]

Including and Excluding followers

You can choose the followers you want to use for responding to read request with the environment variables SLAVE_ENABLED_FOLLOWERS and SLAVE_DISABLED_FOLLOWERS. Whitelist followers you want or blacklist the followers you don’t want:

$ heroku config:set SLAVE_ENABLED_FOLLOWERS="PINK, CRIMSON"
$ heroku config:set SLAVE_DISABLED_FOLLOWERS=COBALT

Without these variables set, all followers will be used for reads. This may be undesirable for you. One example of usage is when adding an additional follower to a live application, where the above ENV vars are used to ensure the new follower is excluded until it is sufficiently caught up to master for duty.

Credit

Thanks to Evan Prothro for writing the original dynamic shards.yml file and octopus.rb initializer file, as well as the original wiki guide for setting up Octopus on Heroku which the article is heavily based on. Thanks also to everyone who has contributed to the Octopus gem: https://github.com/tchandy/octopus/graphs/contributors