Distributing Reads to Followers with the Octopus Gem
Last updated 17 May 2016
Table of Contents
This documentation is not comprehensive and is provided as a point to get started with the Octopus gem. This is advanced topic that will require custom configuration for each application. Official support on usage of this gem is not provided.
Heroku allows you to easily add horizontal scaling to your application by way of adding additional dynos to meet capacity. Similarly, Heroku Postgres allows you to horizontally scale your database by adding read-only followers to your primary database. While these followers are great for analytical purposes, you can also use them as a part of your application for handling read-only queries for your data.
The Octopus gem is a Ruby gem that enables sharding and replication features to ActiveRecord. In this guide we’ll add it into a Ruby on Rails application to distribute read queries to a follower.
- Ruby and Rails experience
- an existing Ruby on Rails application with models set up.
- A Heroku Postgres production database (Standard tier or above) with a Follower.
Estimated time: 15 minutes
An example repository for this guide can be found here: https://github.com/catsby/distributed-reads
In a normal Octopus setup, you hard code your shard and replication definitions in a file called
config/shards.yml. Because Heroku advocates 12 Factor applications however, hardcoding your database information is not recommend. Here we’ll utilize a dynamic
shards.yml file, as well as an initializer file, catered to the Heroku environment.
We start by first adding the Ocotpus gem to your
Gemfile and running
# Gemfile [...] gem 'ar-octopus', require: 'octopus' [...]
$ bundle install Fetching gem metadata from https://rubygems.org/....
Second, we add a tailored
config/shards.yml to dynamically populate the appropriate configurations for Octopus to use your follower(s) for read operations. Copy this file into
shards.yml file can be seen here: https://gist.github.com/catsby/6923840
$ curl -L -o config/shards.yml https://gist.github.com/catsby/6923840/raw/0aaf94ccc383951118c43b9b794fc62e427c2e51/shards.yml
Additionally we’ll add an initializer to
config/initializers to setup Octopus specifically for Heroku, which adds some convenience methods like
Octopus.followers, and setups some additional logging. Copy this file info
$ curl -L -o config/initializers/octopus.rb https://gist.github.com/catsby/6923632/raw/87b5abba2e22c3acf8ed35d06e0ab9ca1bd9f0d0/octopus.rb
octopus.rb file can be seen here: https://gist.github.com/catsby/6923632
Commit those changes in git and test out your setup locally. In development mode Octopus will simulate 2 followers for convenience:
$ heroku local web | Puma starting in single mode... web | * Version 2.6.0, codename: Pantsuit Party web | * Min threads: 5, max threads: 5 web | * Environment: development web | => 2 databases enabled as read-only slaves web | * FOLLOWER 1 web | * FOLLOWER 2
Designating Replicated Models
With this setup, your followers will not be marked as ‘fully replicated’; a fully replicated application will send all writes queries to the primary database, and all reads queries to followers. Depending on your application, this may not be desirable. In this configuration by default you are required to set the appropriate models with
replicated_model, or explicitly query them using methods to have queries sent to your followers.
# app/models/person.rb class Person < ActiveRecord::Base replicated_model end
This will configure the
Person class to be replicated, and read queries will be sent to the follower.
You can explicitly use a follower as well:
$ heroku run rails console Running `rails console` attached to terminal... up, run.2106 => 1 database enabled as read-only slave * PURPLE follower Loading production environment (Rails 4.0.0) irb(main):001:0> Octopus.using(:purple_follower) do irb(main):002:1* Person.first.name irb(main):003:1> end => "Jessica" irb(main):005:0> Person.using(:purple_follower).pluck(:name) => ["Jessica", "Leto", "Paul"]
Including and Excluding followers
You can choose the followers you want to use for responding to read request with the environment variables
SLAVE_DISABLED_FOLLOWERS. Whitelist followers you want or blacklist the followers you don’t want:
$ heroku config:set SLAVE_ENABLED_FOLLOWERS="PINK, CRIMSON" $ heroku config:set SLAVE_DISABLED_FOLLOWERS=COBALT
Without these variables set, all followers will be used for reads. This may be undesirable for you. One example of usage is when adding an additional follower to a live application, where the above ENV vars are used to ensure the new follower is excluded until it is sufficiently caught up to master for duty.
Thanks to Evan Prothro for writing the original dynamic shards.yml file and octopus.rb initializer file, as well as the original wiki guide for setting up Octopus on Heroku which the article is heavily based on. Thanks also to everyone who has contributed to the Octopus gem: https://github.com/tchandy/octopus/graphs/contributors