Abhinav Keswani

This article was contributed by Abhinav Keswani

Abhinav heads up the Bespoke Solutions team at Trineo, a boutique software development shop based in New Zealand and Australia. After way too many years wrangling the vagrancies of massive public facing web infrastructure, Abhinav and the Trineo team focus on developing applications that use modern cloud technology. Trineo are listed Heroku Dev Partners, and Salesforce Consulting Partners.

Queuing in Ruby with Redis and Resque

Last Updated: 28 November 2013

background jobs queuing resque

Table of Contents

The delegation of long running, computationally expensive and high latency jobs to a background worker queue is a common pattern used to create scalable web applications. The aim is to serve end-user requests with fastest possible responses, by ensuring that all long running jobs are handled outside the request/response cycle.

Isolated longer running jobs can be handled by a separate worker pool that can scale independently. These jobs can asynchronously handle tasks like fetching data from remote APIs, reading RSS feeds or handling long running uploads to S3.

Resque (pronounced ‘rescue’) is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and handling those jobs outside of the user request/response cycle.

If you have questions about Ruby on Heroku, consider discussing it in the Ruby on Heroku forums.

Overview

Resque is designed to be used in scenarios that require a high volume of job entries, and provides mechanisms to ensure visibility and reliability of behavior while providing statistics using a web dashboard.

Consider Resque as a good choice for applications that run multiple queues each with many thousands of job entries, where worker behavior can be volatile. Volatile worker behavior is mitigated by forking children processes to handle jobs which ensures that any out of control workers can be dealt with in isolation.

Install locally

To install Redis locally, use the following approach on Mac OS X.

$ brew install redis

To see if you’re already running Redis, check the output of the redis-cli ping command. A PONG response indicates you already have Redis installed and running.

To test your install, firstly ensure that the Redis server is running.

$ redis-server

Then use the following simple commands.

$  redis-cli ping
PONG
$  redis-cli
redis 127.0.0.1:6379> set foo bar
OK
redis 127.0.0.1:6379> get foo
"bar"

To install Redis on Linux, and to see more details on an effective quickstart, refer to the documentation on the Redis site.

The example application mentioned later in this article has a dependency on ImageMagick. To install ImageMagick on OS X:

$ brew install imagemagick

To install ImageMagick on other operating systems, refer to the ImageMagic site for installation of binary releases or from source.

Resque on Heroku

Prior to v1.22 there was a disparity between Resque’s signal handling and Heroku’s process management that caused orphaned and incomplete jobs between worker restarts. Starting with v1.22 Resque’s handling of TERM and QUIT signals is in accordance with UNIX standards and maintains compatibility with the Heroku runtime that previous versions lacked.

To run Resque on Heroku you need to specify the correct Resque version and provide a couple options to the worker process.

Specify version

In your application Gemfile specify Resque v1.22.0.

gem 'resque', "~> 1.22.0"

Run bundle install to update your gems.

Process options

To opt-in to UNIX compatible signal handling in Resque v1.22 you will also need to provide a TERM_CHILD environment variable to the resque worker process. This can be done inline in your application Procfile.

resque: env TERM_CHILD=1 bundle exec rake resque:work

TERM_CHILD tells Resque to pass SIGTERM from the parent to child process to ensure that all child worker process have time to execute an orderly shutdown.

The default period Resque waits before sending SIGKILL to its child processes is 4 seconds. To modify this value and give your workers more time to gracefully shutdown, modify the RESQUE_TERM_TIMEOUT environment variable.

resque: env TERM_CHILD=1 RESQUE_TERM_TIMEOUT=7 bundle exec rake resque:work

Heroku waits 10 seconds after sending SIGTERM to send SIGKILL, so be sure to use a RESQUE_TERM_TIMEOUT value less than 10.

Reference application

For the remainder of this article, a reference application will be cited to illustrate the working parts of an application that uses Resque.

The main premise of the application is to watermark uploaded images and store them on AWS S3. When a user uploads an image, it is saved to S3, and then a Resque job is enqueued to create a watermarked copy of themis user’s image, which is also stored on AWS S3.

The source for this reference application can be found on GitHub

Local installation

To set this app up locally firstly fork it on GitHub and then clone it locally, for example:

$ git clone git://github.com/trineo/resque-example.git

To install application dependencies run:

$ bundle install

Local setup

Sign up to AWS S3 if you haven’t already done so, following these instructions.

To set up your local environment, and connectivity to all backing services, you need to create a .env file.

$ cp .env.example .env

Create two buckets on S3 that respectively will be used to store original and watermarked images. Edit the .env file appropriately to specify:

  • the names of these buckets
  • AWS S3 secret access key, and access key id
  • your local Redis URL (already set to the default Redis value)

Running the app locally

Start Redis in a separate shell.

$ redis-server

Then start the watermarked application.

$ foreman start

This should now start a web process as well as a worker process, both of which are specified in the Procfile of the application.

Verify and use local setup

Using the Resque dashboard, check that the app is running:

To interact with the application and queue a job, get a test image and run the following curl command (where src.jpeg is the test image):

$ curl -F "file=@src.jpeg" "http://localhost:5000/upload"

This should have the following effect:

  • The worker should show it is working
  • An image called src.jpeg should be present in both “originals” and “watermarked” buckets
  • The image in the watermarked bucket should be … watermarked!
  • The index page of the reference app (http://localhost:5000) should show some statistics about file upload and watermarking activity

Queueing jobs

Jobs are queued by the upload post method in the main Sinatra file, resque-example-app.rb. The upload method saves the file to S3 and then enqueues a job to watermark the uploaded file.

Resque.enqueue(Watermark, file_token.key)

Now, any worker that is bound to the same Redis backing service can process this job.

The enqueue method has the following method signature:

Resque.enqueue(klass, *args)

In calling enqueue, the perform method of klass is called, with all the original enqueue arguments. These arguments are serialized as JSON, and therefore it is important to ensure that the arguments can in fact be serialized as JSON. Arguments like symbols, or entire ActiveRecord objects will not work. Try instead to send object IDs or as is the case in the example application, a AWS file token key is sent.

Processing jobs

The Watermark worker

The Watermark class is responsible for watermarking files in the background. Once a job has been enqueued as shown above, and when a worker is ready, the “perform” method of the Watermark class is invoked with the S3 file token being sent as an argument.

Watermark#perform is a class method that instantiates a new instance of this class, such that useful class variables are set up.

def self.perform(key)
  (new key).apply_watermark
end

Perform then chains the invocation of the ‘apply_watermark’ method on this new instance, which then applies and saves the watermarked file.

def apply_watermark
  Dir.mktmpdir do |tmpdir|
    ...
    save_watermarked_file(watermarked_local_file)
  end
end

def save_watermarked_file(watermarked_local_file)
  ...
end

Note that it is considered bad practice to store any files on the Heroku filesystem. What is happening in the reference application, is:

  • The file is transient, which is to say that the filesystem is used to store the file so that it can be watermarked, and once this is complete the file is deleted. (refer to Dir#mktmpdir)
  • Local file processing that is briefly conducted in the background, using the filesystem of a process is not considered to be bad practice (reference)

Define and provision workers

Resque workers are defined the reference application’s Procfile:

resque: env TERM_CHILD=1 bundle exec rake resque:work

These workers can be provisioned by running a simple scaling command such as:

heroku ps:scale resque=2 --app appname

Alternatively, one can run multiple worker threads within a single process/dyno by modifying the Procfile accordingly:

resque: env TERM_CHILD=1 COUNT=2 bundle exec rake resque:workers

Note two main differences: * The COUNT variable specifies the number of worker threads to run * The resque:workers rake task is invoked

Running locally

As per prior instructions, start the application, and issue the given curl command to post a file to the upload handler. Subsequently, this is the log stream generated by the web and resque workers.

20:47:15 web.1    | 127.0.0.1 - - [14/Aug/2012 20:47:15] "POST /upload HTTP/1.1" 200 - 3.1093
20:47:19 resque.1 | Initialized Watermark instance
20:47:19 resque.1 | Opening original file locally: /var/folders/kq/src.jpeg
20:47:19 resque.1 | Writing watermarked file locally: /var/folders/kq/watermarked_src.jpeg
20:47:22 resque.1 | Persisted watermarked file to S3: https://your-watermarked-bucket.s3.amazonaws.com/src.jpeg

Job failures

Resque leaves it to the developer to decide on how to best handle job failure, using a ‘on_failure’ hook. A common approach is to re-enqueue a job if it fails. The reference application shows a simple example of this in the context:

module RetriedJob
  def on_failure_retry(e, *args)
    ...
    Resque.enqueue self, *args
  end
end

class Watermark
  extend RetriedJob
  ...
end

More complex implementations would involve stipulating a number of retries, so that a job is given n number of attempts before giving up, and additionally stipulating a delay between retries. The resque-retry plugin provides much of this functionality.

Failed jobs are visible on the Resque dashboard, which is described below.

Job termination

When worker termination is requested via the SIGTERM signal, Resque throws a Resque::TermException exception. Handling this exception allows the worker to cease work on the currently running job and gracefully save state by re-enqueueing the job so it can be handled by another worker.

Handling this exception on Heroku is recommended due to the frequency of dyno restarts and the likelihood that high-throughput worker processes will encounter termination requests.

The reference application indicates the method to cleanly shut a worker down on receiving the SIGTERM signal.

require 'resque/errors'
...
def self.perform(key)
  (new key).apply_watermark
rescue Resque::TermException
  Resque.enqueue(self, key)
end

Introspection

There are a number of ways to introspect Resque’s behavior. The best place to do this, the in-built Resque web dashboard. As mentioned earlier, the reference application exposes the dashboard here: http://localhost:5000/resque.

The dashboard allows you to inspect the following:

  • queues
  • workers
  • failed jobs and stack traces
  • current working jobs
  • useful redis stats

Using the console, you can inspect the above by using the following commands:

Resque.info
Resque.queues
Resque.redis
Resque.size(queue_name)
Resque.peek(queue_name, start=1, count=1)
Resque.workers
Resque.working

Finally, the reference application itself has a small dashboard on its index page that shows file upload and watermarking activity: http://localhost:5000.

Deployment

To deploy the reference application to Heroku, after it has been cloned locally, firstly create a new Heroku app, and provision a new instance of Redis:

$ heroku create --stack cedar
$ heroku addons:add rediscloud

Push the app to heroku.

$ git push heroku master

Set up the required AWS bucket and security config vars.

$ heroku config:set \
> AWS_S3_BUCKET_ORIGINALS=<insert your originals bucket name> \
> AWS_S3_BUCKET_WATERMARKED=<insert your watermarked bucket name> \
> AWS_ACCESS_KEY_ID=<insert your access key id> \
> AWS_SECRET_ACCESS_KEY=<insert your secret access key>

Check that this worked.

$ heroku config | grep AWS
AWS_ACCESS_KEY_ID:         access key id
AWS_S3_BUCKET_ORIGINALS:   your originals bucket name
AWS_S3_BUCKET_WATERMARKED: your watermarked bucket name
AWS_SECRET_ACCESS_KEY:     secret access key

Scale up the web and background workers.

$ heroku scale web=1 resque=1

Open the app in a browser.

$ heroku open

Open the resque web dashboard in a new browser tab using the “Resque Dashboard” link.

Open your AWS Management Console, and open the watermarked bucket.

Taking note of your heroku app’s name (eg vast-sierra-1234), post an image to the upload endpoint.

$ curl -F "file=@src.jpeg" "http://appname.herokuapp.com/upload"

A watermarked version of the image that was posted should be in your watermarked bucket in a few seconds (refresh the bucket view in the AWS console).

Furthermore, check the index page of the reference application to see a list of public URLs of recently watermarked images. Or, by examining the app logs.

$ heroku logs --tail

Look for a line such as this that indicates a public URL that indicates the location of your watermarked image.

app[resque.1]: Persisted watermarked file to S3: ...

Visit the public URL to see the watermarked image.