Background Tasks in Python with RQ

Last Updated: 28 November 2013

cedar python redis

Table of Contents

RQ (Redis Queue) makes it easy to add background tasks to your Python applications on Heroku. RQ uses a Redis database as a queue to process background jobs. To get started using RQ, you need to configure your application and then run a worker process in your application.

If you have questions about Python on Heroku, consider discussing it in the Python on Heroku forums. Both Heroku and community-based Python experts are available.

Configuration

To setup RQ and its dependencies, install it using pip:

$ pip install rq
Downloading/unpacking rq
  Downloading rq-0.1.2.tar.gz
  Running setup.py egg_info for package rq
  ...
Successfully installed rq

Next, record the new modification to your application’s requirements.txt file:

$ pip freeze > requirements.txt

Create a worker

Now that we have everything we need to create a worker process, let’s create one.

Create a file called worker.py. This module will listen to queued tasks and process them as they are received.

import os

import redis
from rq import Worker, Queue, Connection

listen = ['high', 'default', 'low']

redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')

conn = redis.from_url(redis_url)

if __name__ == '__main__':
    with Connection(conn):
        worker = Worker(map(Queue, listen))
        worker.work()

Now you can run your new worker process:

$ python worker.py

Queuing jobs

To send a job to your new Redis Queue, you need to send your jobs to Redis in your code. Let’s pretend we have a blocking function in an external module, utils.py:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

In your application, create a RQ queue:

from rq import Queue
from worker import conn

q = Queue(connection=conn)

And enqueue the function call:

from utils import count_words_at_url

result = q.enqueue(count_words_at_url, 'http://heroku.com')

The blocking function will automatically be executed in the background worker process.

Deployment

To deploy your new worker system to Heroku, you need to add the worker process to the Procfile in the root of the project.

worker: python worker.py

Additionally, provision an instance of Redis with the Redis To Go addon and deploy with a git push.

$ heroku addons:add redistogo
----> Adding redistogo to secret-samurai-42... done, v10 (free)

$ git push heroku master
Counting objects: 5, done.
Delta compression using up to 4 threads.
...

Once everything’s pushed up, scale your workers according to your needs:

$ heroku scale worker=1
Scaling worker processes... done, now running 1

Troubleshooting

View worker process output by filtering the logs with the -p flag and the name of the worker process type.

$ heroku logs -t -p worker

The worker process can be manually invoked for further isolation.

$ heroku run worker
Running worker attached to terminal... up, run.1