Blitz

This add-on is operated by Spirent Communications

Load Testing from the Cloud

Blitz

Last Updated: 19 March 2014

Table of Contents

Introduction

Blitz is a Heroku add-on for providing self-service, cloud-based, integrated load and performance testing for your app.

In less than 5 minutes, you could be running tests against your web site, API, and mobile apps from around the world, using Blitz results to understand how users are experiencing your app, or setting up alerts to notify you 24x7 when something goes down. Blitz can help at every stage of the dev cycle:

  • Staging. Are all features and capabilities working as designed? Blitz can run load and performance tests against your staging app to help identify performance bottlenecks on a continuous basis as a CI task. It’s also the simplest way to find out how many dynos you need to scale out.
  • Production. Is auto-scaling kicking in, database sharding working as expected, and your wonderful RESTful API not throwing up 500’s at your users? Blitz can continuously monitor your app 24x7 from around the world, emulating a single user or hundreds of users all day, everyday, and notify you if anything goes wrong.
  • Operations. Is your app down? Blitz’s performance monitoring can alert you immediately when something is wrong – and help you find and fix problems by importing logs, APM metrics, and analytics from other tools. Blitz also supports converting monitored data into performance tests you can use to verify any fixes you make.

The basic Blitz plan lets you run tests simulating up to 250 users for free (see all Blitz pricing plans). Test using the Blitz UI or the Blitz API with supported client libraries for Java, Ruby, Python, Play!, and Node.js.

Installing the Blitz add-on

You can install the Blitz add-on for any Heroku application using the CLI:

$ heroku addons:add blitz:250
-----> Adding blitz to sharp-mountain-4005... done, v18 (free)

Blitz will automatically create an account for you on Blitz.io, send you a welcome email, and authorize the associated Heroku app and all custom domains. To confirm a successful install, use heroku config command:

$ heroku config | grep blitz
blitz    => http://user:pass@instance.ip/resourceid

Integrating Blitz

Blitz doesn’t integrate with your app at a code-level and isn’t restricted to a particular language, so you can use Blitz UI with any app hosted on Heroku. We also support several API clients and provide documentation to help you integrate Blitz into your app environment:

After you’ve completed the previous section you’re all set to use the Blitz UI. To use the Blitz API, there is a bit more work to do that involves installing the Blitz gem and adding some unique data to your app to prove to Blitz that you are indeed the devops person.

Using the Blitz UI

To the Blitz UI to run load tests, login to Heroku and use the Heroku web interface to find your app, then open the Add-ons menu and select Blitz:

Or, to launch the Blitz UI from the CLI, type:

$ heroku addons:open blitz
Opening blitz for myapp.heroku.com

Because Blitz supports single sign-on, you will be logged into the blitz.io web interface using your Heroku credentials.

Your first encounter with the Blitz UI is going to be the Blitz bar. The command-line look and feel is intentional, as we find that most developers are not big fans of complex forms.

In the Blitz bar, type something like this:

-p 1-250:60 -r california myapp.heroku.com

This runs a 1-minute rush (sustained load test) against your app from California (or leave out -r california to use a randomly-picked region around the world). Results include:

  • Responses returned by your app:

  • Highlights in easy-to-read, organized format:

  • Reports that you can view, customize and print, or export data to csv:

Looking for a single-click load and performance test? Check out the Blitz Chrome and Firefox browser add-ons that let you instantly run a load test by simply navigating to a page in your app.

Using the Blitz API

When you use the Blitz UI to run load tests, the single-sign on from Heroku takes care of letting Blitz know that it’s OK to run tests against your app. However, when you use the API to run tests, you’ll need to do a bit more work (not much!) to prove to Blitz that you are indeed the devops person for that app.

You’ll need to add some unique data to your app that Blitz can check for, install the Blitz gem, and enter some values from the Blitz Settings/API-Key page. Here’s how:

  1. Login to the Blitz site and browse to Settings > API Key. You should see something like this:

    Make a note of your User-ID and API-Key values as you'll need them in a few moments.

  2. Create a text file with the API-Key as the title, place that file at the root directory of your app, and push your changes to Heroku.

    For example, if my API-Key is 83926748-agid8392-jk8936at-i8392057, I create 83926748-agid8392-jk8936at-i8392057.txt and place next to my index.html. After I’ve committed and pushed that file to Heroku, the text file will be waiting for when Blitz queries it for authentication.

  3. Install the blitz gem and authenticate using your User-ID and API-Key:

    $ gem install blitz
    $ blitz api:init
    Enter your blitz credentials. You can find this in Settings/API Key.
    User-ID: **your-user-id**
    API-Key: *your-api-key*

After authenticating, you are ready to run a load test against your app! From the command line, try this to run a 1-minute rush (sustained load test) on your app from California (or leave out -r california to use a randomly-picked region around the world):

$ blitz curl -p 1-250:60 -r california myapp.heroku.com
rushing from california...

  Time  Users Response     Hits Timeouts   Errors   Hits/s Mbps
  2.5s     11   0.212s        7        0        0
  5.0s     21   0.189s       35        0        0    11.14 0.00
  7.5s     32   0.253s       85        0        0    19.94 0.01
 10.0s     42   0.201s      156        0        0    28.29 0.01
 12.5s     53   0.219s      253        0        3    38.64 0.02
 15.1s     63   0.202s      363        0        4    43.83 0.02
 17.6s     73   0.204s      502        0        4    55.42 0.02
 20.1s     84   0.196s      660        0        4    63.03 0.03
 22.6s     94   0.195s      844        0        4    73.36 0.03
 25.1s    105   0.191s     1049        0        4    81.74 0.03
 30.1s    126   0.260s     1510        0        4    97.68 0.04
 32.6s    136   0.205s     1759        0        4    99.29 0.04
 35.1s    146   0.350s     2036        0        4   110.45 0.05
 37.6s    157   0.206s     2334        0        4   118.81 0.05
 40.1s    168   0.256s     2665        0        4   131.98 0.06
 42.6s    178   0.317s     2931        0        4   106.05 0.05
 45.2s    188   0.270s     3304        0        4   148.49 0.06
 50.2s    210   0.224s     4144        0        4   162.29 0.07
 52.7s    220   0.235s     4583        0        4   175.04 0.07
 55.2s    230   0.236s     4949        0        4   145.94 0.07
 57.7s    241   0.356s     5420       14        4   187.80 0.07
 60.2s    189   0.203s     5886       14        4   185.52 0.08
 62.8s      0   0.203s     5886       14        4     0.00 0.00

To see pretty charts and graphs, run the rush from the Blitz UI.

$ heroku logs -t | grep 'blitz:250 pattern'

Migrating between plans

Application owners should carefully manage the migration timing to ensure proper application function during the migration process.

The basic Blitz plan lets you run unlimited 1-minute load tests simulating up to 250 users for free and is ideal for a single dyno or staging app. If your app is using multiple dynos and/or is production, consider upgrading to a paid plan that can increase the number of users to 1000 or even 5000 (see all Blitz plans).

All add-ons to your app are billed by Heroku and pro-rated to the second. On the Blitz 1000 plan, this means that if you want to see your app scale out to 1,000 concurrent users, it’s only going to cost you $1/hour.

To upgrade to the Blitz 1000 plan:

$ heroku addons:upgrade blitz:1000
-----> Upgrading blitz:1000 to sharp-mountain-4005... done, v18 ($799/mo)
        Your plan has been updated to: blitz:1000

Now run as many load tests as you want up to 1,000 concurrent users. Here is the important part: When done testing, simply switch back to the free plan. Cost to you? $1 for the hour. We do abort running load testing on plan changes, so no cheating! :)

To downgrade back to the Blitz free plan:

$ heroku addons:downgrade blitz:250

Troubleshooting

We’ve listed a few of the common problems that people experience getting Blitz up and running. If you don’t see your question answered here, please visit the Blitz support site.

Authorizing

If you try to run a load test using the API and get something that looks like this:

!! You haven't verified that you are the devops dude for www.artemisclydefrog.com. Make
!! sure the following URL is reachable and returns the string '42'.
!!
!! http://www.artemisclydefrog.com:80/mu-83926748-agid8392-jk8936at-i8392057
!!
!! We tried checking the following URLs and got this
!!
!! 404 | /mu-83926748-agid8392-jk8936at-i8392057
!! 404 | /mu-83926748-agid8392-jk8936at-i8392057/
!! 404 | /mu-83926748-agid8392-jk8936at-i8392057.txt
!!
!! If your app is RESTfully built with sinatra or rails, simply add this route:
!!
!! get '/mu-83926748-agid8392-jk8936at-i8392057' do
!!     '42'
!! end
!!
!! Once this is done, you can blitz www.artemisclydefrog.com all you want.

… it means that you haven’t configured your app to play nice with Blitz just yet. You’ll need to follow the steps described in Using the Blitz API section of the this document.

Where are my Blitz credentials?

Your Blitz credentials authorize you to access the Blitz API. You’ll need to provide these credentials the first time you use the Blitz API. You can find your credentials in the API Key section of Blitz Settings, as shown below:

  • User-ID. This value is automatically set to your Heroku username. In your Heroku app config, this is also the BLITZ_API_USER config variable.
  • API-Key. This value is dynamically generated by Blitz. In your Heroku app config, this is also the BLITZ_API_KEY config variable.

Configuring timeouts

Blitz uses a default connect and read timeout of 1 second for both Sprinting and Rushing. If the TCP connect or the HTTP response takes longer than this timeout, we simply close that connection and mark it as a timeout. The free plan allows you to increase the timeout to a maximum of 10 seconds while paid plans allow timeouts of up to 30 seconds.

To change the default timeout to 10 seconds, use the -T 10000 option.

Timeouts tend to increase with concurrency if you have lock contention on some shared resource. Consider using in-memory caching and avoiding global locks to prevent this from happening. A page that loads under 250ms (for a single user) doesn’t necessarily mean it’s going to load that fast for 100 simultaneous users.

Removing the Blitz add-on

You can remove the Blitz add-on using the CLI.

This will destroy all associated data and cannot be undone!

$ heroku addons:remove blitz
-----> Removing blitz from sharp-mountain-4005... done, v20 (free)

Support

All Blitz support and runtime issues should be logged with Heroku Support at https://support.heroku.com. Any non-support related issues or product feedback is welcome at hello@blitz.io and twitter @blitz_io.

Additional resources

Additional resources are available at: