This add-on is operated by Mu Dynamics Inc
Load Testing from the Cloud
Table of Contents
In less than 5 minutes, you could be running tests against your web site, API, and mobile apps from around the world, using Blitz results to understand how users are experiencing your app, or setting up alerts to notify you 24x7 when something goes down. Blitz can help at every stage of the dev cycle:
- Staging. Are all features and capabilities working as designed? Blitz can run load and performance tests against your staging app to help identify performance bottlenecks on a continuous basis as a CI task. It’s also the simplest way to find out how many dynos you need to scale out.
- Production. Is auto-scaling kicking in, database sharding working as expected, and your wonderful RESTful API not throwing up 500’s at your users? Blitz can continuously monitor your app 24x7 from around the world, emulating a single user or hundreds of users all day, everyday, and notify you if anything goes wrong.
- Operations. Is your app down? Blitz’s performance monitoring can alert you immediately when something is wrong – and help you find and fix problems by importing logs, APM metrics, and analytics from other tools. Blitz also supports converting monitored data into performance tests you can use to verify any fixes you make.
The basic Blitz plan lets you run tests simulating up to 250 users for free (see all Blitz pricing plans). Test using the Blitz UI or the Blitz API with supported client libraries for Java, Ruby, Python, Play!, and Node.js.
Installing the Blitz add-on
You can install the Blitz add-on for any Heroku application using the CLI:
$ heroku addons:create blitz:250 -----> Adding blitz to sharp-mountain-4005... done, v18 (free)
Blitz will automatically create an account for you on Blitz.io, send you a welcome email, and authorize the associated Heroku app and all custom domains. To confirm a successful install, use
heroku config command:
$ heroku config | grep blitz blitz => http://user:email@example.com/resourceid
Blitz doesn’t integrate with your app at a code-level and isn’t restricted to a particular language, so you can use Blitz UI with any app hosted on Heroku. We also support several API clients and provide documentation to help you integrate Blitz into your app environment:
After you’ve completed the previous section you’re all set to use the Blitz UI. To use the Blitz API, there is a bit more work to do that involves installing the Blitz gem and adding some unique data to your app to prove to Blitz that you are indeed the devops person.
Using the Blitz UI
To the Blitz UI to run load tests, login to Heroku and use the Heroku web interface to find your app, then open the Add-ons menu and select Blitz:
Or, to launch the Blitz UI from the CLI, type:
$ heroku addons:open blitz Opening blitz for myapp.heroku.com
Because Blitz supports single sign-on, you will be logged into the blitz.io web interface using your Heroku credentials.
Your first encounter with the Blitz UI is going to be the Blitz bar. The command-line look and feel is intentional, as we find that most developers are not big fans of complex forms.
In the Blitz bar, type something like this:
-p 1-250:60 -r california myapp.heroku.com
This runs a 1-minute rush (sustained load test) against your app from California (or leave out
-r california to use a randomly-picked region around the world). Results include:
Responses returned by your app:
Highlights in easy-to-read, organized format:
Reports that you can view, customize and print, or export data to csv:
Using the Blitz API
When you use the Blitz UI to run load tests, the single-sign on from Heroku takes care of letting Blitz know that it’s OK to run tests against your app. However, when you use the API to run tests, you’ll need to do a bit more work (not much!) to prove to Blitz that you are indeed the devops person for that app.
You’ll need to add some unique data to your app that Blitz can check for, install the Blitz gem, and enter some values from the Blitz Settings/API-Key page. Here’s how:
Login to the Blitz site and browse to Settings > API Key. You should see something like this:
Make a note of your User-ID and API-Key values as you'll need them in a few moments.
Create a text file with the API-Key as the title, place that file at the root directory of your app, and push your changes to Heroku.
For example, if my API-Key is 83926748-agid8392-jk8936at-i8392057, I create 83926748-agid8392-jk8936at-i8392057.txt and place next to my index.html. After I’ve committed and pushed that file to Heroku, the text file will be waiting for when Blitz queries it for authentication.
Install the blitz gem and authenticate using your User-ID and API-Key:
$ gem install blitz $ blitz api:init Enter your blitz credentials. You can find this in Settings/API Key. User-ID: **your-user-id** API-Key: *your-api-key*
After authenticating, you are ready to run a load test against your app! From the command line, try this to run a 1-minute rush (sustained load test) on your app from California (or leave out
-r california to use a randomly-picked region around the world):
$ blitz curl -p 1-250:60 -r california myapp.heroku.com rushing from california... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 11 0.212s 7 0 0 5.0s 21 0.189s 35 0 0 11.14 0.00 7.5s 32 0.253s 85 0 0 19.94 0.01 10.0s 42 0.201s 156 0 0 28.29 0.01 12.5s 53 0.219s 253 0 3 38.64 0.02 15.1s 63 0.202s 363 0 4 43.83 0.02 17.6s 73 0.204s 502 0 4 55.42 0.02 20.1s 84 0.196s 660 0 4 63.03 0.03 22.6s 94 0.195s 844 0 4 73.36 0.03 25.1s 105 0.191s 1049 0 4 81.74 0.03 30.1s 126 0.260s 1510 0 4 97.68 0.04 32.6s 136 0.205s 1759 0 4 99.29 0.04 35.1s 146 0.350s 2036 0 4 110.45 0.05 37.6s 157 0.206s 2334 0 4 118.81 0.05 40.1s 168 0.256s 2665 0 4 131.98 0.06 42.6s 178 0.317s 2931 0 4 106.05 0.05 45.2s 188 0.270s 3304 0 4 148.49 0.06 50.2s 210 0.224s 4144 0 4 162.29 0.07 52.7s 220 0.235s 4583 0 4 175.04 0.07 55.2s 230 0.236s 4949 0 4 145.94 0.07 57.7s 241 0.356s 5420 14 4 187.80 0.07 60.2s 189 0.203s 5886 14 4 185.52 0.08 62.8s 0 0.203s 5886 14 4 0.00 0.00
To see pretty charts and graphs, run the rush from the Blitz UI.
$ heroku logs -t | grep 'blitz:250 pattern'
Migrating between plans
Application owners should carefully manage the migration timing to ensure proper application function during the migration process.
The basic Blitz plan lets you run unlimited 1-minute load tests simulating up to 250 users for free and is ideal for a single dyno or staging app. If your app is using multiple dynos and/or is production, consider upgrading to a paid plan that can increase the number of users to 1000 or even 5000 (see all Blitz plans).
All add-ons to your app are billed by Heroku and pro-rated to the second. On the Blitz 1000 plan, this means that if you want to see your app scale out to 1,000 concurrent users, it’s only going to cost you $1/hour.
To upgrade to the Blitz 1000 plan:
$ heroku addons:upgrade blitz:1000 -----> Upgrading blitz:1000 to sharp-mountain-4005... done, v18 ($799/mo) Your plan has been updated to: blitz:1000
Now run as many load tests as you want up to 1,000 concurrent users. Here is the important part: When done testing, simply switch back to the free plan. Cost to you? $1 for the hour. We do abort running load testing on plan changes, so no cheating! :)
To downgrade back to the Blitz free plan:
$ heroku addons:downgrade blitz:250
We’ve listed a few of the common problems that people experience getting Blitz up and running. If you don’t see your question answered here, please visit the Blitz support site.
If you try to run a load test using the API and get something that looks like this:
!! You haven't verified that you are the devops dude for www.artemisclydefrog.com. Make !! sure the following URL is reachable and returns the string '42'. !! !! http://www.artemisclydefrog.com:80/mu-83926748-agid8392-jk8936at-i8392057 !! !! We tried checking the following URLs and got this !! !! 404 | /mu-83926748-agid8392-jk8936at-i8392057 !! 404 | /mu-83926748-agid8392-jk8936at-i8392057/ !! 404 | /mu-83926748-agid8392-jk8936at-i8392057.txt !! !! If your app is RESTfully built with sinatra or rails, simply add this route: !! !! get '/mu-83926748-agid8392-jk8936at-i8392057' do !! '42' !! end !! !! Once this is done, you can blitz www.artemisclydefrog.com all you want.
… it means that you haven’t configured your app to play nice with Blitz just yet. You’ll need to follow the steps described in Using the Blitz API section of the this document.
Where are my Blitz credentials?
Your Blitz credentials authorize you to access the Blitz API. You’ll need to provide these credentials the first time you use the Blitz API. You can find your credentials in the API Key section of Blitz Settings, as shown below:
User-ID. This value is automatically set to your Heroku username. In your Heroku app config, this is also the
API-Key. This value is dynamically generated by Blitz. In your Heroku app config, this is also the
Blitz uses a default connect and read timeout of 1 second for both Sprinting and Rushing. If the TCP connect or the HTTP response takes longer than this timeout, we simply close that connection and mark it as a timeout. The free plan allows you to increase the timeout to a maximum of 10 seconds while paid plans allow timeouts of up to 30 seconds.
To change the default timeout to 10 seconds, use the
-T 10000 option.
Timeouts tend to increase with concurrency if you have lock contention on some shared resource. Consider using in-memory caching and avoiding global locks to prevent this from happening. A page that loads under 250ms (for a single user) doesn’t necessarily mean it’s going to load that fast for 100 simultaneous users.
Removing the Blitz add-on
You can remove the Blitz add-on using the CLI.
This will destroy all associated data and cannot be undone!
$ heroku addons:destroy blitz -----> Removing blitz from sharp-mountain-4005... done, v20 (free)
All Blitz support and runtime issues should be logged with Heroku Support at https://support.heroku.com. Any non-support related issues or product feedback is welcome at firstname.lastname@example.org and twitter @blitz_io.
Additional resources are available at: