Treasure Data
Last updated June 02, 2022
Table of Contents
Treasure Data Add-On lets Heroku users collect, store and analyze large amounts of data without sinking a lot of time learning about Hadoop, MPP databases and other powerful but time-consuming technologies. With Treasure Data, you can start analyzing your data today, not weeks and months later.
We are proven to scale. We store more than 2 trillion data records for our customers, adding hundreds of millions of data records every hour. Our customers run thousands of queries against their data on our system every day.
The Treasure Data Add-on offers instant setup of ‘Log Everything’ infrastructure for tracking, and understanding the user activities of your apps.
Our customers can collect user activities from their applications instantly and analyze the data through a SQL-like query language (Apache Hive) for better understanding of your users behavior. Typical use cases include:
- Building Reporting Feature to Your Customers
- Daily / Hourly Reports of Your Business Metrics
- Ranking Calculation
- Conversion Path Analytics
Provisioning the add-on
A list of all plans available can be found here.
One configuration parameter that Treasure Data introduces into your Heroku app is TREASURE_DATA_API_KEY
.
Treasure Data can be attached to a Heroku application via the CLI:
$ heroku addons:create treasure-data
-----> Adding treasure-data to sharp-mountain-4005... done, v18 (free)
heroku addons:open treasure-data
will lead you to our web console if you want.
Data Import: just write to STDOUT!
You can import data to Treasure Data by simply writing to STDOUT with a specific format. The format is:
@[database.table] JSON-In-ONE-LINE
Here’s an example in Ruby. The logs are uploaded every 5 minutes.
puts "@[production.login] #{{'uid'=>123}.to_json}"
puts "@[production.follow] #{{'uid'=>123, 'from'=>'@TreasureData', 'to'=>'@Heroku'}.to_json}"
puts "@[production.pay] #{{'uid'=>123, 'item_name'=>'Stone of Jordan', 'category'=>'ring', 'price'=>100, 'count'=>1}.to_json}"
If you can’t see your logs with “heroku logs” command, please check if your stdout is flushed correctly in your code.
CLI Setup
1) TD Toolbelt setup
At first, please download and install the Treasure Data Toolbelt for your development environment.
- Download MacOS Installer
- Download Windows Installer
- Linux (Redhat/CentOS, Debian/Ubuntu)
- or ‘gem install td’ for those who are familiar with Ruby.
2) Heroku CLI setup
heroku-td
CLI plugin is also required to bridge between heroku
CLI and td
CLI. Once you install the CLI plugin, you will be able to execute the heroku td
family of commands.
$ heroku plugins:install https://github.com/treasure-data/heroku-td.git
$ heroku td
usage: heroku td [options] COMMAND [args]
Analyze Your Dataset
Please access to your site to call TD.event.post()
. After several minutes, the data upload is done by td
gem. heroku td tables
shows your uploaded dataset.
$ heroku td tables
+------------+--------+------+-----------+
| Database | Table | Type | Count |
+------------+--------+------+-----------+
| production | login | log | 31232 |
| production | follow | log | 3132 |
| production | pay | pay | 132 |
+------------+--------+------+-----------+
Now you can issue the query into cloud by heroku td query
command. The example query below counts login per day.
$ td query -w -d testdb \
"SELECT \
TD_TIME_FORMAT(time, "yyyy-MM-dd", "PDT") AS day, \
COUNT(1) AS cnt \
FROM login \
GROUP BY TD_TIME_FORMAT(time, "yyyy-MM-dd", "PDT") \
ORDER BY cnt"
+------------+------+
| day | cnt |
+------------+------+
| 2012-05-26 | 4981 |
| 2012-05-27 | 4481 |
| 2012-05-28 | 481 |
+------------+------+
Migrating between plans
Application owners should carefully manage the migration timing to ensure proper application function during the migration process.
Use the heroku addons:upgrade
command to migrate to a new plan.
$ heroku addons:upgrade treasure-data:developer
-----> Upgrading treasure-data:developer to sharp-mountain-4005... done
Your plan has been updated to: treasure-data:developer
developer
is one of our plans. You can find out more about our price plans here.
Removing the add-on
Treasure Data can be removed via the CLI.
Note that your data stays on Treasure Data even if you remove the addon. If you want to purge your data, here is how you do it.
$ heroku addons:destroy treasure-data
-----> Removing treasure-data from sharp-mountain-4005... done, v20 (free)
Support
All Treasure Data support and runtime issues should be submitted via on of the Heroku Support channels. Any non-support related issues or product feedback is welcome at support@treasure-data.com.
Additional resources
- Heroku Addon documentation
- Data Import from Heroku Apps: Multiple Options
- Scheduled Queries
- Result Output