This add-on is operated by teowaki

Analyse everything going on in your application. BigData made easy

Last updated May 04, 2017

Datawaki is an add-on for providing analysis and real-time alerts for events and technical events generated at your application.

Adding Datawaki lets you get insights on what your users are doing within your application. All you need to do is store an entry into your application log with the information you want to analyse later, and Datawaki will take care of everything else.

Once the information is in your log, you won’t need any development or integration to create new reports, customized interactive queries or real-time alerts. By using Datawaki, you will save precious development time. Even non-developers, such as your business and marketing people, will have the power to create their own analysis.

Setting Datawaki is as easy as setting up Google Analytics. After the simple initial set-up, you can use on a daily basis to see how your business is evolving. and to analyse every kind of event. You can also track individual sessions from individual users if you want, rather than having only aggregated data.

Do you want to get an alert the next time a user registers for your application or hires a paid plan? Would you like to see if an individual user has been using your application more after that e-mail you sent her? In which countries did you get more sales last month? Datawaki can help you answer all those questions.

And of course your data is yours. If you want to get a snapshot of all the data you have sent to Datawaki in the past, we will provide it for you.

Right now Datawaki is available only using the web interface, but we are working on providing API access for clients.

A little peek at the technology behind Datawaki

We use Big Data techniques, so you can store as many events as you want, and you will always get a fast response time. With Datawaki you will never get out of space for event-data storage and performance will never slow down, so you can analyse absolutely everything going on without any penalties.

Our Big Data backend is Google BigQuery, which is the same infrastructure powering internal Google services such as Adwords or Gmail. Our engineers have been using BigQuery since it was publicly launched, and one of our founders is one of the few experts in BigQuery, having the official recognition of being a Google Developer Expert, which gives him access to the Google Cloud team to give feedback and to be informed of new features before they are released.

Provisioning the add-on

Datawaki can be attached to a Heroku application via the CLI:

$ heroku addons:create datawaki
-----> Adding datawaki to sharp-mountain-4005... done, v18 (free)

Once Datawaki has been added a DATAWAKI_URL setting will be available in the app configuration and will contain the canonical URL used to access the newly provisioned Datawaki service instance. This can be confirmed using the heroku config:get command.

$ heroku config:get DATAWAKI_URL
https://api.datawaki.com/heroku/resources/resource_id

After installing Datawaki the application should be configured to fully integrate with the add-on.

Generic instructions

To send any event to Datawaki, all you need to do is writing a JSON object to your Heroku log. The format of this message should be:

{
   "datawaki":{
      "origin":"94.174.83.188",
      "operation":"POST",
      "scope":"/account",
      "status":"200",
      "session_ref":"d661542a23412830c9bbbf01a3c932dd",
      "user_ref":"1234",
      "extra":{
      }
   }
}

The JSON object MUST have a root named datawaki, otherwise it will be ignored by Datawaki. All the fields are optional, so feel free to leave blank any fields not applicable for your use case.

If you want to send other data with your event, you can use the extra field to send as many fields as you want. For example, you might want to send information about a purchase with information about the product like this:

{
  "datawaki":{
     "origin":"94.174.83.188",
     "operation":"POST",
     "scope":"/order",
     "status":"201",
     "session_ref":"d661542a23412830c9bbbf01a3c932dd",
     "user_ref":"1234",
     "extra":{
        "article_ref": "black_ford",
        "price": "300",
        "currency": "USD"
     }
  }
}

The session_ref and user_ref fields are designed so you can track behaviour of individual users or of individual sessions. To avoid any privacy issues, we strongly recommend you don’t send information that can be used to identify a user, such as emails, names or social security numbers. Instead, we recommend you send identifiers that make sense for you, but would be useless for any external person.

The origin, operation, scope and status fields can be used for any purposes you want. For example you might want to send this JSON

{
    "datawaki":{
       "origin":"94.174.83.188",
       "operation":"GET",
       "scope":"/newsletter",
       "status":"OK",
       "session_ref":"d661542a23412830c9bbbf01a3c932dd",
       "user_ref":"1234",
       "extra":{
       }
    }
}

or this one

{
    "datawaki":{
       "origin":"London",
       "operation":"DELIVERY",
       "scope":"VC42J-08GJ",
       "status":"DISPATCHED",
       "session_ref":"d661542a23412830c9bbbf01a3c932dd",
       "user_ref":"1234",
       "extra":{
         "destination": "Brighton",
         "post_code": "BN1 1UE",
         "lat": 50.8237,
         "long": -0.1384
       }
    }
}

The semantics of the fields are up to you. Just make sure you are consistent and you are using always the same fields for the same things. Otherwise, when you try to analyse the stored data it will be impossible to make sense of what you have.

Please make sure you have configured your application to properly log into Heroku (logs should be sent to STDOUT). You will find more information about logging in the Heroku logging guide, and specific instructions for different languages in the next sections of this help.

Using with Rails

Open your config.ru file and add the following line, if not already present

$stdout.sync = true

This will disable stdout buffering, so datawaki can receive your log entries in real-time.

Now you will need to make sure your rails Logger is pointing to stdout. If you are using the recommended rails_12factor gem, then you don’t need to do anything else. If not, just add it to your Gemfile

gem 'rails_12factor'

Update application dependencies with bundler.

$ bundle install

To store an event in Datawaki, you only need to write it to the Rails.logger

Rails.logger.info your_object.to_json

Using with plain Ruby

Open your config.ru file and add the following line, if not already present

$stdout.sync = true

This will disable stdout buffering, so datawaki can receive your log entries in real-time.

To store an event in Datawaki, you only need to write it to stdout

puts your_object.to_json

Using with Node.js

All you need to do is sending your JSON object to STDOUT.

console.log(JSON.stringify(yourObject));

Using with Scala

All you need to do is sending your JSON object to STDOUT.

println(jsonString)

Using with Play!

All you need to do is sending your JSON object to STDOUT using the play! standard logger. If you still haven’t configured your play! logger to write to STDOUT, you can do so by visiting this official Play! doc or this Stackoverflow question now you can simply log your JSON object by doing

logger.info(yourJSONObject);

Using with Grails

All you need to do is sending your JSON object to STDOUT. Make sure you have your logger properly configured then just send events to Datawaki by writing to the log.

log.info yourJSONString

Using with Clojure

All you need to do is sending your JSON object to STDOUT.

 (println JSONString)

Using with Java

All you need to do is sending your JSON object to STDOUT.

System.out.println(jsonString);

Using with PHP

All you need to do is sending your JSON object to STDOUT. We suggest you use Monolog, as explained in the second half of this official heroku PHP guide. But you can really use any other method for logging to STDOUT as long as you output a compliant JSON.

$log->addInfo(jsonObject);

Using with Python/Django

All you need to do is sending your JSON object to STDOUT. We suggest using the built-in logger for that. If you still haven’t configured your logger to write to STDOUT, you can do so by visiting this stackoverflow question

now you can simply log your Json object by doing

logger.info(yourJSONString)

Monitoring and logging

Stats and the current state of Datawaki can be displayed via the CLI.

$ heroku datawaki:command
example output

Dashboard

The dashboard can be accessed via the CLI:

$ heroku addons:open datawaki
Opening datawaki for sharp-mountain-4005…

or by visiting the Heroku Dashboard and selecting the application in question. Select Datawaki from the Add-ons menu.

Alternatively, if you are already logged into Heroku, you can navigate to https://datawaki.com in your browser For more information on the features available within the Datawaki dashboard please see the docs at datawaki.

Troubleshooting

If you can’t see your data using the dashboard, double check you are sending a well-formatted JSON to the log with the fields described above.

Migrating between plans

Use the heroku addons:upgrade command to migrate to a new plan.

$ heroku addons:upgrade datawaki:newplan
-----> Upgrading datawaki:newplan to sharp-mountain-4005... done, v18 ($49/mo)
       Your plan has been updated to: datawaki:newplan

Removing the add-on

Datawaki can be removed via the CLI.

$ heroku addons:destroy datawaki
-----> Removing datawaki from sharp-mountain-4005... done, v20 (free)

Before removing Datawaki a data export can be performed by contacting hello@datawaki.com

Support

All Datawaki support and runtime issues should be submitted via on of the Heroku Support channels. Any non-support related issues or product feedback is welcome at hello@datawaki.com.