Live-Debugging Remote Clojure Apps with Drawbridge

Last Updated: 24 February 2015

clojure debugging

Clojure developers are used to a very interactive development environment in which the editor is closely integrated with the running process via a read/eval/print loop (REPL). It’s possible to reproduce this to a degree with actual Heroku dynos using heroku run lein repl, but this creates a new process rather than connecting to existing processes which are serving web requests. This makes it unsuitable for certain debugging tasks. Using nREPL and its drawbridge transport, it’s possible to connect a REPL session over HTTP, giving you direct access to a web dyno.

The code for the app below is available on Github and deployed to Heroku as well.

Setup

This article assumes the use of Leiningen 2. Download the latest release of Leiningen if you haven’t already.

Drawbridge is a transport for nREPL (Clojure’s networked REPL) that is implemented as a Ring handler, making it easily integrated into existing Ring applications. Let’s write a whimsical web application for calculating the magnitude of a given laugh.

$ lein new heroku-ring chortles
$ cd chortles
$ heroku apps:create
Creating hollow-stone-7912... done, stack is cedar-14
http://hollow-stone-7912.herokuapp.com/ | git@heroku.com:hollow-stone-7912.git
Git remote heroku added

Add com.cemerick/drawbridge to the :dependencies list in project.clj:

(defproject chortles "1.0.0-SNAPSHOT"
  :description "Just for laughs"
  :url "http://chortles.herokuapp.com"
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [ring/ring-jetty-adapter "1.1.0"]
                 [com.cemerick/drawbridge "0.0.6"]])

Basic application

The benefits of the live REPL are clearer if the application has some state to inspect; otherwise you might as well use the naive heroku run lein repl approach. Here is the implementation of our app in the file src/chortles/web.clj. It has no routing and simply returns a JSON response.

(ns chortles.web
  (:require [ring.adapter.jetty :as jetty]))

(defonce scores (atom []))

(defn calculate-chortle-magnitude [chortle]
  (let [sub-chortles (re-seq #"(?im)ha+" chortle)
        caps (apply + (for [sub sub-chortles c sub
                            :when (Character/isUpperCase c)] 1))]
    (+ (count sub-chortles) caps)))

(defn percentile [magnitude]
  (* 100.0 (/ (count (filter (partial >= magnitude) @scores))
              (count @scores))))

(defn app [req]
  (let [chortle (slurp (:body req))
        magnitude (calculate-chortle-magnitude chortle)]
    (swap! scores conj magnitude)
    {:status 200
     :headers {"Content-Type" "application/json"}
     :body (format "{\"%s\": %s, \"percentile\": %s}"
                   chortle magnitude (percentile magnitude))}))

(defn -main [& [port]]
  (let [port (Integer. (or port (System/getenv "PORT")))]
    (jetty/run-jetty #'app {:port port})))

Along with returning the magnitude of a given chortle, it also keeps a history of chortles in the scores atom and returns a percentile score for the given chortle compared to everything it has calculated so far. Let’s get this deployed.

$ git init
$ git commit -a -m "Initial commit"
$ git push heroku master

It should be all set and ready to serve responses to curl:

$ curl http://hollow-stone-7912.herokuapp.com -d hahahaha
{"hahahaha": 4, "percentile": 100.0}
$ curl http://hollow-stone-7912.herokuapp.com -d HAHAHAhahaHA
{"HAHAHAhahaHA": 14, "percentile": 100.0}
$ curl http://hollow-stone-7912.herokuapp.com -d HAH
{"HAH": 3, "percentile": 33.33333333333333}

Drawbridge handler

Now you want to be able to access the application’s internal state. This can be done by wrapping the handler with drawbridge middleware back in our web.clj file. The cemerick.drawbridge/ring-handler function needs to be wrapped in a few other standard ring middleware functions to pull the request apart; in a nontrivial app you would probably already have declared these as part of your own code.

(def drawbridge-handler
  (-> (cemerick.drawbridge/ring-handler)
      (keyword-params/wrap-keyword-params)
      (nested-params/wrap-nested-params)
      (params/wrap-params)
      (session/wrap-session)))

(defn wrap-drawbridge [handler]
  (fn [req]
    (if (= "/repl" (:uri req))
      (drawbridge-handler req)
      (handler req))))

(defn -main [& [port]]
  (let [port (Integer. (or port (System/getenv "PORT")))]
    (jetty/run-jetty (wrap-drawbridge app)
                     {:port port :join? false})))

This exposes an nREPL HTTP endpoint via ring middleware under the /repl path. Leiningen’s repl task can now be used to connect once it’s deployed.

$ git commit -a -m "Add nREPL middleware."
$ git push heroku master

Ensure that it’s working fine:

$ curl http://hollow-stone-7912.herokuapp.com -d HAHAHAhahaHA
{"HAHAHAhahaHA": 14, "percentile": 100.0}

Now connect via Leiningen:

$ lein repl :connect http://hollow-stone-7912.herokuapp.com:80/repl
Welcome to REPL-y!
[...]
user> (in-ns 'chortles.web)
#<Namespace chortles.web>
chortles.web> @scores
[14]

All the code entered at the REPL runs in the deployed process, making it ideal for debugging hard-to-reproduce issues.

Requests to apps with multiple dynos will be evenly distributed among the dynos. This makes for potentially-confusing scenarios; it is best to attempt to reproduce issues on a single-dyno app.

Adding authentication

The problem is that at this point anyone could just go in and enter arbitrary Clojure code:

$ lein repl :connect http://hollow-stone-7912.herokuapp.com:80/repl
chortles.web> (defn app [req] {:status 410, :headers {}, :body "DELETED"})

This is certainly not acceptable; clearly some security is necessary. The ring-basic-authentication dependency provides Ring middleware that makes it easy to protect certain parts of an app. Add this it to your project.clj :dependencies with [ring-basic-authentication "1.0.1"] and define an authenticated? function to read credentials from environment variables:

(defn authenticated? [name pass]
  (= [name pass] [(System/getenv "AUTH_USER") (System/getenv "AUTH_PASS")]))

With this function the drawbridge-handler can be protected:

(defn wrap-drawbridge [handler]
  (fn [req]
    (let [handler (if (= "/repl" (:uri req))
                    (basic/wrap-basic-authentication
                     drawbridge-handler authenticated?)
                    handler)]
      (handler req))))

Now add the credentials to the app’s config:

$ heroku config:set AUTH_USER=flynn AUTH_PASS=reindeerflotilla

Push the changes out:

$ git commit -a -m "now with security" && git push

Now the REPL is inaccessible to unauthenticated users:

$ lein repl :connect http://hollow-stone-7912.herokuapp.com:80/repl
ExceptionInfo clj-http: status 401
[...]
Bye for now!

But of course if credentials are provided (here in the URL scheme) it’s a different story:

$ lein repl :connect http://flynn:reindeerflotilla@hollow-stone-7912.herokuapp.com:80/repl
Welcome to REPL-y!
Clojure 1.4.0
[...]
user=> @chortles.web/scores
@chortles.web/scores
[13 22 1 42]

Inspect HTTP requests

It can be helpful for debugging purposes to intercept incoming requests both to inspect and to re-submit them at your leisure. Since Ring represents requests as simple Clojure maps, it’s easy to store these off temporarily. In the REPL, enter this code in the chortles.web namespace:

(defonce requests (atom []))

(defn app [req]
  (swap! requests conj req)
  (let [chortle (slurp (:body req))
        magnitude (calculate-chortle-magnitude chortle)]
    (swap! scores conj magnitude)
    {:status 200
     :headers {"Content-Type" "application/json"}
     :body (format "{\"%s\": %s, \"percentile\": %.2f}"
                   chortle magnitude (percentile magnitude))}))

This is the exact same definition of the app function except that it swaps the req argument into an atom for you to play with later. The changes will take effect immediately with no deploy needed, so make a few requests over curl again:

$ curl http://hollow-stone-7912.herokuapp.com -d hahahaha
{"hahahaha": 4, "percentile": 100.0}
$ curl http://hollow-stone-7912.herokuapp.com -d HAHAHAhahaHA
{"HAHAHAhahaHA": 14, "percentile": 100.0}
$ curl http://hollow-stone-7912.herokuapp.com -d HAH
{"HAH": 3, "percentile": 33.33333333333333}

Then back to the REPL:

user=> @chortles.web/requests
[{:remote-addr "10.102.5.151", :scheme :http, :request-method :post [...]}]

There should be three large request maps. To re-submit one of these requests, simply call the chortles.web/app function on one of them and it will be as if a user had submitted it from a web browser:

user=> (chortles.web/app (first @chortles.web/requests))
{:status 200, :headers {"Content-Type" "application/json"}, :body "{\"\": 0, \"percentile\": 25.00}"}

Note that the requests atom will fill up and could be a potential memory leak, so once you’re done debugging don’t forget to redefine the app function with the old version which doesn’t swap each request onto the requests atom. You can also flush the stored requests list by calling (swap! chortles.web/requests empty).

Other tools that offer Leiningen integration can be connected to this REPL endpoint. For instance, in Emacs invoking C-u M-x run-lisp will prompt for a command. Enter the lein repl [...] invocation above and Emacs will connect so that the standard code evaluation bindings like C-c C-r will evaluate the given code remotely. As nREPL integration becomes more widespread other environments will implement direct connection to remote nREPL servers without delegating that functionality to Leiningen.