Building JS graphs using Backbone and Rickshaw

By Richard Powell, Frontend Engineer at Server Density.
Published on the 19th September, 2013.

Recently I had the pleasure of overhauling our time-series metric graphs, a core feature for Server Density. The overhaul had a few key objectives.

  • Maximise context & insight. Graphs are about the insight which comes from context. Context is provided by the metric, the time-range, the y values and a comparison of metrics to others. Once a graph has context it can reveal patterns and highlight problems quicker than any other method of displaying data. Every line of code in this overhaul comes back to this simple point.
  • Remove complexity. Our agent makes it simple to monitor metrics, we needed to make it simple to render them. There are command line tools available for plotting graphs, but these can be complex and verbose. Or you could roll your own solution, but who’s got time to do that? We needed an interface that removes complexity via an intuitive and snappy interface.
  • Compare multiple metrics in one view. Context is not always limited to a single metric. It might be necessary to compare IO Stats with CPU Stats to reveal the cause of a problem.
  • Filter metrics. A device may have multiple CPU’s and different metrics available for each of those CPU’s. We needed to never exclude data, whilst making it easy to focus on your important metrics and reduce instances of “no data available”.
  • Easily change time-ranges. Problems occur in a set point in time. Only by examining the available metrics surrounding that point can you gain enough insight to diagnose a problem. It needed to be easy to change time-ranges.
  • Persist useful preferences. Different devices have different configurations and different metrics. As such different time-ranges are relevant to different devices. We needed to provide sensible defaults whilst remembering your preferences.

So, how did our design fulfil these objectives? Daniele and Harry did a great job.

Design

Because our design vertically stacks graphs rather than arranging them in a grid format, it’s easy to compare two metrics at the same point in time. To further aid metric comparisons we added tooltips that synchronise with every graph on the page.

Comparing IO stats with disk usage

The above image also demonstrates how we are revealing the metric hierarchy via the drop downs. The user can filter two levels within the metric and they can also turn series on or off via the legend. There’s an inherent flexibility in this system as graphs are highly customisable and can be presented alongside any metric or combination of metrics. This proved especially useful recently when Wes was debugging performance issues using multiple graphs for IO Stats and CPU Stats.

Daniele and Harry were also careful to remove superfluous visual clutter that slows down insight. Everything on the graphs screen is functional, contrast and legibility has been maximised and nothing interferes with the interpretation of data. Now I’m going to talk about how we actually implemented these designs.

Data

We have 3 pieces of data to get out of the back-end and into JavaScript’s warm embrace on the front-end before we could render a graph

  • The persisted graphs
  • Series data for each graph
  • The available metrics

Let’s cover each endpoint now.

Persisted Graph Data

The first task was to get any previously persisted graphs for a device. ┬áThis ensures that any graphs the user adds, removes or edits remains the same the next time they load the page. For this we make a simple API call which returns the last used time-preset (we don’t persist specific dates) and an array of graphs, for example. If the user has never visited this device before we default to some sensible metrics. Here’s an example of the data that we return:

{
    "timePreset": {
        "key": "30 minutes ago",
        "name": "Last 30 minutes"
    },
    "graphs": [{
        "id": 1567570825,
        "key": "ioStats",
        "name": "IO stats",
        "filters": {
            "ioStats": {
                "dm": "all"
            }
        }
    }],
}  

 

From this data we can construct the URL’s to get the rest of the data.

Series Data

For this we developed an API endpoint with queries for device, time range, metric, sub-metric and sub-sub-metric:

/svc/metrics/graphs/:deviceid?start=:time&end=:time&filter={iostats:{dm:all}}

 

Here, we are getting all the ioStats for dm between the start time and end time for the device with that id. The data is returned in a format that matches the metrics query and added to a graphs collection in Backbone. More on that later.

Rickshaw requires its data as a flat array where each object contains the information for a series: name, color, x & y data. However our metrics API is used for more than graphing so we decided it should return more information regarding the nesting of metrics than a flat array would allow. Here’s a simple example with truncated data:

[{
    "name": "IO stats",
    "key": "ioStats",
    "tree": [{
        "name": "dm",
        "tree": [{
            "name": "Kilobytes read",
            "data": [{
                "x": 1378814872,
                "y": 8
            }]
        }, {
            "name": "Average Que Length",
            "data": [{
                "x": 1378814872,
                "y": 16
            }]
        }]
    }]
}]

 

I’m a big believer that when it comes to graphing, the front-end should be as dumb as possible, simply presenting the data it receives. Unfortunately this implementation meant we had to do a little data parsing on the front-end, but certainly nothing too complex.

Available Metrics Data

The final piece of data was the available metrics for that device at that time-range.

A user may configure devices differently and the configuration of that device may change over time. One week a device could be without plugins, the next week it might include several. Hence we needed a way to get the available metrics for a device at the time-range the user has selected. A third and final API call gives us a collection of metrics from which we can populate one, two or three drop-downs depending on how complex that metric is:

Server Density Dropdowns

Backbone

The backbone implementation of these graphs can be split loosely into 3 areas.

The graphs page view. This configures the graphs collection and instantiates new graph views for each persisted graph. Initially it acts as a bridge between the graphs collection and the individual graph views. It handles the event of a user adding a new graph and it listens to events on the graphs collection for removing.

The graph collection. This is a single code interface for 3 collections: the persisted graphs, the graphs and the available metrics. Upon initialisation it tells the persisted graph collection to GET the information regarding what graphs have been persisted. Once this completes, events trigger GET requests for the available metrics and the series data for each persisted graph.

The graph view. This is a base view, one for each graph that manages child views for Rickshaw and for the drop-down area of the graph. It initialises its child views and then passes models to them once they arrive. From that point on the child views subscribe to events on the models and on a mediator object, meaning they are independent of their base views.

This implementation works well as it allows each graph view to render into different states as quickly as possible:

  • Once the persisted graph data has been fetched
  • Once the series data for a graph exists
  • Once the available metrics data exist

Furthermore it groups the code into logical modules: the persistence data, the series data, the metrics data, the page view, each graph view, the rickshaw view, the drop-downs views, each drop-down view and so on. Furthermore the use of the mediator pattern and the delegation pattern ensured that each module could communicate transparently with other modules where needed.

D3, Rickshaw

Rickshaw is a fantastic graphing library for time series graphs built on-top of D3, the data visualisation library. Check out Rickshaw’s demo page to get an idea of what it’s capable of. Rickshaw is fantastic as it does one thing and one thing very well; time-series graphs. But I like Rickshaw for more than that:

  • Its source code & namespace architecture is very easy to understand and extend.
  • The way it requires data to be formatted and options to be specified is very intuitive.
  • It’s very easy to choose only the elements and functionality you want as the series, the xAxis, the yAxis, the legend, the tooltips, interactivity and more are all independent.
  • It uses SVG where needed and plain old HTML where it’s easier.
  • It’s very feature rich.

One example of where Rickshaw is easy to extend is it’s time fixture, which we customise in two ways. First we override the formatTime and formatDate methods, converting the UNIX time stamps to the user’s pre-configured timezone. In these methods, we also specify the date format as HH:mm or D MMM YYYY depending on how large the time-range is. Next we add units to the time fixture so that we can display ticks every 4 hours, every 3 days, every 5 minutes and every 2 minutes, something that suits our time presets but is not available as default in Rickshaw. Here is the unit object for 4 hours:

{
    name: '4 hour',
    seconds: 3600 * 4,
    formatter: function(d) { return self.formatTime(d) }
}

 

Because Rickshaw exposes its series data it’s easy to add interactivity using Backbone views. We do this to handle legend functionality. Yes, Rickshaw provides this, but it wasn’t quite what we wanted. We wanted the ability to toggle a series on or off by clicking it, and to toggle all series on or off in one click. Thankfully this was as simple as adding delegateEvents that loop through each series changing disabled to true or false before calling Graph.update. In the CoffeeScript code example below we are disabling each series but the one that’s first in the legend:

# A method that is triggered when the user selects "disable all"
# Unfortunately we can't disable every series, so the first one remains enabled
disableAllSeriesButFirst: () ->

    # Each Series in the legend
    $legendItems = @$legend().find('li').not('[data-legend-toggle]')

    # Visually enable the first series in the legend
    $firstLegend = $legendItems.first().removeClass('disabled')

    # Use the legends text to find the correct series to enable
    # in Rickshaws series array
    $firstLegendText = $firstLegend.find('.label').text()

    # Visually disable every other series in the legend
    $legendItems.not(':first').addClass('disabled')

    # Loop through each item in Rickshaws series array
    # Add disabled: true || disabled: false
    # depending on a match with the legend text above 
    for series, i in @rickshaw.graph.series
        series.disabled = ($firstLegendText isn't series.name)

    # Keep a count of the disabled series
    # to prevent too many from being disabled
    @disabledSeriesCount = @rickshaw.graph.series.length-1

 

We handled the synchronised tooltips in a similar way. Synchronised tooltips means that when a user hovers on a point a tooltip appears. A tooltip also appears for that point on every other graph revealing all its series information. This is handled via a simple mediator object which each graphs subscribes to. When the user hovers on a point, an event fires which passes the x date that was hovered. Each graph then loops through its own series data to construct its tooltip which is positioned using Rickshaw’s x method like so:

 leftPxPosition = @rickshaw.graph.x(hover.domainX)
@$hoverDetail().css('left', leftPxPosition).show()

Result

We’re quite proud of this round of graphing improvements and we think they have achieved the objectives outlined at the start of this article. The UI is intuitive, changing metrics or time-ranges is a breeze and most importantly the graphs give real insight which makes the debugging and development of devices easier.

But we are not done yet, we’ve got a lot planned. Horizon charts are a feature that sadly didn’t make it into this release, but they are useful because they are clearer than line charts when plotting many series:

Server Density Horizon Graphs

We’ll be working on horizon charts at some-point, but first we plan to allow you to plot any metric(s) from any device(s) on the same graph. From your feedback we know this will be a massive feature. One that will make it easy to gain insight into the relative performance of devices regardless of how your devices are configured.

We’ll also be making smaller improvements to the existing graphs and we’ll be speaking to customers over the coming months to see where else we can make improvements.

Articles you care about. Delivered.

Help us speak your language. What is your primary tech stack?

Maybe another time