Building a color engine for graphing
Sonja Krause-Harder, Frontend Engineer at Server Density.
Published on the 4th April, 2017.
One of the basic features of a monitoring dashboard is to display metrics that came in from the monitored infrastructure. The bread-and-butter tools for this are simple line graphs, and in line graphs, every line needs to be assigned a colour. Ideally, these colours help the user interpret the data on the graph.
The old approach
The approach we used in the last few years was based on the principle that every metric, and therefore every series we display on a graph, has the same importance: a user has hand-picked these metrics when they configured the graph widget, so they are interested in every one of them. Our job is to make them equally visible. To ensure that, we used colours that are as easy to distinguish from each other as possible. Starting at the bottom of the chart, every metric line would get a colour assigned that is guaranteed to be visibly different from the one above. On a new chart, we’d start at the same first colour again.
This means that on most graphs, we used the same colours, and we didn’t have a discernable colour scheme because we cycled through lots of colours as graphs got bigger:
The first issue: colours jumping around
This approach had a side effect that some customers were understandably unhappy about: as new data is coming in and a chart rerenders, the order in which colours are requested may change as well, and a specific metric can be assigned a different colour on refresh.
Users could have a graph that shows the load average of two devices on one chart: the one with the higher load average at the end of the time frame would always get the first colour, and the one with the lower load average the second colour. When the load pattern changed on the devices the metrics swapped places, and they also swapped colours. The user could never mentally attach one colour to one device, but had to use the tooltip every time to find out which is which.
The second issue: same metric having different colours
Another usability issue became apparent when the same metric was represented on more than one graph: it would be assigned a colour depending on its position on each graph.
Almost every time, this meant that the same metric on different graph widgets had a different colour. The user might notice similarities in the shape of the line, but it would be much easier to notice that two charts are sharing a set of metrics when the lines that represent these metrics not only have the same shape, but also the same colour.
As an example, on these two graphs the series that is highlighted on the second graph is also present on the first one, but it has a different colour.
On large dashboards with lots of graph widgets these kinds of visual connections across graphs are hard to make, but in certain situations it would be nice to be able to recognize recurring metrics on different graphs by their colour.
Solution: a stable colour assignment
As a remedy for these problems, we started looking for a way to assign colours to metrics once and for all so that the colour:
- does not change when a graph chart reloads, and;
- is consistent when a certain metric appears on more than one chart.
The first, naive approach was to use the same colour generation algorithm we used per graph, and apply it to the whole account. Every metric that is shown anywhere in the UI on a dashboard would be assigned a colour when it is first rendered, and that colour would then be saved and reused whenever the metric is rendered again in the same or a different context.
This turned out to be impractical for the following reasons:
- We would need a way to save the colour mapping, somewhere.
- If we want the colours to be the same across users on the same account, this would mean a roundtrip to the API to save it in the account. If we didn’t do this, engineers might be shown different colours on a certain dashboard graph than the colleagues sitting next to them, or on the dashboard shown on the large TV in their office.
- Our old algorithm is capable of generating 2000 unique colours before repeating itself, but while these colours are numerically unique and clearly distinguishable from their direct neighbours in the order in which they are generated, there are still very many similar colours in that set. As we would no longer assign colours in a specific order per graph, but across the whole account in a more or less random order, distinguishability could no longer be guaranteed.
Mathematics to the rescue
So we started looking elsewhere. We have a unique string that describes every series that can appear on a graph: the inventory ID (which uniquely identifies the server in our database) concatenated with the name of the metric. What if there was a way to calculate the colour that is assigned to a certain series from this unique string? That way, we could calculate the colours on the fly and wouldn’t have to save anything, anywhere. This calculation would need to be fast, deterministic, and assign colours evenly across a given range.
It turns out that there is a mathematical procedure that does exactly that: hashing. Hashing takes values from a very large, even infinite input set, and puts them into a smaller, finite number of possible slots or so-called buckets. In our case, the infinitely many input values are the unique string identifying the inventory/metric combination. The buckets would be a colour, or for practical reasons, an integer pointing at a specific place in a given colour palette.
To illustrate the concept, a very simple hash calculation could look like this (in pseudocode):
sum = 0 NUMBER_OF_BUCKETS = 10 for every char in input string sum += ascii value of character bucket = sum % NUMBER_OF_BUCKETS
(The % is the modulo operator: it divides an integer by another and returns the remainder.)
The above function would assign the string “abc” the value (61 + 62 + 63) % 10 = 6 and the string “def” the value (64 + 65 + 66) % 10 = 5.
A good hash function for our purpose should not only map many input values onto a smaller set of buckets, but also take care that the distribution is as uniform as possible. This means, that no output values have a higher probability to be selected than others — or, thinking of colours, we want no colour to appear significantly more often than others.
It should also put input values that are close to each other into buckets that are further apart, so that series which have a very similar name don’t get very similar colours. All this means that the hash function we ended up using is a bit more complex than the simple example above, but the concept is the same.
This approach solves the issue of repeatability: the association between a series and its colour is arbitrary, but deterministic. No randomness is involved, the assigned colours are stable.
Real world behaviour
So, how does this approach behave when used with real data?
First, it solved our main problem: metric lines in graphs don’t change their colours any more when the graphs re-render.
Second, we found that the new colour distribution introduces more visual variation between graphs widgets. With the old colour approach every graph would use the same set of colours:
With the new approach, each graph will end up with its own distinctive set of colours, which are not perfectly balanced and gives each of them a specific tint. While this happens randomly and the colours don’t have any semantics assigned to them, it does make distinguishing between graphs visually somewhat easier:
On our own production dashboards the effect is even more visible:
However, giving up colour assignment per graph widget also has a downside: the hashing approach distributes colours evenly across all metrics in the account, but as we have fewer colours available than metric names (the output set is way smaller than the input set) it can very well happen that colours within a graph widget are really close together.
We tweaked the hash function parameters to minimize this effect, but it can and does happen that even graphs with just a few lines end up with them having similar colours:
The colours are assigned regardless of the context in which a particular series appears. Even if the hashing is deterministic, it is arbitrary, and looks random to a human, and sometimes it is less than perfect.
The original motivation for reassessing colours was to have a stable colour distribution on graph widgets. This is the most important aspect of this change for our customers, it works well, and we got positive feedback for this. A specific graph widget will always represent metrics by the same colours, no matter how often the page refreshes with new data, and you will always see the same colours as your colleagues on their screens.
That said, we’re not entirely happy ourselves with the amount of similar colours that appear in graph widgets, especially those which only show a small number of series. It it’s perfectly logical why, with the new approach, a certain graph might show only blue lines for three different metrics, but that doesn’t mean it is good, and we heard your feedback on that aspect too. We know we need to improve that and are already working on ways to solve the issue.