Premium Hosted Website & Server Monitoring Tool.

(Sysadmin / Devops blog)

visit our website

Author Archives: Rob Elkin

About Rob Elkin

Rob is responsible for ensuring things get done but he spends a good portion of his time thinking about buying chips from the local chippy. This is often justified because he cycles into work.
  1. How we use the Server Density Ops Dashboard

    Leave a Comment

    This week, we announced the release of our Ops Dashboard, allowing users to create custom dashboards to visualise their devices and services.

    Naturally, we use Server Density daily to monitor and manage the machines and services that keep us going. As a result, we’ve been making extensive use of the dashboard ourselves to visualise and react to issues, and it’s quickly become one of our favourite features. This post will give you an overview of how we use it on a daily basis, because you might be interested in our set up.

    First, here’s our Ops Dashboard:

    Server Density Ops Dashboard

    We run this on a 42″ TV that is wall mounted in our office, so it is always available and can be seen at a glance. This allows people in the office to pick up on issues quickly, often before an alert is triggered, which helps us improve our response time.

    Service Checks

    The first row contains three service checks:

    dashboard-service-check-1

    dashboard-service-check-2

    dashboard-service-check-3

    These three checks cover our three biggest customer facing areas, the v2 login page for the account we use, the v1 API and the v1 login page for the account we have there. This gives us a good overview of speed degradation and will let us see quickly if any services appear to be down from a customer perspective.

    Alerts

    Next up is a row giving us an overview of issues that might be affecting us:

    dashboard-alerts-1

    dashboard-alerts-2

    dashboard-alerts-3

    This gives us a view of the open alerts on our account. We’ve included the overall figure, but this can fluctuate based on a number of different factors, and usually has a few alerts triggered to indicate non-critical things we should be looking at (e.g. we have alerts for when our disk space gets to 90% as well as 95%, 99%, to allow us to pick up on issues before they start affecting the service). We therefore also include the alert numbers from a number of critical groups that it is important for us to keep an eye on.

    The alerts widget is new this month, and allows you to view the number of open alerts on an account, group and device/service level.

    Our infrastructure runs on Softlayer, so we monitor their status using our cloud status widget, which gives us an overview of the Softlayer RSS feed. This widget supports the major cloud/hosting companies, including AWS, Rackspace, Joyent, Google App Engine, and Digital Ocean, to name just a few.

    Graphing

    Now we move on to a number of graphs, the first three rows of which help us to monitor processing queues and the load on the MongoDB clusters that run v2:

    Queues and mongo load

    This gives us a nice grouping of related metrics, as issues with our queues tend to cause or be caused by issues with MongoDB.

    Finally, we have a grouping of load and memory graphs for our 4 v2 clusters:

    dashboard-load-memory-graphs-1

    dashboard-load-memory-graphs-2

    Using a vertical layout (I’ve split the above screenshots apart for legibility) allows us to see trends across clusters over time, so viewing a spike in load on one cluster can be compared instantly to the load on other clusters to see any patterns that are emerging.

    All these graphs use the graph builder feature, so they are built from a number of different metrics across multiple devices and/or services, which makes them very powerful and information dense.

    So that’s it, now you know how we’re using our dashboard, has it inspired you to add a few new widgets to yours? We’d love to know how dashboards fit into your current workflow. If they haven’t made it in, you can always use our dashboards, they’re free for 15 days.

  2. How to build stunning custom iOS graphs on iPhone and iPad

    5 Comments

    Server monitoring graphs iPad

    A couple of weeks ago, we released a big update to our iOS server monitoring app app, adding graph support on both iPhone + iPad. It’s an update that we’ve been working on for a while, as it brings the app in line with the web UI, whilst making our graphs even better. Here, I’ll explore the graphing library we chose and why; some of our design / UI considerations; and how we’ve implemented everything.

    Graphing Libraries

    We looked at a number of iOS graphing libraries:

    If nothing else, this is a fantastic lesson in online marketing. Of those libraries, Core Plot and iOSPlot are open source, the other three libraries require a paid license. Shinobi Charts have developed the only website with more to it than a page of features or code examples, in our eyes it was instantly ahead. It extended that lead with promises of an easy setup and reasonable pricing compared to the other libraries available. Shinobi also has the best ongoing support (1 year of support included for the same base price as the others without); it is easily customisable; and has good documentation. Visit the last 3 sites for yourself and you’ll see why first impressions really do count for something!

    The average cost for any of the 3 paid libraries is around $999, so why not just go with something open source and therefore free? It was certainly considered, but we ultimately decided that the price tag for Shinobi was worth it to receive the customisation we wanted, the documentation we needed, and the support that we (might have) required. Our business isn’t writing graphing libraries, and whilst it would have been great to contribute to open source projects like Core Plot and iOSPlot, it would have cost us more in time to setup and implement them, than simply paying for Shinobi.

    Goals

    Our main aim was to create an amazing way to access and interact with data visually on the iPad. We also wanted to add iPhone support as well, but the focus was to be on the iPad version. There were a couple of reasons for this:

    • The iPad has the “wow” factor.
    • The iPad has more screen space to interact with.
    • The use case for the iPhone version is that of a quick reference, for when you are out and about and something is happening that you need alerting of, whereas the iPad is something you are more likely to sit down with, and use to dig into details. We see the iPhone version as reactionary for our use case.

    To quote one of the internal discussions we had during planning:

    iPhone

    Accessing the graphs on the iPhone is more likely to be an emergency thing or a quick check of the recent status of a specific server/metric. This is because the screen is considerably smaller and hence, less useful. You’d be unlikely to want to go back over months.

    • Receive an alert and want to check what led up to it being triggered. This would involve checking the graph over a short time period e.g. the last hour or even the last 30 minutes.
    • Want to quickly check the state of a server over the last few hours or maybe last day or so.
    • Emphasis on speed of accessing the data for metrics vs customisability for range/metric/etc.

    iPad

    The iPad is much more useful for viewing historical data because of the larger screen. Quick access to common time periods would be useful, but a full time selection is also needed.

    • In a meeting discussing capacity planning so want to see stats over the last few weeks or months to look at the trending.
    • In a meeting discussing an outage so you want to see the stats leading up to a specific incident.
    • Want to compare metrics against each other (on a metric level e.g. comparing CPU usage and Apache req/s on the same graph from 1 server AND on a multiple server level e.g. comparing CPU on Server A vs Server B).

    Design

    We settled on a design quite early on, allowing us to make good progress. Daniele worked on the design for the current iteration of the app (since we updated it to a native version), and so naturally he designed this update.

    One of the biggest features on the iOS version of the charts is the ability to view more than one chart at a time, an improvement we have planned for the web version of Server Density but haven’t finished yet. Now on iOS, you can drill down charts to just the metrics you want to see, and display them beside other metrics (e.g. compare your load average with your network traffic and disk IO over the same period of time). To enhance this ability to do comparisons, we added pan and zoom syncing to all the charts, so when you pan or zoom one chart, all the others follow suit, making the comparison of spikes incredibly powerful.

    iPhone

    As I mentioned, the iPhone version is for gaining quick access to information, and so we prioritised simplicity and a single track for this side of the app. A prime example of this is the way you can pick the dates to display graphs over. Unlike on the iPad, where you can choose any date and time, on the iPhone we only allow the choice of four static options, as shown below.

    iPhone Time Selection

    This allows users to quickly see what is happening without having to fiddle with small controls. We also provide some background feedback on the currently selected time range by modifying the time selector button.

    Different states for the date range selector

    iPad

    On the iPad, we wanted to provide the best possible graphing experience. We included the ability to quickly switch between devices, kept the graph selector beside the graphs, and included a full featured date selector analogous to the date selector in the web version. We used the Tapku library for the calendar, and enhanced it to allow custom images to be used, as detailed in this pull request.

    Selecting a date and time on the iPad

    Comparing metrics on the iPad

    Metrics Selection

    We needed a way for users to be able to select not only the main metrics that they wanted to see (Network Traffic, Physical Memory) but also to turn on and off the sub metrics (eth0, Memory Used, etc). To do this, we created a nested tableview control, which we released on github a few months ago. We think this is really powerful for users to drill down to exactly what they want to see, and that is enhanced by the pan and zooming sync.

    Developing with Shinobi Charts

    Shinobi charts is everything you would expect from a modern framework, with good documentation, well structured relationships and classes, customisable interface elements, and the datasource and delegate callbacks you would expect. Its delegate and datasource structure should make it very easy to use for anyone remotely versed in using standard UIKit frameworks, e.g. UITableViews.

    Data

    All the data we use to populate our chart interfaces comes directly from our API, so if you need to do something similar for any platform you should be able to get everything you need in the same way the iOS app does.

    Chart Class

    This is the class we use for all the charts that we show in the app. It extends the ShinobiChart class, and allows us to add in our license key in the constructor, and set up a number of defaults that all the charts will use. This class deals with creating a new crosshair and legend on the chart, as well as setting the theme and modifying the axis’ to suit our needs. Here is a list of some useful things to note on the ShinobiChart class that we used to customise our charts:

    • self.crosshair: An instance of SChartCrosshair, which itself contains a tooltip (SChartCrosshairTooltip : UIView)  and has a small circle target with lines extending to the axis. You can override the view property of the Crosshair to create your own “circle target”.
    • self.legend: An instance of SChartLegend, again this is a UIView backed object. Override - (void)drawLegend to do your own custom drawing, using self.chart.datasource to get information on titles and colours, then just draw labels onto the view.
    • ShinobiChart is a UIView backed object itself, so we used that to add a full screen button to the top right corner of each of the charts. That also means that to show your chart, you just need to do [self.view addSubview:chart] in your viewController.
    • self.xAxis and self.yAxis: self explanitory really, but these are the axis objects for the chart. You can set them to be any number of different axis types, we use SChartNumberAxis and SChartDateTimeAxis. We also use the axisPosition property set to SChartAxisPositionReverse for the y axis. We also make the background a clear colour, and set some other view properties to make it look the way we want.
    • self.canvas: If you look closely enough at our charts, you should be able to see that there is a gradient running from top to bottom, so the bottom part of the chart is more opaque. We achieved this by adding a UIImageView as a subview to self.canvas, and sending it to the back. We also added an
      autoresizingMask to that image view, UIViewAutoresizingFlexibleWidth | UIViewAutoresizingFlexibleHeight, which allows it to scale and cover the whole chart.
    • self.title: We set this to the title of the metric group for the chart (e.g. Network, IO Stats). We could then pull the value out of it when creating the legends, giving us a clean way of passing that title around. Related to this is self.titleLabel, which we hid since we were creating the title in the legend ourselves.

    Delegate methods

    We use a controller for the interface you see when interacting with the charts, and each of the charts gets added to a UIScrollView that is part of that controller. When we create the chart, we set the delegate to be the controller that controls the view. This allows us to receive all of the important movement and change notifications, and take appropriate action. This controller is also the delegate for the scrollview, and for a pinch gesture recogniser we have added to the scrollview.

    Breaking down some of the tricks we used:

    • - (BOOL)gestureRecognizer:(UIGestureRecognizer *)gestureRecognizer shouldRecognizeSimultaneouslyWithGestureRecognizer:(UIGestureRecognizer *)otherGestureRecognizer – we return YES here to make sure that the pinch will be recognised. There is a lot going on in this scrollview, and our gesture recogniser wasn’t playing nicely with the pan recogniser that is on the chart, so returning YES here meant that both recognisers get the message. This doesn’t become detrimental however, because one or the other will end up discarding the event because it doesn’t suit (a pan will be discarded by the pinch recognisers, and vice versa).
    • - (void)sChartDidStartZooming:(ShinobiChart *)chart - We actually started using this delegate method from the ShinobiChart library, along with a number of other associated methods, but eventually decided on using a pinch recogniser on the entire view so that even if you weren’t focused directly on a chart, you would still get zooming.
    • - (void)sChartDidStartPanning:(ShinobiChart *)chart/- (void)sChartIsZooming:(ShinobiChart *)chart etc : There are a number of delegate methods for panning and zooming on the charts, and we used them to prevent scrolling on the scrollview when panning and zooming (see the next point)
    • - (void)scrollViewWillBeginDragging:(UIScrollView *)scrollView/ - (void)scrollViewDidEndDragging:(UIScrollView *)scrollView willDecelerate:(BOOL)decelerate : All our charts are in a scrollview, but they can also pan and zoom. That makes for some interesting interactions if left unchecked. We were finding that when you scrolled and also panned, you could move the charts, so we co-opted these methods to disable/enable panning and zooming on the charts while a scroll is happening.
    • - (void)sChart:(ShinobiChart *)chart crosshairMovedToXValue:(id)x andYValue:(id)y : Again, very useful for us to disable scrolling of the scrollview when the crosshair moves. We actually only disable scrolling for .3 seconds here, and then enable it again. This method will be called regularly while the user have a finger down to show the crosshair, so if we enable it again and their finger is still down then it will just be disabled. However, if the user takes their finger up, then we need to allow them to scroll again. There is no delegate method to let you know that a user has finished with the crosshair, and it is very possible that the crosshair could still be there, so we had to use this as a workaround.

    Datasource methods

    We had an existing app, which had a number of different data models for the different types of things we display (e.g. a Device model, Service model, Alert model). We decided that these models were best positioned to provide the data for the charts, so we implemented the SChartDatasource protocol and got to work. All the models extend off of a base called SDItem, so once we had added support to it, we got charting support for both Devices and Services. There are 4 basic delegate methods that should be implemented to get a chart up and running:

    • - (int)numberOfSeriesInSChart:(Chart *)chart
    • - (SChartSeries *)sChart:(Chart *)chart seriesAtIndex:(int)index
    • -(id<SChartData>)sChart:(Chart *)chart dataPointAtIndex:(int)dataIndex forSeriesAtIndex:(int)seriesIndex
    • - (int)sChart:(Chart *)chart numberOfDataPointsForSeriesAtIndex:(int)seriesIndex

    We implemented these methods, which then go back to a datasource singleton that is shared between all devices and services, to get the data for the chart. They are mostly self explanitory, however one thing to point out. seriesAtIndex: returns an SChartSeries, which you can customise by setting the style.lineColor and style.areaColor properties to get different colours of lines on your chart. We created a graph line colours object which would give us a concrete colour based on the sub-item key in the array, meaning that colours would always be the same for a given key, avoiding a colour change when a metric is added/removed.

    Using a custom pinch gesture recogniser

    ShinobiChart has a built in pinch recogniser, which will take care of all the zooming for you, and if you can get away with it, I would absolutely encourage you to use it. If however, you have the complex interactions we have, and/or want fine grained control, you can follow what we have done. It isn’t exactly the same as the Shinobi implementation, but it is very close and gave us the flexibility we wanted to zoom charts from anywhere in the scrollview.

    We start off by adding the gesture recogniser to the scrollview:

    UIPinchGestureRecognizer *pinchRecogniser = [[UIPinchGestureRecognizer alloc] initWithTarget:self action:@selector(pinchHappened:)];
    [pinchRecogniser setDelegate:self];
    [self.graphScrollView addGestureRecognizer:pinchRecogniser];
    

    Then we take the callback we get from the pinch, and use the “scale” property:

    - (void) pinchHappened: (UIPinchGestureRecognizer *)sender
    {
        if(sender.state == UIGestureRecognizerStateChanged)
        {
            [self zoomAllChartsToLevel:sender.scale];
        }
    }
    

    Then, take each charts current zoom level and multiply it by the scale:

    - (void)zoomAllChartsToLevel: (double)level
    {
        Chart *tempChart = nil;
        for (NSString *aKey in charts)
        {
            tempChart = [charts objectForKey:aKey];
            if([tempChart isKindOfClass:[Chart class]])
            {
                [tempChart.xAxis setZoom:(tempChart.xAxis.zoom * level) fromPosition:nil withAnimation:YES];
            }
        }
    }
    

    Limitations of Shinobi

    There isn’t much wrong with the Shinobi library, but here are a few things that irked me when I was working with it. These are things I’ve reported as bugs, and I believe are scheduled to be fixed.

    • Inability to detach axis from charts. You can move an axis around a bit, but it is never really independent of the chart it is attached to, so you can’t take it and add it to a view somewhere else. We had to use a bit of a workaround to get ours looking like it had independent axis’, but I’d rather it just be a UIView backed object that I could add to different views
    • Momentum zooming doesn’t call delegate methods during momentum. This is the sole reason we don’t have double tap to zoom enabled on our charts right now. If you have double tap to zoom enabled, the sChartDidFinishZooming: method will get called as soon as the double tap happens, but then the chart will continue to zoom a little more, which we don’t pick up. This is also the reason that momentum panning is turned off.

    tl;dr

    If you want to do charting on iOS, and can afford the cost ($999), Shinobi Charts is the way to go.