Server Density architecture choices
Our server monitoring service, Server Density, is now over a year old. We’re continuing to release improvements but over the last 12 months, many of the decisions made at the start have come up in customer support requests – why isn’t this available? why does this work like that? This is always the same with any software product and the only way to change core elements is a complete rewrite, or major refactoring. Neither of these is necessary and a complete rewrite is almost always a bad idea. So what choices were made at the beginning?
The Server Density agents sent data to our servers. We never communicate with your servers – everything is one way. The first reason for this is security – it means a breach on our servers will not affect you servers.
Secondly, it means we do not need to maintain a larger server infrastructure because we only have to handle incoming data. We do not need servers all over the world to do availability checks nor do we need to queue requests and ensure every monitored server gets contacted within a specified time period (e.g. checking every minute).
Thirdly, it means we do not require any open ports other than standard HTTP (port 80, or 443 if you choose to use HTTPS). Almost every server has outbound access to the internet so this avoids firewall and network configuration issues. We can utilise a protocol that is well defined and supported, without needing any custom programming.
No auto update
If you install our agent manually then there is no automatic update. You must manually execute a command. This is for security so that we can’t automatically install software on your servers, so a breach on our servers would not affect you.
However, if you use our OS packages (yum/apt) then we use the built in updater to automatically keep the agent up to date. This will be as part of your normal OS package management update process so you should be aware when this happens, and you can disable it or block certain packages. But most importantly, all installs/updates have to be signed with our key pair. We keep this secure so you can be sure that when an update is applied, it is really from us.
HTTP by default, not HTTPS
Our agent communicates with us over HTTP by default, which is not encrypted. Although we provide our web UI URL using HTTPS by default, the agent does not. This follows on from the above discussion about open ports. It makes it very easy to install the agent first time, and you can easily change to use HTTPS by adding a character to the URL in the config.
Core metrics not plugin based
The original version of the agent had no support for plugins and included all the metrics in the agent code out of the box. There were only a few metrics too, which made the agent extremely simple to set up and use. If we were writing the agent again we would probably make it modular but the original prototype of the agent was written quickly to get the product out as a beta.
Agent key based server identification
The agent key uniquely identifies each server by default. This allows you to change the IP and hostname of a server without affecting the monitoring. However, we also include an auto copy mechanism so that if the hostname does change, we assume it is a new server and add it to your account using agent key and hostname combined to uniquely identify it. No server should have the same hostname and hostnames should never change in production.
Every choice is right at the beginning. Some might need to be adapted as customers use the product (like our agent key based identification), others would be changed if you wrote the code again (like our plugins system) and still others would never change. It’s interesting to be able to look back and see how things were different.
Enjoy this post? You may also like Multi data center redundancy – sysadmin considerations