The perils of auto updating
We regularly review the ideas posted in the feedback forum for our server monitoring service, Server Density, and last month we were internally discussing a request to allow auto updates for our Windows agent. Part of our service requires an agent to run on your servers to report data but we have never had an automated built in update mechanism, so we wanted to decide whether to implement this or not.
This is a discussion of the tradeoffs between always having the latest software version installed and the security implications of allowing remote code execution – the agent would have to be told to download code from our update site, then install and run it without user intervention. Our security architecture has always been based on the idea of one way communication – the agent only reports data to us and we never allow code execution to be triggered remotely. In our opinion, this is a major security issue because compromise of our whole service, or just a single account, could cause customer servers to be compromised – something we specifically designed the service to avoid.
There are good reasons for allowing updates, most of these revolve around security patches. This is true for consumer use cases – you want Google to update Chrome silently because the web is a risky environment (and it’s more convenient – Firefox updates were always a pain). And you want Microsoft to keep you up to date if you’re less technical and running Windows so they can avoid viruses. But there are risks, such as injection of malware.
However, the situation for business / server use cases are different from consumers:
- Automated updates can’t be controlled. Often you will want to deploy updates during a specific maintenance window so you can have staff on site, make sure vendor support is open and give customers appropriate notification.
- You need to have a rollback procedure so if an update fails or goes wrong, you can detect it and go back to the previous version. This is much more difficult to control if you don’t know when updates are happening.
- Some types of updates require integration testing e.g. updates to system libraries such as Python or language libraries and dependancies. This is less relevant to a monitoring agent but changes to the core agent might affect your plugins or customisations.
- Unless there is a specific reason to upgrade, you might elect to pin your version to a known stable release and never plan to update.
The way we package our agent for Linux makes all these possible without any further action on our part – we integrate into RPM and Deb package formats so our customers can build the agent into their normal OS deployment process. Windows doesn’t have such a well established system for this, especially for system services (ClickOnce is Microsoft’s recommended method but may not work with system services). RPM and Deb have signing built in, although we also sign our Windows packages.
In the end we decided to decline this request but add producing an MSI to our roadmap. This makes it easy to automate deployment and upgrades so although it still requires a manual download (vs RPM/Deb where we supply hosted repos) it can be managed by the Windows syadmins.
The feedback post remains open to comments so we will review this if someone has any further suggestions but in the end, sysadmins hate updates!
Enjoy this post? You may also like How to configure nginx as a load balancer