Preventing false positives
Written by David Mytton
Any sysadmin who has been woken up in the middle of the night to fix some problem will know that whilst this is an inconvenience itself, it is even more irritating if the problem is actually a false alarm.
Over the last weekend, you may have received a number of false positive alerts from Server Density. These will likely have been “no data received” alerts that resolved themselves after 10-15 minutes. This was caused by an issue with our database, MongoDB, that caused the open file limit to be reached, combined with the fact that we were in the process of migrating our live database to the new hardware.
A temporary fix was implemented on the Sunday which should have resolved the issue for all users, and with the migration to our new super awesome database servers completed earlier today, a long term solution has now been deployed. We are very sorry for the inconvenience this caused
Aside from the technical issues, false positives can still occur but can be easily prevented if you configure your alerts in a specific way. One of the best options for this is the alert delay. This means that when an alert is triggered, you can tell it to wait a certain time period before notifying you. This is useful if you have a short load spike caused by a backup program that you know will last for just 30 seconds. You can still configure your alerts at your regular threshold but set a 2 minute delay so that only if the load remains above the trigger value for 2 minutes or more will you actually be notified.
Alerting configurability is a very important feature and one that we’re told is a key selling point of Server Density. It is also an area we’re working on improved for in future releases. But spending a few minutes to set the right options will ensure you only get notified when you need to so you can stay in bed that little bit longer!
As always, let us know if you have any questions.