Database migration on 2009-07-15
Written by David Mytton
Update 2009-07-16 19:48 GMT: All historical data has now been imported, completing the first phase of the migration. If you spot any problems, please let us know.
Update 2009-07-16 18:15 GMT: The migration has been completed and Server Density is now running on MongoDB for storage of checks data. We have just started the import of old data from MySQL and expect that to complete in a few hours. During this time historical data will be unavailable. A final update will be posted once this has completed.
Update 2009-07-16 01:49 GMT: Unfortunately we were unable to complete the migration yesterday as planned. Although all development work is completed and tested, a dry-run of the migration revealed that MongoDB uses a lot more disk space with the existing MySQL data structure than we had expected. We have now reworked the data-store and so we will be able to complete a second dry run in the morning (UK time), final testing and hopefully do the final migration during the day. Another update will be posted once we have more information.
On Wednesday 15th July 2009, we will be migrating our backend database for check data from MySQL to MongoDB. Although MySQL has served us well to date, it is becoming increasingly difficult to scale and maintain.
Having tested a number of non-RDBMSs, MongoDB has been proven to be the most suitable for our needs. Indeed, in most cases we are seeing query speed increases, with some queries being reduced to executing in under 5ms, on 500k rows. This means page load times will decrease and the speed of API calls will improve significantly (which directly affects the upcoming iPhone application download speeds).
The migration will be in 2 stages. This first stage will be completed on Wednesday and will see all check data being moved to MongoDB. All remaining data (server lists, user profiles etc) will be migrated on another date.
The impact of this migration will be as follows:
- Historical data will be temporarily unavailable. The switch will occur immediately and new data points will be inserted the next time your server agent reports back (within 60 seconds). Older data will be imported on an account-by-account basis. With almost 80GB of check data, this may take a few hours. No data will be lost.
- If you have a low trigger threshold for your “no data” alerts, they may be triggered by the lack of data in the database due to the above point. Data will generally appear again within 60 seconds once your server agent reports back but you may wish to pause alerts to prevent false positives.
As a side note, you may have noticed that we have not yet activated our data retention policy for free accounts. This will come into effect on Wednesday with the switch to MongoDB as we will only import data for the last 1 month for free accounts (paid accounts are unaffected). If you wish to keep this historical data, now is a good time to upgrade your account!
If you have any questions, don’t hesitate to contact us. This post will be updated once the migration is completed.