[AC] Downtime extended

[Colleoni]

We have extended our ETA for the servers to come back up by an hour. We now expect VT, DT, HG, MT, and WE to be back up at 12:00PM PDT / 3:00PM EDT / 7:00PM GMT.

I can also now provide more details on what occured. There were two specific issues.

First, late last night we had a minor hardware failure on Verdantine. There was no noticeable affect on players but the back-end started having issues.

Second, a file on the database cluster that manages all five worlds was not truncating itself properly. The file was associated with Verdantine and caused the issues that occured around 5:00 AM EDT this morning, as well as other issues reported over the weekend. When we caught it, the file had grown to fill almost all available size on the disk.

Once we realized what had happened, we decided to shut down the other four worlds to ensure that there was no data loss. As the other worlds write data to the same disk, there was a chance that they could begin having problems as well. We felt that a few hours of downtime would be better than taking the risk of further issues and a possible rollback on four game worlds.

Currently we are in the process of replacing the failed hardware and allowing the file on the database server to process properly. It is taking longer than anticipated, thus the hour extension to the ETA.

At this time we do not expect there to be any rollbacks. We apologize for this unexpected downtime and thank you for your patience.

[Colleoni]

Just got an update from our Operations folk:

Due to the Verdantine issues, last night's backup did not run on the five worlds that are currently down. We are running the backup now, but it will take more time. So we are pushing the uptime ETA out to 2:00PM PDT / 5:00PM EDT / 9:00PM GMT.