Serverdown message von 3 Uhr Morgens

[Tirami-Su]

This world will be coming down at 06:30pm PDT for a Hotfix. Please logoff before that time. Thank you. - Admin"


Na toll... woolt grad meine loottour loslegen aber nööö...


bin mal gespannt obs nen reroll gibt.. De Silvas unassingt XP sind

232.664.879.964 XP

[Ziv]

Grund für den Hotfix:

First, let me apologize for not getting this message out sooner. Unfortunately I've been juggling too many balls today, and I dropped this one. I'll try not to let it happen again.

Second, as I type this (about 12:45 am, EST), all the worlds are down for a hotfix. The worlds should be back up fairly soon. At this time, we do not believe that a rollback will be neccesary.

So, what happened? Well, as you may have noticed, a couple days ago we began to get many reports of items disappearing out of players' inventories on Leafcull, sometimes right before their eyes. There is a very rare, very intermittent bug that can cause item loss this way; there are also a few ways for your client to get confused and temporarily "lose" items (although it will find them again if you log out and back in) -- but we quickly determines that this problem was different from both of those issues. We immediately began collecting data from players who had been struck with the bug, and after a grueling day of investigative coding, digging through thousands of lines of logs, and collating reports, we found what we believe to be the cause of the problem.

During this process, reports of disappearing inventory items also began to occur on Frostfell. When we figured out what was up, we realized that the other worlds were temporarily safe, but that players on Frostfell and Leafcull were going to keep losing items until we got a fix out. So we took down those worlds immediately. But we also knew that within a few days to a week (depending on the world), the problem would crop up on the other worlds as well. So we needed to get the fix out to all the worlds pronto.

As soon as the fix was done, it went into testing. We didn't think we'd be able to schedule a hotfix until Sunday morning (as Ibn said in his last update), but through the preserverance of MS Test and Operations, we were able to process the hotfix in record time, get it scheduled, and get it out to the worlds. And that's where we are now.

So, what was the problem? In short, some automatic code that handles assigning ID numbers to game objects (like backpacks and drudges) hit a special condition and thereafter failed to function correctly. This meant that new objects were sometimes being assigned existing IDs when they entered the world. For instance, a newly spawned drudge might have the same ID as, say, your backpack. This is obviously bad. Especially when that drudge is killed and the server sends it a message to delete itself, and your backpack intercepts that message by accident ... And your backpack deletes itself, and everything in it.

This sounds disasterous -- and it *is* bad -- but luckily the code to prevent this from happening again is actually fairly simple and quite safe. Of course, we'll be monitoring the worlds carefully after they come back up, and we'll keep you apprised of any further developments.

Thank you for your patience during this issue. And thank you also to the players who submitted clear and concise reports of the problem -- your reports helped us track down the issue more quickly.

[Ziv]

* Regarding folks working overnight Friday to try and resolve this problem. Yes, that's exactly what srand did, that's why we were able to post an update so early Saturday morning.

* Regarding further updates and the State of the Network update. Yes, we absolutely dropped the ball on this. We and MS agreed to update the SotN as soon as the servers started coming down for the hotfix, and we believed we had, but it got missed.

* Regarding the length of time that the servers were down: It was decided to do a full backup of the server data during the downtime, just to be on the safe side.

* Regarding a rollback: At this time we do not plan on rolling back any servers on account of this bug.

For further details on the exact nature of the bug, please see srand's post on this forum.

On behalf of Turbine and MS I'd like to apologize for the situation. It was a very nasty and surprising bug, and many people at both places worked long hours without sleep to get this resolved as quickly as possible. I thank you all for your patience with us over the last few days, and I assure you that we are working hard to make sure that this does not happen again.