These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

Unscheduled Downtime Extension – July 15, 2015

First post First post First post
Author
Previous Topic Next Topic
Alpha Spinecone
Wildcard Aerospace
#6021 - 2015-07-15 23:03:19 UTC  |  Edited by: Alpha Spinecone
That's a point. Any chance we could be filled in on what happened? Not trying to be demanding, just honestly curious.
Timballisto
Prometheus Academy
#6022 - 2015-07-15 23:03:36 UTC
CCP Falcon wrote:

UPDATES:
22:40 UTC - TRANQUILITY IS NOW BACK ONLINE AND ACCEPTING CONNECTIONS!


Great work guys and thanks for the constant updates. Hope that everybody involved gets a well deserved break!
Igtoo
Sebiestor Tribe
Minmatar Republic
#6023 - 2015-07-15 23:03:49 UTC
To all the DEV's, a big Thank You!!!

I'm sure you've had a bloody hard day to get this back up and running with a couple thousand people looking over your shoulder the whole time tutting and shaking their heads. Get yourselves home, put your feet up and have a well deserved (beer, J, tug, whatever your into) you deserve it.
Ultim8Evil
Exit-Strategy
Minmatar Fleet Alliance
#6024 - 2015-07-15 23:04:17 UTC
Out of interest, what was it that broke TQ?

Serious answer appreciated... stupid answers expected (hamsters, fozzie did it, moar duct tape, yawn)

Follow me on Twitter for literally no good reason @TheUltim8Evil
Valentine King
Zarnfell
#6025 - 2015-07-15 23:05:28 UTC
Well done to all those who worked so hard at CCP to get things back up and running. To quote Churchill:

"Never was so much owed by so many to so few"

A lot of peeps appreciate the work - me included

Big smileBig smileBig smileBig smileBig smileBig smile
Green Gambit
Blue Republic
RvB - BLUE Republic
#6026 - 2015-07-15 23:05:43 UTC  |  Edited by: Green Gambit
Pyotr Veliky wrote:
If anyone else asked the same question, it appears that there has been no response. It is standard programming practice to maintain a stable configuration as a backup whenever you deploy new code. It is hard to imagine that they would not plan for this kind of contingency. The bugs in the new configuration are obviously difficult to troubleshoot and fix. Hence, it would seem to make sense to do a rollback to the last stable configuration.


Maybe the bug was actually introduced during the Aegis release yesterday, and it was a series of events that happened during game-play yesterday that left game-state in a way that the cluster can't start. Even if you rolled back today's patches, iot doesn't matter the state is such that the app doesn't run.

At this point your rollback plan becomes restoring the pre-Aegis backup to the DB servers (having seen the Fanfest presentations on the amount of data involved, that's several hours of database restore) then you have to roll out the pre-Aegis code to all the servers, test etc. Then you've got to deal with a playerbase that's pissed because not only has the game been unavailable for half a day, but an entire day's worth of actions has been lost.

So would you enact the rollback? Or would you commit at least _some_ time to trying to come up with a fix?
Moonlightmist
Dissident Aggressors
Mordus Angels
#6027 - 2015-07-15 23:05:57 UTC
Perfect :)
I am on my way home from work :D :D :D
Reverendia Zeddamax
Angry Dragons
Brothers of Tangra
#6028 - 2015-07-15 23:06:08 UTC
Thanks devs!

Now, seriously, where is that Pizza Fund where we can donate??
Riley Masterson
Mechanical Basterds
#6029 - 2015-07-15 23:06:11 UTC
Good work with getting everything back up and running.

Break for dinner earlier next time and get more magic pizza.
Mourn LeBlade
Jupiter Roughriders
#6030 - 2015-07-15 23:06:24 UTC
I once worked over 48 hours straight on a circuit cut-over. Nobody thanked me because it was my job, and I was paid well to do it. Failure was not an option. As a paid professional your paycheck should be its own reward. Unless it's volunteer work, there is no need to prostrate yourselves and say thanks to those fixing what shouldn't be broken. The world has certainly changed. Trophy Generation.

LTCOL LeBlade 177 Division Live Free or Die
Gilan Isana
Science and Trade Institute
Caldari State
#6031 - 2015-07-15 23:07:11 UTC  |  Edited by: Gilan Isana
Alpha Spinecone wrote:

That's a point. Any chance we could be filled in on what happened? Not trying to demanding, just honestly curious.


Dont you think after a solid 12 hour shift the why and wherefores and analysis can wait till they have all had some rest (and probably a few beers!).

Tomorrow will be time enough for what happened, let the guys get home, or the pub.

Oh and CCP - I await delivery of my TIII Pizza delivery drone!! :-p

Well done and thanks again guys.
Crowbiwan
N.A.G.A Corporation
Domain Research and Mining Inst. Logistics
#6032 - 2015-07-15 23:08:08 UTC
In before threads closed.

Seriously down, what, 12hrs'ish?
Pepper Solette
Doomheim
#6033 - 2015-07-15 23:08:26 UTC
Thank you very much CCP for all the hard work and effort getting it back up and running.

I'm sure it was much more frustrating for you than it was for us!

o/

** Miko Sunji:  "There is no better way to find out if you can swim, than swimming for your life."**
Balaur Venatores
Nordului
#6034 - 2015-07-15 23:08:34 UTC
CCP, Felicitări =)
Alpha Spinecone
Wildcard Aerospace
#6035 - 2015-07-15 23:08:56 UTC
Gilan Isana wrote:
Alpha Spinecone wrote:

That's a point. Any chance we could be filled in on what happened? Not trying to demanding, just honestly curious.


Dont you think after a solid 12 hour shift the why and wherefores and analysis can wait till they have all had some rest (and probably a few beers!).

Tomorrow will be time enough for what happened, let the guys get home, or the pub.


Yeah, that's fair enough I guess. Just bein' nosy.
CCP Falcon
#6036 - 2015-07-15 23:09:02 UTC
Crowbiwan wrote:
In before threads closed.

Seriously down, what, 12hrs'ish?


11 hours, 41 minutes by my watch. Hell of a day!

Big smile

CCP Falcon || EVE Universe Community Manager || @CCP_Falcon

Happy Birthday To FAWLTY7! <3
CCP Falcon
#6037 - 2015-07-15 23:09:51 UTC
Gilan Isana wrote:
Alpha Spinecone wrote:

That's a point. Any chance we could be filled in on what happened? Not trying to demanding, just honestly curious.


Dont you think after a solid 12 hour shift the why and wherefores and analysis can wait till they have all had some rest (and probably a few beers!).

Tomorrow will be time enough for what happened, let the guys get home, or the pub.


Our awesome Dev Team are already talking about putting something together for you guys Smile

CCP Falcon || EVE Universe Community Manager || @CCP_Falcon

Happy Birthday To FAWLTY7! <3
Riddler Ronuken
A Game of Memes
#6038 - 2015-07-15 23:12:26 UTC
Mourn LeBlade wrote:
I once worked over 48 hours straight on a circuit cut-over. Nobody thanked me because it was my job, and I was paid well to do it. Failure was not an option. As a paid professional your paycheck should be its own reward. Unless it's volunteer work, there is no need to prostrate yourselves and say thanks to those fixing what shouldn't be broken. The world has certainly changed. Trophy Generation.


I feel the same, but it must be because we're douchebags.
Zarek RedHill
State War Academy
Caldari State
#6039 - 2015-07-15 23:12:38 UTC
Sorry about the rough day, CCP folk. Try to get some rest. Thanks for your hard work and this great game!

If it helps relieve some tension -- my favorite coding-related xkcd - http://imgs.xkcd.com/comics/goto.png


chaosjj
Doomheim
#6040 - 2015-07-15 23:13:02 UTC
CCP Falcon wrote:
Crowbiwan wrote:
In before threads closed.

Seriously down, what, 12hrs'ish?


11 hours, 41 minutes by my watch. Hell of a day!

Big smile



thank alot for keeping us updated!, enjoy a few well earned beers and moar pizza. Cool