These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Information Portal

 
  • Topic is locked indefinitely.
123Next pageLast page
 

Dev Blog: Behind the scenes of a long EVE Online downtime!

First post First post
Author
Previous Topic Next Topic
CCP Falcon
#1 - 2015-08-07 14:49:14 UTC  |  Edited by: CCP Phantom
Sometimes in the world of Internet Spaceships, things don't always go according to plan.

More often than not it's because our wonderful players do something completely unexpected with content we add to the game, but occasionally there's hardware or software issues, or sometimes a combination of both, that result in extended downtime for Tranquility.

On July 25th, one of these instances occurred that resulted in the longest downtime we've had in almost half a decade...

The community requested a little more information about what exactly goes on behind the scenes when something like this happens, and of course for more technical information regarding what went wrong.

So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!"

What follows is a tale of server issues, hotfixes, lumberjacks, live console experiments and apparently, a dog that can code...

Take a look at this blog for further information...

Enjoy!

CCP Falcon || EVE Universe Community Manager || @CCP_Falcon

Happy Birthday To FAWLTY7! <3
Aladar Dangerface
Federal Navy Academy
Gallente Federation
#2 - 2015-08-07 14:55:09 UTC
CCP Falcon wrote:
Sometimes in the world of Internet Spaceships, things don't always go according to plan.

More often than not it's because our wonderful players do something completely unexpected with content we add to the game, but occasionally there's hardware or software issues, or sometimes a combination of both, that result in extended downtime for Tranquility.

On July 25th, one of these instances occurred that resulted in the longest downtime we've had in almost half a decade...

The community requested a little more information about what exactly goes on behind the scenes when something like this happens, and of course for more technical information regarding what went wrong.

So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!"

What follows is a tale of server issues, hotfixes, lumberjacks, live console experiments and apparently, a dog that can code...

Take a look at this blog for further information...

Enjoy!

Thanks for this, and the free SP \o/

I don't need twitter. I'm already following you.
Entity
X-Factor Industries
Synthetic Existence
#3 - 2015-08-07 15:11:20 UTC
CCP Falcon wrote:
So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!


you know sometimes... things... happen when you lock 2 guys into a room...

╦......║...╔╗.║.║.╔╗.╦║.╔╗╔╦╗╔╗

║.╔╗╔╗╔╣.╔╗╠..╠ ╠╗╠╝.║╠ ╠╝║║║╚╗

╩═╚╝║.╚╝.╚╝║..╚╝║║╚╝.╩╚╝╚╝║.║╚╝

Got Item?
Vic Vorlon
Malevelon Roe Industries
Convocation of Empyreans
#4 - 2015-08-07 15:16:04 UTC
Coincidentally, the same kind of thing happened at my work. Our logging system was running very slowly over a network, but nice and fast on our local PCs. It turned out that opening/writing/closing the log file was a lot slower than just keeping the log file open between writes. I'm sure that's a very different thing to what CCP is facing, but its nice to know we all face similar issues sometimes.
CCP Goliath
C C P
C C P Alliance
#5 - 2015-08-07 15:20:10 UTC
Joke's on them, I stopped watching Game of Thrones 3 eps into the last series.

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
Chainsaw Plankton
FaDoyToy
#6 - 2015-08-07 15:26:16 UTC
Sounds like the logs hit the bottle harder than a FanFest attendee

@ChainsawPlankto on twitter
CCP Goliath
C C P
C C P Alliance
#7 - 2015-08-07 15:29:29 UTC
Chainsaw Plankton wrote:
Sounds like the logs hit the bottle harder than a FanFest attendee


Yet still not as fast as me immediately after we opened TQ back up on the day.

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
CCP Masterplan
C C P
C C P Alliance
#8 - 2015-08-07 15:30:57 UTC
CCP Falcon wrote:
So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!"

I hope you realise what you've set in motion here...

"This one time, on patch day..."

@ccp_masterplan  |  Team Five-0: Rewriting the law
Hendrink Collie
Sebiestor Tribe
Minmatar Republic
#9 - 2015-08-07 15:34:34 UTC
CCP Masterplan wrote:
CCP Falcon wrote:
So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!"

I hope you realise what you've set in motion here...


So... who do we lock in the closet to get citadels done in two weeks? Big smile
flakeonium
The Cruciform
Keep Calm and Parley
#10 - 2015-08-07 15:40:28 UTC
oh the cliffhanger. Shocked
i need the conclusion !
Marc Callan
Center for Advanced Studies
Gallente Federation
#11 - 2015-08-07 15:44:34 UTC
Illuminating. But worryingly, I got the distinct impression that CCP figured out what was causing the problem but not why - unless the underlying cause of the logging issues has since been determined?

"We are what we pretend to be, so we must be careful about what we pretend to be." - Kurt Vonnegurt
drunklies
Federal Navy Academy
Gallente Federation
#12 - 2015-08-07 15:46:43 UTC
A blog full of words I technically understand, but have no real comprehension of what they mean.

10/10 tech blog Big smile
Oxide Ammar
#13 - 2015-08-07 15:54:00 UTC
We are starting to make Blogs about that ? What a sad year for EVE development this year Ugh

Lady Areola Fappington:  Solo PVP isn't dead!  You just need to make sure you have your booster, remote rep, cyno, and emergency Falcon alts logged in and ready before you do any solo PVPing.
Lothian Scykle
Nasty-Boyz
Templis CALSF
#14 - 2015-08-07 15:59:41 UTC
This is (to me) a fascinating and refreshingly honest explanation of the problem. Thanks for taking the time to put this together, and for respecting your players enough to give us a peek behind the curtain.

o7o7
Agent Unknown
School of Applied Knowledge
Caldari State
#15 - 2015-08-07 16:01:52 UTC
Ah, software development. It seems to easy to reproduce a problem, but finding the root cause is the fun (and sometimes infuriating) part.

So in the case, the logs, instead of showing nothing™, stabbed random nodes to death? Neat. Maybe it has to do with the *way* it's reporting the logs. I know nothing about the underlying code, but a difference between the generic_logger.warn() and campaign_logger.warn() routines could be something.

It makes me wonder if the Eve problems with Windows 10 in relation to its own logging (root cause was a permissions issue, go figure) has some parallels with the way the server handles things.
tasman devil
Puritans
#16 - 2015-08-07 16:02:40 UTC  |  Edited by: tasman devil
CCP Falcon wrote:
So, we locked CCP Goliath and CCP Masterplan in a room together, and told them "no more Game of Thrones until you write a blog!"

What follows is a tale of server issues, hotfixes, lumberjacks, live console experiments and apparently, a dog that can code...


I for one welcome our new CANINE overlord! :-)



PS: next time you lock two boys together, throw in a sausage and a knife Twisted

I don't belive in reincarnation I've never believed in it in my previous lives either...
Vincent Athena
Photosynth
#17 - 2015-08-07 16:03:07 UTC
I'm guessing this blog was somewhat delayed because you were hoping to find out why logging was being an issue before releasing the blog, so we would get the entire story. But, you still don't know?

Know a Frozen fan? Check this out

Frozen fanfiction
Chribba
Otherworld Enterprises
Otherworld Empire
#18 - 2015-08-07 16:04:55 UTC
The binary tale

★★★ Secure 3rd party service ★★★

Visit my in-game channel 'Holy Veldspar'

Twitter @ChribbaVeldspar
drunklies
Federal Navy Academy
Gallente Federation
#19 - 2015-08-07 16:05:04 UTC
Oxide Ammar wrote:
We are starting to make Blogs about that ? What a sad year for EVE development this year Ugh


A large portion of the community was asking for such a blog the minute the problem was resolved.

Insert line here [This did not take development time away from Eve]
GM Grave
Game Masters
C C P Alliance
#20 - 2015-08-07 16:05:57 UTC
The logs are dead, long live the logs!

Lead GM Grave | Senior Project Lead | CCP Customer Support | EVE Online | DUST 514
123Next pageLast page