These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Information Portal

 
  • Topic is locked indefinitely.
 

Dev Blog: Behind the scenes of a long EVE Online downtime!

First post First post
Author
Previous Topic Next Topic
Dersen Lowery
The Scope
#61 - 2015-08-08 19:21:20 UTC
KenFlorian wrote:
Marc Callan wrote:
Illuminating. But worryingly, I got the distinct impression that CCP figured out what was causing the problem but not why - unless the underlying cause of the logging issues has since been determined?



As a former software developer/IT guy this happens more often than most of us choose to publicly acknowledge. Hat off to CCP for telling us what happened as best they could sort it out. They, more than anybody, would like a perfectly coherent explanation...some of the time it's impossible.


Also, it's a question of priorities. If you're running a service, priority #1 is restoring the service. Once you've done that, you have the luxury of time to figure out the why of it.

Proud founder and member of the Belligerent Desirables.

I voted in CSM X!
GeeShizzle MacCloud
#62 - 2015-08-09 01:08:51 UTC
so TL;DR Tranq is more unique than any of us (players and devs alike) ever thought...
Waffelweib
Hostile Erection
#63 - 2015-08-09 11:21:36 UTC
I loved that Devblog, thx a lot!
It just left me wondering... what are these little white tables on the pizzas?
KenFlorian
Jednota Inc
#64 - 2015-08-09 12:10:27 UTC  |  Edited by: KenFlorian
Waffelweib wrote:
I loved that Devblog, thx a lot!
It just left me wondering... what are these little white tables on the pizzas?



the "tables" keep the top of the box from crushing the pizza...think of it as shield extender.

Note: a single pizza box won't require a shield extender (it's alone in system). When stacked, however, there is a "weight penalty" which requires the support of a shield extender "white table".
KenFlorian
Jednota Inc
#65 - 2015-08-09 13:10:02 UTC  |  Edited by: KenFlorian
KenFlorian wrote:
Waffelweib wrote:
I loved that Devblog, thx a lot!
It just left me wondering... what are these little white tables on the pizzas?


the "tables" keep the top of the box from crushing the pizza...think of it as shield extender.

Note: a single pizza box won't require a shield extender (it's alone in the pizzaverse). When stacked, however, there is a "weight penalty" which requires the support of a shield extender "white table".
unimatrix0030
University of Caille
Gallente Federation
#66 - 2015-08-09 18:13:47 UTC  |  Edited by: unimatrix0030
Thanks for the devblog!
I love to read more of those!
Thanks for all the work you guys put in that day to get the game back up and running!

No local in null sec would fix everything!
Elegbara
White Wolf Enterprises
Harmonious Ascent
#67 - 2015-08-09 18:28:31 UTC
Code formatting broken? Looks like there is some actual python code in the blog, but I only see numbered lines and comments.

Also - "Perforce sync hangs causing delays in our build system" - I thought I am the only one having problems with Perforce.

Open your eyes. And awaken.
Lord Battlestar
CALIMA COLLABORATIVE
Atrox Urbanis Respublique Abundatia
#68 - 2015-08-10 03:42:06 UTC
Very interesting read, thanks for the info!

I once podded myself by blowing a huge fart.
Corraidhin Farsaidh
Federal Navy Academy
Gallente Federation
#69 - 2015-08-10 09:17:25 UTC
It's always interesting to see others peoples pain in these situations, mainly because misery loves company :D

I'd suggest replacing the databases with Oracle and hiring me to build them for you...and nuke the M$ stuff from orbit, it's the only way to be sure...
CCP Goliath
C C P
C C P Alliance
#70 - 2015-08-10 10:01:39 UTC
Dradis Aulmais wrote:
Sounds like Ghost in the machine.

TQ is a very unique system. 12 years old, reborn several times. Code here code there, its own little ecosystem. Its like the ultimate Capsleer.


I kind of love this analogy Big smile

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
Corraidhin Farsaidh
Federal Navy Academy
Gallente Federation
#71 - 2015-08-10 13:01:50 UTC
CCP Goliath wrote:
Dradis Aulmais wrote:
Sounds like Ghost in the machine.

TQ is a very unique system. 12 years old, reborn several times. Code here code there, its own little ecosystem. Its like the ultimate Capsleer.


I kind of love this analogy Big smile


I prefer my analogy of M$ software being a xenomorph that clamps itself to your face before infesting you with a parasite that eats you from the inside.

Then again my view may be somewhat tainted by having bought a laptop with Windows 8 pre-installed a few years back...someone pass me an m41a pulse rifle...
Estella Osoka
Cranky Bitches Who PMS
#72 - 2015-08-10 19:54:25 UTC
If you can't fix it with a hammer, time to get a bigger hammer.
Pestilen Ratte
The Scope
Gallente Federation
#73 - 2015-08-11 12:44:04 UTC
Vincent Athena wrote:
Legacy code?
I'll make a prediction. The channel you used for campaign logging was used in the past for doing something else. You thought that code was removed, but some part of it still remains. When you started campaign logging, some old code woke up, tried to do something related to that channel, and "bad things" resulted.


I agree with this reasoning, except that the nodes didn't fail until the python code went into a sustained loop. If the code in the log changed a binary setting for a channel, the single instance would also have caused the crash. The crash seems to be something to do with the loop.

Alternatively, there could very well be an if statement in the legacy code that triggers when any communication happens on this log channel. Even if that legacy code doesn't break the node when called once, it could disrupt the run loop by leaking memory when called over and over again. Then the node crashes.

Whatever it is, something in your run loop is freaking out when that channel gets used, so I would want to know why that channel was used in the past, meaning for what areas of functionality, and start looking in those areas of code for if channel statements.
KenFlorian
Jednota Inc
#74 - 2015-08-11 21:49:22 UTC
Pestilen Ratte wrote:


Whatever it is, something in your run loop is freaking out when that channel gets used, so I would want to know why that channel was used in the past, meaning for what areas of functionality, and start looking in those areas of code for if channel statements.



Which is why, when I was coding and leading coding teams, extra beer and pizza were the just rewards for in-line comments that explained WHY a given segment of code was written the WAY it was written.

Interpreting what a given line of code is doing is, usually, straightforward. Understanding why it was done this way and not that way or explaining why this will cause something later to perform in a given way....Priceless.

I loved documentation....in-line in the source code. External documentation? Usually worthless.
CCP DeNormalized
C C P
C C P Alliance
#75 - 2015-08-11 22:18:37 UTC
Corraidhin Farsaidh wrote:
It's always interesting to see others peoples pain in these situations, mainly because misery loves company :D

I'd suggest replacing the databases with Oracle and hiring me to build them for you...and nuke the M$ stuff from orbit, it's the only way to be sure...


hmmm, I'd rather we not go that route, thanks :)

CCP DeNormalized - Database Administrator
Corraidhin Farsaidh
Federal Navy Academy
Gallente Federation
#76 - 2015-08-11 22:51:06 UTC
CCP DeNormalized wrote:
Corraidhin Farsaidh wrote:
It's always interesting to see others peoples pain in these situations, mainly because misery loves company :D

I'd suggest replacing the databases with Oracle and hiring me to build them for you...and nuke the M$ stuff from orbit, it's the only way to be sure...


hmmm, I'd rather we not go that route, thanks :)


I'm sensing a vested interest...
KenFlorian
Jednota Inc
#77 - 2015-08-11 23:22:55 UTC
Corraidhin Farsaidh wrote:
I'm sensing a vested interest...



It's always good to know with what (or whom) one's SQL's are joined...outer but most especially the often problematic inner.
Joia Crenca
Science and Trade Institute
Caldari State
#78 - 2015-08-12 05:55:37 UTC
Looked tasty: Pizza!

I'll have to re-read the rest NOT at midnight after a full day P
KenFlorian
Jednota Inc
#79 - 2015-08-12 20:44:08 UTC
CCP DeNormalized wrote:


hmmm, I'd rather we not go that route, thanks :)



What could possibly go wrong?

Ah! I know what you are. You're a carebear, aren't you? (Says the 6 month old capsuleer with nary a kill to his name).

I think I see Carebear Control Corporation on their way and, generally speaking, they know where you live.
CCP DeNormalized
C C P
C C P Alliance
#80 - 2015-08-14 15:51:30 UTC
Actually, Oracle RAC is super sexy stuff and I'd love to get some experience with it!

it kicks the pants off SQL Server's offerings in this regard (although I've never used it and perhaps it's junk - but it sounds awesome!)

CCP DeNormalized - Database Administrator