These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

CCP what's your plan for node crashing

First post
Author
Salpun
Global Telstar Federation Offices
Masters of Flying Objects
#121 - 2013-12-19 23:01:17 UTC
CCP Veritas wrote:
Red Teufel wrote:
I'm wondering what is CCPs long term plan to fix the node crashingQuestion

Long term plan is the same as the short term plan - diagnose the problem when it happens and fix it. Ideally twice; fixing the root cause and also hardening the system against similar faults.

The causes for the previous three node deaths have been patched on TQ, for example.

It's worth noting that in all recent events it hasn't been the high load that directly causes the node death, but rather errors in the programming that only express at high load or in the kind of highly complex scenarios that large fights contain.

Thanks for the info.

I know the normal rule is that bugs and when they get fixed are not told to the players but after the first of the last 3 node crashes a dev said that the root cause would be passed on to the players as it was a normal player action that caused the crash.

Hope the next large fleet fight can work its way thru with out crashingBig smile

If i dont know something about EVE. I check https://wiki.eveonline.com/en/wiki/ISK_The_Guide

See you around the universe.

Leigh Akiga
Kuhri Innovations
#122 - 2013-12-19 23:10:03 UTC
Salpun wrote:
Hope the next large fleet fight can work its way thru with out crashing


We shall see http://i.imgur.com/QkR20JQ.jpg
Onictus
Federal Navy Academy
Gallente Federation
#123 - 2013-12-19 23:13:27 UTC
Red Teufel wrote:
CCP Veritas wrote:
Red Teufel wrote:
I'm wondering what is CCPs long term plan to fix the node crashingQuestion

Long term plan is the same as the short term plan - diagnose the problem when it happens and fix it. Ideally twice; fixing the root cause and also hardening the system against similar faults.

The causes for the previous three node deaths have been patched on TQ, for example.

It's worth noting that in all recent events it hasn't been the high load that directly causes the node death, but rather errors in the programming that only express at high load or in the kind of highly complex scenarios that large fights contain.


so no super computer that uses WHs to process data more quickly darn :P



By definition the TQ cluster ...if not a full fledged super-computer.....is damn close.
Beekeeper Bob
Beekeepers Anonymous
#124 - 2013-12-20 00:06:19 UTC
CCP Veritas wrote:
Red Teufel wrote:
I'm wondering what is CCPs long term plan to fix the node crashingQuestion

Long term plan is the same as the short term plan - diagnose the problem when it happens and fix it. Ideally twice; fixing the root cause and also hardening the system against similar faults.

The causes for the previous three node deaths have been patched on TQ, for example.

It's worth noting that in all recent events it hasn't been the high load that directly causes the node death, but rather errors in the programming that only express at high load or in the kind of highly complex scenarios that large fights contain.


Thank you for finally responding to an important thread. We are glad to hear that you actually are trying to address it, however, considering node crashes have been a problem for as long as there have been large battles, you must understand our concern.

If this is in each case a programming error, then perhaps you need to start looking at bringing in some fresh eyes to look at the code, or figure out a way to stress test it virtually, instead of forcing your subscribers to continually beta test it for you.

Just saying, 10 years in beta is a bit excessive...Lol

And personally, I'd like to start seeing more Dev discussion on Tidi.
As a combatant in the current war, I can't say anyone is happy with the current solution. It's simply not acceptable to force your clientele to spend 10 hours online fighting a one hour battle.
The only reason I tolerate it now is for the opportunity to shoot Goons, that's really the only reason I come back and play this game anymore
It's really a pity too, this game has had so much potential, but the never ending bugs make it impossible for me to recommend this game to anyone.

Signature removed - CCP Eterne

Magna Mortem
The Scope
Gallente Federation
#125 - 2013-12-20 00:09:50 UTC  |  Edited by: Magna Mortem
Jill Chastot wrote:
Magna Mortem wrote:
I have a suggestion that will help you convince CCP that the matter needs more attention.

Flood highsec system with people and activity for a while.
No need for ganking or something and no need crashing nodes.

Just spread TiDi. You have the numbers to pull it off.

I'd love to see the effects of this.

Acta, non verba. Your whine threads are embarassing.
The whining is embarassing.

Stop whining and do something.


Also i don't think null blocs want their player base banned for intentional server disruption or whatever.
So lets leave the silly at the door eh?

Ban players for what? Spreading out in highsec and having fun shooting each other in an alliance wide tournament, with mandatory drone useage? Bull. Stress doesn't kill a node. Bugs do. CCP Veritas said so.
If anything, they'd have to THANK you for your support and your effort helping them making more stable code.

If by chance several highsec systems suffer TiDi... can't blame you for that either.
That's the game. Not your fault.


How big is the CFC again?
Leigh Akiga
Kuhri Innovations
#126 - 2013-12-20 00:10:48 UTC
Beekeeper Bob wrote:
As a combatant in the current war, I can't say anyone is happy with the current solution. It's simply not acceptable to force your clientele to spend 10 hours online fighting a one hour battle


Hey guy- when you bring Aeons as your logistics and Titans as your drone trigger, you might be contributing to the length of an engagement.

Just sayin'

Love you buddy
Vaerah Vahrokha
Vahrokh Consulting
#127 - 2013-12-20 00:42:09 UTC
James Amril-Kesh wrote:
Mashie Saldana wrote:
The long term plan is to have TiDi go down to 1% instead of 10%. No more node crashing.

Looking forward to waiting almost 7 minutes between drone volleys, almost 17 minutes to turn off a MWD, 1 hour 40 minutes waiting out aggression on a gate, 8 hours and 20 minutes for siege cycle, almost 17 hours between doomsdays, etc.

Forget node crashes, at that point downtime will be the interrupting factor.


Maybe that day you'll understand it's a kind way of CCP to tell you to stop blobbing like mad?
Alavaria Fera
GoonWaffe
#128 - 2013-12-20 00:45:32 UTC
Vaerah Vahrokha wrote:
James Amril-Kesh wrote:
Mashie Saldana wrote:
The long term plan is to have TiDi go down to 1% instead of 10%. No more node crashing.

Looking forward to waiting almost 7 minutes between drone volleys, almost 17 minutes to turn off a MWD, 1 hour 40 minutes waiting out aggression on a gate, 8 hours and 20 minutes for siege cycle, almost 17 hours between doomsdays, etc.

Forget node crashes, at that point downtime will be the interrupting factor.


Maybe that day you'll understand it's a kind way of CCP to tell you to stop blobbing like mad?

Don't worry, we can take it

let's do this

Triggered by: Wars of Sovless Agression, Bending the Knee, Twisting the Knife, Eating Sov Wheaties, Bombless Bombers, Fizzlesov, Interceptor Fleets, Running Away, GhostTime Vuln, Renters, Bombs, Bubbles ?

Il Feytid
State War Academy
Caldari State
#129 - 2013-12-20 00:46:53 UTC
You mean the node crashing is due to programming errors and not... Wait what's the current CFC narrative said about their enemies?
Vaerah Vahrokha
Vahrokh Consulting
#130 - 2013-12-20 00:50:14 UTC
Beekeeper Bob wrote:

Thank you for finally responding to an important thread. We are glad to hear that you actually are trying to address it, however, considering node crashes have been a problem for as long as there have been large battles, you must understand our concern.

If this is in each case a programming error, then perhaps you need to start looking at bringing in some fresh eyes to look at the code, or figure out a way to stress test it virtually, instead of forcing your subscribers to continually beta test it for you.


High load concurrency problems are possibly the worst kind of bugs to spot. They are irregular, rare, unpredictable, it takes a lot of sampling just to find out they happened, SiSi mass tests only find the easier ones that stick out more. Plus EvE is ever changing, so nothing stops some more of these errors to actually add to the old ones.
Celeste DeAgama
Doomheim
#131 - 2013-12-20 05:49:46 UTC
James Amril-Kesh wrote:
Soldarius wrote:
Or maybe CCP can simply ban the players involved when it can be confirmed that they were intentionally trying to crash the node.

What's next is we'll be seeing rumors about DBRB being banned from EVE.
And then he'll stream again and the forum trolls will be butthurt.


Goons are entertaining. Think ill cheer for them since most N3 peeps I've known have a large stick in the back. Cry less, shoot more.

Vcheap siggyV
"Post with you main! I disaggree with you and you are denying me the opportunity to hunt you down in game and gank you!"Twisted
Alavaria Fera
GoonWaffe
#132 - 2013-12-20 06:03:55 UTC
Celeste DeAgama wrote:
James Amril-Kesh wrote:
Soldarius wrote:
Or maybe CCP can simply ban the players involved when it can be confirmed that they were intentionally trying to crash the node.

What's next is we'll be seeing rumors about DBRB being banned from EVE.
And then he'll stream again and the forum trolls will be butthurt.


Goons are entertaining. Think ill cheer for them since most N3 peeps I've known have a large stick in the back. Cry less, shoot more.

Vcheap siggyV
"Post with you main! I disaggree with you and you are denying me the opportunity to hunt you down in game and gank you!"Twisted

Don't call Boat entertaining, it encourages him

Triggered by: Wars of Sovless Agression, Bending the Knee, Twisting the Knife, Eating Sov Wheaties, Bombless Bombers, Fizzlesov, Interceptor Fleets, Running Away, GhostTime Vuln, Renters, Bombs, Bubbles ?

Il Feytid
State War Academy
Caldari State
#133 - 2013-12-20 09:20:49 UTC
Alavaria Fera wrote:
Don't call Boat entertaining, it encourages him

I find it highly entertaining. Smile
Kuno Kenshao
Brack Region
#134 - 2013-12-20 09:31:21 UTC
Yep, I am sure it wasnt the 1000's of drones being launched and abandoned at a high rate by the slowcat fleet.

Onictus wrote:
Alavaria Fera wrote:
Boat did it

Smoking gun

Weapons of node destruction



Nothing to do with the 1800 nerds and eleventybillion drones on the field.....nope nada.

crononyx
Aliastra
Gallente Federation
#135 - 2013-12-20 09:50:52 UTC
Tyberius Franklin wrote:
James Amril-Kesh wrote:
CCP Veritas wrote:
The causes for the previous three node deaths have been patched on TQ, for example.

When did that happen?

There was a thread asking about a patch today, possibly then?

That is most likely not client-side fix.
Nami Kumamato
Perkone
Caldari State
#136 - 2013-12-20 09:56:03 UTC  |  Edited by: Nami Kumamato
Apparently the game evolved beyond the developer's control :)
Pretty sure when they created it they didn't have things like Asakai in mind.
However, to this point, it's still pretty unbelievable that a game can pit hundreds (and occasionally thousands)of players in a single coordinated battle.
The game is pretty darn amazing just for that - but now people are asking CCP to allow for even larger scale engagements to be possible - and frankly - I think the current gaming technology simply cannot handle it. Multi-threading would be an improvement but still would not solve the issue. As the CCP man said - the problem is not caused by the load itself but by the errors resulting/appearing in the code when it reaches high loads. Ti-Di addresses that specifically - it slows the flow in order to prevent data from overwriting and generating errors.
But seriously you're talking about 2000 entities fighting 2000 entities resulting in a total of 4000 running scripts (plus adding the neutrals like the system celestial and what not ) and some million or so user command inputs every second. Frankly there are few (commercial) machines at the moment that can handle that and not turn into molten silica.
The only viable solution is for CCP to either multi-thread the code and split the input over different processing channels or to acquire one industrial super-computer that would be used only for cases like Asakai.
However those things cost billions (plus change in maintenance).
Sadly it's a case of maximum available technology and not the unwillingness of the developer - you can't ask CCP to aquire Blue-Gene in order to allow battles in the magnitude of tens of thousands. It will simply not be cost efficient for them or you.

Some game developers have recently started looking into solutions for this and the prevalent idea is that the current client-server architecture is not viable anymore since the load has increased exponentially.
Many propose something similar to cap-chaining - allowing more and more of the node's functions to be equally shared by the clients (something in the lines of a bot-net so to speak), using your own PC's processing power to strengthen the node.

Fornicate The Constabulary !

Lukas Rox
Aideron Technologies
#137 - 2013-12-20 10:46:43 UTC
CCP Veritas wrote:
Red Teufel wrote:
I'm wondering what is CCPs long term plan to fix the node crashingQuestion

Long term plan is the same as the short term plan - diagnose the problem when it happens and fix it. Ideally twice; fixing the root cause and also hardening the system against similar faults.

The causes for the previous three node deaths have been patched on TQ, for example.

It's worth noting that in all recent events it hasn't been the high load that directly causes the node death, but rather errors in the programming that only express at high load or in the kind of highly complex scenarios that large fights contain.


I will be more than happy to participate in some mass tests on Sisi to help you find more bugs like that. Better crash sisi than a would-be Asakai on tranq imo ;-)

Proud developer of LMeve: Industry Contribution and Mass Production Tracker: https://github.com/roxlukas/lmeve | Blogging about EVE on http://pozniak.pl/wp/

SFM Hobb3s
Perkone
Caldari State
#138 - 2013-12-20 15:02:18 UTC
Kuno Kenshao wrote:
Yep, I am sure it wasnt the 1000's of drones being launched and abandoned at a high rate by the slowcat fleet.

Onictus wrote:
Alavaria Fera wrote:
Boat did it

Smoking gun

Weapons of node destruction



Nothing to do with the 1800 nerds and eleventybillion drones on the field.....nope nada.




I see about 8 KM's on our boards for Nulli carriers in the DY battle. In cargo, they have this many drones (not counting any already deployed/abandoned/killled):

1455
730
1758
234
851
397
57
385

Yep. In b4 'omg i keep getting too far from my dronez' and 'omg bombz'. Even if the fight didn't have any tidi there's no justifyable reason for this many drones. If you check km's from a few months ago there never used to be this many...maybe even 1/5 as much.
Il Feytid
State War Academy
Caldari State
#139 - 2013-12-20 17:05:40 UTC
SFM Hobb3s wrote:
Kuno Kenshao wrote:
Yep, I am sure it wasnt the 1000's of drones being launched and abandoned at a high rate by the slowcat fleet.

Onictus wrote:
Alavaria Fera wrote:
Boat did it

Smoking gun

Weapons of node destruction



Nothing to do with the 1800 nerds and eleventybillion drones on the field.....nope nada.




I see about 8 KM's on our boards for Nulli carriers in the DY battle. In cargo, they have this many drones (not counting any already deployed/abandoned/killled):

1455
730
1758
234
851
397
57
385

Yep. In b4 'omg i keep getting too far from my dronez' and 'omg bombz'. Even if the fight didn't have any tidi there's no justifyable reason for this many drones. If you check km's from a few months ago there never used to be this many...maybe even 1/5 as much.

I think you guys carry too much ammo as well. Why are you bringing so much antimatter? And what is with all those navy scorch light missiles?!
ElQuirko
University of Caille
Gallente Federation
#140 - 2013-12-20 17:15:33 UTC
Jill Chastot wrote:
Errors in programming eh?

Well that is quite interesting to know.

What do you expect? No code is flawless.

Dodixie > Hek