These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

Asakai fight round up.
Author
Previous Topic Next Topic
Alizabeth Vea
Doomheim
#41 - 2013-01-29 10:56:13 UTC
Skorpynekomimi wrote:
It seems TiDi is working as intended. The fight, entirely unexpected, happened, and was finished. There was an outcome other than 'the server crashed and dropped all connections for hours'.

Very true. Kudos to CCP for keeping the server running.

Retainer of Lady Newelle and House Sarum.

"Those who step into the light shall be redeemed, the sins of their past cleansed, so that they may know salvation." -Empress Jamyl Sarum I

Virtue. Valor. Victory.
Lord Zim
Gallente Federation
#42 - 2013-01-29 11:04:40 UTC
IHaveCandyGetInTheVan69 wrote:
With no area of effect warp disruption I can't help but think these rules must pale compared to checking every ship on grid to see if it is in range of every bubble / HIC on grid?

I have no idea what you're trying to get at here.

Today, the server thread has to do all processing for the entire exterior of the solar system. This means that if I drop 200 supers and each super lauches 20 fighterbombers, that's 4200 objects which the server has to do all the processing for. Then add the 20 or so titans, the 300 carriers (which of course in turn launches 10 fighters, i.e. 3300 more objects which are flying around in space and must be calculated etc), all the subcap ships which pop drones, fires missiles and guns etc etc etc etc.

Now, compare that to a system where f.ex the solar system instance took care of things like structures (POSes, stations, market orders, warp disruption bubbles, gate guns, PI etc) and which players were in the system, and all the actual player-related processing (ship, position, direction, speed, HP, drones, fighters, fighter bombers, missiles, reps, neuts, etc) was done on player instances located on other nodes, and just the player data itself was updated on the solar system instance. This would mean that if f.ex I initiated warp, my player instance could ask the solar system instance for a list of bubbles and players (including state such as who they're pointing etc), and my player instance would do the calculation to check whether or not I was pointed or in a bubble, instead of the solar system instance. That would scale much better than today's system, but as I have said repeatedly, that would introduce potential syncronization issues

IHaveCandyGetInTheVan69 wrote:
Yes all solutions still have a limit, most that will still be reached, that doesn't mean we shouldn't strive for them, else we could have just said 300 player fights are big, if we do anything to improve it players will just find a new maximum so why bother? Also before you consider such an improvement a feat on CCP's part its still pretty much follows Moore's law, we should expect ~8X more players in a fight now, and even those are playing at a 1/10th of normal speed. Tidi is a bandage, reinforced nodes are a farce.

Of course tidi is a bandaid, same goes for node reinforcing. Fact of the matter is, they've worked, they let us have a fairly epic fight (even though I had to wait around 5+ minutes for anything to actually respond to any commands), and that's probably about the best we can expect out of the current architecture which CCP uses for its server shard. We can probably expect to see another few percent of optimization, but this is also just a bandaid which would lift the performance ceiling by a few percent.

And no, it's not "a farce".

IHaveCandyGetInTheVan69 wrote:
I'd hope CCP are still working on a solution that doesn't make 20minute fights take over 3 hours

I, too, would hope CCP are working on a solution which makes the server shard deal much better with large unexpected fights, but you shouldn't underestimate just how much work those few words entail. It's very easy to do what you're doing now and going "WAAAAAA CCP FIX IT IT SUCKS", or what I've done and spitball a possible structural change which might be the solution which would make everything scale up much better than it does today, the problem lies in actually executing the "fix it" or "redo processing to work in manner X".

IHaveCandyGetInTheVan69 wrote:
I'm pretty sure we won't see client-side processing on eve, for obvious reasons.

I never said "client-side processing", I said "per-player processing". I've no idea why you're talking about client-side processing for critical stuff like this, that'd be positively ********.

Cyno's lit, bridge is up, but one pilot won't be jumping home.

RIP Vile Rat
Buhhdust Princess
Mind Games.
#43 - 2013-01-29 11:08:26 UTC  |  Edited by: Buhhdust Princess
Vincent Pryce: Never Forget.

A Panda Eats Shoots and Leaves

We hope to of not left you disappointed.
Whitehound
#44 - 2013-01-29 11:12:35 UTC
Lord Zim wrote:
Bane Necran wrote:
But more on topic, it's mainly a hardware problem, not software.

Nope, it's a software problem.

The problem is that the finest granularity you can get on the cluster is one node for one solar system. This means that when there's 2500+ shooting, popping drones, navigating, deploying fighterbombers etc, that one node has to deal with all of that. This leads to one node having f.ex 50-100 systems, some hisec, some lowsec, and probably some nullsec/WHs too, because there's nothing really happening in them. This would mean that putting each of them on their own server (we're talking about ...what is it again, 5k systems?) would financially bankrupt CCP, and you would still not have gotten rid of the upper limit of how many people can be in a single system, all you would've done is raise that roof slightly.

The best thing we can probably hope for is that CCP were to redo the whole structure, so instead of having the processing structured around what goes on in a single solar system, the processing is structured around what one person does. That way they can distribute the load much more evenly across all the hardware, and they can develop usage-based heuristics to see which users habitually need a lot of processing power, and which just sit around in jita fapping their market orders.

The problem with this solution, however, is making sure all the required information is syncronized between all participants, and this is A Hard Task to solve. It's basically a can of worms, waiting to be opened. Anyone who's worked with multithreading in any way, shape or form knows why this is.

Short answer: No, because of locality.

The point of grouping things together into solar systems is because this is where most of the interaction happens. If you spread the players randomly out over all the nodes then most player interactions need to run over the local network. If you structure it into solar systems then player interactions are mostly kept within a node and therefore can be processed faster.

Loss is meaningful. Therefore is the loss of meaning likewise meaningful. It is the source of all trolling.
Lord Zim
Gallente Federation
#45 - 2013-01-29 11:50:30 UTC
Whitehound wrote:
Short answer: No, because of locality.

The point of grouping things together into solar systems is because this is where most of the interaction happens. If you spread the players randomly out over all the nodes then most player interactions need to run over the local network. If you structure it into solar systems then player interactions are mostly kept within a node and therefore can be processed faster.

No, that's not why it's structured the way it is, the reason it's structured the way it is is because it's a much easier (and less error-prone) than spreading it across multiple nodes. Last I checked, the EVE cluster is not a realtime cluster, it runs on 1 second ticks, and as such it doesn't care all that much about latency.

And while latency will be lower by keep all processing within the same node/instance, it still scales far, far worse than spreading the processing out over multiple nodes. The network latency is not a huge limiting factor in this picture, especially not when contemplating the increased scalability and resistance to surprise clusterfuck (or even scheduled clusterfuck), which is the main situation which the eve cluster still struggles with.

Cyno's lit, bridge is up, but one pilot won't be jumping home.

RIP Vile Rat
Whitehound
#46 - 2013-01-29 12:04:58 UTC  |  Edited by: Whitehound
Lord Zim wrote:
... the reason it's structured the way it is is because it's a much easier (and less error-prone) than spreading it across multiple nodes...

It is not just easier, it is faster, too. You are also wrong about the 1 second assumption. Only because the interactions of one player is ticked in intervals of one second will you still get many hundreds and thousands of interactions per second, because you have more than 2 players within a solar system.

Loss is meaningful. Therefore is the loss of meaning likewise meaningful. It is the source of all trolling.
Lord Zim
Gallente Federation
#47 - 2013-01-29 12:11:55 UTC  |  Edited by: Lord Zim
Whitehound wrote:
Lord Zim wrote:
... the reason it's structured the way it is is because it's a much easier (and less error-prone) than spreading it across multiple nodes...

It is not just easier, it is faster, too.

And while it is faster to do this on the same node, it is only faster to do this on the same node (and by node I also mean "on the same processor core/in the same process") as long as it can keep up. The instant it can't keep up (i.e. every time tidi kicks in), running it on the same node is slower, because you have access to less processing power than you would if you split it all up with higher granularity.

Whitehound wrote:
You are also wrong about the 1 second assumption.

No, I am not. The server ticks, i.e. the time when this network communication matters, happens on a 1 second interval. I can mash the "warp" button 100 times in 1 second, the server is still going to process just 1 warp command when it is time for the tick.

Whitehound wrote:
Only because the interactions of one player is ticked in intervals of one second will you still get many hundred and thousands of interactions per second, because you have more than 2 players within a solar system.

The commands are received and queued up/dropped as they're duplicates of previous commands, or blocked by previous commands until the calculations are done, which happens on a 1 second interval.

Cyno's lit, bridge is up, but one pilot won't be jumping home.

RIP Vile Rat
Whitehound
#48 - 2013-01-29 12:24:57 UTC  |  Edited by: Whitehound
Lord Zim wrote:
And while it is faster to do this on the same node, it is only faster to do this on the same node (and by node I also mean "on the same processor core/in the same process") as long as it can keep up. The instant it can't keep up (i.e. every time tidi kicks in), running it on the same node is slower, because you have access to less processing power than you would if you split it all up with higher granularity.

The moment a node cannot keep up will it still process many more transactions than the local network could ever carry. The CPU with its memory bus will always have the much higher bandwidth than the local network.

Lord Zim wrote:
No, I am not. The server ticks, i.e. the time when this network communication matters, happens on a 1 second interval. I can mash the "warp" button 100 times in 1 second, the server is still going to process just 1 warp command when it is time for the tick.

Yes, you were, but the point still is that a node has to process all the transactions between the players within a system. If you try to run these transactions over the local network then you are not only trying to squeeze these transactions through a lower bandwidth network, but you are also competing with the rest of the EVE universe, because all the transactions of 40,000-50,000 players would then need to be squeezed through the local network, too. It would be a complete disaster in the design.

Loss is meaningful. Therefore is the loss of meaning likewise meaningful. It is the source of all trolling.
Raiz Nhell
PeregrineXII
#49 - 2013-01-29 12:58:16 UTC
I think the first thing that should happen when a server hits TiDi is to start dropping useless processes, stuff that doesn't directly impact the fight.

I would like to see local only display a pilot list, no chat, no avatars... This stops people shitting up local to jam the server...
I would like to see eve mails stopped for those in the overloaded node, no mail bombing people's clients, the guy running the live feed was getting smashed with mails.
API access and some wallet transactions should be stopped until the node comes out of TiDi... Eve-Kill and bounties can wait till everyone is dead.

Unless of course the servers split the work, one hamster to deal with splurging chat and mail... 76 hamsters dealing with important stuff like who's missile hit who and did it do it before someone's else autocannon reduced it to slag...

I know our FC had us changing overview and bracket settings about 5 jumps out to try lessen the load on individual clients, dunno if it helped...

Even though I got maybe 15 volleys off in the whole time I loved it... Getting out was funny too, all the temp blue standings were getting a bit shaky :)

There is no such thing as a fair fight...

If your fighting fair you have automatically put yourself at a disadvantage.
sYnc Vir
Wolfsbrigade
Ghost Legion.
#50 - 2013-01-29 15:01:03 UTC
I think that given we RF'ed the pos on a whim, Forgot about it completely until the defense fleet was spotted coming in. Asakai turned out pretty damn good fun.

Never did take the pos thou, after the fighting was done people seemed too spent to go finish it off. Aye well.

Anyone got any links to what CCP said of the fight?

I'd be interested in seeing what Team Gridlock thought about it, given the server load and 2800 people in one system with another 1000ish next door..

Don't ask about Italics, just tilt your head.
Lord Zim
Gallente Federation
#51 - 2013-01-29 15:15:56 UTC
Whitehound wrote:
The moment a node cannot keep up will it still process many more transactions than the local network could ever carry. The CPU with its memory bus will always have the much higher bandwidth than the local network.

You seem to be thinking that there's a 1:1 relationship between the data bandwidth a CPU uses when doing tons of calculations, and the amount of data the CPU needs to start doing those calculations. This is incorrect.

Lord Zim wrote:
No, I am not. The server ticks, i.e. the time when this network communication matters, happens on a 1 second interval. I can mash the "warp" button 100 times in 1 second, the server is still going to process just 1 warp command when it is time for the tick.

Yes, you were[/quote]
No, I'm not. EVE's cluster isn't a realtime cluster, the fact it receives commands asyncronously doesn't mean it's executing them instantly, in fact they're queued up and executed when the processing tick starts.

Whitehound wrote:
but the point still is that a node has to process all the transactions between the players within a system. If you try to run these transactions over the local network then you are not only try to squeeze these transactions through a lower bandwidth, but you are also competing with the rest of the EVE universe, because all the transactions of 40,000-50,000 players would then need to be squeezed through the local network, too. It would be a complete disaster in the design.

I'd love to see your logic behind using 40-50k, since most of these 40-50k players wouldn't be interacting with eachother at all, through the wonders of a switched network, and most of them wouldn't require any data whatsoever.

But since you're critical of my way of thinking, what's your idea? How would you structure things to scale fleet fights up to, say, 10k players in the same system, with drones, fighters, fighterbombers, bombs, missiles, smartbombs, ECM bursts, the whole shebang?

Cyno's lit, bridge is up, but one pilot won't be jumping home.

RIP Vile Rat
Whitehound
#52 - 2013-01-29 16:54:48 UTC
Lord Zim wrote:
You seem to be thinking that there's a 1:1 relationship between the data bandwidth a CPU uses when doing tons of calculations, and the amount of data the CPU needs to start doing those calculations. This is incorrect.

CCP has shown in a dev blog that the computations make only a tiny fraction of it! Go find it and read it.

Quote:
No, I'm not. EVE's cluster isn't a realtime cluster, the fact it receives commands asyncronously doesn't mean it's executing them instantly, in fact they're queued up and executed when the processing tick starts.

Yes, you were and you still are wrong - or why do you say "No, I'm not" when you can say "No, I wasn't"?! ... Lol Check your pants for fire!

It then may not be a beauty of a real-time system and thus not find your approval, but EVE is a real-time system when it is required to respond in 1s intervals. Using PC hardware for this is then just fine. Only for things that require a finer granularity, like 1ms or 1us intervals, will you need a real-time OS. I am sure you'll understand this and can see a real-time system here.

Quote:
I'd love to see your logic behind using 40-50k, since most of these 40-50k players wouldn't be interacting with eachother at all, through the wonders of a switched network, and most of them wouldn't require any data whatsoever.

Yes, they would, only not in the way you think.

In your design, when two players are engaged in a 1v1, is there a high chance that each player sits on a different node and so everything they do will have to run across the local network.

Imagine what would happen when it is not a 1v1 fight but a fleet fight with 2000 players and several players firing smartbombs?! One smartbomb would cause hundreds if not thousands of interactions and with players spread across all nodes!

Or take a d-scan request. A d-scan would require transactions with every node to find out who is currently in the system and if they are within scanner range. Imagine the amounts of transactions that are required to travel across the network in such a design!

In short, your design will fall flat on its nose. Why? Because the game mechanics are build on top of the cluster's current architecture!

Quote:
But since you're critical of my way of thinking, what's your idea? How would you structure things to scale fleet fights up to, say, 10k players in the same system, with drones, fighters, fighterbombers, bombs, missiles, smartbombs, ECM bursts, the whole shebang?

I'd say, stop assuming, inform yourself on how the game currently works and trust in CCP to know what they are doing. They won't hire you any time soon to let you test a new design just because you don't understand their current one. So you might as well just study theirs.

Loss is meaningful. Therefore is the loss of meaning likewise meaningful. It is the source of all trolling.
Lord Zim
Gallente Federation
#53 - 2013-01-29 18:10:32 UTC
Whitehound wrote:
In your design, when two players are engaged in a 1v1, is there a high chance that each player sits on a different node and so everything they do will have to run across the local network.

Imagine what would happen when it is not a 1v1 fight but a fleet fight with 2000 players and several players firing smartbombs?! One smartbomb would cause hundreds if not thousands of interactions and with players spread across all nodes!

Or take a d-scan request. A d-scan would require transactions with every node to find out who is currently in the system and if they are within scanner range. Imagine the amounts of transactions that are required to travel across the network in such a design!

In short, your design will fall flat on its nose. Why? Because the game mechanics are build on top of the cluster's current architecture!

What you just described was not my design, that was your interpretation of my design, and it's not even remotely close to what I was thinking. I mean, really, "you have to query all the nodes to figure out who's in which solar system"? That's literally the worst idea I've heard all year.

What I said should be done is take the player processing, i.e. taking the ship position and calculating its new position given the direction it's travelling, what it's shooting, what its setup is like, which drones/fighters/fighterbombers/boms/smartbombs/etc it has running, do range, collision and damage calculations etc out of the solar system process and stick it into something which is more granular and can be spread across more nodes. Let the player instances query the solar system instance for the relevant data, do whatever processing it needs to do to update where missiles are, drones, whether to bump/get bumped etc and send the result back to the solar system instance for use at the next tick.

At no point have I said "make every player isolated objects on the cluster so the only way to figure out where anything is, is to search every node every second", and I'm not going to guess why you thought I would suggest anything even remotely close to this. Because holy **** that's a bad design.

Whitehound wrote:
I'd say, stop assuming

I should say the same to you, since you assumed the most boneheaded interpretation of my idea I've ever seen.

Whitehound wrote:
inform yourself on how the game currently works and trust in CCP to know what they are doing.

I've actually had a quick discussion with Veritas about splitting up the logic in the manner I'm thinking of, he said it was something he'd thought of testing himself, just never gotten around to it yet, and optimizing missiles, drones etc and various issues with switching solar systems took precedence.

Whitehound wrote:
They won't hire you any time soon

I'm not looking to get hired, I'm satisfied with what I'm working on as it is. But thanks for assuming.

Whitehound wrote:
to let you test a new design just because you don't understand their current one. So you might as well just study theirs.

I think I've a better idea of their design, than you have of mine.

Cyno's lit, bridge is up, but one pilot won't be jumping home.

RIP Vile Rat
Whitehound
#54 - 2013-01-29 18:27:13 UTC  |  Edited by: Whitehound
Lord Zim wrote:
I think I've a better idea of their design, than you have of mine.

Cry moar.

Lord Zim wrote:
The best thing we can probably hope for is that CCP were to redo the whole structure, so instead of having the processing structured around what goes on in a single solar system, the processing is structured around what one person does. That way they can distribute the load much more evenly across all the hardware, and they can develop usage-based heuristics to see which users habitually need a lot of processing power, and which just sit around in jita fapping their market orders.

There is no misunderstanding here.

Loss is meaningful. Therefore is the loss of meaning likewise meaningful. It is the source of all trolling.
Lord Zim
Gallente Federation
#55 - 2013-01-29 18:56:51 UTC
Whitehound wrote:
Lord Zim wrote:
I think I've a better idea of their design, than you have of mine.

Cry moar.

That wasn't crying, that was a statement which turned out to be entirely correct.

Whitehound wrote:
Lord Zim wrote:
The best thing we can probably hope for is that CCP were to redo the whole structure, so instead of having the processing structured around what goes on in a single solar system, the processing is structured around what one person does. That way they can distribute the load much more evenly across all the hardware, and they can develop usage-based heuristics to see which users habitually need a lot of processing power, and which just sit around in jita fapping their market orders.

There is no misunderstanding here.

Actually, yes, there is.

Apparently I could've explained things in more detail/clearer, but in my mind I was thinking of using a cut down version of today's solar system instance as a centralized data storage for player data, and make sure it takes care of grid creation etc, but move the processing of player position, ship fit changes, firing, missile/drone/fighter/fighter bomber/bombs/smartbomb activation etc into worker threads.

In short, instead of having the processing structured around what goes on in a single solar system, structure the processing around what one person does. If he fires a drone, his worker process fires it off and keeps track of it, and sends the location data back to the solar system instance. If he tries to warp, he gets the grid's content and does the collision damage checks etc. If he shoots, he gets the coordinate of his target and calculate the distance and damage output and reports back to the solar system instance. If he does a dscan, his process sifts through what data should be presented to the client.

Cyno's lit, bridge is up, but one pilot won't be jumping home.

RIP Vile Rat
Skorpynekomimi
#56 - 2013-01-29 18:57:13 UTC
sYnc Vir wrote:
I think that given we RF'ed the pos on a whim, Forgot about it completely until the defense fleet was spotted coming in. Asakai turned out pretty damn good fun.

Never did take the pos thou, after the fighting was done people seemed too spent to go finish it off. Aye well.

Anyone got any links to what CCP said of the fight?

I'd be interested in seeing what Team Gridlock thought about it, given the server load and 2800 people in one system with another 1000ish next door..


http://community.eveonline.com/devblog.asp?a=blog&nbid=74227
http://community.eveonline.com/devblog.asp?a=blog&nbid=74226

Still seems that TiDi works as intended.

Economic PVP
Whitehound
#57 - 2013-01-29 19:34:32 UTC
Lord Zim wrote:
In short, instead of having the processing structured around what goes on in a single solar system, structure the processing around what one person does.

Such an approach won't help when you want to exploit the full performance of an architecture, which is what you want to do here. Your approach only makes the maintenance of a software design model easier, because new software developers can more easily understand it when it centres around a player (even the game designers will understand it quickly), but there is no additional advantage here, at least not when you want to unleash the full hardware performance.

If you want to exploit a system's full performance then you need to shape the game world with its mechanics literally around the hardware architecture and in the best way possible. This includes making sacrifices to the game play as well as to the software design model. Only then will you be able to get the most out of it.

The price for an approach so close to the hardware is that your code will not be as easy to understand, its software design model will be complex and requires additional learning for every new software developer. Further, a game designer will not always get his wishes fulfilled, because these can interfere with the software model crippling the performance, but those wishes that make it will run optimally on the architecture.

You could argue that it is the job of the software developer to implement every wish of the game designer, but it is a give and take between the two. You could end up with either a super fast system, but boring game mechanics, or, with a slow but feature-rich game.

So I still say to trust CCP and their teams to know what they are doing and to let them figure out what the best compromise is.

Loss is meaningful. Therefore is the loss of meaning likewise meaningful. It is the source of all trolling.
Lord Zim
Gallente Federation
#58 - 2013-01-29 19:47:43 UTC
Which part of my idea "won't exploit the full performance of an architecture", and why?

Cyno's lit, bridge is up, but one pilot won't be jumping home.

RIP Vile Rat
Whitehound
#59 - 2013-01-29 20:06:42 UTC
Lord Zim wrote:
Which part of my idea "won't exploit the full performance of an architecture", and why?

Because you believe you can do a job single-handedly, which takes entire teams for CCP to do and are doing for 10 years now.

For example, do you seriously believe that nobody at CCP knows about multi-threading and that they are not constantly improving their game? You think they are waiting for you to tell them how it is done? No.

Go watch "A Beautiful Mind" and perhaps then you'll understand.

Loss is meaningful. Therefore is the loss of meaning likewise meaningful. It is the source of all trolling.
Lord Zim
Gallente Federation
#60 - 2013-01-29 20:08:18 UTC
So the objection to my idea is "it wasn't thought up by CCP, therefore it sucks"?

Cyno's lit, bridge is up, but one pilot won't be jumping home.

RIP Vile Rat