EVE Information Portal

And since it's taken a nontrivial amount of time to move relative to the 1HZ physics engine, meaning that the odds are very good that your half a second will cross a tick boundary, that means that every move must be followed by a resync with adjacent systems to get everyone back on the same page, right? If one node is off by a server tick, how do you handle that?

CCP Prism X, is there any relation between the "balanced universe" and the perceived unresponsiveness from this thread?

Vmotion at least is truly transparent to the underlying VM. You will see a "pause" but incoming network packets etc. are not dropped ,just queued while the machine is in motion afaik.

I noticed a typo in the 3rd to last image, detailing how the x/y split works to better facilitate the repeated splitting in half.
http://content.eveonline.com/www/newssystem/media/65499/1/wholePowerOfTwoSolution.jpg
In the blue text for the 1st split, 85/64 is not 75.3%. 64/85 is, however. ^^

About the 2 seconds: that's straight from the vendor. So while, in practice, it may not take more than half a second, you still need to design your cluster to be able to handle a 2 second move. Better yet, a 4 second move. If every client disconnects because a move took .7 instead of .5 seconds, you gained nothing.

And to the every solar system on its own VM: yes, that is rather easy to maintain - from the POV of the virtual infrastructure. But it means x30 more connections on the internal end of the session servers. It also means x30 more SQL sessions which probably can't be scaled down by x30. It also means a larger memory foorprint for the entire server (x30 more OS instances) and decreased cache efficiency. That's why I called it a workaround.

vMotion works nicely for applications where you can add redundancy via IP failover. For protocols with standing connections and high degree of time synchronization - let's just say it gets complicated fast.


The tick between different systems runs differently. It probably always has. While all TQ nodes will run with similar latencies against the same NTP to keep the cluster internal clocks sync'ed, I doubt CCP sync'ed the server tick. Unless they use a wallclock second to initialize the first tick after starting the process - which actually they might have, thinking about it.

But this is not really that important inside the cluster. There really only the wallclock has to be sync'ed so timestamps represent consistently the same to all involved. That can be handled, NTP solved that problem decades ago.

The move is much more likely to desync the tick-count between server and client dogma simulation. The clients would be some seconds ahead of the server.
Here the server could:
- skip forward to the clients, discarding input for the skipped ticks
- skip forward, (try to) apply the entire input queue to the next processed tick
- issue all clients to roll back to his tick, discarding input
- signaling the clients a higher TiDi level than the server actually runs at until the it has caught up again
In any case, the server would have to be notified by the infrastructure that it has been moved, since the eve clients are untrusted terminals and the server can not trust them even if every connected client agrees that the server-tick is off by the same offset.

Btw, I think it's awsome that we as players sit here talking about TQ's cluster architecture.

[Edit]
The most elegant solution would actually be to have the infrastructure send an "intent to move" message to the server. The server could then set TiDi to 100%, completely freezing the universe (similar to how it works at downtime), trigger some "Sol-Node is being moved" message on the client, signal "ready to move" back to the infrastructure. After the move, the the infrastructure would send a "move complete" message, and the server would lift the 100% TiDi and continue the game.

Can I just pop in and say having read all 9 pages of this thread I wish more threads were like this on the forums.

Constructive talking among a diverse group of some very and some not so very knowledgeable members, no one having tantrums, throwing teddies out of prams, nothing but reasoned arguments.

Some putting forward what if’s, others debating and showing why this would not be possible but leaving room for further debate in case they missed something.

Must be the spirit of Christmas or something, well done to all.

It does mean dropped packets though. /That/ is what I was saying.

I've submitted a bug report, referencing the dev blog, and outlining the scenario in which traffic control will reject a wormhole jump while the ship's mass is still counted toward the hole's mass limit (as if the jump was successfully made). I can't find a bug report number to list here.

tell me if i'm wrong, and if traffic control does not affect mass limit totals under any node/load condition.

still waiting for confirmation that failed wormhole jumps with traffic control messages count against the wormhole mass, but will be looked into. (meanwhile there will be support tickets, handled by uninformed customer service staff)

Christ, these changes are awful.
"We noticed that inter-node jumps are less expensive than intra-node jumps"
And then proceeds to put adjacent systems on the same node to increase intra-node jumps.