These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Information Portal

 
  • Topic is locked indefinitely.
 

Dev blog: Tranquility Tech III

First post First post
Author
Previous Topic Next Topic
Puer Servus
Republic Military School
Minmatar Republic
#61 - 2015-10-14 05:50:25 UTC
Cor'len wrote:
Bienator II wrote:
Thats prob why ccp seems to see MT as low priority atm.


Actually, CCP would love to multithread the ~space code~ (can't remember the component name, haha). But it's practically impossible to get a consistent result; operations must be done in sequence, otherwise you get dead ships killing living ships, and other ~exciting~ edge cases.

This is the ultimate limiter on EVE performance. They might conceivably be able to MT the processing of different grids in a single system, but everything that happens on a single grid must execute in a deterministic fashion, and in the correct order.

Plus, even if that wasn't a problem, they run Stackless Python, with the beloved global interpreter lock which effectively prevents multithreading.


tl;dr CCP wants to multithread all the things, but it's so hard it's bordering on impossible. Hence the effort to not have big fights.


Actually there is a ~space code~ multithread project called The Destiny Dispatcher.

http://www.youtube.com/watch?v=NEJbwZCgNgU&t=1h16m00s
http://www.youtube.com/watch?v=UcEUB6h4Br0&t=8m10s

Can devs give any update on this project?
Corraidhin Farsaidh
Federal Navy Academy
Gallente Federation
#62 - 2015-10-14 07:58:45 UTC  |  Edited by: Corraidhin Farsaidh
The only problem I can see is with the database...should have been Oracle RAC cluster with ASM and Dataguard Active-Active failover :D

Ed; Admittedly the Oracle licensing structure makes me think it was designed by the Mittani himself (or one of his little minions) but everything has it's drawbacks...
CCP DeNormalized
C C P
C C P Alliance
#63 - 2015-10-14 09:43:49 UTC
Gospadin wrote:
xrev wrote:
Gospadin wrote:
I'm shocked that a system designed to deploy in 2016 is even using rotating drives. That data must be REALLY cold.

It's called auto-tiering. The hot storage blocks reside on the fast SSD's or the internal read cache. When blocks of data aren't touched, they move to slower disks that are still more cost effective if you look to volume for your buck. Compared to Ssd's, hard disks suck at random i/o but serial streams will do just fine.


I know how it works.

It's just interesting to me that TQ's cold data store is satisfied with about 10K IOPS across those disk arrays. (Assuming 200/disk for 10K SAS and about 50% utilization given their expected multipath setup and/or redundancy/parity overhead)


the DB averages around 2K IOPS during a regular run and while we spike upwards of 60-70K IOPS during startup, typically things are somewhat calm (2,000 batches per second @ the DB layer isn't massive by any means, but it's also far from quiet)

So in the end we have a flash tier of 5+ TB with a DB that's only 3 TB, plus we have over 700 GB of RAM for buffer pool space.

We really just don't need anything faster :)

CCP DeNormalized - Database Administrator
CCP Gun Show
C C P
C C P Alliance
#64 - 2015-10-14 09:57:35 UTC
xrev wrote:
San storage enthousiast here :)

Keep in mind before upgrading from 4 or 8 Gbps to 16 Gbps to check the available and used buffer credits in combination with the average transmitted package size. Default FC frame size is supposed to be 2112 bytes and uses 1 buffercredit to transmit. With the increased speed, more buffercredits are used to fully utilize the line. When they're used, nothing will be transmitted until buffers are freed up.
Had a customer some time ago who had upgraded the linespeed and in combination with synchronous storage replication, production came to a halt because of the switches running out of buffers.

So CCP, please double check the framesizes and buffercredits before upgrading the speed ;)

Oh, just reminded, Iirc, the 48B-5 switches have 2 Condor3 asics, dividing the 48 ports in 0-23 on Asic1, and 24-47 on asic2. Keep in mind there's a default oversubscription on the asics of 1.5:1. So make sure to distribute the FC ports of a host evenly over both asics to control the performance :)

Cool stuff though. Make pictures when it arrives in the datacenters!


Excellent advice , Thanks for the constructive feedback Blink
CCP DeNormalized
C C P
C C P Alliance
#65 - 2015-10-14 10:00:05 UTC  |  Edited by: CCP DeNormalized
Corraidhin Farsaidh wrote:
The only problem I can see is with the database...should have been Oracle RAC cluster with ASM and Dataguard Active-Active failover :D

Ed; Admittedly the Oracle licensing structure makes me think it was designed by the Mittani himself (or one of his little minions) but everything has it's drawbacks...


Oracle RAC is super sexy for sure! But as you say, the licensing is nuts and at this point we just don't see any need to switch.

Cost / benefit just isn't there, and really MS SQL is quite reliable and has great HA/DR features with AlwaysOn and Availablity Groups.

we'll be doing some fun tests where we have a 4 node cluster with multiple AlwaysOn read secondary's: 2 nodes in our primary data center with a 3rd node in our DR Site and finally a 4th Node hosted @ amazon.

This is level of DR/BC that we're happy with :)

CCP DeNormalized - Database Administrator
virm pasuul
Imperial Academy
Amarr Empire
#66 - 2015-10-14 10:23:46 UTC
I wonder how many techies and possibly even sales & management of your equipment vendors actually play Eve?
I bet among Eve players you probably have as much tech workers as a fairly high tier vendor. Add in Eve Serenity players and you have the manufacture base covered too.
Xian Reevs
Armilies Corporation
Domain Research and Mining Inst.
#67 - 2015-10-14 10:38:19 UTC  |  Edited by: Xian Reevs
I love when they talk dirty like that. Geek porn.

I am just wondering if anyone have even considered Hyper-V clustering? And no, I don't want to start a Hyper-V vs SEXi discussion. ;)

CCP DeNormalized wrote:
We really just don't need anything faster :)


There is no such thing as "fast enough" for an IT enthusiast!
Corraidhin Farsaidh
Federal Navy Academy
Gallente Federation
#68 - 2015-10-14 10:53:16 UTC
CCP DeNormalized wrote:
Corraidhin Farsaidh wrote:
The only problem I can see is with the database...should have been Oracle RAC cluster with ASM and Dataguard Active-Active failover :D

Ed; Admittedly the Oracle licensing structure makes me think it was designed by the Mittani himself (or one of his little minions) but everything has it's drawbacks...


Oracle RAC is super sexy for sure! But as you say, the licensing is nuts and at this point we just don't see any need to switch.

Cost / benefit just isn't there, and really MS SQL is quite reliable and has great HA/DR features with AlwaysOn and Availablity Groups.

we'll be doing some fun tests where we have a 4 node cluster with multiple AlwaysOn read secondary's: 2 nodes in our primary data center with a 3rd node in our DR Site and finally a 4th Node hosted @ amazon.

This is level of DR/BC that we're happy with :)


What's funny is you most likely have better DR availability and testing than most of the major banks I've worked at. I wish they would pay half the attention to detail as you have :D

Ed: Actually I don't. I'd be out of a job then....
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#69 - 2015-10-14 11:23:41 UTC
Oracle licensing. ewww.

1 CPU license required per 2 cores. (for most intel kit) at around 32K per license.
Then there's 15k for each RAC license (1 per 2 cores).
And then there can be all the addins, which can double that easily. And to become liable for licensing, it can be as simple as 'run one query' or 'change one configuration setting')

Yes, I'm bitter

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Nafensoriel
Brutor Tribe
Minmatar Republic
#70 - 2015-10-14 12:05:21 UTC
virm pasuul wrote:
I wonder how many techies and possibly even sales & management of your equipment vendors actually play Eve?
I bet among Eve players you probably have as much tech workers as a fairly high tier vendor. Add in Eve Serenity players and you have the manufacture base covered too.



You would be amazed at how many physicists, geneticists, chemists, etc play EVE... Not even considering the "just graduated line members" we have access to a veritable horde of skilled technical and amazing minds.

It's not even limited to the sciences. Pick a topic and start a discussion and chances are you'll find someone in EVE online whos a kindred spirit to your discipline.
Natya Mebelle
Center for Advanced Studies
Gallente Federation
#71 - 2015-10-14 12:41:33 UTC  |  Edited by: Natya Mebelle
Every now and then I read an Eve tech article to see how much I understand. It sounds foreign and astounding as usual, because I really have no point of reference even with the statistics for what will be done and all.
But for all that tech talk, it leaves me with little that I can actually take away from the devblog to look forward to :c Even browsing the 4 pages on the forum didn't get me anywhere further.

So it leaves me with two big questions:

One... because I'm curious! How much does the entire thing cost in total?

Two... what will ACTUALLY change for us players?

Will session timers will be further reduced?
Will transition between systems be faster?
Will we get less dreaded "Socket closed" or other connectivity error messages? Because I know not every single one of them is related to the user.
Will there be less random Tidi now in remote systems that are hardly populated just because the performance is needed elsewhere?
And most importantly... are there plans with all that new power under the hood to fight the war against the 1second server tick? c:

oh and one that I forgot: With all the "redundancy" in the best way, is there a chance to look at no downtime or even faster downtime than we have nowadays?
Alundil
Rolled Out
#72 - 2015-10-14 13:12:10 UTC
Bienator II wrote:
Mara Rinn wrote:
Berahk wrote:
How much closer does this server setup bring us to never needing downtime?


Most important question in the thread :D

http://community.eveonline.com/news/dev-blogs/death-to-downtimes/


having DT only every second day would be a start :P

Congratulations. You've succeeded in nerfing capital escalations by 50%

I'm right behind you
Corraidhin Farsaidh
Federal Navy Academy
Gallente Federation
#73 - 2015-10-14 13:16:27 UTC
Nafensoriel wrote:
virm pasuul wrote:
I wonder how many techies and possibly even sales & management of your equipment vendors actually play Eve?
I bet among Eve players you probably have as much tech workers as a fairly high tier vendor. Add in Eve Serenity players and you have the manufacture base covered too.



You would be amazed at how many physicists, geneticists, chemists, etc play EVE... Not even considering the "just graduated line members" we have access to a veritable horde of skilled technical and amazing minds.

It's not even limited to the sciences. Pick a topic and start a discussion and chances are you'll find someone in EVE online whos a kindred spirit to your discipline.


One of the reasons why I like the game :)
Corraidhin Farsaidh
Federal Navy Academy
Gallente Federation
#74 - 2015-10-14 13:27:44 UTC
Steve Ronuken wrote:
Oracle licensing. ewww.

1 CPU license required per 2 cores. (for most intel kit) at around 32K per license.
Then there's 15k for each RAC license (1 per 2 cores).
And then there can be all the addins, which can double that easily. And to become liable for licensing, it can be as simple as 'run one query' or 'change one configuration setting')

Yes, I'm bitter


Addins which include the Grid Control stuff which is pretty much a necessity! I too am not at all bitter about it. Mainly because I never have to pay for it, my clients do :D
Tiddle Jr
Brutor Tribe
Minmatar Republic
#75 - 2015-10-14 13:36:31 UTC
i'm excited! And you?

"The message is that there are known knowns. There are things we know that we know. There are known unknowns. That is to say there are things that we now know we don't know. But there are also unknown unknowns. There are things we don't know" - CCP
Esrevid Nekkeg
Justified and Ancient
#76 - 2015-10-14 14:02:23 UTC  |  Edited by: Esrevid Nekkeg
Nafensoriel wrote:
... It's not even limited to the sciences. Pick a topic and start a discussion and chances are you'll find someone in EVE online whos a kindred spirit to your discipline.
Ships Carpenter here (woodworking on pricey yachts in my case)...Big smile
Natya Mebelle wrote:
Every now and then I read an Eve tech article to see how much I understand. It sounds foreign and astounding as usual, because I really have no point of reference even with the statistics for what will be done and all.
But for all that tech talk, it leaves me with little that I can actually take away from the devblog to look forward to :c Even browsing the 4 pages on the forum didn't get me anywhere further. ...
Same here. But I am more than willing to relax in the back seat of this limo enjoying the ride wile the technicians discuss the things going on under the hood.

Thanks CCP for the hard and undoubtedly necessary work on keeping TQ reliably going, now and in the future!

Here I used to have a sig of our old Camper in space. Now it is disregarded as being the wrong format. Looking out the window I see one thing: Nothing wrong with the format of our Camper! Silly CCP......
Aryth
University of Caille
Gallente Federation
#77 - 2015-10-14 14:51:45 UTC
In the writeup I don't see why IBM. Did you bake these off against UCS and they were faster? Or just whitebox.

Maybe your performance needs are very niche but I have yet to see a bakeoff where IBM won against almost anyone.

Leader of the Goonswarm Economic Warfare Cabal.

Creator of Burn Jita

Vile Rat: You're the greatest sociopath that has ever played eve.
Freelancer117
So you want to be a Hero
#78 - 2015-10-14 15:45:24 UTC
Gratz on moving to DDR4 Cool

Your new server cpu's are over a year on the market and mid range, hope you got a good price.

source: http://ark.intel.com/products/family/78583/Intel-Xeon-Processor-E5-v3-Family#@All

You mentioned Eve Forever and Dust514 wil run on Eve proxies, is CCP games still lasor focused
on tying Dust514 and New Eden capusleers together in A Future Vision with enhanced interactions ?

Regards, a Freelancer

Eve online is :

A) mining simulator B) glorified chatroom C) spreadsheets online

D) CCP Games Pay to Win at skill leveling, with instant gratification

http://eve-radio.com//images/photos/3419/223/34afa0d7998f0a9a86f737d6.jpg

http://bit.ly/1egr4mF
CCP DeNormalized
C C P
C C P Alliance
#79 - 2015-10-14 15:56:25 UTC  |  Edited by: CCP DeNormalized
Freelancer117 wrote:
Gratz on moving to DDR4 Cool

Your new server cpu's are over a year on the market and mid range, hope you got a good price.

source: http://ark.intel.com/products/family/78583/Intel-Xeon-Processor-E5-v3-Family#@All


Regards, a Freelancer



CPU's are E7-8893 v3, not E5

http://ark.intel.com/products/84688/Intel-Xeon-Processor-E7-8893-v3-45M-Cache-3_20-GHz

Launch Date Q2'15

Errr, at least the DB CPU's are :) I don't really care so much about the others :)

CCP DeNormalized - Database Administrator
Freelancer117
So you want to be a Hero
#80 - 2015-10-14 16:01:11 UTC
CCP DeNormalized wrote:


CPU's are E7-8893 v3, not E5

http://ark.intel.com/products/84688/Intel-Xeon-Processor-E7-8893-v3-45M-Cache-3_20-GHz

Launch Date Q2'15

Errr, at least the DB CPU's are :) I don't really care so much about the others :)


Hehe, wouldn't mind switching my I7 cpu to one of those ! Cool

Eve online is :

A) mining simulator B) glorified chatroom C) spreadsheets online

D) CCP Games Pay to Win at skill leveling, with instant gratification

http://eve-radio.com//images/photos/3419/223/34afa0d7998f0a9a86f737d6.jpg

http://bit.ly/1egr4mF