These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

How big is the Eve database please?

First post First post First post
Author
Previous Topic Next Topic
Luscius Uta
#61 - 2014-10-19 19:52:42 UTC
CCP DeNormalized wrote:
virm pasuul wrote:
CCP DeNormalized wrote:
virm pasuul wrote:
I've recently started to look after a work database that's 24Gb and growing by 100Mbish per day.
It's a tiny drop in the ocean compared to Eve but I keep looking at it and thinking "How can I stop it growing?"
I am considering turning the users off because they are spoiling my database.


we discuss similar things :)

An on-going project is trimming the market orders table - we tried to talk the game designers into just letting us truncate the table but they wanted no part of it :)


Make everything on the market a fixed price, then the table will become much smaller. Seems a good compromise to me :)


In this market order clean up, we noticed that there is one crazy player that has an open order for 7 years now!!!! Any order expires after 90 days unless you update it...

So this player has been modifying this one order, at least once every 90 days, for the last 7 frickin years!! haha

There are 3 items left in his sell order - I wanted to just go buy them so there wouldn't be such a big gap in ID's :)

I got over ruled on that as well

(i sent a mail to community about this guy saying he (or she?) should get some kinda medal, so we'll see what comes of that)


The only reason why EVE needs achievements

Workarounds are not bugfixes.
CCP DeNormalized
C C P
C C P Alliance
#62 - 2014-10-19 22:04:27 UTC
Jandice Ymladris wrote:

I wonder how much data our assets are occupying on the server? As unlike other MMO's, we can have stuff everywhere indefinatly, so I suspect it would be a big drain on data storage? Or not really as it's more static?



the inventory system is around 250 GB - this includes multiple indexes on the same data though so we (or you) can access it faster, etc...

every item/stack per location has a unique row in the DB, for a total of around 2.2 billion rows at the moment :)

CCP DeNormalized - Database Administrator
Tzeentchful Nakrar
ANZAC ALLIANCE
Goonswarm Federation
#63 - 2014-10-20 01:59:36 UTC
CCP DeNormalized wrote:
Jandice Ymladris wrote:

I wonder how much data our assets are occupying on the server? As unlike other MMO's, we can have stuff everywhere indefinatly, so I suspect it would be a big drain on data storage? Or not really as it's more static?



the inventory system is around 250 GB - this includes multiple indexes on the same data though so we (or you) can access it faster, etc...

every item/stack per location has a unique row in the DB, for a total of around 2.2 billion rows at the moment :)


That is a lot of rows. You guys must have some really insane servers to serve that amount of data at the seed you do.
I do have a quick question. What is the oldest item a players has?
Bienator II
madmen of the skies
#64 - 2014-10-20 03:59:18 UTC
i am wondering if the DB would be still useable when every item in eve would be unstacked. Since if it would ... can we pretty please have manufacturer info and other metadata for items? ;)

how to fix eve: 1) remove ECM 2) rename dampeners to ECM 3) add new anti-drone ewar for caldari 4) give offgrid boosters ongrid combat value
Max Kolonko
Caldari Provisions
Caldari State
#65 - 2014-10-20 05:56:36 UTC
Bienator II wrote:
i am wondering if the DB would be still useable when every item in eve would be unstacked. Since if it would ... can we pretty please have manufacturer info and other metadata for items? ;)


that trit

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
WhyYouHeffToBeMad IsOnlyGame
#66 - 2014-10-20 08:27:01 UTC
Sakurako Kimino wrote:
CCP DeNormalized wrote:
Toriessian wrote:
Jandice Ymladris wrote:
CCP DeNormalized wrote:
it is a sell order, that started with 20 units - 17 sold and 3 left :)

it's only like 1.6 mil per unit and not the highest in the region.


Must be something very rarely traded then! Or perhaps the creator just likes to see how long he can keep running it... Still, darn impressive lasting that long.


Its about the right price for synth boosters.


Perhaps, but this isn't those!! :)



if you tell us the item we might buy them out lol

that is why he doesn't want to tell which item it is so to not ruin the EVE record for the longest ongoing sell order. Roll

Everything's a game if you make it one - Uriel Paradisi Anteovnuecci

CCP: Continously Crying Playerbase - Frostys Virpio
Varisto
North Star Science And Industry
#67 - 2014-10-20 08:34:14 UTC
So you run on SSD that database? Now in curious how long does it take you to kill one drive? SSD dont usually like very much on constantly changing data...
CCP DeNormalized
C C P
C C P Alliance
#68 - 2014-10-20 10:28:26 UTC
Varisto wrote:
So you run on SSD that database? Now in curious how long does it take you to kill one drive? SSD dont usually like very much on constantly changing data...


I've asked our SAN guys that very question in the past :)

They say upwards of 5 year life span depending on how many writes, etc... keep in mind these are enterprise grade SSD's (with the insane prices to boot) and have extra 'stuff' to keep them alive longer :)

100% moar stuff!!! yeaaa!!!! hehe

We've yet to lose one, but I believe we're getting close to end of life and may likely replace in the coming year or so.

CCP DeNormalized - Database Administrator
CCP DeNormalized
C C P
C C P Alliance
#69 - 2014-10-20 10:40:58 UTC
Tzeentchful Nakrar wrote:

That is a lot of rows. You guys must have some really insane servers to serve that amount of data at the seed you do.
I do have a quick question. What is the oldest item a players has?


The most insane part of our DB is ram IMO - we're running 512 GB of ram, so it's able to pull a tremendous amount of data into memory and that gives us great speed.

That much RAM coupled with fast SSD's (and properly indexed data) and you can do quite a bit :)

In terms of oldest item, it's a very difficult thing to find. Up until 4 years ago or so, we were recycling ID's (32 bit) so we are really able to say beyond that.

Now we use 64 bit and can trace how old items are, but I think it's fair to say that there are items in hangars from the beginning :)

CCP DeNormalized - Database Administrator
Johan Civire
Caldari Provisions
Caldari State
#70 - 2014-10-20 11:09:05 UTC
This is good reading stuff :D i like it. But about that order can we have a name or what do he or she sells or is that forbidden? i like to know now ? :D
Jandice Ymladris
Aurora Arcology
#71 - 2014-10-20 11:10:04 UTC
I'm surprised the inventory 'only' takes 259 GB of space, considering we tend to litter stuff everywhere as our characters age (I admit I'm terrible at it, think my asset list expands over a hundred stations....) So sounds like a pretty efficient system you got (I guess it helps that you don't need to 'generate' a history for the items as well, at least the ones you can stack)

On the other hand 2.2 billion rows, and one row per item/stack per location, that's one humongous list.... Albeit I presume this includes inactive accounts as well.


That ram tho... 512 GB! That just boggles my mind. Makes me wonder what amounts of ram you normally see to run servers? Or is this big amount an EvE thing to keep it a one shard thing?

Providing a new home for refugees in the Aurora Arcology
Max Kolonko
Caldari Provisions
Caldari State
#72 - 2014-10-20 11:21:23 UTC
Jandice Ymladris wrote:



That ram tho... 512 GB! That just boggles my mind. Makes me wonder what amounts of ram you normally see to run servers? Or is this big amount an EvE thing to keep it a one shard thing?


Rams going into hundreds or thousands of gigs for dedicated database clusters are slowly becoming standard for large databases. I think google was one of the first to utilize it in their hadoop design.

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
Hengle Teron
Red Sky Morning
The Amarr Militia.
#73 - 2014-10-20 11:22:27 UTC
CCP DeNormalized wrote:


The most insane part of our DB is ram IMO - we're running 512 GB of ram, so it's able to pull a tremendous amount of data into memory and that gives us great speed.
:)

Mind blown
Arline Kley
PIE Inc.
Khimi Harar
#74 - 2014-10-20 11:25:25 UTC
Jandice Ymladris wrote:
That ram tho... 512 GB! That just boggles my mind. Makes me wonder what amounts of ram you normally see to run servers? Or is this big amount an EvE thing to keep it a one shard thing?


512GB of RAM is pretty big; the cluster I am running here is something like 120GB RAM for 9 seperate servers (each virtualized) whilst the data is sat on a 8TB SAN running tradtional HDD's. We did try to get SSD's but sadly the costs were prohibitive.


I would give my left arm to work in CCP's datacentre.

"For it was said they had become like those peculiar demons, which dwell in matter but in whom no light may be found." - Father Grigori, Ravens 3:57
CCP DeNormalized
C C P
C C P Alliance
#75 - 2014-10-20 11:25:32 UTC
Johan Civire wrote:
This is good reading stuff :D i like it. But about that order can we have a name or what do he or she sells or is that forbidden? i like to know now ? :D


I wouldn't tell, no... community may choose to do something with this info, but if not it'll just be one more mystery of the universe :)

CCP DeNormalized - Database Administrator
Kousaka Otsu Shigure
#76 - 2014-10-20 11:26:41 UTC
One of my side projects is creating an out-of-game contacts list. Is our ingame contacts list cap somehow related to how the database works?

I'm at ~24k hisec contacts at the moment and i must admit, im such a noob with databases.

Archiver, Software Developer and Data Slave

Current Project Status: What can I make with these minerals?
CCP DeNormalized
C C P
C C P Alliance
#77 - 2014-10-20 11:43:53 UTC
Kousaka Otsu Shigure wrote:
One of my side projects is creating an out-of-game contacts list. Is our ingame contacts list cap somehow related to how the database works?

I'm at ~24k hisec contacts at the moment and i must admit, im such a noob with databases.


In most cases any sort of cap would be done for performance reasons, either due to the DB or the client itself. I can't see the DB being an issue here however.

Mind you, if you were to have 24,000 contacts in-game, i would think the overhead of loading those in the client and doing whatever happens with those, would definitely have a negative impact on client side performance :)

CCP DeNormalized - Database Administrator
Wacktopia
Fleet-Up.com
Keep It Simple Software Group
#78 - 2014-10-20 12:46:10 UTC
Chribba wrote:
CCP DeNormalized wrote:
DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...

We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.

We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total.

So that's why you're not letting me read mails further back than a month Sad Stop purging and give me my mails back! Lol


See, all the cool kids are deleting their data. :)

Kitchen sink? Seriousy, get your ship together -  Fleet-Up.com
Arcos Vandymion
Ministry of War
Amarr Empire
#79 - 2014-10-20 14:25:34 UTC
Jandice Ymladris wrote:
I'm surprised the inventory 'only' takes 259 GB of space, considering we tend to litter stuff everywhere as our characters age (I admit I'm terrible at it, think my asset list expands over a hundred stations....) So sounds like a pretty efficient system you got (I guess it helps that you don't need to 'generate' a history for the items as well, at least the ones you can stack)


I perferr my cchaos concentrated into one abysmal eye of terror... I did have to rent freighter services a handful of times allready to ferry stuff between stations.

Currently everything is on one station and some assests in null that i can't get sold ...




Regarding the contacts - I recently did a single mission for the majority of corporations in EVE and whenever I open my standings tab now (which really could need nifty little arrows to expand or collapse the tree) it needs a while. Can't imagine what the game would do if you had a lsit of 24k to pull and display.
Gospadin
Bastard Children of Poinen
#80 - 2014-10-20 17:29:10 UTC  |  Edited by: Gospadin
Varisto wrote:
So you run on SSD that database? Now in curious how long does it take you to kill one drive? SSD dont usually like very much on constantly changing data...


There are essentially no actual field usage models that will burn out the latest generation of enterprise SSDs with 10DWD or above ratings. 10 DWD to an 800GB drive is 100MB/s continuously, 24/7/365. In a normal 80/20 read/write workload typical of many databases, essentially the drive will run bus-saturated for the entire warranty duration.

Most people don't understand what a petabyte of data written to a single device looks like, while drives launched a year ago are good for multiple petabytes per drive. Build an array out of them, and you're talking multiple tens of petabytes. That rating accounts for all internal data movement as well, meaning those petabytes are true user petabytes.