These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
123Next pageLast page
 

How big is the Eve database please?

First post First post First post
Author
virm pasuul
Viziam
Amarr Empire
#1 - 2014-10-17 13:07:10 UTC
Just being curious. How big in Gb is Tranquility please?
I'm guessing big :)
Doc Fury
Furious Enterprises
#2 - 2014-10-17 13:10:26 UTC  |  Edited by: Doc Fury
It's as big as a baby, Ren!

There's a million angry citizens looking down their tubes..at me.

Ramona McCandless
The McCandless Clan
#3 - 2014-10-17 13:11:24 UTC
Its not as big as a Texan Bar

"Yea, some dude came in and was normal for first couple months, so I gave him director." - Sean Dunaway

"A singular character could be hired to penetrate another corps space... using gorilla like tactics..." - Chane Morgann

Hengle Teron
We Shoot Things. Sometimes
Two Maidens One Chalice
#4 - 2014-10-17 14:19:13 UTC
Can you tell me that in cubic meters? please !
Erin Crawford
#5 - 2014-10-17 14:29:50 UTC
as big as a Fedo's bottom! Shocked

"Those who talk don’t know. Those who know don’t talk. "

Guttripper
State War Academy
Caldari State
#6 - 2014-10-17 15:45:33 UTC
Definitely smaller than most male teenager's pornography collection...
The Forum Alt
Hedion University
Amarr Empire
#7 - 2014-10-17 16:08:30 UTC
I have it on good authority that the size of the database is smaller than 1000 petabytes but bigger than 1 byte.
Chribba
Otherworld Enterprises
Otherworld Empire
#8 - 2014-10-17 16:20:33 UTC
Last time I heard it was a few TB, that was some years ago, so with Jita's daily market transactions I would guess 10x the size now.

/c

★★★ Secure 3rd party service ★★★

Visit my in-game channel 'Holy Veldspar'

Twitter @ChribbaVeldspar

Unsuccessful At Everything
The Troll Bridge
#9 - 2014-10-17 16:23:43 UTC
Im sure the database is the same size in Great Britain as it would be anywhere else in the world.




Since the cessation of their usefulness is imminent, may I appropriate your belongings?

CCP DeNormalized
C C P
C C P Alliance
#10 - 2014-10-17 16:24:51 UTC
DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...

We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.

We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total.

CCP DeNormalized - Database Administrator

Chribba
Otherworld Enterprises
Otherworld Empire
#11 - 2014-10-17 16:26:52 UTC
CCP DeNormalized wrote:
DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...

We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.

We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total.

So that's why you're not letting me read mails further back than a month Sad Stop purging and give me my mails back! Lol

★★★ Secure 3rd party service ★★★

Visit my in-game channel 'Holy Veldspar'

Twitter @ChribbaVeldspar

CCP DeNormalized
C C P
C C P Alliance
#12 - 2014-10-17 16:28:08 UTC
Chribba wrote:
Last time I heard it was a few TB, that was some years ago, so with Jita's daily market transactions I would guess 10x the size now.

/c


Over the course of the last year, the market tables have grown by about 500 GB :)

CCP DeNormalized - Database Administrator

ISD LackOfFaith
ISD Community Communications Liaisons
#13 - 2014-10-17 16:47:16 UTC  |  Edited by: ISD LackOfFaith
Disclaimer: what follows is a nerd's best guess, not any official figures from CCP.

TL;DR: Roughly 3.5 TB (3500 GB) of data for strictly describing the universe and stuff running. Roughly 100 TB (100000 GB) for total storage across all of TQ. See below for full details.



A MySQL dump of the static database (item types, solar systems, orbital objects, etc) is around 300 MB.

Total so far: 300 MB

The Community Toolkit downloads for Types, Renders, and Icons are an extra 600 MB or so, zipped. Unzipped at a conservative 50% compression rate, that's 1.2 GB of data just for various static images.

Total so far: 1.5 GB

While we're assuming all static data is also part of the database, we can add the size of a regular install too; say, 10 GB.

Total so far: 11.5 GB

Somewhat less dynamic are character portraits, each of which are saved in varying sizes, from 1024x1024 JPGs down to 32x32 JPG icons. Based on my own portrait, i'm going to estimate the total per-character of these is 250 KB. In 2013, Eve reached "over 500,000 subscribers". Assuming an average of 2.5 characters per account (not everyone uses all 3 slots, but many do), that's 1,250,000 characters. That means 312.5 GB in just character portraits.

Total so far: 324 GB

Of course, that's active subscriptions. How many trial players (or player alts) started accounts that are now inactive? Eve is a pretty rough niche game, so let's say there are 1,500,000 of those. That's another 375 GB of portraits.

Total so far: 699 GB

Now, let's look at ships. The 10 year commemoration video estimates 40,000,000 ships lost. Let's add another 5,000,000 for a violent year. Assuming the number of ships lost per day has been going up linearly since the game's launch (it hasn't, but let's assume), that means that after the (roughly) 4200 days of Eve's lifespan, we're getting about 21,000 lost ships per day. Every time you lose a ship, how many more do you have sitting around? I have no access to those stats, and no idea where to get them, but I'm going to ballpark 10 for the average. 10 ships per 1,250,000 active character is 12,500,000 ships. Add one noobship for every inactive account and you get 14,000,000 ships.

Each ship has a bunch (20-30) of attributes associated with it. Each of those is a 64-bit (8 byte) integer value. 25 attributes per ship, with 8 bytes per attribute and 14,000,000 ships, that's 2.8 GB of data.

Total so far: 702 GB (note: I have only considered assembled ships so far)

But many of those ships have modules fit to them! Let's estimate 8 modules/rigs fit per ship. Each one of those also has a bunch of attributes, and blah blah blah... another 1 GB of stuff.

Total so far: 703 GB

What about other items? Well... Eve-central has a 200 MB gzipped market dump for every day. I'm not going to bother downloading and unzipping those, and call it 1 GB per day. If that's the data size of just the items being traded, how much stuff is there in game? How many buy/sell orders does the average character have going at any one time, compared to their items? Well, no more than 5, unless they have trading skills trained (few non-dedicated traders do). Let's just call it 5 orders. Eve players are pack rats and keep lots of stuff scattered all around the universe. I just checked and it looks like I have many thousands of things scattered about. Though, compensating for alts and new players, I'm going to ballpark 2,000 items (stacks of things) average per character. That means that on average, players have 500 item more items than they do market orders. If an item takes roughly the same amount of data to store, then that's 500 GB of data about peoples' stuff. Plus 1 GB of the actual current orders.

Total so far: 1,304 GB

What about stateful stuff in space? NPCs? Drones? Wrecks? Corpses? Large collidable objects? Asteroids? Mission structures? Wormholes? POSes? Player outposts? Mobile depots? Bubbles? This number is extremely hard to gauge because many of these aren't simple items. They all have positional data, and probably a variety of extra stuff stored server-side to make them work. Their number also varies wildly from system to system. I'm not sure how to approach it, but I'm going to ballpark that it takes multiple times the space it takes to store peoples' stuff (which is relatively static and boring). Let's call it just under 3x, so 2,000 GB.

Total so far: 3,304 GB

You could add logs and other non-critical data to make it an even 3.5 TB.

I can't think of anything else that needs storing, and at this point I've done so much estimation that the answer may be off by quite a lot.

Still... 3.5 TB is not that much, is it? I mean, on Newegg, a 2 TB solid state drive costs just $3300. For about $6000-7000 you could store all of Eve's data on a very fast drive, right?

Well... not so much. Eve is a big place, and a single SSD has no way to keep up with how intensely it would have to be used if it served as the central repository. Also, can you imagine what would happen if the drive broke? You would need extensive replication of all that data both for performance and for backup purposes. On top of that, remember that Tranquility is distributed across 60 (?) nodes, each with a local copy of high-demand data (your ships and stuff in space) plus other cached data.

Considering all the duplication of data necessary to have a system as robust as TQ (yeah, don't laugh, TQ is amazingly robust for what it is), I would estimate there to be something on the order of 100 TB of data floating around in storage between all the machines in TQ put together. Maybe a bit less. Maybe a bit more. Who knows, I don't work there.

Hope this helps!

ISD LackOfFaith

Captain

Community Communication Liaisons (CCLs)

Interstellar Services Department

I do not respond to Eve Mail or anything other than the forums.

TheMercenaryKing
Ultimatum.
Almost Awesome.
#14 - 2014-10-17 16:47:38 UTC
CCP DeNormalized wrote:
DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...

We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.

We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total.


How often do you backup the servers? Daily via deduplication?

I can't imagine your RAID setup so the throughput must be pretty good for all the transactions. SSDs or a lot of 15K drives?
15k drives have about 175 IOPS at 4K block size i think. I doubt you are using 4K, probably another block size for optimization.
Bienator II
madmen of the skies
#15 - 2014-10-17 16:49:54 UTC
* stacked his loot to make the DB smaller

how to fix eve: 1) remove ECM 2) rename dampeners to ECM 3) add new anti-drone ewar for caldari 4) give offgrid boosters ongrid combat value

CCP DeNormalized
C C P
C C P Alliance
#16 - 2014-10-17 16:59:00 UTC
ISD LackOfFaith wrote:
Disclaimer: what follows is a nerd's best guess, not any official figures from CCP.

TL;DR: Roughly 3.5 TB (3500 GB) of data for strictly describing the universe and stuff running. Roughly 100 TB (100000 GB) for total storage across all of TQ. See below for full details.


Great effort LoF!

As mentioned, the core DB is 2.8 TB currently, and the image DB is around 1.3 TB.

the inventory system takes up about 250 GB (a single table w/ 2-3 indexes)
the market system takes up around 300 GB (2-3 tables and multiple indexes)
in-game mail system is around 300 GB as well (we delete most of chribba's mail just to annoy him :))

CCP DeNormalized - Database Administrator

CCP DeNormalized
C C P
C C P Alliance
#17 - 2014-10-17 17:08:28 UTC
TheMercenaryKing wrote:


How often do you backup the servers? Daily via deduplication?

I can't imagine your RAID setup so the throughput must be pretty good for all the transactions. SSDs or a lot of 15K drives?
15k drives have about 175 IOPS at 4K block size i think. I doubt you are using 4K, probably another block size for optimization.


DB is split across 2 tiers of storage, SSD (enterprise level intel disks in v7000 enclosures) for the heavily accessed stuff and 15k SAS disks for the second 'slower' tier.

I'm not that knowledgeable abuot the SAN side and how the block sizes are handed there, but as the LUN's are given to us, we setup our volumes @ 64K block size as this is the optimal allocation for SQL (64k = 1 extent = 8x8KB pages)

We do 2 full backups per day (written across 8 files for disk melting IO dumps) which take a little over 1 hour each... Hits the disks for around 400 MB/s

CCP DeNormalized - Database Administrator

MiliasColds
Strategic Incompetence
Blue Sun Interstellar Technologies
#18 - 2014-10-17 17:09:12 UTC  |  Edited by: MiliasColds
it's not on an SSD it's on RAMSAN AFAIK, so think more like 100k / TB (i think) but hey it's super super fast :P

edi: seems i was wrong and beaten to the punch by 1s
virm pasuul
Viziam
Amarr Empire
#19 - 2014-10-17 17:09:22 UTC
TY DeNormalized, and thank you for caring for our sandpit.
CCP DeNormalized
C C P
C C P Alliance
#20 - 2014-10-17 17:12:47 UTC
MiliasColds wrote:
it's not on an SSD it's on RAMSAN AFAIK, so think more like 100k / TB (i think) but hey it's super super fast :P

edi: seems i was wrong and beaten to the punch by 1s


We used to run on the RamSan's but stopped about 3 years ago. They were completely overkill for our needs and at the time expanding was rather difficult.

We had 3 units, 2x 128 GB pure flashbased units and sometime later added a third 2 TB array (this wasn't flash based if I remember).

CCP DeNormalized - Database Administrator

123Next pageLast page