These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

How big is the Eve database please?

First post First post First post
Author
Previous Topic Next Topic
MiliasColds
Strategic Incompetence
Blue Sun Interstellar Technologies
#21 - 2014-10-17 17:18:17 UTC
CCP DeNormalized wrote:
MiliasColds wrote:
it's not on an SSD it's on RAMSAN AFAIK, so think more like 100k / TB (i think) but hey it's super super fast :P

edi: seems i was wrong and beaten to the punch by 1s


We used to run on the RamSan's but stopped about 3 years ago. They were completely overkill for our needs and at the time expanding was rather difficult.

We had 3 units, 2x 128 GB pure flashbased units and sometime later added a third 2 TB array (this wasn't flash based if I remember).



least i wasn't just horribly misinformed, simply out of date :), but dat 1M randon IOPS /s :P
virm pasuul
Imperial Academy
Amarr Empire
#22 - 2014-10-17 17:18:38 UTC
I've recently started to look after a work database that's 24Gb and growing by 100Mbish per day.
It's a tiny drop in the ocean compared to Eve but I keep looking at it and thinking "How can I stop it growing?"
I am considering turning the users off because they are spoiling my database.
Hengle Teron
Red Sky Morning
The Amarr Militia.
#23 - 2014-10-17 17:25:22 UTC
CCP DeNormalized wrote:
ISD LackOfFaith wrote:
Disclaimer: what follows is a nerd's best guess, not any official figures from CCP.

TL;DR: Roughly 3.5 TB (3500 GB) of data for strictly describing the universe and stuff running. Roughly 100 TB (100000 GB) for total storage across all of TQ. See below for full details.


Great effort LoF!

As mentioned, the core DB is 2.8 TB currently, and the image DB is around 1.3 TB.

the inventory system takes up about 250 GB (a single table w/ 2-3 indexes)
the market system takes up around 300 GB (2-3 tables and multiple indexes)
in-game mail system is around 300 GB as well (we delete most of chribba's mail just to annoy him :))

ok, but how much is that in cubic meters... ?
Adrie Atticus
Caldari Provisions
Caldari State
#24 - 2014-10-17 17:38:52 UTC
Hengle Teron wrote:
CCP DeNormalized wrote:
ISD LackOfFaith wrote:
Disclaimer: what follows is a nerd's best guess, not any official figures from CCP.

TL;DR: Roughly 3.5 TB (3500 GB) of data for strictly describing the universe and stuff running. Roughly 100 TB (100000 GB) for total storage across all of TQ. See below for full details.


Great effort LoF!

As mentioned, the core DB is 2.8 TB currently, and the image DB is around 1.3 TB.

the inventory system takes up about 250 GB (a single table w/ 2-3 indexes)
the market system takes up around 300 GB (2-3 tables and multiple indexes)
in-game mail system is around 300 GB as well (we delete most of chribba's mail just to annoy him :))

ok, but how much is that in cubic meters... ?


0.02 m3 per module, not sure how many, but I'd guess it's a full rack; 1.43 m3 per full 42U rack.
ISD LackOfFaith
ISD Community Communications Liaisons
ISD Alliance
#25 - 2014-10-17 17:40:25 UTC
CCP DeNormalized wrote:
ISD LackOfFaith wrote:
Disclaimer: what follows is a nerd's best guess, not any official figures from CCP.

TL;DR: Roughly 3.5 TB (3500 GB) of data for strictly describing the universe and stuff running. Roughly 100 TB (100000 GB) for total storage across all of TQ. See below for full details.


Great effort LoF!

As mentioned, the core DB is 2.8 TB currently, and the image DB is around 1.3 TB.

the inventory system takes up about 250 GB (a single table w/ 2-3 indexes)
the market system takes up around 300 GB (2-3 tables and multiple indexes)
in-game mail system is around 300 GB as well (we delete most of chribba's mail just to annoy him :))

Woo! Hooray ballparking!

Also, I completely forgot about the mail system, oops.

ISD LackOfFaith

Captain

Community Communication Liaisons (CCLs)

Interstellar Services Department

I do not respond to Eve Mail or anything other than the forums.
CCP DeNormalized
C C P
C C P Alliance
#26 - 2014-10-17 17:44:29 UTC
virm pasuul wrote:
I've recently started to look after a work database that's 24Gb and growing by 100Mbish per day.
It's a tiny drop in the ocean compared to Eve but I keep looking at it and thinking "How can I stop it growing?"
I am considering turning the users off because they are spoiling my database.


we discuss similar things :)

An on-going project is trimming the market orders table - we tried to talk the game designers into just letting us truncate the table but they wanted no part of it :)

CCP DeNormalized - Database Administrator
Chainsaw Plankton
FaDoyToy
#27 - 2014-10-17 17:46:50 UTC
CCP DeNormalized wrote:
MiliasColds wrote:
it's not on an SSD it's on RAMSAN AFAIK, so think more like 100k / TB (i think) but hey it's super super fast :P

edi: seems i was wrong and beaten to the punch by 1s


We used to run on the RamSan's but stopped about 3 years ago. They were completely overkill for our needs and at the time expanding was rather difficult.

We had 3 units, 2x 128 GB pure flashbased units and sometime later added a third 2 TB array (this wasn't flash based if I remember).




I remember seeing something about the Ramsan in a devblog a while back. all i know is thing sounded shiny

@ChainsawPlankto on twitter
virm pasuul
Imperial Academy
Amarr Empire
#28 - 2014-10-17 17:55:51 UTC
CCP DeNormalized wrote:
virm pasuul wrote:
I've recently started to look after a work database that's 24Gb and growing by 100Mbish per day.
It's a tiny drop in the ocean compared to Eve but I keep looking at it and thinking "How can I stop it growing?"
I am considering turning the users off because they are spoiling my database.


we discuss similar things :)

An on-going project is trimming the market orders table - we tried to talk the game designers into just letting us truncate the table but they wanted no part of it :)


Make everything on the market a fixed price, then the table will become much smaller. Seems a good compromise to me :)
Caviar Liberta
The Scope
Gallente Federation
#29 - 2014-10-17 17:57:53 UTC
TheMercenaryKing wrote:
CCP DeNormalized wrote:
DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...

We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.

We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total.


How often do you backup the servers? Daily via deduplication?

I can't imagine your RAID setup so the throughput must be pretty good for all the transactions. SSDs or a lot of 15K drives?
15k drives have about 175 IOPS at 4K block size i think. I doubt you are using 4K, probably another block size for optimization.


The intial back up would be the most time consuming. Setting up the back to only copy any files that has been changed or added would make the task a lot easier then.
CCP DeNormalized
C C P
C C P Alliance
#30 - 2014-10-17 18:02:28 UTC
Caviar Liberta wrote:
TheMercenaryKing wrote:
CCP DeNormalized wrote:
DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...

We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.

We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total.


How often do you backup the servers? Daily via deduplication?

I can't imagine your RAID setup so the throughput must be pretty good for all the transactions. SSDs or a lot of 15K drives?
15k drives have about 175 IOPS at 4K block size i think. I doubt you are using 4K, probably another block size for optimization.


The intial back up would be the most time consuming. Setting up the back to only copy any files that has been changed or added would make the task a lot easier then.


we do different things w/ each backup - the secondary one gets shipped to a different server for reporting purposes, but you are correct, a differential would make sense in most cases here

CCP DeNormalized - Database Administrator
CCP DeNormalized
C C P
C C P Alliance
#31 - 2014-10-17 18:04:39 UTC
virm pasuul wrote:
CCP DeNormalized wrote:
virm pasuul wrote:
I've recently started to look after a work database that's 24Gb and growing by 100Mbish per day.
It's a tiny drop in the ocean compared to Eve but I keep looking at it and thinking "How can I stop it growing?"
I am considering turning the users off because they are spoiling my database.


we discuss similar things :)

An on-going project is trimming the market orders table - we tried to talk the game designers into just letting us truncate the table but they wanted no part of it :)


Make everything on the market a fixed price, then the table will become much smaller. Seems a good compromise to me :)


In this market order clean up, we noticed that there is one crazy player that has an open order for 7 years now!!!! Any order expires after 90 days unless you update it...

So this player has been modifying this one order, at least once every 90 days, for the last 7 frickin years!! haha

There are 3 items left in his sell order - I wanted to just go buy them so there wouldn't be such a big gap in ID's :)

I got over ruled on that as well

(i sent a mail to community about this guy saying he (or she?) should get some kinda medal, so we'll see what comes of that)

CCP DeNormalized - Database Administrator
Max Kolonko
Caldari Provisions
Caldari State
#32 - 2014-10-17 18:08:20 UTC
CCP Denormalized - best CCP name for the job :)

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
Jandice Ymladris
Aurora Arcology
#33 - 2014-10-17 18:13:55 UTC
I love threads like this! Very informative & you learn alot about the technical aspects of EvE (and you get to see some CCP folks that are usually very good at hiding)

Providing a new home for refugees in the Aurora Arcology
Gilbaron
The Scope
Gallente Federation
#34 - 2014-10-17 18:30:17 UTC
I'm curious about that 7 year old market order. Is it an overpriced sellorder, an underpriced buyorder or an item that nobody ever buys?
virm pasuul
Imperial Academy
Amarr Empire
#35 - 2014-10-17 18:32:08 UTC
It's gotta be a finger slip order that's hoping to catch a drunken finger slip. The quantity is small and it probably cost a lot to set up - hence the desire not to pay to set it up again.
Chainsaw Plankton
FaDoyToy
#36 - 2014-10-17 18:36:18 UTC
I don't think I've had a market order last over 180 days, usually after a few weeks of modifying orders I get bored, go do something else, and forget about it

@ChainsawPlankto on twitter
CCP DeNormalized
C C P
C C P Alliance
#37 - 2014-10-17 18:39:20 UTC
it is a sell order, that started with 20 units - 17 sold and 3 left :)

it's only like 1.6 mil per unit and not the highest in the region.

CCP DeNormalized - Database Administrator
Jandice Ymladris
Aurora Arcology
#38 - 2014-10-17 18:47:21 UTC
CCP DeNormalized wrote:
it is a sell order, that started with 20 units - 17 sold and 3 left :)

it's only like 1.6 mil per unit and not the highest in the region.


Must be something very rarely traded then! Or perhaps the creator just likes to see how long he can keep running it... Still, darn impressive lasting that long.

Providing a new home for refugees in the Aurora Arcology
virm pasuul
Imperial Academy
Amarr Empire
#39 - 2014-10-17 18:55:33 UTC  |  Edited by: virm pasuul
Hey DeNormalized when is CCP going to set up a database administrator's player event?
We could all drink beer and complain about users touching the database.

It could be the only Eve player event ever where no one was allowed to log onto Eve.
CCP DeNormalized
C C P
C C P Alliance
#40 - 2014-10-17 19:07:39 UTC
virm pasuul wrote:
Hey DeNormalized when is CCP going to set up a database administrator's player event?
We could all drink beer and complain about users touching the database.

It could be the only Eve player event ever where no one was allowed to log onto Eve.


We do it every year @ fanfest during our virtual worlds admin round table/QandA :)

CCP DeNormalized - Database Administrator