These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Information Portal

 
  • Topic is locked indefinitely.
Previous page123Next page
 

Dev blog: EVE Universe Static Data Export

First post
Author
Previous Topic Next Topic
Makoto Priano
Kirkinen-Arataka Transhuman Zenith Consulting Ltd.
Arataka Research Consortium
#21 - 2014-02-14 17:36:48 UTC
Benny Ohu wrote:

Since CCP has talked about a content arc over the next few expansions that involves building stargates and new space, this might be foundation for developing that feature? I don't know computer :(


Odds are, yes. ;) But I'd love to hear a dev talk about possible applications. ;)

Itsukame-Zainou Hyperspatial Inquiries: exploring the edge of the known, advancing the state of the art. Would you like to know more?
Akrasjel Lanate
Immemorial Coalescence Administration
Immemorial Coalescence
#22 - 2014-02-14 17:57:10 UTC
Quote:
What does this mean for the regular pod pilot? Players should not notice any difference what-so-ever.


Yep i'm sure after DT all EVE will be broken LolRoll

CEO of Lanate Industries

Citizen of Solitude
Max Kolonko
Caldari Provisions
Caldari State
#23 - 2014-02-14 19:02:34 UTC
CCP Xhagen wrote:
Benny Ohu wrote:
CCP Xhagen wrote:
They should only be off by a meter or two.

not just one metre but possibly two??

D: gearing up to shot statue

Only if you can find it again!


Touche

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
JP Nakamura
Union of Intergalactic Miners and Nano Assemblers
#24 - 2014-02-14 19:26:52 UTC
Gilbaron wrote:
so, is this one of the "what could possibly go wrong" updates ?


Yes/No ... in this case their just mucking about with the Server, not the Client, so its only HALF as likely something will go wrong P

(Then again, we are talking about the underlying structure of the universe ... so all bets are off Lol)

CCP: 10+ years of Harvesting players Tears  (latest efforts being Source Limited Edition, and Alliance Logo Revised Policies)
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#25 - 2014-02-14 19:46:43 UTC
I've just restored the SQL server backup, and it still has the map* tables in it. They're empty though.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
CCP Nobody
C C P
C C P Alliance
#26 - 2014-02-14 20:16:18 UTC
Makoto Priano wrote:
So, the questions: is this the change needed to allow for creation of structures to not be tied to downtime, a la player-created stations? And would this change now allow players-- with new game mechanics, of course --to destroy stations and stargates? Or would this allow, say, wormhole signatures to retain their IDs across downtime?

Basically: obviously this has a lot of utility. Is this being deployed primarily for Team Gridlock purposes, or is its primary aim backbone for the summer expansion?

The data that was in this migration was "only" information about static objects in space.

That is information that doesn't change for regions, constellations, solarSystems, suns, secondary suns, planets, moons, stations, asteroid belts and stargates. There is no way to add/remove information from this data set since it is built into static files.

What this will give us is an easier way to modify the universe, without making development hard for the rest of the company.

JP Nakamura wrote:
Yes/No ... in this case their just mucking about with the Server, not the Client, so its only HALF as likely something will go wrong

Actually this has quite the change for clients as well :) But hopefully nobody will notice anything different, except the starmap being a bit faster...
Atum
Eclipse Industrials
Quantum Forge
#27 - 2014-02-14 20:49:17 UTC
Steve Ronuken wrote:
I've just restored the SQL server backup, and it still has the map* tables in it. They're empty though.

I noticed that as well... haven't had a chance to play with the new sqlite db though, it's been a busy day at the office.
Billy Hix
Dreddit
Test Alliance Please Ignore
#28 - 2014-02-14 21:47:04 UTC  |  Edited by: Billy Hix
Steve Ronuken, you seem to know your stuff. What sort of future changes does this update allow for that couldn't easily be done without the changes?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#29 - 2014-02-14 22:09:45 UTC
Billy Hix wrote:
Steve Ronuken, you seem to know your stuff. What sort of future changes does this update allow for that couldn't easily be done without the changes?



Adding celestials, removing celestials. Changing the links between systems. Adding systems (well, with other changes too.)

This will have more effect on the development process, than on actual deployments, as it'll let them handle the changes in a properly staged fashion, with the changes in data being very easy to document.

Change management in databases is a pain in the ass.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Daedalus III
Caldari Provisions
Caldari State
#30 - 2014-02-14 23:35:34 UTC
I'm wondering; if parts of the database has to be created each time the server is started, won't this risk extending the DT which you have worked so hard on reducing?
Andy Zola
Pator Tech School
Minmatar Republic
#31 - 2014-02-14 23:35:42 UTC
Does this roll up into the build your own stargate teaser at the last Fanfest? This change will make it easier to add new areas/celestials on the fly?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#32 - 2014-02-15 00:33:39 UTC
New conversions.

https://www.fuzzwork.co.uk/dump/sqlite-latest.sqlite.bz2
https://www.fuzzwork.co.uk/dump/postgres-latest.sql.bz2
https://www.fuzzwork.co.uk/dump/mysql-latest.tar.bz2

https://www.fuzzwork.co.uk/dump/rubicon-1.2-94438/


For mysql, you'll find the universe data in a separate file to import. the others have it embedded.

I'm /pretty/ sure it'll work, though the conversion tool I used had some data type issues. Seriously considering picking up a commercial tool for this. Smile

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Atum
Eclipse Industrials
Quantum Forge
#33 - 2014-02-15 00:46:13 UTC
Daedalus III wrote:
I'm wondering; if parts of the database has to be created each time the server is started, won't this risk extending the DT which you have worked so hard on reducing?

Not really... they'll only need to regenerate the map db file when something actually changes, which probably isn't all that often. Do the regen on a patch day, and nobody would know the difference.
JP Nakamura
Union of Intergalactic Miners and Nano Assemblers
#34 - 2014-02-15 09:29:07 UTC
Atum wrote:
Daedalus III wrote:
I'm wondering; if parts of the database has to be created each time the server is started, won't this risk extending the DT which you have worked so hard on reducing?

Not really... they'll only need to regenerate the map db file when something actually changes, which probably isn't all that often. Do the regen on a patch day, and nobody would know the difference.


Exactly. It sounds like all they are doing is moving from the Current System:
Manually Input Data --> Populate Database that doesn't change
where if you need changes, you change the database directly

to the New System:
Manually modify config files in Source Control System --> Programmatically populate database from Configuration files
where if you need changes, you change the config files and rebuild the database tables

This just lets you put the source files in source control, allows you to rebuild the DB from scratch if you need to (as opposed to just rebuilding from a dump of the database), and lets you generate other forms for the data to be used, which is probably the hint we saw about changes to the client above. Wouldn't be surprised if they are using the SQL lite files in the client vs. however they did it before (but thats just a complete guess).

CCP: 10+ years of Harvesting players Tears  (latest efforts being Source Limited Edition, and Alliance Logo Revised Policies)
JP Nakamura
Union of Intergalactic Miners and Nano Assemblers
#35 - 2014-02-15 09:35:46 UTC
...
CCP Nobody wrote:

JP Nakamura wrote:
Yes/No ... in this case their just mucking about with the Server, not the Client, so its only HALF as likely something will go wrong

Actually this has quite the change for clients as well :) But hopefully nobody will notice anything different, except the starmap being a bit faster...


I stand corrected from someone who certainly should know better. Smile

If you can talk about it, is the Client using the SQL Lite files in place of the caching mechanism it was using before (no idea what)?
Is that why CCP is moving to providing dumps in this format, since getting the data to the client in this format already needs to happen, so giving it to us is "free" (so to speak)?

(Just curious as a programmer ... we always seem to be curious how the Universe works ... even when its a virtual Universe P)

CCP: 10+ years of Harvesting players Tears  (latest efforts being Source Limited Edition, and Alliance Logo Revised Policies)
CCP Nobody
C C P
C C P Alliance
#36 - 2014-02-15 10:44:13 UTC
JP Nakamura wrote:

I stand corrected from someone who certainly should know better. Smile

If you can talk about it, is the Client using the SQL Lite files in place of the caching mechanism it was using before (no idea what)?
Is that why CCP is moving to providing dumps in this format, since getting the data to the client in this format already needs to happen, so giving it to us is "free" (so to speak)?

(Just curious as a programmer ... we always seem to be curious how the Universe works ... even when its a virtual Universe P)

We use a wide array of files for the Client (and Server), depending on what we need at the time (f.ex cpickles and homebrewed binary format).
And with our build system we can pretty easily create additional files for the static data export, that will get built every time someone changes the underlying data. So there is actually a tiny bit of work that goes into generating the build script for the SDE files, but once that is done, it is all autogenerated.

We decided to provide this dump in sqlite format just because we thought that it would be easy for the 3rd party dev community to utilize that in clever ways, since you guys already use sqlite.

Steve Ronuken wrote:

Adding celestials, removing celestials. Changing the links between systems. Adding systems (well, with other changes too.)

This will have more effect on the development process, than on actual deployments, as it'll let them handle the changes in a properly staged fashion, with the changes in data being very easy to document.

Change management in databases is a pain in the ass.

You sir, certanly know your stuff Smile
Har Harrison
Garoun Investment Bank
Gallente Federation
#37 - 2014-02-15 12:57:23 UTC
CCP Xhagen wrote:
Benny Ohu wrote:
CCP Xhagen wrote:
They should only be off by a meter or two.

not just one metre but possibly two??

D: gearing up to shot statue

Only if you can find it again!

It should only be a maximum of a meter or two away from where it was last time...
Chirjo Durruti
Doomheim
#38 - 2014-02-15 13:07:22 UTC
Now if you could change the DATADUMP as well from MSSQL to sth more accessible on non-Windows operating systems, i would be very grateful. Playing on Windows - but i develop on Linux.

HOWTO: No More Tears (solo) http://www.youtube.com/watch?v=BdA4ciUrH-k If you can get me a better crew than THIS: http://www.youtube.com/watch?v=yPrtQ9AdoM0 convo me.
Har Harrison
Garoun Investment Bank
Gallente Federation
#39 - 2014-02-15 13:13:25 UTC
CCP Nobody wrote:
JP Nakamura wrote:

I stand corrected from someone who certainly should know better. Smile

If you can talk about it, is the Client using the SQL Lite files in place of the caching mechanism it was using before (no idea what)?
Is that why CCP is moving to providing dumps in this format, since getting the data to the client in this format already needs to happen, so giving it to us is "free" (so to speak)?

(Just curious as a programmer ... we always seem to be curious how the Universe works ... even when its a virtual Universe P)

We use a wide array of files for the Client (and Server), depending on what we need at the time (f.ex cpickles and homebrewed binary format).
And with our build system we can pretty easily create additional files for the static data export, that will get built every time someone changes the underlying data. So there is actually a tiny bit of work that goes into generating the build script for the SDE files, but once that is done, it is all autogenerated.

We decided to provide this dump in sqlite format just because we thought that it would be easy for the 3rd party dev community to utilize that in clever ways, since you guys already use sqlite.

Steve Ronuken wrote:

Adding celestials, removing celestials. Changing the links between systems. Adding systems (well, with other changes too.)

This will have more effect on the development process, than on actual deployments, as it'll let them handle the changes in a properly staged fashion, with the changes in data being very easy to document.

Change management in databases is a pain in the ass.

You sir, certanly know your stuff Smile

Yes and no - have a table with a version number in it and it isn't hard to work out what statements need to be executed to upgrade/downgrade a database (DDL). That's what entity framework does (POCO first).

However, this isn't a database change management issue so much as it is a config file management one. People have had XML, csv etc... files checked into source control for ages. Difference here is that the files need to be in a database instead of on a file system.
Salpun
Global Telstar Federation Offices
Masters of Flying Objects
#40 - 2014-02-15 13:51:44 UTC  |  Edited by: Salpun
Noticed a change in the training queue all partial skills are yellow now, all the time, even if that sorting option is off.

Is this an intended change without patch notes its hard to tell?

I would keep the change, I like it but it does jump out at you if your not expecting it, but if it was not on purpose its a bug which might have other ramifications.

In game bug report not written. I'll submit an out of game one if needed.

If i dont know something about EVE. I check https://wiki.eveonline.com/en/wiki/ISK_The_Guide

See you around the universe.
Previous page123Next page