These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

lpDatabase v0.11 - SQL release of in-game LP Stores (Kronos 1.0)

First post
Author
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#21 - 2013-02-19 02:42:15 UTC  |  Edited by: Sable Blitzmann
Update: Trying to figure out which corps are more popular so as to have a more focused verification process. I will be verifying the 4 navies next, and Lai Dai was also suggested for the research agents. In the mean time, I have been working on my own ISK/LP conversion site. \o/
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#22 - 2013-02-19 04:31:26 UTC
Sable Blitzmann wrote:
I have been working on my own ISK/LP conversion site. \o/


\o/

Always worth having more than version kicking around. And this way, I don't need to feel guilty for anything. Big smile

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#23 - 2013-02-19 18:12:36 UTC  |  Edited by: Sable Blitzmann
Steve Ronuken wrote:
Sable Blitzmann wrote:
I have been working on my own ISK/LP conversion site. \o/


\o/

Always worth having more than version kicking around. And this way, I don't need to feel guilty for anything. Big smile


Oh good, so I won't have to feel guilty for using your EMDR script to fuel it either. =P Credit where credit it due tho, thanks for introducing me to EMDR. This thing kicks ass!

On to the topic at hand, I was asking around in the missioning forum which corps are popular so that they can be verified, where one pilot (Carniflex) said that each faction basically has 3 different types of stores: military, industrial, and exploration, and that the corps that use each store should be more or less the same. I tested this out by showing the four main factions and their corps, along with the number of LP Store offers in each corp. I figure if a corp has the same number of LP offers, then it's a good chance they share the same LP Store. This is fairly useful information and does show a lot of similarities, however it's not as cut and dry as "3 variations per faction".

I will, however, be using this as a guide. What this means is that I will use a verified corp to verify another corp without visiting the store ingame. This will cut down the time it takes to verify as I can just tab between two tabs in quick succession to find any discrepancies between two like corps. For example, I've already verified Modern Finances, and I see that Caldari Business Tribunal, Expert Distribution, Prompt Delivery, etc have similar stores (145 items each). I can use Modern Finances as a template for the verification of those corps. However, this of course opens up the possibility of errors as they will not be completely verified, only verified by relation. Such errors include missing items, added items, incorrect isk/lp cost or req items... However, from what I've seen so far with verification, lp/isk cost remains stable between the corps, as does req items, so the chance of an LP store differentiating is slim.

I will also be adding a column in `lpVerified` denoting how the corp was verified: 1 will be manual verification and 2 will be verification by relation to a like corp.
Feanos
Brutor Tribe
Minmatar Republic
#24 - 2013-02-20 11:07:39 UTC
So, as someone who was doing this before UK left FW, mostly for grins, if you want, I've got a copy of my internal DB with the information for the various corpID's:
sqlite> select distinct(corpID) from corpOffers;
1000002
1000010
1000019
1000033
1000049
1000053
1000054
1000055
1000057
1000059
1000062
1000067
1000072
1000074
1000079
1000082
1000083
1000086
1000093
1000094
1000100
1000104
1000130
1000160
1000165
1000172
1000179
1000182

There's, in general, three main LP store types for the various factions, from my research/testing:
FW - Mostly military/Datacores
Combat - This is your standard L4 combat bread/butter
R&D - These give those nice little datachips for making ships that everyone likes

If you're interested in the data I farmed, it's SQLite, with the following formatting:
sqlite> .schema
CREATE TABLE corpOffers(
corpID INTEGER NOT NULL,
offerID INTEGER NOT NULL);
CREATE TABLE offers(
offerID INTEGER NOT NULL,
typeID INTEGER NOT NULL,
iskCost INTEGER NOT NULL,
lpCost INTEGER NOT NULL,
reqItems TEXT,
qty INTEGER NOT NULL,
PRIMARY KEY(offerID),
UNIQUE(offerID));

Located at: http://snipanet.com/lpdb.db

Offers are unique to what they give/etc, so you end up with a relatively small offers table, with the corpOffers pushing data into this fairly readily, also makes it really easy to look up with some relatively clean joins. I'll see about dumping more corp data into this when I get a chance. There's been a cloaky in my system all day, so I'll get an alt busy soonish :D
Ydnari
Estrale Frontiers
#25 - 2013-02-20 23:31:16 UTC
a quick one: some duplicates snuck back in

delete from lpStore where storeID in (6827, 6826, 28784);

consider:

select s.corporationID, s.typeID, s.quantity, s.lpCost, s.iskCost, group_concat(i.typeID), group_concat(distinct s.storeID)
from lpstore.lpStore s
left outer join lpstore.lpRequiredItems i on (s.storeID = i.parentID)
group by s.corporationID, s.typeID, s.quantity, s.lpCost, s.iskCost
having count(distinct s.storeID) > 1

which catches any exact duplicates

--

Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#26 - 2013-02-21 00:20:33 UTC
Ydnari wrote:
a quick one: some duplicates snuck back in

delete from lpStore where storeID in (6827, 6826, 28784);

consider:

select s.corporationID, s.typeID, s.quantity, s.lpCost, s.iskCost, group_concat(i.typeID), group_concat(distinct s.storeID)
from lpstore.lpStore s
left outer join lpstore.lpRequiredItems i on (s.storeID = i.parentID)
group by s.corporationID, s.typeID, s.quantity, s.lpCost, s.iskCost
having count(distinct s.storeID) > 1

which catches any exact duplicates


Thanks!

I already found and fixed the Quafe and Senate duplicates (both had 2 Fed Navy 100MN MWD and were missing a 10MN MWD). Thank you for letting me know about the CONCORD duplicate; just goes to show even verified corps have issues x_x. I've deleted the duplicate, and will most likely release v0.4 tonight/tomorrow with over 100 corps vverified (using the new template method described above)

Awesome query as well. Will definitely save it and go back to it regularly.
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#27 - 2013-02-21 01:02:02 UTC  |  Edited by: Sable Blitzmann
Updated OP with new version (v0.4).

Also, here's a nifty image: http://eve-files.com/dl/260764 the 4 main factions and their verification progress. Green is verified, purplish is verified with template. This does not include smaller factions (Mordu's Legion, etc) because I'm lazy.

Verified:
- Brutor Tribe
- Joint Harvesting
- Theology Council
- Six Kin Development
- Lai Dai Protection Service
- School of Applied Knowledge

Verified via Relation:
- CBD Corporation
- Prompt Delivery
- Hyasyoda Corporation
- Deep Core Mining Inc.
- Poksu Mineral Group
- Minedrill
- Caldari Provisions
- Kaalakiota Corporation
- Wiyrkomi Corporation
- Top Down
- Rapid Assembly
- Perkone
- Caldari Steel
- Zainou
- Nugoeihuvi Corporation
- Echelon Entertainment
- Ishukone Corporation
- Zero-G Research Firm
- Propel Dynamics
- Expert Distribution
- CBD Sell Division
- Sukuuvestaa Corporation
- Caldari Constructions
- Expert Housing
- Caldari Funds Unlimited
- State and Region Bank
- Chief Executive Panel
- Mercantile Club
- Caldari Business Tribunal
- House of Records
- Science and Trade Institute
- Sebiestor Tribe
- Republic Fleet
- Republic Justice Department
- Republic Security Services
- Minmatar Mining Corporation
- Core Complexion Inc.
- Boundless Creation
- Eifyr and Co.
- Native Freshfood
- Carthum Conglomerate
- Imperial Armaments
- Viziam
- Zoar and Sons
- Noble Appliances
- Ducia Foundry
- HZO Refinery
- Inherent Implants
- Amarr Certified News
- Nurtura
- Further Foodstuffs
- Royal Amarr Institute
- Imperial Chancellor
- Amarr Civil Service
- Ministry of Assessment
- Amarr Trade Registry
- Court Chamberlain
- Emperor Family
- Kador Family
- Kor-Azor Family
- Ardishapur Family
- Civic Court
- TransStellar Shipping
- Federal Freight
- Inner Zone Shipping
- Material Acquisition
- Astral Mining Inc.
- Combined Harvest
- Quafe Company
- CreoDron
- Roden Shipyards
- Allotek Industries
- Poteque Pharmaceuticals
- Impetus
- Egonics Inc.
- The Scope
- Aliastra
- Bank of Luminaire
- Garoun Investment Bank
- President
- Senate
- Supreme Court
- Federal Administration
- Hedion University
- Center for Advanced Studies
- Republic Military School
- Republic University
Ydnari
Estrale Frontiers
#28 - 2013-02-21 02:33:34 UTC
I've gone through the data and the store in-game for Federation Navy and found no errors or omissions.

This should verify Federation Intelligence Office and Federation Customs as well; I use those stores a lot and have never noticed any difference.

Can't speak for Federation Navy Academy, it's probably the same but haven't checked, I haven't got any LP with them.

--

Skalarik
Republic Military School
Minmatar Republic
#29 - 2013-02-21 14:28:50 UTC
Hi, nice job you've done here.
For your next db-dump, would it be possible to limit the 'INSERT INTO ...' statements to insert max. 1000 rows per statement? I work with MsSql and Oracle databases (XE versions) and these can't take more than 1000...if not, well i'll just continue to insert them manually.
Tonto Auri
Vhero' Multipurpose Corp
#30 - 2013-02-21 17:45:12 UTC
Even a thousand rows is a big number. Better split them in packs of 50-100 rows, with intemittent commits.

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison

Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#31 - 2013-02-21 19:41:46 UTC
Skalarik wrote:
Hi, nice job you've done here.
For your next db-dump, would it be possible to limit the 'INSERT INTO ...' statements to insert max. 1000 rows per statement? I work with MsSql and Oracle databases (XE versions) and these can't take more than 1000...if not, well i'll just continue to insert them manually.


Will do. =)
Neugeniko
Insight Securities
#32 - 2013-03-04 15:02:06 UTC  |  Edited by: Neugeniko
This is excellent! A little while back I was looking for this data to work out isk/lp. It seems like a few people are working on this already, so ill do what I can to help with the core data. Torrinos system is pretty popular for mission runners seeking a caldari 0.5 system with a few lvl 4 agents. Homegaurd and Spacelane patrol LP stores are there, ill compare these to the data up on fuzzworks and see if its accurate.

Neug
Neugeniko
Insight Securities
#33 - 2013-03-05 21:35:43 UTC
Compared the data on Fuzzworks to actual LP Store for Home Gaurd.

Omissions
=========
[1 x Zainou 'Snapshot' Rockets RD-905],[79,375 LP],[79,375,000 ISK]

[Caldari Navy Dual 150mm Railgun],[24,000 LP],[9,600,000 ISK]
1 x Dual 150mm Railgun I
63 x Federation Navy Fleet Captain Insignia I
106 x Federation Navy Fleet Major Insignia I

[1 x Zainou 'Gypsy' Target Painting TG-905],[79,375 LP],[79,375,000 ISK]

[1 x Zainou 'Gypsy' Target Painting TG-903],[10,875 LP],[10,875,000 ISK]

[1 x Zainou 'Gypsy' Target Painting TG-901],[375 LP],[375,000 ISK]

Taking these into consideration I also checked Spacelane Patrol, LP store is exactly the same as Home Gaurd.

Hope this helps.

Neug

DaOpa
Static Corp
#34 - 2013-04-05 18:41:24 UTC


Hundreds of hours of personal work to generate LP Store DB, only to be undone by a scraper and others who are republishing scraped data .....


I'm lost for words....

Cry
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#35 - 2013-04-06 00:34:52 UTC
DaOpa wrote:


Hundreds of hours of personal work to generate LP Store DB, only to be undone by a scraper and others who are republishing scraped data .....


I'm lost for words....

Cry


I appreciate the work you put into Ellatha, and it had been a great resource, however I'm not gonna mince words: it is fairly outdated with errors and, the biggest of all, proprietary which makes it useless for anything but simple lookups. I needed something tangible to manipulate, and I assume Zanto Snix did as well. Sorry if you feel slighted. =(

In other news, I took a smalk break from EVE, and thus from this project, which explains lack of posts. But I have verified a few more corps and will be releasing the next version sometime soon. \o/ progress
Tonto Auri
Vhero' Multipurpose Corp
#36 - 2013-04-06 12:01:36 UTC
Sable Blitzmann wrote:
DaOpa wrote:


Hundreds of hours of personal work to generate LP Store DB, only to be undone by a scraper and others who are republishing scraped data .....


I'm lost for words....

Cry


I appreciate the work you put into Ellatha, and it had been a great resource, however I'm not gonna mince words: it is fairly outdated with errors and, the biggest of all, proprietary which makes it useless for anything but simple lookups.

Even for simple lookups, it was hardly useful. My every attempt to narrow search to at least a single page of results failed in the past.

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison

Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#37 - 2013-04-14 23:43:59 UTC
v0.5 is out, OP updated. This release took much longer than I had hoped, but as previously stated I took about a month long break from EVE. 85% of the corps are now verified, the remaining ones are mostly the small factions.

Achievements:
- Minmatar faction completely verified.
- SOE verified (since a lot of missioners like SOE, figured this was a priority)
- Got a couple nullsec factions verified without my alts dying in their shuttle! \o/

v0.5 - More verifications. Nothing new.
Added:
- MP-805 added to Caldari Navy store
- Caldari Navy Dual 150mm Railgun (typeID = 15815) added to some Caldari corps
- Federation Customs seems to be the same LP store as the fed navy, but was missing a chunk of items (BY-810, SE-605, EG-605, EE-605, PJ-805, SA-705). Added these items after verifying they do indeed exist in LP store (then verified LP store with relation to Fed Navy)
- Ocular Filter - Standard/Improved to Intaki Commerce and Bank
- WD-901 to Amarr Navy
Fixed:
- Mega Navy Issue for non-FW stores (storeID: 151, 153, 157, 156), changed iskCost (20000000 -> 200000000)
- storeID: 9389 & 9390 (iskCost: 16200, lpCost: 10800000) (verified with Trust Partners, not Thukker Mix)
- typeID: 28817 (iskCost 13400000 -> 113400000) for Intaki faction
- typeID: 28819 (iskCost 14200000 -> 214200000) for Intaki faction
- Required typeID: 16183 (set all quantities to 5 instead of mix of 1 & 5)
- Large Blaster/Rail Spec lpCost (3000 -> 9000) for Intaki faction
Verified:
- Caldari Navy
- Federation Navy
- Duvolle Laboratories
- Sisters of EVE
- Pator Tech School
- Trust Partners
- Intaki Bank
- Outer Ring Excavations
- Imperial Academy
- Intaki Syndicate
Verified via Relation:
- Spacelane Patrol
- State War Academy
- Wiyrkomi Peace Corps
- Peace and Order Unit
- Internal Security
- Ishukone Watch
- Home Guard
- Corporate Police Force
- Federal Navy Academy
- Federal Intelligence Office
- Federation Customs
- The Sanctuary
- Food Relief
- Vherokior Tribe
- Urban Management
- The Leisure Group
- Republic Parliament
- Krusual Tribe
- Freedom Extension
- Intaki Commerce
- Sarum Family
- Ministry of Internal Order
- Ministry of War
- Amarr Navy
- Intaki Space Police
Uppsy Daisy
State War Academy
Caldari State
#38 - 2013-04-15 22:17:38 UTC
Now that is dedication.. I salute you/
Jess Technite
Almost Absolute
#39 - 2013-04-25 00:42:34 UTC  |  Edited by: Jess Technite


That seems nice, but I use PostgreSQL and that scripts aren't possible :P (or I don't know how to do it). I have coded this for VB.NET 2010, as the program that I'm doing (not only LP Store, also T1 / T2 manufacturing) is in that platform. Should also work for older versions:

http://pastebin.com/raw.php?i=3FhJhJZT

Change the field names to whatever you want (are easily understandable) and the result will be 1584 distinct offers, much better that almost 30.000. The process that iterates all the original table have a duration of 45 minutes :S

EDIT: Now only have a duration of 2 minutes \o/
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#40 - 2013-04-27 17:32:22 UTC  |  Edited by: Sable Blitzmann
Just a heads up, the database schema will most likely be changing with the next release. I'm gonna base it mostly on Ydnari's approach (https://forums.eveonline.com/default.aspx?g=posts&m=2546104#post2546104). Since a lot of the data in the database is redundant, I aim to slim it up a bit.

I'm doing this because my lpStore application (which I've been procrastinating with releasing, ugh) would benefit from it. Let me know if there's any concerns with this new approach

Jess Technite wrote:
The process that iterates all the original table have a duration of 45 minutes :S


Really? Yikes! x_x