EVE Technology Lab

Hello thar fellow internet spaceship pilot!

I have put together a database that represents information regarding the various LP Stores in-game. This is an ongoing project to verify the accuracy of the data and keep it up-to-date with the release of expansions.

Initial data was based on a CSV file provided by Zanto Snix, which was apparently a web scrape of DaOpa's Ellatha LP store. As both of these were severely outdated at the time with many errors and typos, much of the information needed to be fixed. Additionally, due to both Ellatha and chruker.dk (who owns this?) being proprietary with the database not released to the community, I decided to make this available to everyone. =D

tl;dr:
lpDatabase.zip v0.11 (sql, json)

History

Recently, I needed to be able to take LP Store offers and manipulate the data for a project I'm working on. So I went on a trek to find the database somewhere. CCP doesn't include this information in their dump, so I figured surely someone, somewhere would have made one and posted a SQL file. Alas, turns out I was wrong (or maybe my GoogleFu was not up to par). The only hits I got back were the likes of the LP Store Database (http://www.ellatha.com/eve/LP_Stores.asp) and, indirectly, http://games.chruker.dk/eve_online/default.php (I didn't find an LP store, but most LP items show their requirements on their respective item page)

The problems with these sites was that they were not available in SQL format, and attempts to contact them for it did not work. I cannot work with webpage data - I need it in cold, hard SQL!

I finally came across usable data, thanks to Zanto Snix and his Google Doc (link). It looked like a scrape from Ellatha's LP Store site, and seemed to be more or less complete. And I can download to a .csv file! Now I have real data, and a real file format to work with. But it can't end there

The Google Doc (and indeed, most LP stores out there) was fairly outdated. Some of the LP store offers did not have typeIDs, you couldn't find some of the required items in the database, and then there were just plain typos and duplicates. The majority of the issues stems from the recent module name changes.

Anyway, to correct these issues I set up a web app that would parse the CSV file and, if it ran into an issue (couldn't find item in database, possible typos/bad format, etc), it would let me know and present a form to correct it. Worked out pretty well, but still took quite a while. After I finally finished with that part, I tried to cross reference all LP store offers with chruker.dk/eve_online (contains outdated item names, however typeIDs were still valid. Some items aren't represented as LP offers on that site tho) and then remove duplicate entries.

Once that was done, it was a matter of parsing the file into MySQL. I'm sure there's some of you that would appreciate this data, so I'm making it available.

However, I need help in 1) verifying the information is correct, and 2) keeping the database up to date. It's not often CCP adds or removes items in the database, however it could happen. If they change the name, it'll still be up to date as I use the typeIDs, however if they change the actual items or quantity, it'll need to be updated. So if you notice wrong information or items that need adding, post them here (with proof (image) preferably) and I'll correct it asap and post a new file.

Use
These projects are using lpDatabase. Let me know if you make / find one and want it added to the list! Currently all sites use the data to present it and calculate isk/lp in some way.

fuzzwork/lpstore - the first usage of the database, released just days after. Uses EMDR for pricing information.
blitzmann/lpStore - my own version, and the project that make me want to gather data for a database in the first place. Open source.
noonecares/LpStore - ialocin's version, using EVE Central's API.

I just got me a copy of it.
Hmmmm, I'm really tempted into adding an LP store browser in EVEMon. I'll scan the data and see what's possible.

Maybe I'll even release an MSSQL version.

Thanks anyways for the work done.

Great data, thanks for making it available.

I've played with it a bit, and worked it together with the pricing data I'm gathering (I think I need to work on the caching a little, to reduce the load and speed it up. But that's all behind the scenes stuff) to produce a calculator for working out what is worth buying. It falls down on the blueprints and similar items which aren't on the market. Or have components which aren't on the market.

http://www.fuzzwork.co.uk/lpstore/

Just saw your post on Reddit. Wonderful work! I replied with a few questions of my own, I'd be interested in hearing your reply.

Thank you for reporting on some inaccuracies. I'll look into them and release a point release soon. =)

Thanks for the info!

Yeah, implants are a mess. I've already fixed the LE-605, along with a few other things. My alt is sitting in Jita, I'll verify the CN Stasis Webifier bit and fix it

Your way of storing the data is interesting. I was going to lay out the data the way (or similar) however I don't know just how uniform all the LP store offers are across the board. For example, customs offic gantry blueprints are 3,000/10,000,000 at the 4 FW LP Stores, but double the price at CONCORD (and that's confirmed). So I'm reluctant to group offers together like that until they can be verified. Then again, CONCORD (and the FW) stores are kinda unique; I'm sure the vast majority of offers are duplicates.

I would be interested in that script if you could post it. I may be able to use it.

Also, what do you mean by this:

I thought I removed all the duplicates. =(

Ah, thanks. I'll definitely look into it. =)

I wonder if the LP stores is somewhere in the cache/client files? Would make it so much easier to validate...
I would rather write a tool to do the boring work, then do it by hand... not that I'm volunteering or anything :P

I do not think the info is available in the Cache, otherwise someone would have done something like this by now. I agree, it would make it so much easier. If someone who is familiar with the cache/client files wants to look into this or provide a little more insight, that's be great.



Thanks for the help! I've set up a quick web app that helps with the process. I check each item in the in-game store with the web app, and if it's the same I can just remove it from the DOM. If it needs to be changed, it presents the store ID and stuff. I also have a query running on each item that checks the other costs and if there's differences it tells me so I can investigate. So it helps a bit.

If you can be a bit more accurate or detailed with your notes (such as every change you make with IDs and stuff) that'd be much easier to follow.

I'll be putting up updated dumps tonight.

I was meaning copy the web page (from mine. guess I should have been clearer. Or posted in my own thread. oops.) into a spreadsheet. because from there, you can make notes more easily.


I've added an extra table just to maintain a list of which stores I've gone through and checked.. But that's not an important one.

Anyway, some sql. Those three stores looked pretty close to right. corporationids of: 1000180, 1000077, 1000030.

Don't just cut and paste it. The lpRequiredItems insert depends on the id inserted in the row before. It's because of the missing Caldari heavy assault missile launcher from the modern finances store.

update lpStore set lpcost=1850,iskcost=1850000 where typeid=27315;
insert into lpStore (corporationid,typeid,quantity,lpcost,iskcost) values (1000030,28377,1,24000,9600000);
insert into lpRequiredItems (parentid,typeid,quantity) select 28779,typeid,quantity from lpRequiredItems where parentid=8791;
update lpRequiredItems set typeID=15594 where parentid=5525 and quantity=21;
update lpRequiredItems settypeid=17244 where parentid=6228;
update lpRequiredItems set typeid=17244 where parentid=6228;
update lpStore set typeid=24641,iskcost=10875000,lpcost=10875 where id=3762;

Uploaded v0.3. Biggest thing here is that all 4 FW militia stores are now verified

Verified in v0.3:
- Ytiri
- State Protectorate
- FedMart
- Chemal Tech
- 24th Imperial Crusade
- Federal Defense Union
- Tribal Liberation Force