These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

lpDatabase v0.11 - SQL release of in-game LP Stores (Kronos 1.0)

First post
Author
Previous Topic Next Topic
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#61 - 2013-11-24 03:23:34 UTC
http://eve-files.com/dl/266076

- added json directory. Every corp has it's own json file. Every offer is in the following format:

offerID:
-- typeID
-- qty
-- lpCost
-- iskCost
-- reqItems: (if applicable)
---- typeID : qty

Also, I've update the SQL dump to also include a comment on the lpStore table that represents the version of the data. You can all this via

SELECT table_comment
FROM INFORMATION_SCHEMA.TABLES
WHERE table_schema = :dbName
AND table_name = lpStore
Drailen
Doomheim
#62 - 2013-11-24 18:25:02 UTC
JSON format works like a charm! Thanks for the quick update.
Taal Khurin
Acorn Harvesting
#63 - 2013-11-28 00:07:33 UTC
You completely ripped the http://www.ellatha.com/eve/LP_Stores.asp sites data that someone spent ages putting together and are now touting it as your own?

For shame!
Mikokoel
Mining Industry Exile Foundation
Synergy of Steel
#64 - 2013-11-28 11:54:38 UTC
You fail to see that those LP websites from ellatha and chruker are outdated and that accessing their information needs complicated HTML scraping?

He offers a well thought out database with up-to-date information, in various formats, and you think it's a rip-off?

For shame!

Mikokoel | Head FC League of Unaligned Master Pilots
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#65 - 2013-11-28 12:25:00 UTC
Taal Khurin wrote:
You completely ripped the http://www.ellatha.com/eve/LP_Stores.asp sites data that someone spent ages putting together and are now touting it as your own?

For shame!



You failed to see that the source of the information is all mentioned in the first post?

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
TestBox
Center for Advanced Studies
Gallente Federation
#66 - 2013-11-28 16:34:03 UTC
Mikokoel wrote:
You fail to see that those LP websites from ellatha and chruker are outdated and that accessing their information needs complicated HTML scraping?

He offers a well thought out database with up-to-date information, in various formats, and you think it's a rip-off?

For shame!


So lets use this in a different example -


1. Use a Robot to Scrap all of "http://eveboard.com/" characters (Chribba's awesome site).

2. Dump it into a excel spreadsheet and post it here.

3. Someone else converts that spreadsheet into SQL and other forms. find errors in some data and fixes it.

4. Makes a new thread saying Look I have awesome Character data for all you guys to use in various formats ..

5. Make a sight mention the data is from Chribba's site ...

You find this alright for people to do?
Taal Khurin
Acorn Harvesting
#67 - 2013-11-28 23:10:17 UTC
Mikokoel wrote:
You fail to see that those LP websites from ellatha and chruker are outdated and that accessing their information needs complicated HTML scraping?

He offers a well thought out database with up-to-date information, in various formats, and you think it's a rip-off?

For shame!


I know the person that compiled all the data personally and they were in the process of updating it when they noticed your thread.
They are still very active and were very miffed about it i can tell you.
Elmore Jones
New Eden Mining Organisation
The Craftsmen
#68 - 2013-11-29 09:18:33 UTC
While I can understand that finding you have a rival project might be off putting, this kind of rage posting isn't helping. They're both working on the same information from the same game. It's going to be a very similar end result which in no way implies bot scraping from another persons effort.

+++ Reality Error 404 - Reboot Cosmos +++
Leokokim
Mining Industry Exile Foundation
Synergy of Steel
#69 - 2013-11-29 14:18:38 UTC
TestBox wrote:

5. Make a sight mention the data is from Chribba's site ...

You find this alright for people to do?


You mean like here where it is in the second paragraph of the first page? (right below the TL:DR)
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#70 - 2013-11-29 23:56:12 UTC
Just to clear a few things up...

I've made several mentions, in the original post, to where most of this data came from, and various method I used to obtain it. I never once claimed that I was the one that gathered it all. However, I believe it's a bit unfair to portray me as someone who is profiting off of someone else's work.

- The original data was extremely out of date. I wish I kept a count on the inaccuracies I found when I was preparing for initial release. It was mind boggling, we're talking a few expansions old. Especially the whole module renaming phase; oh god that was horrible trying to investigate old names and finding their correct typeID. I'm glad to hear that the person has since made an effort to update their own database.

But I have spent hours upon hours going through and fixing inaccuracies. I have spent hours flying from station to station to personally, and manually verify most of the corps as everything was outdated. I don't feel as if I simply scraped the data, packaged it up in a neat wrapper and stuck my name on it with credits in fine print. Kudos to those that took the time to gather all the initial data and put it up. Unfortunately, stuff like that rots, and I needed up to date information.

- It was proprietary. IE: absolutely useless for anything. I needed a format that I could manipulate. So I created one. I could have kept it to myself, as the other LP stores out there have done, but I decided to release it to the greater EVE community. Sue me.

Again, kudos to those that took the time to gather initial data. But it was way out of date and proprietary. Anyone is welcome to contribute to this project if they wish so that the whole eve community can benefit.
Taal Khurin
Acorn Harvesting
#71 - 2013-11-30 01:09:55 UTC
I guess my only real gripe that there was no real appreciation for the original developers of the data.

I'm sure if a certain person whoses name began with "D" would get a mention he would be ok :)
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#72 - 2013-11-30 01:32:09 UTC  |  Edited by: Sable Blitzmann
Then DaOpa should speak up for himself. He's already commented in this thread and gave basically the same response you and TestBox did. I'd rather not have this thread derailed by people who think I slighted someone's work when they themselves don't bring up the issue.

That being said, I will revise the original post to better reflect that this is a derivative work, as looking over it for the first time since I posted it it mentions it somewhat in passing, under a history heading. This can be improved. =)
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#73 - 2013-11-30 01:56:40 UTC
And now, back to the discussion at hand...

I'm currently looking into using the LogServer to perhaps validate data and keep this thing up to date. If you open the LP Store with the log server running, it starts to print out pretty much everything you need. However, it seems to truncate after a specific size of information has printed to the LogServer.

Here's a sample of the data that pops up while accessing the LP Store of University of Caille:

http://pastebin.com/MZxQ4m9D

I'm not familiar with the LogServer at all, and I'm not sure if all information is sent to the logserver which then truncates the info before printing, or if the info is first truncated by the client then sent to the LogServer. I do know, tho, that there are ways, or used to be ways, to hook into the log server and read info as it's printed (see: the now defunct BACON software for local monitoring). And, as far as I remember, that software was not against the EULA (this may or may have not changed, I have no idea).

If anyone has any experience with this or with the LogServer, please let me know. If some software is developed that can achieve this, I can then make it a quick trek to visit the stations that need visiting and just dump the data.

Cheers!
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#74 - 2014-01-21 15:39:11 UTC
No update, just giving this topic a little activity so it doesn't auto-lock.

Using the LogServer was a dead end, unfortunately. Oh well.
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#75 - 2014-01-27 22:16:11 UTC
v0.9 - Rubicon 1.1
Added:
- Nestor: 1,000,000LP/100,000,000ISK for SoE corp, and 800,000LP/80,000,000ISK for The Sanctuary
- Nestor BPC: 600,000LP/150,000,000ISK for SoE corp, and 400,000LP/100,000,000ISK for The Sanctuary

Can't upload to eve-files currently, so this is being hosted on my server.
Taleden
North Wind Local no. 612
#76 - 2014-02-19 21:38:12 UTC
Thanks for making this data so much easier to use. I've just started pulling it into my own projects, and in case anyone is curious (or feels inclined to spot check it), here's the current clustering of corporations who share exactly the same set of offers:

Quote:

Royal Khanid Navy
Freedom Extension
Krusual Tribe
Pator Tech School
Republic Parliament
The Leisure Group
Urban Management
Vherokior Tribe
Intaki Bank
Intaki Commerce
CBD Corporation
CBD Sell Division
Caldari Business Tribunal
Caldari Funds Unlimited
Chief Executive Panel
Echelon Entertainment
Expert Distribution
Expert Housing
House of Records
Mercantile Club
Modern Finances
Nugoeihuvi Corporation
Prompt Delivery
Science and Trade Institute
State and Region Bank
Ytiri
True Creations
CONCORD
Ardishapur Family
Carthum Conglomerate
Ducia Foundry
Further Foodstuffs
HZO Refinery
Imperial Armaments
Inherent Implants
Joint Harvesting
Noble Appliances
Nurtura
Royal Amarr Institute
Viziam
Zoar and Sons
Pend Insurance
Federal Defense Union
Ammatar Fleet
Caldari Navy
Corporate Police Force
Home Guard
Internal Security
Ishukone Watch
Peace and Order Unit
Spacelane Patrol
State War Academy
Wiyrkomi Peace Corps
Nefantar Miner Association
Imperial Shipment
Khanid Transport
The Sanctuary
Ammatar Consulate
True Power
Thukker Mix
Aliastra
Bank of Luminaire
Egonics Inc.
FedMart
Federal Administration
Federal Freight
Garoun Investment Bank
Impetus
Inner Zone Shipping
President
Quafe Company
Senate
Supreme Court
The Scope
TransStellar Shipping
University of Caille
Food Relief
Guristas
Archangels
Dominations
Guardian Angels
Brutor Tribe
Republic Fleet
Republic Justice Department
Republic Military School
Republic Security Services
Intaki Space Police
Intaki Syndicate
Sisters of EVE
Serpentis Corporation
Lai Dai Corporation
Khanid Innovation
Khanid Works
24th Imperial Crusade
Federal Intelligence Office
Federal Navy Academy
Federation Customs
Federation Navy
Tash-Murkon Family
Allotek Industries
Astral Mining Inc.
Center for Advanced Studies
Chemal Tech
Combined Harvest
CreoDron
Material Acquisition
Poteque Pharmaceuticals
Roden Shipyards
Amarr Constructions
Boundless Creation
Core Complexion Inc.
Eifyr and Co.
Minmatar Mining Corporation
Native Freshfood
Republic University
Sebiestor Tribe
Six Kin Development
State Protectorate
Outer Ring Excavations
Amarr Certified News
Amarr Civil Service
Amarr Trade Registry
Civic Court
Court Chamberlain
Emperor Family
Hedion University
Imperial Chancellor
Kador Family
Kor-Azor Family
Ministry of Assessment
Theology Council
Tribal Liberation Force
Lai Dai Protection Service
Caldari Constructions
Caldari Provisions
Caldari Steel
Deep Core Mining Inc.
Hyasyoda Corporation
Ishukone Corporation
Kaalakiota Corporation
Minedrill
Perkone
Poksu Mineral Group
Propel Dynamics
Rapid Assembly
School of Applied Knowledge
Sukuuvestaa Corporation
Top Down
Wiyrkomi Corporation
Zainou
Zero-G Research Firm
Duvolle Laboratories
Blood Raiders
Mordu's Legion
Salvation Angels
Guristas Production
Amarr Navy
Imperial Academy
Ministry of Internal Order
Ministry of War
Sarum Family
Serpentis Inquest
Trust Partners
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#77 - 2014-03-09 22:35:03 UTC
No new activity, just a simple bump to keep thread alive.

Apparently there will be a new ship skinning thing, with one of them available in CONCORD LP store. I will be monitoring this and will release a new version if needed. =)
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#78 - 2014-03-26 05:55:51 UTC
Took a little while due to other things, but v0.10 is out with the addition of the Police Pursuit Navy Comet in the CONCORD LP Store, woop woop

Download here:
http://eve-files.com/dl/267604
Vapuk Kripit
Brutor Tribe
Minmatar Republic
#79 - 2014-04-11 01:08:52 UTC
This looks to be an amazingly useful tool to a noob like me.

However, at the risk of not looking like a stats major, how would one access this data please.

Without ever having used sql - I have no idea how to access this (probable) cornucopia.

Thank you - and even if no reply eventuates. Awesome work anyway.

thanks
Sable Blitzmann
24th Imperial Crusade
Amarr Empire
#80 - 2014-04-11 03:32:40 UTC  |  Edited by: Sable Blitzmann
You can load the SQL file into a MySQL database, where you can manipulate it from there. I'm not sure about other DB programs. I also supply a version in JSON format which should be easy to parse.

It really depends what you want to use this data for. You will need to research into both these options for this data to be useful. There's not a whole lot you can do with it out of the box - you must devise a way to store it / access it and use it for your own project. On top of that, you probably want access to the names of the items which is provided by CCP through their static database dump

I might be able to create CSV files based on the tables, so that those with spreadsheet experience can use the info...