These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
12Next page
 

API / market price data to SQL
Author
Previous Topic Next Topic
EveOrg
Doomheim
#1 - 2011-11-01 09:53:37 UTC  |  Edited by: EveOrg
EveOrg App has been updated, 0.9.8.4 - 18 March 2012

* no registration required, fully free to use personally, corp or alliance wide
* improved API data processing, now more efficient and reliable
* API data scraping expanded to encompass all available API data
* Your choice of database server, MSSQL or MySQL.

The EOApp runs under .Net (v3.5) on Windows (AFAIK it does not work on mono). It does three things - in a fancy multi-threaded way Cool ...

API 2DB
You provide a SQL database and put some API keys in it. API2DB will continually scrape the API for all available data for all the keys and keep the SQL database up to date with the latest data 24/7. Some data (like market orders and sovereignty) also build historically to provide a useful data mine.

Every piece of data available from the API is scraped - lock stock the f@#!ing lot - private, public, even useful anonymous data is scraped about characters and corps - e.g. your wallet transaction data reports selling an item to character "EveOrg" then as much data about EveOrg as possible is scraped from public API calls (like CharacterInfo).

What can you use this for? What can't you use it for! Monitor and plan your trading. Gather intel on corp applicants. Alliance wide detection of hostile activities. Backend to a website. Monitor corp contracts. Easily queryable security logs. Anything you can dream up to do with the data is just a SQL query away :)

Market 2DB
Monitors your Eve client cache folder for market data, new data is added to a table in the SQL database. You can rapidly build up a database of accurate up to date market orders/prices/etc by clicking through the market in-game. There is also an In Game Browser auto-market page that will automatically click through the entire market (or market group) while you are afk - you can even pause/resume the sweep if you need to pewpew.

API Proxy
Acts as a proxy for the API. Point your other API consuming applications at it instead of CCP and always get the latest data (rather than the API rejecting apparently repeated calls).

It's pretty much at late beta stage. Thanks to some early beta testers from a thread in the old forums it now fairly reliable and efficient Big smile So I'm making it available for public consumption. Free of charge. No registration. Donations of isk are of course appreciated Lol

Please report bugs in this thread, or to me via Eve mail, of drop into the in-game channel eveorg.net. Hope people find it useful!
EveOrg
Doomheim
#2 - 2011-11-01 09:56:05 UTC
I would also be interested in hearing from people wanting to integrate other tools/services. Like centralized market data etc.
Jenn Makanen
Doomheim
#3 - 2011-11-01 13:03:43 UTC  |  Edited by: Jenn Makanen
Have you thought about using odbc for the database connection? (Disclaimer: I have no idea how hard this would be)

I've been playing with the python scrapers to get stuff into a remote mysql DB, and this tool looks like it would handle it better.
EveOrg
Doomheim
#4 - 2011-11-01 22:05:43 UTC
The connection is over ODBC, you give it an ODBC connection string. It talks SQL to the DB - it's expecting an MS SQL DB not sure how it would fair with other DBs. Please try and let me know :)

It works fine with the free version of MS SQL Express if it's a cost thing.
Jenn Makanen
Doomheim
#5 - 2011-11-01 22:47:03 UTC
Oh neat Big smile I'll give it a shot later.

Not a cost thing in this case. Just an accessibility thing. If I load the data onto the mysql database, I can get at it from work, as it's on a dedicated server. Gives me more time to play with it, that's not Eve time Blink
Jenn Makanen
Doomheim
#6 - 2011-11-01 23:29:33 UTC
damn. can't get a connection string it'll like.

Oh well. Time to install sql server, and see about a nightly upload from that Big smile
EveOrg
Doomheim
#7 - 2011-11-02 00:46:57 UTC
None of this any help? MySQL connection strings

Or shoot me an evemail with details of the server if it's publically accessible I'll see what I can do.

I'll spend some time this weekend setting up mySQL on a linux VM and see if I can make it work(tm). Smile
Jenn Makanen
Doomheim
#8 - 2011-11-02 00:58:32 UTC
If I remember right I've got a full sql server license kicking around somewhere. (microsoft academy has it's uses Big smile) so I'll give that and some replication software a go. I did give the various connection string types a go, but either they died on the driver or provider portion, or just tried using sql server.

Nothing important really. Just me being a little lazy and not working on the python sample I've got. If you do get it working, I will be grateful however Big smile
EveOrg
Doomheim
#9 - 2011-11-02 01:06:41 UTC  |  Edited by: EveOrg
No need for a full license, SQL Express is up to the task and completely free. Cool

I will have a go at making it work with MySQL, if nothing else the more options people have for deployment the more useful it is - and the more chance of generous isk donations being heaped upon me Lol

Plus I've been meaning to try this linux thing for a while...
EveOrg
Doomheim
#10 - 2011-11-05 02:36:54 UTC
Jenn Makanen wrote:
Have you thought about using odbc for the database connection? (Disclaimer: I have no idea how hard this would be)

I've been playing with the python scrapers to get stuff into a remote mysql DB, and this tool looks like it would handle it better.


New version of EveOrg App out - supporting MySQL :) Now you can configure for MSSQL or MySQL compatibility. All your API data in the database of your choice!

Also fixed a few bugs and that Smile

EveOrg App
6ie
The Kiwis
#11 - 2012-01-15 02:18:23 UTC
Looks awesome. Setting this up to talk to my MYSQL DB right now. Will let you know how it works out.
Kepuh Harjo
Gradient
Electus Matari
#12 - 2012-01-18 10:36:44 UTC
Hi EveOrg!

Great app, really love it!

I sent you several in-game mails with bugs/fixes/etc... but havn't received anything back yet. I've also joined the in-game support channel, but likely having timezone issues with connecting.

I'd love to work with you on the project (have done my own .net MSSQL project and was ready to start again when I found yours).



As a note, dumper.exe crashes every single time it tries to read a file. Does this work for anyone? (Win 7 64)
6ie
The Kiwis
#13 - 2012-01-19 13:58:12 UTC
Market 2DB doesn't seem to work:

80c.cache... ERROR Unknown column 'cacheTS' in 'field list'
3c90.cache... 0 orders (PS.Q=0)
2448.cache... 0 orders (PS.Q=0)
81f7.cache... ERROR Unknown column 'cacheTS' in 'field list'
f1a8.cache... ERROR Unknown column 'cacheTS' in 'field list'
c900.cache... 0 orders (PS.Q=0)
a9e3.cache... 0 orders (PS.Q=0)
a87f.cache... 0 orders (PS.Q=0)
Kepuh Harjo
Gradient
Electus Matari
#14 - 2012-01-19 16:48:01 UTC
so umm, i'm a little dumb at times :)

http://dl.stackfoundry.org/libevecache/

when i went here to download, i took the top file, without noticing the date... guess I just assumed it was sorted by date desc...

silly me :)

[16:39:05] 6ietrader > this is the dumper i use http://dl.stackfoundry.org/libevecache/libevecache-win32-bin-0.2.1.zip

6ie to the rescue via the in-game chat channel and now my public orders are being populated!

VICTORY IS MINE!!! (or well, his... but whatever) :)
Aineko Macx
#15 - 2012-02-06 20:37:46 UTC
Bug when running the MySQL DB under a *nix system:
In these systems, table names are case sensitive, but EONapp creates all lower case tables, then calls tables with camel case name, and consequently errors out.

Initial startup checks...
Validating API database... Initialising schema... OK.
... EveOrg DB appears OK.
Validating Eve database... OK.
... Eve DB appears OK.
Validating Market database... OK.
Initialising... OK.
Validating static data with EveOrg.Net...
AgentIDs ERROR!
Table 'eonapp.eoncache_AgentIDs' doesn't exist

iveeCore 3.0: The PHP engine for industrial activities and CREST library
Kepuh Harjo
Gradient
Electus Matari
#16 - 2012-02-07 21:57:30 UTC  |  Edited by: Kepuh Harjo
i have the same issues... just with a binary collation MS SQL Server

first i had to rename all the EONCACHE tables to lower case, just to get the test config to pass, but now when I try to add api keys it complains about the ccpuser table not found.

the table name in the db is ccpUser yet the code is looking for ccpuser

I was able to make the nessary changes to the last version to handle my case-sensitive server collation, but this version seems to have alot more mis-matched upper/lower case issues :(

can't use this version at all sadly :(
6ie
The Kiwis
#17 - 2012-02-07 23:01:20 UTC
An issue with the new version when enabling Market2DB:

Exception: FUNCTION eonappdb.dateadd does not exist
Imuran
Zentor Industries
#18 - 2012-02-09 16:39:13 UTC
Falling at the first hurdle here

Validating API database... Initialising schema... ERROR!
Can't create table 'EonAppDB.xmlcachetimes' (errno: 150)


MySQL database running on a QNAP NAS
Nathan Gorden
State War Academy
Caldari State
#19 - 2012-02-18 06:04:10 UTC
6ie wrote:
An issue with the new version when enabling Market2DB:

Exception: FUNCTION eonappdb.dateadd does not exist



I use MySQL and I have this exact same problem. I sent a message to the developer, but I have not gotten a response back. I really hope this gets fixed soon as this is an extremely valuable tool.
Nathan Gorden
State War Academy
Caldari State
#20 - 2012-02-27 05:45:59 UTC
An update to this Application. IT DOES WORK!

I spent hours trying to get the MySQL feature to work...I eventually gave up.
The MySQL portion of this App is currently broken.
I took the time to install MS SQL Server 2008 Express and the Market 2DB function works just fine. Likewise with Market Sync, it works. As much as I would like to use MySQL, MSSQL will have to suffice.
12Next page