These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
Previous page12
 

API / market price data to SQL
Author
Previous Topic Next Topic
EveOrg
Doomheim
#21 - 2012-03-18 11:02:02 UTC
Hi, sorry been a while since last update. Had to go away for a month abroad to work. Back and with an update to EveOrg. Will attempt to answer some questions posted above...

dumper.exe crashing. I found a new version of dumper from the libevecache website fixed that (something changed!)

Kepuh Harjo, having to recapitalise all the fieldnames in code would be a big job, maybe next update sorry.

6ie, fixed 'cacheTS' bug in this version.

6ie, Nathan Gorden, fixed dateadd bug. Sorry you had to go to the trouble of installing ms-sql ;)

Sorry about the lowecase/camelcase problems. I will address them in a future update.
6ie
The Kiwis
#22 - 2012-03-18 11:48:23 UTC
Cool - will install soon and report back!
Xelia Otsada
Omnicorp Unlimited
#23 - 2012-04-14 17:40:45 UTC  |  Edited by: Xelia Otsada
Good morning. I was looking to try your application, EONAPP v0.9.8.4 . I was able to create the MS SQL and users without a problem.

Update: The below issue was resolved by EveOrg.net (it was a server side fix.) The Market2DB functionality portion works without hitch. Initial feedback from our trader is that it works flawlessly and has saved her a good chunk of time doing a manual run of evelibcache dumper.dumping the information and loading it to the database. Kudos to EveOrg for the cache scraping tool for Market Orders!


However, when I go to test the config I get the following error after it passes the DB checks:

Starting version 0.9.8.4 at 2012-04-14 17:24:45 (UTC)
Configuration
EODB: Data Source=xxxx\xxxxxxx; Initial Catalog=EONAppDB;user id=xxx;password=;
is: MSSQL
EveDD: Data Source=xxxx\xxxxxxx; ;Initial Catalog=Crucible_1.6_65449;user id=xxx;password=;
is: MSSQL
MarketDB: Data Source=xxxx\xxxxxxx;Initial Catalog=EONAppDB;user id=xxx;password=;
is: MSSQL
Data: {directory path removed}EONAppDataStore
Admin: xxxx
SMTP: xxxx
API: http://api.eve-online.com
ProxP: 30303
Stats: True
Dumper: {directory path removed}dumper.exe
Eve: {directory path removed}c_games_xxxx_tranquility
MSyncID: not configured!
Initial startup checks...
Validating API database... OK.
... EveOrg DB appears OK.
Validating Eve database... OK.
... Eve DB appears OK.
Validating Market database... OK.
Initialising... OK.
Validating static data with EveOrg.Net...
AgentIDs downloading new data (remote)... ERROR!
The remote server returned an error: (404) Not Found.

I tried disabling my firewall and still encountered the same error. Can you provide details on what remote server it is trying to go to get the AgentIDs download?

Cheers,
Xelia
6ie
The Kiwis
#24 - 2012-04-27 00:07:30 UTC
hmmm i'm getting this as of today..

Starting version 0.9.8.4 at 2012-04-27 00:05:29 (UTC) Configuration EODB: server=c...
...
...
Initialising... OK.Validating static data with EveOrg.Net... AgentIDs checking... ERROR!Conversion from string "\StaticData" to type 'Integer' is not valid.

Due to the Escalation DB changes?
EveOrg
Doomheim
#25 - 2012-05-07 06:21:05 UTC
Yes, more than likely the DB changes. Giant new update has been released that takes care of that, and some other things. Including..

Totally Refactored Market Sync

Massively more scalable and efficient market sync feature. Sync your market price database with the central server and get universe wide price data (assuming someone has uploaded it). Yes, a bit like Eve-Central et al.

This central price data is also available to anyone who wants to query it with some code/script, instructions on the Market Prices page. Or you can just let the app do the work then query your own DB yourself.

I just noticed a forum thread about combining market data from the other market data websites, guess I'll have a read of that and see what's going on there...
Khorkrak
KarmaFleet
Goonswarm Federation
#26 - 2012-05-07 13:56:49 UTC  |  Edited by: Khorkrak
What is "scraping the API" exactly. Second time I've seen this term used and have only heard of it in the context of EVE. Searched on the web and couldn't find anything defining this jargon. I'm assuming that it means querying an API for all possible values. Is that correct? What does that have to do with "scraping"? Is there a negative connotation to this as scraping doesn't sound particularly pleasant?

Developer of http://www.decloaked.com and http://sourceforge.net/projects/pykb/
EveOrg
Doomheim
#27 - 2012-05-08 11:12:12 UTC
Khorkrak wrote:
What is "scraping the API" exactly. Second time I've seen this term used and have only heard of it in the context EVE. Searched on the web and couldn't find anything defining this jargon. I'm assuming it means querying an API for all possible values. Is that correct? What does that have to do with "scraping"? Is there are negative connotation to this as scraping doesn't sound particularly pleasant?


Heh you are correct, scraping - also known as harvesting - basically getting as much data as possible.

EON App does that. You put Eve api keys into the database, it fills the database with all the API data it can get.

Similar for the market scraper, if you enable the sync you'll get all the market data for all the items in all the regions you want.

Did you have a preferred less 'negative' term for it?!?
Dragonaire
Here there be Dragons
#28 - 2012-05-08 14:46:04 UTC
How about using data mining which is more what you're doing. Scraping is usually used when you are take data from another source then didn't intend the data to be used that way but if you're using people's ApiKeys etc it's not scraping.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Desmont McCallock
#29 - 2012-05-08 15:09:35 UTC  |  Edited by: Desmont McCallock
Scraping is what we do, through the various libraries we have created for diff programming languages, getting data out of EVE cache files. EVE cache files where created to be used by the EVE client only but we figured out their structure and can now harvest the same data the EVE client uses.
Dragonaire
Here there be Dragons
#30 - 2012-05-08 15:13:52 UTC
Grabbing stuff from the cache files would be scraping alright but using ApiKeys to get it from the APis is not which was the main point plus suggesting a better name for that.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.
Desmont McCallock
#31 - 2012-05-08 15:17:32 UTC
I was supplementing your post not controversing it, Dragonaire.
EveOrg
Doomheim
#32 - 2012-05-08 20:52:14 UTC
Dragonaire wrote:
How about using data mining which is more what you're doing. Scraping is usually used when you are take data from another source then didn't intend the data to be used that way but if you're using people's ApiKeys etc it's not scraping.


Data mining is what you do with the data once it's in the database. Scraping is the process of collecting the data. I think we're getting a bit semantic here ;) How about API spider, or crawler, or vacuum?
Previous page12