These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Why are we not combining market data, player devs?

Author
Dragonaire
Here there be Dragons
#21 - 2011-10-11 02:36:38 UTC
Like any user supplied data even the API data could be faked from the client but like you said it would make more information available which is a good thing and can be used to help detect the bad data. One reason I made the suggestion is I know at least in the case on Yapeal it spends a lot of time doing nothing but waiting on the API servers or the database server but it doesn't use much of the outbound network bandwidth so if people were interested I could look at adding it as an option. I can also see it being something any other applications that work with the API data could add. If all local applications that have marketing functions could easily add API and/or cache anonymous data uploading it would make a lot more data available and then the biggest problem is for the sites to handle the load Blink

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Muscaat
EVE Markets
#22 - 2011-10-11 10:41:12 UTC
I've written a cache scraper and multi-site uploader; binaries and source at http://www.eve-markets.net/uploader.php - currently supports EVE Central, EVE MarketData and EVE Marketeer. So this thread is (as they say) relevant to my interests Blink
Callean Drevus
Perkone
Caldari State
#23 - 2011-10-11 20:54:46 UTC
Muscaat wrote:
I've written a cache scraper and multi-site uploader; binaries and source at http://www.eve-markets.net/uploader.php - currently supports EVE Central, EVE MarketData and EVE Marketeer. So this thread is (as they say) relevant to my interests Blink


But... you know, when I start it it does exactly nothing. I wish I could get it to work. I do support your idea at the very least, especially since java is by definition multiplatform, which is awesome for an uploader!

BTW, sorry for not doing anything this weekend guys, RL is getting in the way.

Developer/Creator of EVE Marketeer

Garo Hertee
Eclipse Industrials
Quantum Forge
#24 - 2011-10-16 10:22:14 UTC  |  Edited by: Garo Hertee
Is it just me or are Eve Marketeer and Eve Marketdata both down? I'm not sure what the development status of these sites is right now but when they're considered production systems, it's crucial that they have a very high uptime.

Edit: I'm getting a 504 from Eve Marketeer and a 503 from Eve Marketdata.
Callean Drevus
Perkone
Caldari State
#25 - 2011-10-16 13:50:21 UTC
Garo Hertee wrote:
Is it just me or are Eve Marketeer and Eve Marketdata both down? I'm not sure what the development status of these sites is right now but when they're considered production systems, it's crucial that they have a very high uptime.

Edit: I'm getting a 504 from Eve Marketeer and a 503 from Eve Marketdata.


Let's just say that EVE Marketeer is a production system, but that the production system is overloaded ATM. Fixes are under way, but they require a major rewrite of the codebase (meaning I'm busily writing now). In the meantime, EVE Marketeer is sometimes a bit erratic in terms of access, though it's never truly down. In the sense that it will eventually be accessible again without me having to reboot the server.

EVE Marketdata somehow always has a period of intermittent 503's somewhere around downtime. I think it has something to do with the database being regenerated.

Developer/Creator of EVE Marketeer

Garo Hertee
Eclipse Industrials
Quantum Forge
#26 - 2011-10-16 14:01:20 UTC
Callean Drevus wrote:
Let's just say that EVE Marketeer is a production system, but that the production system is overloaded ATM. Fixes are under way, but they require a major rewrite of the codebase (meaning I'm busily writing now). In the meantime, EVE Marketeer is sometimes a bit erratic in terms of access, though it's never truly down. In the sense that it will eventually be accessible again without me having to reboot the server.


Thanks, that's good to hear. From my point of view, that means that Eve Marketeer can't compete with Eve Central right now but that there's light at the end of the tunnel to get a replacement which is actively supported. I'll look forward to news on how your hard work is progressing.

Eve Marketdata looked attractive from the point of view that its API was almost identical to Eve Central's. Is Eve Marketeer's also similar?
Callean Drevus
Perkone
Caldari State
#27 - 2011-10-16 16:57:02 UTC
Garo Hertee wrote:
Eve Marketdata looked attractive from the point of view that its API was almost identical to Eve Central's. Is Eve Marketeer's also similar?


I imagine there are some differences in the order of fields, but the concept is the same. If you have any requests, I'll be happy to implement them.

Developer/Creator of EVE Marketeer

Alpha Schodemeiss
Brutor Tribe
Minmatar Republic
#28 - 2011-10-16 16:57:06 UTC
I for one agree that a univeral uploader for all sites would be far more sensible.

Having just looked at the code for Contribtastic (Eve Centrals Uploader), it would be very simple to add new sites to the upload tree.

Infact, at one time, it was posting to Eve Metrics (now offline) anyway, so, this code could just be changed.
Garo Hertee
Eclipse Industrials
Quantum Forge
#29 - 2011-10-16 17:41:53 UTC
I've only ever done server-side programming, so I'm out of my depth with the uploaders, but having a universal uploader would be fantastic. I would immediately switch from Contribtastic to a universal uploader to help get data to the newer sites.
Garo Hertee
Eclipse Industrials
Quantum Forge
#30 - 2011-10-16 18:22:00 UTC
I was just thinking, how about a list of sites with checkboxes? That way, if a site goes offline, you can just untick it until the uploader gets its next revision.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#31 - 2011-10-16 18:32:45 UTC
Ideally it would have a standard API for upload (or possibly change the order round). and be able to have new sites added via a config file. I know that I'd think about setting up my own personal site for the items and regions I'm interested it.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Garo Hertee
Eclipse Industrials
Quantum Forge
#32 - 2011-10-16 21:26:46 UTC
It would be nice if adding a new site for upload was a case of adding a new config file into a directory rather than having to reprogram the uploader. When you start the uploader, it could read in the files from the config directory and let you turn each one on or off. That would let people who wanted to start their own personal sites the ability to simply add a config file of their own. It could be listed in the config file which regions that site was interested in.
Scrapyard Bob
EVE University
Ivy League
#33 - 2011-10-20 04:41:55 UTC
On a semi-related note, folks need to be spreading the word in their corporations / contacts that more people need to run the upload scanners (preferably one that feeds multiple websites). There's a lot of holes in the data on the various sites - even for regions in hi-sec. The Amarr trade-hub data is usually a few days out of date, places like Everyshore or Tash-Murkon are generally a week or more out of date.

(Personally, I try to make sure that Sinq/Heim are up to date at least every 2 days. But more redundancy would help a lot.)
Callean Drevus
Perkone
Caldari State
#34 - 2011-10-21 16:17:43 UTC
EVE Marketeer now contains a definition of an upload API. It can be found on the API page. It is the same format used by EVE Marketdata, which made it the logical choice. If you have any comments or suggestions, please state them here :)

Developer/Creator of EVE Marketeer

Dragonaire
Here there be Dragons
#35 - 2011-10-22 23:40:59 UTC
So after looking at the format used by the above two sites I came up with an idea for how to layout the JSON version of the same thing. Here's a couple of example for everyone to look at:
Quote:
{$site.root}api/upload?data=
{"upload_type":"orders","region_id":1,"type_id":1,"upload_key":"abc",
"developer_key":"xyz","version":"0.1","generated_at":"2011-10-22 15:46:00",
"log":[
[2288279261,"s",30002053,60005686,1299999.99,40,40.0,1,"2011-09-10 12:16:33",90,32767],
[2322655203,"s",30002053,60005686,1299998.54,88,25.0,1,"2011-10-18 21:59:51",90,32767],
[2052179646,"b",30002053,60005686,351001.11,100,39.0,1,"2011-10-09 08:38:12",90,32767]
]
}

Quote:
{$site.root}api/upload?data=
{"upload_type":"history","region_id":1,"type_id":1,"upload_key":"abc",
"developer_key":"xyz","version":"0.1","generated_at":"2011-10-22 16:01:00",
"log":[
["2011-09-10 12:16:33",0.01,2.00,0.30,404,50],
["2011-10-18 21:59:51",0.06,7.00,0.80,909,1]
]
}


The above have been formated for easier reading. To better understand the columns in "log" you'll probably need to have a look at page from EMK when it's back up or get it from repos and just open in as local file in browser. You can find it in tpl/api/display.html. I gave the examples as GET request but POST is much better. I made this just to get some comments and with a minimum of changes to see if anyone else had ideas how it might be improved on.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Dragonaire
Here there be Dragons
#36 - 2011-10-23 00:48:18 UTC  |  Edited by: Dragonaire
Ok sorry for the double post but I think I made an improvement already myself by adding a little more meta-data to them. I also updated the formating to make it a little clearer but of course forums screw up most of it as usual X
Quote:
{
"upload_type":"orders","region_id":1,"type_id":1,"upload_key":"abc",
"developer_key":"xyz","version":"0.1","generated_at":"2011-10-22 15:46:00",
"log":{
"header":["order_id","buysell","solar_system_id","station_id","price",
"vol_entered","vol_remaining","min_volume","issue_datetime","duration",
"range"],
"rows":[
[2288279261,"s",30002053,60005686,1299999.99,40,40.0,1,"2011-09-10 12:16:33",90,32767],
[2322655203,"s",30002053,60005686,1299998.54,88,25.0,1,"2011-10-18 21:59:51",90,32767],
[2052179646,"b",30002053,60005686,351001.11,100,39.0,1,"2011-10-09 08:38:12",90,32767]
]
}
}

Quote:
{
"upload_type":"history","region_id":1,"type_id":1,"upload_key":"abc",
"developer_key":"xyz","version":"0.1","generated_at":"2011-10-22 16:01:00",
"log":{
"header":["history_date","low_price","high_price","avg_price","volume",
"orders"],
"rows":[
["2011-09-10 12:16:33",0.01,2.00,0.30,404,50],
["2011-10-18 21:59:51",0.06,7.00,0.80,909,1]
]
}
}

A nice thing about this version is it's easier for humans to understand and you could possibly reorder the columns in the rows, and add or remove some of them should the need arise in the future.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

Alain Kinsella
#37 - 2011-10-24 07:22:17 UTC  |  Edited by: Alain Kinsella
Your 'history' JSON has 'avg_price' listed twice in the header. I assume that's a typo, and the second one is volume.

Given I know no JSON whatsoever, it was still readable (much more so than XML, sorry guys).

Otherwise I support this effort, though I currently do not run an uploader (thought HQ2 had it already, but not checked all the new things yet).

@ Callean - My own guess for eve-central's performance would be that hardware was thrown at it. At my work the higher-end DB systems use RAM disks (not SSD, actual RAM-based external disks - connected by 2-4 FC links in Solaris Multipathing). Vendors like FusionIO are starting to break that mold for the lower-end (just would be nice if they had SPARC drivers on Solaris).

"The Meta Game does not stop at the game. Ever."

Currently Retired / Semi-Casual (pending changes to RL concerns).

Dragonaire
Here there be Dragons
#38 - 2011-10-24 08:21:18 UTC
Thanks for the catch on typo I've updated post to correct.

Finds camping stations from the inside much easier. Designer of Yapeal for the Eve API. Check out the Yapeal PHP API Library thread.

otto leading
Sebiestor Tribe
Minmatar Republic
#39 - 2011-10-24 23:51:57 UTC
Just an odd thought. Since the data originates on the CCP servers, and since CCP has introduced a new, much more granular api system, and since they are currently in the mood to listen to the player base...

Why not start a concerted lobby for them to open the pricing api to the market information providers. There, one source of data, eliminates cache scraping, file uploads and it is accurate. And most of all, it is the primary and complete set of data.

otto
Tonto Auri
Vhero' Multipurpose Corp
#40 - 2011-10-25 01:35:11 UTC
otto leading wrote:
Why not start a concerted lobby for them to open the pricing api to the market information providers.

Because they said not once that such information is supposed to be collected by players, as a part of gameplay process?

Two most common elements in the universe are hydrogen and stupidity. -- Harlan Ellison