These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Devsite blog: Nom nom tasty market data!

First post First post
Author
Previous Topic Next Topic
Querns
Science and Trade Institute
Caldari State
#21 - 2014-11-20 17:26:11 UTC
Sven Viko VIkolander wrote:
Querns wrote:
Sven Viko VIkolander wrote:
This is pretty awesome, but I am confused why this data is being released. I thought Foxfour et al were opposed to making pubic any data sets that might negatively impact current gameplay styles? For instance, Foxfour et al decided not to make all killmails public, partly because some players, apparently, liked to prevent some of their kills/losses from reaching public boards.

How does releasing market data negatively impact yours or anyone's current gameplay style?


Some traders do a lot of research on markets to find the best trades across regions. Currently, the data on sites like eve central is spotty at best for any non-major trade hub, requiring in-game work. I doubt this will affect my play styles in any way, and it will benefit most people, but then again when it came to public kill information it was largely the complaints of a very small group of players that seemed to sway against making the data public. When public market data was suggested in the past at times like fanfest there were also some worries raised about whether this will make running market bots easier.

Considering that the alternative of using the IGB to prime the client's cache to be scraped is being eliminated, I would think that folks would be grateful for a way to continue culling market data for regional arbitrage. I mean, what's the alternative? No market data at all?

One thing is definitely for certain — eliminating cache scraping will unimpeachably harm automated pricing alerts. Note that I didn't say "bot" specifically — there is a perfectly legal way to use the IGB to continually poll orders, then have an alert be raised or a sound played, alerting a flesh and blood user when their orders have been undercut. With a five minute cache timer (and indeed no guarantee that the market data in question is even relatively current; it could very easily be delayed) you will see a marked reduction in automatic price updates.

This post was crafted by the wormhole expert of the Goonswarm Economic Warfare Cabal, the foremost authority on Eve: Online economics and gameplay.
Thorbit Klefman
The Dirty Alpaca Fur Traders
The Last Alpaca
#22 - 2014-11-20 17:34:36 UTC
would like to request access to this to test, sisi clientid: ca7b1206e7e94dd9b0fa1976ab379b64 named "AlpacaWaffe Test"
Fiesertyp Lennelluc
Federal Navy Academy
Gallente Federation
#23 - 2014-11-20 17:37:56 UTC
I'd like to get access, too

appname: HyperCRESTTest
ID on sisi: 3a474b8137f7413b92812c8bce96b054
CCP FoxFour
C C P
C C P Alliance
#24 - 2014-11-20 17:45:07 UTC
Thorbit Klefman wrote:
would like to request access to this to test, sisi clientid: ca7b1206e7e94dd9b0fa1976ab379b64 named "AlpacaWaffe Test"


Done

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
CCP FoxFour
C C P
C C P Alliance
#25 - 2014-11-20 17:45:47 UTC
Fiesertyp Lennelluc wrote:
I'd like to get access, too

appname: HyperCRESTTest
ID on sisi: 3a474b8137f7413b92812c8bce96b054


Done

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Turk Fezzik
Sons of Seyllin
Pirate Lords of War
#26 - 2014-11-20 18:14:33 UTC
Sign me up.

TURKTESTAPP_WINPHONE8.1
907a4993e1c84ca483a8ab80bd1a3bdb
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#27 - 2014-11-20 18:28:56 UTC
Just so it's here:

I'm working up a php class to hide all the nasty crest bits away, so you can just say 'gimme data' and it does. An initial draft is in the following gist. https://gist.github.com/fuzzysteve/78c8952afc3d70d5f388

It's fairly clean, requiring a refresh token (when you sign in, asking for the scope, you'll be given one), and following the paths through from the base url.

I'll be expanding it, and putting it into a composer based system at some point in the nearish future. Along with a little code you can use to generate your refresh token. It would also be possible to just use a regular oauth login, which stores that token, and hands it over.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Zifrian
The Frog Pond
Ribbit.
#28 - 2014-11-20 18:38:20 UTC
It seems like your target audience for this is websites? I guess I'm wondering who will use this and why. If it's an industry or market site, then they would likely want all the data for the entire game. Limited data is helpful but users generally like all kinds of crazy stuff. I guess you can limit that but then, why use this resource when you can use Eve central?

I'm confused of the value for this sort of thing if you limit it. If you don't then I don't see how it doesn't replace Eve central. Heck even if you do, what's stopping five people from splitting up the data pulls across the entire maket? What's to stop a webpage from just replicating the data? And if they don't, why would anyone go to eve central when they can just use the website that has access to this data?

I may be missing the point here but what I'm confused in what the goal is. What is wrong with the status quo that this solves? My biggest worry is it screws up eve central and then I'm left with bumpkis for prices in my app...and several years of a hobby project down the drain.

Maximze your Industry Potential! - Download EVE Isk per Hour!

Import CCP's SDE - EVE SDE Database Builder
CCP FoxFour
C C P
C C P Alliance
#29 - 2014-11-20 18:47:34 UTC
Turk Fezzik wrote:
Sign me up.

TURKTESTAPP_WINPHONE8.1
907a4993e1c84ca483a8ab80bd1a3bdb


Done

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Ydnari
Estrale Frontiers
#30 - 2014-11-20 18:54:37 UTC
Quote:
Web sites such as EVE Central and their APIs will be around for a long time to come. We are however fairly hopeful EVEMon can stop cache scraping after this.


So it's going to be OK for EVE Central to read from the CREST endpoint, and offer that data out through its existing APIs?

EVEMon cache scraping is where it gets the vast majority of its data from right now.

--
CCP FoxFour
C C P
C C P Alliance
#31 - 2014-11-20 18:57:14 UTC
Ydnari wrote:
Quote:
Web sites such as EVE Central and their APIs will be around for a long time to come. We are however fairly hopeful EVEMon can stop cache scraping after this.


So it's going to be OK for EVE Central to read from the CREST endpoint, and offer that data out through its existing APIs?

EVEMon cache scraping is where it gets the vast majority of its data from right now.


Yes. EVE Central can get the data from CREST and share it via it's own API's. :)

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#32 - 2014-11-20 19:00:41 UTC
Zifrian wrote:
It seems like your target audience for this is websites? I guess I'm wondering who will use this and why. If it's an industry or market site, then they would likely want all the data for the entire game. Limited data is helpful but users generally like all kinds of crazy stuff. I guess you can limit that but then, why use this resource when you can use Eve central?

I'm confused of the value for this sort of thing if you limit it. If you don't then I don't see how it doesn't replace Eve central. Heck even if you do, what's stopping five people from splitting up the data pulls across the entire maket? What's to stop a webpage from just replicating the data? And if they don't, why would anyone go to eve central when they can just use the website that has access to this data?

I may be missing the point here but what I'm confused in what the goal is. What is wrong with the status quo that this solves? My biggest worry is it screws up eve central and then I'm left with bumpkis for prices in my app...and several years of a hobby project down the drain.



This can work quite happily with an app, rather than a website. It's just a bunch of web requests. The only 'complicated' bit is getting a refresh token, which is set up to work with your clientid and secret. Which isn't particularly complicated to do.

It could replace Eve Central. Though a site providing an aggregation function is still of immense use to everyone who uses spreadsheets.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Ydnari
Estrale Frontiers
#33 - 2014-11-20 19:12:30 UTC
CCP FoxFour wrote:
Ydnari wrote:
Quote:
Web sites such as EVE Central and their APIs will be around for a long time to come. We are however fairly hopeful EVEMon can stop cache scraping after this.


So it's going to be OK for EVE Central to read from the CREST endpoint, and offer that data out through its existing APIs?

EVEMon cache scraping is where it gets the vast majority of its data from right now.


Yes. EVE Central can get the data from CREST and share it via it's own API's. :)


Alrighty. I was thinking there was a clause in the developer license agreement that prohibited this, but comparing the old version I'd read and the current version up on the site there appears to be changes to the wording that no longer prohibit offering APIs based on the licensed data, and if you confirm it too, then all's well.

I guess that means EMDR is OK to read from CREST too, and that'd be the most efficient way to switch over not just EVE Central but also everything else using EMDR.

--
CCP FoxFour
C C P
C C P Alliance
#34 - 2014-11-20 19:21:03 UTC
Ydnari wrote:
CCP FoxFour wrote:
Ydnari wrote:
Quote:
Web sites such as EVE Central and their APIs will be around for a long time to come. We are however fairly hopeful EVEMon can stop cache scraping after this.


So it's going to be OK for EVE Central to read from the CREST endpoint, and offer that data out through its existing APIs?

EVEMon cache scraping is where it gets the vast majority of its data from right now.


Yes. EVE Central can get the data from CREST and share it via it's own API's. :)


Alrighty. I was thinking there was a clause in the developer license agreement that prohibited this, but comparing the old version I'd read and the current version up on the site there appears to be changes to the wording that no longer prohibit offering APIs based on the licensed data, and if you confirm it too, then all's well.

I guess that means EMDR is OK to read from CREST too, and that'd be the most efficient way to switch over not just EVE Central but also everything else using EMDR.


Yes, I pushed very hard to make sure that clause was removed. This community thrives on sharing and working together.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Ydnari
Estrale Frontiers
#35 - 2014-11-20 19:39:22 UTC
The final numeric ID in the href for the order is the order ID as per the XML endpoints, right? So we can cross-reference with our orders to mark our own ones?

Can we have volEntered in the data (the original quantity on the order)? It's in the in-game market "Export to File" data and the XML endpoints.

--
Muscaat
EVE Markets
#36 - 2014-11-20 20:11:51 UTC
I've run EVE Markets for more than six and a half years. CCP Foxfour, I can't tell you how delighted I am with this new API! Smile

I wonder how EMDR fits into this new future. Will people write CREST clients that pump data into it, or will people consider it superseded? (Just when I thought I had some time to take a break from dev projects P)
Aryth
University of Caille
Gallente Federation
#37 - 2014-11-20 20:33:22 UTC
If you thought the cartels were bad before then this should horrify you.

For everyone else, horray.

Leader of the Goonswarm Economic Warfare Cabal.

Creator of Burn Jita

Vile Rat: You're the greatest sociopath that has ever played eve.
Desmont McCallock
#38 - 2014-11-20 21:40:28 UTC
This is HUGE news. If EMDR decides to switch to the CREST API for the market orders too then we (EVEMon) have no reason to keep scrapping the cache for them.
J3ST3R
Dark Light Inc
Caretakers
#39 - 2014-11-20 23:07:59 UTC
To Bad we can't get ccp fox4's team to sort out corp votingQuestion
Pete Butcher
The Scope
Gallente Federation
#40 - 2014-11-21 07:50:07 UTC
I approve this apiBig smile sign me up!

Name: Evernus Dev
Id: d153969c683746dd8aefcfba5860e18c

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool