These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Devsite blog: Nom nom tasty market data!

First post First post
Author
Previous Topic Next Topic
CCP FoxFour
C C P
C C P Alliance
#81 - 2014-11-26 12:34:11 UTC
Pete Butcher wrote:
CCP FoxFour wrote:
Pete Butcher wrote:
Seems like either I'm doing something wrong or Sisi is trolling me. I fetch region sell/buy urls using "regions" endpoint: http://public-crest-sisi.testeveonline.com/regions/ (the result is eg. http://public-crest-sisi.testeveonline.com/market/10000032/orders/buy/), then I try to use that url to fetch some orders (using "itemTypes" endpoint url to specify the type: http://public-crest-sisi.testeveonline.com/inventory/types/), so the result is a GET request to http://public-crest-sisi.testeveonline.com/market/10000032/orders/buy/?type=http://public-crest-sisi.testeveonline.com/inventory/types/10631/ . And then I get a 404. Am I missing something?


You're trying to do it via public CREST. It has to be done via authed CREST. There is some information back a page talking about it.


So, am I supposed to replace any of those url for something else? I already pass the access token to every request, so I assumed everything will "just work".
The dev blog only says that the endpoints are not public and need OAuth - already doing that.


Sorry about that. The second example includes the URL for authed CREST: https://api-sisi.testeveonline.com

On TQ that will be https://crest-tq.eveonline.com/ I believe.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Pete Butcher
The Scope
Gallente Federation
#82 - 2014-11-26 13:02:24 UTC
CCP FoxFour wrote:
Pete Butcher wrote:
CCP FoxFour wrote:
Pete Butcher wrote:
Seems like either I'm doing something wrong or Sisi is trolling me. I fetch region sell/buy urls using "regions" endpoint: http://public-crest-sisi.testeveonline.com/regions/ (the result is eg. http://public-crest-sisi.testeveonline.com/market/10000032/orders/buy/), then I try to use that url to fetch some orders (using "itemTypes" endpoint url to specify the type: http://public-crest-sisi.testeveonline.com/inventory/types/), so the result is a GET request to http://public-crest-sisi.testeveonline.com/market/10000032/orders/buy/?type=http://public-crest-sisi.testeveonline.com/inventory/types/10631/ . And then I get a 404. Am I missing something?


You're trying to do it via public CREST. It has to be done via authed CREST. There is some information back a page talking about it.


So, am I supposed to replace any of those url for something else? I already pass the access token to every request, so I assumed everything will "just work".
The dev blog only says that the endpoints are not public and need OAuth - already doing that.


Sorry about that. The second example includes the URL for authed CREST: https://api-sisi.testeveonline.com

On TQ that will be https://crest-tq.eveonline.com/ I believe.


Thanks, works fine now. However, I would suggest returning the proper url from the regions endpoint - the one returned now will never work anyway.
Also, how about that order id and type id in the results (or better yet, ALL the data that is available in the market logs exported in-game)?Smile

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
CCP FoxFour
C C P
C C P Alliance
#83 - 2014-11-26 13:08:11 UTC
Pete Butcher wrote:
CCP FoxFour wrote:
Pete Butcher wrote:
CCP FoxFour wrote:
Pete Butcher wrote:
Seems like either I'm doing something wrong or Sisi is trolling me. I fetch region sell/buy urls using "regions" endpoint: http://public-crest-sisi.testeveonline.com/regions/ (the result is eg. http://public-crest-sisi.testeveonline.com/market/10000032/orders/buy/), then I try to use that url to fetch some orders (using "itemTypes" endpoint url to specify the type: http://public-crest-sisi.testeveonline.com/inventory/types/), so the result is a GET request to http://public-crest-sisi.testeveonline.com/market/10000032/orders/buy/?type=http://public-crest-sisi.testeveonline.com/inventory/types/10631/ . And then I get a 404. Am I missing something?


You're trying to do it via public CREST. It has to be done via authed CREST. There is some information back a page talking about it.


So, am I supposed to replace any of those url for something else? I already pass the access token to every request, so I assumed everything will "just work".
The dev blog only says that the endpoints are not public and need OAuth - already doing that.


Sorry about that. The second example includes the URL for authed CREST: https://api-sisi.testeveonline.com

On TQ that will be https://crest-tq.eveonline.com/ I believe.


Thanks, works fine now. However, I would suggest returning the proper url from the regions endpoint - the one returned now will never work anyway.
Also, how about that order id and type id in the results (or better yet, ALL the data that is available in the market logs exported in-game)?Smile


If you go from the root of the authed CREST it's all fine.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Max Kolonko
Caldari Provisions
Caldari State
#84 - 2014-11-26 21:59:14 UTC  |  Edited by: Max Kolonko
CCP FoxFour wrote:
Max Kolonko wrote:
Why do we pass type as uri to markettypes crest instead of just adding /type/1245 at the end of uri?


I will probably add an id parameter as well, but defiantly not as part of the URI.

The reason for this not being part of the URI though is because we never want you constructing URIs. Adding a parameter yes, but constructing them no. You should be able to get to everything by following links from the root of the API. In this case it's go to the regions, get links to the region markets. Then get a list of all types on the market and pass the link to that type as a parameter to the regions market link you got early. In that flow you never construct a URI. One of the big advantages of this is that if we ever decide to change the route to a resource, such as moving the types from /types/ to /inventory/types/ which I just did today, you would not have to ever update your application. Because you got the links from CREST and never constructed a URI it would all just work.

Hope that makes sense.


ok, thank You, but still have some more questions as I'm still trying to wrap my head around concept of following links from main page to the one I want:

My imaginery application will list prices for 1 item, for example PLEX in the region of The Forge.

I dont want my user to go from main market group via subgropus untill i find plex - i just want him to see plex orders in that region.

So how will list of calls looks like? I go to main crest page and get regions api link, than go to that link and look for the forge region link and follow it and inside follow market orders api link.

Now I have valid link for The Forge market orders and I need plex inventory link.

So I go back to main crest page and follow inventory api link, then go look for group type, subtype, etc... untill i get into right market branch containing plex and look for plex there.

If i understand corectly i have to know where to look for plex on market and still dont know how to change when sudenly our plex example will move to diffrent market branch.

So all in all, unless I have it all wrong I have to follow tons of crest calls even though I know plex item ID and forge region ID.

pls tell me I have it all wrong

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#85 - 2014-11-26 23:16:49 UTC  |  Edited by: Steve Ronuken
To get a plex price you:

Get an access token.

Use it to:
Access https://api-sisi.testeveonline.com/

Get the itemTypes endpoint (currently https://api-sisi.testeveonline.com/inventory/types/, but that can change). You'll have to bounce through each of the next links, to build up the whole thing. (no bouncing through groups. the endpoint returns raw items, over (currently) 23 pages.

Ideally store that for a while so you don't need to keep hitting it.

From there, you look up the href for PLEX (by name)

Then, you hit the region endpoint (again, finding it from the main list)
From there you can find the endpoint for the appropriate region (by name)

From that, you get the market order endpoints.

You now have what you need to get the price for plex. Yes, it's possible to cheat and assemble the urls yourself. It may not be the best idea though. It'll work for a while, but if they need to update the details, then it'll break.


FoxFour: Can you add the id numbers for the things to the endpoints, in addition to the names? for me, the perfect presentation would be
items:[0:{"href": "https://api-sisi.testeveonline.com/inventory/types/0/","name": "#System"},
etc



Possibly add the market group href to the item types too, where it exists. If possible.

Oh, and possibly increase the expiry time for the access tokens for SiSi. Browsing the tree with something like postman is a bit of a pain, with the 300 second expiry.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#86 - 2014-11-26 23:55:22 UTC
If you want to browse the endpoints, and you can lay your hands on a refresh token, I'd suggest picking up the packaged Postman chrome extension. http://www.getpostman.com/

It allows for things like retaining headers, which, once you have an access token, makes life substantially easier for clicking around, rather than having to fill in the authorization header again and again.

It's missing some functionality which would be nice, but I'm not currently willing to pay for the jetpacks extension for it. Even if it is just 10 bucks.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Pete Butcher
The Scope
Gallente Federation
#87 - 2014-11-27 08:23:49 UTC
Steve Ronuken wrote:
FoxFour: Can you add the id numbers for the things to the endpoints, in addition to the names? for me, the perfect presentation would be
items:[0:{"href": "https://api-sisi.testeveonline.com/inventory/types/0/","name": "#System"},


Also, solar system id and volume entered would be nice.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Max Kolonko
Caldari Provisions
Caldari State
#88 - 2014-11-27 10:43:05 UTC  |  Edited by: Max Kolonko
Steve Ronuken wrote:
To get a plex price you:

Get an access token.

Use it to:
Access https://api-sisi.testeveonline.com/

Get the itemTypes endpoint (currently https://api-sisi.testeveonline.com/inventory/types/, but that can change). You'll have to bounce through each of the next links, to build up the whole thing. (no bouncing through groups. the endpoint returns raw items, over (currently) 23 pages.

Ideally store that for a while so you don't need to keep hitting it.

From there, you look up the href for PLEX (by name)

Then, you hit the region endpoint (again, finding it from the main list)
From there you can find the endpoint for the appropriate region (by name)

From that, you get the market order endpoints.

You now have what you need to get the price for plex. Yes, it's possible to cheat and assemble the urls yourself. It may not be the best idea though. It'll work for a while, but if they need to update the details, then it'll break.


FoxFour: Can you add the id numbers for the things to the endpoints, in addition to the names? for me, the perfect presentation would be
items:[0:{"href": "https://api-sisi.testeveonline.com/inventory/types/0/","name": "#System"},
etc



Possibly add the market group href to the item types too, where it exists. If possible.

Oh, and possibly increase the expiry time for the access tokens for SiSi. Browsing the tree with something like postman is a bit of a pain, with the 300 second expiry.


So You have to potentialy make something like up to 30 calls just to get links even tough You already know id's of stuff You need. We really need parameters that we can add to link, even in the form of ?foo=bar



Also, I cant find on Dev page any info how to access auth crest

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
Pete Butcher
The Scope
Gallente Federation
#89 - 2014-11-27 10:57:04 UTC
Max Kolonko wrote:
[quote=Steve Ronuken]Also, I cant find on Dev page any info how to access auth crest


There is none, unfortunately. Basically, you need to use https://api-sisi.testeveonline.com as base url instead of http://public-* and pass the access token obtained through SSO (there is a page for that in the resources).

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool
Max Kolonko
Caldari Provisions
Caldari State
#90 - 2014-11-27 11:37:34 UTC
Steve Ronuken wrote:
If you want to browse the endpoints, and you can lay your hands on a refresh token, I'd suggest picking up the packaged Postman chrome extension. http://www.getpostman.com/

It allows for things like retaining headers, which, once you have an access token, makes life substantially easier for clicking around, rather than having to fill in the authorization header again and again.

It's missing some functionality which would be nice, but I'm not currently willing to pay for the jetpacks extension for it. Even if it is just 10 bucks.



trying to use that extension to get access token.

I set my client id and secret. Changed callback url in my app settings on sisi to https://www.getpostman.com/oauth2/callback, now i'm left with to fields i'm not sure what to put thee:
autorisation URL
access token URL

I assume that access token URL is https://sisilogin.testeveonline.com/oauth/token/ but what is the other one?

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#91 - 2014-11-27 12:13:55 UTC
Yes, 30 odd requests to get it.

However, that's what caching is for. One single lookup takes a while. But once it's all done once, then you don't need to redo it all for something else.

I'm currently looking at options for my library for this. Currently wavering between slapping it into a database (for portability), into memcache/redis for speed, or using something like Guzzle to just use webcaching.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#92 - 2014-11-27 12:20:06 UTC
Kali Izia
GoomWaffe
#93 - 2014-11-27 12:21:08 UTC
Max Kolonko wrote:
I set my client id and secret. Changed callback url in my app settings on sisi to https://www.getpostman.com/oauth2/callback, now i'm left with to fields i'm not sure what to put thee:
autorisation URL
access token URL

I assume that access token URL is https://sisilogin.testeveonline.com/oauth/token/ but what is the other one?

/oauth/authorize and /oauth/token are the relevant URLs

But PSA from testing this: Since the dev site doesn't actually let you select CREST Access, making any changes will revert your key back to a normal SSO only one.

FoxFour - Please add access back to a6c586907bbf4879a203f2f6304a8213 :)
Max Kolonko
Caldari Provisions
Caldari State
#94 - 2014-11-27 12:23:51 UTC
Steve Ronuken wrote:
Token url is https://sisilogin.testeveonline.com/oauth/token
The authorization url is probably https://sisilogin.testeveonline.com/oauth/authorize



its working now, thx

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
CCP FoxFour
C C P
C C P Alliance
#95 - 2014-11-27 13:00:06 UTC
Kali Izia wrote:
Max Kolonko wrote:
I set my client id and secret. Changed callback url in my app settings on sisi to https://www.getpostman.com/oauth2/callback, now i'm left with to fields i'm not sure what to put thee:
autorisation URL
access token URL

I assume that access token URL is https://sisilogin.testeveonline.com/oauth/token/ but what is the other one?

/oauth/authorize and /oauth/token are the relevant URLs

But PSA from testing this: Since the dev site doesn't actually let you select CREST Access, making any changes will revert your key back to a normal SSO only one.

FoxFour - Please add access back to a6c586907bbf4879a203f2f6304a8213 :)


Whats the client name again?

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Kali Izia
GoomWaffe
#96 - 2014-11-27 14:30:23 UTC
POSthing Market
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#97 - 2014-11-27 22:32:52 UTC
https://github.com/fuzzysteve/CrestLibrary is the beginning of a library for dealing with CREST.

No magic strings, CCP can change the locations of endpoints (other than the first) as much as they want, and it'll continue to work.

They just can't change the query parameter needed for the market endpoint.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Max Kolonko
Caldari Provisions
Caldari State
#98 - 2014-11-27 23:08:58 UTC  |  Edited by: Max Kolonko
Steve Ronuken wrote:
https://github.com/fuzzysteve/CrestLibrary is the beginning of a library for dealing with CREST.

No magic strings, CCP can change the locations of endpoints (other than the first) as much as they want, and it'll continue to work.

They just can't change the query parameter needed for the market endpoint.


Looks like i'm gona drill You with some questions :)

I have php 5.4.34 for the reference in case its php version issues

I entered my credentials into setup file (app id and secret), left url unchenged and cleared refresh token value (left empty string) as i understand there are no refresh tokens for now.

I set back callback url to url of the file i'm running and i got this error:


Fatal error: Uncaught exception 'GuzzleHttp\Exception\ClientException' with message 'Client error response [url] https://sisilogin.testeveonline.com/oauth/token/?grant_type=refresh_token&refresh_token= 
[status code] 400 [reason phrase] Bad Request' in /home/bud-net/ftp/praca_dyplomowa/crest/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:68 Stack trace: 
#0 /home/bud-net/ftp/praca_dyplomowa/crest/vendor/guzzlehttp/guzzle/src/Subscriber/HttpError.php(31): 
GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Message\Request), Object(GuzzleHttp\Message\Response)) 
#1 /home/bud-net/ftp/praca_dyplomowa/crest/vendor/guzzlehttp/guzzle/src/Event/Emitter.php(105): GuzzleHttp\Subscriber\HttpError->onComplete(Object(GuzzleHttp\Event\CompleteEvent), 'complete') 
#2 /home/bud-net/ftp/praca_dyplomowa/crest/vendor/guzzlehttp/guzzle/src/Event/RequestEvents.php(75): GuzzleHttp\Event\Emitter->emit('complete', Object(GuzzleHttp\Event\CompleteEvent))
#3 /home/bud-net/ftp/praca_dyplomowa/crest/vendor/guzzlehttp/guzzle/src/ in /home/bud-net/ftp/praca_dyplomowa/crest/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php on line 68



I then changed the URL variable to 'https://sisilogin.testeveonline.com/' and error changed. I stil didn't get redirected to eve login page:

### ### 
Warning: Invalid argument supplied for foreach() in /home/bud-net/ftp/praca_dyplomowa/crest/vendor/fuzzysteve/crestlibrary/src/CrestLibrary/Client.php on line 111

Warning: Invalid argument supplied for foreach() in /home/bud-net/ftp/praca_dyplomowa/crest/vendor/fuzzysteve/crestlibrary/src/CrestLibrary/Client.php on line 90

Warning: Invalid argument supplied for foreach() in /home/bud-net/ftp/praca_dyplomowa/crest/vendor/fuzzysteve/crestlibrary/src/CrestLibrary/Client.php on line 168

Warning: Invalid argument supplied for foreach() in /home/bud-net/ftp/praca_dyplomowa/crest/vendor/fuzzysteve/crestlibrary/src/CrestLibrary/Client.php on line 168
string(20) "{"buy":[],"sell":[]}"

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#99 - 2014-11-27 23:15:00 UTC  |  Edited by: Steve Ronuken
Callback?


The url shouldn't need to be changed at all. you just need to fill in the app id, secret and a refresh token.

If you use postman to do the initial authentication, you should have a refresh token available to you (no idea how it'll make it available to you, I'm afraid)

This code doesn't do the initial auth for you, it's for use after that, once you've authed, and have the token. (it's provided at the same time as you get an access token)

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
Max Kolonko
Caldari Provisions
Caldari State
#100 - 2014-11-27 23:16:13 UTC  |  Edited by: Max Kolonko
Steve Ronuken wrote:
Callback?


The url shouldn't need to be changed at all. you just need to fill in the app id, secret and a refresh token.

If you use postman to do the initial authentication, you should have a refresh token available to you (no idea how it'll make it available to you, I'm afraid)

This code doesn't do the initial auth for you, it's for use after that, once you've authed, and have the token.


ahhh, got it... postman shows refresh token as blank :( so i'll wait for someone with more expierience figure it out :) I can only assume that You plan on expanding it to full auth support including login process?

Read and support: Don't mess with OUR WH's What is Your stance on WH stuff?