These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

EVE API and Public CREST discussion

First post First post First post
Author
Max Kolonko
Caldari Provisions
Caldari State
#681 - 2014-12-14 22:16:48 UTC
Steve Ronuken wrote:
The Content-Type is used to specify specific version of the data returned (Well, the Accept header is. The content-type header is the response for that.


Whatever the intent is, R does not see it as JSON unless it have valid application/json content-type. I'm to new to this stuff to question it, that's what I observed. Luckily I can just tell it that its jason whatever the content-type is.

OR its just a problem with a binary response, where it have to be explicitly stated even if content-type was app/json?? like I said, I'm to new to both: OAUTH and R. I'm just glad that I managed to do something useful for me that actually works pretty stable so far :)

As I go along my course on R I might try to play with data from the market for some simple modeling and having it working from within R environment and not having to get the data from elsewhere is great as it speeds up analysis
ISD Dorrim Barstorlode
ISD Community Communications Liaisons
#682 - 2014-12-15 00:18:05 UTC
Removed a redundant post.

ISD Dorrim Barstorlode

Senior Lead

Community Communication Liaisons (CCLs)

Interstellar Services Department

Aineko Macx
#683 - 2014-12-15 12:23:05 UTC
Aineko Macx wrote:
https://public-crest.eveonline.com/market/prices/ is not returning data for typeID=34396 (Self-Assembling Nanolattice).

This is a thing. Can't calculate Confessor builds atm because of the missing data.
TheSmokingHertog
Julia's Interstellar Trade Emperium
#684 - 2014-12-15 18:02:14 UTC  |  Edited by: TheSmokingHertog
*

"Dogma is kind of like quantum physics, observing the dogma state will change it." ~ CCP Prism X

"Schrödinger's Missile. I dig it." ~ Makari Aeron

-= "Brain in a Box on Singularity" - April 2015 =-

Kali Izia
GoomWaffe
#685 - 2014-12-16 05:36:39 UTC
Aineko Macx wrote:
Aineko Macx wrote:
https://public-crest.eveonline.com/market/prices/ is not returning data for typeID=34396 (Self-Assembling Nanolattice).

This is a thing. Can't calculate Confessor builds atm because of the missing data.

That's not really an API issue since the data isn't in-game either.
I can't recall how often the prices get updated but it just means that script hasn't run yet.
Epitropy
Epitropy Corporation
#686 - 2014-12-16 06:09:34 UTC
Is there documentation about the authentication process to access the publicData scope on auth'ed CREST? I'm looking for buy/sell orders. It seems to be different than the SSO protocol, as there wouldn't be a user login step.
Kali Izia
GoomWaffe
#687 - 2014-12-16 08:38:18 UTC
Epitropy wrote:
Is there documentation about the authentication process to access the publicData scope on auth'ed CREST? I'm looking for buy/sell orders. It seems to be different than the SSO protocol, as there wouldn't be a user login step.

You still need to authenticate as a user. You more or less follow the normal SSO steps once (I posted the exact differences here), after that you can store the refresh token and won't have to log in again.

For a server-side app that will only use public data, you can just log in yourself once and store your own refresh token if you don't want to make users log in. The refresh token doesn't expire and will stay valid unless you revoke access.

Once you have the refresh token, for any future calls you'll only need to call /oauth/token to get an access token. And then you can access the CREST resources.
Ortho Loess
The Legion of Spoon
Curatores Veritatis Alliance
#688 - 2014-12-16 11:38:47 UTC
I've hit an issue with walking through CREST to get URIs, rather than constructing them.

I'm trying to get info about a system, given its ID (obtained from XML API). To get a URI for my system, I can go to the CREST root, find the regions endpoint, find the correct region, find the correct constellation, then the system will be one of the available hrefs.

Problem: I don't know the region or constellation, all I have is a solar system ID

It is trivial to construct a link to it. I can just go to /solarsystems/{id}

To do it without manually coding in that link would require either walking through to a random solarsystem and then doing string operations to extract the part I need from its URI, or walking through the entire universe map, and get all the solar systems into my own DB. In the second case I'll still have to do a string operation to extract the ID from the end of the href, as id does not seem to be returned anywhere.

Parsing the provided URIs to extract information seems even more likely to break and harder to bug-fix than just constructing the route.

I assume that finding details of a system by its id is something that should be easy from CREST, given that all the data is there and it must be a fairly common use. This is a very common use of the SDE (and would be trivial to accomplish), but I'm trying to move as much as I can away from SDE to CREST.

To me it seem like the easiest solution would be to add the solarsystems endpoint to the root list (and maybe give it an index)

Am I really stupidly missing something?
Max Kolonko
Caldari Provisions
Caldari State
#689 - 2014-12-16 16:47:58 UTC
Same goes for items, regions, constelations, market groups, etc.
Max Kolonko
Caldari Provisions
Caldari State
#690 - 2014-12-16 16:49:55 UTC
Also - looking for names have added problem of names in general changing overtime (look at item tiericide for example)
Ydnari
Estrale Frontiers
#691 - 2014-12-16 20:59:49 UTC
CCP FoxFour wrote:
Admittedly one of the main reasons we want to do this is because we have to maintain a separate configuration for both the HTTP and HTTPS version of the service. That sucks and we don't want to.

For your concerns of mobile power consumption: Once Authed CREST is out with actual private data, I hardly see people using Public CREST on mobile. They would be getting private data through Authed CREST. If you are getting private data from Authed CREST it's not worth the added complexity of getting data from both Authed and Public CREST. Add in some basic client side caching, and things should be fine on mobile. Public CREST will mainly be used by... well I am not entirely sure at this point now that Authed CREST is out with a much higher rate limit.

HTTPS shouldn't be any harder to work with these days. I don't know of any added complexity to working with it.


Do you have any comment on the oddity with the CREST SSL configuration that makes it awkward to access from a relatively common configuration of OpenSSL and clients compiled against it?

https://forums.eveonline.com/default.aspx?g=posts&m=4998975

I don't know the cause, but it's the only HTTPS endpoint I've seen that has ever exhibited it, but there are various reports of the same symptoms on a small minority of other sites from a casual Googling.

Whether it's useful or not, here's a link to the findings from a popular SSL tester. You probably want to drop SSL3 support server-side for starters:
https://www.ssllabs.com/ssltest/analyze.html?d=public-crest.eveonline.com&s=87.237.38.231

--

Aineko Macx
#692 - 2014-12-17 08:14:52 UTC  |  Edited by: Aineko Macx
Ydnari wrote:
Do you have any comment on the oddity with the CREST SSL configuration that makes it awkward to access from a relatively common configuration of OpenSSL and clients compiled against it?

https://forums.eveonline.com/default.aspx?g=posts&m=4998975

I don't know the cause, but it's the only HTTPS endpoint I've seen that has ever exhibited it, but there are various reports of the same symptoms on a small minority of other sites from a casual Googling.

Whether it's useful or not, here's a link to the findings from a popular SSL tester. You probably want to drop SSL3 support server-side for starters:
https://www.ssllabs.com/ssltest/analyze.html?d=public-crest.eveonline.com&s=87.237.38.231

This is a problem in protocol negotiation. For some reason in some constellations php-curl doesn't negotiate TLS properly, so you have to set it explicitly:
CURLOPT_SSL_CIPHER_LIST => 'TLSv1'

@CCP FoxFour:
Ciphersuites are configured inconsistently across the different https domains, public-crest for instance is still running with rc4-md5. And SSLv3 ofc.
EdwardNardella
Deep Core Mining Inc.
Caldari State
#693 - 2014-12-22 20:08:55 UTC
Idea: Provide Documentation
Description: Write and publish comprehensive documents detailing how to access the system. Provide examples on how to integrate it into applications, websites and spreadsheets. Provide detailed descriptions of all the features of the system and how to use each of the features.
Fifth Blade
Jump Drive Appreciation Society
#694 - 2014-12-22 20:19:31 UTC
EdwardNardella wrote:
Idea: Provide Documentation
Description: Write and publish comprehensive documents detailing how to access the system. Provide examples on how to integrate it into applications, websites and spreadsheets. Provide detailed descriptions of all the features of the system and how to use each of the features.

You're not from around these parts are you m80?

Working as coded.
Ortho Loess
The Legion of Spoon
Curatores Veritatis Alliance
#695 - 2014-12-23 12:19:11 UTC  |  Edited by: Ortho Loess
I was pretty hopeful about the new dev site. When it went up, it had some useful info on some of the services and promises of more to come. However, even this new source of documentation has already been allowed to go out of date. e.g. It still says that the returned auth token has a null value for the refresh token and that the auth code only last 5 minutes, not 30.

Just yesterday, I managed to forget the tq root for authed CREST. Amazingly this is not in the documentation (nor are any other roots...) and not very easy to guess. Like most other info, I found it by trawling through forum threads.

FoxFour is doing excellent work on the third party tools, but SOMEONE needs to write documentation

p.s. FoxFour on holiday for Christmas? Not had a response to my last post yet :(

[edit: Turns out I just didn't look long enough, found all the root urls on the FAQ page (feel they should still be in the relevant areas as well though...)]
Greygal
Redemption Road
Affirmative.
#696 - 2014-12-24 16:42:49 UTC
Feel free to move this post if it's in the wrong thread.

Came across an endpoint that is throwing a server error:

http://public-crest.eveonline.com/inventory/groups/11/


message: "Unexpected exception thrown of type type 'exceptions.TypeError' ",
key: "unexpectedException",
exceptionType: "InternalServerError",
refID: "50acbc30-6ec4-4015-9101-3e2b616053d1"


(note removed some brackets that were causing the forums to yell at me about potential html code)

This endpoint is within the http://public-crest.eveonline.com/inventory/groups/ endpoint, and it's label is "Asteroids OLD"

GG

What you do for yourself dies with you, what you do for others is immortal.

Free weekly public roams & monthly NewBro new player roams!

Visit Redemption Road or join mailing list REDEMPTION ROAMS for information

bogdica19 bodica
Ministry of War
Amarr Empire
#697 - 2014-12-29 09:14:24 UTC
Sorry if i'm not on the right forum thread, but i was wondering if there is anyone that can help me get some guide, documentation or examples on how to use the data retrieved by the api key in a little application. Bassically i want to make an application that people can use on phones and tablets to view any details regarding their character(isk, assets, skill training and so on). I'll be using mainly php/javascript.
If anyone can point me in the right direction that will be really nice.
Thanks for your time!

P.S. i can be contacted at [email protected]
Aineko Macx
#698 - 2015-01-12 18:36:41 UTC  |  Edited by: Aineko Macx
I'm gonna try to elaborate on a few things that I think are suboptimal about the current CREST implementation. Please understand as constructive criticism.

Unfortunately atm CREST is not self-describing enough to allow for clients to be developed in a more meta/generic way. There is the OPTIONS call which gives some info about the responses, but lots of knowledge about how to use the individual endpoints ends up having to be hardcoded into the client. Does a certain endpoint require authentication? What scope? Is it available to third party developers at all? What are the necessary/possible request parameters?

As an example, we can take a look at the market orders endpoint.

  • AFAIK there is no programmatically accessible place that specifies that it should be requested with a "type" parameter with value collected from the market types endpoint, so it has to be hardcoded.
  • Why the parameter value can't be the actual typeID is beyond me. Also the current design ends up being inconsistent: The regionID is part of the actual URL, the type href is a param. For consistency it should be either /market/10000002/orders/buy/34/ or /market/orders/buy/?regionID=10000002&typeID=34. Either would have been so much nicer than the current implementation.
  • Moreover, the info returned by CREST is misleading: If we do a GET request on it without parameters, the error response says "Both a regionID and typeID is required". The regionID is not a parameter and there is no "typeID" parameter, only "type".
  • There is no possibility to query for a single or selected type's market hrefs. You have to get and parse all market type pages. Equally, you can't get a region, alliance or other similar endpoint without going through their respective collection pages first if you adhere to not constructing URLs.
  • The result sets are not keyed by ID, you have to traverse potentially all items to their IDs to find a specific one.
  • The pagination is silly. Since we cannot query for specific subsets, we will HAVE to query all pages anyway, before searching for the needed item. It only adds complexity and more requests. It should be optional with client specified max page size. Also there's no page index, so fancy things like parallel async CURL get are excluded from the start (unless you construct URLs).
  • User comfort is no argument for pagination, because there should be a cache in between and 1000 items per page are not user friendly anyway.
  • Most cache expiry timers are unnecessarily short.

I have mentioned it before, but for completeness: The lack of OAuth client credential flow is "disturbing", requiring users to get a character-tied refresh token for otherwise public data.

CREST market order data has one big downside compared to what it is intended to replace, i.e. scraped data aggregated via EMDR:
With EMDR, item market data frequency correlates strongly with actual market activity. Trit in Jita gets frequent updates, obscure items in Feythabolis do not, and the data typically comes in when a user actually interacted with the market, i.e. caused changes. This is completely decoupled with CREST. There is no efficient way of getting the data that actually changed, or the hottest items. One has to scan the whole thing with 1.5 million requests. EMDR achieves the same effect with a fraction of the traffic and load (yes, the EMDR data is incomplete, but hardly in the relevant parts).
Unfortunately I don't see an obvious/easy solution for this apart from CCP offering a push service, which is what EMDR is, and completely different from CREST. Since I don't see this happening, the outcome will be that consumers will permanently scan as fast they can until they or CCP runs into load problems.
CCP FoxFour
C C P
C C P Alliance
#699 - 2015-01-15 11:49:01 UTC
Aineko Macx wrote:
I'm gonna try to elaborate on a few things that I think are suboptimal about the current CREST implementation. Please understand as constructive criticism.

Unfortunately atm CREST is not self-describing enough to allow for clients to be developed in a more meta/generic way. There is the OPTIONS call which gives some info about the responses, but lots of knowledge about how to use the individual endpoints ends up having to be hardcoded into the client. Does a certain endpoint require authentication? What scope? Is it available to third party developers at all? What are the necessary/possible request parameters?

As an example, we can take a look at the market orders endpoint.

  • AFAIK there is no programmatically accessible place that specifies that it should be requested with a "type" parameter with value collected from the market types endpoint, so it has to be hardcoded.
  • Why the parameter value can't be the actual typeID is beyond me. Also the current design ends up being inconsistent: The regionID is part of the actual URL, the type href is a param. For consistency it should be either /market/10000002/orders/buy/34/ or /market/orders/buy/?regionID=10000002&typeID=34. Either would have been so much nicer than the current implementation.
  • Moreover, the info returned by CREST is misleading: If we do a GET request on it without parameters, the error response says "Both a regionID and typeID is required". The regionID is not a parameter and there is no "typeID" parameter, only "type".
  • There is no possibility to query for a single or selected type's market hrefs. You have to get and parse all market type pages. Equally, you can't get a region, alliance or other similar endpoint without going through their respective collection pages first if you adhere to not constructing URLs.
  • The result sets are not keyed by ID, you have to traverse potentially all items to their IDs to find a specific one.
  • The pagination is silly. Since we cannot query for specific subsets, we will HAVE to query all pages anyway, before searching for the needed item. It only adds complexity and more requests. It should be optional with client specified max page size. Also there's no page index, so fancy things like parallel async CURL get are excluded from the start (unless you construct URLs).
  • User comfort is no argument for pagination, because there should be a cache in between and 1000 items per page are not user friendly anyway.
  • Most cache expiry timers are unnecessarily short.

I have mentioned it before, but for completeness: The lack of OAuth client credential flow is "disturbing", requiring users to get a character-tied refresh token for otherwise public data.

CREST market order data has one big downside compared to what it is intended to replace, i.e. scraped data aggregated via EMDR:
With EMDR, item market data frequency correlates strongly with actual market activity. Trit in Jita gets frequent updates, obscure items in Feythabolis do not, and the data typically comes in when a user actually interacted with the market, i.e. caused changes. This is completely decoupled with CREST. There is no efficient way of getting the data that actually changed, or the hottest items. One has to scan the whole thing with 1.5 million requests. EMDR achieves the same effect with a fraction of the traffic and load (yes, the EMDR data is incomplete, but hardly in the relevant parts).
Unfortunately I don't see an obvious/easy solution for this apart from CCP offering a push service, which is what EMDR is, and completely different from CREST. Since I don't see this happening, the outcome will be that consumers will permanently scan as fast they can until they or CCP runs into load problems.



AFAIK there is no programmatically accessible place that specifies that it should be requested with a "type" parameter with value collected from the market types endpoint, so it has to be hardcoded.

No but this is coming. https://gist.github.com/Regner/f360e9972fbfd05dad7a Each resource will have a section for params.

Why the parameter value can't be the actual typeID is beyond me.

I am hoping to get around to adding that as well. If your application is built only using CREST then often times you don't even have the ID, just a href. So we built it to use those. A lot of third-party devs happen to pull information from the SDE or the old XML API and have typeIDs and it is valid in those cases to want to use them.

Also the current design ends up being inconsistent: The regionID is part of the actual URL, the type href is a param. For consistency it should be either /market/10000002/orders/buy/34/ or /market/orders/buy/?regionID=10000002&typeID=34. Either would have been so much nicer than the current implementation.

Actually this isn't inconsistent at all. Technically if we follow the design pattern /market/10000002/orders/buy/ should return all market buy orders for the region regardless of type. The type parameter is a filter. /market/regionID/orders/buy/orderID is the next step in the URI. /market/regionID/orders/buy/ is also where the post request goes for creating a new buy order. /market/regionID/orders/buy/ is a collection or buy orders, and following RESTful design should, and does just not exposed to third-party devs yet, a way of viewing an individual item from that collection. The parameters should then filter, which is what we have in the above design. The only questionable part of that is the region ID being part of the URI and not a parameter, but that's because of the way our backend works. the market is divided into regions and must be viewed per region, it's not an optional parameter. Again, type should really be optional... that might be the inconsistent part.

[b]Moreover, the info returned by CREST is misleading: If we do a GET request on it without parameters, the error response says...

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Aineko Macx
#700 - 2015-01-15 13:58:02 UTC  |  Edited by: Aineko Macx
No but this is coming. https://gist.github.com/Regner/f360e9972fbfd05dad7a Each resource will have a section for params.

That's a step in the right direction, but still a long way from making endpoints generically accessible by clients.

I am hoping to get around to adding that as well. If your application is built only using CREST then often times you don't even have the ID, just a href. So we built it to use those. A lot of third-party devs happen to pull information from the SDE or the old XML API and have typeIDs and it is valid in those cases to want to use them.

I don't see why you'd want to get rid of / not expose IDs in the first place. More below.

Correct, but you can fetch those pages once per runtime of your application, or if your application runs through downtime, monitoring the serverVersion attribute from the root endpoint.

Depending on the app having the option of doing single item lookups would be desirable. Also since the clients respect cache timers you have occasional long lags while the app has to refetch all necessary collections after they expired.

This is true because again CREST was built to assume you are using CREST to get all of your information. So how would you possibly know about IDs? Because something has linked you there before. For example I had someone point out that the only way to find their market order information from CREST was to parse all of the orders just to find theirs. Thats because they had the order ID from the XML API. If done entirely in CREST you would have viewed the characters market orders, and that would have just linked to the market order directly. So yes, it's a big more work when not working entirely in CREST.

The thing is you're not keying by anything. Id, href, name... anything would be better than nothing for quick array access.

But more importantly, consider not moving away from IDs but introducing them consistently. They are universal across the XML API, the SDE and all other existing services and certainly won't hurt CREST. Ids are much nicer to handle. My guess is that most apps will handle IDs internally, not hrefs, except where they have to.

Client specified page sizes is out of the question as it ruins our ability to cache the pages, which we do on the NGINX layer.

Yup, I didn't have much hope on this one. Maybe the option to not paginate unless its a huge result set? That would cover the static multipage data so caching on your end wouldn't be an issue.

Now THATS a first I have heard. Care to give some examples?

The static data, i.e. the SDE equivalent comes with 1h expiry timer as far as I've seen. Since it only changes on expansions the timer could be much higher.

All the issues mentioned so far can be dealt with a well written client library, it's just that the API is not particularly consumer friendly. And the received data has to be massaged quite a bit depending on the use. The code I'm writing ends up having two caching layers, one for the immediate CREST responses, one for the concatenated pages, properly keyed by IDs and with only the useful data objects. This could be simpler if the API was more consumer oriented.

I would talk to the EVE Central guys about this. I know they built their service to crawl all the market data but they also prioritize it in a way where types that are viewed often are refreshed more often.

Yes, going by some metric to determine update frequency, for instance average number of transactions, is a workable approach. However it fails on sudden activity spikes for items that were previously rarely traded. This is an issue for 0.0 markets.

Thx for replying to my wall of text.