EVE Technology Lab

Removed a redundant post.

*

Is there documentation about the authentication process to access the publicData scope on auth'ed CREST? I'm looking for buy/sell orders. It seems to be different than the SSO protocol, as there wouldn't be a user login step.

I've hit an issue with walking through CREST to get URIs, rather than constructing them.

I'm trying to get info about a system, given its ID (obtained from XML API). To get a URI for my system, I can go to the CREST root, find the regions endpoint, find the correct region, find the correct constellation, then the system will be one of the available hrefs.

Problem: I don't know the region or constellation, all I have is a solar system ID

It is trivial to construct a link to it. I can just go to /solarsystems/{id}

To do it without manually coding in that link would require either walking through to a random solarsystem and then doing string operations to extract the part I need from its URI, or walking through the entire universe map, and get all the solar systems into my own DB. In the second case I'll still have to do a string operation to extract the ID from the end of the href, as id does not seem to be returned anywhere.

Parsing the provided URIs to extract information seems even more likely to break and harder to bug-fix than just constructing the route.

I assume that finding details of a system by its id is something that should be easy from CREST, given that all the data is there and it must be a fairly common use. This is a very common use of the SDE (and would be trivial to accomplish), but I'm trying to move as much as I can away from SDE to CREST.

To me it seem like the easiest solution would be to add the solarsystems endpoint to the root list (and maybe give it an index)

Am I really stupidly missing something?

Also - looking for names have added problem of names in general changing overtime (look at item tiericide for example)

This is a problem in protocol negotiation. For some reason in some constellations php-curl doesn't negotiate TLS properly, so you have to set it explicitly:
CURLOPT_SSL_CIPHER_LIST => 'TLSv1'

@CCP FoxFour:
Ciphersuites are configured inconsistently across the different https domains, public-crest for instance is still running with rc4-md5. And SSLv3 ofc.

You're not from around these parts are you m80?

Working as coded.

Feel free to move this post if it's in the wrong thread.

Came across an endpoint that is throwing a server error:

http://public-crest.eveonline.com/inventory/groups/11/

message: "Unexpected exception thrown of type type 'exceptions.TypeError' ",
key: "unexpectedException",
exceptionType: "InternalServerError",
refID: "50acbc30-6ec4-4015-9101-3e2b616053d1"

(note removed some brackets that were causing the forums to yell at me about potential html code)

This endpoint is within the http://public-crest.eveonline.com/inventory/groups/ endpoint, and it's label is "Asteroids OLD"

GG

I'm gonna try to elaborate on a few things that I think are suboptimal about the current CREST implementation. Please understand as constructive criticism.

Unfortunately atm CREST is not self-describing enough to allow for clients to be developed in a more meta/generic way. There is the OPTIONS call which gives some info about the responses, but lots of knowledge about how to use the individual endpoints ends up having to be hardcoded into the client. Does a certain endpoint require authentication? What scope? Is it available to third party developers at all? What are the necessary/possible request parameters?

As an example, we can take a look at the market orders endpoint.

• AFAIK there is no programmatically accessible place that specifies that it should be requested with a "type" parameter with value collected from the market types endpoint, so it has to be hardcoded.
• Why the parameter value can't be the actual typeID is beyond me. Also the current design ends up being inconsistent: The regionID is part of the actual URL, the type href is a param. For consistency it should be either /market/10000002/orders/buy/34/ or /market/orders/buy/?regionID=10000002&typeID=34. Either would have been so much nicer than the current implementation.
• Moreover, the info returned by CREST is misleading: If we do a GET request on it without parameters, the error response says "Both a regionID and typeID is required". The regionID is not a parameter and there is no "typeID" parameter, only "type".
• There is no possibility to query for a single or selected type's market hrefs. You have to get and parse all market type pages. Equally, you can't get a region, alliance or other similar endpoint without going through their respective collection pages first if you adhere to not constructing URLs.
• The result sets are not keyed by ID, you have to traverse potentially all items to their IDs to find a specific one.
• The pagination is silly. Since we cannot query for specific subsets, we will HAVE to query all pages anyway, before searching for the needed item. It only adds complexity and more requests. It should be optional with client specified max page size. Also there's no page index, so fancy things like parallel async CURL get are excluded from the start (unless you construct URLs).
User comfort is no argument for pagination, because there should be a cache in between and 1000 items per page are not user friendly anyway.

• Most cache expiry timers are unnecessarily short.

I have mentioned it before, but for completeness: The lack of OAuth client credential flow is "disturbing", requiring users to get a character-tied refresh token for otherwise public data.

CREST market order data has one big downside compared to what it is intended to replace, i.e. scraped data aggregated via EMDR:
With EMDR, item market data frequency correlates strongly with actual market activity. Trit in Jita gets frequent updates, obscure items in Feythabolis do not, and the data typically comes in when a user actually interacted with the market, i.e. caused changes. This is completely decoupled with CREST. There is no efficient way of getting the data that actually changed, or the hottest items. One has to scan the whole thing with 1.5 million requests. EMDR achieves the same effect with a fraction of the traffic and load (yes, the EMDR data is incomplete, but hardly in the relevant parts).
Unfortunately I don't see an obvious/easy solution for this apart from CCP offering a push service, which is what EMDR is, and completely different from CREST. Since I don't see this happening, the outcome will be that consumers will permanently scan as fast they can until they or CCP runs into load problems.

No but this is coming. https://gist.github.com/Regner/f360e9972fbfd05dad7a Each resource will have a section for params.

That's a step in the right direction, but still a long way from making endpoints generically accessible by clients.

I am hoping to get around to adding that as well. If your application is built only using CREST then often times you don't even have the ID, just a href. So we built it to use those. A lot of third-party devs happen to pull information from the SDE or the old XML API and have typeIDs and it is valid in those cases to want to use them.

I don't see why you'd want to get rid of / not expose IDs in the first place. More below.

Correct, but you can fetch those pages once per runtime of your application, or if your application runs through downtime, monitoring the serverVersion attribute from the root endpoint.

Depending on the app having the option of doing single item lookups would be desirable. Also since the clients respect cache timers you have occasional long lags while the app has to refetch all necessary collections after they expired.

This is true because again CREST was built to assume you are using CREST to get all of your information. So how would you possibly know about IDs? Because something has linked you there before. For example I had someone point out that the only way to find their market order information from CREST was to parse all of the orders just to find theirs. Thats because they had the order ID from the XML API. If done entirely in CREST you would have viewed the characters market orders, and that would have just linked to the market order directly. So yes, it's a big more work when not working entirely in CREST.

The thing is you're not keying by anything. Id, href, name... anything would be better than nothing for quick array access.

But more importantly, consider not moving away from IDs but introducing them consistently. They are universal across the XML API, the SDE and all other existing services and certainly won't hurt CREST. Ids are much nicer to handle. My guess is that most apps will handle IDs internally, not hrefs, except where they have to.

Client specified page sizes is out of the question as it ruins our ability to cache the pages, which we do on the NGINX layer.

Yup, I didn't have much hope on this one. Maybe the option to not paginate unless its a huge result set? That would cover the static multipage data so caching on your end wouldn't be an issue.

Now THATS a first I have heard. Care to give some examples?

The static data, i.e. the SDE equivalent comes with 1h expiry timer as far as I've seen. Since it only changes on expansions the timer could be much higher.

All the issues mentioned so far can be dealt with a well written client library, it's just that the API is not particularly consumer friendly. And the received data has to be massaged quite a bit depending on the use. The code I'm writing ends up having two caching layers, one for the immediate CREST responses, one for the concatenated pages, properly keyed by IDs and with only the useful data objects. This could be simpler if the API was more consumer oriented.

I would talk to the EVE Central guys about this. I know they built their service to crawl all the market data but they also prioritize it in a way where types that are viewed often are refreshed more often.

Yes, going by some metric to determine update frequency, for instance average number of transactions, is a workable approach. However it fails on sudden activity spikes for items that were previously rarely traded. This is an issue for 0.0 markets.

Thx for replying to my wall of text.