These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

EVE API and Public CREST discussion

First post First post First post
Author
brammator
SUB ZERO.
#701 - 2015-01-17 10:46:47 UTC  |  Edited by: brammator
Could you add list of base urls to https://developers.testeveonline.com/resource/crest ? Also, https://developers.testeveonline.com/resource/crest/public-crest need some explanation.

EDIT.

I mean, it would be very helpful to know:

Are those limits applied for ip or for application? For example, I may have different apps at one host, so should I worry about total amount of requests?

What is "burst"?

Is there any signs that I am near limit? Maybe you could add some header with numbers to crunch.
Kali Izia
GoomWaffe
#702 - 2015-01-17 23:01:32 UTC
The URLs are on https://developers.eveonline.com/resource/faq

Rate limits are per IP on public CREST.
On authed CREST there are rate limits but they're high enough that you can pretend they don't exist (for market data only, you should still implement rate limiting for all other endpoints).

For burst rates, read up on the leaky bucket and token bucket algorithms.
Your app can easily keep track of its own rate limits using either one of those.
brammator
SUB ZERO.
#703 - 2015-01-18 12:58:43 UTC
I'm using eveapi ( https://github.com/ntt/eveapi ) python library for XML api along with my own caching linrary ( https://github.com/brammator/glomp )

Storing api call results with (host, path, params) key goes well with XML eve api: it separates TQ from SiSi by host, inherently does security (via keyID=, vCode= in api call params) etc.

It doesn't go well with CREST: security code ("bearer" token) TTL is too short and passed differently (via HTTP header, not api call parameter), enpoints are duplicated on different hosts ("public" and "authed").

I could rewrite CREST API library to add refresh_token or app secret code to "params" list, but is it good solution? And what about data host deduplication?
Ortho Loess
The Legion of Spoon
Curatores Veritatis Alliance
#704 - 2015-01-18 13:39:56 UTC
CREST and the XML APIs are very different. (Very, very different) if you want to write a CREST library, there will be relatively little you can reuse from a library written for XML.

I don't know of a python CREST library as yet, there may be one though.
brammator
SUB ZERO.
#705 - 2015-01-18 14:32:11 UTC
There is some good libs in development, I even abandon mine in favor of one of them ;-) Data is data, anyway, and api libs look pretty familiar, XML and CREST.
Frankster Ijonen
Butlerian Crusade
#706 - 2015-01-21 14:46:36 UTC
Ortho Loess wrote:
I don't know of a python CREST library as yet, there may be one though.


I've been playing with https://github.com/jgoldshlag/crespy - its simple but elegant. No SSO auth though.
CCP FoxFour
C C P
C C P Alliance
#707 - 2015-01-21 15:04:33 UTC
https://wiki.eveonline.com/en/wiki/CREST_Libraries

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Sentenced 1989
#708 - 2015-02-02 14:18:19 UTC
Am I missing here something obvious or I'm just blind and or inexperienced with REST?

Is there any list of all the endpoints and parameters they take? So far I've found all the endpoints when I access CREST root, but I don't see which parameters are needed for which endpoint and how are they called, is it "id", "ids", "name", "names", etc...
Aineko Macx
#709 - 2015-02-17 13:21:43 UTC
Here's one example of how the current API design is impractical for users:

The scenario is loading the solarsystems endpoint for a given system name, a fairly standard task.
But thanks to the RESTful design, there's no endpoint listing systems by ID and name, but rather a hierarchy of regions pointing to constellations pointing to solar systems. Since we don't know in which region or constellation the system is, you have to traverse all of them, including the solarsystem endpoints themselves to get the system name, resulting in over 9200 calls to the API.

Of course the consumer can massage the data into a useful form and cache/persist that, but the problems of what should be a trivial API remain. Especially when you consider it is intended to be able to replace the SDE at some point. This is symptomatic of the problems one keeps uncovering while working with CREST, so I'd like to point to this article (API design matters), ask the CREST developer(s) to read it and to memorize "Is there anything I could reasonably do for the caller I am not doing?".

Another example is the wars endpoint:
The endpoint lists over 400k wars. There's no attribute by which you can find/filter them like dates, aggressor or defender. Guess what you're forced to do to find specific ones...
Ortho Loess
The Legion of Spoon
Curatores Veritatis Alliance
#710 - 2015-02-17 23:53:20 UTC
Aineko Macx wrote:
Here's one example of how the current API design is impractical for users:

The scenario is loading the solarsystems endpoint for a given system name, a fairly standard task.
But thanks to the RESTful design, there's no endpoint listing systems by ID and name, but rather a hierarchy of regions pointing to constellations pointing to solar systems. Since we don't know in which region or constellation the system is, you have to traverse all of them, including the solarsystem endpoints themselves to get the system name, resulting in over 9200 calls to the API.

[...]


There's one extra issue with the current hierarchy:

In theory it would be good for a selection interface. If a user wants a system, they select a region, then a constellation, then finally a system. However, this is going to completely break down, due to the fact that almost no one ever uses the constellation names. I couldn't name a single constellation in EVE, and couldn't pick the name of the constellation I live in out of a list of those in Providence

Realistically, for the vast majority of use cases, you either want the whole list of solar systems by name or all those in a specific region, there's no reason why a single call shouldn't be able to do either of these.

That's in an ideal world, once CREST is fully implemented and the XML API and SDE are gone. Until then, we will always be using data keyed on IDs, so that needs to be in the main list too.
Lucas Kell
Solitude Trading
#711 - 2015-02-19 12:02:55 UTC
That's a common problem with RESTful APIs. Because you have to be able to arrive at an entry URI then walk through the API, it has to be broken down into the steps it takes to provide you the information with each step, then to go back up a level you need to backtrack. They are great for providing individual pieces of functionality or information, but pretty terrible at just providing a bulk of information, which the regular EVE API does quite well. I wonder how useful to sites like EVE-central the market API will actually be in the long run, as I do for zKB and kill data if that is ever made fully browsable.

The Indecisive Noob - EVE fan blog.

Wholesale Trading - The new bulk trading mailing list.

Aineko Macx
#712 - 2015-02-20 09:27:11 UTC
The following two things could improve the situation a lot:
- Parallel to the RESTful structuring of the API, introduce endpoints with flat collections (no hierarchies)
- Collection endpoints need to be filterable via parameters. This hurts caching, but improves utility/usability immensely, bringing CREST a little bit closer to what the SDE can provide.

And another matter: We really need some clarification about what "substantially higher" rate limit means. I will soon unleash a CREST library with parallel async call support that can reach upward of 120 reqs per second on small responses if not curtailed.
Cadiz Aragon
Brutor Tribe
Minmatar Republic
#713 - 2015-02-23 00:11:34 UTC
So in marketSellOrders, the type object contains id_str & id fields, but in itemTypes only href and name fields are found. Are these not supposed to be the same object? It would be really helpful to find the id_str and id fields in the itemTypes endpoint. Thanks.
Vivien Meally
Liga Freier Terraner
Northern Coalition.
#714 - 2015-03-12 11:32:57 UTC
Hello,
is it possible to get Character info by id or by name like in old api /eve/CharacterID.xml.aspx und /eve/CharacterInfo.xml.aspx work?
Ortho Loess
The Legion of Spoon
Curatores Veritatis Alliance
#715 - 2015-03-12 15:37:35 UTC
Vivien Meally wrote:
Hello,
is it possible to get Character info by id or by name like in old api /eve/CharacterID.xml.aspx und /eve/CharacterInfo.xml.aspx work?


The list of endpoints includes one for characters, but it's not implemented yet.

It's coming, but so far, of the main 3, we only have alliances (released because it was needed for the AT stuff).
Aan Avast
The Scope
Gallente Federation
#716 - 2015-03-13 07:51:36 UTC
Idea: Allow us to query the current active missions
Description: i would like to access this information to have a better experience while mission running. It would allow me to look up eve-survival information automatically without the hassle of manual interaction.
xHjfx
The Legion of Spoon
Curatores Veritatis Alliance
#717 - 2015-03-22 05:12:43 UTC
FoxFour - Can we have closed entities included in the ContactList XML? or a Standings endpoint :p

I poked you on this ages ago but it appears to have gotten lost on your list :(
Aineko Macx
#718 - 2015-03-25 15:21:03 UTC  |  Edited by: Aineko Macx
Seeing that we'll soon have more than one authentication scope, can we please get a data field in the returned data of the OPTIONS call that tells us which scope is required for a certain endpoint?

Why is there nested json encoded data in the OPTIONS call response?

Why is the requested representation ignored? Regardless of the Accept string passed for the representation, the newest version is always returned (you can even pass garbage, with no effect).
CCP FoxFour
C C P
C C P Alliance
#719 - 2015-03-25 16:32:54 UTC
Aineko Macx wrote:
Seeing that we'll soon have more than one authentication scope, can we please get a data field in the returned data of the OPTIONS call that tells us which scope is required for a certain endpoint?

Why is there nested json encoded data in the OPTIONS call response?

Why is the requested representation ignored? Regardless of the Accept string passed for the representation, the newest version is always returned (you can even pass garbage, with no effect).


1) Maybe, I will look into it.
2) Dunno, didn't create it.
3) For the options call?

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Aineko Macx
#720 - 2015-03-25 17:34:02 UTC  |  Edited by: Aineko Macx
CCP FoxFour wrote:
3) For the options call?

No, for GET requests. So far I've only seen it working for the root endpoint, reacting to older version requests. ItemType for instance ignores it. Requesting with bad strings is also silently ignored.