These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

EVE API and Public CREST discussion

First post First post First post
Author
Malus Sentio
Subsidy H.R.S.
Xagenic Freymvork
#421 - 2014-05-29 22:48:10 UTC
CCP FoxFour wrote:
Malus Sentio wrote:
Idea: Allow us to query entire killmail history
Description: We need to be able to pull entire killmail histories to be able to develop better killboards that can compete with the established ones. Killboards play a massive part in telling the stories that happen in EVE.


Thats not a very good argument considering established killboards (aka zKillboard) offers an API to pull all the killmails they have.

You don't need us to help you compete.

To be honest, they also store them in a slightly better and more efficient way. If they were not offering an API to get killmails, which is actually easier than getting them from us because you don't need API keys and such, then I would be more considering of this.



Firstly, thank you for even replying. This thread (aside from the WH nonsense) is really refreshing to see.

I appreciate what you're saying but there is no way I would want to spend hundreds of hours to develop something that is reliant on an external 3rd party, especially one that I am actually competing with. It would require me to trust them and be beholden to any changes/censorship they make. Furthermore any server issues they might have, changing of licensing/introduction of fees or if they just get bored and shutdown.

Also, there is the issue of completeness, which is something I believe noone can offer yet aside from CCP.

TheSmokingHertog
Julia's Interstellar Trade Emperium
#422 - 2014-05-30 01:24:59 UTC
CCP FoxFour wrote:
Ideki wrote:
Hey FoxFour, love the Dev Spolight.
always nice to be able to put a real face on a name Big smile


:) Glad you enjoyed it.


That he did not say :P...

"Dogma is kind of like quantum physics, observing the dogma state will change it." ~ CCP Prism X

"Schrödinger's Missile. I dig it." ~ Makari Aeron

-= "Brain in a Box on Singularity" - April 2015 =-

Sentenced 1989
#423 - 2014-05-30 02:00:53 UTC  |  Edited by: Sentenced 1989
CCP FoxFour wrote:
....
Awesome! That is exactly the way it should work and huge thanks for the user agent! :D

Any chance of sharing a link to what you are working on?


Yea, I'm pulling incursion data from your CREST into my server to see which incursions are up so I can created MOTD formatted messages that people can use (here is link:http://eve.qsna.eu/post.php?post_id=incursion_constellations_layout

So thank you for making this CREST endpoint, really helps out, otherwise I have to populate the selection box with about 40-50 options, this way it never has more then 3.

Ill prolly release the full source and might just make some youtube movie explaining how it works so maybe some part of it will be beneficial to somebody else trying to figure out this stuff.

Also, one small advice, create one page or wiki or whatever (but somewhere where only devs can modify) and please put a sample call for each API/CREST function. So talking here about sample call in normal browser. It would help many new devs to actually start, since it took me long time to even figure out how the older API calls worked since I couldn't find any example (not that much long once I decided to figure it out, but sometimes I just went there, saw the documentations and said, ah fu** it, some other time). Just having an list with functions isn't as clear as having an example line which can do query by itself by just copy paste into browser.
Sentenced 1989
#424 - 2014-06-01 17:18:29 UTC  |  Edited by: Sentenced 1989
As promised, here is how I handle crest request.

Excuse my hard english accent on video.
Also keep in mind my video stream cut so I talked for 10 more minutes without recording :D
Basically what is left to show, is when I lower the number in last_update.txt by 3600 or more, it will force new update from server.

http://youtu.be/mPwg4-lwQpg

So this is solution how to access eve CREST only on request (so not doing it every hour if there is no need) and how to check if there was any request in last hour to use local copy of data instead of making new request.



Source files:
http://pastebin.com/FjLPjaUK (main file)
http://pastebin.com/52ZNfqYm (proxy file)

Also, CCP don't get mad because I clicked 4-5 times get data in second :)
Wollari
Dirt Nap Squad
#425 - 2014-06-04 12:55:23 UTC  |  Edited by: Wollari
Problem with the war details endpoint:

1) http://public-crest.eveonline.com/wars/69195/

--> no timeFinished

2) https://api.eveonline.com/corp/corporationSheet.xml.aspx?corporationID=1595673139

--> memberCount = 0

3) Client

--> Not involved in any wars

I bet this war is over but got never closed

DOTLAN EveMaps | Your out-of-game map, navigation toolset, sov database, etc. since 2008

Wollari
Dirt Nap Squad
#426 - 2014-06-04 13:40:28 UTC  |  Edited by: Wollari
I found lots of mutual wars that are more then 2-3 years old from very small corporations (that might not exist any longer).

IMHO there should be a small maintenance fee for both parties instead of no-fee. Then all mutual wars from corps that no longer exists or only exists of non-subscribed alts would automatically end. But I know this could become a problem when one party want's to end the war single sided ... so bad anyway

btw. here's a broken war from 2014

war: https://public-crest.eveonline.com/wars/354187/
dead corp: https://api.eveonline.com/corp/corporationSheet.xml.aspx?corporationID=98110087

I managed to open the info window of the closed corporation
* [Closed] Heavy-Snowflake Collision
* 41 Active Wars
* 33 Finished Wars

DOTLAN EveMaps | Your out-of-game map, navigation toolset, sov database, etc. since 2008

Squizz Caphinator
The Wormhole Police
#427 - 2014-06-04 14:53:44 UTC
Page iteration on killmails isn't working.

These two links are identical:
https://public-crest.eveonline.com/wars/339618/killmails/all/
https://public-crest.eveonline.com/wars/339618/killmails/all/?page=2

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com

iskflakes
#428 - 2014-06-06 02:09:56 UTC
Idea: Loyalty points in the API
Description: Provide an endpoint to retrieve character loyalty points from various factions/corporations. Possible places to put this information include the /char/Standings endpoint, the account balance endpoint or character sheet.

Use case: Loyalty point spending/tracking applications. Character tracking applications (e.g. EVEMon).

-

Mann im Mond
Deep Core Mining Inc.
Caldari State
#429 - 2014-06-09 08:51:25 UTC
Idear: Add System list with the Incursion Profile to the Crest incursions endpoint or create a new one

If the Incursion gets closed and you need to change the current fokus for your Incursion group it is very helpfull if you get a dockup in a system with the Profile you prefere (Vanguards for me) , but atm you have to fly there at first to set the fokus and with my slow bs it take much more time than other pilots need. So it would be awesome to get the profile information over the API and set a proper dockup.

Protag Alduin
HF Salvage and Mining Corp.
#430 - 2014-06-10 07:06:27 UTC
Anyone else think that the number of planetary interaction API calls needed just to write a simple PI monitor is a bit excessive? I know planets don't change all that often, at least for me, but I think it would be nice if I could just send each of the endpoints a list of planet IDs rather than a single ID. That means making only 4 calls per character when a planet has been updated. Right now I have to make a PlanetaryColonies call, then a PlanetaryPins call for each planet, then a call to the other two endpoints for each updated planet for each character being monitored. I could manually stagger the calls instead of using the cachedUntil timer, but that just feels like I'm making too many calls, especially if my program is monitoring other endpoints.
Ideki
Wolf Brothers INC
United Neopian Federation
#431 - 2014-06-10 12:25:11 UTC
well I find that's it a lot of webcalls too.
But at least now we have a PI API.
That's better than nothing.
So I am not complaining too much.

So I am grateful to CCP FoxFour for his work.Big smile
Protag Alduin
HF Salvage and Mining Corp.
#432 - 2014-06-10 13:56:55 UTC
Ideki wrote:
well I find that's it a lot of webcalls too.
But at least now we have a PI API.
That's better than nothing.
So I am not complaining too much.

So I am grateful to CCP FoxFour for his work.Big smile


Absolutely. I didn't mean for it to come off as ungrateful. I love that I have access to the information now. It just made me feel like maybe I was using the interface wrong or something because I was having to call it so much.
Hawelt
Warpspeed Shipping Inc.
#433 - 2014-06-13 05:30:58 UTC  |  Edited by: Hawelt
So I've been playing around with the public crest market history end point here is an example and some observations:



  1. Threadnought: At over 20 pages I'll ignore 90% of them. Sorry if this feedback is made redundant by earlier posts. Maybe consider a seperate forum section with 1 thread for each api endpoint and some documentation in the original post.

  2. Documentation: abysmal. Its about the same level as the eviction notice for planet earth in HHGTTG: Burried in some forgotten basement filing cabinet on a planet in alpha centauri.

  3. Parameter names: /types/ implies the ability of providing a list, doesnt seem to work

  4. Lack of time related parameters. I'd like to fetch the latest update for a given region and/or list of types since foo-day without fetching everything from last year. Might exist but then again the documentation doesn't.

  5. Unclear semantics: We get data for a timespan but its labeled by a timestamp:
  6. {"totalCount_str": "397", "items": [{"volume_str": "3231", "orderCount": 2064, "lowPrice": 517592000.0, "highPrice": 527000000.0, "avgPrice": 523901000.0, "volume": 3231, "orderCount_str": "2064", "date": "2013-05-01T00:00:00"},

    Does 'date' denote the beginning, middle or the end or neither one of those ? At what time becomes the latest data point available ? Did I mention the lack of documentation already ?


I currently have an interface that fetches data from crest when it presumes to be missing data combined with a cron job to update previously requested (type,region) pairs daily. As it stands its both unreliable and inefficient due to the points mentioned above.

Sorry for ranting so much about the documentation, its pretty awesome to have this api. But is there any chance of getting a zmq api to subscribe to in order to receive updates as they become available?
CCP FoxFour
C C P
C C P Alliance
#434 - 2014-06-13 17:14:15 UTC
Hawelt wrote:
So I've been playing around with the public crest market history end point here is an example and some observations:



  1. Threadnought: At over 20 pages I'll ignore 90% of them. Sorry if this feedback is made redundant by earlier posts. Maybe consider a seperate forum section with 1 thread for each api endpoint and some documentation in the original post.

  2. Documentation: abysmal. Its about the same level as the eviction notice for planet earth in HHGTTG: Burried in some forgotten basement filing cabinet on a planet in alpha centauri.

  3. Parameter names: /types/ implies the ability of providing a list, doesnt seem to work

  4. Lack of time related parameters. I'd like to fetch the latest update for a given region and/or list of types since foo-day without fetching everything from last year. Might exist but then again the documentation doesn't.

  5. Unclear semantics: We get data for a timespan but its labeled by a timestamp:
  6. {"totalCount_str": "397", "items": [{"volume_str": "3231", "orderCount": 2064, "lowPrice": 517592000.0, "highPrice": 527000000.0, "avgPrice": 523901000.0, "volume": 3231, "orderCount_str": "2064", "date": "2013-05-01T00:00:00"},

    Does 'date' denote the beginning, middle or the end or neither one of those ? At what time becomes the latest data point available ? Did I mention the lack of documentation already ?


I currently have an interface that fetches data from crest when it presumes to be missing data combined with a cron job to update previously requested (type,region) pairs daily. As it stands its both unreliable and inefficient due to the points mentioned above.

Sorry for ranting so much about the documentation, its pretty awesome to have this api. But is there any chance of getting a zmq api to subscribe to in order to receive updates as they become available?



  1. 1 thread per endpoint is not feasible, there are far to many.
  2. Yes, but since the beginning of time CCP has basically stayed away from providing documentation for third parties. This goes for the SDE, API, CREST, and everything else. I do hope we can get some proper documentation for CREST out at some point, but don't expect anything from us for the older API and SDE.
  3. CREST is a restful API, /types/ is not a parameter. Parameter would be something after a ? such as ?page=2.
  4. I don't have any plans to provide anything like that for the market history resource. I do however have plans to release a resource that contains all market types and their current average price. That may help but it wont contain information like order count or anything like the history endpoint.
  5. Yea... that should probably drop the seconds, minutes, and hours leaving just the date. /shrug not enough of an issue for me to go fix it when the list of other things to fix (see war killmails being totally broken and PI endpoint needing lots of help) to put it very high on my list.
  6. Date denotes the day that information is for. This market history information is for per day, so yea. In your example above the volume, order count, and price information is all for the day of 2013-05-01.
  7. I recommend running all cron jobs for the market data at 01:00 every day and just leaving it at that.
  8. I have no idea what zmq is. But um... no not really. I recommend following this thread, specifically the third post.



@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Hawelt
Warpspeed Shipping Inc.
#435 - 2014-06-14 08:20:21 UTC  |  Edited by: Hawelt
CCP FoxFour wrote:
CREST is a restful API, /types/ is not a parameter. Parameter would be something after a ? such as ?page=2.


What is the proper RESTful api terminology for the typeID numbers in an uri like http://public-crest.eveonline.com/market/10000002/types/34/history/ ?

And is there a way of specifying more than one typeID and if not why is it /types/ instead of /type/ ? (Sorry for nitpicking)
When reading types I immediately assumed that just like with the zkb api I could get data for more than one typeID at a time and tried http://public-crest.eveonline.com/market/10000002/types/34,35,36,37,38/history/ with no success.


CCP FoxFour wrote:

I don't have any plans to provide anything like that for the market history resource. I do however have plans to release a resource that contains all market types and their current average price. That may help but it wont contain information like order count or anything like the history endpoint.


Yesterday I've sequentially fetched history data for the regions The Forge, Heimatar, Sinq Laison, Metropolis, Domain, Essence with all type IDs that have a marketGroupID. I haven't timed it exactly but it took somewhere around 4 hours with downtime causing some grief too.
At the very minimum I'd like to do daily updates for two to four regions for about 5000-10000 types because historic prices and volume for the last seven days are part of a ranking function I'd like to use when looking for profitable trades between market hubs.

As long as nobody complains about it I'm okay with hammering the server for an hour or two but it seems rather inefficient when I just want one out of 400ish elements in the individual results.

CCP FoxFour wrote:

I have no idea what zmq is. But um... no not really. I recommend following this thread, specifically the third post.


Its a messaging protocol which allows for various producer/consumer models. Currently the way of getting fresh market order data (as gathered by evemon et al. users) EMDR is using zmq to dispatch json strings to consumers in real-time.
For things that change rapidly like the currently active market orders it makes a lot of sense. For the case of yesterdays historic market data a restful api endpoint that doesnt involve me throwing way 99.75% of the results would be perfectly fine. :D

CCP FoxFour wrote:

I recommend running all cron jobs for the market data at 01:00 every day and just leaving it at that.


Awesome. I assume thats in UTC ?
CCP FoxFour
C C P
C C P Alliance
#436 - 2014-06-14 17:40:24 UTC
Well ****, forums just ate my response and I have to run. Will try and remember to respond properly on Monday.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Darkblad
Doomheim
#437 - 2014-06-14 18:20:08 UTC
CCP FoxFour wrote:
Well ****, forums just ate my response and I have to run. Will try and remember to respond properly on Monday.
Don't hesitate, make your Forums Gold

NPEISDRIP

CCP FoxFour
C C P
C C P Alliance
#438 - 2014-06-14 19:01:27 UTC
Darkblad wrote:
CCP FoxFour wrote:
Well ****, forums just ate my response and I have to run. Will try and remember to respond properly on Monday.
Don't hesitate, make your Forums Gold


Have that at work but not at home. :(

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Hawelt
Warpspeed Shipping Inc.
#439 - 2014-06-14 19:16:02 UTC
Darkblad wrote:
CCP FoxFour wrote:
Well ****, forums just ate my response and I have to run. Will try and remember to respond properly on Monday.
Don't hesitate, make your Forums Gold


That no-nom feature sounds delicious. Only a preventive application of copy & pasted saved me earlier today.
Risingson
#440 - 2014-06-16 15:16:34 UTC
Please add allianceID + allianceName to ConquerableStationList ?