These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

EVE API and Public CREST discussion

First post First post First post
Author
CCP FoxFour
C C P
C C P Alliance
#461 - 2014-07-01 18:04:05 UTC
Jinn Aideron wrote:
CCP FoxFour wrote:
Public CREST:

  • Added market/Prices
  • ** Returns a list of all items that have an average market price or adjusted market price and what their average or adjusted market price is.
    ** Average price is the same price you see when you select an item in your inventory.
    ** Adjusted price is what the industry formulas use.

Can someone point me to what adjusted is in this context? Adjusted how? Was this maybe mentioned elsewhere? Thank you.


Um... it just is that. It's an adjusted price, adjusted in what way we have not said publicly, that we use for things internally. One of those things we use it for is industry cost calculations. Due to that we chose to expose it to you guys since you will need it to calculate industrial costs.

It's use may grow over time so we have simply called it adjustedPrice.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Jinn Aideron
#462 - 2014-07-02 01:51:00 UTC
CCP FoxFour wrote:
Jinn Aideron wrote:
..
Um... it just is that. It's an adjusted price, adjusted in what way we have not said publicly, that we use for things internally. One of those things we use it for is industry cost calculations. Due to that we chose to expose it to you guys since you will need it to calculate industrial costs.

It's use may grow over time so we have simply called it adjustedPrice.

Alright, if it is undisclosed, I can stop sifting the forums/blogs for posts I missed. Blink

Does this mean industry cost calculations won't be predictable, forward-projectable for us then, being in the dark about how this value is derived, that we can only work on the last publication?

Stealth deletes are bad.

Rust Connor
Industrias PapaCapim
#463 - 2014-07-02 10:42:28 UTC  |  Edited by: Rust Connor
CCP FoxFour wrote:
This is whats on Sisi right now for the Crius release.
Public CREST:

  • Added market/Prices
  • ** Returns a list of all items that have an average market price or adjusted market price and what their average or adjusted market price is.
    ** Average price is the same price you see when you select an item in your inventory.
    ** Adjusted price is what the industry formulas use.
  • Added industry/facilities
  • ** Lists all public facilities within New Eden
  • Added industry/specialities
  • ** Lists all specialities that can be associated with teams and what groups they modify
  • Added industry/specialities/specialityID
  • ** Details what groups a specific speciality modifies
  • Added industry/teams
  • ** Lists all teams in active use in the known universe (not WH systems)
  • Added industry/systems
  • ** Lists the cost index of installing jobs in the known universe (not WH systems)
  • Added the CREST root to public CREST
  • ** Simply going to https://public-crest-sisi.eveonline.com/ will now return the CREST root


Industry/systems worked fine. Is there a minimum value for costIndex? I mean, for a system that has 0h productions right now. And what means costIndex? :)
Its the "fraction of global job hour"? I guess Ive missed some post about it.

Market/Prices didnt work.... "Not found error".
CCP FoxFour
C C P
C C P Alliance
#464 - 2014-07-02 11:32:00 UTC  |  Edited by: CCP FoxFour
Rust Connor wrote:
CCP FoxFour wrote:
This is whats on Sisi right now for the Crius release.
Public CREST:

  • Added market/Prices
  • ** Returns a list of all items that have an average market price or adjusted market price and what their average or adjusted market price is.
    ** Average price is the same price you see when you select an item in your inventory.
    ** Adjusted price is what the industry formulas use.
  • Added industry/facilities
  • ** Lists all public facilities within New Eden
  • Added industry/specialities
  • ** Lists all specialities that can be associated with teams and what groups they modify
  • Added industry/specialities/specialityID
  • ** Details what groups a specific speciality modifies
  • Added industry/teams
  • ** Lists all teams in active use in the known universe (not WH systems)
  • Added industry/systems
  • ** Lists the cost index of installing jobs in the known universe (not WH systems)
  • Added the CREST root to public CREST
  • ** Simply going to https://public-crest-sisi.eveonline.com/ will now return the CREST root


Industry/systems worked fine. Is there a minimum value for costIndex? I mean, for a system that has 0h productions right now. And what means costIndex? :)
Its the "fraction of global job hour"? I guess Ive missed some post about it.

Market/Prices didnt work.... "Not found error".


works for me: http://public-crest-sisi.testeveonline.com/market/prices/

Edit: be sure you get the trailing slash when compiling an actual URL.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Rust Connor
Industrias PapaCapim
#465 - 2014-07-02 11:41:43 UTC
Thanks!
It's case sensitive. I've tried "market/Prices/".....

And the costIndex? Any brief explanation?
CCP FoxFour
C C P
C C P Alliance
#466 - 2014-07-02 11:54:49 UTC
Rust Connor wrote:
Thanks!
It's case sensitive. I've tried "market/Prices/".....

And the costIndex? Any brief explanation?


Yea, really the best thing to do is follow links from the root to get where you want to go. That way the URL is mostly irrelevant.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

CCP FoxFour
C C P
C C P Alliance
#467 - 2014-07-03 11:40:00 UTC
Just deployed a fix to sisi to fix the cost and date formatting of the 4 (corp/char and running/history) industry endpoints. Let me know if there are any issues.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Teoshen
Transcendent Innovations Incorporated
#468 - 2014-07-03 16:59:40 UTC
Thank you, CCP FoxFour, for your attention and work on the APIs. I hope that some day the management at CCP/EVE realizes how essential all the third party sites are to the livelihood of the game and thanks you for not leaving us out in the cold.
CCP FoxFour
C C P
C C P Alliance
#469 - 2014-07-04 13:24:34 UTC  |  Edited by: CCP FoxFour
Hey guys,

I just want everyone to know that I will personally be on vacation from now until July 28th or so. This does mean I will be away for the Crius deployment. I have made sure they people here know whats up and can take care of any problems that arise with that.

If you happen to come across any other issues on Sisi though that need to be fixed before Crius and you are not getting a response on the forums poke me on twitter. I will try and monitor this thread, but yea... vacation. :)

I think we are looking really good for the Crius deployment though and am really excited to see what you guys do with everything we have done. :D

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

CCP Nullarbor
C C P
C C P Alliance
#470 - 2014-07-04 14:02:40 UTC
Rust Connor wrote:
Thanks!
It's case sensitive. I've tried "market/Prices/".....

And the costIndex? Any brief explanation?


I have a blog on its way to explain how to use cost index + adjusted price to calculate the price for installing industry jobs. Stay tuned.

CCP Nullarbor // Senior Engineer // Team Game of Drones

Taleden
North Wind Local no. 612
#471 - 2014-07-10 18:04:03 UTC
Is there any "best-practice" limit on the number of requests we send to CREST in a given time period?

I'm specifically thinking of things like market history scans, which may require 5,000-10,000 requests, per region, potentially every day for really serious marketeers. Server load can be reduced somewhat by submitting many requests through the same HTTP/TCP connection without opening a new one each time, but even so, should they be throttled on the client end, or can we rely on CREST to respond as quickly as it wants to?

Along a similar line, maybe it would make sense to offer market history queries for shorter time periods? Anyone who wants to pull that data frequently may also be storing it locally, so they don't need the entire past year for every item every time; a 7- or 30-day window to just pull new data would make for less redundant data transfer overhead.
CCP FoxFour
C C P
C C P Alliance
#472 - 2014-07-11 09:12:34 UTC
Taleden wrote:
Is there any "best-practice" limit on the number of requests we send to CREST in a given time period?

I'm specifically thinking of things like market history scans, which may require 5,000-10,000 requests, per region, potentially every day for really serious marketeers. Server load can be reduced somewhat by submitting many requests through the same HTTP/TCP connection without opening a new one each time, but even so, should they be throttled on the client end, or can we rely on CREST to respond as quickly as it wants to?

Along a similar line, maybe it would make sense to offer market history queries for shorter time periods? Anyone who wants to pull that data frequently may also be storing it locally, so they don't need the entire past year for every item every time; a 7- or 30-day window to just pull new data would make for less redundant data transfer overhead.


Here is some basic information: https://forums.eveonline.com/default.aspx?g=posts&t=344803&find=unread

Basically keep it to 10 RPS on CREST until we have the developer singups going. 503's are generally because we are rate limiting you. If you hit the wars endpoint several wars may timeout due to being stupid big and not loaded yet. If you get that timeout issue just wait 10 minutes and try again.

Keep in mind you should not request WH regions, and you should only request types that have a marketGroupID set. If they don't have a market group ID set they are not on the market and you are just wasting time.

I suppose we could offer different length market requests, but it's not really needed. The pages are usually cached in nginx (I suggest you do your cron jobs around 01:05 in the morning like everyone else) and so are loaded stupid fast. We then ensure the results are sorted so you only need to parse the first result.

One of the constraints we have on our caching system is we cache endpoints for X amount of time, and thats defined outside the scope of the endpoint, using a decorator. One of the things I want to get, and am pushing for, is the ability to set the client cache, what you guys see, and then also set the nginx cache differently. Doing that I could tell you guys it's cached for 1 hour but do things like expire the cache on the server at midnight. This means that if something is requested at 01:00 it stays cached for 23 hours. I cannot cache something for 23 hours right now because if it's requested at 23:55 and then cached for 23 hours you guys will stab me. If I get that kind of caching control, doing an endpoint that does something like take a region ID, return all market types and their data for yesterday, would be doable. I cannot do that now just due to how much time each request would take. If I can guarantee it's only hit once (while twice actually, once per nginx box) per day, screw it who cares.

So yea, TL;DR: time better spent on other things, once we get better caching control we will do even cooler things. :D

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Taleden
North Wind Local no. 612
#473 - 2014-07-11 13:41:27 UTC
CCP FoxFour wrote:
Taleden wrote:
Is there any "best-practice" limit on the number of requests we send to CREST in a given time period?


Here is some basic information: https://forums.eveonline.com/default.aspx?g=posts&t=344803&find=unread

Basically keep it to 10 RPS on CREST until we have the developer singups going. 503's are generally because we are rate limiting you. If you hit the wars endpoint several wars may timeout due to being stupid big and not loaded yet. If you get that timeout issue just wait 10 minutes and try again.

Keep in mind you should not request WH regions, and you should only request types that have a marketGroupID set. If they don't have a market group ID set they are not on the market and you are just wasting time.


Thanks muchly, hadn't seen that thread. Haven't gotten any 503s yet either but I added a rate limiter and user-agent just in case.
Squizz Caphinator
The Wormhole Police
#474 - 2014-07-11 13:52:58 UTC
What is the cache time on market calls such as this?

http://public-crest.eveonline.com/market/10000002/types/34/history/

I thought it was supposed to be an hour, but even after 04:00 the latest date provided was still only 2014/07/09 rather than 2014/07/10.

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com

CCP FoxFour
C C P
C C P Alliance
#475 - 2014-07-12 07:02:33 UTC
Squizz Caphinator wrote:
What is the cache time on market calls such as this?

http://public-crest.eveonline.com/market/10000002/types/34/history/

I thought it was supposed to be an hour, but even after 04:00 the latest date provided was still only 2014/07/09 rather than 2014/07/10.


You can always check the header to find the cache time. I don't know what item that is, but mabe there were none traded on that day.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Louis Vitton
Viziam
Amarr Empire
#476 - 2014-07-12 09:12:46 UTC
Type 34 = Tritanium.
I would say there would of been transactions over the length of a day.
CCP FoxFour
C C P
C C P Alliance
#477 - 2014-07-12 09:33:23 UTC
Louis Vitton wrote:
Type 34 = Tritanium.
I would say there would of been transactions over the length of a day.


Well then... Does resource match what's displayed in the client?

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Bubbles Udan
Caldari Provisions
Caldari State
#478 - 2014-07-13 07:09:02 UTC
CCP FoxFour wrote:
:/ Thanks for the heads up. I will look into this when I get back from vacation in August. I want to review the whole PI API as there are a few defects with it.

Some of the PI API problems are probably related to PI itself.

Actual extractor is pulling in 21371 units this cycle(1), 1 hour cycles,over a 2 day period it will pull in 575209 units for 11504 units per hour.

API says it is pulling 5056 units per cycle. At no point in production will it be pulling that few units per cycle.

I tried seeing if it was a factor of something.. no or maybe it was my skills and it stored what it predicted before I started extracting.. no. Maybe thats the units a single head is pulling in? I haven't tested that.

Outside of the API, PI still makes no sense even inside the game client. Same extractor on its route says 36400 units outgoing. That is WAY more my extractor will output for any cycle. The API returns the same numbers for routes and links as the game does at least.

Would it just be possible to add a field to the extractor that shows the program output total?

That being said, everything else in in the PI API works flawless. The contents of facilities and storage match up and work perfect. Being able to catch when I forget to set a route for a extractor I just moved has already made it worth it :P
CCP FoxFour
C C P
C C P Alliance
#479 - 2014-07-13 07:31:28 UTC
Bubbles Udan wrote:
CCP FoxFour wrote:
:/ Thanks for the heads up. I will look into this when I get back from vacation in August. I want to review the whole PI API as there are a few defects with it.

Some of the PI API problems are probably related to PI itself.

Actual extractor is pulling in 21371 units this cycle(1), 1 hour cycles,over a 2 day period it will pull in 575209 units for 11504 units per hour.

API says it is pulling 5056 units per cycle. At no point in production will it be pulling that few units per cycle.

I tried seeing if it was a factor of something.. no or maybe it was my skills and it stored what it predicted before I started extracting.. no. Maybe thats the units a single head is pulling in? I haven't tested that.

Outside of the API, PI still makes no sense even inside the game client. Same extractor on its route says 36400 units outgoing. That is WAY more my extractor will output for any cycle. The API returns the same numbers for routes and links as the game does at least.

Would it just be possible to add a field to the extractor that shows the program output total?

That being said, everything else in in the PI API works flawless. The contents of facilities and storage match up and work perfect. Being able to catch when I forget to set a route for a extractor I just moved has already made it worth it :P


That number is actually what's used. We do some math with to get the graph. When I am back I will see about just releasing said math.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Bubbles Udan
Caldari Provisions
Caldari State
#480 - 2014-07-13 07:43:32 UTC
Oh wow, that would be amazing. Thank you!