These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Testing a major revision to CREST

First post First post
Author
Previous Topic Next Topic
CCP FoxFour
C C P
C C P Alliance
#41 - 2016-04-27 18:38:39 UTC
Aineko Macx wrote:
CREST is currently throwing massive amounts of 500 errors. Also the market history and order endpoints continue to be extremely slow as described.


Yea, we had some issues post deployment. Things should be in a good state now. Let me know if there are any issues. :)

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Ydnari
Estrale Frontiers
#42 - 2016-04-27 18:45:53 UTC  |  Edited by: Ydnari
Having trouble with corp XML API keys since the deployment, anyone else seeing that? Character/Account keys seem to be working OK though. Corp keys throwing 221 access error, both on old valid keys and freshly created ones.

edit: seems to have cleared up now

--
Aineko Macx
#43 - 2016-04-28 10:38:34 UTC
Since the performance issue of market history and orders remains, I did some timed tests:
Full history scan of "The Forge": 7m
Full order scan of "The Forge": 16m

Full history scan of "Cloud Ring": 45m
Full order scan of "Cloud Ring": 109m (a handful of 30 second timeouts also occurred)

Before around April 17th, one could do history + orders for any region in ~6 minutes (100reqs/s).

Note that all these numbers are from runs with 10 parallel requests. Interestingly, on the slow items/regions, it doesn't matter if 10 or 2 parallel requests are used, they run at about the same speed, like if the backend is handling them in serialized fashion.

iveeCore 3.0: The PHP engine for industrial activities and CREST library
CCP FoxFour
C C P
C C P Alliance
#44 - 2016-04-29 04:36:51 UTC
Aineko Macx wrote:
Since the performance issue of market history and orders remains, I did some timed tests:
Full history scan of "The Forge": 7m
Full order scan of "The Forge": 16m

Full history scan of "Cloud Ring": 45m
Full order scan of "Cloud Ring": 109m (a handful of 30 second timeouts also occurred)

Before around April 17th, one could do history + orders for any region in ~6 minutes (100reqs/s).

Note that all these numbers are from runs with 10 parallel requests. Interestingly, on the slow items/regions, it doesn't matter if 10 or 2 parallel requests are used, they run at about the same speed, like if the backend is handling them in serialized fashion.


Are you sending an authorization token? Can you tell me your IP address? Are you using the new combined orders resource?

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Aineko Macx
#45 - 2016-04-29 05:49:08 UTC  |  Edited by: Aineko Macx
I am not sending auth tokens. I just tested sending one but no notable change. The numbers are from the separate orders endpoints (so they are comparable with pre-citadel times), but I just started using the combined one and will run more tests. I'll PM you my current IP.
Currently I'm experiencing lot's of timeouts and 500's, so my scraper doesn't run for long.

EDIT: Yes, timeout are a big issue, I'm running into them on simple single page requests to collection endpoints.

iveeCore 3.0: The PHP engine for industrial activities and CREST library
Aineko Macx
#46 - 2016-04-29 11:57:21 UTC  |  Edited by: Aineko Macx
Nice, just had a fast run on "Cloud Ring". Whatever you guys did, it helped, thx! Big smile

EDIT: Yep, with the combined orders endpoint we are now back to 6 minute region scan.

iveeCore 3.0: The PHP engine for industrial activities and CREST library
CCP FoxFour
C C P
C C P Alliance
#47 - 2016-04-29 14:08:59 UTC
Aineko Macx wrote:
Nice, just had a fast run on "Cloud Ring". Whatever you guys did, it helped, thx! Big smile

EDIT: Yep, with the combined orders endpoint we are now back to 6 minute region scan.


I am also working on a resource that is all orders for all types in a given region. Should be interesting.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Steve Ronuken
Fuzzwork Enterprises
Vote Steve Ronuken for CSM
#48 - 2016-04-29 14:18:44 UTC
CCP FoxFour wrote:
Aineko Macx wrote:
Nice, just had a fast run on "Cloud Ring". Whatever you guys did, it helped, thx! Big smile

EDIT: Yep, with the combined orders endpoint we are now back to 6 minute region scan.


I am also working on a resource that is all orders for all types in a given region. Should be interesting.



\o/

Op Success!

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter
CCP FoxFour
C C P
C C P Alliance
#49 - 2016-04-29 14:31:43 UTC
Current test setup looks like this:

{
"totalCount_str": "20000",
"items": [
{
"buy": false,
"issued": "2016-04-29T14:16:02",
"price": 1200000000.01,
"volumeEntered": 3,
"stationID": 60000970,
"volume": 3,
"range": "region",
"minVolume": 1,
"duration": 90,
"type": 1944,
"id": 4051697671
},

20,000 entries per page. The largest region is The Forge and as of last night it had just shy of 287,000 orders. So even on that region it would only be 15 requests to get all of it. Smaller regions would of course be less. Hopefully can get this out next release.

I also stripped out all the links and such to make this response as small as possible.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Jare Kin
The Corporate Tax Shelter
#50 - 2016-04-30 21:33:47 UTC  |  Edited by: Jare Kin
Hey Foxfour,

according to [1] I need to send a custom http header "Authorization: Basic XYZ" to turn the authorization code from SSO into an access/refresh token to be able to access CREST.

I do the SSO client side with XMLHttpRequest but the requirement of the Authorization header makes modern browsers (edge, firefox) do a preflight request that looks like this:

OPTIONS https://login-tq.eveonline.com/oauth/token/ HTTP/1.1
Access-Control-Request-Headers: authorization, content-type
Access-Control-Request-Method: POST


Your SSO service then answers with 404.

Would it be possible to setup the SSO service to answer with the following response instead:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, OPTIONS
Access-Control-Allow-Headers: Authorization, Content-Type


Your CREST service is doing this already, which is very nice.

If you want to read further about CORS, have a look at [2]

Thanks & bye, Jare Kin.

[1] http://eveonline-third-party-documentation.readthedocs.io/en/latest/sso/authentication.html
[2] https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS
CCP FoxFour
C C P
C C P Alliance
#51 - 2016-05-01 01:27:23 UTC
Jare Kin wrote:
Hey Foxfour,

according to [1] I need to send a custom http header "Authorization: Basic XYZ" to turn the authorization code from SSO into an access/refresh token to be able to access CREST.

I do the SSO client side with XMLHttpRequest but the requirement of the Authorization header makes modern browsers (edge, firefox) do a preflight request that looks like this:

OPTIONS https://login-tq.eveonline.com/oauth/token/ HTTP/1.1
Access-Control-Request-Headers: authorization, content-type
Access-Control-Request-Method: POST


Your SSO service then answers with 404.

Would it be possible to setup the SSO service to answer with the following response instead:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: POST, OPTIONS
Access-Control-Allow-Headers: Authorization, Content-Type


Your CREST service is doing this already, which is very nice.

If you want to read further about CORS, have a look at [2]

Thanks & bye, Jare Kin.

[1] http://eveonline-third-party-documentation.readthedocs.io/en/latest/sso/authentication.html
[2] https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS


It is a 404 which is an error because the query string is to long. We have an open ticket internally to look into fixing this. We could probably look into adding CORS headers to it as well, but CREST doesn't have CORS headers for errors either.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Jare Kin
The Corporate Tax Shelter
#52 - 2016-05-01 09:43:07 UTC
Hey Foxfour,

thanks for your quick answer and the explanation.

In that case I would vote for getting rid of the 404 so you don't need to add CORS headers to a non-200 response.

I worked around that 404 by using a server-side script that relays my POST request:

PHP:
$ch = curl_init($_GET['url']);

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

if (array_key_exists('PHP_AUTH_USER', $_SERVER) && array_key_exists('PHP_AUTH_PW', $_SERVER))
    curl_setopt($ch, CURLOPT_USERPWD, $_SERVER['PHP_AUTH_USER'].':'.$_SERVER['PHP_AUTH_PW']);

curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);
  
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  
$contents = curl_exec($ch);
if($contents === false)
    echo 'ERROR: '.curl_error($ch);
else
    echo $contents;
  
curl_close($ch);


I don't like it, cause I need a server-side script to run. Until now I had only static files + window.localStorage for my fleet app :(

I hope fixing that 404 doesn't cost you any effort.

Thanks again & bye, Jare Kin.
Cornbread Muffin
The Chosen - Holy Warriors of Bob the Unforgiving
#53 - 2016-05-03 09:05:30 UTC
CCP FoxFour wrote:
Aineko Macx wrote:
Nice, just had a fast run on "Cloud Ring". Whatever you guys did, it helped, thx! Big smile

EDIT: Yep, with the combined orders endpoint we are now back to 6 minute region scan.


I am also working on a resource that is all orders for all types in a given region. Should be interesting.


Victory! Thanks, this should be great.
Aineko Macx
#54 - 2016-05-03 14:37:02 UTC  |  Edited by: Aineko Macx
There's currently no navigable way to reach the new Station endpoint. I believe station hrefs should be listed in the System responses.
The same is true for the aggregated character stats. Can it be linked from the Character endpoint please?
Finally, it would be nice if token verify and the aggregated stats responses hat proper content types set.

Also, are there plans for enabling the corporations endpoint sometime?

iveeCore 3.0: The PHP engine for industrial activities and CREST library
Rectar en Meunk
Center for Advanced Studies
Gallente Federation
#55 - 2016-05-04 12:55:09 UTC
Requests to squads fail.

 In [17]: oauth.get(fleet_url).json()
Out[17]: 
{'isFreeMove': False,
'isRegistered': False,
'isVoiceEnabled': False,
'members': {'href': 'https://crest-tq.eveonline.com/fleets/1016611228925/members/'},
'motd': '',
'wings': {'href': 'https://crest-tq.eveonline.com/fleets/1016611228925/wings/'}}


In [18]: oauth.get("https://crest-tq.eveonline.com/fleets/1016611228925/wings/").json()
Out[18]: 
{'items': [{'href': 'https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/',
  'id': 2021811228925,
  'id_str': '2021811228925',
  'name': 'Wing 1',
  'squads': {'href': 'https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/squads/'},
  'squadsList': [{'href': 'https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/squads/3041611228925/',
    'id': 3041611228925,
    'id_str': '3041611228925',
    'name': 'Squad 1'}]}],
'pageCount': 1,
'pageCount_str': '1',
'totalCount': 1,
'totalCount_str': '1'}

In [19]: oauth.get("https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/squads/").json()
Out[19]: 
{'exceptionType': 'BadRequestError',
'key': 'noHandlerMatch',
'message': 'Did not find handle function to match GET None */* for class FleetSquads'}

In [20]: oauth.get("https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/squads/3041611228925/").json()
Out[20]: 
{'exceptionType': 'BadRequestError',
'key': 'noHandlerMatch',
'message': 'Did not find handle function to match GET None */* for class FleetSquad'}
Rectar en Meunk
Center for Advanced Studies
Gallente Federation
#56 - 2016-05-04 14:27:13 UTC
That's not the only one failing. https://gist.github.com/6dd8372d5b3391d4ba8203e818c1b853 

The call to https://crest-tq.eveonline.com/races/8/ failed with {"message": "Did not find handle function to match GET None */* for class Race", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/fleets/1016611228925/members/93709888/ failed with {"message": "Did not find handle function to match GET None */* for class FleetMember", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/characters/93709888/navigation/waypoints/ failed with {"message": "Did not find handle function to match GET None */* for class Waypoints", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/characters/93709888/notifications/ failed with {"message": "Did not find handle function to match GET None */* for class CharacterNotifications", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/bloodlines/8/ failed with {"message": "Did not find handle function to match GET None */* for class Bloodline", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/squads/3041611228925/ failed with {"message": "Did not find handle function to match GET None */* for class FleetSquad", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/ failed with {"message": "Did not find handle function to match GET None */* for class FleetWing", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/squads/ failed with {"message": "Did not find handle function to match GET None */* for class FleetSquads", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
CCP FoxFour
C C P
C C P Alliance
#57 - 2016-05-04 15:02:58 UTC
Rectar en Meunk wrote:
That's not the only one failing. https://gist.github.com/6dd8372d5b3391d4ba8203e818c1b853 

The call to https://crest-tq.eveonline.com/races/8/ failed with {"message": "Did not find handle function to match GET None */* for class Race", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/fleets/1016611228925/members/93709888/ failed with {"message": "Did not find handle function to match GET None */* for class FleetMember", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/characters/93709888/navigation/waypoints/ failed with {"message": "Did not find handle function to match GET None */* for class Waypoints", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/characters/93709888/notifications/ failed with {"message": "Did not find handle function to match GET None */* for class CharacterNotifications", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/bloodlines/8/ failed with {"message": "Did not find handle function to match GET None */* for class Bloodline", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/squads/3041611228925/ failed with {"message": "Did not find handle function to match GET None */* for class FleetSquad", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/ failed with {"message": "Did not find handle function to match GET None */* for class FleetWing", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}
The call to https://crest-tq.eveonline.com/fleets/1016611228925/wings/2021811228925/squads/ failed with {"message": "Did not find handle function to match GET None */* for class FleetSquads", "key": "noHandlerMatch", "exceptionType": "BadRequestError"}


Um... I don't know about the races or bloodlines stuff but the rest makes sense. The error message describes it perfectly. There is no GET for those endpoints. Only PUT or POST.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.
Rectar en Meunk
Center for Advanced Studies
Gallente Federation
#58 - 2016-05-04 16:02:19 UTC
CCP FoxFour wrote:
Um... I don't know about the races or bloodlines stuff but the rest makes sense. The error message describes it perfectly. There is no GET for those endpoints. Only PUT or POST.

Oops. I'll go read the spec again.
Bruce Warhead
Incursion Citadel Network
#59 - 2016-05-06 13:01:39 UTC
Is there any chance to get a way, to give Boss to an other character using the crest api?
Or if there is one I must have missed it and would really appreciate a hint at it.

Other wise thanks a lot for making the fleet crest endpoint happen! I am currently working on updating the Waitlist manager I wrote for Warp To Me to use it.
So we can actually just press a button to invite people to the correct squad instead of needing to drag their portrait.
It will make things so much easier!
Big smile
https://www.youtube.com/watch?v=IKLRqdENDww your api at use!
CCP FoxFour
C C P
C C P Alliance
#60 - 2016-05-06 13:44:47 UTC
Bruce Warhead wrote:
Is there any chance to get a way, to give Boss to an other character using the crest api?
Or if there is one I must have missed it and would really appreciate a hint at it.

Other wise thanks a lot for making the fleet crest endpoint happen! I am currently working on updating the Waitlist manager I wrote for Warp To Me to use it.
So we can actually just press a button to invite people to the correct squad instead of needing to drag their portrait.
It will make things so much easier!
Big smile
https://www.youtube.com/watch?v=IKLRqdENDww your api at use!


:D :D :D :D :D

You have no idea how amazing that is to see! :D Thank you for sharing.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.