These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
12Next page
 

api.eveonline.com update, gzip enabled

First post
Author
Previous Topic Next Topic
CCP illurkall
C C P
C C P Alliance
#1 - 2012-12-17 09:29:06 UTC
Hello there

As you may remember our attempt at deploying internal caching changes to the API failed some weeks back.

One of the changes we were going to make was to enable gzip compression on the API. To make a short story even shorter we have enabled gzip compression on the API without deploying the new caching layer changes or the other minor changes listed before.

Please let me know if this causes any issues.

We will revisit the other deployment next yearSmile
Verite Rendition
F.R.E.E. Explorer
#2 - 2012-12-17 10:44:32 UTC
Oh nice; I'm glad to see that could be rolled out and not held up by the unrelated problems with the cache layer. Thanks for the heads up, Illurkall. Smile
Squizz Caphinator
The Wormhole Police
#3 - 2012-12-17 17:16:16 UTC
Like! like like like like!

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com
Louis Vitton
Viziam
Amarr Empire
#4 - 2012-12-18 04:51:26 UTC
Thank you for the update
Desmont McCallock
#5 - 2012-12-18 09:42:53 UTC
@CCP illurkall or whoever knows anything about it
I apologize in advance for hijacking the thread but I just received a crash report in EVEMon and by looking at it I discovered a potential inconsistency in the Assets API call results.

When the result contains a 'Plastic Wrap' (typeid = 3468) the rawQuantity value is anything but '-1'.
AFAIK rawQuantity was added to specify the BP type (original or copy) as described in this blog.
Am I missing something?
CCP Creber Cattus
C C P
C C P Alliance
#6 - 2012-12-18 16:21:51 UTC
Desmont McCallock wrote:
@CCP illurkall or whoever knows anything about it
I apologize in advance for hijacking the thread but I just received a crash report in EVEMon and by looking at it I discovered a potential inconsistency in the Assets API call results.

When the result contains a 'Plastic Wrap' (typeid = 3468) the rawQuantity value is anything but '-1'.
AFAIK rawQuantity was added to specify the BP type (original or copy) as described in this blog.
Am I missing something?


Singleton items have the rawQuantity set as a negative number (and the quantity for those items are always one). The type of the item defines what the negative value means. Most types have just -1 but like you said then blueprint are different (-1 for original and -2 for copies). Since Retribution the Plastic Wrap uses the negative range for the total volume of all items inside the plastic wrap and is kept there as a negative value (divide it by -100 to get the m3).

So all checks that check if an item is a singleton should be for “less than zero” but not for “equal -1”.
xHjfx
The Legion of Spoon
Curatores Veritatis Alliance
#7 - 2012-12-18 17:34:02 UTC
CCP illurkall wrote:
Hello there

As you may remember our attempt at deploying internal caching changes to the API failed some weeks back.

One of the changes we were going to make was to enable gzip compression on the API. To make a short story even shorter we have enabled gzip compression on the API without deploying the new caching layer changes or the other minor changes listed before.

Please let me know if this causes any issues.

We will revisit the other deployment next yearSmile


OMG Best Xmas present ever... TY :)
Marcel Devereux
Aideron Robotics
Aideron Robotics.
#8 - 2012-12-18 20:11:48 UTC
Is there any other issues with the API? A ton of people are reporting that Aura has stopped updating since yesterday. I'm seeing intermittent hangs on my phone as well.
Desmont McCallock
#9 - 2012-12-18 20:44:22 UTC
CCP Creber Cattus wrote:
Desmont McCallock wrote:
@CCP illurkall or whoever knows anything about it
I apologize in advance for hijacking the thread but I just received a crash report in EVEMon and by looking at it I discovered a potential inconsistency in the Assets API call results.

When the result contains a 'Plastic Wrap' (typeid = 3468) the rawQuantity value is anything but '-1'.
AFAIK rawQuantity was added to specify the BP type (original or copy) as described in this blog.
Am I missing something?


Singleton items have the rawQuantity set as a negative number (and the quantity for those items are always one). The type of the item defines what the negative value means. Most types have just -1 but like you said then blueprint are different (-1 for original and -2 for copies). Since Retribution the Plastic Wrap uses the negative range for the total volume of all items inside the plastic wrap and is kept there as a negative value (divide it by -100 to get the m3).

So all checks that check if an item is a singleton should be for “less than zero” but not for “equal -1”.

Does that mean that the rawQuantity value can be a 64bit integer, just for the Plastic Wrap?
Marcel Devereux
Aideron Robotics
Aideron Robotics.
#10 - 2012-12-18 20:52:56 UTC
Actually it looks like this change is the cause. I forgot that I already had the code in Aura to handle GZip:

get.setHeader("Accept-Encoding", "gzip");
get.setHeader("User-Agent", "gzip");
...
Header contentEncoding = response.getFirstHeader("Content-Encoding");
if (contentEncoding != null && contentEncoding.getValue().indexOf("gzip") != -1) {
stream = new GZIPInputStream(stream);
}

I see that I am getting back 'Content-Encoding: gzip' for some of the feeds (APIKeyInfo and AccountStatus). Others are just hanging and the connection times out. Even those that are returning are taking a super long time (MemberTracking just too 151 seconds to return).

If I comment out this code everything works like it use to and nothing is timing out or taking a long time.
Desmont McCallock
#11 - 2012-12-19 06:05:39 UTC
No issues with EVEMon though. Probably, Marcel, it's the library you use for unzipping. In EVEMon and what I proposed to CCP illurkall, is to use zlib compatible libs and not native framework solutions.
CCP Creber Cattus
C C P
C C P Alliance
#12 - 2012-12-19 09:41:37 UTC
Desmont McCallock wrote:

Does that mean that the rawQuantity value can be a 64bit integer, just for the Plastic Wrap?


No, it is still 32 bit integer.
if I remember correctly then the current maximum size of plastic wrap is little under 1 million m3 and a 32 bit integer can handle 21,4 million (since it is stored as m3/100).
CCP illurkall
C C P
C C P Alliance
#13 - 2012-12-19 09:54:39 UTC
This thread is now confusing...

On the subject of compression. What I ended up using was the built in IIS 7.5 GZIP compressor. Didn't find any reports of problems on the internet and it has served us well in the past for static file compression.
Desmont McCallock
#14 - 2012-12-19 11:03:29 UTC
CCP Creber Cattus wrote:
Desmont McCallock wrote:

Does that mean that the rawQuantity value can be a 64bit integer, just for the Plastic Wrap?


No, it is still 32 bit integer.
if I remember correctly then the current maximum size of plastic wrap is little under 1 million m3 and a 32 bit integer can handle 21,4 million (since it is stored as m3/100).
Ok, that clears things a lot. Hijacking of the thread terminated.
Marcel Devereux
Aideron Robotics
Aideron Robotics.
#15 - 2012-12-19 14:24:04 UTC  |  Edited by: Marcel Devereux
CCP illurkall wrote:
This thread is now confusing...

On the subject of compression. What I ended up using was the built in IIS 7.5 GZIP compressor. Didn't find any reports of problems on the internet and it has served us well in the past for static file compression.


I can deflate the stream just fine. What is happening is when I make the request it just doesn't return. This is affecting every Aura user right now. If I comment out these two lines:

get.setHeader("Accept-Encoding", "gzip");
get.setHeader("User-Agent", "gzip");

everything starts working again. The only three feeds that ever return are APIKeyInfo, AccountStatus, and MemberTracking. The other feeds eventually hit my timeout.
Desmont McCallock
#16 - 2012-12-19 14:28:05 UTC
User-Agent should be "Aura" and in EVEMon case I sent Accept-Encoding as "gzip, deflate". In case this helps.
Marcel Devereux
Aideron Robotics
Aideron Robotics.
#17 - 2012-12-19 15:06:29 UTC
Desmont McCallock wrote:
User-Agent should be "Aura" and in EVEMon case I sent Accept-Encoding as "gzip, deflate". In case this helps.


Some servers require that User-Agent to be gzip for it to work (read CCP"s shouldn't be failing if I setting the User-Agent to gzip). I will try adding deflate but again that shouldn't really matter (read this code works with ever server except CCP's).
Marcel Devereux
Aideron Robotics
Aideron Robotics.
#18 - 2012-12-19 20:06:57 UTC
I'm seeing this error a bunch:

javax.net.ssl.SSLException: Read error: ssl=0x5722ed28: I/O error during system call, Connection reset by peer
Marcel Devereux
Aideron Robotics
Aideron Robotics.
#19 - 2012-12-19 20:12:50 UTC
Ok adding deflate seems to have fixed the issue but it is taking a ton of time (5+ seconds) for each feed. I disabled it and things went back to normal (<1 second for most feeds). I'm calling foul on your end.
Dei
Cosmic Core Industries
#20 - 2012-12-19 20:13:17 UTC
I've been using gzip, * as accept encoding (in case illurkall pulls it again), and it has been running fine. Deflate isn't necessary unless you really want to support that encoding.
12Next page