These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

EVE API - HTTP compression
Author
Previous Topic Next Topic
Ydnari
Estrale Frontiers
#1 - 2011-09-23 20:13:35 UTC
Just resurrecting this from the old forum as it didn't get a CCP answer - and the monetisation licensing terms are due by now so there may be more incentive for people to look at mobile apps; so:

Quote:
Some of the API calls produce relatively large XML output, such as the Assets list, this can easily end up being a megabyte or more. There's others that you might call more frequently that add up too.

This could be greatly reduced by enabling compression in the response (if requested and supported by the client in the usual way, standard HTTP stuff). XML compresses well due to the large amount of repeated text in it; around 10:1 compression for the assets API response.

Obviously this trades off decreased bandwidth with increased processing at the server.

The size of the response doesn't really matter when it's being run on a desktop or from a server, but when you're writing a mobile app you have to account for these things - 3G connections aren't particularly fast, and tend to have monthly caps on them.

Since there seems to be a little movement in the API area with the Contracts API, thought I'd raise this issue and see if CCP will consider it.

--
Karbowiak
State War Academy
Caldari State
#2 - 2011-09-24 01:29:25 UTC  |  Edited by: Karbowiak
If you dare, you can use http://api.evsco.net - it gzips everything, and also caches it all, till the cached_until time is run out.
It IS a public proxy, and it is run by EVSCO.

So anyone with privacy concerns should stay away from it, also if you generate too many errors i will block your ip (or totally take it offline from the public).

Also, it does not have any https access, not that it really matters anyway :)
Ydnari
Estrale Frontiers
#3 - 2011-09-24 13:34:47 UTC
Thanks, that's an option, but it would be much better if it were compressed at source by CCP.

--
PsyKzz
Queen'S DucK
#4 - 2011-09-26 12:07:44 UTC
Ydnari wrote:
Thanks, that's an option, but it would be much better if it were compressed at source by CCP.


I just want to add that compression requires extra CPU cycles.
Do you really want that on our precious servers?

Just to save a little bandwidth?

Meh.
Lutz Major
Austriae Est Imperare Orbi Universo
#5 - 2011-09-26 13:02:07 UTC
PsyKzz wrote:

I just want to add that compression requires extra CPU cycles.
Do you really want that on our precious servers?

Just to save a little bandwidth?
The additional load is on the web server and not on the back end. Compression is done in a few cycles and would save about 90% of the current bandwidth used.

I've an average of 2MB per account or 192KB zipped. I don't mind it. If you find a spot in Austria without 3G access you must be six feed under Ugh ... but there are other countries were you don't have that network coverage and every byte counts.
Ydnari
Estrale Frontiers
#6 - 2011-09-26 19:06:06 UTC
PsyKzz wrote:
Ydnari wrote:
Thanks, that's an option, but it would be much better if it were compressed at source by CCP.


I just want to add that compression requires extra CPU cycles.
Do you really want that on our precious servers?

Just to save a little bandwidth?


Yes please, provided it doesn't cause degredation of service. I don't have a full view of CCP's server configuration to put forward any sort of claim whether it will or won't. If the API webservers are relatively lightly loaded (not totally unreasonable to expect since the majority of the work is querying the backend SQL database, probably?) then this may be fine. If they're shared for other things, it may not fit and would be bad. Compressed resources may fit better in memcached, giving them a better hit rate.

Who knows? CCP does, cos they know how they have their servers set up. So I'm asking.



If you're referring to non-CCP servers processing API responses, the way you request compression within HTTP means you can also request that compression is NOT used, so it doesn't have have any effect on them unless they specifically add a header. Decompression is a very small overhead in either case.

--
PsyKzz
Queen'S DucK
#7 - 2011-10-05 03:02:40 UTC
I wasn't aware of the total impact, i just knew their would be one.
If the impact is small enough and they have the spare cycles i don't see a reason why not.

Meh.