EVE General Discussion

Hello everyone,

Those of you who use one of the many applications and websites that make use of the EVE API probably noticed that this Thursday they were not working particularly well or at all for most of the day. In fact the EVE API was largely non functional for about 14 hours and I would like to shed a little light on what happened and what we are doing to prevent future occurrences.

The popularity of the EVE API has grown tremendously over the past few years with an ever larger number of websites and applications making use of the information that is available via the API. This is a really good thing and opens endless possibilities for interacting with and presenting in-game information on a variety of portals. But as the popularity has grown exponentially so has the load on the service as a whole and at times to be honest we have struggled to keep up with this growth from both a hardware and software perspective. In between spurts of hardware upgrades and software improvements there have been periods where the usage has grown in leaps and bounds and thus left the API struggling to keep up and vulnerable to sudden peaks and shifts in usage.

This is more or less at the core of what happened Thursday. One of the larger more popular applications that uses the API is EVEMon. It is used by tens of thousands of players (including a lot of CCP devs) to view skills and attributes, monitor skill training and wallet balance etc. In version 1.5.0 of EVEMon released last Friday the behavior of the application changed slightly so that it would start aggressively retrying if it did not get a properly XML formatted CCP error response back in the case of an API error.

At the start of extended downtime on Thursday the API was not gracefully shut down and presented the applications with a default HTTP error page instead of a properly formatted XML one. This caused several thousand EVEMon clients all over the internet to go into overdrive and bombard the server with anywhere from 2 to 50-60 requests per client per second. Once the API was back online after downtime it never stood a chance at recovering from the massive flood as the servers could not keep up, filled up to their max connection limit and keeled over despite our best efforts to increase throughput and capacity. Once the servers reached their connection limit (which took only a few seconds) they no longer responded with a properly formatted XML response but a generic one which only served to sustain the request flood. Now this was greatly escalated by the fact that the API was already very fragile and vulnerable to such an overload condition as we had not taken care to plan for such an event predictable as it may be when I reflect back on this. So the API only had enough capacity to handle around 5x normal average load whereas this was on the order of 20x or more and we had no built-in means to throttle connection attempts.

Once we realized what was happening and after some attempts at restoring service on our own we reached out to the EVEMon devs which responded ultra quickly (massive props due) and released an updated version in less than two hours after we contacted them. A fix was ready at around 19:30 GMT and soon after started to propagate and we saw the load decrease until ultimately shortly after midnight full service was restored and things were back to normal.

Now needless to say there are a number of obvious improvements that need to be made to the API and to the supporting webserver infrastructure as a result. Top of the list is the ability to throttle inbound connections and to selectively allow certain types of requests. We are also planning to greatly increase the capacity of the API servers and be ready to throw in additional resources as needed with short notice. We encountered some issues troubleshooting what exactly was causing this so additional debugging tools and the ability to accurately trace through the application has been added to the shortlist of things to improve.

In closing I would like to make it absolutely clear that even though it may have been the EVEMon application that caused the increased traffic it ultimately could have been any application and we were just plain and simple not prepared and ill equipped to handle this properly. That is definitely a lesson learned and we will be putting a lot of effort in the next days and weeks into making the API more robust as it is a critical part of our game environment and will only continue to become more important as we evolve.

Yours,
Red Button

PURPLE BUTTON! LIKE A BOSS!

Thanks for the update

/c

Thanks for the write up! I'm glad it wasn't me that caused the problem

Oh dang and bwast. Another popuwar myth evapowates.

Sowwy cwibba.