EVE Technology Lab

2015-04-21 00:09:17 --- CRITICAL: Pheal\Exceptions\APIException [ 0 ]: General Error: Scotty the docking manager heard you were talking **** about him behind his back and refuses to service your request. ~ APPPATH\vendor\3rdpartyeve\phealng\lib\Pheal\Core\Result.php [ 95 ] in C:\www\siggy\application\vendor\3rdpartyeve\phealng\lib\Pheal\Pheal.php:294



WHY YOU SO MEAN SCOTTY


~5000 requests a day rejected from a service that hits 1.5k simultaneous users and with two layers of cache timers

CharacterAffialiation API over the last 3-4 months suddenly decided to throw mean scotty at me. It just returns that error in the XML body randomly.

https://gist.github.com/marekr/8bf683dd5590447ffb57

It appears to affect specific character IDs randomly and then go back to working for their IDs.
All the other APIs requests I do appear to be fine at the same time. I don't get back any unusual HTTP status codes nor timeouts.

Generally I query that API to validate an SSO user's corporation because SSO only returns characterID. I revalidate it every so often. while they are actively using the service. Generally it was every 2 hours when the cached expired. I saw the announcement the timer got reduced to 15 minutes and I may be polling more often now (I multiplied timer by 2 previously) and I need to correct back but I've had the issue for months now before the change.


It ends up returning the scotty hates you message for about 5000 requests a day. There are so many requests only because the service is eager to retry to actually get data or else users will have to be blocked from service. Doing security for 3rd parties with EVE is hard.


Is this just the world's most poorly written rate limited error message or something? :/


I'm considering making it email me when an character affiliation API starts failing so I can see if I can observe it from a different IP.

Hehe, thought so. I'm just super paranoid about rate limits given how much data I shuffle around.


I actually sent you an email at your CCP Games email address on April 22nd after reading that sticky thread on API Errors about this with the subject "[3rd party dev] api errors for siggy". Did you get it?


Otherwise I can evemail you more details.

I did miss your previous email. I have done some digging and found out whats causing the problem. Now I just need to fix it. :D

Being the terrible noob with C# I am, mainly a Python person here, I was unaware the C# hates your stupid guts if you try and edit a collection while iterating over it.

When I take the list of IDs you provide I check each ID to see if it's cached. If it is not found in the cache I add it to a new list called unCachedCharacterIDList. If that list is greater than 0 in length I get a lock and recheck the cache. To recheck the cache I loop over that unCachedCharacterIDList list and if the ID is found in cache on the second check I try and remove it from the uncached list. C# doesn't like you doing that. Whelp.

So this issue would happen if two requests came in for the same character ID very close to each other. The first request would succeed and the second request, having checked the cache before the first request finished, would not have found it but found it on the second attempt to check the cache. It would however just **** off and die at that point ruining all my plans for greatness.

Working on a fix now.

Testing seems to be fine on Sisi. It is a bit of a hard issue to replicate but I tried a few things to do it. Going to push to TQ.

3 hours, the bug is fixed and fix is live. Jesus christ I have no words.

Glad to hear it. :)

You play hots? I had higher expectations...