These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Technology Lab

 
  • Topic is locked indefinitely.
 

Authed crest - Intermittent 502 bad gateway on navigation endpoint.

First post
Author
Excellion
Nexus Mercator
#1 - 2016-03-09 19:37:23 UTC  |  Edited by: Excellion
I have been using the authenticated CREST API for over a month now to set waypoints for my characters. Usually this works entirely fine, but some evenings the only navigation API responses that crest returns are "502 Bad Gateway" errors.

HTTP/1.1 502 Bad Gateway
Server: nginx/1.2.0
Date: Wed, 09 Mar 2016 19:24:22 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Cache-Status: BYPASS

Gateway error communicating with EVE proxy

I ran into this issue a handful of times now and every time it appears that the problem itself just disappears the next day or the day after without any modifications to my own application. I can connect just fine to https://crest-tq.eveonline.com, and (deliberately) passing an incorrect authorization bearer will yield the expected "Invalid authorization" error. Only when the bearer is correct and the request is passed to the navigation API it will return above responses (Tested this both in my own code and in SoapUI).

Public crest on the other hand works like a charm when this issue pops up. I suspect this is just an infrequent issue with crest itself by now but i am wondering if anyone else is running into this when using the navigation endpoint?
Taris Atram
Open University of Celestial Hardship
Art of War Alliance
#2 - 2016-03-23 03:29:34 UTC
I started getting these tonight as well, hitting random authenticated endpoints on crest-tq. I know I'm not doing anything even close to the rate limit. At this point, I'm assuming it is something wrong with gateway proxy on CCP's side, they have been having issues with it as of late from what I understand.

I can't find too many mentions of 502 errors in any CREST related posts on the forum.
Pete Butcher
The Scope
Gallente Federation
#3 - 2016-03-23 07:53:57 UTC
Taris Atram wrote:
I can't find too many mentions of 502 errors in any CREST related posts on the forum.


Welcome to the world of CREST:

https://forums.eveonline.com/default.aspx?g=posts&m=6388293#post6388293
https://forums.eveonline.com/default.aspx?g=posts&m=6155371#post6155371
https://forums.eveonline.com/default.aspx?g=posts&m=6145624#post6145624
https://forums.eveonline.com/default.aspx?g=posts&m=6396832#post6396832

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool

Ken Bekle
Jednota Inc
#4 - 2016-03-30 10:26:43 UTC
If I am lucky, my CREST access runs once per 24 hour period without a 502.

Ken Bekle
Jednota Inc
#5 - 2016-03-30 22:35:10 UTC
Are there any actions one can take to increase chance of not getting a 502 on Market Import functionality of CREST?
Rtg Quack
xell network seven
#6 - 2016-04-07 19:42:30 UTC
Bump.

Same for me. Only getting 502 bad gateway if loggen on the Eve Client. If not I never run into the problem.

Weird. Any solution available?
Squizz Caphinator
The Wormhole Police
#7 - 2016-04-08 14:41:23 UTC
Ken Bekle wrote:
Are there any actions one can take to increase chance of not getting a 502 on Market Import functionality of CREST?


Retry with a few seconds pause until you get a proper answer. This is literally the approach I take on zKillboard. In a 5 minute time span I can have anywhere from 200-1000 CREST requests with about a 0.1% error rate.

https://github.com/zKillboard/zKillboard/blob/master/classes/CrestTools.php#L18-L39

Various projects I enjoy putting my free time into:

https://zkillboard.com | https://evewho.com

KenFlorian
Jednota Inc
#8 - 2016-04-09 12:10:26 UTC
Thanks, Squizz!
Rtg Quack
xell network seven
#9 - 2016-04-12 18:28:09 UTC
Hi Squizz , Thanks for the hint.

Unfortunately it does not help in my case, gave it a lot of tries in the last days.

The authed CREST behaviour seems pretty reproducable.

  1. Login against EVE-outh2 is reliable, works fine in all cases.
  2. If using then my application without running Eve client, no problems, never got the 502/bad Gateway message
  3. If logging in to Eve client I have a 50% chance of getting 502/bad Gateway messages afterwards
  4. First login to Eve client then to my application I have a chance of >75% of getting 502/bad Gateway
  5. Keep logged in with my application but logoff / logon the Eve client often fix the issue with a high likeliness


I'm open to any suggestion, have not the faintest clue on how to fix that.
CCP FoxFour
C C P
C C P Alliance
#10 - 2016-04-13 14:26:48 UTC
I don't have much to say about this except that I have made some very major changes to the CREST backend that will be coming with the citadel release. I hope that it will address many of these issues. :(

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Omniscient Fury
Whole Squid
#11 - 2016-04-20 07:26:02 UTC
CCP FoxFour wrote:
I don't have much to say about this except that I have made some very major changes to the CREST backend that will be coming with the citadel release. I hope that it will address many of these issues. :(


Can you confirm that Rtg's behaviour above is expected or a bug that's fixed in the Citadel release you mentioned? I'm seeing identical behaviour; if the EVE client is open and logged in then that user cannot authenticate via CREST in my web application (or using the IGB, ironically).
CCP FoxFour
C C P
C C P Alliance
#12 - 2016-04-25 08:22:14 UTC
Omniscient Fury wrote:
CCP FoxFour wrote:
I don't have much to say about this except that I have made some very major changes to the CREST backend that will be coming with the citadel release. I hope that it will address many of these issues. :(


Can you confirm that Rtg's behaviour above is expected or a bug that's fixed in the Citadel release you mentioned? I'm seeing identical behaviour; if the EVE client is open and logged in then that user cannot authenticate via CREST in my web application (or using the IGB, ironically).


I cannot as I cannot reproduce that and it should NOT be possible.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Omniscient Fury
Whole Squid
#13 - 2016-04-25 23:47:47 UTC  |  Edited by: Omniscient Fury
CCP FoxFour wrote:
Omniscient Fury wrote:
CCP FoxFour wrote:
I don't have much to say about this except that I have made some very major changes to the CREST backend that will be coming with the citadel release. I hope that it will address many of these issues. :(


Can you confirm that Rtg's behaviour above is expected or a bug that's fixed in the Citadel release you mentioned? I'm seeing identical behaviour; if the EVE client is open and logged in then that user cannot authenticate via CREST in my web application (or using the IGB, ironically).


I cannot as I cannot reproduce that and it should NOT be possible.

Do you have any suggestion as to how to troubleshoot this? There's two people in this thread and two more in the other one that are having this issue.

I get a 502 bad gateway returned if the EVE client is open and so do all of my users.

EDIT: To give more info on this... sorry, didn't get muchh sleep. :P


  • Authorization code is successfully obtained (user gets redirected, accepts, redirects back)
  • Access token is successfully obtained
  • API endpoints are walked and the decode endpoint located (https://crest-tq.eveonline.com/decode/)
  • GET request to the decode endpoint is made with an "Authorization: Bearer $" header
  • An error 502 bad gateway is thrown


Going back through logs I've also gotten this on a POST to https://login.eveonline.com/oauth/token with Authorization: Basic and Host: login.eveonline.com set, although that one returns a 504.

Double edit: I can confirm that Pyfa has the same issue. If you're currently in-game doing the export fitting results in a 502 bad gateway.
Lquid Drisseg
KarmaFleet
Goonswarm Federation
#14 - 2016-04-28 20:13:09 UTC  |  Edited by: Lquid Drisseg
+1 for Same CREST issue as described above.

CCP FoxFour wrote:
... it should NOT be possible.

It will never get fixed while your thinking like that. Lol

Steps to reproduce:

1. Install develop build of Pathfinder.
2. Install Pathfinder correctly.
3. Configure Crest Logins and create a proper Crest application with the 2 scopes Pathfinder uses.
4. Login to EveOnline with the character you plan on logging into Pathfinder with.
5. Try to login via SSO in Pathfinder.
6. Experience 502 error
7. Logout of Eve with the character.
8. Try logging into Pathfinder via SSO.
9. It magically works.

This exact process has been confirmed with many different people/characters over multiple days and in multiple corporations. If you are logged into eve with the character you are trying to get data from crest with, its very likely that Crest will 502 error when you try to hit /decode.
CCP FoxFour
C C P
C C P Alliance
#15 - 2016-04-29 03:49:48 UTC
HOLY ******* ****! JHAGSDUYASTD&*AS*D^ASD&*ASDGASD

Carbon Alabel was able to give me a reproducible test case and I finally have a working theory on why this is happening and why it happens on TQ but not Sisi or local test servers and why it was so hard to track down. I need to wait for the network admins to get into the office, turns out sane people are not awake at 0345 to answer questions about work.

IF this theory proves good I have an idea on how to fix it. The code fix wont ship until Tuesday next week at the earliest since that is the next deployment time for non-critical things but I may be able to convince people to make a change to our network configs today if this theory proves out.

Holy ******* ****... FAH P*HF*YHAUIDS!!!!!!!!!

p.s. always match your test environments to production

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Carbon Alabel
Gemini Talon
The Curatores Veritatis Auxiliary
#16 - 2016-04-29 07:17:58 UTC
I'm pretty sure that it was only reproducible by accident, but yay!
Hope you got this nailed down Smile
CCP FoxFour
C C P
C C P Alliance
#17 - 2016-04-29 15:21:01 UTC
I hereby declare 502's caused by being logged in as fixed. Please correct me if I am wrong.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Omniscient Fury
Whole Squid
#18 - 2016-05-01 04:56:33 UTC
CCP FoxFour wrote:
I hereby declare 502's caused by being logged in as fixed. Please correct me if I am wrong.



Thanks! So very much appreciated. Sorry for my terrible reporting/brain dumping, it was one thing on a very long list (and I assumed I was just being bad).

Thanks again.
CCP FoxFour
C C P
C C P Alliance
#19 - 2016-05-01 07:35:19 UTC
Omniscient Fury wrote:
CCP FoxFour wrote:
I hereby declare 502's caused by being logged in as fixed. Please correct me if I am wrong.



Thanks! So very much appreciated. Sorry for my terrible reporting/brain dumping, it was one thing on a very long list (and I assumed I was just being bad).

Thanks again.


No thank you. Thank you for reporting it, giving details, being persistent, all of it. Some of these things are incredibly hard to track down and it helps to have threads like this.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Rtg Quack
xell network seven
#20 - 2016-05-06 19:10:05 UTC
Congrats, +1, like, whatoever, you nailed him down.

Thanks and a big hug for that.


Now I have to take the next step, the /characters endpoint seems to be gone. Or at least it became invisble for me. I'll get into detail on a dedicated thread (still open).