Issues, Workarounds & Localization

There appears to be a gradually-getting-worse widespread problem with something on CCP's end (or the network very close to them -- it is impossible for end-users to tell**) that's affecting the following things intermittently:

http://community.eveonline.com/ and http://www.eveonline.com/
EVE Launcher
XML API / CREST

The issue manifests itself either as very long timeouts (60-90 seconds) when trying to establish a TCP connection to these resources, or immediate errors (e.g. "Connection reset by peer" or similar). The latter is likely caused by TCP RST being sent back from the server to the client either during handshake or immediately after (and before any actual payload is sent). For EVE Launcher, another symptom is the launcher starting up in "Offline Mode".

Twitter account @EVE_status mentions nothing.

Some threads discussing all of these so far -- but this is not a full list:

https://forums.eveonline.com/default.aspx?g=posts&t=419294 (brief technical details in post #8)
https://forums.eveonline.com/default.aspx?g=posts&t=419478
https://forums.eveonline.com/default.aspx?g=posts&t=419492
https://forums.eveonline.com/default.aspx?g=posts&t=419530
https://forums.eveonline.com/default.aspx?g=posts&t=419253
https://forums.eveonline.com/default.aspx?g=posts&t=419505

Is anyone at CCP on call? Work weekends? Is there a NOC? Is monitoring set up for these endpoints from external sources (i.e. Internet-bound) and monitoring tied to that? How can we reach out to CCP to inform them of this? Else my gut feeling is that it's just going to stay like this until Monday when someone rolls into work and finds something quite busted.

** -- Please do not bother with traceroutes or mtrs. Border routers and/or firewalls block ICMP echo and/or response time-exceeded messages, not to mention IP routing is asymmetric and we can only see one half of the path -- someone with access to the EVE servers would need to be doing the same back to you and then both results compared for it to be effective.

Anything is possible, but I'd think the possibility of that would be unlikely -- who would implement that form of throttling on a Friday? :-)

I did some packet captures for the EVE Launcher issue and can confirm it's the same issue that's affecting the community.eveonline.com and www.eveonline.com websites (read: manifests itself the same way):

No.     Time           Source                srcport Destination           dstport Protocol Length Info
      3 0.024649000    192.168.1.50          1758    87.237.39.208         443     TCP      78     1758 > 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=16 TSval=0 TSecr=0 SACK_PERM=1
      4 0.044623000    87.237.39.208         443     192.168.1.50          1758    TCP      60     443 > 1758 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460
      5 0.044639000    192.168.1.50          1758    87.237.39.208         443     TCP      54     1758 > 443 [ACK] Seq=1 Ack=1 Win=65535 Len=0
      6 0.064743000    87.237.39.208         443     192.168.1.50          1758    TCP      60     443 > 1758 [RST, ACK] Seq=1 Ack=1 Win=16384 Len=0

Here we can clearly see CCP's 87.237.39.208 server (a.k.a. client.eveonline.com (verified this per DNS capture)) and my workstation completing the full 3-way handshake (SYN, SYN+ACK, ACK) followed immediately by CCP's server abruptly closing the connection with TCP RST. This isn't a clean shutdown (FIN, FIN+ACK), this is a deliberate RST. (Yes, the above capture involves a machine behind NAT, but rest assured when I say the packet capture I did on the WAN interface of my router showed the same behaviour).

Things to take note of:

1) client.eveonline.com has an A record TTL of 300, so I imagine the IP may vary/change for people. No idea if the issue is specific to just 87.237.39.208 or not, but my money is on that it isn't (because I've seen this happen for different CCP destinations within different CIDR blocks (see aforementioned linked thread, re: post #8))).

2) There is no payload -- this happens immediately after TCP handshake. So this isn't a case of, say, an HTTPS webserver rejecting a request or throttling based on the HTTP request itself, or an expired SSL certificate. This instead would be something like a very broken webserver daemon or a load balancer misbehaving. The possibilities are endless though (layer 7 filtering device, something like a Sandvine unit in the network path, etc.), and impossible for an end-user to determine.

3) I've manually tried to reproduce this using openssl s_client -showcerts -servername client.eveonline.com -connect 87.237.39.208:443 -- as expected, this works most of the time but still intermittently, which further supports my theory that CCP may just have a one (or more?) broken server(s) behind a load balancer. But it could just as equally be something with the LB itself.

When things do work, I notice Microsoft IIS is involved (yes, this is across HTTPS/port 443, but webservers these days wait for the client to send data before determining if it should be SSL or plaintext -- this is me manually using plaintext)):

GET / HTTP/1.0
Host: client.eveonline.com

HTTP/1.1 200 OK
Cache-Control: public, max-age=38
Content-Type: text/html; charset=utf-8
Expires: Sun, 19 Apr 2015 21:09:09 GMT
Last-Modified: Sun, 19 Apr 2015 21:08:09 GMT
Vary: *
Server: Microsoft-IIS/7.5
X-AspNetMvc-Version: 5.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sun, 19 Apr 2015 21:08:30 GMT
Connection: close
Content-Length: 40446

{snip HTML response}

I will note that in the HTML response, there is Javascript that references https://login.eveonline.com/ and https://crest.eveonline.com -- the latter is interesting because it could explain how the XML API / CREST is being impacted. I wouldn't have thought the Launcher used CREST, but chances are it does.

4) Other threads on the forum show people in the UK, and all throughout the US (different countries, states, and ISPs), and on different OSes (I'm on XP, others are on 7 or newer) are experiencing this. It's possible there's a commonality network-path-wise, but it's much more likely it's something on CCP's end. I just wish I knew who I could poke via Twitter etc. so they could engage their Systems folks to figure out what's up.

P.S. -- No CCNA, just a 20-years-in-the-field UNIX SA who's well-trained in networking as well. :-)

My launcher has evolved into an unplayable state, and download speeds are so slow it's impossible to patch. Started getting minor issues about Friday as well.

Yea it would be nice if they would at least make a thread stating what is going on.. or at the very least that they know about it and are working out the problem. I don't much care for the silent treatment/avoidance.

Hey look.. there is an elephant in the room....

Having the Launcher problems (TQ OFFline) since the last patch and it is getting worse.
Yesterday all my overview settings were gone and all notification settings, as well as password protected channels were resetted/ closed. That was kind of the switch between inconvenience to annoyance.

what kind of logs would help CCP in a bug report?