These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

Unexpected Dowtime Update - Tranquility Is Back Online

First post
Author
Previous Topic Next Topic
Alexander Eisenhower
The Scope
Gallente Federation
#181 - 2013-06-03 13:41:40 UTC
I appreciate the fact that CCP is willing to deal with issues absolutely when there is an issue instead of pretending like there is no issue like i have seen on several other game servers in my time. Keep up the good work.

And really guys, whats 24 hrs in the scheme of my training? Not very significant really.
Popsikle
Caffeine Commodities Company
#182 - 2013-06-03 13:42:45 UTC
THC Trader wrote:
Popsikle wrote:
Phobos Eversor wrote:
Quote:
We apologize for any inconvenience that this extended downtime has caused.


You... you what?

You stayed up all night for us. You gave up your weekend for us. You updated twitter every 3 hours for us. Your coffee pot valiantly died in the line of duty for us.

You don't need to apologize to us; You need a nap, a raise, and a new coffee pot.

Thanks for the hard work.


While I do appreciate CCP and normally support them (fanboi) this situation was completely avoidable. They took all of the servers down because there was a patchable exploit in their infrastructure. The keyword in that last sentence is patchable. We (collectively) pay them lots of money, and this sort of flaw in the platform should have been identified before its too late. Yes, I know some exploits are 0-day and such, but since they patched it it had to be an exploit that was on someones radar. Suggesting someone gets a raise because of this is the opposite of what should happen tbh. Someone should get a yelling at and possible termination for not being on top of security.

That said they did not stay up for us, they stayed up because its their job. They could not work around the clock to get the servers patched and expect the userbase to keep paying them. Now CCP isn't a giant machine, and I am sure a lot of the people were sympathetic with us because they do also play the game, but even if they weren't i'm sure it was an "ALL HANDS ON DECK" call from above. I know I have been in situations similar where sure, I want to make things available to customers because it sucks for everyone things are down but everyone is working on the problem because that's what they get paid to do.

I really do appreciate the information on why they took everything down instead of the bs line about this being a DDoS attack. That is what agitated me most yesterday was that they were blatantly lying to everyone about what happened. Glad to see they came clean about the issue today!

I do also agree with the decision to shut everything down to figure out what the hell happened and get it fixed in how every long it takes before bringing it back online. That was very responsible of them, and I appreciate them protecting my information ;)

You're ********. It was definitely a DDoS attack. The attack took advantage of a (until now) unknown vulnerability in their servers. They took the servers down to evaluate the situation, and make sure customer data had not been compromised, while also working on a patch to remove the vulnerability. CCP handled the situation very well, and you're a jackass for claiming otherwise. Go back to WoW with your entitlement attitude. See what happens when your 2000 man WoW server gets DDoSed. Bet it's down for a month.



Sorry, bittervet (here since beta with my 6 accounts) so going back to anything isn't an option. They fact is they patch and exploit in their setup. You set something on the interwebs that's exploitable its going to get exploited. Its not the attackers fault your **** wasn't secured...Why was a back-end service accessible from the internet anyway?

Also, trad the whole post, I think they handled the situation fine, and did what they were supposed to do when an exploit is found, but they did not need to feed us the bullshit line that a DDoS is taking everything out. DDoS's cause latency and fill pipes, sometimes take out services but are generally not crafted attacks, they are generic attacks sent out by drones. This was a crafted attack and they knew that from the start which is why they shut down TQ (which is what any sensible company would have done) and then did not let the attacker know they were on to what was going on, which is a good thing BUT call a spade a spade and drop the word DDoS from their updates. "We are currently undergoing a sustained attack and are investigating it" would have been way more appropriate than what they said.
SN50049177 1
Re-Hab
#183 - 2013-06-03 13:43:13 UTC
Ace Uoweme wrote:
SN50049177 1 wrote:
I don't want an extra day or anything, just that my carefully planned skilltraining is going to work out as I planned it.


Especially when your toons training queued stopped for hours during/after/inbetween the shutdown, when trying to beat the deadline for June 4th. Was suppose to finish 40 days of training at 1am on the 4th, now it pretty darn close to the server shutdown, and barely enough to insert/train the remaining skill which has to wait for the other skill to finish.

If it wasn't for the expansion deadline it wouldn't be much of a problem. But this forced using up remaps to beat the clock, and the clock stopped even. :(


Yup...
Lady Areola Fappington
#184 - 2013-06-03 13:43:46 UTC
Popsikle wrote:

I really do appreciate the information on why they took everything down instead of the bs line about this being a DDoS attack. That is what agitated me most yesterday was that they were blatantly lying to everyone about what happened. Glad to see they came clean about the issue today!



Now, I know the last few hours have been stressful, but you MAY have missed the first line in Mr. Jónasson's announcement. Allow me to share it.

At 02:05 UTC June 2nd, CCP became aware of a significant and sustained distributed denial-of-service attack (DDoS) against the Tranquility cluster (which houses EVE Online and DUST 514) and web servers.

They got DDoSed. The DDoS took advantage of a vulnerability in some back-end code. It happens, it's unexpected, nothing CCP or anyone else could have done to prevent it.


This has been an excellent reaction on CCP's part. We were informed, service came back up, and you're getting "compensation" for the troubles.


A typical company response to something like this would be "We are currently having an unplanned downtime. We have no ETA on return. Please see EULA section XX:XX where it says we don't promise any service uptime". Following that, a massive forums purge of any questions remotely relating to what is happening.

7.2 CAN I AVOID PVP COMPLETELY? No; there are no systems or locations in New Eden where PvP may be completely avoided. --Eve New Player Guide
Popsikle
Caffeine Commodities Company
#185 - 2013-06-03 13:43:58 UTC
Adam1337 wrote:
baltec1 wrote:
Popsikle wrote:


While I do appreciate CCP and normally support them (fanboi) this situation was completely avoidable. They took all of the servers down because there was a patchable exploit in their infrastructure.


It is impossible to lock all the back doors. Just last week China stole several top secret designs from the US military.

If an organisation with a trillian $ budget cant stop these things from happening then what chance does a small icelandic gaming company have?

Just be glad they fixed it.



With the greatest of respect, CCP team are highly trained competent team of skilled software and hardware engineers, they should stand a better chance at protecting there networks than the poorly trained low skill operatives employed by a government.




lol. you owe me a keyboard...
Arabelli Tanis
The Scope
Gallente Federation
#186 - 2013-06-03 13:44:08 UTC  |  Edited by: Arabelli Tanis
EVE is a service. People pay for that service. It's CCP's job to make sure people get the service they're paying for. If that means coming in on their days off to do so, then that's just part of the job. I appreciate their hard work, but some people seem to be giving reactions as if they expected CCP to just wait until normal business hours to address this situation - and thus are so surprised and thankful that they did their jobs on the weekend instead. I hate that it sounds so negative, but the truth hammer is rarely gentle.

However, I do feel CCP deserves thanks / props / credit for their communication through this situation. Most companies seem to fail to realize that their player base can actually be quite patient as long as they aren't left in the dark. Most companies don't handle unexpected downtime communication very well... CCP did a wonderful job letting everyone know what was wrong, why it was wrong, what they were doing to try to fix it - and providing status updates along the way. THAT is where they went above and beyond, and for that I am grateful.


I just hope that whoever decided to rent a botnet for the weekend and DDoS CCP got their jollies off and won't be doing it again.
Hakaru Ishiwara
Republic Military School
Minmatar Republic
#187 - 2013-06-03 13:44:10 UTC
baltec1 wrote:
Moretic wrote:



how do you know it wasnt their fauilt?
They say it wasnt their fault but thats what they say, it can just as well been a ccp worker who lost his/her brain.
in the end we are paying for a service, which was not accessible.


Why would they ddos themselves?

You self entitalment types need to just go away, CCP pulled an all nighter dredging through the entire game line by line on a weekend.

The only gift I want is the name of whoever did this. CCP dont owe us anything.
To think that CCP doesn't have personnel on-duty during "weekend" hours is ludicrous. While CCP may have had to bring in additional "off-duty" resources to remedy the issue(s), these heroes are employees (and contractors) doing their jobs.

IMHO, it is more honest to say 'thank you for doing your jobs and bringing the EVE entertainment service back online w/in ~36 hours of the initial problems.'

+++++++ I have never shed a tear for a fellow EVE player until now. Mark “Seleene” Heard's Blog Honoring Sean "Vile Rat" Smith.
Jeann Valjean
The Scope
Gallente Federation
#188 - 2013-06-03 13:46:20 UTC
Popsikle wrote:
Its not the attackers fault your **** wasn't secured...


"Help, help! My computer just started attacking EVE because it wasn't secured! How do I stop it?!?!?!"

Thanks for my morning rofl.
Goin Off
Manson Family
Advent of Fate
#189 - 2013-06-03 13:47:35 UTC
Good job getting us back online and thanks for all of your hard work over the weekend. It sucks that their are a$$hats in this world that do these despicable things. Me thinks that they should get a life and use their talents towards the good of mankind as opposed to harming others.

Thanks for doing your due diligence in protecting our proprietary information and if taking the servers down to protect your membership from harm then by all means do so at any time.

On a personal note it was somewhat distressing at the time the server went down that I along with our fleet of alliance T3 BC's had just landed on a hostile pos to reinforce it after whelping their fleet. In looking back the stress over wondering whether or not my alliance mates and I would still be in a ships seems silly now.

CCP you gotta do what you gotta do and, if it's in the best interest of to pull the plug, then just do it.

Thanks for bringing us such a fun game and the opportunity to build friendships ALL OVER THE WORLD.

Compensation for the down time is not necessary but will be greatly appreciated if offered. Good luck with tomorrows expansion and THANKS AGAIN!

Goin Off

BTW, We'll be back to RF that CO2 pos real soon!
Lara Dantreb
Reisende des Schwarzschild Grenze
#190 - 2013-06-03 13:47:56 UTC
CCP folks, sorry for your loss and for the inconvenience. And thank you for bringing back to us our little computer hobby Big smile

---   Buying T2 ship bpos since 2005  ---
Apollo Lawless
Federal Navy Academy
Gallente Federation
#191 - 2013-06-03 13:48:18 UTC  |  Edited by: Apollo Lawless
Glad the server is up lucky I had a 5 day skill in queue!
Thank you guys for working hard to bring the server up! Big smile

All this talk about free SP! A future CONCORD Cruiser would do just as fine Pirate

Et tu, Capsuleer?
Surfin's PlunderBunny
Sebiestor Tribe
Minmatar Republic
#192 - 2013-06-03 13:57:09 UTC
This was like coming home from summer camp to find your parents moved Cry

"Little ginger moron" ~David Hasselhoff 

Want to see what Surf is training or how little isk Surf has?  http://eveboard.com/pilot/Surfin%27s_PlunderBunny
Melissa Turner
The Scope
Gallente Federation
#193 - 2013-06-03 13:57:26 UTC
you know im really getting sick and tired of folks out there getting mad about folks wanting to know they'll get "compensated" for an unexpected service disruption... its called Customer Service and is a great policy to follow in order to make sure customers are happy. for those that dont know.. when sony was hit with the PSN attack. not only did they apologize to their 77 million effected gamers, they also gave 5 free full games, free playstation plus service, and a special msg at E3 about the entire situation. it was excellent customer service and ccp should rightfully follow that trend.
Athelas Loraiel
Viziam
Amarr Empire
#194 - 2013-06-03 14:03:17 UTC
Thank you CCP; I hope that the culprit gets hanged by the ball hairs.
May his occasional GFs miss their periods.
Repeatedly and at the same time.

No recompensation needed.
Adam1337
Lauren Abbey Industries
#195 - 2013-06-03 14:03:29 UTC
Melissa Turner wrote:
you know im really getting sick and tired of folks out there getting mad about folks wanting to know they'll get "compensated" for an unexpected service disruption... its called Customer Service and is a great policy to follow in order to make sure customers are happy. for those that dont know.. when sony was hit with the PSN attack. not only did they apologize to their 77 million effected gamers, they also gave 5 free full games, free playstation plus service, and a special msg at E3 about the entire situation. it was excellent customer service and ccp should rightfully follow that trend.




In the PSN attack, tell me how any accounts where compromised, was it suspect 10 million..?

So far as we know not one account has been compromised in the Tranquility attack, so yes I agree a token gift to say thank you to all would be a nice gesture, but I think getting 5 free games for my Playstaion and an access pass to the plus service is a little to much to expect.

Let Odyssey roll out, if it arrives without issue then be happy if you get some sp, if it has issues I am confident some big bundle of gifts might be in order...
Alatari Yassavi
Viziam
Amarr Empire
#196 - 2013-06-03 14:06:34 UTC
Melissa Turner wrote:
you know im really getting sick and tired of folks out there getting mad about folks wanting to know they'll get "compensated" for an unexpected service disruption... its called Customer Service and is a great policy to follow in order to make sure customers are happy. for those that dont know.. when sony was hit with the PSN attack. not only did they apologize to their 77 million effected gamers, they also gave 5 free full games, free playstation plus service, and a special msg at E3 about the entire situation. it was excellent customer service and ccp should rightfully follow that trend.


I agree that it would be great customer service for them to do, and they probably will. The problem I have is that people EXPECT them to do it. Because they have been wronged by the evil game people. That is where i draw issue. It should not be a thing of "I demand to be comped". It should be a thing of "Oh nice they care enough to actually give it to us".

Quit your bitching and fly!
SN50049177 1
Re-Hab
#197 - 2013-06-03 14:07:04 UTC  |  Edited by: SN50049177 1
Melissa Turner wrote:
you know im really getting sick and tired of folks out there getting mad about folks wanting to know they'll get "compensated" for an unexpected service disruption... its called Customer Service and is a great policy to follow in order to make sure customers are happy. for those that dont know.. when sony was hit with the PSN attack. not only did they apologize to their 77 million effected gamers, they also gave 5 free full games, free playstation plus service, and a special msg at E3 about the entire situation. it was excellent customer service and ccp should rightfully follow that trend.


So you're sick and tired of hearing ppl ask for compensation, yet you really think CCP should give it... Or did I misunderstand?

You know you're implicitly asking right?
Melissa Turner
The Scope
Gallente Federation
#198 - 2013-06-03 14:11:49 UTC
Adam1337 wrote:
Melissa Turner wrote:
you know im really getting sick and tired of folks out there getting mad about folks wanting to know they'll get "compensated" for an unexpected service disruption... its called Customer Service and is a great policy to follow in order to make sure customers are happy. for those that dont know.. when sony was hit with the PSN attack. not only did they apologize to their 77 million effected gamers, they also gave 5 free full games, free playstation plus service, and a special msg at E3 about the entire situation. it was excellent customer service and ccp should rightfully follow that trend.




In the PSN attack, tell me how any accounts where compromised, was it suspect 10 million..?

So far as we know not one account has been compromised in the Tranquility attack, so yes I agree a token gift to say thank you to all would be a nice gesture, but I think getting 5 free games for my Playstaion and an access pass to the plus service is a little to much to expect.

Let Odyssey roll out, if it arrives without issue then be happy if you get some sp, if it has issues I am confident some big bundle of gifts might be in order...



those that were compromised received direct communication and as well was advised to contact their banks. and since then still remain loyal sony players. and not one even came forward and blasted sony for something that was not their fault.. so even that 10 mil number was with a ? mark. havent heard anything about it since then.. so yeah excellent customer service.. im pretty sure xsnot one will cuss you out when you ask it to connect to eve online.. oh wait... that's right ccp gave them a middle finger.. i wonder why? hmmmmmm and besides you must protect yourself these days.. so dont even use a real CC when there's readily available psn time cards.. just sayin.. again excellent customer service providing an alternative.
Daemon Ceed
Ice Fire Warriors
#199 - 2013-06-03 14:14:14 UTC  |  Edited by: Daemon Ceed
There are 3 guarantees in life: death, taxes, and that there will be big whiny babies in Eve.

That being said, lets have a nice hot cup of realism here. There are vulnerabilities in everything and it's damn near impossible to patch every single one of them. So long as software is written by fallible beings (humans), there will be exploits and holes. Like with any company that deals with computing, they often are reliant on the software companies that run their underlying software to patch their code and root out vulnerabilites. Unless you wrote ALL of the code yourself (web server, database engines, network protocol stack, etc) you will lose certain amounts of control over what holes are in the code itself. I think CCP did a great job working to get all of us Eve addicts back online. They also took responsibility for the exploit being an issue with their own code. Most companies would just point the finger at a 3rd party software vendor to save face - but CCP did the honest thing and shouldered the blame and took ownership of the problem.

For those who aren't familiar with DDoS attacks and haven't had to spend the effort to thwart one, they are one of the hardest attacks to mitigate. I'm actually surprised they were able to get back online as fast as they did, all things considered. I think the offer to compensate the customers is warranted, because afterall we pay for the service - and we appreciate it. So to the whiners: pull those tight fitting thongs from your cracks and HTFU. One could only wish all companies had the uptime of CCP.
SN50049177 1
Re-Hab
#200 - 2013-06-03 14:14:19 UTC  |  Edited by: SN50049177 1
Why is everyone still here?... the servers are up you know?... eh wait, hold on... oh nevermind Roll