These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

Unexpected Dowtime Update - Tranquility Is Back Online

First post
Author
Kitty Bear
Deep Core Mining Inc.
Caldari State
#221 - 2013-06-03 15:11:16 UTC
Clause 11



but I won't complain if there is compensation Cool
Korinne
The Partisan Brigade
#222 - 2013-06-03 15:14:22 UTC
So people got socket punched and forced to log off in space with aggression timers, then people couldn't log in. Then they took down the server and rebooted, but still certain people couldn't log in, so they took it back down.

This being the case, why didn't CCP just reset Tranq to the previous downtime? I ask because in that interim where certain people still couldn't log in, and even before when only certain people were being kicked and locked out, alot of stuff happened. Ships were exploded, pods lost, structures destroyed, reinforcement timers ticked away, towers went offline.

Normally, this would be business as usual being Eve, **** happens. However when the **** only happens because people aren't physically capable of logging in to defend their ****, while other people are somehow magically able to log in to destroy it, it raises questions.

I know CCP has stated they're talking about compensation, but physical compensation means little when compared to reinforcement timers, and the fact that it will take DAYS of time to take back what was lost; days I could be spending farming or whatever.

Given all that, it seems like the most reasonable thing to do would have been to reset the server to the previous downtime, but that didn't happen. Why didn't that happen? That would be fair and it would hurt NOBODY, whereas letting stuff go on hurts alot of people and is wholly unfair. Perhaps if the problem were localized on the customer's end, I could understand, but it wasn't. The problem was on CCP's end, and therefore could not have been remediated by anything I could have done.
Ray Morrison
Doomheim
#223 - 2013-06-03 15:16:43 UTC
GREAT JOB CCP !!!!!!!!!!!!!!!!!!! and thank you very very much for all the updates. Tis a shame the hamsters died What?, and the coffee pot burned up so thank you very much for you hard and tireless work getting Eve back up and running. As my nation says "Ahééhee'" (Navajo/native american)
Callic Veratar
#224 - 2013-06-03 15:20:48 UTC
I think a reasonable compensation would be a 1-Day PLEX card. One per account. If isk is important, sell it, if time is important, use it, if collecting is important, save it forever.

If events like this happen in the future, pass out more 1-Day cards (unless it starts happening regularly...).
Mr Epeen
It's All About Me
#225 - 2013-06-03 15:25:15 UTC
There was a weekend event in another game that I'm to embarrassed to admit I play. I didn't even realize Tranquility was down until it was almost up again.

Mr Epeen Cool
DeAira Skord
Brutor Tribe
Minmatar Republic
#226 - 2013-06-03 15:26:16 UTC
So, someone griefed the entire Eve Online community. Since sociopathic behavior is encouraged in Eve, then this is perfectly acceptable, right?

Antecursor Venatus
Classis Argentum
#227 - 2013-06-03 15:28:54 UTC
I´d like to echo the thanks for a problem well handled. Good job CCP!


So I couldn´t log in to EVE for a day - big deal.
I don´t know about anyone else, but EVE isn´t the only game I play, and computer gaming isn´t the only thing I do to relax.
If not being able to play EVE for a day is a disaster for you, I´d suggest visiting a psychatrist - NOW! Because you sir, have a serious problem Blink

Also, to those that want compensation:
We are training chars for months, years even, to get them to where we want them.
We train single skills that take weeks to complete, and you are complaining about 20 hours lost? Are you nuts?

Now don´t get me wrong, if CCP hands out free SP, I won´t reject them, but I will not be asking for them, cause I don´t think any compensation is needed.
Immortalmyth
Noble Guardians
#228 - 2013-06-03 15:30:08 UTC
for all the people thanking ccp it seems to me that your overlooking something,,,

the launcher for us english speakers was in russian...

a ddos attack wouldnt affect the launcher on your local computer, unless the launcher was trying to
use bit torrent type connections to distribute ccp software,

so what kind of thanks does ccp deserve wen the apparent stealing of your bandwidth is uncovered..


-imm
Mark Rain
Deep Core Mining Inc.
Caldari State
#229 - 2013-06-03 15:38:01 UTC
Quote:
We will be looking at ways to compensate players in both EVE and DUST for the outage and expect to announce what that compensation will be very soon.


http://community.eveonline.com/news/news-channels/eve-online-news/tranquility-downtime-on-sunday-june-2-and-monday-june-3/



Looks like no one will be left hanging on skill downtime....thanks CCP :)
1esus isa4abdul
COMMUNISTS
#230 - 2013-06-03 15:38:30 UTC
Adaahh Gee wrote:
Awesome, it seems that New Eden has been frozen, a wreck is still in place from before the attack, as is my alts C3 high sec exit. No more scanning needed for now then!


Is it just my imagination or has anybody else´s skilling been "frozen" too?
No Alibi
Sometimes Here
#231 - 2013-06-03 15:40:04 UTC

.
If you open the "bin\ExeFile.exe", you can log in no worries.

That is what people were doing yesterday, the launcher wasn't working (for whatever reason) but you could still login using the other method.

BTW thanks CCP for getting it back up and running.[/quote]

Wait..What?!?Shocked How did you log into an OFFLINE server??? HMMM???Lol

I fly by the seat of my pants, No wonder my ass is always on fire!

drillerkiller2004
Brazilian Vultures
Ferrata Victrix
#232 - 2013-06-03 15:43:43 UTC
Neddy Fox wrote:
drillerkiller2004 wrote:
I very much see CCP as a vendor offering me Software As A Service. Whilst I commend CCP for all of their hard work, and totally appreciate why they disconnect the cluster from the internet during the DDOS; it does **** me off that they haven't put adequate protection in place to stop this from happening.

They take a serious amount of money from us, the least they could do is protect their network from such attacks.

CCP I suggest you look into buying something like Cisco IPS Intrusion Detection System devices, then use a third party such as Qinetic to monitor your network for such attacks. As is you have failed the community a second time. Get your 5h1t sorted out!


LOL. Ignorance is bliss. I too run a Cisco IPS to protect our small network. You think it withstands DoS or DDos? Or do you think they don't know what happened?
Before giving advice, get all the information that's needed to give that. And YOU don't have it.



Shut up
Great Revenge
Silverhawk Industries
#233 - 2013-06-03 15:49:12 UTC  |  Edited by: Great Revenge
While I was rather sad to be unable to play EvE after only recently re-signing up. I completely understand and wholeheartedly agree with your decision.

I also understand you will likely be unable to give 100% of the information regarding the attack we would all LOVE to hear! Shocked

Keep up the good work, perhaps we can all think of yesterday's event as a fire drill to which the Dev Team responded wonderfully.

Bravo.

In regards to protecting yourself in the future....I have come to the realization that people consider rules as "meant to be broken" and a Fort Knox network as "a challenge to be overcome"

Nothing is 100% secure, only 99.9%. Expect that one day the .1% haxor may come knocking on your door. It's not a matter of "if" it's a matter of "when."
Unsuccessful At Everything
The Troll Bridge
#234 - 2013-06-03 15:50:26 UTC
I would just like to offer my sincere thanks to all the employees at CCP who sacrificed their weekend to fix the issues spawning from the DDoS attack. You guys do a bang up job, and dont get nearly the credit you all deserve.
I would also like to apologize in advance for the certain percentage of the playerbase who feel its necessary to demand compensation. Frankly it was out of line for those demands to be made while you guys were knee deep in server trying to get things running again. In my honest opinion, no compensation is due. So once again I apologize for that contingent of players who will 'New Topic' bomb the forums with entitled whine threads.

To show my good faith and how sincere I am about the thank you, I will refrain from trolling on the forum today.

Dont get used to it. Big smile

Thanks again, UaE.

Since the cessation of their usefulness is imminent, may I appropriate your belongings?

THC Trader
Doomheim
#235 - 2013-06-03 16:04:44 UTC  |  Edited by: THC Trader
Popsikle wrote:
THC Trader wrote:
Popsikle wrote:
Phobos Eversor wrote:
Quote:
We apologize for any inconvenience that this extended downtime has caused.


You... you what?

You stayed up all night for us. You gave up your weekend for us. You updated twitter every 3 hours for us. Your coffee pot valiantly died in the line of duty for us.

You don't need to apologize to us; You need a nap, a raise, and a new coffee pot.

Thanks for the hard work.


While I do appreciate CCP and normally support them (fanboi) this situation was completely avoidable. They took all of the servers down because there was a patchable exploit in their infrastructure. The keyword in that last sentence is patchable. We (collectively) pay them lots of money, and this sort of flaw in the platform should have been identified before its too late. Yes, I know some exploits are 0-day and such, but since they patched it it had to be an exploit that was on someones radar. Suggesting someone gets a raise because of this is the opposite of what should happen tbh. Someone should get a yelling at and possible termination for not being on top of security.

That said they did not stay up for us, they stayed up because its their job. They could not work around the clock to get the servers patched and expect the userbase to keep paying them. Now CCP isn't a giant machine, and I am sure a lot of the people were sympathetic with us because they do also play the game, but even if they weren't i'm sure it was an "ALL HANDS ON DECK" call from above. I know I have been in situations similar where sure, I want to make things available to customers because it sucks for everyone things are down but everyone is working on the problem because that's what they get paid to do.

I really do appreciate the information on why they took everything down instead of the bs line about this being a DDoS attack. That is what agitated me most yesterday was that they were blatantly lying to everyone about what happened. Glad to see they came clean about the issue today!

I do also agree with the decision to shut everything down to figure out what the hell happened and get it fixed in how every long it takes before bringing it back online. That was very responsible of them, and I appreciate them protecting my information ;)

You're ********. It was definitely a DDoS attack. The attack took advantage of a (until now) unknown vulnerability in their servers. They took the servers down to evaluate the situation, and make sure customer data had not been compromised, while also working on a patch to remove the vulnerability. CCP handled the situation very well, and you're a jackass for claiming otherwise. Go back to WoW with your entitlement attitude. See what happens when your 2000 man WoW server gets DDoSed. Bet it's down for a month.



Sorry, bittervet (here since beta with my 6 accounts) so going back to anything isn't an option. They fact is they patch and exploit in their setup. You set something on the interwebs that's exploitable its going to get exploited. Its not the attackers fault your **** wasn't secured...Why was a back-end service accessible from the internet anyway?

Also, trad the whole post, I think they handled the situation fine, and did what they were supposed to do when an exploit is found, but they did not need to feed us the bullshit line that a DDoS is taking everything out. DDoS's cause latency and fill pipes, sometimes take out services but are generally not crafted attacks, they are generic attacks sent out by drones. This was a crafted attack and they knew that from the start which is why they shut down TQ (which is what any sensible company would have done) and then did not let the attacker know they were on to what was going on, which is a good thing BUT call a spade a spade and drop the word DDoS from their updates. "We are currently undergoing a sustained attack and are investigating it" would have been way more appropriate than what they said.


Since you clearly don't know what you're talking about, I'll spell it out for you.

For one, CCP obviously didn't know the vulnerability was present until it had been attacked. This has caused plenty of issues for them, I highly doubt they would intentionally put vulnerabilities in their server or knowingly allow them to exist. There's little that can be done to prevent them 100% of the time. Mistakes happen.

For two, per wikipedia, a Denial of Service attack is defined as follows: "In computing, a denial-of-service attack (DoS attack) or distributed denial-of-service attack (DDoS attack) is an attempt to make a machine or network resource unavailable to its intended users. Although the means to carry out, motives for, and targets of a DoS attack may vary, it generally consists of efforts to temporarily or indefinitely interrupt or suspend services of a host connected to the Internet."

Any Denial of Service attack is crafted. There are some well known ones that target basic protocols such as TCP, and are included in just about any botnet. However, the most effective DoS attack is one that is specifically tailored to the service you are attacking. There are attacks that are crafted for the HTTP protocol, ICMP protocol, etc. They exploit core weaknesses in the protocol itself. There are other attacks that target a specific version of a specific server. These attacks generally make the server do excessive amounts of work, resulting in all available resources being exhausted. This prevents the server from processing legitimate requests. It still denies service, and thus it is still a Denial of Service. The only difference between a Denial of Service, and Distributed Denial of Service, is one uses a single computer to launch the attack, and the other uses multiple computers. I would trust CCP knows how to differentiate between the two.
Aidan Brooder
Dynasphere Ltd.
#236 - 2013-06-03 16:05:53 UTC
Oh, it was a DDoS attack? I thought they had brought the servers down so the GMs could haul all the Ice away from the belts and hide it.

Great job CCP. While it was a shame the servers were down, it was fun to linger at EVE radio and get the various conspiracy theories... ("The servers are down because you are all bad people and touch yourself at night. Stop doing that!")

Blog: http://aidanbrooder.wordpress.com My EVE Playlist on YouTube: http://www.youtube.com/playlist?list=PLSNuHY7z8n1q1BdLvW2verIfH8vvWtz_x

Max Sneak
LightningStrikesTwice
Invidia Gloriae Comes
#237 - 2013-06-03 16:07:55 UTC  |  Edited by: Max Sneak
Parris Colidi wrote:
As with other players..

Thx CCP for the working overnight/weekend to get this back up.

As an IT professional myself, and having to things like this numerous times, i can feel your pain..

Time for a large number of stiff drinks..

Cheers..



Same - been working in this industry - mmmm over 3 decades (damn)

/me segues to when he got his first glass 10MB winchester disk drive... oh those halcyon days

anyways +1

oh and you forgot to add the chunks of cheese that go with the drinks

and yes 'Well Done' CCP
Bugsy VanHalen
Society of lost Souls
#238 - 2013-06-03 16:28:51 UTC  |  Edited by: Bugsy VanHalen
No Alibi wrote:

.
If you open the "bin\ExeFile.exe", you can log in no worries.

That is what people were doing yesterday, the launcher wasn't working (for whatever reason) but you could still login using the other method.

BTW thanks CCP for getting it back up and running.


Quote:
Wait..What?!?Shocked How did you log into an OFFLINE server??? HMMM???Lol

What was SiSi up? were you playing on that server?

TQ was down and unable to connect for most of the weekend for me. And the exefile.exe in the bin folder did not get around it. That is how I have logged in all the time since the new launcher was released. I only use the launcher for patching.

CCP, With this being the last weekend before Odyssey I had planed on spending the entire weekend, or at least as much time as I could ICE mining. I am sure many others were doing the same. All four of my accounts were in the belts mining went the servers went down. i was not able to reconnect until this morning just before the normal sever down time. Semi AFK mining as I do when mining ICE in high sec, I would done at least 12 hours of mining over the weekend with an 4 account fleet of 3 Macks and an ORCA.

My intention was to mine enough ICE for POS fuel to survive until things leveled out after the expansion. Not the expansion is coming tomorrow, and I have no ICE. not your problem but mine, I will survive, but any compensation would be greatly appreciated.

I do not in any way Blame CCP for this, such an attack is not something you can see coming, or prepare for. if you had addressed the issues that gave them the needed hole to pull this off before it happened, they would have just found another hole.

But you may want to wake up, and consider what the possible reasons behind this attack may have been. this may just be the paranoid conspiracy theorist in me, But I do not want to see all the time I have spent developing my characters in EVE go down the tubes. I hate starting fresh in a new game. Especially when I have been playing for several years.

There are many many players not at all happy about some of the changes coming with Odyssey. CCP's only answer to these complaints is suck it up and adapt.

Well it would seem to me somebody with the skills to do something about it, did not like that reply. maybe this had nothing to do with it, but if that is the case it may not be over. Obviously someone with the skills and knowledge needed, was ignorant enough to pull off this attack. I do not believe that the fact it was the last weekend before Odyssey was a coincidence.
Someone who has rage quit over Odyssey may be behind this.

If someone was frustrated enough with the ignorance shown by CCP to player concerns over the coming changes to not only quit the game, but has the skills to lash out at CCP in this manner, this may not be over. i really hope CCP did a good job patching those holes this weekend.
Ray Morrison
Doomheim
#239 - 2013-06-03 16:35:49 UTC
drillerkiller2004 wrote:
Neddy Fox wrote:
drillerkiller2004 wrote:
I very much see CCP as a vendor offering me Software As A Service. Whilst I commend CCP for all of their hard work, and totally appreciate why they disconnect the cluster from the internet during the DDOS; it does **** me off that they haven't put adequate protection in place to stop this from happening.

They take a serious amount of money from us, the least they could do is protect their network from such attacks.

CCP I suggest you look into buying something like Cisco IPS Intrusion Detection System devices, then use a third party such as Qinetic to monitor your network for such attacks. As is you have failed the community a second time. Get your 5h1t sorted out!


LOL. Ignorance is bliss. I too run a Cisco IPS to protect our small network. You think it withstands DoS or DDos? Or do you think they don't know what happened?
Before giving advice, get all the information that's needed to give that. And YOU don't have it.



Shut up


The Lulz Boat ‏@LulzSec 14 Jun 11
We just wiped out the login server for Eve Online, and it accidentally took their website out at the same time: http://www.eveonline.com/


Collapse Reply
Retweet

Favorite


More


Diaishi Akye
State War Academy
Caldari State
#240 - 2013-06-03 16:42:18 UTC
Braldt Ironeye wrote:
CCP Falcon wrote:
As of 10:15 UTC today, Tranquility is back on line after an unexpected overnight downtime.

Our engineers have worked overnight to resolve the cause of the downtime, and we are currently monitoring the cluster to ensure that the issues which caused the situation are fully resolved.

We apologize for any inconvenience that this extended downtime has caused.

You can read more about what caused the problem in a statement from our Chief Operating Officer here.


I don't care what everyone else has been complaining about with this thread, and the previous one. I myself will say this.

Thank you for your hard work, especially on a Sunday to fix and block the DDoS attack.. Your work is appreciated, even if it's not stated very often. Now, tell your engineers to go get some sleep.

-Braldt

I'll Drink a Coffee to that, well said! =0)

Those who cannot remember the past are condemned to repeat it without a sense of ironic futility. Errol Morris