These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

Unscheduled Downtime Extension – July 15, 2015

First post First post First post
Author
Howard Carpenter
Doomheim
#4481 - 2015-07-15 20:47:37 UTC  |  Edited by: Howard Carpenter
CCP Falcon wrote:
[img]http://i.imgur.com/2P4UoRb.gif[/img]

Hey guys,

As you’re all aware, Tranquility is still offline after today’s scheduled daily downtime.

This is due to an issue with a code deployment today while patching for a few fixes in EVE Online. The issue caused the cluster to fail to start up correctly, and as such, Tranquility is currently unavailable for play for both EVE and DUST.

Our operations and development team are working on this issue as we speak, and we’re hoping to have Tranquility back online for you guys as soon as possible.

Right now, we don’t have an ETA for when the cluster will be back online and accepting connections, but we’ll keep you updated in this thread.

Cheers,

- F

UPDATES:

13:00 UTC - AFTER SEVERAL FAILED STARTUPS, WE ARE CURRENTLY INVESTIGATING A STARTUP ISSUE
13:30 UTC - INVESTIGATION IS STILL ONGOING - NO FURTHER ETA ON UPTIME FOR TRANQUILITY
13:37 UTC - STARTUP FAILED AGAIN - INVESTIGATION IS ONGOING
14:00 UTC - INVESTIGATION IS STILL ONGOING
14:30 UTC - WE ARE CURRENTLY IN THE PROCESS OF REBOOTING AFTER FAULTFINDING AND CHANGES
15:00 UTC - FURTHER ATTEMPTS TO START UP HAVE FAILED - INVESTIGATION IS ONGOING
15:30 UTC - INVESTIGATION STILL ONGOING - NO ETA ON SERVER UPTIME
16:00 UTC - A FURTHER FAILED STARTUP AFTER A NUMBER OF CHANGES - INVESTIGATION ONGOING
17:00 UTC - INVESTIGATIONS STILL ONGOING - NO ETA ON SERVER UPTIME
17:30 UTC - WE ARE CURRENTLY FAULTFINDING - NO ETA ON SERVER UPTIME
18:00 UTC - AFTER TESTING ON SINGULARITY WE WILL BE REBOOTING TQ TO TEST A FIX
18:30 UTC - WE ARE CURRENTLY STILL FAULTFINDING ON TQ - NO ETA FOR UPTIME AT THIS STAGE
19:00 UTC - WE ARE CURRENTLY TESTING A FIX FOR THE ISSUES ON TQ - NO ETA FOR UPTIME YET
19:30 UTC - TRANQUILITY IS STILL CURRENTLY BEING EXPERIMENTED ON - WE WILL CONTINUE TO UPDATE
20:00 UTC - PIZZA HAS BEEN ACQUIRED AND WE ARE CURRENTLY REFUELLING OUR DEVS!
20:30 UTC - DEVS REFUELLED - TESTING HAS RESUMED!


PIZZA-POWER! Nothing can go wrong!
Take your time - don't waste your health on this game
MrLuciferi
Cambridge Mountain Rescue
The Monarchy
#4482 - 2015-07-15 20:47:51 UTC
Grinder2210 wrote:
MrLuciferi wrote:
99.9% sure there won't be any reimbursements. Suck it up kids, and welcome to life. This can happen to anyone, and any company.




Na i think with the length of time we have been down now there prob will be some sort of reimbursements Prob skill points witch is CCP's normal go to in the sort of soituation


Maybe, hence my 99.9%. It's nothing I count on will happen. If they want to, and feel like it, we will probably get something. But I don't expect do be given something for free. Smile
Leeloo Killik
Fweddit
Free Range Chikuns
#4483 - 2015-07-15 20:47:56 UTC
Equinox WhiteWolf wrote:
Jove Delegate wrote:
Mdes Ormand wrote:
01100101 01110110 01100101 01110010 01111001 01110100 01101000 01101001 01101110 01100111 00100000 01110100 01101000 01100001 01110100 00100000 01101000 01100001 01110011 00100000 01100001 00100000 01100010 01100101 01100111 01101001 01101110 01101110 01101001 01101110 01100111 00100000 01101000 01100001 01110011 00100000 01100001 01101110 00100000 01100101 01101110 01100100 00101110 00100000 01101110 01101111 01110111 00100000 01101101 01111001 00100000 01110100 01101001 01101101 01100101 00100000 01110100 01101111 00100000 01100111 01101111 00101110 00100000 01110011 01100101 01100101 00100000 01111001 01101111 01110101 00100000 01110000 01100101 01101111 01110000 01101100 01100101



0110110001100001011101000110010101110010001000000110111100110111



01200011010111



I see what you did there. Naughty naughty
Qprime
Center for Advanced Studies
Gallente Federation
#4484 - 2015-07-15 20:48:03 UTC
@ CCP Falcon

After reading all... well, a good chunk of this thread, there are a few things I'd like to say (woot -- first time bothering to post on the forums btw)

1) Good for you for feeding your employees CCP. Nice to see a company treating it's workers like humans. Serious points in my book for that.

2) On the topic of all the whining about getting compensated for this DT, while that would be nice there is really only one thing I honestly want out of all this: A devblog or article going into serious detail about what dafuq happened and exactly why there was no rollback option. I feel that is a perfectly reasonable request -- and it gives you, CCP Falcon, something REALLY useful to do in the next few hours until this finally blows over. Instead of sitting here shooting the ****, go interview the techs (that aren't pulling their hair out) and find out just what is wrong and start composing an article methinks.

3) Tell the techs for me: Good luck and godspeed on getting this fixed.

~Q
aPoQ
Marzipan Meerkat Flying Circus
#4485 - 2015-07-15 20:48:03 UTC
Quattras Peione wrote:
Srly guys, enjoy the pizza. We know you're working hard for it. Ignore the gripes and know the grownups in the room really appreciate the work you're putting in just so we can have internet spaceships.


Mummy pays for your sub then...

If you bought a plane ticket and on arrival at the airport it wasn't there you'd go spanner. This is no different.

CCP adopting the casual approach to informing us of the situation AND their casual approach at change control (i.e rollback procedure? support plan? ) would be totally fine if this were a free game. It is not. Subs = customers, ergo require a level of service delivery.
Interweb space ships or airplane tickets, you bought into a service and they, in this case CCP need to deliver. This should be a ******* massive wake up call to the current state of the process they use to roll out changes in a production environment.

Someone / people have dropped the ball and as such someone really needs to step up and sort this out. This current issue can not be solved with a wave of a finger BUT mitigating issues reoccurring should now be a priority.
This used to happen a fair bit back in the day, what is worrying is we are seeing it happen again despite the obvious improvements to hardware and software.

Major outages such as this really should be, and in most cases are, a thing of the past.

CCP step up and explain what actually is happening to rectify this and why it occurred in the first place, and it needs to be someone the fine pilots of this great game respect.

Fobbing us off with lip service is insulting.

ps having ordered the pizza i wonder your reaction if they were cold, mashed up and inedible....guessing you would want it sorted??! What?
Lt Motion
Royal Amarr Institute
Amarr Empire
#4486 - 2015-07-15 20:48:05 UTC  |  Edited by: Lt Motion
Emma Verondi wrote:
Nathan32Derby UK wrote:
20:00hrs ... PIZZA !!!! Fckin' PIZZA !!!! are you having a laugh? Maybe the players who pay your wages should be a bit " go slow " when it comes to resubscribing. 3 or 4 days won't hurt us, but it'll sure hurt CCP in the pocket.

Here's a hint, undo the patch that you applied ( that's knackered ) and run the servers without the new patch until you can afford to pay for proper coders to write patches that don't crash the system.


You have obviously never worked with IT
Consider the timers etc, how would they react on a full code and DB-rollback?
If the server is unrecoverable, I am sure they will reload a previous backup, but that's nothing short of a last resort



To the original poster. Please just chill out mate, these things happen to all companies. Even fortune 500 companies can go isolated over an entire country if a network failure happens

Going out for a smoke and thinking about the problem often times gets me the solution at work, so itl be the same for our friends in Iceland. Let them catch their breath and approach this with a fresh eye.

Go go CCP, you guys can do this !. get us back on so we can keep killing each other

(whilst writing this im making peace with the fact that ill probably lose the 3 bil I made before DT)
Yasemin Hanim
Blackwater USA Inc.
Pandemic Horde
#4487 - 2015-07-15 20:48:08 UTC
CCP Falcon wrote:
[img]http://i.imgur.com/2P4UoRb.gif[/img]

Hey guys,

As you’re all aware, Tranquility is still offline after today’s scheduled daily downtime.

This is due to an issue with a code deployment today while patching for a few fixes in EVE Online. The issue caused the cluster to fail to start up correctly, and as such, Tranquility is currently unavailable for play for both EVE and DUST.

Our operations and development team are working on this issue as we speak, and we’re hoping to have Tranquility back online for you guys as soon as possible.

Right now, we don’t have an ETA for when the cluster will be back online and accepting connections, but we’ll keep you updated in this thread.

Cheers,

- F

UPDATES:

13:00 UTC - AFTER SEVERAL FAILED STARTUPS, WE ARE CURRENTLY INVESTIGATING A STARTUP ISSUE
13:30 UTC - INVESTIGATION IS STILL ONGOING - NO FURTHER ETA ON UPTIME FOR TRANQUILITY
13:37 UTC - STARTUP FAILED AGAIN - INVESTIGATION IS ONGOING
14:00 UTC - INVESTIGATION IS STILL ONGOING
14:30 UTC - WE ARE CURRENTLY IN THE PROCESS OF REBOOTING AFTER FAULTFINDING AND CHANGES
15:00 UTC - FURTHER ATTEMPTS TO START UP HAVE FAILED - INVESTIGATION IS ONGOING
15:30 UTC - INVESTIGATION STILL ONGOING - NO ETA ON SERVER UPTIME
16:00 UTC - A FURTHER FAILED STARTUP AFTER A NUMBER OF CHANGES - INVESTIGATION ONGOING
17:00 UTC - INVESTIGATIONS STILL ONGOING - NO ETA ON SERVER UPTIME
17:30 UTC - WE ARE CURRENTLY FAULTFINDING - NO ETA ON SERVER UPTIME
18:00 UTC - AFTER TESTING ON SINGULARITY WE WILL BE REBOOTING TQ TO TEST A FIX
18:30 UTC - WE ARE CURRENTLY STILL FAULTFINDING ON TQ - NO ETA FOR UPTIME AT THIS STAGE
19:00 UTC - WE ARE CURRENTLY TESTING A FIX FOR THE ISSUES ON TQ - NO ETA FOR UPTIME YET
19:30 UTC - TRANQUILITY IS STILL CURRENTLY BEING EXPERIMENTED ON - WE WILL CONTINUE TO UPDATE
20:00 UTC - PIZZA HAS BEEN ACQUIRED AND WE ARE CURRENTLY REFUELLING OUR DEVS!
20:30 UTC - DEVS REFUELLED - TESTING HAS RESUMED!



Do not touch the keyboard with Pizza Hands plz ! Lol
Aldran Gentlharp
Unidentified Murderers and Young Hack Offenders
#4488 - 2015-07-15 20:48:09 UTC  |  Edited by: Aldran Gentlharp
Here some Cap booster charges for the Dev Big smile

Cap Booster
Grisha Altol
Viziam
Amarr Empire
#4489 - 2015-07-15 20:48:21 UTC
Warlof Tutsimo
Daerie Inc.
#4490 - 2015-07-15 20:48:21 UTC
CCP Falcon wrote:
20:30 UTC - DEVS REFUELLED - TESTING HAS RESUMED!


My snowball tell me that you're on the good way ! Stay up with my dwarf https://www.youtube.com/watch?v=NVhOulJ9lo4
ksolis2
Salt Miners Union
SONS of BANE
#4491 - 2015-07-15 20:48:26 UTC
Kesinik wrote:
Dr. Dice wrote:
What kind of pizza?


All of them https://twitter.com/CCP_Falcon/status/621407777467052032



forget the servers now, i want some of that pizza now they better share that
Dr. Dice
Aurum-Sol
#4492 - 2015-07-15 20:48:26 UTC
Well I'm off to get pizza...
Kitsune Koraka
Caldari Provisions
Caldari State
#4493 - 2015-07-15 20:48:35 UTC
Drgzs wrote:
Nathan32Derby UK wrote:
20:00hrs ... PIZZA !!!! Fckin' PIZZA !!!! are you having a laugh? Maybe the players who pay your wages should be a bit " go slow " when it comes to resubscribing. 3 or 4 days won't hurt us, but it'll sure hurt CCP in the pocket.

Here's a hint, undo the patch that you applied ( that's knackered ) and run the servers without the new patch until you can afford to pay for proper coders to write patches that don't crash the system.


Yes, 20:00. Yes, Pizza. yes, Fckin' PIZZA !!!! Because, unlike you, they deserve it. Calm your **** and "can i haz your stuff'?



out of order tbh.
Maetel Lithium
Undead Dragons
#4494 - 2015-07-15 20:48:41 UTC
I just had Pizza while I was out. Yum. Programmer fuel.
Algarion Getz
Aideron Corp
#4495 - 2015-07-15 20:48:55 UTC
CCP PLS, CALL CCP VERTIAS, HE FIXES SERVER IN 5MIN

LOOK AT HIS 1337 MLG CODING SKILLZ https://www.youtube.com/watch?v=wBX6cAtpkC8
Sagnon
Greysno
#4496 - 2015-07-15 20:49:01 UTC
After Pizza, is it not nappy nap time?
!Even DEVS need their nappies.
Shocked
Krantwak Pino
Center for Advanced Studies
Gallente Federation
#4497 - 2015-07-15 20:49:07 UTC
300 paGES CONFIRMED ?
Vincent Athena
Photosynth
#4498 - 2015-07-15 20:49:10 UTC
I told my wive that she should not scared, but the ENTIRE UNIVERSE has ceased to exist.

She gave me her standard "You're crazy" look.

Know a Frozen fan? Check this out

Frozen fanfiction

MCKen3311 Daugherty
F0RCED ENTRY
Solyaris Chtonium
#4499 - 2015-07-15 20:49:11 UTC
While we are waiting, here's my first attempt at Jackdaw Solo PvP... more than likely going to tweak my fit to make it better...

https://www.youtube.com/watch?v=2p4sUDEkR1s
Bea DenSotreni
Mine Mine Meins Inc.
#4500 - 2015-07-15 20:49:12 UTC
01000011 01101111 01101110 01110110 01100101 01110010 01110100 00100000 01000010 01101001 01101110 01100001 01110010 01111001 00100000 01110111 01101001 01110100 01101000 00100000 01101000 01110100 01110100 01110000 00111010 00101111 00101111 01110111 01110111 01110111 00101110 01000011 01101111 01101110 01110110 01100101 01110010 01110100 01000010 01101001 01101110 01100001 01110010 01111001 00101110 01100011 01101111 01101101 00100000 00101000 01001111 01101110 01101100 01101001 01101110 01100101 00100000 01000010 01101001 01101110 01100001 01110010 01111001 00100000 01010100 01110010 01100001 01101110 01110011 01101100 01100001 01110100 01101111 01110010 00101001