These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

UPDATED - Unscheduled Downtime - November 4th

First post First post
Author
Previous Topic Next Topic
Malchristus
Sebiestor Tribe
Minmatar Republic
#81 - 2012-11-04 03:57:33 UTC
Daemon Ceed wrote:
nortelslug wrote:


100% uptime HA costs HUNDREDS OF MILLIONS of dollars, plus full hardware replacement in multiple datacenters at regular intervals + running a backup internet. For $200/month@400K players, it MIGHT be doable.

**** happens. Move on with your life and come back tomorrow. The world did not end.


Hundreds of millions? Not quite. You can get very close HA (the OS's would have to reboot) with a VMWare cluster running your hosts and redundant SAN's connected to the cluster. I spoke to a Dev @ Eve Vegas and the issue running on VMWare ESXi was that when a node migrates from one server to another it either hangs on re-building the node. It seems more of a software issue than anything else, but if they really wanted to I'm sure they could find a solution to the problem.

Plainly put, relying on physical hosts to run everything on bare metal leads to long downtimes. Utilize virtualization, snapshotted data storage or continuous replication, and you can effectively ensure 99.99% update and razor thin RTO/RPO's for a few hundred thousand for a medium sized business. Not hundreds of millions. I do it at my job every day and have never had a time where systems weren't live within 5 minutes or less during a catastrophic failure. We also use Site Recovery so with a single click of a button we can bring our systems online at a secondary site within minutes should we our primary datacenter gets cratered.


Yes but do you get laid ........... ever?
Helena Russell Makanen
DRRUSSEL
#82 - 2012-11-04 03:57:43 UTC
Gyala Frey wrote:
And I just resubbed today after being gone over a year. Bad omen?


Yes and all of Eve hates you. P

"If a miner needs to go to the bathroom, for instance, I ask that they dock up first, or at the very least ask the Supreme Protector for permission to go."  -  James 315 - aka - the miner bumper

Ando Gart
Almost Dangerous
Wolves Amongst Strangers
#83 - 2012-11-04 03:58:01 UTC
Oh come on, youre keeping me up at 5 am....
Theo Ramone
Stryker Industries
Stryker Group
#84 - 2012-11-04 03:58:03 UTC
Hottspitta jR wrote:
Mrs Doubtfire wrote:
From Twitter:

EVE Online ‏@EveOnline

This is looking to be a potential hardware issue. More updates soon as we learn more. Sorry for the current lack of pew pew!


Please eleborate.

How on earth would a single isolated hardware issue, be a problem to your servers? I would expect your systems to be redundant??


Even the most redundant systems have their moment's. You can never make something 100% solid. Can only try your best to achieve. This is pretty damn rare... Doesn't happen like this every year...


Thats not entirely true.

Redundancy costs money. How redundant do you want to be?
Helena Russell Makanen
DRRUSSEL
#85 - 2012-11-04 03:58:22 UTC
Malchristus wrote:
Daemon Ceed wrote:
nortelslug wrote:


100% uptime HA costs HUNDREDS OF MILLIONS of dollars, plus full hardware replacement in multiple datacenters at regular intervals + running a backup internet. For $200/month@400K players, it MIGHT be doable.

**** happens. Move on with your life and come back tomorrow. The world did not end.


Hundreds of millions? Not quite. You can get very close HA (the OS's would have to reboot) with a VMWare cluster running your hosts and redundant SAN's connected to the cluster. I spoke to a Dev @ Eve Vegas and the issue running on VMWare ESXi was that when a node migrates from one server to another it either hangs on re-building the node. It seems more of a software issue than anything else, but if they really wanted to I'm sure they could find a solution to the problem.

Plainly put, relying on physical hosts to run everything on bare metal leads to long downtimes. Utilize virtualization, snapshotted data storage or continuous replication, and you can effectively ensure 99.99% update and razor thin RTO/RPO's for a few hundred thousand for a medium sized business. Not hundreds of millions. I do it at my job every day and have never had a time where systems weren't live within 5 minutes or less during a catastrophic failure. We also use Site Recovery so with a single click of a button we can bring our systems online at a secondary site within minutes should we our primary datacenter gets cratered.


Yes but do you get laid ........... ever?


ROFL good one! Lol

"If a miner needs to go to the bathroom, for instance, I ask that they dock up first, or at the very least ask the Supreme Protector for permission to go."  -  James 315 - aka - the miner bumper

Gyala Frey
Center for Advanced Studies
Gallente Federation
#86 - 2012-11-04 03:58:28 UTC
The Doctor Galferrin wrote:
Gyala Frey wrote:
And I just resubbed today after being gone over a year. Bad omen?


yeah you broke it man



<--- Sad Panda
Kalkoken Sukarala
Kalkoken Sukarala Corporation
#87 - 2012-11-04 03:58:33 UTC
0 seconds is a long time...
Kelseru
Boojum-Corp
#88 - 2012-11-04 03:58:59 UTC
Malchristus wrote:
[quote=Daemon Ceed]

Yes but do you get laid ........... ever?



Are you kidding me? this man makes bank. He's got ladies crawling over eachother to get at him. Remember to be nice to the nerds. Chances are you will be working for one someday.
Tiger Bite
Data Sheets
#89 - 2012-11-04 03:59:05 UTC
DEV & CCP,

I would like to report that the link at the log in screen for the eve client is broken... it links to:

https://https//forums.eveonline.com/default.aspx?g=posts&t=169175

It should link to:

https://forums.eveonline.com/default.aspx?g=posts&t=169175


Regards,
Tiger Bite
Ando Gart
Almost Dangerous
Wolves Amongst Strangers
#90 - 2012-11-04 03:59:13 UTC
Kalkoken Sukarala wrote:
0 seconds is a long time...

Waiting for 0 seconds is like dividing by 0.
Takor Ma
Sebiestor Tribe
Minmatar Republic
#91 - 2012-11-04 03:59:17 UTC
WHYYYY IT NOT STARTINGGG!!!!!!!!!!!!!!!!!!!!!!!
Robyn Green
Genesis Corp
#92 - 2012-11-04 03:59:39 UTC
nortelslug wrote:

This.

100% uptime HA costs HUNDREDS OF MILLIONS of dollars, plus full hardware replacement in multiple datacenters at regular intervals + running a backup internet. For $200/month@400K players, it MIGHT be doable.


No player really knows, or ever really will know, what it costs for CCP to run all the equipment and so on that they do. Those sorts of things I imagine would be known only to the highest of the high at CCP. But hundreds of millions I imagine would be an over estimate.
Gyala Frey
Center for Advanced Studies
Gallente Federation
#93 - 2012-11-04 04:00:03 UTC
Ando Gart wrote:
Kalkoken Sukarala wrote:
0 seconds is a long time...

Waiting for 0 seconds is like dividing by 0.


UNDEFINED
Rocketfish
State War Academy
Caldari State
#94 - 2012-11-04 04:00:03 UTC
Daemon Ceed wrote:
[quote=nortelslug]
Hundreds of millions? Not quite. You can get very close HA (the OS's would have to reboot) with a VMWare cluster running your hosts and redundant SAN's connected to the cluster. I spoke to a Dev @ Eve Vegas and the issue running on VMWare ESXi was that when a node migrates from one server to another it either hangs on re-building the node. It seems more of a software issue than anything else, but if they really wanted to I'm sure they could find a solution to the problem.

Plainly put, relying on physical hosts to run everything on bare metal leads to long downtimes. Utilize virtualization, snapshotted data storage or continuous replication, and you can effectively ensure 99.99% update and razor thin RTO/RPO's for a few hundred thousand for a medium sized business. Not hundreds of millions. I do it at my job every day and have never had a time where systems weren't live within 5 minutes or less during a catastrophic failure. We also use Site Recovery so with a single click of a button we can bring our systems online at a secondary site within minutes should we our primary datacenter gets cratered.


+10 for Vmotion
The Doctor Galferrin
Viziam
Amarr Empire
#95 - 2012-11-04 04:00:23 UTC
Gyala Frey wrote:
The Doctor Galferrin wrote:
Gyala Frey wrote:
And I just resubbed today after being gone over a year. Bad omen?


yeah you broke it man



<--- Sad Panda


dont be sad have a cookie!!
Sten Beldrulf
Sebiestor Tribe
Minmatar Republic
#96 - 2012-11-04 04:00:31 UTC
This thread is slowly becoming Jita 4-4
Vaude Villian
Brutor Tribe
Minmatar Republic
#97 - 2012-11-04 04:00:42 UTC
Clambumper June
Republic University
Minmatar Republic
#98 - 2012-11-04 04:00:43 UTC
Trokiel wrote:
Well, at least this gave me enough time to go into the kitchen and make a huge batch of chili and then go outside for some fresh air!


Should have used that Bean-o before eatting the chili. Farts are good for digestion, but bad for relationships and aroma therapy...
oien cabrereas
Gold Trading Inc..
#99 - 2012-11-04 04:00:54 UTC
Sten Beldrulf wrote:
This post is slowly becoming Jita 4-4


LOL
Rocketfish
State War Academy
Caldari State
#100 - 2012-11-04 04:01:12 UTC
W00t! I logged in!