These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

Test Server Feedback

 
  • Topic is locked indefinitely.
Previous page123Next page
 

Singularity temporarily down

First post First post
Author
Previous Topic Next Topic
CCP Habakuk
C C P
C C P Alliance
#21 - 2014-06-24 10:57:47 UTC  |  Edited by: CCP Habakuk
Starfury Shang wrote:
James Amril-Kesh wrote:
CCP Habakuk wrote:
Well, serious news: the database of Singularity is irreversibly damaged and needs to be re-mirrored. Oops

How does something like that happen, and is TQ ever at risk for that?


I would assume SiSi isn't backed up as it is not a 'production' server, so if TQ had problems they would restore a backup and roll forward any logs.


This, and also the fact that the setup for Singularity is far from optimal due to very limited disk space - which puts it at much higher risk.

CCP Habakuk | EVE Quality Assurance | Team Five 0 | (Team Gridlock)

Bug reporting | Mass Testing
Sales Alt negrodamus
Sanctuary of Shadows
#22 - 2014-06-24 11:35:30 UTC
CCP Habakuk wrote:
This, and also the fact that the setup for Singularity is far from optimal due to very limited disk space - which puts it at much higher risk.


is that what happened? :)

trust me you guys wouldn't be the first to have a database suck up a disk and crash.
elitatwo
Zansha Expansion
Brave Collective
#23 - 2014-06-24 11:41:26 UTC  |  Edited by: elitatwo
Starfury Shang wrote:
-snip-
How does something like that happen, and is TQ ever at risk for that?

I would assume SiSi isn't backed up as it is not a 'production' server, so if TQ had problems they would restore a backup and roll forward any logs.


Nobody who works in the industry would ever put a high availability database at any risk. There are or there should be many redunant backup systems in place, everything runs or should be running on raid system and all harddrives that might be involved get changed no matter what after 2 years of constant operation.

Those drives will or should be wiped with a shredding program so any data on those drives are no longer recoverable by any forensic recovery program.

After that the drives can be decomissioned or used for the admins to play with or send away to be destroyed or recycled.

There is or should be a desaster recovery protocol in place which involves the backups or other means of recovery.

The reason why all backups have to be always redundant is, so that if one of the backup systems fails, no production service needs to be halted or taken offline.

There other protocols in place for high availabitly services (heartbeat anyone??) which I won't talk about but I can assure you that our hamsters on Tranquility are well guarded.

And why do I respond in the first place?

Well my last job was exactly that, high availability server maintenance and administration, so I do know what I am talking about.

Eve Minions is recruiting.

This is the law of ship progression!

Aura sound-clips: Aura forever
Olivia Berett
Advanced Weapon Supplement League
Fraternity.
#24 - 2014-06-24 14:57:58 UTC
Lots of people in EVE like the EVE style because they are IT professionals. So most of us have a good idea on how things are. Those who dont, also dont care.

The time and ammount of things you do are not argument for assuming what you say is true.

People who works at CCP may be taget of critics, but they are also the ones DOING, so im pretty sure they know what and how they are doing what they do.

We should focus on helping with tests and not elucidating people who either know or dont care how EVE works.

Helpful informations would be time tables and details about what specific information over the EVE itself.

Discussing concepts and how CCP could do it differently really doesnt help anything.

Why do I say that ?

Cuz there is always people to dish out a lot of garbage sitting on years of doing this or that, and most of it is just quibling over "Whats today ? This day or the present day ?".

Seriously, this aint funny anymore.
elitatwo
Zansha Expansion
Brave Collective
#25 - 2014-06-24 16:50:25 UTC
Olivia Berett wrote:
-snip-

Cuz there is always people to dish out a lot of garbage sitting on years of doing this or that, and most of it is just quibling over "Whats today ? This day or the present day ?".

Seriously, this aint funny anymore.


Wasn't meant to be criticism but an assurance our Tranquility server is save and well guarded.

SiSi is more of a playtoy and if or when SiSi crashes, so be it, we get an early mirror, everything is fine.

Eve Minions is recruiting.

This is the law of ship progression!

Aura sound-clips: Aura forever
xRyokenx
GoonWaffe
Goonswarm Federation
#26 - 2014-06-24 21:52:13 UTC
Yay for re-mirror! Any updates on if the restart is back on track for tomorrow/today?
Regan Rotineque
The Scope
Gallente Federation
#27 - 2014-06-24 21:56:29 UTC
CCP Habakuk wrote:
ISD Ezwal wrote:
MailDeadDrop wrote:
Would it be possible to alter the mirror script so that *highsec* POSes (and the jobs in them) aren't nuked when the mirror is taken? There's no intel to protect for highsec POSes (unlike lowsec, nullsec, and WH POSes). This would allow you to test the Crius upgrade script's handling of in-progress jobs. It also would make life incrementally easier for people to test POS things on Sisi.

MDD
Question forwarded to CCP, fao CCP Habakuk.


Normally I would deny this request, but I will bring it up with our DB specialists, as we are messing around with the mirror preparation scripts anyway. It is still quite unlikely as deleting only some POS structures is much slower and more failure prone than deleting whole DB tables.



Although sounding like a good idea it is not.....war targets and others will be able to unmolested to hs towers and scan them to see what you have in them. There is a reason that all this data needs to be wiped. Nobody should gain a strategic advantage using SISi. I realize its a pita to set up new POS online stuff and restart jobs but its also a good test of some pretty old code that should be tested and retested with each release.
Bio Akachi
PHOENIX-SPZOO
#28 - 2014-06-24 22:08:52 UTC
Maybe It`s time to invest more money for better hardware and security system 1k users (average) on test server = problems?! Shut it down permanentlyTwisted - only jokingSmileBear
Hommycyde
Science and Trade Institute
Caldari State
#29 - 2014-06-24 22:52:05 UTC
Someone broke it.....Ugh
CCP Habakuk
C C P
C C P Alliance
#30 - 2014-06-25 01:40:21 UTC  |  Edited by: CCP Habakuk
Regan Rotineque wrote:
CCP Habakuk wrote:
ISD Ezwal wrote:
MailDeadDrop wrote:
Would it be possible to alter the mirror script so that *highsec* POSes (and the jobs in them) aren't nuked when the mirror is taken? There's no intel to protect for highsec POSes (unlike lowsec, nullsec, and WH POSes). This would allow you to test the Crius upgrade script's handling of in-progress jobs. It also would make life incrementally easier for people to test POS things on Sisi.

MDD
Question forwarded to CCP, fao CCP Habakuk.


Normally I would deny this request, but I will bring it up with our DB specialists, as we are messing around with the mirror preparation scripts anyway. It is still quite unlikely as deleting only some POS structures is much slower and more failure prone than deleting whole DB tables.



Although sounding like a good idea it is not.....war targets and others will be able to unmolested to hs towers and scan them to see what you have in them. There is a reason that all this data needs to be wiped. Nobody should gain a strategic advantage using SISi. I realize its a pita to set up new POS online stuff and restart jobs but its also a good test of some pretty old code that should be tested and retested with each release.



Very good argument. Yeah, even high sec POS can hold secrets.

PS: Short update on the progress: Everything is going as planned so far. Tomorrow we will have a few further tests in VIP mode for making sure that all industry jobs were upgraded correctly and then we will need to prepare the DB for public use (delete all POS and similar). My current estimate would be late Wednesday or (more likely) Thursday morning.

CCP Habakuk | EVE Quality Assurance | Team Five 0 | (Team Gridlock)

Bug reporting | Mass Testing
Orny
Science and Trade Institute
Caldari State
#31 - 2014-06-25 02:42:44 UTC
Tks for the update CCP Habakuk
Potions Master
GearBunny
#32 - 2014-06-25 03:16:17 UTC
CCP Habakuk wrote:
Regan Rotineque wrote:
CCP Habakuk wrote:
ISD Ezwal wrote:
MailDeadDrop wrote:
Would it be possible to alter the mirror script so that *highsec* POSes (and the jobs in them) aren't nuked when the mirror is taken? There's no intel to protect for highsec POSes (unlike lowsec, nullsec, and WH POSes). This would allow you to test the Crius upgrade script's handling of in-progress jobs. It also would make life incrementally easier for people to test POS things on Sisi.

MDD
Question forwarded to CCP, fao CCP Habakuk.


Normally I would deny this request, but I will bring it up with our DB specialists, as we are messing around with the mirror preparation scripts anyway. It is still quite unlikely as deleting only some POS structures is much slower and more failure prone than deleting whole DB tables.



Although sounding like a good idea it is not.....war targets and others will be able to unmolested to hs towers and scan them to see what you have in them. There is a reason that all this data needs to be wiped. Nobody should gain a strategic advantage using SISi. I realize its a pita to set up new POS online stuff and restart jobs but its also a good test of some pretty old code that should be tested and retested with each release.



Very good argument. Yeah, even high sec POS can hold secrets.

PS: Short update on the progress: Everything is going as planned so far. Tomorrow we will have a few further tests in VIP mode for making sure that all industry jobs were upgraded correctly and then we will need to prepare the DB for public use (delete all POS and similar). My current estimate would be late Wednesday or (more likely) Thursday morning.


I usually prep my towers ahead of time when I know there's going to be a mirror so nothing of importance gets lost, but I have had a thought on preserving said data for those who need it/surprise mirrors like this one. Would it be fairly easy to backup the pos content table before you delete it and make a quick script to automate delivery of said items to a station hangar? So you could just dock up in a station, go to the 'deliver' channel and type 'deliver' and anything item-wise you had at your towers gets dropped in your personal hangar (if you have the roles for it) in containers labeled after where they were or something. (Don't know how much lag or whatever this would cause, but just a database list copy/add from one spot to another right?)

Just a thought that would help those who keep important things at their towers. (IE: Wormholeers, NPC Nullsec dwellers, Lowsec pioneers, etc...)
James Amril-Kesh
Viziam
Amarr Empire
#33 - 2014-06-25 04:06:31 UTC
CCP Habakuk wrote:
Starfury Shang wrote:
James Amril-Kesh wrote:
CCP Habakuk wrote:
Well, serious news: the database of Singularity is irreversibly damaged and needs to be re-mirrored. Oops

How does something like that happen, and is TQ ever at risk for that?


I would assume SiSi isn't backed up as it is not a 'production' server, so if TQ had problems they would restore a backup and roll forward any logs.


This, and also the fact that the setup for Singularity is far from optimal due to very limited disk space - which puts it at much higher risk.

How often is TQ backed up? Every downtime, or less frequently?

Enjoying the rain today? ;)
elitatwo
Zansha Expansion
Brave Collective
#34 - 2014-06-25 09:54:31 UTC
James Amril-Kesh wrote:
-snip-
How often is TQ backed up? Every downtime, or less frequently?


e2 waves her arms like a Jedi-

This is not for nosy goofswarm folks to know. You don't want to sell killer-sticks. You want to go home and think about your life.

Eve Minions is recruiting.

This is the law of ship progression!

Aura sound-clips: Aura forever
Sulpa
Blacksmith Industry 778
#35 - 2014-06-25 11:08:48 UTC
i Would like this account activated please
Wedge Rancer
The Rogue Space Force
#36 - 2014-06-25 12:09:09 UTC
Sulpa wrote:
i Would like this account activated please

This is defiently not the thread to post that.

Also I saw VIP Mode on SISI earlier + a patch. Hopefully this means CCP are making progress on getting SISI fixed :D
Supreme clientele
The Dikembe Mutombo Shotblocking Team
#37 - 2014-06-25 14:27:51 UTC
he said thursday......Roll
MailDeadDrop
Archon Industries
#38 - 2014-06-25 14:28:26 UTC
Regan Rotineque wrote:
CCP Habakuk wrote:
ISD Ezwal wrote:
MailDeadDrop wrote:
Would it be possible to alter the mirror script so that *highsec* POSes (and the jobs in them) aren't nuked when the mirror is taken? There's no intel to protect for highsec POSes (unlike lowsec, nullsec, and WH POSes). This would allow you to test the Crius upgrade script's handling of in-progress jobs. It also would make life incrementally easier for people to test POS things on Sisi.

MDD
Question forwarded to CCP, fao CCP Habakuk.


Normally I would deny this request, but I will bring it up with our DB specialists, as we are messing around with the mirror preparation scripts anyway. It is still quite unlikely as deleting only some POS structures is much slower and more failure prone than deleting whole DB tables.



Although sounding like a good idea it is not.....war targets and others will be able to unmolested to hs towers and scan them to see what you have in them. There is a reason that all this data needs to be wiped. Nobody should gain a strategic advantage using SISi. I realize its a pita to set up new POS online stuff and restart jobs but its also a good test of some pretty old code that should be tested and retested with each release.


Forgive me for being blunt, but you're nuts. Nothing prevents a "war target or other" from going to your tower in highsec on TQ and seeing what you have online. War targets would use a cloaked ship or an out-of-corp alt. "Others" will simply go look.

And as for your dismissive attitude indicating it's but a small "pita", you've obviously not dealt with the problem of *LOSING* BPOs on Sisi. You see, when the POS is nuked, all of the blueprints in jobs on it are obliterated, too. Do you have any idea how much a hassle it is to re-acquire those blueprints? Not to mention that you lose all the blueprints' research, too.

Hababuk, can you at least return the blueprints to the station?

MDD
GazBoi08
x13
Pandemic Horde
#39 - 2014-06-25 14:30:18 UTC
MailDeadDrop wrote:
Regan Rotineque wrote:
CCP Habakuk wrote:
ISD Ezwal wrote:
MailDeadDrop wrote:
Would it be possible to alter the mirror script so that *highsec* POSes (and the jobs in them) aren't nuked when the mirror is taken? There's no intel to protect for highsec POSes (unlike lowsec, nullsec, and WH POSes). This would allow you to test the Crius upgrade script's handling of in-progress jobs. It also would make life incrementally easier for people to test POS things on Sisi.

MDD
Question forwarded to CCP, fao CCP Habakuk.


Normally I would deny this request, but I will bring it up with our DB specialists, as we are messing around with the mirror preparation scripts anyway. It is still quite unlikely as deleting only some POS structures is much slower and more failure prone than deleting whole DB tables.



Although sounding like a good idea it is not.....war targets and others will be able to unmolested to hs towers and scan them to see what you have in them. There is a reason that all this data needs to be wiped. Nobody should gain a strategic advantage using SISi. I realize its a pita to set up new POS online stuff and restart jobs but its also a good test of some pretty old code that should be tested and retested with each release.


Forgive me for being blunt, but you're nuts. Nothing prevents a "war target or other" from going to your tower in highsec on TQ and seeing what you have online. War targets would use a cloaked ship or an out-of-corp alt. "Others" will simply go look.

And as for your dismissive attitude indicating it's but a small "pita", you've obviously not dealt with the problem of *LOSING* BPOs on Sisi. You see, when the POS is nuked, all of the blueprints in jobs on it are obliterated, too. Do you have any idea how much a hassle it is to re-acquire those blueprints? Not to mention that you lose all the blueprints' research, too.

Hababuk, can you at least return the blueprints to the station?

MDD



uhh BPO's are seeded..
Regan Rotineque
The Scope
Gallente Federation
#40 - 2014-06-25 15:05:03 UTC



Seeded BPOs plus its a test environment and almost everything is 100 isk......

reminds me of the guy that once argued that all his minerals were missing....im like go buy 100 thanatos and melt them

that being said i by no means was being dismissive.....i have run towers all over eve....and when a mirror happens i too hate that i have to go set stuff up again. However its a few hours to get things up and since everything is cheap and seeded it really is not that inconvenient. In fact there are things I have tweaked when i set tham up. I feel for our WH brothers because they have nonother options in their WH when SiSi is wiped.....but us highsec and null folks have zip to complain about......though the dude that builds supers and titans ❤️ i haz sum feelings fer u
Previous page123Next page