These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Alliance Tournament Discussion

 
  • Topic is locked indefinitely.
 

EVE TV Stream issues 15th July

First post
Author
CCP Loxy
C C P
C C P Alliance
#1 - 2012-07-15 22:21:21 UTC  |  Edited by: CCP Navigator
Hey everyone,

So it definitively wasn't in the plan to have to post another one of these threads, unfortunately more issues hit us today and I'd like to bring you up to speed.

For the record, during the tournament itself I'm directing the show and don't have hands on the actual streaming. However as producer of the event I take full responsibility for the issues we've had this weekend, more testing should have been done and more redundancy should have been in place. After 6 years of streaming the tournament I have no excuses for letting this happen.

I'd also like to formally state again that we have no reason to believe any issues have been the fault of Own3d.tv who have continued to offer their support and assistance in finding the issue, and I'd like to thank CCP Bro and CCP Mindstar along with others for jumping on the problem and finding solutions.

After making changes in reaction to yesterday's problems we found we were still having issues as soon as begun the stream. We immediately made some more drastic changes which we weren't keen to make the evening before but we thought had to fix the issue. The stream continued to have issues which still at this point we can't explain.

At this point we switched to the backup streaming PC even with no clear indication that there was anything wrong with the main server. The backup was running identically to the main but held the stream. In addition we brought in our old encoding software and established a second stream to Own3d.tv as an alternative to the backup also failing.

In the end it seems the backup streaming pc worked fine and we had no issues in that regard for the rest of the day. Why exactly the first pc was having these issues is still unexplained, this is the same pc that has been streaming for weeks and we can find no faults.

Which brings us to the next problem, what I personally refer to as the "oh wow, that's literally the worst thing that could ever happen to us" scenario.

After several hours of a perfect stream the EVE TV studio lost all internet connectivity.

Obviously this is bad.

While investigations into the cause are still ongoing the result was obvious, we couldn't stream you the match. After review the tournament referee declared the result of the match valid and that takes precedence over actually streaming the fight. So the result will stand with unfortunately no visual record.

We now have 5 days to the next weekend, it goes without saying we will strive to test as much as possible and deliver the feed you expect from us. Putting aside the streaming issues the show has going incredibly smoothly so we're obviously disappointed that you haven't been able to enjoy it properly. Please tune in next week for a hopefully smooth and awesome finale to the Tenth Alliance Tournament.

- CCP Loxy

Video Producer & Director of EVE TV

Rynnik
Evasion Gaming
The Ancients.
#2 - 2012-07-15 22:24:31 UTC
Classy post.

Hope you get it all worked out for next weekend, and grats for running an awesome tourney. It is so good that there is tons of rage when we can't enjoy it with crystal clear streaming!

Thanks for running down the issues for us.
Poetic Stanziel
The Scope
Gallente Federation
#3 - 2012-07-15 22:34:53 UTC
If you need to brute force test, announce a date and time that you'd like people to start streaming some sort of test and I'm sure a few thousand of us will be more than happy to help out.
TeaDaze
Sablenacht
#4 - 2012-07-15 22:41:22 UTC  |  Edited by: TeaDaze
Can we assume that the matches that didn't get broadcast on the stream were still recorded locally and will appear on youtube in due time?

As to why one identical server works when the other one fails - that's just computers for you ;)

Edit.
Not just in due time, but up already - great job :)

Served on CSM4 and CSM5, AT7 "Expert"

Skogen Gump
Sebiestor Tribe
Minmatar Republic
#5 - 2012-07-15 22:57:02 UTC
Hey Loxy;

I thought it was particularly of note that these problems only started occuring, a) When you had up to 24 players on the field and b) at least firstly, only during actual matches.

Could it be possible that there's some timing issue between the box rendering the clients and the expected latency on the codec ?

Alternatively, is it possible you've got a busted/overheating router or switch ?

Lastly, not intending to teach you the art of egg sucking but as these timeouts were quite noticeable at our end, can you correlate the time to any cisco logging, windows event logging, DNS cache errors or anything like that ? Similarly any alert events from the NIC in the primary streaming PC ?

I now that sort of investigation is a real time consuming pain, but it might just go some way to pointing you in the right direction.

Best of luck and a heart felt thank you for the heroic efforts you guys made to get this working for us, on a sunday evening of all days.

You all officially rock :)
Z1gy
Vindicator Corporation
#6 - 2012-07-15 23:46:23 UTC
looking forward for next weekend matches and hope you guys resolve the issues by then
Hathrul
School of Applied Knowledge
Caldari State
#7 - 2012-07-16 00:26:42 UTC
CCP Loxy wrote:
Hey everyone,

So it definitively wasn't in the plan to have to post another one of these threads, unfortunately more issues hit us today and I'd like to bring you up to speed.

For the record, during the tournament itself I'm directing the show and don't have hands on the actual streaming. However as producer of the event I take full responsibility for the issues we've had this weekend, more testing should have been done and more redundancy should have been in place. After 6 years of streaming the tournament I have no excuses for letting this happen.

I'd also like to formally state again that we have no reason to believe any issues have been the fault of Own3d.tv who have continued to offer their support and assistance in finding the issue, and I'd like to thank CCP Bro and CCP Mindstar along with others for jumping on the problem and finding solutions.

After making changes in reaction to yesterday's problems we found we were still having issues as soon as begun the stream. We immediately made some more drastic changes which we weren't keen to make the evening before but we thought had to fix the issue. The stream continued to have issues which still at this point we can't explain.

At this point we switched to the backup streaming PC even with no clear indication that there was anything wrong with the main server. The backup was running identically to the main but held the stream. In addition we brought in our old encoding software and established a second stream to Own3d.tv as an alternative to the backup also failing.

In the end it seems the backup streaming pc worked fine and we had no issues in that regard for the rest of the day. Why exactly the first pc was having these issues is still unexplained, this is the same pc that has been streaming for weeks and we can find no faults.

Which brings us to the next problem, what I personally refer to as the "oh wow, that's literally the worst thing that could ever happen to us" scenario.

After several hours of a perfect stream the EVE TV studio lost all internet connectivity.

Obviously this is bad.

While investigations into the cause are still ongoing the result was obvious, we couldn't stream you the match. After review the tournament referee declared the result of the match valid and that takes precedence over actually streaming the fight. So the result will stand with unfortunately no visual record.

We now have 5 days to the next weekend, it goes without saying we will strive to test as much as possible and deliver the feed you expect from us. Putting aside the streaming issues the show has going incredibly smoothly so we're obviously disappointed that you haven't been able to enjoy it properly. Please tune in next week for a hopefully smooth and awesome finale to the Tenth Alliance Tournament.

- CCP Loxy


thanks for the post :) much appreciated

how about we just stress test it somewhere during the week?^^
Psichotic
Caldari Provisions
Caldari State
#8 - 2012-07-16 00:43:37 UTC
Don't worry, between the bad camerawork and 1/3 of the screen taken up by gayness we can't really see the fights anyway. Although, stable audio would be helpful.
Swidgen
Republic University
Minmatar Republic
#9 - 2012-07-16 03:17:54 UTC  |  Edited by: Swidgen
More testing should have been done.

The story of CCP's life, eh?

This latest snafu didn't affect me in the least because I vowed not to watch any matches after day 1 as long as MB3 was involved in the broadcasts. I've been staying true to my word. Feel bad for those who wanted to watch, though. It's unacceptable, really.
Aineko Macx
#10 - 2012-07-16 05:39:18 UTC
Looking forward to seeing non-interlaced video.
Daneel Trevize
Give my 11percent back
#11 - 2012-07-16 10:51:40 UTC  |  Edited by: Daneel Trevize
So, officially no footage from Match 11 with DarkSide vs BrickSquad?
Superintendent Chalmers
GoonWaffe
Goonswarm Federation
#12 - 2012-07-16 13:41:36 UTC
What was with the audio issues right after the stream broke and came back the first time? There were at times:

  • Ground loop / 50Hz hum in left channel
  • High gain on player ads leading to distortion
  • levels all over the place, sometimes distorting, sometimes way too low
  • Monitor cross-feeds / studio not muted while on commentators
  • Some weird echo (feedback from stream?)


Is this playout related or from other issues?

-- siggy

Thecla Elarik
Blue Republic
RvB - BLUE Republic
#13 - 2012-07-17 10:27:36 UTC
Superintendent Chalmers wrote:
What was with the audio issues right after the stream broke and came back the first time? There were at times:

  • Ground loop / 50Hz hum in left channel
  • High gain on player ads leading to distortion
  • levels all over the place, sometimes distorting, sometimes way too low
  • Monitor cross-feeds / studio not muted while on commentators
  • Some weird echo (feedback from stream?)


Is this playout related or from other issues?


Will add my observations:

* Ground loop hum was continuously present in the live cast, not in the youtube uploads.
* Gain too high often happened after a stream crash
* Audio levels for ads, actual match and studio time did vary a lot
* Did not notice this..
* Echo only present after stream crash


Also never give Greyscale a pen or something else to fidget around with on air :)
Seismic Stan
Freebooted Junkworks
#14 - 2012-07-18 15:58:21 UTC
Watching the commentator roam from the UK, audio and visual seems smooth, although it's impossible to tell if it's in synch without a talking head.

I've got the feed running on both my laptop and my iPhone. Oddly, despite the fact they are both running wirelessly on the same connection, there is a 30 second difference between the two signals.
Caviar Liberta
The Scope
Gallente Federation
#15 - 2012-07-18 23:21:35 UTC  |  Edited by: Caviar Liberta
Seismic Stan wrote:
Watching the commentator roam from the UK, audio and visual seems smooth, although it's impossible to tell if it's in synch without a talking head.

I've got the feed running on both my laptop and my iPhone. Oddly, despite the fact they are both running wirelessly on the same connection, there is a 30 second difference between the two signals.


Possibly because of compressed digital streaming? In the house I could have 3 tv's on the the same channel but you will a small delay between them since the digital boxes aren't perfectly sync'd.