These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE General Discussion

 
  • Topic is locked indefinitely.
 

CCP, Singularity and You: How we can all make better use of the test servers

First post
Author
Previous Topic Next Topic
CCP Goliath
C C P
C C P Alliance
#81 - 2012-06-05 09:39:35 UTC
Culmen wrote:
The problem with CCP taking feedback is that bittervets are probably the majority users of the test server.
Thus it's impossible to distinguish a small segment of bittervets whining and something that is going to be hated by the majority of EVE, at least while it's at the test server phase.

For example:
I started a thread on the new UI the day it went live on the test server.
Most of the gripes and simple fixes that CCP is now starting to implement were voiced in that thread.
Mean while the dev blog thread was stuffed with people say "WOW! SO COOL!".

So its 10 people who are probably biased bittervets vs 100 people who are only seeing the surface.

Really the best way to get around this is to hide changes behind a check box, thats defaulted to on.
Though that has the hazard of people de-selecting it for purposes of familiarity rather than the feature itself being broken.


This just isn't possible in most cases (recent examples where it's not possible - font, unified inventory, FW revamp). The game evolves (had to restrain myself from writing eve-volves) and parts of it will change. Familiarity will be developed again.

Also I think you're making a big assumption about the test server users here, and not one that I agree with.

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
CCP Guard
C C P
C C P Alliance
#82 - 2012-06-05 10:58:00 UTC
One thing I like (apart from being ahead of Punkturis in the likes department) is early dev blogs. It depends on the feature how much value they hold and how difficult they are to squeeze out, sometimes it's really hard. But when we've had early dev blogs for things like ship balancing for example, we've been able to use the feedback to spot things and change course on some details before deployment.

Understandably there's always a certain amount of hesitation when it comes to blogging about concepts or really rough early plans because they are bound to change and may even get dropped or pushed back. But I think you guys are less likely to get all "heyforky" (that's a word right?) about that than the stuff we don't talk about enough, right? :) And an early, ongoing discussion is what we all want. So let's try to get more of those.

CCP Guard | EVE Community Developer | @CCP_Guard
Luis Graca
#83 - 2012-06-05 11:02:17 UTC
CCP Guard wrote:
One thing I like (apart from being ahead of Punkturis in the likes department) .



Thats just a matter of time Cool



CCP Guard wrote:
Understandably there's always a certain amount of hesitation when it comes to blogging about concepts or really rough early plans because they are bound to change and may even get dropped or pushed back.



Why not make small videos and add them to the blogs?
Mr Kidd
Center for Advanced Studies
Gallente Federation
#84 - 2012-06-05 11:21:27 UTC  |  Edited by: Mr Kidd
Excellent post by the OP. However, if you're not going to ask the questions foremost on the minds of those interested in such topics because "they're not appropriate" you need to question why you do it at all. Why was it inappropriate? You already admitted event wasn't even remotely related to such a topic. Thanks for the effort. But no thanks to typical journalistic, "I only want to ask powder puff questions because I don't want to lose my spot next time". That's self-aggrandizing at the expense of your readers and the issues that drive the topic for which you report. We get enough of that in the real news media.

I don't test on Sisi. Never have. Never have I felt compelled to do so for the sake of the game. That's entirely due to having played this game for years while watching exactly the bugs and atrocious features that were reported weeks in advance of a release make it into TQ exactly in the state that feedback indicated would be a problem.

I think what the interviewer got was a hopeful wish of what Sisi could be. But, the reality is that in 9 years it's never been that so there's little point to believe it will ever be that. If people aren't using Sisi in a way that helps CCP then CCP only has itself to blame.

I'm just not going to waste my time.

Don't ban me, bro!
CCP Guard
C C P
C C P Alliance
#85 - 2012-06-05 11:23:43 UTC
Luis Graca wrote:

CCP Guard wrote:
Understandably there's always a certain amount of hesitation when it comes to blogging about concepts or really rough early plans because they are bound to change and may even get dropped or pushed back.



Why not make small videos and add them to the blogs?


That's sometimes very beneficial. Like in this recent blog about the effects bar for example. We may also do some "In development" videos to talk about things that are still fresh on the drawing board...could be fun?

P.s. Where's the love in that signature, dude! ;)

CCP Guard | EVE Community Developer | @CCP_Guard
Luis Graca
#86 - 2012-06-05 11:26:40 UTC
CCP Guard wrote:
Luis Graca wrote:

CCP Guard wrote:
Understandably there's always a certain amount of hesitation when it comes to blogging about concepts or really rough early plans because they are bound to change and may even get dropped or pushed back.



Why not make small videos and add them to the blogs?


That's sometimes very beneficial. Like in this recent blog about the effects bar for example. We may also do some "In development" videos to talk about things that are still fresh on the drawing board...could be fun?

P.s. Where's the love in that signature, dude! ;)




Then don't speak do it
Morgan North
Dark-Rising
Wrecking Machine.
#87 - 2012-06-05 11:32:36 UTC
Mr Kidd wrote:
Excellent post by the OP. However, if you're not going to ask the questions foremost on the minds of those interested in such topics because "they're not appropriate" you need to question why you do it at all. Why was it inappropriate? You already admitted event wasn't even remotely related to such a topic. Thanks for the effort. But no thanks to typical journalistic, "I only want to ask powder puff questions because I don't want to lose my spot next time". That's self-aggrandizing at the expense of your readers and the issues that drive the topic for which you report. We get enough of that in the real news media.

I don't test on Sisi. Never have. Never have I felt compelled to do so for the sake of the game. That's entirely due to having played this game for years while watching exactly the bugs and atrocious features that were reported weeks in advance of a release make it into TQ exactly in the state that feedback indicated would be a problem.

I think what the interviewer got was a hopeful wish of what Sisi could be. But, the reality is that in 9 years it's never been that so there's little point to believe it will ever be that. If people aren't using Sisi in a way that helps CCP then CCP only has itself to blame.

I'm just not going to waste my time.


Actually, sometimes, they are... Even more bugged. Like a lot more bugged. Like in the sense that a single release makes your entire character sheet froze up when loaded. Amongst other things.
Anna Shoul
#88 - 2012-06-05 11:41:50 UTC
Early dev blogs about what's going on with Singularity would be nice, so that people who go there remember to log in and test the problems out of the new stuff -- but I would very much love to see a changelog with every Singularity patch, so that if I do go there, I can mess with everything in detail.

There should be less worry about public opinion about what might happen, and more emphasis on actually setting up an experimental feature -> test server -> bug reports -> final feature cycle, and getting things done properly by the time they hit production. Rough plans are rough plans, everyone should be able to grasp that much. Rough plans remaining rough by the time they actually go live irks far more people.
Peter Tjordenskiold
#89 - 2012-06-05 11:46:51 UTC
Tbh a test server demands a systematic recording of issues and evaluation of new features incl. an information in both directions. That's why we have bug reporting, but why is there no overview for the bugs on Sisi reported by players? And another question is, why survived so many bugs on SiSi?

The evaluation of new features is a mess. Instead of making a systematic survey we reporting something in a forum. That's not working that way. And even when CCP is trying to give us a surprise, this product is too complex to work in this manner. The surprise has to be the idea about a new feature, not the feature itself. Designing new features is always a risk because of side effects.

it's not easy to get information from a crowd that is willing to work on Sisi. A Sisi without a systematic evaluation of results is a wasted ressource.
CCP Goliath
C C P
C C P Alliance
#90 - 2012-06-05 11:49:58 UTC
Anna Shoul wrote:
Early dev blogs about what's going on with Singularity would be nice, so that people who go there remember to log in and test the problems out of the new stuff -- but I would very much love to see a changelog with every Singularity patch, so that if I do go there, I can mess with everything in detail.

There should be less worry about public opinion about what might happen, and more emphasis on actually setting up an experimental feature -> test server -> bug reports -> final feature cycle, and getting things done properly by the time they hit production. Rough plans are rough plans, everyone should be able to grasp that much. Rough plans remaining rough by the time they actually go live irks far more people.


Patch notes for Sisi is impractical, but I write a list of features that are on Sisi on the forums - do you think that making this list into a devblog would be more appropriate?

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
CCP Goliath
C C P
C C P Alliance
#91 - 2012-06-05 11:54:23 UTC
Peter Tjordenskiold wrote:
Tbh a test server demands a systematic recording of issues and evaluation of new features incl. an information in both directions. That's why we have bug reporting, but why is there no overview for the bugs on Sisi reported by players? And another question is, why survived so many bugs on SiSi?

The evaluation of new features is a mess. Instead of making a systematic survey we reporting something in a forum. That's not working that way. And even when CCP is trying to give us a surprise, this product is too complex to work in this manner. The surprise has to be the idea about a new feature, not the feature itself. Designing new features is always a risk because of side effects.

it's not easy to get information from a crowd that is willing to work on Sisi. A Sisi without a systematic evaluation of results is a wasted ressource.


Are you asking for a public bug database? That's not going to happen. The overhead of whittling out what can and can not be public facing would make it more trouble than it's worth. I've actually had a long running conversation with Trebor about how we can make public facing bug reporting easier and more transparent but the long and short of it is - we can't/won't.

Surveys only tell you so much and are really easily gamed. How do you ensure that the survey taker even bothered to use the feature? At least on the forums its really easy to identify when someone has just looked at a screenshot or read a thread and gone full blown ragepost without actually getting their hands dirty and finding out their own opinion.

It's easy to get information, it's hard to whittle out the good posts from the noise. Especially when people start creating their own threads rather than using a coordinated thread that a team has set up for themselves.

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
CCP Goliath
C C P
C C P Alliance
#92 - 2012-06-05 11:54:54 UTC
CCP Guard wrote:
One thing I like (apart from being ahead of Punkturis in the likes department) is early dev blogs. It depends on the feature how much value they hold and how difficult they are to squeeze out, sometimes it's really hard. But when we've had early dev blogs for things like ship balancing for example, we've been able to use the feedback to spot things and change course on some details before deployment.

Understandably there's always a certain amount of hesitation when it comes to blogging about concepts or really rough early plans because they are bound to change and may even get dropped or pushed back. But I think you guys are less likely to get all "heyforky" (that's a word right?) about that than the stuff we don't talk about enough, right? :) And an early, ongoing discussion is what we all want. So let's try to get more of those.


I believe the word you are looking for is "pitchforky", or "hayforky" (not a word)

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
Cutter Isaacson
DEDSEC SAN FRANCISCO
#93 - 2012-06-05 11:55:06 UTC
CCP Goliath wrote:
Anna Shoul wrote:
Early dev blogs about what's going on with Singularity would be nice, so that people who go there remember to log in and test the problems out of the new stuff -- but I would very much love to see a changelog with every Singularity patch, so that if I do go there, I can mess with everything in detail.

There should be less worry about public opinion about what might happen, and more emphasis on actually setting up an experimental feature -> test server -> bug reports -> final feature cycle, and getting things done properly by the time they hit production. Rough plans are rough plans, everyone should be able to grasp that much. Rough plans remaining rough by the time they actually go live irks far more people.


Patch notes for Sisi is impractical, but I write a list of features that are on Sisi on the forums - do you think that making this list into a devblog would be more appropriate?



That is precisely the sort of thing I was calling for in my earlier post.

"The truth is usually just an excuse for a lack of imagination." Elim Garak.
Salpun
Global Telstar Federation Offices
Masters of Flying Objects
#94 - 2012-06-05 11:55:16 UTC  |  Edited by: Salpun
CCP Goliath wrote:
Anna Shoul wrote:
Early dev blogs about what's going on with Singularity would be nice, so that people who go there remember to log in and test the problems out of the new stuff -- but I would very much love to see a changelog with every Singularity patch, so that if I do go there, I can mess with everything in detail.

There should be less worry about public opinion about what might happen, and more emphasis on actually setting up an experimental feature -> test server -> bug reports -> final feature cycle, and getting things done properly by the time they hit production. Rough plans are rough plans, everyone should be able to grasp that much. Rough plans remaining rough by the time they actually go live irks far more people.


Patch notes for Sisi is impractical, but I write a list of features that are on Sisi on the forums - do you think that making this list into a devblog would be more appropriate?

It would get more eyes on it that way.Twisted
A daily news letter type devblog with vidio comments would be perfict and get people exited.

New forum section would work alsoTwisted

If i dont know something about EVE. I check https://wiki.eveonline.com/en/wiki/ISK_The_Guide

See you around the universe.
Salpun
Global Telstar Federation Offices
Masters of Flying Objects
#95 - 2012-06-05 11:57:57 UTC
CCP Goliath wrote:
Peter Tjordenskiold wrote:
Tbh a test server demands a systematic recording of issues and evaluation of new features incl. an information in both directions. That's why we have bug reporting, but why is there no overview for the bugs on Sisi reported by players? And another question is, why survived so many bugs on SiSi?

The evaluation of new features is a mess. Instead of making a systematic survey we reporting something in a forum. That's not working that way. And even when CCP is trying to give us a surprise, this product is too complex to work in this manner. The surprise has to be the idea about a new feature, not the feature itself. Designing new features is always a risk because of side effects.

it's not easy to get information from a crowd that is willing to work on Sisi. A Sisi without a systematic evaluation of results is a wasted ressource.


Are you asking for a public bug database? That's not going to happen. The overhead of whittling out what can and can not be public facing would make it more trouble than it's worth. I've actually had a long running conversation with Trebor about how we can make public facing bug reporting easier and more transparent but the long and short of it is - we can't/won't.

Surveys only tell you so much and are really easily gamed. How do you ensure that the survey taker even bothered to use the feature? At least on the forums its really easy to identify when someone has just looked at a screenshot or read a thread and gone full blown ragepost without actually getting their hands dirty and finding out their own opinion.

It's easy to get information, it's hard to whittle out the good posts from the noise. Especially when people start creating their own threads rather than using a coordinated thread that a team has set up for themselves.

It would take some work on the forum code but some forums have a combine feature. Just combine the threads and add a list of thread names that link to OP on the first page.

If i dont know something about EVE. I check https://wiki.eveonline.com/en/wiki/ISK_The_Guide

See you around the universe.
Luis Graca
#96 - 2012-06-05 12:00:07 UTC
CCP Goliath wrote:
Anna Shoul wrote:
Early dev blogs about what's going on with Singularity would be nice, so that people who go there remember to log in and test the problems out of the new stuff -- but I would very much love to see a changelog with every Singularity patch, so that if I do go there, I can mess with everything in detail.

There should be less worry about public opinion about what might happen, and more emphasis on actually setting up an experimental feature -> test server -> bug reports -> final feature cycle, and getting things done properly by the time they hit production. Rough plans are rough plans, everyone should be able to grasp that much. Rough plans remaining rough by the time they actually go live irks far more people.


Patch notes for Sisi is impractical, but I write a list of features that are on Sisi on the forums - do you think that making this list into a devblog would be more appropriate?



i don't think a dev blog would work however you guys could do a sticky/closed threat were you guys could simple write what's being tested and the date of singularity serv, after you guys are done with the tests and start a new one simply update that post.


This way people would know what testes are being conducted and the date of their skills in a single place without have to search 1 by 1

Even better create a EVE sisi folder like "EVE Information Center" "DUST 514 Information Center" about sisi test this way you will gain more input information instead of chat with some players
Velicitia
XS Tech
#97 - 2012-06-05 12:04:06 UTC
CCP Guard wrote:
One thing I like (apart from being ahead of Punkturis in the likes department) is early dev blogs. It depends on the feature how much value they hold and how difficult they are to squeeze out, sometimes it's really hard. But when we've had early dev blogs for things like ship balancing for example, we've been able to use the feedback to spot things and change course on some details before deployment.

Understandably there's always a certain amount of hesitation when it comes to blogging about concepts or really rough early plans because they are bound to change and may even get dropped or pushed back. But I think you guys are less likely to get all "heyforky" (that's a word right?) about that than the stuff we don't talk about enough, right? :) And an early, ongoing discussion is what we all want. So let's try to get more of those.



I think you're talking about a "pitchfork" there Guard (possibly used in the company of torches, works best if you have an angry mob with you)... and you're right, we won't use pitchforks.

1400mm artillery is more our style Big smile

One of the bitter points of a good bittervet is the realisation that all those SP don't really do much, and that the newbie is having much more fun with what little he has. - Tippia
Makari Aeron
Imperial Shipment
Amarr Empire
#98 - 2012-06-05 12:33:36 UTC  |  Edited by: Makari Aeron
I believe dev blogs about prototype features would be a good thing ONLY if announced with a clear disclaimer. Then have a thread attached that is for constructive criticism only in order to find out which features the players like the most and if possible, prioritize said features.

EDIT: a great way to get new ideas from players would be to do like I've heard tales of from "ye olden EVE days", where a representative would go from region to region in an invincible ship convoing players, corps, and alliances to discuss with them how they felt about the game and what features they would like to see in the game. I would love to see that put back into action, as would quite a few of the "old time-y" EVE players I know.

CCP RedDawn: Ugly people are just playing life on HARD mode. Personally, I'm playing on an INFERNO difficulty.

CCP Goliath: I often believe that the best way to get something done is to shout at the person trying to help you. http://goo.gl/PKGDP
jonnykefka
Adhocracy Incorporated
Adhocracy
#99 - 2012-06-05 13:07:36 UTC
One thing that came up in the unified inventory threads was people posting videos of attempting to do various things with the new UI and where it broke down. I believe some devs called this incredibly useful because it showed genuine use cases and incredibly clear reproduction steps.

Unfortunately, making such a video right now is a matter of getting 3rd-party software, recording massive video files, editing them and uploading them. Bug reporting also has some issues in that is only post-hoc. You have to do the buggy thing, then report the bug and hope that the BH/devs can dig far enough back in the logs to figure out what's wrong.

While an in-game video recording tool would be difficult and probably impractical, how about a way of "flagging", at least on Sisi, "I'm about to do something that I believe will lead to a buggy behavior" in the bug report interface, which leaves an entry in the log or creates some kind of more detailed log that we can then send to you. If nothing else it will give you guys easier access to our repro steps and if you can get it to record things like mouse coordinates and clicks possibly even reconstruct a video of what it is we were doing.

In short, the bug reporting, while already light-years ahead of the old manually-start-logserver-and-find-the-report-bugs-website setup, could be cleaned up to make it easier for us to show you what we're doing and easier for you to figure out what's going wrong.
Iamien
Deep Core Mining Inc.
Caldari State
#100 - 2012-06-05 13:20:15 UTC
It's quite simple. SiSi is for testing technical implementations. CCP does not use the SiSi as a tool to test game design. Game design in eve is done by the producers in a vacuum.

For example, a single designer can choose to buff t2 armor plates an arbitrary amount, with no methodology behind the values.

The developers and bug hunters are solely tasked with facilitating an efficient technical implementation of the game design described to them. This only works if said game design has been rigorously thought out and vetted. If the idea is not fully formulated before being passed to the devs, the idea quickly gets to a point where a significant(yet essential) change can be overlooked.

Simply put, the community has very little input into design in the time of of idea to initial implementation on tranquility. It's all up to the lead producers who are hopefully improving each expansion.