These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

Upcoming Feature and Change Feedback Center

 
  • Topic is locked indefinitely.
 

Project Discovery - Changes to the scoring system

First post First post
Author
Previous Topic Next Topic
Memphis Baas
#81 - 2016-03-31 00:22:17 UTC
SurrenderMonkey wrote:
This is a good change, however, I feel like the training samples need to be harder. A lot harder.

I can actually predict, with pretty close to perfect accuracy (exceptions are, e.g., microtubules, where the unknown samples are usually pretty obvious, too), whether or not a sample will be a training sample just by its appearance.


I disagree.

First, it's a game, let people get enjoyment from seeing their accuracy increase and their rewards increase. Because the rewards are crap anyway, so going from 50k to 100k is still pretty crappy.

Second, how would you feel with your confidence of "perfect accuracy" but only 50% in the score at the top of the screen? At 25 slides per level, anyone reaching level 40 has done 1000 slides, and should be a freaking expert.

Third and last, if they make them a lot harder I will screenshot all of them and then reference the image and "guess" correctly next time. Thus, making the slides harder just reinforces the wrong kind of gameplay.

I don't want to learn more subcellular morphology than it's already been presented or than I know from general knowledge. Project Discovery isn't easy, and we're working for free, so make it harder and a lot of people will just stop working. Because EVE is a game, not work.
Beta Maoye
#82 - 2016-03-31 02:52:38 UTC
Memphis Baas wrote:

Third and last, if they make them a lot harder I will screenshot all of them and then reference the image and "guess" correctly next time. Thus, making the slides harder just reinforces the wrong kind of gameplay.


Sooner or later, according to the tradition of Eve, someone will post a full list of answers to known samples on the web. Lol
SurrenderMonkey
State Protectorate
Caldari State
#83 - 2016-03-31 14:50:58 UTC
Memphis Baas wrote:
SurrenderMonkey wrote:
This is a good change, however, I feel like the training samples need to be harder. A lot harder.

I can actually predict, with pretty close to perfect accuracy (exceptions are, e.g., microtubules, where the unknown samples are usually pretty obvious, too), whether or not a sample will be a training sample just by its appearance.


I disagree.

First, it's a game, let people get enjoyment from seeing their accuracy increase and their rewards increase. Because the rewards are crap anyway, so going from 50k to 100k is still pretty crappy.

Second, how would you feel with your confidence of "perfect accuracy" but only 50% in the score at the top of the screen? At 25 slides per level, anyone reaching level 40 has done 1000 slides, and should be a freaking expert.

Third and last, if they make them a lot harder I will screenshot all of them and then reference the image and "guess" correctly next time. Thus, making the slides harder just reinforces the wrong kind of gameplay.

I don't want to learn more subcellular morphology than it's already been presented or than I know from general knowledge. Project Discovery isn't easy, and we're working for free, so make it harder and a lot of people will just stop working. Because EVE is a game, not work.





I don't have perfect accuracy - what I said is that I can predict whether or not a given image is a training sample with close to perfect accuracy. They're invariably highly precise examples of their given type. Visually, they stick out like a sore thumb.

Consequently, they don't do a particularly good job of training the player to identify the comparatively "noisy" unknown samples.

I'm not saying they should be harder for any balance related reason - I'm saying they need to be more closely representative of the samples we're actually trying to identify to serve any effective purpose.

"Help, I'm bored with missions!"

http://swiftandbitter.com/eve/wtd/
Memphis Baas
#84 - 2016-03-31 22:31:02 UTC  |  Edited by: Memphis Baas
I agree in principle with that, but they have to make sure that the example slides contain only the single cell feature that they're exemplifying. Blur and distort all you want, but if the example for plasma membrane shows a cell with plasma membrane and cytoplasm, then we'll mis-grade all the normal samples based on how the green appears in the example slide, cytoplasm included.

Because we don't really know microbiology. Golgi strands look a little like mitochondria strands, and the single sentence of text explaining each of them doesn't really stress just how critical it is to note if the squiggles are spread out through the cell or only in the vicinity of the nucleus (actually touching it).

To me it's more important that the example slides function as good examples, than it is that they function as an accurate scoring mechanism.

They can just get rid of the accuracy score altogether and just give everyone 99k xp every slide, it's not like that would make any difference. Might even convince some more people to try some slides.

EDIT: An idea for your "training using real, noisy samples" would be if we could request "show me a few slides that have a high consensus for mitochondria", with the slide and the consensus shown. That way we can look at a portfolio of already-rated slides and figure out how things look "in the wild" so to speak.

But even then, we're also dealing with the (lack of) skill of the person preparing the samples. Blur, artifacts, nucleus oozing out, cytoplasm oozing out, brightness too low, brightness too high, too much zoom (only one cell shown), too little zoom (a million cells shown) - they're not giving examples of how to deal with these situations.
Lugh Crow-Slave
#85 - 2016-04-16 21:25:58 UTC
Can we get examples of what something isn't to go along with what something is

BLOPS Hauler
The 16.8km Bubble
Shallanna Yassavi
qwertz corp
#86 - 2016-04-25 01:52:29 UTC
Memphis Baas wrote:
SurrenderMonkey wrote:
This is a good change, however, I feel like the training samples need to be harder. A lot harder.

I can actually predict, with pretty close to perfect accuracy (exceptions are, e.g., microtubules, where the unknown samples are usually pretty obvious, too), whether or not a sample will be a training sample just by its appearance.


I disagree.

First, it's a game, let people get enjoyment from seeing their accuracy increase and their rewards increase. Because the rewards are crap anyway, so going from 50k to 100k is still pretty crappy.

Second, how would you feel with your confidence of "perfect accuracy" but only 50% in the score at the top of the screen? At 25 slides per level, anyone reaching level 40 has done 1000 slides, and should be a freaking expert.

Third and last, if they make them a lot harder I will screenshot all of them and then reference the image and "guess" correctly next time. Thus, making the slides harder just reinforces the wrong kind of gameplay.

I don't want to learn more subcellular morphology than it's already been presented or than I know from general knowledge. Project Discovery isn't easy, and we're working for free, so make it harder and a lot of people will just stop working. Because EVE is a game, not work.



Discovery isn't a game, though. We're actually telling researchers what their microscope looked at in the lab, and they're probably going to use those results develop medicines and diagnose tricky diseases based on the data we give them. The higher the quality of results we give, the better they can do exactly that.

What makes it feel like a game is how we get shinies and boosters from it, and how it "keeps score."
With the accuracy system the way it is, it's not all that difficult to game, because most of the test slides are pretty obvious. Some of them aren't obvious, and some of the obvious ones claim they weren't test samples.

A signature :o
Memphis Baas
#87 - 2016-04-25 22:20:22 UTC
Shallanna Yassavi wrote:
Discovery isn't a game, though. We're actually telling researchers what their microscope looked at in the lab, and they're probably going to use those results develop medicines and diagnose tricky diseases based on the data we give them.


It is a game because it's not a job, and it's not a job because we don't get paid with RL money. So they asked a bunch of gamers to click some buttons inside our preferred game, and they are giving no feedback after you've clicked. Sounds like an EVE minigame to me, much like archeology or mining.

It's a game because:

- you have to start up the EVE game client to access PD.
- you get ISK and item rewards from it.
- we can "game" the choices if we want to.

- there are no repercussions (What if everyone clicking cytoplasm results in the "future medicine" being a carcinogen, for example? Are they going to arrest any of us?).

- there are no supervisors and there is no peer review.
- there is no pay.

You want everyone to be serious and treat PD like a serious research project. Maybe that's what they want, too. Or maybe, they get better results if 50,000 people click it like a game, than if only 50 click it seriously and everyone else says "screw it, this is no fun, let's shoot some ships."
Tanya Anatolia
Minerva's Ward
#88 - 2016-04-27 16:16:36 UTC
Skimmed through a good share of this thread, but with good reason, I have a personal gripe about a bad slide. Normally I'm fine with one that's a little grey zone or hard to classify, and I admit I've gotten my fair share wrong, but this one got right in my craw:

http://imgur.com/M3YioIz

I see two instances of a cytokinetic bridge and one instance of a lactin filament, but it's classified as intermediate filaments. My accuracy score took a huge hit from that one, and I take pride in my accuracy score.

It's low-effort ISK and a chance at a shiny lab coat, so I'm still "satisfied" but this particular slide took a saki bowl worth of wine and some complaining in corp chat before I felt better about it.
Moondo
Unforeseen Consequences.
Valkyrie Alliance
#89 - 2016-04-28 01:54:01 UTC
Really, PD???

http://imgur.com/a/KH5W7

REALLY??!!?!?!?
HPA Illuminator
H P A
C C P Alliance
#90 - 2016-04-29 13:38:42 UTC
Tanya Anatolia wrote:
Skimmed through a good share of this thread, but with good reason, I have a personal gripe about a bad slide. Normally I'm fine with one that's a little grey zone or hard to classify, and I admit I've gotten my fair share wrong, but this one got right in my craw:

http://imgur.com/M3YioIz

I see two instances of a cytokinetic bridge and one instance of a lactin filament, but it's classified as intermediate filaments. My accuracy score took a huge hit from that one, and I take pride in my accuracy score.

It's low-effort ISK and a chance at a shiny lab coat, so I'm still "satisfied" but this particular slide took a saki bowl worth of wine and some complaining in corp chat before I felt better about it.


Aye, there are two ckn bridges visible, but as they aren't stained in the green channel, one should not label them. What you do see is a small intermediate filament stretching over the nucleus, so I'd say it's correct (maybe we should add ccv too though).

Having said that, we created a reddit thread for reporting incorrect samples, to easier collect and follow up at:

https://www.reddit.com/r/Eve/comments/4gtoyg/project_discovery_collecting_incorrect_control/
HPA Illuminator
H P A
C C P Alliance
#91 - 2016-04-29 13:39:47 UTC
Moondo wrote:
Really, PD???

http://imgur.com/a/KH5W7

REALLY??!!?!?!?


This sample has been reported, and i agree it should be nuclear bodies (many).

If you take screen dumps, please make sure to include the ID in bottom right corner (and report them to https://www.reddit.com/r/Eve/comments/4gtoyg/project_discovery_collecting_incorrect_control/).
HPA Illuminator
H P A
C C P Alliance
#92 - 2016-04-29 13:47:20 UTC
Drake Carver wrote:
TL:DR >>This post is one big sloppy thank you for this wonderful game. Skip to the end for the actual question.


From the beginning I have supported project discovery for its purpose and a fascination with the microscopic world.

From my experience so far I find it disheartening that the system can and will be gamed however I am certain methods will arise and adapt to optimize for the best results for the hardworking researchers that will actually make use of the results to refine our knowledge to be less of a guessing game in whatever we MMO player assist in.

Now I may be sorely mistaken in the things I am about to suggest but there are some functions that I feel would at least allow serious project discovery players to get a little more satisfaction from a job well done.

I often find that sometimes I have made an obvious mistake only after checking all the marks.

If there could be a way to report that I am truly mistaken in my result so that I cannot knowingly skew the results I would be thankful. I would still gladly accept the consequences of reductions in accuracy or the nullification of any gains I would have stood to earn. Letting obviously bad results through does not feel right and if I could make reparations I would.


TL:DR >>Can there be a button for reporting that a mistake has been made but it was realized only after submitting.


I like the idea, will add to idea-document :) However, it's completely up to CCP whether they can and want to implement it.
Tanya Anatolia
Minerva's Ward
#93 - 2016-04-29 14:54:44 UTC  |  Edited by: Tanya Anatolia
I'm glad you're working so actively on quality assurance. This crowdsourcing project was a great idea and it makes me feel good not only that I'm helping science progress, but that I'm making ISK in my downtime.

as far as in-game goes, I'm a miner, one of several in my Corp. I never AFK mined to begin with so this just helped break up the tedium, but I have noticed it's greatly reduced the amount of AFK mining others do. Cranking these sheets out with a >60% accuracy score is almost enough to double the income of people with lower skills and cheaper modules.

I still have my eyes on that shiny labcoat, but at my current rate of LP gains it'll take a while to get there, how long will project discovery be going on for?
HPA Illuminator
H P A
C C P Alliance
#94 - 2016-04-30 13:46:11 UTC
Tanya Anatolia wrote:
I'm glad you're working so actively on quality assurance. This crowdsourcing project was a great idea and it makes me feel good not only that I'm helping science progress, but that I'm making ISK in my downtime.

as far as in-game goes, I'm a miner, one of several in my Corp. I never AFK mined to begin with so this just helped break up the tedium, but I have noticed it's greatly reduced the amount of AFK mining others do. Cranking these sheets out with a >60% accuracy score is almost enough to double the income of people with lower skills and cheaper modules.

I still have my eyes on that shiny labcoat, but at my current rate of LP gains it'll take a while to get there, how long will project discovery be going on for?



Happy to hear your thoughts on PD! I don't know how long it will be on, but CCP has said that as long as it's popular they'll keep it.

In terms of improving accuracy, maybe this would be helpful? We'll also be doing an introductory PD class at EVE uni on Tuesday (first ever :)).