EVE General Discussion

With Project Discovery now well under way, I must say I am surprised by the community's ability to accurately report on a significant portion of samples (for a community of trolls, scammers and schemers, it isn't bad at all!). I am very much enjoying the work thus far, particularly those challenging, niggly slides! I do have some suggestions from my experience thus far:

To Project Discovery coordinators:

1. The hardware the programme is running on needs to be upgraded or a queueing system setup for requests. Spending a significant amount of time on a slide only to find that your interpretation is rejected by the server is remarkably irksome.

2. Providing the cell line for each slide would be very helpful. While most players are not familiar with cell types, as they gain experience they will become familiar with distribution patterns in certain cells. I think the benefits here will outweigh those of 'blinding' in the medium to long term. I don't know many cytologists who look at samples not knowing the cell type. Examples of issues here include cells with small nucleoli, nucleolar number and cytoplasmic form.

3. While the selection of (particularly cytoplasmic) targets is nice, it doesn't quite encompass all options, leading to community confusion. With protein localisation being such a complex area, this was bound to happen from the outset. I have, for example, found cytoplasmic proteins localising toward the nuclear envelope with no obvious binding to Golgi or ER. Further options could also be opened up to experienced screeners.

4. A grading system would be infinitely helpful for experienced cytologists. At present there is no way to indicate protein distribution between nucleus and cytoplasm. A protein, for example, may be more prominent in the nucleoplasm than in the cytoplasm. To help ensure correct application, this feature could be introduced at 'Level 20'. Even if players aren't accurate due to subjectivity, the ratios of FITC binding are likely to be similar.

5. Players are having a hard time using the channels. I often find that obvious nuclear binding has been missed by multiple players (due to FITC being hidden under the Hoechst stain). Helping players more with this in the tutorial will likely resolve the issue.

6. Example images would benefit immensely from the ability to switch channels on and off.

7. A way to flag samples for attention would be good. Many times I have come across images where the community have missed something (like nucleolar rim or fibrillar binding). You could set yourselves a threshold for when you choose to review them.

8. The community are frequently omitting to use 'intercellular variation' where appropriate. Perhaps a couple in the tutorial would help. Actually if I remember correctly one was missed in the tutorial (maybe image 9, but I don't recall).

9. More information on the project would be wonderful! At present, players are forced largely to Google. A webpage with basic information describing what the project is, something about protein localisation and functions for the layperson, what the projected outcomes are and why the research is important would be lovely. It would also be nice to know more about how you are controlling the experiment (eg. analysis by your own screeners and multiple images from the same HCA) and how the data will be filtered and analysed.

10. Feedback. This is sorely missing at present. Being able to review slides which were subsequently scored by an expert would help greatly. These should be categorised by our personal correct and incorrect entries (this part is CCP's area). I understand this will be a significant task to implement, but essential if you are serious about building an adroit community of screeners. The effort now will pay off as fewer slides will require review later on since experienced members of the community will not be making the same errors constantly, thus reaching accurate consensus more often.

11. Stick with it! There will always be teething issues with an ambitious project like this one. The community will improve with experience. Re-testing some of the early slides in a few weeks may return better results. Encouraging and rewarding player accuracy is paramount!


To CCP:

1. The rewards system needs to be non-linear to encourage correct identification. Consider a logarithmic scale. A good cytologists is worth more than 20 button mashers.

2. Accuracy would be better based on more than simply how many are in agreement with the community. The community makes mistakes. Not only this but some players are gaming the system by cancelling out the difficult slides to gain better 'accuracy'. The system for determination would benefit from being more robust and rewarding those who successfully identify the less common markers accurately. Consider working together with Prof. Lundberg's team to improve the system.

3. Really nice work with the interface - I'm impressed!


Well that's my 20 cents worth for today!

My thoughts: The thing is a Pavlovian Device. It's there to study your behavior. It doesn't matter if you're matching up protoplasm, or the variety of stripes on zebras. That's not the object. The object is you...and your pet mouse.

I think more than one third of slides are borderline cases. Cytoplasm is difficult to be distinguished from background noise. Nucleus can co-exit with cytoplasm even though the description said cytoplasm is seen throughout the whole cell, except in the nucleus. Nucleoplasm and Nucleoli are possible to appear next to each other. I can't tell the difference between cytoskeleton(microtubule ends) and focal adhesions. Sometimes I cannot decide whether it is aggresome or microtubule organizing center. I have hardtime to choose between cytoskeleton(actin filaments) and cell junctions. Intermediate filaments and endoplasmic reticulum look like twins to me. These are not true or false questions. The classification process is more like an art rather than science. No wonder computer performs badly in this job.

I believe many players want to do it right, but have no idea how to do it. It would be great if more information or training can be provided. I believe our community can be great contributors to this never-seen-before wonderful citizen science project.

To show you the correct answer would mean for them to know the correct answer. If they knew it, then there would be no project discovery. That's the point of the project. To discover, to find out. And, depending on the majority of checks, they will then analyse those proteins.

It doesn't matter what others choose. If you choose right, most of the times you will select what others selected as well,
As long as you don't loose accuracy, you are on the right track.

I believe the issue for many of the slides is not that the experts can not reach consensus but that the volume of slides is simply too much to handle for a scientific team. In this regard guidance for us newbies is essential to ensure that we are generally in consensus. Major issues persist in false positive flagging of background noise and unwashed stain artefact (so many I have seen artefact reported as aggresome. This is not the fault of players - they simply aren't accustomed to looking at fluorescence microscopy.

They do not need us because they cant figure out the correct answer, but because they need a way to solve many images. But this only works if we are trained correctly. And for this we need more training samples and better feedback.
If the only feedback is from other users, that also have no good training, then there is no reason to assume that our training will converge to correct solutions. It will probably converge somewhere, but probably more into some kind of pattern that allows a fast consenus, not a correct one.

I would love to have a more comprehensive tutorial, even something like a test with some pointers on the errors that I made so I can learn where to look to better myself. Right now I'm not sure where to look to analyse my mistakes and grow from them.

So far at analyst rank 8 I usually only see red filaments. Maybe its just random but, again, I'd like to have some pointers.

Anyways I applaud the initiative as a whole and I want to support it.
I would even watch a 1 hour video explaining the classification for 3 dozens samples, or some reviews on the community statistics by a professional so I can have a better understanding of things...

Keep doing things like this pls