These forums have been archived and are now read-only.

The new forums are live and can be found at https://forums.eveonline.com/

EVE Information Portal

 
  • Topic is locked indefinitely.
 

Dev blog: An Update on Bug Reporting, Part 3

First post First post
Author
Previous Topic Next Topic
Kenneth Feld
Habitual Euthanasia
Pandemic Legion
#41 - 2014-08-20 00:16:10 UTC
I submitted a bug in Dec 2013, never heard a thing about it, kinda gave up on bug reports

Looks today and find out nothing has been touched on it. Although i re submitted it on a separate pilot.

23 accounts and 41 pilots, anyway, we could make it so you can see all your bug reports in one area. i found several duplicates today as well, that i didn't even remember writing


Also, added 7 more that i had been putting off writing cause well, not getting feedback kinda leads you to believe no one knows or cares, although seeing one untouched for 9 months kinda does the same thing...
Salpun
Global Telstar Federation Offices
Masters of Flying Objects
#42 - 2014-08-20 00:51:55 UTC  |  Edited by: Salpun
Currently having to reenter my credentials to access the info pages.
And no option to go back once in the page.
Fixed only happened the first time but please take a look at the coding thanks.
Edit:
Started happening again. Its like the access is timing out really fast.

If i dont know something about EVE. I check https://wiki.eveonline.com/en/wiki/ISK_The_Guide

See you around the universe.
CCP Habakuk
C C P
C C P Alliance
#43 - 2014-08-20 00:54:28 UTC
Noriko Mai wrote:
To be honnest, I stopped writing new reports because I thought no one looks into it. After a few months I could not remember what my report was about, just that I've written one. I just checked my bugreports and most of them are attechend to a defect. So someone actually looked at it and something may/will happen. It's is so satisfying. (No joke or sarcasm) *hugs* will write bugreports again. Sorry for not believing Smile

EDIT: I just checked all of them and some are fixed or never made it from SiSi to TQ.

  • EBR-15230 is fixed.
  • EBR-16144 is fixed.
  • EBR-16465 is fixed.
  • EBR-19126 is fixed.

So if I'm not mistaken or my reports are only a small part of a bigger defect, all of them can be closed.


I had a short look and it seems like all four of those got fixed because of your bug reports. Thank you! Big smile

CCP Habakuk | EVE Quality Assurance | Team Five 0 | (Team Gridlock)

Bug reporting | Mass Testing
CCP Habakuk
C C P
C C P Alliance
#44 - 2014-08-20 01:17:22 UTC
Kenneth Feld wrote:
I submitted a bug in Dec 2013, never heard a thing about it, kinda gave up on bug reports

Looks today and find out nothing has been touched on it. Although i re submitted it on a separate pilot.

23 accounts and 41 pilots, anyway, we could make it so you can see all your bug reports in one area. i found several duplicates today as well, that i didn't even remember writing


Also, added 7 more that i had been putting off writing cause well, not getting feedback kinda leads you to believe no one knows or cares, although seeing one untouched for 9 months kinda does the same thing...


Yeah, unfortunately we were not successful in being able to process all bug reports within the last year and we did not decide to mass-close the remaining ones (which might have looked better).
If you find a very old bug report not closed and you have additional info to it - or you found out that it was not a bug: Please add a comment to it. I'll try to go through bug reports which were updated and see if I can do anything with them.

Btw, some background info: The main part of the bug reports are processed by our volunteer bug hunters and smaller parts are done by us in QA and by our testing partner. Due to several reasons (including the signup being closed for a long time due to technical difficulties) we had less active bug hunters than in previous years - this had quite some impact on our ability to process bug reports. In the last few months the situation improved a lot due to several new and some very active bug hunters (Thanks go especially to BH Nonlocality) - but we could need several additional volunteers. Follow the links in the top post, if you are interested in helping as bug hunter.

CCP Habakuk | EVE Quality Assurance | Team Five 0 | (Team Gridlock)

Bug reporting | Mass Testing
Rionan Nafee
#45 - 2014-08-20 07:11:09 UTC
Thank you for your improvements and the infos about them.


One suggestion:

How would it be if the Bug Hunters/Dev Teams can "rate" the bug reports and give comments about the quality of the submitted bug reports?

So players can see how usefull their reports were and how they can improve their next ones therefore the neverending work of the Bug Hunters and Devs can be make easier.
CCP Goliath
C C P
C C P Alliance
#46 - 2014-08-20 10:12:56 UTC
David Laurentson wrote:
CCP Goliath wrote:

Thanks very much for telling us how you feel. Sorry that you don't think we take Sisi feedback on board, but I assure you the teams are as diligent as possible when it comes to reading the threads and discussing the ideas in them! That being said, the items in the hypothetical list would already have been acknowledged by us to be issues, so does that go some way to mitigating your concerns?


I think his concern is that those issues were acknowledged when they hit TQ, and closed in error on SiSi. Not ignored in threads, Bug Reports were raised, and closed, despite them actually being bugs.

Do you do any investigation of TQ-impacting bugs to see if they were raised on SiSi, and how they were handled? If not, it might be worthwhile: you might have some internal communication work to do, or need to change how you set targets (I can only guess, not working at CCP!).


This might be to do with our version control then. Some bugs will be closed while still being a problem on TQ, because we have verified that the fix works on one of our test environments (Sisi is one, but there are many others)

We do perform RCA (root cause analysis) on certain escaped issues (escaped describes the defect "escaping" from the test environment into the live environment). This is usually facilitated by me, with the involved team(s), and we just go over how the defect managed to get to TQ, strictly looking at the facts and leaving any blame or recrimination out of the picture (one of the worst things you can do, especially in QA, is create an environment where failure has massive negative consequences so that people become afraid of it - that's how problems stay hidden instead of being discussed and solved). Once we know how it happened, we work out what lessons we can take from this so as to not have it happen again, and usually make sure we have a test case or two in our regression suite to cover the defect and areas around it.

Having hard targets with defects is a very double edged sword - some targets are positive, such as "no A or B class bugs on release", others are impossible and dangerous, such as "0 bugs for every feature" or "a good QA should find X bugs per day".

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
CCP Goliath
C C P
C C P Alliance
#47 - 2014-08-20 10:13:45 UTC
Che Biko wrote:
CCP Goliath wrote:
Additionally, editing a bug once it has been attached or closed will be futile, as we will no longer be checking that report.
Then why not remove the edit option/button for those reportsQuestion People may not know about the futility, and think "The button/option is still there, so it must serve a purpose."


We likely will in the next iteration, just didn't have time before Tier went on summer vacation Smile

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
CCP Goliath
C C P
C C P Alliance
#48 - 2014-08-20 10:18:32 UTC
Tvashnar Crendraven wrote:
CCP Goliath wrote:


Bug reports don't get "fixed" per se - they either become or are attached to defects, which are fixed, but there is no way currently to relay the state of the defect to the bug report attached to it. Going back over old bug reports to set them to fixed would be very time consuming.


No doubt. Bugs have costs, and those costs are higher if you ship them to customers.

"'Closed" does not tell me whether I should not be seeing the issue anymore; or whether my understanding is broken and it is not actually a bug or any number of other things which might make me feel that the time I spent to file the bug was not wasted.

Simply closing the issue without comment or a tag like 'fixed', 'not-repro', 'by design' or even 'raised a bug in our real bug tracking system' means that this nice new interface into our bug reports is *useless*.


This seems to be a common sentiment, and we will discuss it internally and see if adding the resolution field (which would show you the Closed resolution, but wouldn't provide any details on fixes, for reasons I've explained in other posts) is something we can fit in to the next iteration. "Attached" means "Raised a bug in our real bug tracking system", relatively speaking.

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
CCP Goliath
C C P
C C P Alliance
#49 - 2014-08-20 10:19:53 UTC
Circumstantial Evidence wrote:
CCP Goliath wrote:
Bug reports don't get "fixed" per se - they either become or are attached to defects, which are fixed, but there is no way currently to relay the state of the defect to the bug report attached to it. Going back over old bug reports to set them to fixed would be very time consuming.
Please work on automating this :) I have bugs both attached / "I-know-fixed", and attached / "I-don't-know-anything". I would like to see a column with a one word report / state of the internally-written defect our bug is attached to.


Impossible without rewriting part of our Jira instance I believe. This isn't a desired solution as it makes plugin maintenance and ongoing support much more complex and risky. Sorry, but at least for now this isn't going to happen. Sad

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
CCP Goliath
C C P
C C P Alliance
#50 - 2014-08-20 10:24:10 UTC
Vincent Athena wrote:
I like the idea of being able to vote on legacy bugs. But, if you have no way of setting a bug to at status of "fixed" how will this list be populated? You will not know which bugs to include in the list as not a single one will have any indication that it has been fixed!

I would think that if a bug can be attached to a defect, then when that defect has been fixed, that information could automatically flow back to the bug report.

Also the status "closed" is just not informative enough. If something has "insufficient information", I would like to know, so I can try explaining it again. If it cannot be reproduced, I would like to try again, maybe with some other method for repro that might work for you. If its "working as intended" I would like to know so I can do a feature request in F & I.


The legacy bugs would be in our database, so would have resolutions, and obviously we can see when defects are fixed - the thing here is that defects are invisible to you, and what you see are called bug reports. The jargon is pretty important when discussing both at the same time to differentiate.

Unfortunately the defect tracking software doesn't work like you describe with the flow of issues, at least at this time, but it does get worked on continually by Atlassian, so who knows what the future may bring!

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
CCP Goliath
C C P
C C P Alliance
#51 - 2014-08-20 10:31:24 UTC
Rionan Nafee wrote:
Thank you for your improvements and the infos about them.


One suggestion:

How would it be if the Bug Hunters/Dev Teams can "rate" the bug reports and give comments about the quality of the submitted bug reports?

So players can see how usefull their reports were and how they can improve their next ones therefore the neverending work of the Bug Hunters and Devs can be make easier.


This is something that's come up in our meetings before. On the one hand, as you say it's great to give people positive feedback when they submit a report that leads to us fixing or discovering an issue. On the other hand, the maintenance around such a system (you'd need rules, and checks, and consistency, etc) as well as the work that would have to go into creating it, might be a little much, or the wrong effort in the wrong place... It's definitely not entirely ruled out, but right now I don't have a good idea of how it would work well either.

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
Momiji Sakora
Omni Galactic
Central Omni Galactic Group
#52 - 2014-08-20 10:44:07 UTC  |  Edited by: Momiji Sakora
This is great progress, been asking and wanting something like this for a long while. So glad to see it actually implemented!

I would like a bit of feedback in the general "This was closed because XYZ" or "Resolved in Hyperion build #####" etc. I don't think we need personalised responses of course, that'd be a bit too much.

EDIT: I can see why this might not be implemented anytime soon, but it would be a welcome addition. Generally when working with Jira we've merged duplicates and then used the link option to link it to duplicates. It'd just require a copy paste fix, obviously it's working out if the time would be better spent elsewhere.
Rionan Nafee
#53 - 2014-08-20 16:11:36 UTC
CCP Goliath wrote:
Rionan Nafee wrote:
Thank you for your improvements and the infos about them.


One suggestion:

How would it be if the Bug Hunters/Dev Teams can "rate" the bug reports and give comments about the quality of the submitted bug reports?

So players can see how usefull their reports were and how they can improve their next ones therefore the neverending work of the Bug Hunters and Devs can be make easier.


This is something that's come up in our meetings before. On the one hand, as you say it's great to give people positive feedback when they submit a report that leads to us fixing or discovering an issue. On the other hand, the maintenance around such a system (you'd need rules, and checks, and consistency, etc) as well as the work that would have to go into creating it, might be a little much, or the wrong effort in the wrong place... It's definitely not entirely ruled out, but right now I don't have a good idea of how it would work well either.

Thank you for this information.

Is there any other way possible to know if the submitted bug reports are usefull or only a pain in the ass for the bug hunters?
CCP Goliath
C C P
C C P Alliance
#54 - 2014-08-20 16:44:40 UTC
Rionan Nafee wrote:
CCP Goliath wrote:
Rionan Nafee wrote:
Thank you for your improvements and the infos about them.


One suggestion:

How would it be if the Bug Hunters/Dev Teams can "rate" the bug reports and give comments about the quality of the submitted bug reports?

So players can see how usefull their reports were and how they can improve their next ones therefore the neverending work of the Bug Hunters and Devs can be make easier.


This is something that's come up in our meetings before. On the one hand, as you say it's great to give people positive feedback when they submit a report that leads to us fixing or discovering an issue. On the other hand, the maintenance around such a system (you'd need rules, and checks, and consistency, etc) as well as the work that would have to go into creating it, might be a little much, or the wrong effort in the wrong place... It's definitely not entirely ruled out, but right now I don't have a good idea of how it would work well either.

Thank you for this information.

Is there any other way possible to know if the submitted bug reports are usefull or only a pain in the ass for the bug hunters?


At this time there definitely isn't an easy way to relay that info back to you, barring commenting "great report" or similar on the report, which will absolutely have to be left up to the individuals processing the reports (but will be encouraged)

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
Salpun
Global Telstar Federation Offices
Masters of Flying Objects
#55 - 2014-08-20 16:47:05 UTC
CCP Goliath wrote:
Rionan Nafee wrote:
CCP Goliath wrote:
Rionan Nafee wrote:
Thank you for your improvements and the infos about them.


One suggestion:

How would it be if the Bug Hunters/Dev Teams can "rate" the bug reports and give comments about the quality of the submitted bug reports?

So players can see how usefull their reports were and how they can improve their next ones therefore the neverending work of the Bug Hunters and Devs can be make easier.


This is something that's come up in our meetings before. On the one hand, as you say it's great to give people positive feedback when they submit a report that leads to us fixing or discovering an issue. On the other hand, the maintenance around such a system (you'd need rules, and checks, and consistency, etc) as well as the work that would have to go into creating it, might be a little much, or the wrong effort in the wrong place... It's definitely not entirely ruled out, but right now I don't have a good idea of how it would work well either.

Thank you for this information.

Is there any other way possible to know if the submitted bug reports are usefull or only a pain in the ass for the bug hunters?


At this time there definitely isn't an easy way to relay that info back to you, barring commenting "great report" or similar on the report, which will absolutely have to be left up to the individuals processing the reports (but will be encouraged)


Even showing how many comments have been attached to the reports would be enough to show that the items is getting worked on. Having that number in the main bugreport list would save the bandwitdh required for people to check their bug reports for comments.

If i dont know something about EVE. I check https://wiki.eveonline.com/en/wiki/ISK_The_Guide

See you around the universe.
Soldarius
Dreddit
Test Alliance Please Ignore
#56 - 2014-08-20 18:55:19 UTC
http://content.eveonline.com/www/newssystem/media/66330/1/bugreportpic1.png

Confirming that TEST has won Eve. Either that, or TEST is a bug and should be... corrected. You decide.

Also, cool stuff. Knowing where you're bur report is in the system is a good thing. People like feedback and knowing that they have been heard.

http://youtu.be/YVkUvmDQ3HY
Kenneth Feld
Habitual Euthanasia
Pandemic Legion
#57 - 2014-08-22 02:31:41 UTC
well, I got my bug report "Attached" but attached to what?
EBR-22458
What is going to get done?

The bug report is kinda one of those it used to be this way, now it is that way...

Attached doesn't tell me if the old way is correct or the new way is correct.....

How the heck am i supposed to know whether to shut up and wait 6 years for it to revert to the old way or 4 years for it to be fixed the new way?


At this rate, going back to old style forum whining is still the best answer......
CCP Goliath
C C P
C C P Alliance
#58 - 2014-08-22 09:50:49 UTC
Kenneth Feld wrote:
well, I got my bug report "Attached" but attached to what?
EBR-22458
What is going to get done?

The bug report is kinda one of those it used to be this way, now it is that way...

Attached doesn't tell me if the old way is correct or the new way is correct.....

How the heck am i supposed to know whether to shut up and wait 6 years for it to revert to the old way or 4 years for it to be fixed the new way?


At this rate, going back to old style forum whining is still the best answer......


Attached means that it has either had a defect created from it, or attached to an existing defect we already had. This is basically the best outcome of a bug report. As I've explained further up the thread, there is no way for us to currently relay the status of defects that bug reports are attached to, to you. In the case of this issue, you filed it on the 19th August, we processed it on the 21st and CCP Claymore made a defect from it. That defect is now assigned to a developer and waiting to be fixed (note, I do not know the nature of the intended fix). If the report was invalid, it would show as Closed rather than Attached.

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath
Nicen Jehr
Subsidy H.R.S.
Xagenic Freymvork
#59 - 2014-08-25 00:24:41 UTC  |  Edited by: Nicen Jehr
CCP Goliath wrote:
When you say extend the public bug tracker idea to feature requests, do you mean that you would vote on features that we have proposed, or that you would submit features of your own? We already have a place for the latter - in case you aren't aware, it's the Features and Ideas forum.
I'm proposing automating both. Right now, to get an idea to CCP karkur and CCP Punkturis they have to read the Little Things thread and notice the post and presumably make a note in your internal to-do list tracking.

I propose instead a features and suggestions app where pilots use SSO to prove identity and can post specific ideas. If you want to keep discussion on the forums then you could programatically make a features and suggestions thread for each post. But the goal is for me to post 100 little things, and have other pilots upvote the 10 most desired ideas, so your devs don't have to sort through the less-desired 90.

Little Things to improve ≡⋁≡ | My Little Things posts
CCP Goliath
C C P
C C P Alliance
#60 - 2014-08-25 10:19:23 UTC
Nicen Jehr wrote:
CCP Goliath wrote:
When you say extend the public bug tracker idea to feature requests, do you mean that you would vote on features that we have proposed, or that you would submit features of your own? We already have a place for the latter - in case you aren't aware, it's the Features and Ideas forum.
I'm proposing automating both. Right now, to get an idea to CCP karkur and CCP Punkturis they have to read the Little Things thread and notice the post and presumably make a note in your internal to-do list tracking.

I propose instead a features and suggestions app where pilots use SSO to prove identity and can post specific ideas. If you want to keep discussion on the forums then you could programatically make a features and suggestions thread for each post. But the goal is for me to post 100 little things, and have other pilots upvote the 10 most desired ideas, so your devs don't have to sort through the less-desired 90.


It's an interesting notion for sure. Out of my remit, but will definitely pass it on to others.

CCP Goliath | QA Director | EVE Illuminati | @CCP_Goliath