tweaking the review process (was: signals2 review results)
Hi Paul, On Fri, Nov 21, 2008 at 1:00 AM, Paul Baxter <pauljbaxter@hotmail.com> wrote:
On Thu, Nov 20, 2008 at 11:40 PM, vicente.botet <vicente.botet@wanadoo.fr> wrote:
I would like to make some suggestion to improve the review management:
<valid points snipped>
Whilst I understand your points, unlike our day job, people do have other more pressing commitments and, for many, the extensive effort given to reviewing boost libraries is time given in an ad-hoc manner.
Whilst it is good to have the timetable and a specific method for submitting reviews, I don't think we need to go further and be more regimented. The existing flexibility is a positive part of the process, IMHO.
I also tend to prefer flexibility when people are volunteering their time.
[ I do have problems with some libraries actually getting to the point of review before interest, maturity of interface or implementation are right, but that doesn't apply here and has been covered in previous discussions.]
I think I was guilty of this / have been bit by this myself. Anyway, there are two things that I think can help here: * requiring a certain number of committed reviewers before scheduling a review * the review manager being more active in examining the library before the actual review About the latter - in both reviews I managed, there was a number of issues (e.g., documentation shortfalls) that came up during the review that I, as review manager, could have discussed privately with the author beforehand had I done a more thorough review of the library. These issues could have been fixed before a review was scheduled, and the reviews could have been more focused on other issues. The review process page states: "The Review Manager... Checks the submission to make sure it really is complete enough to warrant formal review. See the Boost Library Requirements and Guidelines. If necessary, work with the submitter to verify the code compiles and runs correctly on several compilers and platforms." Well, in both cases I had examined the library, tried it on several compilers, read the docs, glanced at the implementation, etc. But it wasn't a thorough review. I think, had I actually gone through the reviewer's list of questions and wrote a full review, it would have given the authors some idea of possible areas that can be improved before the review (being careful not to react prematurely in possibly contentious areas where the opinion of just one person is not enough). Doing something like this would require additional effort from the review manager, but I think it would result in a much better review (and one where it is possibly easier to make a decision because there are fewer remaining problems). Best, Stjepan
Stjepan Rajko wrote:
Hi Paul,
On Fri, Nov 21, 2008 at 1:00 AM, Paul Baxter <pauljbaxter@hotmail.com> wrote:
On Thu, Nov 20, 2008 at 11:40 PM, vicente.botet <vicente.botet@wanadoo.fr> wrote:
I would like to make some suggestion to improve the review management: <valid points snipped>
Whilst I understand your points, unlike our day job, people do have other more pressing commitments and, for many, the extensive effort given to reviewing boost libraries is time given in an ad-hoc manner.
Whilst it is good to have the timetable and a specific method for submitting reviews, I don't think we need to go further and be more regimented. The existing flexibility is a positive part of the process, IMHO.
I also tend to prefer flexibility when people are volunteering their time.
[ I do have problems with some libraries actually getting to the point of review before interest, maturity of interface or implementation are right, but that doesn't apply here and has been covered in previous discussions.]
I think I was guilty of this / have been bit by this myself.
Anyway, there are two things that I think can help here: * requiring a certain number of committed reviewers before scheduling a review * the review manager being more active in examining the library before the actual review
About the latter - in both reviews I managed, there was a number of issues (e.g., documentation shortfalls) that came up during the review that I, as review manager, could have discussed privately with the author beforehand had I done a more thorough review of the library. These issues could have been fixed before a review was scheduled, and the reviews could have been more focused on other issues. The review process page states: "The Review Manager... Checks the submission to make sure it really is complete enough to warrant formal review. See the Boost Library Requirements and Guidelines. If necessary, work with the submitter to verify the code compiles and runs correctly on several compilers and platforms."
Well, in both cases I had examined the library, tried it on several compilers, read the docs, glanced at the implementation, etc. But it wasn't a thorough review. I think, had I actually gone through the reviewer's list of questions and wrote a full review, it would have given the authors some idea of possible areas that can be improved before the review (being careful not to react prematurely in possibly contentious areas where the opinion of just one person is not enough). Doing something like this would require additional effort from the review manager, but I think it would result in a much better review (and one where it is possibly easier to make a decision because there are fewer remaining problems).
Best,
Stjepan
Do you think it would help if the Wizards stressed this need and requested an affirmation of it as part of the lead in to the review? John
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Fri, Nov 21, 2008 at 10:52 AM, John Phillips <phillips@mps.ohio-state.edu> wrote:
Stjepan Rajko wrote:
* the review manager being more active in examining the library before the actual review
About the latter - in both reviews I managed, there was a number of issues (e.g., documentation shortfalls) that came up during the review that I, as review manager, could have discussed privately with the author beforehand had I done a more thorough review of the library. These issues could have been fixed before a review was scheduled, and the reviews could have been more focused on other issues. The review process page states: "The Review Manager... Checks the submission to make sure it really is complete enough to warrant formal review. See the Boost Library Requirements and Guidelines. If necessary, work with the submitter to verify the code compiles and runs correctly on several compilers and platforms."
Well, in both cases I had examined the library, tried it on several compilers, read the docs, glanced at the implementation, etc. But it wasn't a thorough review. I think, had I actually gone through the reviewer's list of questions and wrote a full review, it would have given the authors some idea of possible areas that can be improved before the review (being careful not to react prematurely in possibly contentious areas where the opinion of just one person is not enough). Doing something like this would require additional effort from the review manager, but I think it would result in a much better review (and one where it is possibly easier to make a decision because there are fewer remaining problems).
Best,
Stjepan
Do you think it would help if the Wizards stressed this need and requested an affirmation of it as part of the lead in to the review?
Definitely. I'm not sure whether stressing this requirement would turn off potential review managers because of the added effort, but I think it would make the overall review process better. Stjepan
Stjepan Rajko wrote:
On Fri, Nov 21, 2008 at 10:52 AM, John Phillips <phillips@mps.ohio-state.edu> wrote:
Stjepan Rajko wrote:
[snip] Do you think it would help if the Wizards stressed this need and requested an affirmation of it as part of the lead in to the review?
Definitely. I'm not sure whether stressing this requirement would turn off potential review managers because of the added effort, but I think it would make the overall review process better.
Stjepan
I don't want to discourage anyone who will do a good job of managing a review, but if we decide this is a necessary step in providing a good review then I'm not upset by discouraging anyone who won't do it. John
on Fri Nov 21 2008, John Phillips <phillips-AT-mps.ohio-state.edu> wrote:
Stjepan Rajko wrote:
On Fri, Nov 21, 2008 at 10:52 AM, John Phillips <phillips@mps.ohio-state.edu> wrote:
Stjepan Rajko wrote:
[snip] Do you think it would help if the Wizards stressed this need and requested an affirmation of it as part of the lead in to the review?
Definitely. I'm not sure whether stressing this requirement would turn off potential review managers because of the added effort, but I think it would make the overall review process better.
Stjepan
I don't want to discourage anyone who will do a good job of managing a review, but if we decide this is a necessary step in providing a good review then I'm not upset by discouraging anyone who won't do it.
I'm not sure just what's being proposed here (so this might be off-base), but it's important not to make anything unnecessarily burdensome. We need to rely on peoples' intelligence and sense of responsibility without making egregious rules. For example, a review manager should not -- in general -- need to do a comprehensive review herself; she has enough on her plate already. 95% of the time it should be enough to make a judgement based on the other reviews: both their votes and their quality, depth, etc. At the end of the review period, if she is truly left unsure whether the library should be accepted, a review manager might go through the process herself, but otherwise a comprehensive review from the manager should not be required. I think it would be good if the review wizards could make it clear that they are available to provide guidance in case review managers have questions about how to do their jobs well. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
participants (3)
-
David Abrahams -
John Phillips -
Stjepan Rajko