[GSoC] Proposal reviewing
Mentors, We need to establish some guidelines for reviewing proposals. They won't be complicated :) I'm opting not to use the mentors mailing list for this announcement. I know that all the mentors are subscribed here. Here's how this should work: 1. Identify the set of proposals for projects that you are interested in mentoring. 2. Evaluate the proposal and write your evaluation as a PRIVATE comment. 3. If you have questions of the student, you can write PUBLIC comments or send them an email. 4. Rank the proposal (I think 5 is the best this year). That would be the project that you want to mentor. Towards the end of the review process, we will select the 10 best proposals. This doesn't guarantee that we'll *fund* 10 proposals, just that I've requested 10 slots. I would like, this year, if all students got feedback regarding their proposals even if they are clearly rejects. The reason for writing evaluations privately, for now, is that it will help me write a public summary for proposals that are not accepted. I'll figure out how this is going to work over the course of the week :) Specific evaluation criteria are up to you (the mentors). You may ask students to modify good proposals. You do not need to ask *all* students to improve their proposals. At the end of the week, you should have a list of 1 or 2 proposals that *you yourself* will mentor. If you are not offering to mentor the project, please don't review it as a 5. Otherwise, we end up with a case where we have to assign a mentor to a project that they aren't interested in. I don't think that this has worked out very well in the past. Also, would prefer that comments regarding specific proposals not be sent to the list. I don't think it would be a good idea for a flood of "this proposal is AWFUL" emails to start popping up. I'm all for transparency, but I don't think airing value judgements would be a beneficial for community development :) Questions or comments involving evaluation criteria and process are fine. Andrew
On 09/04/2011 20:05, Andrew Sutton wrote:
Here's how this should work: 1. Identify the set of proposals for projects that you are interested in mentoring. 2. Evaluate the proposal and write your evaluation as a PRIVATE comment. 3. If you have questions of the student, you can write PUBLIC comments or send them an email. 4. Rank the proposal (I think 5 is the best this year). That would be the project that you want to mentor.
I've started ranking some bad proposals as a 1, since they removed the negative point system. I was told by the GSoC folks that we should use the average score to rank proposals rather than the cumulative one.
At the end of the week, you should have a list of 1 or 2 proposals that *you yourself* will mentor. If you are not offering to mentor the project, please don't review it as a 5. Otherwise, we end up with a case where we have to assign a mentor to a project that they aren't interested in. I don't think that this has worked out very well in the past.
Why is that scheme necessary? The score and the list of possible mentors are separate things. We could just select the top 10 projects with the highest average that have at least one mentor assigned to them, resolving the cases where the same mentor is assigned multiple times manually.
Why is that scheme necessary? The score and the list of possible mentors are separate things.
We could just select the top 10 projects with the highest average that have at least one mentor assigned to them, resolving the cases where the same mentor is assigned multiple times manually.
That's eventually how the process is going to end up working out. I'm just trying to make sure that each mentor gets their own choice of proposal rather than just trying to pick the top 10. That's not necessarily the same as the top 10 with mentors assigned, but it probably will be.
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Andrew Sutton Sent: Monday, April 11, 2011 10:41 PM To: boost@lists.boost.org Cc: Mathias Gaunard Subject: Re: [boost] [GSoC] Proposal reviewing
Why is that scheme necessary? The score and the list of possible mentors are separate things.
We could just select the top 10 projects with the highest average that have at least one mentor assigned to them, resolving the cases where the same mentor is assigned multiple times manually.
That's eventually how the process is going to end up working out. I'm just trying to make sure that each mentor gets their own choice of proposal rather than just trying to pick the top 10. That's not necessarily the same as the top 10 with mentors assigned, but it probably will be.
I'm confused (as usual :-) Should I be commenting and assigning a score to ALL proposals, adding a comment "none?". In many cases I don't have an informed view, or others have said it all already. And doing this will increase the torrent of emails, under which I am already floundering! I'm not convinced that a scoring system will prove necessary at all? Paul
Should I be commenting and assigning a score to ALL proposals, adding a comment "none?". In many cases I don't have an informed view, or others have said it all already. And doing this will increase the torrent of emails, under which I am already floundering!
I would only recommend reviewing proposals that you're interested in reviewing. You might end up ranking a proposal that another mentor has asked a student to clarify or improve. That might give it a lower score than it ultimately deserves. Think of it as a very slow race condition. You should be able to turn off email updates on your Profile page. It took me a while to figure that out too.
I'm not convinced that a scoring system will prove necessary at all?
I'm not either :) It changes from year to year...
On 2011-04-12 08:31, Andrew Sutton wrote:
Should I be commenting and assigning a score to ALL proposals, adding a comment "none?". In many cases I don't have an informed view, or others have said it all already. And doing this will increase the torrent of emails, under which I am already floundering! I would only recommend reviewing proposals that you're interested in reviewing. You might end up ranking a proposal that another mentor has asked a student to clarify or improve. That might give it a lower score than it ultimately deserves. Think of it as a very slow race condition.
I agree. Also, I find the "scoring with '1' really is '-5'" a little confusing, and thus right now I'm treating score >=1 as below threshold.
You should be able to turn off email updates on your Profile page. It took me a while to figure that out too.
I'm not convinced that a scoring system will prove necessary at all? I'm not either :) It changes from year to year...
Google really manages to keep the GSoC UI a mess, doesn't it. Stefan -- ...ich hab' noch einen Koffer in Berlin...
I agree. Also, I find the "scoring with '1' really is '-5'" a little confusing, and thus right now I'm treating score >=1 as below threshold.
Do you mean <= 1? I think 0 should mean unrated.
I'm not either :) It changes from year to year...
Google really manages to keep the GSoC UI a mess, doesn't it.
They're trying to keep us on our toes. I would prefer we had something like EasyChair, but apparently they felt like they needed to reinvent reviewing. Hopefully it will get better.
On 2011-04-12 08:54, Andrew Sutton wrote:
I agree. Also, I find the "scoring with '1' really is '-5'" a little confusing, and thus right now I'm treating score>=1 as below threshold. Do you mean<= 1?
Doh, yes of course !
I think 0 should mean unrated.
OK. Unrated implies no-one is interested in it, right ? So score >=2 is a filter for proposals that induce "positive interest".
I'm not either :) It changes from year to year... Google really manages to keep the GSoC UI a mess, doesn't it. They're trying to keep us on our toes. I would prefer we had something like EasyChair, but apparently they felt like they needed to reinvent reviewing. Hopefully it will get better.
Aren't we wishing for that each year ? So far it hasn't got better, just differently bad. (I'd really prefer to focus on the proposals rather than the process. But hey, look where we are having this conversation ! ;-) ) Stefan -- ...ich hab' noch einen Koffer in Berlin...
On 12/04/2011 12:48, Paul A. Bristow wrote:
I'm confused (as usual :-)
Should I be commenting and assigning a score to ALL proposals, adding a comment "none?". In many cases I don't have an informed view, or others have said it all already. And doing this will increase the torrent of emails, under which I am already floundering!
Do you really need to send a "None" comment to score a proposal? Surely if that's necessary that's a melange bug.
participants (4)
-
Andrew Sutton -
Mathias Gaunard -
Paul A. Bristow -
Stefan Seefeld