Talk Academia

ENV2021 wrote: ↑

Sun Jan 26, 2020 4:30 pm

Hi there,
a friend of mine is an MC evaluator and I would like to share with you what he explained to me about the evaluation process.
Three evaluators per proposal plus a chair that supervises the process. Each evaluator may evaluate a few proposals (10-15).
Each evaluator produces a report individually without sharing his comments; only when the three evaluators have submitted their reports it starts a consensus phase when the evaluators discuss their comments about the proposal. When they reach a consensus, and only then, they propose the score, always under the supervision of the chair. They are not allowed to see the evaluation report of the previous submitted proposal (if any) nor to know the old score, actually, this report remains undisclosed; the chair may ask the evaluators to rediscuss specific points if he considers it necessary.
Evaluators have to follow a specific point list and that is why it is so important to fill out the proposal including each and every point specified in the guideline, being specific, concrete. And you have to say the right things where they are expected to be in the proposal, not elsewhere in the text. If you state something in the "excellence" part that should have been stated in the "impact" part, well...that is your fault.
Regarding the CV, effectively the number of papers doesn't care, as it said in the guideline you may be penalised only if according to your research career you have not produced enough; evaluators check your CV to have an idea of what you have done before, especially to determine if your training programme is appropriate and if you effectively are going to learn something new which is the most important thing in this fellowship, besides project excellence. He also told me that many of the projects he reviewed were excellent but even in this case if you fail addressing all and each point they have to start to decrease the score, and few tenths of a point means getting or not the fellowship or even getting a bad score.
Said that..of course, subjectivity is impossible to remove but at least I think this is the most well-designed evaluation process I know.

ENV2021 wrote: ↑

Sun Jan 26, 2020 4:30 pm

Hi there,
a friend of mine is an MC evaluator and I would like to share with you what he explained to me about the evaluation process.
Three evaluators per proposal plus a chair that supervises the process. Each evaluator may evaluate a few proposals (10-15).
Each evaluator produces a report individually without sharing his comments; only when the three evaluators have submitted their reports it starts a consensus phase when the evaluators discuss their comments about the proposal. When they reach a consensus, and only then, they propose the score, always under the supervision of the chair. They are not allowed to see the evaluation report of the previous submitted proposal (if any) nor to know the old score, actually, this report remains undisclosed; the chair may ask the evaluators to rediscuss specific points if he considers it necessary.
Evaluators have to follow a specific point list and that is why it is so important to fill out the proposal including each and every point specified in the guideline, being specific, concrete. And you have to say the right things where they are expected to be in the proposal, not elsewhere in the text. If you state something in the "excellence" part that should have been stated in the "impact" part, well...that is your fault.
Regarding the CV, effectively the number of papers doesn't care, as it said in the guideline you may be penalised only if according to your research career you have not produced enough; evaluators check your CV to have an idea of what you have done before, especially to determine if your training programme is appropriate and if you effectively are going to learn something new which is the most important thing in this fellowship, besides project excellence. He also told me that many of the projects he reviewed were excellent but even in this case if you fail addressing all and each point they have to start to decrease the score, and few tenths of a point means getting or not the fellowship or even getting a bad score.
Said that..of course, subjectivity is impossible to remove but at least I think this is the most well-designed evaluation process I know.

Kenniz wrote: ↑

Sun Jan 26, 2020 4:57 pm

ENV2021 wrote: ↑

Sun Jan 26, 2020 4:30 pm

Hi there,
a friend of mine is an MC evaluator and I would like to share with you what he explained to me about the evaluation process.
Three evaluators per proposal plus a chair that supervises the process. Each evaluator may evaluate a few proposals (10-15).
Each evaluator produces a report individually without sharing his comments; only when the three evaluators have submitted their reports it starts a consensus phase when the evaluators discuss their comments about the proposal. When they reach a consensus, and only then, they propose the score, always under the supervision of the chair. They are not allowed to see the evaluation report of the previous submitted proposal (if any) nor to know the old score, actually, this report remains undisclosed; the chair may ask the evaluators to rediscuss specific points if he considers it necessary.
Evaluators have to follow a specific point list and that is why it is so important to fill out the proposal including each and every point specified in the guideline, being specific, concrete. And you have to say the right things where they are expected to be in the proposal, not elsewhere in the text. If you state something in the "excellence" part that should have been stated in the "impact" part, well...that is your fault.
Regarding the CV, effectively the number of papers doesn't care, as it said in the guideline you may be penalised only if according to your research career you have not produced enough; evaluators check your CV to have an idea of what you have done before, especially to determine if your training programme is appropriate and if you effectively are going to learn something new which is the most important thing in this fellowship, besides project excellence. He also told me that many of the projects he reviewed were excellent but even in this case if you fail addressing all and each point they have to start to decrease the score, and few tenths of a point means getting or not the fellowship or even getting a bad score.
Said that..of course, subjectivity is impossible to remove but at least I think this is the most well-designed evaluation process I know.

thanks for sharing this info.
my question would be: do they get all the reviews at the same time or after another? because getting them after another would heavily impair the results in my opinion. this is, because you may rate the first one better than it should be before reading the others and vice versa, you get what i mean.

ENV2021 wrote: ↑

Sun Jan 26, 2020 5:41 pm

Well...I don't think the story is different...during the consensus phase they try to reach a... consensus, I think it is normal that everyone evaluates each point based on their expertise and even on the other proposals they have to evaluate (which are not the same for the three of them)...each evaluator defends his reasons and it doesn't mean that if you think you are right you cannot stand for it or being convinced that you are misjudging...I think it is normal and fair. Has someone said before he told me that the evaluators have to provide their report before a specific date, so I imagine none knows the other comments in advance.

gipi wrote: ↑

Sun Jan 26, 2020 6:47 pm

What do you mean by general aspect?