2019 Marie Curie Individual Fellowship (H2020-MSCA-IF-2019)

[ENV2021]

Hi there,
a friend of mine is an MC evaluator and I would like to share with you what he explained to me about the evaluation process.
Three evaluators per proposal plus a chair that supervises the process. Each evaluator may evaluate a few proposals (10-15).
Each evaluator produces a report individually without sharing his comments; only when the three evaluators have submitted their reports it starts a consensus phase when the evaluators discuss their comments about the proposal. When they reach a consensus, and only then, they propose the score, always under the supervision of the chair. They are not allowed to see the evaluation report of the previous submitted proposal (if any) nor to know the old score, actually, this report remains undisclosed; the chair may ask the evaluators to rediscuss specific points if he considers it necessary.
Evaluators have to follow a specific point list and that is why it is so important to fill out the proposal including each and every point specified in the guideline, being specific, concrete. And you have to say the right things where they are expected to be in the proposal, not elsewhere in the text. If you state something in the "excellence" part that should have been stated in the "impact" part, well...that is your fault.
Regarding the CV, effectively the number of papers doesn't care, as it said in the guideline you may be penalised only if according to your research career you have not produced enough; evaluators check your CV to have an idea of what you have done before, especially to determine if your training programme is appropriate and if you effectively are going to learn something new which is the most important thing in this fellowship, besides project excellence. He also told me that many of the projects he reviewed were excellent but even in this case if you fail addressing all and each point they have to start to decrease the score, and few tenths of a point means getting or not the fellowship or even getting a bad score.
Said that..of course, subjectivity is impossible to remove but at least I think this is the most well-designed evaluation process I know.

[Kenniz]

Hi there,
a friend of mine is an MC evaluator and I would like to share with you what he explained to me about the evaluation process.
Three evaluators per proposal plus a chair that supervises the process. Each evaluator may evaluate a few proposals (10-15).
Each evaluator produces a report individually without sharing his comments; only when the three evaluators have submitted their reports it starts a consensus phase when the evaluators discuss their comments about the proposal. When they reach a consensus, and only then, they propose the score, always under the supervision of the chair. They are not allowed to see the evaluation report of the previous submitted proposal (if any) nor to know the old score, actually, this report remains undisclosed; the chair may ask the evaluators to rediscuss specific points if he considers it necessary.
Evaluators have to follow a specific point list and that is why it is so important to fill out the proposal including each and every point specified in the guideline, being specific, concrete. And you have to say the right things where they are expected to be in the proposal, not elsewhere in the text. If you state something in the "excellence" part that should have been stated in the "impact" part, well...that is your fault.
Regarding the CV, effectively the number of papers doesn't care, as it said in the guideline you may be penalised only if according to your research career you have not produced enough; evaluators check your CV to have an idea of what you have done before, especially to determine if your training programme is appropriate and if you effectively are going to learn something new which is the most important thing in this fellowship, besides project excellence. He also told me that many of the projects he reviewed were excellent but even in this case if you fail addressing all and each point they have to start to decrease the score, and few tenths of a point means getting or not the fellowship or even getting a bad score.
Said that..of course, subjectivity is impossible to remove but at least I think this is the most well-designed evaluation process I know.

thanks for sharing this info.
my question would be: do they get all the reviews at the same time or after another? because getting them after another would heavily impair the results in my opinion. this is, because you may rate the first one better than it should be before reading the others and vice versa, you get what i mean.

[SimpaLif]

Percentiles MSCA IF 2018:

https://ec.europa.eu/research/participa ... les_en.pdf

[megasphaera]

Hi there,
a friend of mine is an MC evaluator and I would like to share with you what he explained to me about the evaluation process.
Three evaluators per proposal plus a chair that supervises the process. Each evaluator may evaluate a few proposals (10-15).
Each evaluator produces a report individually without sharing his comments; only when the three evaluators have submitted their reports it starts a consensus phase when the evaluators discuss their comments about the proposal. When they reach a consensus, and only then, they propose the score, always under the supervision of the chair. They are not allowed to see the evaluation report of the previous submitted proposal (if any) nor to know the old score, actually, this report remains undisclosed; the chair may ask the evaluators to rediscuss specific points if he considers it necessary.
Evaluators have to follow a specific point list and that is why it is so important to fill out the proposal including each and every point specified in the guideline, being specific, concrete. And you have to say the right things where they are expected to be in the proposal, not elsewhere in the text. If you state something in the "excellence" part that should have been stated in the "impact" part, well...that is your fault.
Regarding the CV, effectively the number of papers doesn't care, as it said in the guideline you may be penalised only if according to your research career you have not produced enough; evaluators check your CV to have an idea of what you have done before, especially to determine if your training programme is appropriate and if you effectively are going to learn something new which is the most important thing in this fellowship, besides project excellence. He also told me that many of the projects he reviewed were excellent but even in this case if you fail addressing all and each point they have to start to decrease the score, and few tenths of a point means getting or not the fellowship or even getting a bad score.
Said that..of course, subjectivity is impossible to remove but at least I think this is the most well-designed evaluation process I know.

Wow I thought each reviewer will look only at 2-3 proposals. 10 to 15 is a lot.

That is a different version of the story. A friend of mine which is also a msca if reviewer told me that usually in the consensus phase the reviewer that have similar opinon try to convince the third one to change scores. Let's say two of them wants to give high scores and a third one is giving low scores: the two reviewers will spend all the time they have to convince the third one to increase scores.

[megasphaera]

Hi there,
a friend of mine is an MC evaluator and I would like to share with you what he explained to me about the evaluation process.
Three evaluators per proposal plus a chair that supervises the process. Each evaluator may evaluate a few proposals (10-15).
Each evaluator produces a report individually without sharing his comments; only when the three evaluators have submitted their reports it starts a consensus phase when the evaluators discuss their comments about the proposal. When they reach a consensus, and only then, they propose the score, always under the supervision of the chair. They are not allowed to see the evaluation report of the previous submitted proposal (if any) nor to know the old score, actually, this report remains undisclosed; the chair may ask the evaluators to rediscuss specific points if he considers it necessary.
Evaluators have to follow a specific point list and that is why it is so important to fill out the proposal including each and every point specified in the guideline, being specific, concrete. And you have to say the right things where they are expected to be in the proposal, not elsewhere in the text. If you state something in the "excellence" part that should have been stated in the "impact" part, well...that is your fault.
Regarding the CV, effectively the number of papers doesn't care, as it said in the guideline you may be penalised only if according to your research career you have not produced enough; evaluators check your CV to have an idea of what you have done before, especially to determine if your training programme is appropriate and if you effectively are going to learn something new which is the most important thing in this fellowship, besides project excellence. He also told me that many of the projects he reviewed were excellent but even in this case if you fail addressing all and each point they have to start to decrease the score, and few tenths of a point means getting or not the fellowship or even getting a bad score.
Said that..of course, subjectivity is impossible to remove but at least I think this is the most well-designed evaluation process I know.

thanks for sharing this info.
my question would be: do they get all the reviews at the same time or after another? because getting them after another would heavily impair the results in my opinion. this is, because you may rate the first one better than it should be before reading the others and vice versa, you get what i mean.

I think they get them at the same time because they have a very specific deadline to send results back.

[ENV2021]

Well...I don't think the story is different...during the consensus phase they try to reach a... consensus, I think it is normal that everyone evaluates each point based on their expertise and even on the other proposals they have to evaluate (which are not the same for the three of them)...each evaluator defends his reasons and it doesn't mean that if you think you are right you cannot stand for it or being convinced that you are misjudging...I think it is normal and fair. Has someone said before he told me that the evaluators have to provide their report before a specific date, so I imagine none knows the other comments in advance.

[SimpaLif]

Well...I don't think the story is different...during the consensus phase they try to reach a... consensus, I think it is normal that everyone evaluates each point based on their expertise and even on the other proposals they have to evaluate (which are not the same for the three of them)...each evaluator defends his reasons and it doesn't mean that if you think you are right you cannot stand for it or being convinced that you are misjudging...I think it is normal and fair. Has someone said before he told me that the evaluators have to provide their report before a specific date, so I imagine none knows the other comments in advance.

There must be some tuning after all. What if do they have many applicants with high scores?

[MSCA_CHEM_2019]

I think that although there is an evaluation manual, each evaluator will evaluate according to his/her criteria, so the evaluation is very subjective. This is reflected in the variations of scores in the re submission. In addition, I think it has been commented in this post or last year (I don't remember well), that proposals with sections of impact and implementation with general aspects but with a solid scientific proposal get the scholarship.

[gipi]

What do you mean by general aspect?

[MSCA_CHEM_2019]

What do you mean by general aspect?

I think people were talking about less specific comments.