How to Assess

The Discovery and Linkage Program Assessor Handbook provides Instructions for Detailed and General Assessors, for FL19, FT19, IC19, IH19 and DE20 – PDF format (1MB) – Word format (172KB)  

The Assessor Handbook (Version 5.0) provides Instructions for Detailed and General Assessors, for ARC Centres of Excellence commencing in 2020 (EI20 and CE20) – PDF format (318KB) – Word format (167KB)  

The Assessor Handbook (Version 4.0) provides Instructions for Detailed and General Assessors, for LP18, SR18, DE19, DP19, IN19 and LE19 – PDF format  (1 MB) – Word format  (173KB) 

Back to top

How to use RMS

RMS Handbook for Assessors – PDF format (2MB) – Word format (2MB) 

Please visit the RMS page for further information.

Back to top

How do I provide a good Detailed Assessment

  • Objective constructive and professional comments
  • Detailed comments (one or two sentences are not sufficient)
  • Sufficient information to allow Applicants to provide a rejoinder to your comments
  • Comments that match ratings—for example, if you have given significant criticisms an ‘A’ rating is unlikely to be appropriate. Further, if a ‘D’ rating is given, criticisms and comments justifying the rating should be given. Assessors are encouraged to identify specific aspects of Applications that warrant the rating given, rather than provide broad general comments.
  • Ensure that the rating given matches the description in the rating scale of the particular scheme.
  • Observe conflict of interest rules and declare anything you are concerned about to the ARC.

Example 1—Good Assessment for a strong Application

Investigators (A)

The Chief Investigators comprise an outstanding team of researchers with complementary skill sets, and extensive experience in researching the four country case studies at the heart of this Applicaiton. I also note that the Project Leader has maximised her research-only position to produce high quality research outputs, including a major review of aid effectiveness commissioned by UNESCO. One CI has had two career interruptions for the birth of her two children, and has published high quality articles in The International Journal of Demography and Population. A third CI has undertaken two secondments to the Asian Development Bank to advise the Bank on best practice for the delivery of aid on the ground.

Project Quality and Innovation (A)

The project plan outlines a highly ambitious study of the role of international aid agencies in reducing poverty in four similar internal regions in the four different countries. It will utilise a cutting edge Mixed Methods approach to triangulate data from the 2014 global survey on aid effectiveness, with data mining of relevant documentation, plus ethnographic studies of donors, managers and recipients. The plan is innovative and highly promising and should generate exciting new data and insights into those case studies with generalisable results.

Feasibility (B)
This important project may have underestimated the time required to undertake the fieldwork, especially as it’s not clear from the Application whether the preliminary work necessary to undertake the research such as liaising with regional governors has been undertaken. Such access may be problematic, and lack of access may delay or seriously compromise the project.

Benefit (A)
This is an important Application with an innovative approach that should engender new public policies leading to improved outcomes. It is also value for money.

Example 2—Good Assessment for a weak Application

Investigators (D)

The lead CI has a significant number of publications, particularly in recent years. However the quality of these publications is not clear (information needed in F15.4). There is little/no evidence of previous track record in the specific research area of the Application (i.e. construction worker heat stress/comfort). This is supported by the fact that no publications have been highlighted as being relevant to this Application. These issues are also relevant to CI two. More than 10 publications are listed in F15.3 for the lead CI. It is not clear what this CI’s most significant contributions are from the lone sentence given in F15.5. Neither CI appear to have had research funding in the past from the information provided in the Application and therefore are a high risk as there is no proven track record of managing and completing projects of this scale. Despite the lead CI only obtaining their PhD in recent years, they have worked as a full time researcher for close to 10 years and the number of publications is fairly limited. The second CI does have some relevant experience; however this has not been clearly described/highlighted. The significance and impact of the second CI’s research is not described (F15.4). There is also no record of collaboration between the two CIs.

Project Quality and Innovation (E)

The project addresses an important issue, however it is considerably underdeveloped and amazing that the Institution allowed it to be submitted in its current state. This reflects poorly not only on the CIs but also on the Institution. The summary contains several spelling errors and this is reflective of the general poor state of this Application. Also, the A6 summary duplicates that in A5, which (as stated in the instructions) is not allowed. The Application is lacking reference to previous work that has looked at the issue. It is hard to believe, being such an important and industry-wide issue, that no-one has considered solutions to this in the past. Figure 2-7 provide the impression that the garment has already been designed. It is not clear if these diagrams are based on previous work of the CIs or the work of others. How is this project developing something that is innovative and new if the garment already exists? Figure 3-7 also lack a detailed description. What do these show? Section 2.5 of the Application indicates that two PhD students will work on the project. It is not clear how the second student is being funded as only funding for one stipend has been requested. How do reference 12 and 13 of the Application provide data that is relevant to the research proposed? The timeline appears to be lacking thought - surely not all tasks will take exactly 6 months to complete? There are many spelling and grammatical errors as well as repetition throughout the Application that should have been addressed prior to submission.

Feasibility (E)

There is a severe lack of detail describing the benefits that will stem from the research. What are the priority goals addressed? What journals will be targeted? Which conferences? What is the audience of the web-site? Other avenues of dissemination may be more appropriate, such as industry seminars/training.

Benefit (D)

This is little evidence of ‘value for money’ in this Application and the medium and longer term outcomes are not clear. For example, how will it provide new benefits that are not already targeted by existing work in this field?

Improvements

Budget details and justifications are severely lacking in detail. In the justification you must explain why the funding is essential for the project and what it will be used for (e.g. $10,000 for the PhD students), not just repeat the information provided in the budget. There is no indication of the value of in-kind/cash support to be provided from the University. What is the specific topic that the PhD student would work on? The section on role of personnel should describe the roles and responsibilities of the CIs and student, not just be a summary of their experience.

Back to top

What not to do in an assessment

Remember that while your assessment is anonymous to the applicant it will be seen by a panel of your peers, the General Assessors. In order to ensure you provide the most helpful information in your assessment it is important that you do not:

  • Include your ratings in the text
  • Write assessments that are so brief as to be unhelpful
  • Write assessments that just quote the wording from the rating scale.
  • Identify yourself, either directly or indirectly, or refer to other researchers or Applications in a way that can identify them
  • Include text that appears to be defamatory,  distastefully irrelevant (such as gratuitous criticism of a researcher) or discriminatory against any particular group
  • Include comments about the potential ineligibility of a Application or of research integrity issues. This information should be provided to the ARC by email, as eligibility or research integrity considerations are kept strictly separate from the assessment process.
  • Do not just quote the rubric or restate the metrics provided by the applicant – give an assessment of the projects quality and innovative value.

What not to do—Example 1

Investigator(s)

‘Good relevant experience’

 Project Quality and Innovation

‘Not at the forefront but solid’

 Feasibility and Benefit

‘Generally OK’

Benefit

'Unclear'

What not to do—Example 2

Investigator(s)

“I have serious doubts about the lead CI’s ability to deliver this project. I was unfortunate to collaborate with him on a number of publications in recent years, and partnered on an ill-fated DP Application last year which fell apart due to his overbearing and tendentious manner. If this is how he behaves towards other researchers it is unlikely he will be able to bring this project, which requires extensive collaboration across institutions, to fruition.”

Back to top

Conflicts of Interest

ARC Conflict of Interest and Confidentiality Policy.

Timeframes associated for different types of Conflicts of Interest for ARC assessment purposes are outlined in Identifying and Handling a Conflict of Interest in NCGP page.

Back to top

Policies, Codes and Guidelines

Policies

Codes and Guidelines

Back to top

Scheme Specific Information

Key documents relating to the Application submission and selection process are available below for individual schemes via the links below:

Back to top

Other Information

Back to top

Contacts

For information about joining the ARC Assessor Community or updating your contact details or availability, please contact the ARC Peer Review Team.

For information about assessing a Application, please contact the ARC Peer Review Team. 

For information about using RMS please email the RMS Helpdesk or ring +612 02 6287 6789.

Back to top

Content Last Modified: 15/11/18

Back to top