Section Navigation
General Information
Breadcrumb Navigation
Peer Review Processes
Introduction
Principles
Peer Review Roles
Peer Review Process
Introduction
The ARC funds research and researchers under the National Competitive Grants Program (NCGP). The NCGP consists of two elements – Discovery and Linkage. Within these elements are a range of schemes structured to provide a pathway of incentives for researchers to build the scope and scale of their work and collaborative partnerships. The majority of funding decisions under the NCGP are made on the basis of peer review.
Principles
The ARC’s peer review processes involve:
- peer review by experts who assess individual research proposals within their field of research or across a broader disciplinary area on the basis of established selection criteria
- processes under which arrangements for assessment are clearly articulated in published documentation
- robust conflicts of interest processes to ensure conflicts are managed and transparent
- the receipt of proposals in confidence, except where required to be released under law, for example under the Freedom of Information Act 1982
- the provision of a right of reply to assessments (in most ARC schemes).
Peer Review Roles
Overview
Depending on the scheme, peer review of proposals may be undertaken by: members of the ARC College of Experts (CoE); Ozreaders; Intreaders and/or other eminent researchers. A description of assessor roles is provided in section 3.2 below.
The ARC CoE members are currently divided among five panels: Biological Sciences and Biotechnology (BSB), Engineering, Mathematics and Informatics (EMI), Humanities and Creative Arts (HCA), Physics, Chemistry and Earth Sciences (PCE), and Social Behavioural and Economic Sciences (SBE).
Under larger schemes, such as Discovery Projects and Linkage Projects, all five panels of the College of Experts convene for selection meetings. In other schemes, such as Australian Laureate Fellowships and Linkage Infrastructure, Equipment and Facilities schemes, a single interdisciplinary Selection Advisory Committee (SAC). SAC members may be ARC College of Expert members and/or other eminent researchers.
Description of key roles
Characteristic |
Description |
|---|---|
College of Experts |
|
| Description | The ARC College of Experts consists of 78 members who are experts of international standing drawn from the Australian research community, including higher education, industry and public sector research organisations. |
| Role | Members of the ARC College of Experts:
|
| Workload | Each year members of the College of Experts submit assessments to the ARC, assign Intreaders to Discovery Projects proposals and participate in two Canberra-based meetings over eight days. CoE members may also be appointed to Selection Advisory Committees for various ARC schemes and carryout additional assessment and reviewer duties as well as attend additional selection meetings. |
| Remuneration | Salary |
| Selection | Members are appointed, by the Minister, for terms of between one and three years. Every year the ARC conducts a competitive nomination process to replace retiring members. |
Ozreaders |
|
| Description | Ozreaders are Australian-based readers whose research interests span the broad field/s of the research projects they are asked to assess. The ARC assessor database currently contains approximately 1000 active Ozreaders. |
| Role | Ozreaders assess proposals against selection criteria set out in scheme funding rules and are requested to score, rank and provide written comments for each proposal by submitting an electronic assessment form through the ARCs online application system. |
| Workload | Ozreaders are sent approximately 20 proposals (across different schemes) each year for assessment. |
| Remuneration | Ozreaders are provided with a small payment for their services. |
| Selection | Ozreaders are reviewed annually by ARC Executive Directors. New Ozreaders are selected through CoE member nominations and ARC consideration of new project participants (i.e. each year the ARC reviews the participants of proposals which were successful the previous year and, where suitable, adds funded researchers not already listed to the Ozreader database). |
Intreaders |
|
| Description | Intreaders are internationally acknowledged experts in the specific field of a proposed research project. Chosen for their specialist knowledge and outstanding research reputation, Intreaders are asked to assess in the context of international research and benchmarks. The ARC assessor database contains approximately 18,000 Intreaders. |
| Role | Intreaders assess proposals against selection criteria set out in scheme funding rules and are requested to score, rank and provide written comments for each proposal by submitting an electronic assessment form through the ARCs online application system. |
| Workload | Intreaders may assess up to 5 proposals per year (average of 2.5 proposals per year). |
| Remuneration | None |
| Selection | Intreaders are updated annually by CoE members. CoE members select new Intreaders during the Discovery Projects assignment process. |
Peer Review Process
Release of guidance material
Every funding round begins with the development of funding rules – the blueprint describing the eligibility criteria and accountability requirements for funding proposals. The funding rules are fundamental to the smooth administration of the funding scheme. In developing the funding rules, the ARC considers feedback from the research sector. This includes responses to the ARC’s annual consultation on funding rules and feedback from Deputy and Pro-Vice Chancellors (Research). The draft funding rules are considered by the ARC Chief Executive Officer (CEO) before final approval is sought from the Minister.
Submission of proposals
Submission
Most ARC proposals require an application form to be completed using the ARC online application system. The proposal application form and the ‘Instructions to Applicants’ documents assist researchers in electronically submitting their proposals for funding through either system.
Eligibility
To be considered for funding, proposals must meet eligibility requirements specified within ARC scheme funding rules. Proposals may only be submitted by applicants who are eligible organisations listed in the scheme funding rules. Proposals must also be completed according to format and submission requirements specified in scheme ‘Instructions to Applicants’ documents.
Most ARC scheme funding rules describe a number of situations where it is possible for applicants to submit an eligibility exemption request or an eligibility advice request for the purposes of obtaining an exemption or advice from the ARC regarding a prospective proposal. The closing time for submission of such requests is set out in the funding rules. The main types of eligibility exemption and advice requests are: requests seeking a relaxation of the qualification and/or timing requirements for a fellowship candidate and requests seeking advice on the suitability of a potential proposal that includes medical and/or dental research.
Requests not to assess
Applicants may provide written notification to the ARC naming any person or persons whom they do not wish to assess their proposal. Normally, requests must contain detailed justification and be submitted to the ARC by the closing time for the proposal. The process for submitting these requests are set out in scheme funding rules. The responsible Executive Director considers each request and the justification provided, but may choose not to give effect to any a request.
Restrictions on proposals
The ARC has implemented restrictions on re-applying applicants in the Discovery Projects scheme. A proposal may not be submitted if it is essentially the same as an unsuccessful proposal submitted in the previous round and was ranked in group E (bottom 25% of unsuccessful proposals).
Receipt and assignment of proposals
Receipt
Once received by the ARC proposals are vetted by ARC staff for eligibility. If a proposal is deemed to be potentially ineligible, the proposal is sent for consideration by the ARC eligibility committee. The relevant administering organisation is advised when a proposal is identified as potentially ineligible. The administering organisation may provide further information if required. All proposals, regardless of their eligibility status, undergo the same peer review process.
Assignment
For schemes with multiple selection panels, as soon as the closing time for proposals has passed, proposals are matched to a selection panel using the Fields of Research (FOR) codes applicants have selected within the proposal. ARC Executive Directors review the appropriateness of the proposals in each panel and may transfer proposals to different panels if required. They also identify interdisciplinary proposals that will require assessors from more than one selection panel. Once these processes are finalised, the responsible Executive Director assigns at least two CoE members to each proposal. For schemes with a single interdisciplinary panel, the responsible Executive Director assigns at least two SAC members to each proposal.
Each scheme varies in its use of assessors depending on the objectives of the scheme, the scale of funding support requested and the timing of the overall process. In most schemes, Executive Directors assign proposals to an appropriate number of Ozreaders. In Discovery Projects, CoE members also assign up to four Intreaders per proposal.
As far as possible, the ARC aims to assign assessors to proposals whose areas of expertise best matches the research described in the proposal. However there are sometimes cases where the ARC has misunderstood an assessor’s area of expertise. The ARC advises all assessors that they may decline to assess a proposal if they consider a proposal to involve research that is outside their area of expertise.
Conflict of Interest
Assessors may also decline to assess proposals if they consider it to be a conflict of interest to do so. The ARC advises all assessors that they should not assess any proposal for which they have an apparent or potential conflict of interest and provides guidance on the types of situations that should be avoided.
The ARC's Confidentiality Obligations and Conflict of Interest Guidelines are published on the ARC’s website. The guidelines are designed to ensure that all members of the ARC's CoE and SACs are aware of the confidentiality obligations associated with College activities and that conflicts of interest are identified and addressed in a rigorous and transparent way. The ARC requires CoE and SAC members to provide assurances to the ARC that they will abide by the ARC's confidentiality requirements and will disclose to it any conflicts of interest related to their official duties as members. The ARC keeps a register of the private interests of CoE members.
Assessment process
Assessment forms
CoE members, Ozreaders and Intreaders submit online assessment forms to the ARC through its online application system. They are each asked to assess the merits of a proposal against selection criteria set out in scheme funding rules. The structure of the assessment form varies from scheme to scheme, but typically assessment forms request scores and written comments for each selection criteria.
Selection criteria
The selection criteria for proposals are set out in the scheme funding rules. Selection criteria and weightings vary from scheme to scheme to fit the nature and objectives of each scheme. However, most scheme selection criteria including investigator track record and proposed program of research components.
When assessing the track records of researchers nominated in a proposal, assessors are advised that it is important to take into account the opportunities that the researchers have had to build their research profile. For example, researchers who are relatively early in their career, or who have had an interrupted career, should have these facts taken into account. Similarly, researchers whose current or previous conditions of employment have allowed them a relatively large amount of research time should have this taken into account.
Scoring
Guidance is provided to assessors on the degree of merit associated with particular scoring bands. A total weighted score, which takes into account the set criteria weightings, is automatically calculated when the assessor saves their assessment.
Recognising the subjective character of scores, the ARC provides the following advice to assessors on the degree of merit associated with specified scoring bands as a guide.
- Scoring band 91-100: Outstanding – of the highest merit, at the forefront of international research in the field. Less than 2% of proposals should receive scores in this band.
- Scoring band 86-90: Excellent – Strongly competitive at international levels. Less than 20% of proposals should receive scores in this band
- Scoring band 81-85: Very good – An interesting, sound, compelling proposal. Approximately 30% of proposals will have a score in this band. Approximately 50% of proposals will have scores above 80
- Scoring band 76-80: Good – A sound research proposal, but lacks a compelling element. Approximately 30% of proposals are likely to fall into this band
- Scoring band 71-75: Fair – The proposal has potential, but requires development to be supportable. Up to 20% of proposals are likely to have a score below 75.
- Scoring band 0-70: Flawed – the proposal has one or more flaws.
Rejoinder
In most ARC schemes, once the initial assessments by assessors are completed and submitted to the ARC, researchers are invited to respond to the written comments from the assessors via a rejoinder process. Scores and assessor details are not released to applicants. Rejoinders allow applicants and nominated participants to clarify any misunderstanding or difference of opinion about perceived weaknesses in the proposal. It also provides an important feedback mechanism that may help researchers in future proposals. The assessors' reports and the rejoinders also assist the ARC CoE members in reviewing proposals.
Selection Meetings
In the Discovery Projects and Linkage Projects schemes, following the rejoinder process, CoE members attend selection meetings to rank the proposals and determine to what extent the competitive proposals ought to be funded. A recommendation for funding is made to the ARC CEO.
There are two main CoE meetings each year:
- In April a small meeting is held to consider proposals submitted to the second round of the Linkage Projects schemeand assign Intreaders to Discovery Projects proposals
- In August a larger meeting is held to consider proposals submitted in the annual round of Discovery Projects and the first round of Linkage Projects.
SAC meetings for other ARC schemes are also held periodically throughout the year, depending on the timelines of particular schemes. They may include some members of the CoE.
Ranking of proposals
For most ARC schemes, a Weighted Average Percent Rank (WAPR) formula is used to combine CoE, Ozreader and Intreader assessments submitted for each proposal. This provides a single value for each proposal, ranging from 0 to 100, allowing proposals to be sorted into an initial ranked list for each selection panel.
To determine a final ranked list, the selection panel considers and discusses proposals – particularly those around the funding margin. Selection panel members may revise the ranking of a proposal in response to a number of factors including: assessor reports; persuasive rejoinder arguments; and panel discussions comparing the relative merits of proposals against the selection criteria. Selection panel members also take into account any WAPR anomalies.
Assignment of budgets
Once a CoE selection panel has determined the final ranked list of proposals, members determine funding recommendations for all proposals in the fundable range. Selection panel members consider in detail each proposal budget request and recommend an appropriate level of funding. Selection panel members may recommend less than requested funding in cases where the proposal budget is considered to be inflated, is inadequately justified or includes items that are prohibited. Each scheme’s funding rules set out what type of funding may be requested and any budget item restrictions.
Awards outcomes
ARC CEO’s recommendation
On advice from the CoE or SAC, the ARC CEO makes funding recommendations to the Minister. Approval of proposal expenditure rests with the Minister. Generally, the Minister acts on advice from the ARC CEO; however, if the Minister does not approve a proposal, a public explanation for that decision will be provided. An approval process is in place to enable the Minister to meet this undertaking.
Notification
Once the Minister has approved and announced the funding recommendations, the ARC may notify organisations of the individual outcomes of their proposals. On its website, the ARC publishes a selection report, providing an analysis of funding outcomes for the scheme round, and listings of successful proposals. Emails and hard copy packages are sent to the relevant research offices to provide a summary of each organisation’s individual proposal outcomes.
Hard copy packages also contain individual notification letters for distribution to the first-named researcher of each proposal. Researchers are notified whether their proposal was successful, unsuccessful, placed on a reserve list or deemed to be ineligible.
Feedback to applicants
For ineligible proposals, researchers are advised of the reason(s) a proposal was deemed to be ineligible. In the case of unsuccessful proposals, feedback outlining the reason(s) a proposal was considered uncompetitive may be included. The type of feedback provided varies from scheme to scheme.
Funding agreements
Successful administering organisations are sent a Funding Agreement, outlining the terms and conditions by which any successful proposals must be administered. Funding for successful proposals (now projects) commences as soon as two copies of the Funding Agreement have been signed by the appropriate delegate of the administering organisation and counter-signed by an ARC Executive Director.
Appeals process
The funding rules for the various ARC funding schemes make provision for an appeals process. Appeals will be considered only against process issues and not against committee decisions or assessor ratings and comments. Appeals must be made on the appeals form available from the ARC's website. The form must be lodged through the relevant administering organisation's research office and be received, within 28 days of the date on the letter notifying the outcomes of proposals.
The ARC appoints an Appeals Committee to hear appeals and make recommendations to the ARC's CEO. In general the functions of the Appeals Committee are to:
- consider valid appeals submitted to the ARC to determine whether there has been any error in the administrative process relating to the selection process
- determine whether any such errors led to a defect in decision-making by the ARC and/or the Minister which adversely affected the proposal
- recommend to the ARC whether to uphold or dismiss an appeal
- provide advice to the ARC in relation to how its administrative processes could be modified or improved.
