ARC Evaluation Strategy
Purpose of this Strategy
The purpose of this Strategy is to guide and strengthen ARC evaluation activity and to reinforce evaluative thinking in ARC policy and program processes.
This Strategy is designed to ensure that ARC evaluation activity is high quality, robust and consistent, and is well targeted to inform decision making and continuous improvement in the ARC’s development and implementation of policies and programs, particularly in its administration of the National Competitive Grants Program (NCGP). It also aims to ensure that ARC evaluation activity supports the Chief Executive Officer (CEO) in meeting the responsibility under the Public Governance, Performance and Accountability Act 2013 (PGPA Act) to measure and assess the entity’s performance.
What is evaluation and why is it important?
Evaluation is the systematic and objective assessment of the appropriateness, effectiveness and efficiency of a policy or program. It is an essential part of policy and program management, and is crucial to supporting continuous improvement in the ARC’s delivery of its purpose.
Evaluation provides evidence of what has been done well and what could be done better, the extent to which objectives have been achieved, and what the impact of a policy or program has been. This evidence of performance is then used to inform ongoing decision making regarding policy and program development and implementation. As such, evaluation represents both the end and the beginning of a policy or program cycle.
To be most effective, evaluation should therefore be built into program design and undertaken as an integral component of program management. It should not be treated as an ad hoc activity, but as part of the committed focus across the ARC on strengthening evaluative thinking—taking a disciplined approach to using good evidence to help make sound judgments and informed decisions. In this respect, all ARC staff members have a role to play in contributing to evaluative thinking in their work
Evaluation in the ARC
The nature of ARC business
The administration of research funding under the NCGP is a major part of the ARC’s business, with research grants themselves representing a significant financial investment by government. In addition, the ARC’s business includes developing and implementing policy related to the administration of the NCGP and to Australia’s overall research effort, as well as providing research grants services for other agencies. Evaluating performance in these areas is important for achieving the best outcomes for government, the ARC, the university research sector, industry and other research partners, research end users and the public.
Monitoring and review processes embedded in ARC policy and program management help to measure performance. This involves collecting and analysing a wide variety of data and consulting with stakeholders (for example, funded researchers and university administrators)—both on an ongoing basis and regularly during funding scheme cycles—to inform program and policy changes and initiatives. While these review processes do not constitute evaluation as such, they do contribute to evaluative thinking in the ARC by supporting evidence-based decision making and continuous improvement. The business intelligence produced is also an essential input to evaluation activity. Evaluation and review should therefore be treated as complementary, but distinct, elements of ARC business.
External stakeholders are critical to ARC business as the recipients of grants, and also play a fundamental role in many program and policy processes, for example, in conducting peer review and assessment of applications. It follows that stakeholder feedback will usually be an indispensable component of evaluation activity. Opportunities to seek and incorporate relevant stakeholder input for evaluation purposes should be welcomed, including through regular program or policy consultations.
Grant funding is administered under a number of schemes with varying objectives, timeframes and potential policy implications. Some aspects of program management, such as grant application and assessment processes, vary between schemes. Conversely, there are thematic issues or policy objectives that may apply to multiple or all schemes, such as supporting Indigenous researchers or encouraging industry collaboration.
When planning and undertaking evaluation of ARC policy and program activity, a range of evaluation questions, topics and timeframes may therefore need to be considered. Evaluations need to be appropriately prioritised and targeted (as part of the Strategic Evaluation Plan, described below) to produce findings that can meaningfully inform decision making in policy and program processes.
Legislation and guidelines
The requirement for performance measurement and assessment is underscored in a number of key legislative and other documents that apply to the administration of ARC business. Effective evaluation plays an important role in helping to meet this requirement, and is given a particular emphasis in relation to grants administration.
The Australian Research Council Act 2001 (ARC Act) outlines the functions of the ARC, which include making recommendations to the Minister on research funding, administering research funding and providing advice on research matters.
The PGPA Act establishes requirements for all Commonwealth entities (including the ARC) to meet high standards of governance, performance and accountability, and to provide meaningful information to the Parliament and the public. In accordance with the CEO’s responsibilities under the PGPA Act, the ARC Corporate Plan outlines performance measures for the ARC’s functions, and the ARC Annual Report provides information on the ARC’s performance against those measures. Evaluation serves as a valuable input to inform and contribute to these processes.
The Commonwealth Grants Rules and Guidelines 2017 (CGRGs) establish the overarching Commonwealth grants policy framework and outline expectations and requirements for Commonwealth entities (including the ARC) in relation to all processes involved in grants administration, including evaluation. In particular, the CGRGs outline:
- the importance of establishing performance and evaluation measures as part of robust planning and design of grants
- the need for an outcomes orientation in grants administration, including clearly specified outcome, output and input measures to ensure effective and efficient evaluation of grants administration
- the importance of evaluating a grant—including consideration of the continued appropriateness of government outcomes and entity strategic directions as a result of the grant’s impact—before initiating further grant opportunities
- the role of grantees in participating in evaluation processes.
Additional guidance on evaluation in the context of grants administration, which informs the ARC’s approach, is provided by the Department of Finance in Resource Management Guide 412: Australian Government Grants – Briefing, Reporting, Evaluating and Election Commitments.
Scope—what we evaluate
The scope of this Strategy includes the evaluation of ARC policy and program performance in relation to the functions of administering the NCGP and providing advice on research matters, as outlined in the ARC Act, as well as the provision of grants services for other agencies.
Examples of evaluation topics may include:
- effectiveness in achieving individual funding scheme objectives and delivering intended outputs, outcomes and benefits
- effectiveness of ARC programs and policies in addressing specific thematic issues or whole-of-government priorities
- alignment of the NCGP with other research funding programs
- efficiency of funding scheme administration
- appropriateness and effectiveness of ARC policies on research matters.
The ARC’s activity in assessing the quality, engagement and impact of research (through the Excellence in Research for Australia and Engagement and Impact assessment exercises) is outside the scope of this Strategy, but may be included when the Strategy is reviewed in the future.
ARC Evaluation Principles
To ensure that evaluations are meaningful and useful for decision makers responsible for ARC policies and programs, all ARC evaluation activity should be informed by, and conducted in accordance with, the following principles.
Evaluation should be built into policy and program processes. Evaluation planning should be completed early in the program or policy design phase to help establish defined, measurable outcomes. The timing of evaluation activity and reporting should be aligned with timeframes for program and policy implementation to maximise efficiency and quality in data collection and ensure the timeliness and usefulness of findings.
2.Good planning and management
Evaluations should be undertaken in accordance with good project management practices. They should be appropriately resourced and given sufficient time to produce meaningful findings. Governance processes should be in place to ensure appropriate approval and oversight of evaluation activity, and to reinforce the commitment of ARC senior executives to evaluation.
3. Fitness for purpose
The scale, resources and methodology applied to an evaluation should be appropriate and proportionate to the policy or program being evaluated. Considerations include the strategic importance, size, lifecycle, anticipated outcomes, impact, and risk of the policy or program, as well as the availability of data and the need to ensure the evaluation represents value for money.
Evaluations should be high quality and evidence based. Robust, well justified and documented research and analytical methods should be applied in all evaluation activity, including both quantitative and qualitative approaches. All stakeholder engagement forming part of the methodology should be targeted and timely. Reliable, relevant data should be used meaningfully to inform evaluation findings.
5. Objectivity and independence
Evaluation should be undertaken with the aim of providing an objective assessment to inform policy and program decision making—it is not a promotional exercise. It should involve and be guided by internal and external stakeholders, including program managers, but should be conducted by evaluators who are independent of the areas responsible for the policy or program being evaluated.
The final reports of all evaluations should be published internally and made available to all ARC staff, unless there are strong reasons to limit their distribution. Whenever possible, evaluation reports should also be published externally to demonstrate the ARC’s commitment to accountability and to strengthen the confidence of stakeholders and the public in ARC policies and programs.
Building evaluation into the policy and program processes due for assessment is essential to ensure consistency with the ARC Evaluation Principles. In particular, embedding evaluation planning into the early stages of program and policy design facilitates high quality and efficient evaluation conduct, and supports a focus on achieving meaningful outcomes from every evaluation.
A key component of good evaluation planning is the use of a logic model to anchor the implementation of the program or policy to its objectives and intended outcomes, and to provide a basis to formulate the questions that the evaluation needs to address.
Integrating evaluation planning into policies and programs also helps to identify the data and information that needs to be gathered—potentially as part of the policy or program implementation process—and analysed to generate meaningful findings that will inform the development of further initiatives or changes to future program rounds.
As represented in Figure 1, integrating evaluation and program or policy lifecycles in this way helps to support continuous improvement.
Figure 1: Integrating evaluation into the program/policy lifecycle
It is important to note that the evaluation lifecycle does not always align with the program or policy lifecycle in the manner depicted in Figure 1. For example, an evaluation may be conducted on only the design and planning, or the implementation, of a program or policy. Every evaluation needs to be planned and tailored to address its particular topic effectively.
In the context of the ARC’s administration of grants, the program lifecycle may refer to a number of different processes, depending on the scope and topic of a particular evaluation, for example:
- the application and assessment process of a funding scheme, up to the point of funding being awarded (which may involve a timeframe of only a few months)
- progress or changes in a funding scheme’s implementation, up to the point of regular mid-term review processes
- the outcomes of a complete funding scheme round, up to the point of receiving and analysing final reports from all completed research projects (which may involve a timeframe of several years).
For evaluations of ARC performance in relation to broader thematic issues or whole-of-government priorities that cut across multiple funding schemes, integrating evaluation into different program cycles may be particularly complex. Flexibility is required to manage such evaluations effectively.
What is important is to ensure all evaluations are well planned and targeted to inform evidence-based decision making, and that they aim to contribute to improved evaluative thinking in the ARC.
Strategic Evaluation Plan
To embed an integrated approach to evaluation in the ARC, a Strategic Evaluation Plan is prepared annually, which identifies both short and longer term evaluation priorities and provides guidance on evaluation practice to ensure the timeliness and usefulness of evaluation activity. The ARC Leadership Group, chaired by the CEO, is responsible for approval of the Plan to ensure activity is best targeted to support the CEO in measuring and assessing performance, as required under the PGPA Act.
The Plan outlines a program of evaluation activity over a 4 year period. It is reviewed annually—in consultation with a broad range of ARC internal stakeholders—to maintain its responsiveness to ARC policy and program needs, including emerging priorities in the administration of grants, as well as broader government policy and research sector imperatives. Identification and prioritisation of evaluation activity in the Plan also takes into account potential overlaps with planned audit activity and major reviews to ensure that evaluation topics are well defined and avoid duplication, and resources are directed appropriately.
Where practicable, evaluations are coordinated with relevant policy and program management timeframes, including relevant funding scheme timeframes, to maximise the effectiveness and efficiency of planning, data collection and analysis. In all cases, the timeframes for delivery of evaluation findings are targeted to achieve the greatest impact for decision makers.
The program of planned activity—including evaluations conducted internally and by external providers—must be achievable within ARC resource and budget constraints.
The Plan is intended to be an internal document available to all ARC staff.
Implementation and governance
Implementation of this Strategy—including governance, strategic planning and conducting evaluations—requires involvement and commitment across the ARC in order to strengthen our evaluative thinking and achieve the best outcomes from evaluation activity. Specific roles and responsibilities are outlined below.
Program Evaluation Section (within Research Excellence Branch)
- Implementation and review of this Strategy, including consultation across the ARC
- Development and review of the Strategic Evaluation Plan, including consultation across the ARC
- Development of plans for individual evaluations, in consultation with line areas
- Project management of evaluations, and conducting internal evaluations—including contract management (where relevant), data collection, monitoring, stakeholder engagement, evaluation analysis, preparing final reports and communicating findings—in consultation with line areas
- Providing advice on evaluation priorities and processes, including advice and support to embed evaluation in policy and program practice
- Providing updates to the ARC Leadership Group (ALG) on evaluation planning, progress and reporting.
Research Excellence Branch
- Responsibility for program and policy evaluation in the ARC (the Research Excellence Branch is independent of grant program and NCGP policy functions within the ARC organisational structure).
- Branch Manager, Research Excellence Branch, provides day-to-day oversight of the Program Evaluation Section and is the delegate for financial approvals.
Corporate Services Branch
- Providing advice on expenditure for contracted evaluations
- Assisting with the promotion of evaluative thinking within the ARC, including training opportunities for staff.
- Internal and external communication regarding evaluation in the ARC, including publishing this Strategy, Strategic Evaluation Plan and finalised evaluation reports
- Assisting with external stakeholder engagement
Programs Branch, Policy and Strategy Branch and Grants Services Section (within Corporate Services Branch)
- Input to development and review of this Strategy
- Input to development and review of the Strategic Evaluation Plan, including advice on:
- program and policy priorities and timeframes
- coordination with regular program and policy monitoring and review processes
- alignment or overlap with planned audit activity (to avoid duplication)
- Working with Program Evaluation Section on the development of plans for individual evaluations
- Incorporating evaluative thinking into policy and program planning and implementation (for example, in monitoring and review processes)
- Supporting Program Evaluation Section in managing evaluations, including data collection and extraction, monitoring, and stakeholder engagement
- Using evaluation findings to inform decision making on policy and programs.
ICT Services Branch
- Supporting Program Evaluation Section (in consultation with Policy and Strategy Branch) in data collection and extraction
- Providing advice and assistance in the use of systems and the development of evaluation data reporting tools.
Program, Strategy and Executive Committee
- Providing advice on this Strategy
- Providing advice on Strategic Evaluation Plan
- Providing advice on individual evaluation plans, findings and reports.
ARC Leadership Group (chaired by CEO)
- Setting overall direction and objectives of evaluation in the ARC
- Approval of this Strategy
- Approval of the Strategic Evaluation Plan
- Approval of annual budget allocations for evaluation activity
- Reviewing and noting plans for individual evaluations
- Monitoring progress of individual evaluations
- Endorsement of final evaluation reports
- Providing leadership in implementing actions and changes that respond to evaluation findings and recommendations
- Providing leadership in promoting evaluative thinking in the ARC.
Review of this Strategy
This Strategy will be reviewed every two years, in consultation with relevant ARC internal stakeholders, for approval by the ARC Leadership Group. The next review date is the second half of 2022.
In addition, the design and implementation of this Strategy and the Strategic Evaluation Plan will themselves be evaluated in 2024 (five years after their introduction) to ensure their effectiveness and appropriateness and to inform improvements in ongoing ARC evaluation activity.