Purpose of this Strategy

The purpose of this Strategy is to guide and strengthen ARC evaluation activity and to reinforce evaluative thinking in ARC policy and program processes. 

This Strategy is designed to ensure that ARC evaluation activity is high quality, robust and consistent, and is well targeted to inform decision making and continuous improvement in the ARC’s development and implementation of policies and programs, particularly in its administration of the National Competitive Grants Program (NCGP). It also aims to ensure that ARC evaluation activity supports the Chief Executive Officer (CEO) in meeting the responsibility under the Public Governance, Performance and Accountability Act 2013 (PGPA Act) to measure and assess the entity’s performance.

What is evaluation and why is it important?

Evaluation is the systematic and objective assessment of the appropriateness, effectiveness and efficiency of a policy or program. It is an essential part of policy and program management, and is crucial to support continuous improvement in the ARC’s delivery of its mission.

Evaluation provides evidence of what has been done well and what could be done better, the extent to which objectives have been achieved, and what the impact of a policy or program has been. This evidence of performance is then used to inform ongoing decision making regarding policy and program development and implementation. As such, evaluation can represent both the end and the beginning of a policy or program cycle.
 
To be most effective, evaluation should therefore be built into program design and undertaken as an integral component of program management. It should not be treated as an ad hoc activity, but as part of the committed focus across the ARC on strengthening evaluative thinking—taking a disciplined approach to using good evidence to help make sound judgments and informed decisions.

Evaluation in the ARC

The nature of ARC business 

The administration of research funding under the NCGP is a major part of the ARC’s business, with research grants themselves representing a significant financial investment by government. In addition, the ARC’s business includes developing and implementing policy related to administration of the NCGP and to Australia’s overall research effort. Evaluating performance in these areas is important to help achieve the best outcomes for government, the ARC, the university research sector, industry and other research partners, research end users and the public.

Monitoring and review processes that are embedded in ARC policy and program management help to measure performance. This involves collecting and analysing a wide variety of data and consulting with stakeholders (for example, funded researchers and university administrators)—both on an ongoing basis and regularly during funding scheme cycles—to inform program and policy changes and initiatives. While these review processes do not constitute evaluation as such, they do contribute to evaluative thinking in the ARC by supporting evidence based decision making and continuous improvement. The business intelligence produced is also an essential input to evaluation activity. Evaluation and review should therefore be treated as complementary, but distinct, elements of ARC business. 

External stakeholders are critical to ARC business as the recipients of NCGP support, and also play a fundamental role in many program and policy processes, for example, in conducting peer review and assessment of research proposals. It follows that stakeholder feedback will usually be an indispensable component of evaluation activity. Opportunities to seek and incorporate relevant stakeholder input for evaluation purposes should be welcomed, including through regular program or policy consultations. 

NCGP funding is administered under a number of schemes with varying objectives, timeframes and potential policy implications. Some aspects of program management also vary between schemes, for example, grant application and assessment processes. Conversely, there are also thematic issues or policy objectives that may apply to multiple or all schemes, for example, supporting Indigenous researchers or encouraging industry collaboration. 

When planning and undertaking evaluation in the ARC, a range of evaluation questions, topics and timeframes may therefore need to be considered. Evaluations need to be appropriately prioritised and targeted (as part of the Strategic Evaluation Plan, described below) to produce findings that can meaningfully inform decision making in policy and program processes.

Legislation and guidelines 

The requirement for performance measurement and assessment is underscored in a number of key legislative and other documents that apply to the administration of ARC business. Effective evaluation plays an important role in helping to meet this requirement, and is given a particular emphasis in relation to grants administration. 

The Australian Research Council Act 2001 (ARC Act) outlines the functions of the ARC, which include making recommendations to the Minister on research funding, administering research funding and providing advice on research matters.

The PGPA Act establishes requirements for all Commonwealth entities (including the ARC) to meet high standards of governance, performance and accountability, and to provide meaningful information to the Parliament and the public. In accordance with the CEO’s responsibilities under the PGPA Act, the ARC Corporate Plan outlines performance measures for the ARC’s functions, and the ARC Annual Report provides information on the ARC’s performance against those measures. Evaluation activity serves as a valuable input to inform and contribute to these processes.

The Commonwealth Grants Rules and Guidelines 2017 (CGRGs) establish the overarching Commonwealth grants policy framework and outline expectations and requirements for Commonwealth entities (including the ARC) in relation to all processes involved in grants administration, including evaluation. In particular, the CGRGs outline:

  • the importance of establishing performance and evaluation measures as part of robust planning and design of grants
  • the need for an outcomes orientation in grants administration, including clearly specified outcome, output and input measures to ensure effective and efficient evaluation of grants administration
  • the importance of evaluating a grant—including consideration of the continued appropriateness of government outcomes and entity strategic directions as a result of the grant’s impact—before initiating further grant opportunities
  • the role of grantees in participating in evaluation processes.

Additional guidance on evaluation in the context of grants administration, which informs the ARC’s approach, is provided by the Department of Finance in Resource Management Guide 412: Australian Government Grants – Briefing, Reporting, Evaluating and Election Commitments

Scope—what we evaluate

The scope of this Strategy includes the evaluation of ARC policy and program performance in relation to the functions of administering the NCGP and providing advice on research matters, as outlined in the ARC Act.

Examples of evaluation topics may include:

  • effectiveness in achieving individual funding scheme objectives and delivering intended outputs, outcomes and benefits
  • effectiveness of ARC programs and policies in addressing specific thematic issues or whole-of-government priorities
  • alignment of the NCGP with other research funding programs
  • efficiency of funding scheme administration
  • appropriateness and effectiveness of ARC policies on research matters.

The ARC’s activity in assessing the quality, engagement and impact of research (through the Excellence in Research for Australia and Engagement and Impact assessment exercises) is currently outside the scope of this Strategy, but may be included when the Strategy is reviewed in the future. 

ARC Evaluation Principles

To ensure that evaluations are meaningful and useful for decision makers responsible for ARC policies and programs, all ARC evaluation activity should be informed by, and conducted in accordance with, the following principles.

1. Integration

Evaluation should be built into policy and program processes. Evaluation planning should be completed early in the program or policy design phase to help establish defined, measurable outcomes. The timing of evaluation activity and reporting should be aligned with timeframes for program and policy implementation to maximise efficiency and quality in data collection and ensure the timeliness and usefulness of findings.

2.Good planning and management

Evaluations should be undertaken in accordance with good project management practices that would apply to any other project. They should be appropriately resourced and given sufficient time to produce meaningful findings. Governance processes should be in place to ensure appropriate approval and oversight of evaluation activity, and to reinforce the commitment of ARC senior executives to evaluation.

3. Fitness for purpose

The scale, resources and methodology applied to an evaluation should be appropriate and proportionate to the policy or program being evaluated. Considerations include the strategic importance, size, lifecycle, anticipated outcomes, impact, and risk of the policy or program, as well as the availability of data and the need to ensure the evaluation represents value for money.

4. Rigour

Evaluations should be high quality and evidence-based. Robust research and analytical methods should be applied in all evaluation activity, including both quantitative and qualitative approaches. All stakeholder engagement forming part of the methodology should be targeted and timely. Reliable, relevant and consistent data should be used meaningfully to inform evaluation findings.

5. Objectivity and independence

Evaluation should be undertaken with the aim of providing an objective assessment to inform policy and program decision making—it is not a promotional exercise. It should involve and be guided by internal and external stakeholders, including program managers, but should be conducted by evaluators who are independent of the areas responsible for the policy or program being evaluated.

6. Transparency

The final reports of all evaluations should be published internally and made available to all ARC staff, unless there are strong reasons to limit their distribution. Whenever possible, evaluation reports should also be published externally, to demonstrate the ARC’s commitment to accountability and strengthen the confidence of stakeholders and the public in ARC policies and programs.

Integrating evaluation

Building evaluation into the policy and program processes due for assessment is essential to ensure consistency with the ARC Evaluation Principles. In particular, embedding evaluation planning into the early stages of program and policy design facilitates high quality and efficient evaluation conduct, and supports a focus on achieving meaningful outcomes from every evaluation. 

A key component of good evaluation planning is the use of a logic model, which anchors the implementation of the program or policy to its objectives and intended outcomes, and can provide a basis to formulate the questions that the evaluation needs to address.

Integrating evaluation planning into policies and programs also helps to identify the data and information that needs to be gathered—potentially as part of the policy or program implementation process—and analysed to generate meaningful findings that will inform the development of further initiatives or changes to future program rounds.

As represented in Figure 1, integrating evaluation and program or policy lifecycles in this way helps to support continuous improvement.

Figure 1: Integrating evaluation into the program/policy lifecycle

It is important to note that the evaluation lifecycle will not always align with the program or policy lifecycle in the manner depicted in Figure 1. For example, an evaluation may be conducted on only the design and planning, or the implementation, of a program or policy. Every evaluation needs to be planned and tailored to address its particular topic effectively.

In the context of NCGP administration, the program lifecycle may refer to a number of different processes, depending on the scope and topic of a particular evaluation, for example:

  • the application and assessment process of a funding scheme, up to the point of funding being awarded (which may involve a timeframe of only a few months)
  • progress or changes in a funding scheme’s implementation, up to the point of regular mid-term review processes
  • the outcomes of a complete funding scheme round, up to the point of receiving and analysing final reports from all completed research projects (which may involve a timeframe of several years).

For evaluations of NCGP performance in relation to broader thematic issues or whole-of-government priorities that cut across multiple funding schemes, integrating evaluation into different program cycles may be particularly complex. Flexibility will be required to manage such evaluations effectively. 

What is important is to ensure all evaluations are well planned and targeted to inform evidence based decision making, and that they aim to contribute to improved evaluative thinking in the ARC.

Strategic Evaluation Plan

To embed an integrated approach to evaluation in the ARC, a Strategic Evaluation Plan will be developed, which identifies both short and longer term evaluation priorities to ensure the timeliness and usefulness of evaluation activity. The Senior Management Group, chaired by the CEO, is responsible for approval of the Plan to ensure activity is best targeted to support the CEO in measuring and assessing performance as required under the PGPA Act. 

The Plan will outline a program of evaluation activity over a four year period and will be reviewed annually to maintain its responsiveness to emerging priorities in NCGP administration and broader government policy. It will therefore require flexibility to allow planned activity in future years to be revised strategically.

The Plan will be developed in consultation with a broad range of ARC internal stakeholders so that evaluation activity is identified and prioritised in accordance with policy and program needs. Development and review of the Plan should take into account relevant funding scheme objectives, processes, funding levels and duration, as well as broader government and research sector imperatives. Potential overlaps with planned audit activity and major reviews should also be considered to ensure evaluation topics are well defined and avoid duplication, and resources are directed appropriately.

Where practicable, evaluations should be coordinated with relevant policy and program management timeframes, including relevant funding scheme timeframes, to maximise the efficiency of planning, data collection and analysis. In all cases, the timeframes for delivery of evaluation findings will be targeted to achieve the greatest impact for decision makers.

The program of planned activity—including evaluations conducted internally and by external providers—must be achievable within ARC resource and budget constraints.

The Plan is intended to be an internal document available to all ARC staff. Planned evaluations may be listed in the ARC Corporate Plan.

Implementation and governance

Implementation of this Strategy—including governance, strategic planning and conducting evaluations—requires involvement and commitment across the ARC in order to strengthen our evaluative thinking and achieve the best outcomes from evaluation activity. Specific roles and responsibilities are outlined below.

Program Evaluation Section (within Corporate Services Branch)

  • Implementation and review of this Strategy, including consultation across the ARC
  • Development and review of Strategic Evaluation Plan, including consultation across the ARC 
  • Development of plans for individual evaluations, in consultation with line areas
  • Project management of evaluations, and conducting internal evaluations—including contract management (where relevant), data collection, monitoring, stakeholder engagement, evaluation analysis, preparing final reports and communicating findings—in consultation with line areas
  • Providing advice on evaluation priorities and processes, including advice on embedding evaluation in policy and program practice
  • Providing updates to the Program, Strategy and Executive Committee and Senior Management Group on evaluation progress.

Corporate Services Branch

  • Responsibility for program and policy evaluation in the ARC (Corporate Services is independent from program and policy functions within ARC organisational structure)
  • Providing advice on expenditure for contracted evaluations
  • Internal and external communication regarding evaluation in the ARC, including publishing this Strategy, Strategic Evaluation Plan and final evaluation reports
  • Assisting with external stakeholder engagement
  • Assisting with the promotion of evaluative thinking within the ARC, including training opportunities for staff.

Programs Branch and Policy and Strategy Branch

  • Input to development and review of this Strategy 
  • Input to development and review of Strategic Evaluation Plan, including advice on:
    • program and policy priorities and timeframes
    • coordination with regular program and policy monitoring and review processes
    • alignment or overlap with planned audit activity (to avoid duplication)
  • Working with Program Evaluation Section on development of plans for individual evaluations
  • Incorporating evaluation questions in policy and program planning and implementation (for example, in monitoring and review processes)
  • Supporting Program Evaluation Section in management of evaluations, including data collection, monitoring and stakeholder engagement
  • Using evaluation findings to inform decision making on policy and programs.

Program, Strategy and Executive Committee

  • Providing advice on this Strategy
  • Providing advice on Strategic Evaluation Plan
  • Providing advice on individual evaluation plans, findings and reports.

Senior Management Group (chaired by CEO)

  • Setting overall direction and objectives of evaluation in the ARC
  • Approval of this Strategy
  • Approval of Strategic Evaluation Plan
  • Approval of annual budget allocations for evaluation activity
  • Reviewing and noting plans for individual evaluations
  • Monitoring progress of individual evaluations
  • Endorsement of final evaluation reports
  • Providing leadership in implementing actions and changes that respond to evaluation findings and recommendations
  • Providing leadership in promoting evaluative thinking in the ARC.

Review of this Strategy

This Strategy will be reviewed every two years, in consultation with relevant ARC internal stakeholders, for approval by the Senior Management Group. The next review date is 30 June 2020.

Contact details

Stakeholder Relations
Australian Research Council 
Phone: 02 6287 6600
communications@arc.gov.au
www.arc.gov.au 
Level 2, 11 Lancaster Place, Canberra Airport ACT 2609 
GPO Box 2702, Canberra ACT 2601