24 December 2013

Professor Aidan Byrne, CEO Australian Research Council

It is important for taxpayers to know if their money is being invested wisely—all Australians want to know that they are getting a return for the investment made by its Government.

The general community hears a lot about research—research looking into deep space, cancer, coral reefs and climate change, dementia and the nation's groundwater supply, just to name a few—but how can the community be assured that the multi-billion dollar investment in research is justified?

The Australian Research Council (ARC) is responsible for Excellence in Research for Australia—more commonly known as ERA—Australia's national evaluation of research quality.

ERA aims to identify and promote excellence across the full spectrum of research activity in Australia's higher education institutions by assessing the quality of research undertaken in those institutions.

The most recent ERA data indicates that the annual commitment to the higher education sector includes around $1.3 billion through Australian Competitive Grants, $850 million invested by others in the public sector, $800 million by industry and other international agencies and another $120 million was invested into cooperative research centres. Additionally universities received a block grant allocation of the order of $1.5 billion.

Given the size of this investment, it is important to ensure that university research is of the highest quality. It is equally important that researchers, industry and the Australian public are confident that ERA is robust, effective and value for money.

ERA has wide spread acceptance in the research community but some commentators continue to disparage it, often based on out-dated or misleading information about the ERA process. The most common claims are that it is: too costly; a blunt measure of quality; or that it leads to perverse research outcomes including disengagement from industry. Letting such comments go unchecked undermines the confidence that ERA quite rightly gives industry and government that its investments in Australian research provide good economic returns and social benefits.

To date the cost of ERA has been $43.5 million, which has funded a 2009 trial, two full rounds in 2010 and 2012, and preparations so far for ERA 2015.  Given the enormous amount of research activity over this period, and that for the three years from 2008 to 2010 universities received $8.77 billion in research investment, ERA is clearly cost effective—at less than 0.5% cost to verify that investment. From the ERA 2012 results, we also know that over 95% of research related to this investment was rated at or above world standard.

The ARC continually works with universities to streamline ERA and take advantage of existing data collections and performance measures. The Australian Scheme for Higher Education Repositories and the Implementation Assistance Program provided universities almost $42 million to help them develop digital repositories and data systems to ensure their reporting systems were ERA-ready. All universities continue to benefit from these systems, which are important for their own internal research management and strategy purposes.  The benefits will only increase in the future and will ease the effort required of universities to participate in ERA.

Those that claim ERA is a blunt measure of research are simply wrong. Highly experienced and qualified committees of experts in the various research fields make the decisions about ERA ratings. The decisions are based on extensive information including reports from peer reviewers or comprehensive citation analysis and benchmarking. Universities also provide contextual information about research income, patents, and esteem measures for their staff.

Unlike other national systems, which use citation data in an ad hoc way, the ARC has developed a world leading citation methodology. The use of peer review or citation analysis for a particular discipline is strongly guided by detailed and on-going consultation with the research community. 

This pioneering of evaluation methodology involving metrics and peer review has attracted considerable international attention.  A 2010 OECD publication described ERA's methodology as 'state of the art'.  The ERA methodology guards against unintended research behaviours and does not direct researchers about where to publish. Indeed, ERA evaluates books, book chapters, journal articles and a wide variety of non-traditional research outputs including music, creative performances, and policy documents.

ERA recognises and rewards quality research that appears in applied journals or specialist publishing outlets.  Based on sector feedback, ERA has not made use of journal rankings since 2010, and while some universities have continued to use them internally, it is the ARC's firm view that this should stop.

Finally, far from creating disincentives for industry collaboration with universities, ERA is a positive for industry engagement. A comparison of the results of ERA 2010 and ERA 2012 shows that despite the pressures that the GFC and other economic conditions have placed on businesses, industry and university engagement in Australia remains strong. The ERA 2010 and ERA 2012 results show increases in industry income for universities and that this income is concentrated in the areas and institutions that achieve higher quality ratings in ERA.

The ARC is now gearing up for ERA 2015, the third full evaluation of research quality across Australia's higher education institutions. The ARC has also released an independent report undertaken by ACIL Allen Consulting that paints a more complete picture of the influences, benefits and impacts of ERA on Australia's university sector.

The review is very positive about the benefits of ERA and the potential for further benefit, given the reasonably recent introduction of ERA. Benefits identified in the report include:

  • ERA is improving the quality of research by focussing behaviour of researchers on measures of research quality.
  • ERA is driving the performance of Australian universities in the global research rankings (in part because it has created a culture which focusses its activities in the areas that are measured by these indices.
  • ERA is one of the primary mechanisms that Government, public and private sectors have to account for their expenditure on higher education research sector.
  • ERA is an investment in intangible knowledge-based assets and a means of tracking that investment over time.
  • ERA is assisting with the concentration of research effort by universities into areas of strength. ERA provides input to the strategic plans of the majority of universities, which includes identifying and fostering emerging research areas and areas of competitive advantage (as well as areas of underperformance).

The Report concludes that these and other beneficial effects are likely to multiply over time.

It is imperative that when industry and other members of society rely on university research to develop products, or address the pressing social, health, environmental and economic issues of our time that the underlying research is of the highest quality.

We should not allow ourselves to support or rely on the outcomes of poor quality research.  

ERA is the best mechanism that the Australian people, Government and the private sector have to ensure this.  ERA is a rich data source and I encourage all stakeholders to use the information and resources it produces.


Image: Excellence in Research for Australia—Benefits Realisation Review