What is Success?
What is Success?
The Australian Research Council (ARC) plays an important role in the provision of Government support for research in Australia. As a grants agency we support the sector though our competitive peer reviewed funding schemes, and this year we will deliver over $800 million to the most dynamic researchers in Australia.
Our grants are competitively awarded to individuals, research teams and large scale centres through two broad arms: the Discovery Program, with a primary focus on pure research endeavour; and the Linkage Program, which creates links outside universities, with industry and other partners and stimulates research impact.
The profile of disciplines seeking funding from the ARC is illustrated in the Figure 1. What is interesting from this data is the relative constancy of the discipline mix over the last decade, with only one area, Medical and Health sciences, increasingly coming to the ARC to ask for more. What is also evident is the relatively low spend on the humanities with only 8% of requests coming from this area.
While the relative success rates for each of our discipline panels is approximately the same, it is almost universally true that no discipline area thinks it is getting its fair share! Notwithstanding this criticism the ARC has a high degree of contribution from researchers in the sector and the sector’s confidence in the ARC’s decision making processes is high.
Alternative messages do however get significant airtime and there are critics who think the system is too complicated, too wasteful and the success rates are generally too low.
While we have taken on the task to simplify the application process it will come as no surprise that I don’t agree with these criticisms and I certainly reject the criticism that the system is broken.
We know the system works because the ARC not only supports quality research and research careers through grants, but it also—through an independent process—measures the quality of the outputs from the Government investment in the Higher Education sector. This process is Excellence in Research for Australia, more commonly known as ERA. ERA provides the country’s most detailed assessment of research activity, which drills right down to the discipline level in every university in Australia.
Unlike other national systems, which use citation data in an ad hoc way, the ARC has developed a world leading, ‘state-of-the-art’, methodology. The use of peer review and/or citation analysis for a particular discipline is strongly guided by detailed and on-going consultation with the research community.
Importantly ERA can track the investment in the sector through the competitive grant process. From the ERA 2012 results, we know that over 95% of the competitive grant research support (largely through the ARC and the NHMRC) was directed into research areas rated at or above world standard, with 73% rated at above or well above. Clearly a process that achieves these outcomes is not broken.
The diverse mix of disciplines shown in the figure illustrates one of the challenges facing the ARC in its annual grant cycle. We need to be able to combine a broad understanding of what constitutes high quality research with an expert perspective from credible and disinterested reviewers close to the particular discipline area. This challenge is further compounded by the large volume of grants processed every year by the ARC—this year over 7000. Although some commentators would argue that this is justification for separating the discipline components of the ARC into separate bodies, as is done in the UK, this would result in an inherently inefficient process because of duplication, and one that would be less capable of accommodating research in multidisciplinary areas.
Applications for funding at the ARC go through a rigorous and competitive assessment process. Each of these applications is looked at by a number of assessors who are experts in their fields, then assessed again by the selection panels drawn from our College of Experts. In the feedback I receive from the panels at the end of these selections my confidence in the rigour and robustness of the process is continually reinforced.
The competition here is important. It ensures that both applicants and assessors strive to achieve the best outcomes in the allocation of a scarce resource. This competition is essential to maintain the vitality of the Australian higher education research system and the strength of its universities generally. This sector provides wealth, both directly and indirectly to the nation. While other models of resource allocation are possible, many would lock in funding based on historical patterns and inhibit the support of creative and innovative ideas that the nation could benefit from. Models that smear the resource uniformly run the risk of lowering the overall quality of the system to a mediocre standard.
Other commentators lament the poor success rate for the ARC, notwithstanding the fact that is far better that many comparable international agencies.
The success rate for ARC grants is, of course, largely beyond the control of the ARC. It is a function of the demand for grants from the academic community, the cost of doing research at an international level and the resource available. The last component is dependent on the policy settings of the government of the day and to some degree the capacity of the sector to convince those controlling expenditure that the research undertaken within the sector provides value to the nation at large. While some argue that the success rates can easily be fixed by an increase of government funds the problem is more complicated than that. An increase in money will address the problem in the first instance, but it will also stimulate demand potentially resulting in an even more competitive system and lowering success rates even further. The introduction of additional funds into the system needs to be done with care and consultation to ensure that desired benefits are realised.
There has been little change in the success rate for our core scheme, Discovery Projects, over the past seven years. This year the success rate was 19.9% compared to a peak of 22.7% in 2010. This change has occurred not because of any withdrawal of funds from the program, but as a result of a conscious decision to address a trend which saw the average return rate per awarded grant fall to a low of 50% in 2010. Here the return rate is the fraction of funds allocated to a successful grant compared to the amount requested. It is my view that low return rates are damaging to the system. They reduce the capacity of researchers to undertake the research they commit to, they require a co-investment from the institution that puts additional strain on and already stretched system and most damagingly, it encourages a spiral of over-asking and ambit bidding anticipating a reduction in allocated funds. Although the success rate this year was just under 20% it was achieved with a much healthier average return rate of 63%. Focussing on the success rate alone can disguise important trends to the return rate, which are actually improving.
There has been a significant decrease in the success rate for our Future Fellowships scheme this year. This has been a direct consequence of a dramatic increase in applications, from around 600 in 2012 to over 1200 in 2013. This alone was responsible for pushing the success rate down from nearly 35% to 16% this year. Obviously the major driver of this change was the terminating nature of the Future Fellowships scheme.
Changes to success rates can occur for other reasons. There is much discussion in the sector at the moment about addressing the overhead of the grant application process by increasing the duration of grants. It is important to realise that, in an environment where the pool of money in constant this can only be achieved with a decrease in success rates. There are good arguments for increasing grant duration, predominantly around the increased capacity for more significant outcomes with a greater opportunity to achieve solutions to more complicated problems. This does however tie up funds in fewer projects resulting in lower success rates. While in a steady state the actual duration of the grant is not an issue, with the same average number of researchers supported, it is difficult to transition quickly to a regime of longer grants without an additional input of money. Care must be taken in any transition phase to ensure that particular researcher cohorts, such as early career researchers, are not disadvantaged as success rates drop. Lower success rates tend to result in a greater fraction of grants being awarded to more senior researchers (and to those in more research intensive universities). While this may give some assurance to these researchers and a relatively minor relief from the burden of the application process, unless longer grants are introduced with care, early career researchers are likely to initially suffer under such a regime and lower success rates will result in even more disappointed researchers. The fraction of new awardees for our current Discovery Projects scheme, together with the Discovery Early Career Researcher Award is around 28%. It would be disappointing to see this number fall significantly.
Some argue that low success rates mean an enormous amount of time is ‘wasted’ in applying. While I do accept the argument that the needless filling out of forms is not a useful process, I do not accept the notion that preparing a grant application is time wasted. The reflection and consideration of the challenges in a particular discipline area, the assessment of the state of knowledge in a particular field and the consideration of how one’s particular contribution may transform a subject are never in my view wasteful. Quite the contrary, they are crucial components of a proper, productive, and beneficial academic system. The claims of wastefulness similarly neglect the advantages and benefits that a competitive system confers in selecting the best grants to support in an environment when there will never be sufficient money to support every project.
It is my view that although there is some benefit in allowing longer grants and, while it would be better to have a slightly higher success rate, the ‘wastefulness’ of the grants process is best addressed by improving the efficiency of the grant application process and, where possible, removing questions or streamlining applications so that researchers can focus on the project and the benefits their work can produce. The ARC is being proactive on this front. We are currently working closely with our sister agency, the NHMRC, to explore ways to streamline processes across the sector and we are rebuilding our ICT systems to enable much smarter application processes. This will take time, but we aim to make a difference.
The competitive grants process is a challenging system, but one that does produce outstanding results. In the course of my research career I have seen at close hand the disappointment associated with such a process. In the short time that I have been at the ARC I see this even more acutely, particularly in the significant number of good applications from people that I know are good researchers. This is offset, however, in the extraordinary stories and promise that lies behind the thousand or so new grants that we do support each year. I am inspired every time I go through the project descriptions at the conclusion of each round. The statistics say that some might not be successful, but all have the potential to be transformative and it gives me great pride to be part of the process that gives so much support to so many outstanding researchers that we are so fortunate to have in this country. The future looks very bright for research and innovation in Australia.
Professor Aidan Byrne, CEO of the Australian Research Council
This article was printed in the January-March 2014 edition of Australian Quarterly, as part of its feature on The Great Grant Debate: The future of Australian research.