What is Success?

The Australian  Research Council (ARC) plays an important role in the provision of Government support for research in Australia. As a grants agency we support the sector  though our competitive peer reviewed funding schemes, and this year we will  deliver over $800 million to the most dynamic researchers in Australia.

Our grants are competitively awarded to individuals, research teams and large scale centres through two broad arms: the Discovery Program, with a primary focus on pure research endeavour; and the Linkage Program, which creates links outside universities, with industry and other partners and stimulates research impact.

The profile of disciplines seeking funding from the ARC is illustrated in the Figure 1. What is interesting from this data is the relative constancy of the discipline mix over the last decade, with only one area, Medical and Health sciences, increasingly coming to the ARC to ask for more. What is also evident is the relatively low spend on the humanities with only 8% of requests coming from this area.

Figure 1:

While the relative  success rates for each of our discipline panels is approximately the same, it is almost universally true that no discipline area thinks it is getting its fair share! Notwithstanding this criticism  the ARC has a high degree of contribution from researchers in the sector and the sector’s confidence in the ARC’s decision making processes is high.

Alternative messages do however get significant airtime and there are critics who think the  system is too complicated, too wasteful and the success rates are generally too low.

While we have taken on the task to simplify the application process it will come as no surprise that I don’t agree with these criticisms and I certainly reject the criticism that the system is broken.

We know the system  works because the ARC not only supports quality research and research careers through grants, but it also—through an independent process—measures the quality of the outputs from the Government investment in the Higher Education sector. This process is Excellence in Research for Australia, more commonly known as ERA. ERA provides the country’s most detailed assessment of research activity, which drills right down to the discipline level in every university in  Australia.

Unlike other national systems, which use citation data in an  ad hoc way, the ARC has developed a world leading, ‘state-of-the-art’, methodology. The use of peer review and/or citation analysis for a particular discipline is strongly guided by detailed and on-going consultation with the research community. 

Importantly ERA can track the investment in the sector through the competitive grant process. From the ERA 2012 results, we know that over 95% of the competitive grant research support (largely through the ARC and the NHMRC) was directed into research areas rated at or above world standard,  with 73% rated at above or well above. Clearly a process that achieves these outcomes is not broken. 

The diverse mix of disciplines shown in the figure illustrates one of the challenges facing the ARC in its annual grant cycle. We  need to be able to combine a broad understanding of what constitutes high quality research with an expert perspective from credible and disinterested reviewers close to the particular discipline area. This challenge is further compounded by the large volume of grants processed every year by the ARC—this  year over 7000. Although some commentators would argue that this is justification for separating the discipline components of the ARC into separate bodies, as is done in the UK, this would result in an inherently inefficient  process because of duplication, and one that would be less capable of  accommodating research in multidisciplinary areas.

Applications for funding at the ARC go through a rigorous  and competitive assessment process. Each of these applications is looked at by a number of assessors who are experts in their fields, then assessed again by  the selection panels drawn from our College of Experts. In the feedback I receive from the panels at the end of these selections my confidence in the rigour and robustness of the  process is continually reinforced.

The competition here is important. It ensures that both applicants and assessors strive to achieve the best outcomes in the allocation of a scarce resource. This competition is essential to maintain the vitality of the Australian higher education research system and the strength of its universities generally. This sector provides wealth, both directly and indirectly to the nation. While other models of  resource allocation are possible, many would lock in funding based on  historical patterns and inhibit the support of creative and innovative ideas that the nation could benefit from. Models that smear the resource uniformly  run the risk of lowering the overall quality of the system to a mediocre standard.

Other commentators lament the poor success rate for the ARC, notwithstanding the fact that is far better that many comparable international  agencies.

The success rate for ARC grants is, of course, largely beyond the control of the ARC. It is a function of the demand for grants from  the academic community, the cost of doing research at an international level and the resource available. The last component is dependent on the policy settings of the government of the day and to some degree the capacity of the  sector to convince those controlling expenditure that the research undertaken within the sector provides value to the nation at large. While some argue that  the success rates can easily be fixed by an increase of government funds the problem is more complicated than that. An increase in money will address the problem in the first instance, but it will also stimulate demand potentially  resulting in an even more competitive system and lowering success rates even  further. The introduction of additional funds into the system needs to be done with care and consultation to  ensure that desired benefits are realised.

There has been little change in the success rate for our  core scheme, Discovery Projects, over  the past seven years. This year the success rate was 19.9% compared to a peak  of 22.7% in 2010. This change has occurred not because of any withdrawal of funds from the program, but as a result of a conscious decision to address a trend which saw the average return rate per awarded grant fall to a low of 50% in 2010. Here the return rate is the fraction of funds allocated to a successful grant compared to the amount  requested. It is my view that low return rates are damaging to the system. They  reduce the capacity of researchers to undertake the research they commit to, they require a co-investment from the institution that puts additional strain on and already stretched system and most damagingly, it encourages a spiral of  over-asking and ambit bidding anticipating a reduction in allocated funds. Although  the success rate this year was just under 20% it was achieved with a much  healthier average return rate of 63%. Focussing  on the success rate alone can disguise important trends to the return rate, which are actually improving.

There has been a significant decrease in the success rate for our Future Fellowships scheme this year. This has been a direct consequence of a dramatic increase in applications, from around 600 in 2012 to over 1200 in 2013. This alone was responsible for pushing the success rate down from nearly 35% to 16% this year. Obviously the major driver of this change was the terminating nature of the Future Fellowships scheme.

Changes to success rates can occur for other reasons. There  is much discussion in the sector at the moment about addressing the overhead of  the grant application process by increasing the duration of grants. It is important to realise that, in an environment where the pool of money in  constant this can only be achieved with a decrease in success rates. There are good arguments for increasing grant duration, predominantly around the  increased capacity for more significant outcomes with a greater opportunity to achieve solutions to more complicated problems. This does however tie up funds in fewer projects resulting in lower success rates. While in a steady state the actual duration of the grant is not an issue, with the same average number of researchers supported, it is  difficult to transition quickly to a regime of longer grants without an  additional input of money. Care must be  taken in any transition phase to ensure that particular researcher cohorts, such as early career researchers, are not disadvantaged as success rates drop.  Lower success rates tend to result in a greater fraction of grants being awarded to more senior researchers (and to those in more research intensive  universities). While this may give some assurance to these researchers and a relatively minor relief from the burden of the application process, unless  longer grants are introduced with care, early career researchers are likely to  initially suffer under such a regime and lower success rates will result in even more disappointed researchers. The fraction of new awardees for our current Discovery Projects scheme, together with the Discovery Early Career Researcher Award is around 28%. It  would be disappointing to see this number fall significantly.

Some argue that low success rates mean an enormous amount of  time is ‘wasted’ in applying. While I do accept the argument that the needless  filling out of forms is not a useful process, I do not accept the notion that preparing a grant application is time  wasted. The reflection and consideration of the challenges in a particular  discipline area, the assessment of the state of knowledge in a particular field  and the consideration of how one’s particular contribution may transform a  subject are never in my view wasteful. Quite the contrary, they are crucial  components of a proper, productive, and beneficial academic system. The claims of wastefulness similarly neglect the advantages and benefits that a competitive system confers in selecting the best grants to support in an environment  when there will never be sufficient money to support every project.

It is my view that although there is some benefit in  allowing longer grants and, while it would be better to have a slightly higher success rate, the ‘wastefulness’ of the grants process is best addressed by  improving the efficiency of the grant application process and, where possible, removing questions or streamlining applications so that researchers can focus  on the project and the benefits their work can produce. The ARC is being  proactive on this front.  We are currently working closely with our sister agency, the NHMRC, to explore ways to  streamline processes across the sector and we are rebuilding our ICT systems to  enable much smarter application processes. This will take time, but we aim to  make a difference.

The competitive grants process is a challenging system, but  one that does produce outstanding results. In the course of my research career  I have seen at close hand the disappointment associated with such a process. In  the short time that I have been at the ARC I see this even more acutely,  particularly in the significant number of good applications from people that I  know are good researchers. This is offset, however, in the extraordinary stories and promise that lies behind the  thousand or so new grants that we do support each year. I am inspired every  time I go through the project descriptions at the conclusion of each round. The statistics say that some might not be successful, but all have the potential to  be transformative and it gives me great pride to be part of the process that gives  so much support to so many outstanding researchers that we are so fortunate to  have in this country. The future looks  very bright for research and innovation in Australia.

Professor Aidan Byrne, CEO of the Australian Research Council

This article was  printed in the January-March 2014 edition of Australian Quarterly, as part of its feature  on The Great Grant Debate: The future of Australian research.