Title: Investigating the Capabilities of Generative AI in Grant Proposal Evaluation: UKRI’s Pioneering Methodology
In light of a rising number of grant submissions and constrained funding resources, the UK’s principal funding organization, UK Research and Innovation (UKRI), is looking into the incorporation of generative AI within their grant proposal evaluation framework. Annually, UKRI allocates more than £8 billion for research grants, but in the last seven years, the total of funded research and innovation projects has decreased by half, while applications have increased by over 80%. To enhance its peer review mechanism, UKRI is examining whether AI could ease the burden on human reviewers by potentially forecasting the results of grant submissions.
Guided by Mike Thelwall, a data scientist from the University of Sheffield, a research group will analyze as many as 2,000 grant proposals submitted to UKRI starting in October. The findings from this initiative could considerably reshape peer review methodologies by leveraging large language models (LLMs) to estimate the scores and verdicts that human reviewers would assign.
This initiative is a continuation of earlier work conducted by Thelwall and his team to ascertain AI’s ability to support the UK’s Research Excellence Framework, which revealed a 72% correlation between AI-generated scores and those provided by human evaluators. The optimal target for practical use stands at 95% accuracy, indicating further advancements are necessary before AI can effectively assist in peer review.
Mohammad Hosseini from Northwestern University points out that LLMs encounter difficulties in generating original concepts, as they depend on existing information. There are also concerns that a lack of clarity regarding the criteria used by AI could lead to backlash or manipulation from grant applicants. Ultimately, Thelwall suggests that AI may serve well in tiebreaking scenarios, functioning as an additional reviewer, or managing quick desk rejections to lessen the workload on experts.
The la Caixa Foundation in Barcelona serves as a model, utilizing AI for initial screenings, although most submissions still go through comprehensive expert peer evaluation. This approach has resulted in considerable time savings for reviewers, indicating that AI holds the promise to enhance the efficiency of the grant review process.
UKRI’s efforts to integrate AI into grant proposal evaluations could signal a transformative advancement in efficiency and innovation in research funding, vital for meeting the growing demands on peer review systems as application volumes increase.