Current status of the REF and bibliometrics [October 2009]

The bibliometric pilot was designed to test the most radical feature of the proposed structure of the Research Excellence Framework and was carried out relatively early in the development process. Since the summer of 2008 when the pilot was carried out development of the final REF specification has continued and a final consultation document was issued in September 2009. The consultation document envisages a much smaller role for bibliometrics that the original REF proposal and in the 'Brief guide to the proposals' document that accompanies the consultation HEFCE outlines the proposed status of bibliometrics in the statement:

"We conducted a substantive pilot exercise to test how to use citation information in the REF. We concluded that citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality, but there is considerable scope for it to inform and enhance the process of expert review. We propose that:

• Those UOAs for which robust data is available will make use of citation information. Sub-panels will decide this in advance. We expect that medicine, science and engineering panels will do so, but that the arts, humanities and a number of other panels will not.
• We will provide the relevant panels with citation information about the number of times that submitted outputs have been cited, and with appropriate benchmarks.
• These panels will use the information to inform and supplement their review of the outputs, to assist with achieving consistency, international benchmarking and where possible reducing workloads.
• There will be clear guidelines on using the data robustly to take account of the known limitations and to avoid bias (for example, citations are less meaningful for recently published outputs, and are not available for certain types of output). Panels will not make judgements about the quality of outputs solely on the basis of citation information; expert judgement must be applied. All submitted outputs will be treated equally, whether or not there is citation information available for them."