PKSG submitted the following statement in response to the consultation as part of the Lord Stern review of the Research Excellence Framework (https://www.gov.uk/government/consultations/research-excellence-framework-review-call-for-evidence )
1. What changes to existing processes could more efficiently or more accurately assess the outputs, impacts and contexts of research in order to allocate QR? Should the definition of impact be broadened or refined? Is there scope for more or different use of metrics in any areas?
The definition of impact in REF is too narrow and is not conducive to innovative scientific ideas. Emphasis is placed on short-run impact and the quick use of the research results by the non-academic society. However, it is well-known from the history of science that the many important ideas are not easily accepted when they are first presented and often it takes years or decades until they have a clear impact.
In economics, two heterodox economists who saw the crisis coming, Hyman Minsky and Wynne Godley, had a very limited impact before the Global Financial Crisis, but since the crisis they are widely cited in the relevant academic literature and the financial press and they have informed research, e.g. at the Bank of England. Thus arguably their work has had a significant impact. If these economists had been assessed based on the REF criteria, they would not have been able to develop their research.
REF should thus recognise that not all types of research are capable of having a quick impact. Research has to be financed not only based on achieved outcomes, but also based on its potential to change fundamentally our way of thinking and understanding that has the potential to exert an impact in the longer term.
6. In your view how does the REF process influence, positively or negatively, the choices of individual researchers and / or higher education institutions? What are the reasons for this and what are the effects? How do such effects of the REF compare with effects of other drivers in the system (e.g. success for individuals in international career markets, or for universities in global rankings)? What suggestions would you have to restrict gaming the system?
In the discipline of Economics the REF has contributed to a narrowing of the discipline in terms of what theories and methodologies are acceptable. This process began already with the RAE, which has strengthened the dominance of a particular theory, neoclassical economics, which is characterised by an axiomatic, utilitarian approach and has efficient market clearing processes as their focus. The RAE and REF thus have contributed to an academic culture that has established an intellectual monoculture. Alternative approaches such as post-Keynesian economics, Feminist economics, Ecological Economics, Evolutionary economics etc get marginalised. This makes it difficult for young researchers to survive academically unless they work within the mainstream/neoclassical approach (because they cannot publish in the ‘best’ journals etc). These processes have been document (e.g. Lee 2007, Lee et al 2013). These tendencies have been reinforced with the move from RAE to REF. Most universities, having a close eye on their position in league tables, will now not submit staff unless the expected GPA for outputs is at least 2.5. The standard reference for the GPA has become various journal ratings, in particular the ABS (Association of Business Schools) journal list. In that list the only unambiguously non-mainstream journals at the 3* level are the Cambridge Journal of Economics and Ecological Economics, with not a single non-mainstream journal being rated as 4*.
As a consequence of this hardly any non-mainstream economists gets submitted to the Economics panel. Rather they are being squeezed into other UoA. To illustrate, take the Cambridge Journal of Economics, which is the most prestigious general non-mainstream journal. For the REF2014 there were four submissions of CJE articles to the Economics panel, 45 in Business and 7 in Politics and International Relations. In other words, high-quality critical economics approaches are more welcomed outside economics than in the discipline.
The dominance of neoclassical or mainstream economics has had three important implications: first, you scholars are discouraged to take up non-mainstream approach and, if they do, their career paths are blocked; second, this has contributed to the narrowing of the field, which has results in bad economic policy advice and the blindness of the profession with respect to the factors that caused the Global Financial Crisis 2008/09; third it has resulted in a narrowing of the teaching of economics against which there has been a wave of student, exemplified in the manifesto of the International Student Initiative for Pluralism in Economics.
It is clear that the present approach to evaluating research in economics has resulted in a narrowing of discipline and does undermine the position of research that take critical approaches. Of foremost importance is to ensure a more diverse composition of the evaluation panel in terms of the theoretical background of the panellists.