The REEF Index - measuring the efficiency of Research and Education

RCA has developed the REEF Index. REEF is the acronym for Research and Education Efficiency Frontier Index (REEF Index). It provides a new approach to analysis of higher education outcomes. It measures the efficiency of performance of institutions against the two key ‘outputs’ or ‘products’ of the university sector – Education (or Teaching) and Research – but is neutral as to where an institution chooses to position itself in respect of the ‘education – research’ mix.

The REEF Index allows institutions to track their performance over time: against their own efficiency, against others, and in comparison to the performance “efficiency frontier”, from which the analysis takes its name. It can also allow institutions to drill down to compare the relative efficiency of different parts of their institution.

WHAT IS THE REEF INDEX

The REEF Index is a rating, not a ranking process, it acknowledges and takes account of the fact that Universities vary in their approach - the analysis is agnostic as to whether there is a teaching or research focus or some combination of the two. This is one of its comparative advantages.

At its base, the REEF Index is a metric benchmarking process using several techniques including Data Envelopment Analysis (DEA) to create competitive benchmarks using core overall performance indicators. The core overall performance indicators are:

  1. Uniquely authored publications per $m;
  2. Students educated per $m;
  3. Uniquely authored publications per full-time equivalent (FTE) Academic Staff member.
  4. Students per FTE Academic Staff member.

In addition to the two outputs (teaching and research), the REEF Index recognizes that much of the production of outputs involves input which are ‘naturally occurring joint and common costs’.

The core overall performance indicators allow a frontier of performance to be identified, and against which all institutions can be compared.

The analysis uses data from (1) proprietary institutional, (2) Federal Government sources including the discontinued HERDC data set or (3) other research datasets available. From 2015, a newly created RCA data set for research has been created. As with HERDC, the RCA research dataset is calibrated to one unique authored published article. The RCA data approach has been subjected to validation processes. In this connection, the correlation RCA 2014 and HERDC 2014 was 0.983. 

WHAT IS EFFICIENCY?

It is the delivery of a given level of output with less input OR an increased level of output for a given level of input all other things being equal. Efficiency can also be a combination of these two.

WHAT IS ANALYSED?

The REEF Index started as a project to examine the costs and efficiency in universities as part of a program to support the higher education sector – particularly to enhance research productivity.

Two input variables are used in the analysis:

  1. Total university expenditure;
  2. Academic staff time.

University Expenditure is corrected for (1) expenditure not involved in academic outcomes e.g. investment losses and asset impairments; and, (2) in the time-series analysis is deflated using the ABS Index for education. Academic staff time is the EFT academic staff of each institution.

WHAT IS AN INSTITUTION ANALYSED AGAINST?

Institutions are analysed in a number of ways varying to support individual institutional needs. These analyses involve either a ‘change’ analysis over time or a ‘level’ analysis comparing an institution with other benchmarks. As would be expected, the econometrics is somewhat different depending on the type of analysis.

The first type of analysis is where the level of efficiency of a university is analysed against other peer institutions (that is the current position of university X relative to university Y at the same time point). The peer institutions may be seen as competitor institutions or comparator institutions.

Institutions can also are analysed in comparison to the “efficiency frontier” performance – that is best performance in the industry at a given point in time. Some universities are at the efficiency frontier, others are a distance away from the frontier. A university might be ‘at the frontier’ or within 5% of the frontier (wherever that might lie for the university given its research / teaching mix), or it might be only operating at 50 or 60% of the level of efficiency of the ‘best practice’ institutions.

The third form of analysis is where efficiency is examined against an institution’s own efficiency over time. For example, some universities may argue that they are a ‘special case’ where comparisons with peers are argued to be invalid. In these situations, the REEF Index provides a ‘change’ analysis where the analysis is for the same university over time.

HOW MIGHT THE REEF INDEX BE OF VALUE?

The Index is a new form of benchmark, based on overall efficiency, relative to the institution and to others rather than just scale or capacity. Government may seek to have efficiency measure that are not just cash flow based. Seeking genuine underlying efficiency may be seen a better targeted mechanism to give effect to public policy settings. Arguably, it is a measure that more easily passes “pub test” type questions such as “Which university gives the taxpayer the best value for each dollar invested?”.

Other interested stakeholders, such as philanthropic trusts, state government and private sector partners and other potential partners, may use the REEF Index as a criterion for establishing or continuing a relationship with a university.

Governments and their agencies with responsibilities may be users, including the relevant Government departments together with agencies such as Auditors-General.

Those charged with university management, governance and oversight such as university councils may use the REEF Index to generate efficiency measures as key KPIs.

 

HOW IS THE REFF INDEX DIFFERENT TO WHAT IS IN THE MARKET NOW?

The REEF Index analysis will often demonstrate a differential between the efficiency of the use of academic staff and the use of non-academic staff resources (embedded within total resources).

Existing rankings generally assess, in various way, the relative ‘prestige’ of universities not there efficient use of either total or academic resources. There is no element of the ‘good use of resources – including, where relevant, public funding.

HOW CAN AN INSTITUTION USE THE REEF INDEX TO IMPROVE INTERNAL EFFICIENCY?

University councils may use the REEF Index to generate efficiency measures as key KPIs. It helps answer questions such: Is our efficiency where we want it to be? What is our position relative to university X and Y? Is our ‘back office’ as efficient as our academic endeavour? Is our VC running an efficient organization? Can we credibly say to donors that we are amongst the most efficient organizations you can give money to? If the organization or government is seeking a partner, can we credibly say we are the most efficient? Will the taxpayer and government agencies see us as both effective in our outcomes (rankings, entrance standards and employability outcomes amongst others) and efficient? Will the Auditor-General see the institution al achieving credible outcomes in a performance auditing context?

Many of the same questions are relevant to University Executives. Such people may use efficiency KPIs in addition to other management tools. In addition to the above, might a VC want to know: is faculty X as efficient as faculty Y? The analysis goes well beyond simple cash flow to answer that question. Does my university need seek greater efficiencies in ‘back office’ activities? Is it efficiency of academic staff that is the higher priority? Amongst many other questions.

World University Ranking Systems and the REEF Index

 There are a number of international ranking systems for universities. The REEF Index as a new index, offers a novel and arguably highly unique approach. The rankings field is relatively new with the first international ranking being “Asia’s Best Universities” that operated from 1997 – 2000. The current ranking systems all use different methodologies with the most well known being:

  1. Times Higher Education World University Rankings
  1. QS World University Rankings
  1. Academic Ranking of World Universities

There are a large number of other ranking systems including national ranking systems in use within countries, such as through well established annual reporting processes, such as in the US and UK. Most ranking systems relate to issues of ‘prestige’ not efficiency and the good use of resources.

There is a high correlation between the outcomes of the various major ranking systems.

A review by RCA of the over 50 major international and national ranking systems shows that none are reported to use overall efficiency measures in the way the REEF Index is able to create an index and a frontier of efficiency.

This puts the REEF Index at a comparative advantage as a new index, with a unique offering to institutions and the global higher education system.

READ OUR REEF INDEX POWERPOINT PRESENTATION

READ SOME OF THE PUBLICITY ABOUT REEF INDEX