Mock REF Reviewer System

Final report for the #455 Mock REF Reviewer System computing project.

This was requested by Head of School. She was looking for a secure database where pre-determined panel members could be authorised to access submitted research papers and assign a rating to them (1 to 4 stars) - this to be followed by a moderation process with a final rating recorded. The selection of up to six research papers which each author has submitted for the REF, linked to their name, and attached a ranking (1 to 6) as well as a one hundred word summary could ideally be extracted from PURE, especially since the papers are already recorded in full detail in PURE researchers may not be happy to be asked to supply those details again in the new system. If that was the case then this system just needs a mechanism to assign panel members to reviewers and for panel members to record their ratings of the papers assigned to them. The project would first need to assess what information could be extracted automatically from PURE. The system would be required for mid-April (2018).

The University was not providing a system to do this, leaving it to individual Schools. There was a tight time scale for implementation. The first stage involved establishing what we could extract from PURE and how automated this could be. While PURE provides a reasonable API this specifically excluded access to the REF data required. However it was possible to write reports that generated CSV and to automatically schedule and send these reports by email, much as we do in BIS although this required IS to enable back end support for the specific user account that wanted to schedule and send data by email. Unfortunately the reports must be run under a particular user account, currently mine. The XML report definitions are held within the project Git repository so anybody else could get the necessary access and upload the reports and run them instead of me.

The expectation (see below) was that College would produce something for the actual REF submission (in 2021) and there was also a tight deadline, so we wanted something that could be thrown together fairly quickly. Also the work flow was not entirely clear at the outset, so we also wanted something that could be quickly altered while in use. So I decided to use a Theon managed service. This meant the database schema and corresponding UI could be prototyped and turned into a production system very quickly, but also meant the live system could be easily updated to account for any late design changes. Furthermore we also got a RESTful API for free by doing this, and if a more custom UI turned out to be necessary down the line it would have been relatively quick to put together using the existing API (although in the end this was never needed as Theon UI was sufficient). A data set and work flow was proposed and subsequently refined in consultation with Steve Renals, who then carried out all further testing.

The system was implemented, went through a few revisions and was then used successfully for the mock REF. Originally the intention was then to throw the system away. However it now looks like it will be used again for the next mock (2019). It is also now highly unlikely the University or College will provide any central system for doing the REF, and the support that could have been added within PURE for the REF rating processes is not going to be added (a decision by the multi University consortium involved). So this system may well continue being used even for the actual REF, no doubt subject to further changes. The system however was built fully managed and backed up so is already a standard production service so this is not really an issue. This specific project will however be wound up, further work done as misc devel or as a new project if sufficiently extensive.

In total this project took 82.25 hours of which: 65 hours was to design/implement the original system which was then used for the mock REF; a further 17.25 hours subsequent to the mock REF was to make changes to the rating system and to add additional access control structure so Institute directors could view the papers/ratings associated with members of their own institute only.

In conclusion using Theon seemed to be a very effective way to rapidly prototype and develop a system like this, to establish what data was needed and how the suggested work flow would actually work in practice. Being so lightweight to prototype it is also then inconsequential to throw away and re-implement as a dedicated system for the task based on the final data set and work flow that evolved from prototyping (even though in this case that was never done as Theon UI plus the provided API were sufficient, for the mock anyway).

-- TimColles - 18 Jan 2019

Topic revision: r1 - 18 Jan 2019 - 09:50:28 - TimColles
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies