User-facing documentation can be found at http://computing.help.inf.ed.ac.uk/msc-project-submission.

Introduction

This page documents the service provided by https://projsubs.inf.ed.ac.uk/, http://project-archive.inf.ed.ac.uk/ and https://project-submit.inf.ed.ac.uk/. The former holds internally visible only and restricted links to the UG4/5 projects and MSc dissertations, along with associated submission records as stored by the UG4/5 and MSc project submission web forms and associated marking records as stored by the UG4, UG5 and MSc Webmark marking forms. The latter holds the publically visible links to UG4/5 projects and MSc dissertations that have been approved for wider publication (some only within the School however, i.e. requiring authentication).

Structure

The projsubs service is managed entirely by LCFG configuration using the dice/options/projsubs-server.h header file. This includes a local projsubs package (held in the SVN dice repository) which contains the CGI scripts that are used both for submissions and also for generating the link pages and public archive. The Webmark forms are held separately under the Webmark system.

The server holds no master data itself; this is all held in AFS group space (group/teaching/projects/{msc|ug4}projects).

The server does hold:

• live generated link pages (YEAR-current.txt for projsubs)
• a cache of project documents (YEAR.html for both projsubs and the public archive).

The caches can all be re-generated on demand by running the CGI scripts manually as described below.

Access

• Only members of staff (staff) should have access to the authenticated projsubs pages.
  • Access granted by web/wwwinf/staffonly capability.
  • In addition, the scripts prevent staff access to project materials for previous years' projects where the submitting student has not granted permission for their continued use (this allows support to perform manual deletion/archiving when appropriate, while enforcing the letter of the agreement).
• Only some members of administrative staff, the projects coordinator and external examiners (examiner-ext) should have access to the authenticated current assessment year projsubs page (YEAR-current.txt).
  • Access granted by web/wwwinf/projsubs capability.
  • URL for external examiners should be projsubs/cgi-bin/submission_index.cgi?file=/ug4/YEAR-CURRENT.txt
• Access to the project-archive pages is open to all anonymously.

As mentioned above, the actual HTML pages (containing project listings) are generated by the CGI scripts held in cgi/. They are run periodically by a cronjob belonging to wwwrun.

Marking Information

Assessment documents are submitted by the webmark forms into the appropriate upload directory in the AFS group space (group/teaching/projects/{msc|ug4}projects/upload). Further details are in the Webmark configuration for these forms.

Manual Runs

Examining crontab for the wwwrun user will reveal the up-to-date details for running the projsubs scripts. These will probably be something like the following:

  /liveroot/scripts/genhtcapabilities > /liveroot/conf/group.capabilities
  KRB5CCNAME=/tmp/krb5cc_projsubs
  MAILTO=rat-unit@inf.ed.ac.uk
  /usr/bin/kinit -k -t /etc/projsubs.keytab projsubs/goyle.inf.ed.ac.uk && /usr/bin/aklog

In summary, any year or project type can be regenerated manually with something along the lines of:

  wwwrun$ cd /liveroot/cgi/
  wwwrun$ export year=<20nn>
  wwwrun$ export ptype=(ug4|msc)
  wwwrun$ SCRIPT_NAME=/cgi-bin/submission_links_${ptype}.cgi ./submission_links_${ptype}.cgi

Additional options to the script control the output. Above generates the authenticated index page for the specified year(s). These are visible to all authenticated staff. They only contain links to the submitted project PDF and project materials (links to materials are only generated for the current year). Adding the mode=live option will generate the assessment page for the current year. This is only accessable by the projsubs admin group (relevant administrative staff, projects coordinator and external examiners) and additonally contains links to the Webmark mark returns as well as showing the actual marks. Using the public=yes option will generate the publically visible index pages and copy the project PDF documents. To regenerate https://project-archive.inf.ed.ac.uk/PTYPE/YEAR-outstanding.html, set public=outstanding. Only projects that meet necessary criteria are included in the publically visible index and cache. This is determined by reference to a data file regularly generated by Theon into AFS space: /afs/inf.ed.ac.uk/group/inf-db/generated/projsubs/studentlist.csv.

Webmark

Webmark data (data4wmk.csv) for the UG4 / MInf and MSc marking forms are generated by the above scripts. If this file is empty or missing, webmark forms will complain with the error, "Project Upload Directory data could not be retrieved: please try again or contact support". To regenerate these files, you can perform a manual run as above with mode=live and ptype set to ug4 (which includes Minf) or msc as appropriate;

 [...]
  wwwrun$ SCRIPT_NAME=/cgi-bin/submission_links_${ptype}.cgi mode=live ptype=(msc|ug4) ./submission_links_${ptype}.cgi

Project materials delivery - changing the current year.

Only the current year's project materials are accessible by default, and the "current year" is not established automatically as the timings may change from year to year. As submissions for the new year start, it will fall to Computing Support to change the definition of "current year" by editing the CGI script which delivers the files. Unfortunately this is currently a tedious manual process which involves a new RPM build each year. The process is as follows:

• check out the SVN dice repository for DICEComp:trunk/projsubs.
• edit the current_year variable in the files cgi/submission_links_ug4.cgi or cgi/submission_links_msc.cgi as appropriate.
• commit the changes.
• use lcfg-reltool to build a new release and pkgsubmit to distribute it.
  • (typically lcfg-reltool microversion; lcfg-reltool srpm; pkgforge submit -B inf srpm)
• update the dice/options/projsubs-server.h header to use the new release.

Hiding Submissions

To hide submissions from the list (accidental duplicate submissions, or revised versions, are common requests) simply navigate to the appropriate upload directory, find the project in question (the URL given by the retrieval system will help you find this) and touch hide in the appropriate directory.

Note: This only works for the new system at project-archive.inf.ed.ac.uk. To hide an old submission, simply manipulate the CVS directly in web/publications/thesis/offline/file.pdf and rename the file to something obscure. That change won't get undone as this stuff is no longer being re-generated by the database.

Making Submissions

Computing staff can make submissions as any other user on their behalf by using the ?as=USERNAME option to the submission CGI.

Previous Years (Public Archive)

No annual maintenance should be required for the project-archive document root (as opposed to projsubs; see above). The cron job generates pages for a sliding window of years, including only the current and previous two sessions. This does mean that the oldest years' project archives are not automatically updated. If these are lost (e.g. on rebuilding the service from scratch) they can be recreated by giving the appropriate year parameter to the appropriate script as above, see the LCFG header or crontab entries on the server for details. Data on all years is held and backed up in the master copy in the AFS group space.

Project Submission Removal

Submitted projects can have flags set on submission meaning they are not to be retained after the assessment processes have been completed (and correspondingly not made public). University record retention rules state that projects should be deleted six weeks after final assessment, except where an appeal is outstanding in which case they must be kept and only deleted six weeks after the appeal process has been completed. For example, the following could be used to identify those projects marked for special attention:

egrep "delete on" /afs/inf.ed.ac.uk/group/teaching/projects/ug4projects/upload/2017*/record

With "delete on" then the submitted project materials and the dissertation PDF MUST be deleted six weeks from final assessment (including any appeal). The wording on the submission form which the student must tick on submission states "one year after" however (not six weeks) so we follow that instead. An example of what to do is show below. Note that from 2017 submissions onwards the "assess" flag is deprecated and never used. The student either ticks to allow both dissertation and materials to be user in perpetuity or they tick for both to be deleted one year after assessment has completed.

# Run this at the end of the October of YEAR for submissions of YEAR - 1,
# in the example below it is October 2018 so we are deleting 2017 submissions
# Delete last years projects (dissertation and materials) where student
# indicated at submission that "I ask the School of Informatics not to publish
# or re-use either my dissertation or project materials. These should be used
# only for assessment of my project, and deleted 1 year after the Board of
# Examiners meeting."
cd /afs/inf.ed.ac.uk/group/teaching/projects/mscprojects/upload
for r in `grep "@delete on" -l 2017*/record`; do D=`dirname $r`; rm $D/msc_proj.dir $D/msc_proj.pdf; done
cd /afs/inf.ed.ac.uk/group/teaching/projects/ug4projects/upload
for r in `grep "@delete on" -l 2017*/record`; do D=`dirname $r`; rm $D/ug4_proj.dir $D/ug4_proj.pdf; done

The marks associated with projects are not to be retained after the assessment processes have been completed. University record retention rules state that project marks should be deleted six weeks after final assessment, except where an appeal is outstanding in which case they must be kept and only deleted six weeks after the appeal process has been completed. The following can be used to delete all mark files associated with the projects for a year.

# Run this at the end of November of YEAR for submissions after checking there are no outstanding appeals,
# in the example below it is November 2018
cd /afs/inf.ed.ac.uk/group/teaching/projects/mscprojects/upload
find . \( -name Mark.\*.csv -o -name Mark.\*.pdf \) -a -path ./2018\* -exec rm {} \;
cd /afs/inf.ed.ac.uk/group/teaching/projects/ug4projects/upload
find . \( -name Mark.\*.csv -o -name Mark.\*.pdf \) -a -path ./2018\* -exec rm {} \;

Submission Log

You can generate a quick submission log in CSV with something hacky like the following:

tf=$(mktemp /tmp/recordXXXXX) && \
  grep @ $(date +%Y)*/record > $tf && \
  join -o '1.3,1.4,1.5,1.6,1.7,1.8,2.3,2.4' <(grep '@author' $tf) <(grep '@submitat' $tf) \
    | sed -e 's/^\([^[]*\) \+\[\([^]]*\)\] \+\(.*\)$/"\1","\2","\3"/' > ./submitlog.csv && \
  rm $tf

However a nice python module to parse these files is available if you require higher confidence in the data.

Things that can go wrong with the submission forms, UG4/5 and MSc

Operational experience shows that project submissions tend to come all at once. These are things to look out for:

• Check the AFS quota of the submission directories here /afs/inf.ed.ac.uk/group/teaching/projects
• Check that the form states the current maximum upload size (2GB, not 4GB as previously stated at the moment)
• Check projsubs.inf.ed.ac.uk for temporary partition space. If this runs out, the forms fail to submit.
  • This can be found at project-submit.inf.ed.ac.uk:/var/tmp (as of 2018, mounted as a tmpfs partition).

-- GrahamDutton - 19 Mar 2014