Note this is not the
practical submission system "submit": this is describing the Honours and MSc project reports/dissertations and project materials.
User-facing documentation can be found at
http://computing.help.inf.ed.ac.uk/msc-project-submission.
Introduction
This page documents the service provided by
https://projsubs.inf.ed.ac.uk/,
http://project-archive.inf.ed.ac.uk/ and
https://project-submit.inf.ed.ac.uk/. The former holds internally visible only and restricted links to the UG4/5 projects and MSc dissertations, along with associated submission records as stored by the
UG4/5 and
MSc project submission web forms and associated marking records as stored by the
UG4,
UG5 and
MSc Webmark marking forms. The latter holds the publically visible links to UG4/5 projects and MSc dissertations that have been approved for wider publication (some only within the School however, i.e. requiring authentication).
Structure
The
projsubs
service is managed entirely by LCFG configuration using the
dice/options/projsubs-server.h
header file. This includes a local
projsubs
package (held in the SVN
dice
repository) which contains the CGI scripts that are used both for submissions and also for generating the link pages and public archive. The Webmark forms are held separately under the
Webmark system.
The server holds no
master data itself; this is all held in AFS group space (
group/teaching/projects/{msc|ug4}projects
).
The server does hold:
- live generated link pages (
YEAR-current.txt
for projsubs)
- a cache of project documents (
YEAR.html
for both projsubs and the public archive).
The caches can all be re-generated on demand by running the CGI scripts manually as described below.
Access
- Only members of staff (
staff
) should have access to the authenticated projsubs
pages.
- Access granted by
web/wwwinf/staffonly
capability.
- In addition, the scripts prevent staff access to project materials for previous years' projects where the submitting student has not granted permission for their continued use (this allows support to perform manual deletion/archiving when appropriate, while enforcing the letter of the agreement).
- Only some members of administrative staff, the projects coordinator and external examiners (
examiner-ext
) should have access to the authenticated current assessment year projsubs
page (YEAR-current.txt
).
- Access granted by
web/wwwinf/projsubs
capability.
- URL for external examiners should be
projsubs/cgi-bin/submission_index.cgi?file=/ug4/YEAR-CURRENT.txt
- Access to the
project-archive
pages is open to all anonymously.
As mentioned above, the actual HTML pages (containing project listings) are generated by the CGI scripts held in
cgi/
. They are run periodically by a cronjob belonging to
wwwrun
.
Marking Information
Assessment documents are submitted by the webmark forms into the appropriate upload directory in the AFS group space (
group/teaching/projects/{msc|ug4}projects/upload
). Further details are in the Webmark configuration for these forms.
Manual Runs
Examining crontab for the
wwwrun
user will reveal the up-to-date details for running the
projsubs
scripts. These will probably be something like the following:
/liveroot/scripts/genhtcapabilities > /liveroot/conf/group.capabilities
KRB5CCNAME=/tmp/krb5cc_projsubs
MAILTO=rat-unit@inf.ed.ac.uk
/usr/bin/kinit -k -t /etc/projsubs.keytab projsubs/goyle.inf.ed.ac.uk && /usr/bin/aklog
In summary, any year or project type can be regenerated manually with something along the lines of:
wwwrun$ cd /liveroot/cgi/
wwwrun$ export year=<20nn>
wwwrun$ export ptype=(ug4|msc)
wwwrun$ SCRIPT_NAME=/cgi-bin/submission_links_${ptype}.cgi ./submission_links_${ptype}.cgi
Additional options to the script control the output. Above generates the authenticated index page for the specified year(s). These are visible to all authenticated staff. They only contain links to the submitted project PDF and project materials (links to materials are only generated for the current year). Adding the
mode=live
option will generate the assessment page for the current year. This is only accessable by the projsubs admin group (relevant administrative staff, projects coordinator and external examiners) and additonally contains links to the Webmark mark returns as well as showing the actual marks. Using the
public=yes
option will generate the publically visible index pages and copy the project PDF documents. To regenerate
https://project-archive.inf.ed.ac.uk/PTYPE/YEAR-outstanding.html, set
public=outstanding
. Only projects that meet necessary criteria are included in the publically visible index and cache. This is determined by reference to a data file regularly generated by Theon into AFS space:
/afs/inf.ed.ac.uk/group/inf-db/generated/projsubs/studentlist.csv
.
Webmark
Webmark data (
data4wmk.csv
) for the UG4 / MInf and MSc marking forms are generated by the above scripts. If this file is empty or missing, webmark forms will complain with the error, "
Project Upload Directory data could not be retrieved: please try again or contact support". To regenerate these files, you can perform a manual run as above with
mode=live
and
ptype
set to
ug4
(which includes Minf) or
msc
as appropriate;
[...]
wwwrun$ SCRIPT_NAME=/cgi-bin/submission_links_${ptype}.cgi mode=live ptype=(msc|ug4) ./submission_links_${ptype}.cgi
Project materials delivery - changing the current year.
Only the current year's project materials are accessible by default, and the "current year" is not established automatically as the timings may change from year to year. As submissions for the new year start, it will fall to Computing Support to change the definition of "current year" by editing the CGI script which delivers the files.
Unfortunately this is currently a tedious manual process which involves a new RPM build each year. The process is as follows:
- check out the SVN dice repository for DICEComp:trunk/projsubs.
- edit the
current_year
variable in the files cgi/submission_links_ug4.cgi
or cgi/submission_links_msc.cgi
as appropriate.
- commit the changes.
- use
lcfg-reltool
to build a new release and pkgsubmit
to distribute it.
- (typically
lcfg-reltool microversion; lcfg-reltool srpm; pkgforge submit -B inf srpm
)
- update the
dice/options/projsubs-server.h
header to use the new release.
Hiding Submissions
To hide submissions from the list (accidental duplicate submissions, or revised versions, are common requests) simply navigate to the appropriate upload directory, find the project in question (the URL given by the retrieval system will help you find this) and
touch hide
in the appropriate directory.
Note: This only works for the new system at project-archive.inf.ed.ac.uk. To hide an old submission, simply manipulate the CVS directly in
web/publications/thesis/offline/file.pdf
and rename the file to something obscure. That change won't get undone as this stuff is no longer being re-generated by the database.
Making Submissions
Computing staff can make submissions as any other user on their behalf by using the
?as=USERNAME
option to the submission CGI.
Previous Years (Public Archive)
No annual maintenance should be required for the
project-archive
document root (as opposed to
projsubs
; see above). The cron job generates pages for a sliding window of years, including only the current and previous two sessions. This does mean that the oldest years' project archives are not automatically updated. If these are lost (e.g. on rebuilding the service from scratch) they can be recreated by giving the appropriate year parameter to the appropriate script as above, see the LCFG header or crontab entries on the server for details. Data on all years is held and backed up in the master copy in the
AFS
group space.
Project Submission Removal
Submitted projects can have flags set on submission meaning they are not to be retained after the assessment processes have been completed (and correspondingly not made public).
University record retention rules state that projects should be deleted six weeks after final assessment, except where an appeal is outstanding in which case they must be kept
and only deleted six weeks after the appeal process has been completed. For example, the following could be used to identify those projects marked for special attention:
egrep "delete on" /afs/inf.ed.ac.uk/group/teaching/projects/ug4projects/upload/2017*/record
With "delete on" then the submitted project materials and the dissertation PDF
MUST be deleted six weeks from final assessment (including any appeal). The wording
on the submission form which the student must tick on submission states "one year after" however (not six weeks) so we follow that instead. An example of what to do
is show below. Note that from 2017 submissions onwards the "assess" flag is deprecated and never used. The student either ticks to allow both dissertation and materials
to be user in perpetuity or they tick for both to be deleted one year after assessment has completed.
# Run this at the end of the October of YEAR for submissions of YEAR - 1,
# in the example below it is October 2018 so we are deleting 2017 submissions
# Delete last years projects (dissertation and materials) where student
# indicated at submission that "I ask the School of Informatics not to publish
# or re-use either my dissertation or project materials. These should be used
# only for assessment of my project, and deleted 1 year after the Board of
# Examiners meeting."
cd /afs/inf.ed.ac.uk/group/teaching/projects/mscprojects/upload
for r in `grep "@delete on" -l 2017*/record`; do D=`dirname $r`; rm $D/msc_proj.dir $D/msc_proj.pdf; done
cd /afs/inf.ed.ac.uk/group/teaching/projects/ug4projects/upload
for r in `grep "@delete on" -l 2017*/record`; do D=`dirname $r`; rm $D/ug4_proj.dir $D/ug4_proj.pdf; done
The marks associated with projects are not to be retained after the assessment processes have been completed. University record retention rules state that project marks
should be deleted six weeks after final assessment, except where an appeal is outstanding in which case they must be kept and only deleted six weeks after the appeal
process has been completed. The following can be used to delete all mark files associated with the projects for a year.
# Run this at the end of November of YEAR for submissions after checking there are no outstanding appeals,
# in the example below it is November 2018
cd /afs/inf.ed.ac.uk/group/teaching/projects/mscprojects/upload
find . \( -name Mark.\*.csv -o -name Mark.\*.pdf \) -a -path ./2018\* -exec rm {} \;
cd /afs/inf.ed.ac.uk/group/teaching/projects/ug4projects/upload
find . \( -name Mark.\*.csv -o -name Mark.\*.pdf \) -a -path ./2018\* -exec rm {} \;
Submission Log
You can generate a quick submission log in CSV with something hacky like the following:
tf=$(mktemp /tmp/recordXXXXX) && \
grep @ $(date +%Y)*/record > $tf && \
join -o '1.3,1.4,1.5,1.6,1.7,1.8,2.3,2.4' <(grep '@author' $tf) <(grep '@submitat' $tf) \
| sed -e 's/^\([^[]*\) \+\[\([^]]*\)\] \+\(.*\)$/"\1","\2","\3"/' > ./submitlog.csv && \
rm $tf
However a nice python module to parse these files is available if you require higher confidence in the data.
Things that can go wrong with the submission forms, UG4/5 and MSc
Operational experience shows that project submissions tend to come all at once. These are things to look out for:
- Check the AFS quota of the submission directories here
/afs/inf.ed.ac.uk/group/teaching/projects
- Check that the form states the current maximum upload size (2GB, not 4GB as previously stated at the moment)
- Check projsubs.inf.ed.ac.uk for temporary partition space. If this runs out, the forms fail to submit.
- This can be found at
project-submit.inf.ed.ac.uk:/var/tmp
(as of 2018, mounted as a tmpfs partition).
--
GrahamDutton - 19 Mar 2014