Submit for Computing Staff

This page covers computing support processes relating to the practical submission system and is incomplete.

  • Student instruction regarding submit is provided alongside coursework instructions or directly by computing support.
  • Teaching Staff should refer to the Practical Submission guide for advice on processing submissions.
  • Computing staff should see also the LabExamsProc page for structural information though note that the submission root directory is different for the two processes.

New for 2019/20 -- opt-in submit

Newer mechanisms for course work submission are available so from session 2019/20 the use of submit will be on an opt-in basis, and from session 2020/21 the expectation is that it will be removed altogether.

In order for a course to appear at least one piece of associated work must be flagged appropriately. At the moment this needs the course_work_3g.feedback column to be set to true on that piece of work, then after the usual 24hr delay (unless forced) submit will list the course and just that piece of work will be available for submission. We should only do this once the course lecturer has requested that their course needs to continue to use submit through the ITO - direct requests to us should be bumped to the ITO to deal with first.

In due course the feedback column will be directly exposed on the Course desktop on Theon and the ITO will then be able to manage this themselves. Until then computing support with Theon access can use the tool ~gdutton/bin/theon_submit_enable which automates some of the process,.

Differences between submit and examsubmit

Almost all of the information on this page refers to both commands. The only differences are:

  • location: different home directories containing independently checked out copies (note the CVS repository supplying them is the same)
  • automation: submit data is updated automatically, examsubmit must be updated using ./syncdata

Submit Control Files

In the submit home directory, the following files are important to the process:

  • exnamefile - updated nightly by portal conduit TP067_Submit_Exercises (mastered in Theon)
  • modgrp - updated nightly by portal conduit TP068_Submit_Groups (mastered in Theon)
  • master - updated nightly by portal conduit TP066_Submit_Master (mastered in Theon)
  • accnames - updated manually using the set_submit_file_names command (mastered in CVS).

Course/Exercise are controlled by the master and exnamefile files. Before course and exercises can be used for submission purposes, the ITO must have created the necessary data in the School Database (if they have not done so already).

Theon mastered data can be examined using the CourseWork desktop: the Course/Acronym value determines the course name, and the Work/Tag value determines the exam title for the purposes of submit. Remember that resits belong to the same session as the previous diet of exams, and not the "current" session, which by August / Sep will have rolled over. Theon uses the concept of "overrun session" to track this.

Acceptable Filenames: Each exam and type restricts submissions to a set of filenames, set by the course organiser and subject to change each year. The filenames must subsequently by updated by command. See the accnames section for details.

Synchronisation and Data updates

These four files are generated by the following processes:

(click for bigger)

General advice

This applies to all of the above files:

  • Read any README files in the submit directories. Field notes have been placed here in an attempt to fix gaps in the process as requirements have changed over time!
  • Remember that all of these control files are stored in and updated via a CVS repository, whose ultimate location can be determined from the CVS subdirectory.
  • Don't assume that any files are hand-editable. In some cases this can cause more trouble than it's worth even for a quick fix!
  • Updating from CVS requires the use of the ./syncdata command, which is automated for submit but manual for examsubmit. This must be run on the CVS server.

syncdata

Though the CVS repository is updated periodically, as shown, the syncdata script must be run on the CVS server for the changes to take effect. This script updates the local CVS working copy for submit to see.

Coursework submit data is updated hourly on the CVS server automatically; it's LCFG managed (and owned by "gdutton" temporarily while we work out how to give the appropriate group to a service account) and invoked by the name .datafiles. This automated run emails rat-unit+submit, so there should not normally be any need to run it manually. However computing staff can harmlessly run syncdata on the CVS server to reassure themselves that all is well. This script will also warn of any uncommitted modification: remember that uncommitted changes will be overwritten, or can cause conflicts in the future.

Exam submit data is stored in an alternative submit root, and this root must always be updated manually using the syncdata script in the exam root, by computing staff, in advance of an exam. In any event if the ITO have had to make last-minute changes to a practical, or any changes time prior to an exam, it will be necessary to manually sync the submission data. This can be done by logging into cvs.inf and running ./syncdata in the appropriate submit or examsubmit directory.

In either case above, you must have the cvs_submit role. Check the changes have been made by looking for the course and exercise in the master file. Additionally, test runs of submit (including prodding its built-in bash tab completion) or examsubmit will reveal what's been configured.

accnames

The accnames file is built exclusively using the set_submit_file_names command, which checks out the oknames subdirectory from the CVS repository holding submit's data. Operational documentation on this command can be found in the and in the man page (man set_submit_file_names).

This script simply checks files out of CVS, launches an editor, and commits the changes. You must have the cvs_submit role for this to work. If you have made any changes it will be subsequently necessary to manually sync the submission data running the syncdata process described above. Check that these changes have been made by examining the file examsubmit/accnames.

However this process can corrupt itself, and in any event the directory tree is (still!) full of year/module/ garbage from the "old regime" that occasionally gets in the way and prevents new filenames from being applied. If this is the case, visit the CVS server for the submit home files and check out oknames into the home directory (or elsewhere if you insist) with:

# cvs co oknames

From here you can rm and cvs rm any conflicting directories or files. Once you've done this, check back in, run set_submit_file_names again, and finally run ./syncdata to manually update the accnames file properly.

Conflicts

If manual edits have been made to the files, they might end up in a CVS conflict state. This can cause everything from missing exercises to submit hangs. If in doubt, check the CVS status and restore from a repository copy!

Manual Refresh

If you need to force an update made by the ITO in a hurry, you'll need to push through the entire process. It's not encouraged but it might be better than no submission, or than manually hacking files. With reference to the diagram above, refreshing involves to following steps in order:

Refresh Conduits

On the portal server, assume apache. Examine the environment used to execute conduits with crontab -l and use export to set all values (e.g. PORTAL_ROOT, etc.), before running:

  portal RUN <conduit>

CVS Push

Having run the conduits, you can push the data to the CVS server. Again on the portal server, assume postgres and again examine and set the execution environment for the submitdata2cvs.sh script. Just run this script without arguments (and watch for errors). After a bit of a shake up, currently (Oct 2016) this is what you need to run:

PORTAL_ROOT=/disk/data/portal /usr/lib/hypatia/portal/support/submitdata2cvs.sh

CVS Update

The final step is to visit the CVS server and perform a CVS update as described update.

Start / End of Year process

Assuming outgoing session xx-yy and current session yy-zz (i.e. yy is the current calendar year...):

As root on the appropriate fileserver:

# cd /disk/ptn197/aisubmit
These steps just help keep the example obvious at all times -- you could just use the year numbers without resorting to these variables.
# export OLD="xx-yy"
# export NEW="yy-zz"
Make the new directory:
# mkdir submissions.$NEW
Fix permissions, assuming the old ones were okay:
# chmod --reference=submissions.$OLD submissions.$NEW
# chown --reference=submissions.$OLD submissions.$NEW
Check everything's consistent between old and new directories:
# ls -ld submissions*
Now update to the new year; this commits the change:
# ln -sin submissions.$NEW submissions
Archive the submit log, and safely create a new one:
# touch submit-log.$NEW
# chown --reference submit-log submit-log.$NEW
# chmod --reference submit-log submit-log.$NEW
# mv submit-log submit-log.$OLD && mv submit-log.$NEW submit-log
Archive everything:
# tar -zcf ARCHIVE/submissions.17-18.tar.gz submissions.17-18

Now, sanity check the contents of ARCHIVE before deleting submissions.xx-yy and submit-log.xx-yy.

Troubleshooting

If you've left it too late and new submissions are piling into the previous year's submissions folder, fear not. A simple procedure to filter out the old ones would be something along the lines of:

# cd /disk/ptn197/aisubmit/
# cp -ar submissions.xx-yy/ ARCHIVE/submissions.xx-yy
# mkdir submissions.yy-zz/
# rm submissions && ln -s submissions.yy-zz submissions
# cd submissions.xx-yy/
# find . -mindepth 3 -maxdepth 3 -type d -newermt 'start/of/teaching'; do mv $d ../submissions/yy-zz/$d
Topic revision: r18 - 30 Jul 2019 - 12:29:21 - GrahamDutton
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies