Final Project Report for Project 164 - Prometheus AFS PTS Conduit


The initial goal of this project was to produce the conduit, data store and associated code which would allow the AFS PTS database to be synchronised with the main Prometheus Olympus data store. This would allow users to be added automatically to the PTS database when they appeared in Prometheus, rather than being added by hand. The scope of the project was soon enlarged to encompass the synchronisation of both the PTS and the VL databases, allowing for the automatic creation and deletion of home directories but the name of the project was never changed. It was felt that it would be beneficial for members of the CO community outwith the main Prometheus development team to create conduits as a means of promulgating knowledge of the workings of Prometheus to a wider audience and so this project was assigned to the Services Unit and in particular myself.

Initial progress

The project began in T1 2010 with the aim of having the conduits and associated data stores in service by the start of the 2010-2011 academic year. The need to acquire knowledge of both the internal workings of Prometheus and Perl based Object Orientated programming mean that initial progress was slow but as necessary knowledge was acquired, the pace accelerated and the deadline of September 2010 was met successfully.

Finished? Not Quite

Although the PTS and VL conduits and data stores were in place and working satisfactorily, there still remained the small matter of the test suites before the project could be signed off. Each Prometheus conduit and data store is required to have a test suite used to test that the conduit or data store is working as expected. Changing the contents of the actual databases associated with the data stores would be highly undesirable and so some way of emulating the behaviour of the underlying database, without changing the code path followed in the conduit or data store, must be found. Most Prometheus data stores front what are at heart some variety of LDAP database and it is a relatively simple matter to create and populate a test LDAP server for use with the test suites in these cases. Bringing up a test AFS cell is a far more daunting prospect and after much research and cogitation, it was decided that the only practical solution to this problem was to use the Perl Test::Mock modules to emulate calls to the Perl AFS:: module. This took a considerable amount of effort, exceeding that which had been required for the writing of the conduits and data stores in the first place.

Since the conduits and data stores. were in place and working, the writing of the test suites was treated as a lower priority task. This added to the total effort required to complete the project since there was a certain amount of refreshing of knowledge required each time the task was taken up again. In hindsight, it would have been far better to give the writing of the test suites the same priority as the writing of the conduits and data stores received and get the project finished as quickly as possible.

Nevertheless, by the end of 2013, the job was done. The test suites had been written, the code submitted to Gerrit and approved, all seemed set fair to finally get the project signed off. Except for one small fly in the ointment. When the project began, there was a choice of two Perl modules to use to talk to the AFS file system, AFS:: and AFS::Command. The former interfaced directly with the AFS API, the latter worked by executing the relevant AFS command line commands and storing the outputs from those commands into a variety of structures. There was overwhelming agreement in the AFS community that the AFS:: module was the one to use and so that is what happened. By the end of 2013, popular opinion had swung round 180 degrees. The AFS:: module had not been updated for some time and it was clear that forthcoming kernel changes would break the module completely. By contrast, the AFS::Command module remained well supported and regularly updated. There was no choice but to switch the conduits, data stores and test suites (which, it will be recalled went to great lengths to emulate the calls to the various AFS:: methods) to use AFS::Command.

As before, the conduits and data stores were completed and deployed in a relatively short timeframe while the test suite took rather longer. This was not because of any intrinsic difficulty in switching the test suites to AFS::Command, indeed it became clear that from the test suite point of view, it would have made much more sense to have used AFS::Command in the first place. This is because the AFS::Command module offers the facility to specify alternative commands to run in place of the AFS binaries such as VOS and PTS. It is therefore easy to write and make use of emulated commands. The test suites have now been written and all code committed to Gerrit and successfully reviewed.

Time taken

It should be noted when considering the time taken for this project that a significant part of this time was spent in re-familiarisation with the project. Over the 6 year period, a total of 624 hours was spent on the project, the per year split being as follows

Year Effort (hours)Sorted descending
Total 624
2012 218
2010 164
2013 121
2014 43
2011 42
2015 36

Lessons Learned

  • Just get it finished

Once the conduits and data stores were in service, it seemed that there were alway higher priority things to work on than the test suite. Maintaining focus on the project would have saved a lot of effort in the long run.

  • Choose your modules carefully

It seems clear now that the AFS::Command should have been used from the start. It should have been obvious that using this module would greatly ease the writing of the test suites

-- CraigStrachan - 9 Feb 2016

This topic: DICE > WebHome > DevelopmentMeeting > FinalProjectReports > FinalProjectReport164
Topic revision: r5 - 12 Feb 2016 - 09:42:58 - CraigStrachan
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies