Completion Report for School Database Revamp: Phase 1B & 2B

Despite the description the scope of this project and final report is now confined to the re-implementation of the back-end technology for managing changes in the School Database structure, UI interface and Portal reporting (the rest has been done). The existing technology was originally developed 20 years ago, and while during the re-factoring to Theon in 2010 had to be updated, this was only done minimally as the basic structure and workflow was unchanged. In order to support new features at that point the existing structures were re-used by forcing changes into them - re-using redundant fields or placing additional interpretations on existing fields making for a baroque and complex change process, for example see: Desktop Data, Coupler Data and Change Management.

Furthermore crude script wrappers were built around the existing technology to map the output to that needed by Theon which were both slow and hard to test/develop with. The technologies did not lend themselves easily to multiple people using them and the base data was not version controlled. This led to a development cycle meaning small changes could take hours to make, including multiple cycles of deployment (usually live), test and fix. Naturally this discouraged change, and resulted in changes being batched up into large release chunks to minimise the pain.

The intention of this project was to replace the old technology which something which would be simpler and significantly faster to use, automate as much as possible and streamline the development cycle and ultimately allow wider usage (be able to devolve change management).

This has largely been achieved, although note that in the end integration/replacement of the Portal technology was de-scoped - partly because it would have added considerably to the project duration but more because the Portal technology, old as it was, was already suitably structured for wider re-use with a better development cycle and also because the direction to take and technology to use was not at all clear and would have in itself constituted a considerable amount of effort purely to investigate.

In summary the original aims which have all been met are:

  • Replace the twenty year old multiuser single master 'TECDB'
  • Replace the surrounding cobbled together 'TECDB' build/test framework
  • Speed up the development cycle
  • Introduce a local simplified development environment (production server not used for testing)
  • Improved and better integrated 'Desktop' configuration
  • Improved and better integrated 'Coupler' configuration
  • Introduce a mechanism for 'schema' version control
  • Remove all legacy infdb specific aspects, make solution completely db agnostic
  • Improved automated documentation generation
  • Provide full technical documentation
  • Provide a 'schema' upgrade mechanism
    • this has only been partially met - demonstrable and usable, but not fully comprehensive, functionality

In summary the original aims which have not been met are:

  • Automated package and distribution management
    • this was implemented and is still included in the tool kit and does work in principle however it needs refinement, is largely undocumented and is not recommended for use
  • Integrated Portal/Reporting
    • as mentioned above this was entirely de-scoped

Effort

This project consumed 22 weeks of effort. It was originally costed at about ten weeks. Some of this effort (maybe about three weeks) can be attributed to porting Theon and the School Database service to SL7 as well as making specific compatibility changes to the School Database itself. Another reason for additional effort was due to the spread out nature of the project where insignificant time was committed to do the project in a block so it was done in multiple stages over three years: initial design, documentation, first pass (prototype) implementation, design changes, re-implementation, final testing, documentation. Each stage was done after a significant time elapsed from the previous requiring re-familiarisation effort - not until the re-implementation stage was significant time blocked out to focus on this project and this was when the majority of progress was made and effort expended. Although it can also be said that having elapsed time between each stage allowed for reflection and re-evaluation, probably improving the final design as a result.

Future Work

There is a long TODO list. However these can all be considered functional enhancements, the primary feature set is done and any known critical issues have been addressed. There is a case for further documentation, but again the core stuff is there. So its really polishing and we conclude it is not worth holding this project open waiting on this (potentially near endless) list of things still to be done.

Conclusion

We have to now quantify whether the effort was justified. Certainly the development cycle is massively improved - what took hours now takes minutes and changes can be repeatedly revised and tested on a local development host away from the live production service. Simple changes are no longer blocked, previously this could be for months, waiting on a suitable release window time frame but can be rolled out immediately. The underlying technology and framework is free of twenty year old baroqueness and far more robust. Developers now have a consistent and well documented front-end and tool kit to manage all changes. All changes are version controlled and multiple developers can be working on changes at the same time. Making changes, in particular changes to the UI desktops or the Coupler (feed management) is vastly easier than before. There remain some rough edges, in particular relating to direct upgrade.

The original lifespan of the School Database service was put at ten years in order to justify the investment cost. We are now into the eighth year of deployment and Theon will clearly be in use for a further two years. So the change management will provide at least two years worth of improved development cycle. Five years would have been preferable to justify the cost, but it is quite likely that Theon will go on for another five years, although probably in a reducing capacity (certainly as regards teaching administrative support).

-- TimColles - 10 Oct 2017

Topic revision: r2 - 12 Oct 2017 - 08:03:15 - TimColles
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies