Final Report : remove obsolete deps for LCFG client (274)

The Goal

The primary goal for this project was to remove the LCFG client dependency on obsolete W3C::SAX XML modules by switching to the XML::LibXML module which is based on libxml. The plan was to create a new object-oriented API for accessing and manipulating the profile/component/resource and packages information for an LCFG client profile. The intention being that, whilst initially the API would only be used for the rdxprof client process, it would be designed in such a way that it could be used by all LCFG software which needs to be able to read and write profile data in the various supported formats (both client and server side). Reducing the numerous separate implementations to a single code base would mean that as well as improving maintainability this would open the way to future new feature development since any new features would only need to be implemented once.

The Project

The Final Products

This project has produced new C and Perl core libraries and the LCFG client has been updated to use those new libraries:

1. Core C libraries

2. Core Perl libraries

3. Updated client

Other Benefits

Improvements to code quality
C is a lot less forgiving about vaguely defined behaviour than Perl so as part of the process of writing new C code many features needed to be better defined. This process also revealed a few bugs (that are now fixed) which could lead to corruption or misinterpretation of resource data on the client side.

Improvements to profile data quality
The testing phase uncovered many genuine bugs in various component schemas which are not being caught by the current LCFG compiler. In particular it has become clear that the resource tag validation in the LCFG server needs to be improved, of course, the hope would be that one day it will be converted to using the new core libraries.

Personal benefits
This project has given me many opportunities to enhance my programming knowledge and skills. I have gained a much deeper knowledge of the C programming language and have learnt a lot about combining C with Perl by using XS code. I have also learnt how to create parsers using the flex and bison tools. Along the way I have become familiar with tools such as valgrind which is used to check for memory leaks and the Coverity static analysis tool.

Bootstrapping improvements
Whilst porting some of the LCFG utilities (e.g. qxprof, qxpack) to the new libraries a number of circular build dependency issues were identified which make porting to new platforms difficult. To improve bootstrapping on new platforms (e.g. EL8) various scripts and Perl modules have been moved between packages.

General Thoughts

Writing documentation for a large API can be tedious but I also found it to be a very useful code review process. For example, once you notice that there are 4 different functions which can be used to serialise resource information in the various required formats you soon realise that the APIs should be consistent so that it is easy to swap from one to the other. It is also useful for spotting areas where common code can be shared and reused.

Writing a large library in isolation would not be a good idea. In this project the C and Perl libraries were developed in parallel, the Perl layer "uses" the functionality provided by the C library and this ensures that it has a sensible and easy to use API. Similarly the Perl libraries were much improved through the work of porting the standard qxprof, sxprof and qxpack utilities. I hadn't originally planned to port those tools but the work has proved to be very useful. In my opinion, creating a good API needs more than just an awareness of the original stated requirements, it needs real code which actually uses the functionality. That could be in the form of a test suite but tests are often only focussed on small parts of a library. Testing small chunks of code is important but porting a whole script gives the API a much more complete work out and provides an understanding of how others would approach using a new library.

The Future

In no particular order here are some thoughts on potential future work:

  • Finish porting Perl code which uses LCFG::Resources over to the new LCFG::Client::Resources module
  • Introduce a new LCFG::Client::Component framework to replace LCFG::Component. That would use the new APIs and also incorporate many other improvements and ideas which have been suggested over the years.
  • Port the LCFG server to using the libraries. It might be possible to do parts without doing a full rewrite (e.g. the package list handling). Doing this would allow us to start developing new server functionality again.
  • Improve the type handling for more power/flexibility. Currently it's not possible to do this all in the C layer, most validation is still done in the Perl layer. Adding new types is just done by limiting possible strings with perl regexps.
  • Switch component resource data structure from linked list to hash for greater speed. Doing this for the packages data structure proved to be a big improvement and it would probably be essential for porting the LCFG server.
  • Add support for diffing lists of packages.
  • Port updaterpms to the new libraries.
  • Introduce a python framework. It should be possible to wrap the C library with python modules.


-- StephenQuinney - 23 Nov 2017

Edit | Attach | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r3 - 24 Nov 2017 - 12:05:47 - StephenQuinney
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies