Conduits Framework

The Conduits Framework used in Hypatia bundles individual conduits and header files, the automated conduit generation of web pages and other data files via CVS/NFS/Email (called outgoing, never used directly but rather through the Portal) and the automated processing of external data feeds from Email (called incoming). The conduits framework is part of the Hypatia repository and packaged with it. The framework can run on any client and can bootstrap itself entirely from suitable configuration information. This document describes the framework. Things subject to change are italicised.

Directory Structure

conduits

Each file in this directory contains the implementation of a conduit. Almost all are implemented in gurgle but there are a few bash types as well. The conduit files are generally named FNNN where F is the uppercase letter code representing the type of output the conduit has and NNN is the unique reference number for the conduit. For example A002 is conduit 002 generating ASCII Plaintext. For a list of internally understood output type codes see conduits/outgoing/forms.txt. Conduits do not have to be named following this convention, and many are not (such as those in the Portal).

includes

Header files containing definitions which are common to multiple conduits (e.g. some low level conduit name handling functions and blocks of text for each file output type). These are included by conduits if needed.

supports

Support files for conduits. Generally ASCII text database input sources. Data which is static and not otherwise available. These files contain small amounts of legacy data that is not (or was not at the time) represented directly in the database.

outgoing

Scripts and configuration files implementing the outgoing automated conduit processing. This process is externally driven by managed crontab entries.

incoming

Scripts and configuration files implementing the incoming automated conduit processing, described in more detail on ConduitsIncoming. This process is externally driven by managed mail filtering for example.

Installed Structure

The framework is packaged as an RPM and deployed by the relevant hypatia header files. The installed structure is as below.

  /usr/lib/hypatia/conduits/
    conduits/
    includes/
    supports/
    incoming/
    outgoing/
  /etc/conduits/
    DBHOST
  /usr/bin/incoming
  /usr/bin/outgoing
  /usr/bin/runcond

The structure under /usr/lib/hypatia/conduits is as in the repository, except that configuration files are moved out into /etc/conduits and linked to. The DBHOST configuration file sets the database server. By default in the deployed package this is empty meaning all the scripts and commands will reference a local postgresql server. In LCFG however DBHOST is configured to point at the School Database server meaning all the scripts and commands will reference this. The commands in /usr/bin are the primary script interfaces to the incoming=/=outgoing processes and runcond for manual conduit execution.

Instances

While /usr/lib/hypatia/conduits is the master copy, it is not used in live script runs. Each of incoming, outgoing and runcond create automatically an instance copy (built bit by bit as needed) of /usr/lib/hypatia/conduits when they run. This is to facilitate easy testing, bootstrap on restart and testing of modifications as a non-system user prior to re-building the package and deployment.

While runcond being a user utility builds the instance in the current users home directory, incoming and outgoing being system utilities build their instance in temporary space (currently /dev/shm/incoming and /dev/shm/outgoing respectively to get the performance benefits on an in-memory file system).

It is possible to configure the actual instance location used to override the default by setting an environment variable prior to running. For incoming the environment variable containing the instance path is INCOMING and for outgoing it is OUTGOING. For runcond the default is ~/.conduits which can be changed by setting MY_CONDUITS.

An instance copy has a directory structure much like the installed structure except that the conduits, includes and supports directories are empty at the outset. They will be populated on demand by copying files as needed. If the system installed files are upgraded (and newer) they will be re-copied when next needed. An instance directory structure may also have additional directories for generated output (outputs, holding, and logs for log files for example).

An instance copy can always be safely deleted in its entirety as it can always be automatically re-created on running one of the primary interface scripts.

Changing Conduits

There should be no need to alter any of the conduits at this layer. All changes would normally be made to Portal conduits.

-- TimColles - 30 Nov 2018