Conduits Framework
The Conduits Framework used in Hypatia bundles individual conduits and header files, the automated conduit generation of web pages and other data files via CVS/NFS/Email (called
outgoing
, never used directly but rather through the Portal) and the automated processing of external data feeds from Email (called
incoming
). The conduits framework is part of the Hypatia repository and packaged with it. The framework can run on any client and can bootstrap itself entirely from suitable configuration information. This document describes the framework.
Things subject to change are italicised.
Directory Structure
conduits
Each file in this directory contains the implementation of a conduit. Almost all are implemented in
gurgle
but there are a few
bash
types as well. The conduit files are generally named
FNNN
where
F
is the uppercase letter code representing the type of output the conduit has and
NNN
is the unique reference number for the conduit. For example
A002
is conduit
002
generating ASCII Plaintext. For a list of internally understood output type codes see
conduits/outgoing/forms.txt
. Conduits do not have to be named following this convention, and many are not (such as those in the Portal).
includes
Header files containing definitions which are common to multiple conduits (e.g. some low level conduit name handling functions and blocks of text for each file output type). These are included by conduits if needed.
supports
Support files for conduits. Generally ASCII text database input sources. Data which is static and not otherwise available. These files contain small amounts of legacy data that is not (or was not at the time) represented directly in the database.
outgoing
Scripts and configuration files implementing the
outgoing
automated conduit processing. This process is externally driven by managed crontab entries.
incoming
Scripts and configuration files implementing the
incoming
automated conduit processing, described in more detail on
ConduitsIncoming. This process is externally driven by managed mail filtering for example.
Installed Structure
The framework is packaged as an RPM and deployed by the relevant hypatia header files. The installed structure is as below.
/usr/lib/hypatia/conduits/
conduits/
includes/
supports/
incoming/
outgoing/
/etc/conduits/
DBHOST
/usr/bin/incoming
/usr/bin/outgoing
/usr/bin/runcond
The structure under
/usr/lib/hypatia/conduits
is as in the repository, except that configuration files are moved out into
/etc/conduits
and linked to. The
DBHOST
configuration file sets the database server. By default in the deployed package this is empty meaning all the scripts and commands will reference a local postgresql server. In LCFG however
DBHOST
is configured to point at the School Database server meaning all the scripts and commands will reference this. The commands in
/usr/bin
are the primary script interfaces to the
incoming=/=outgoing
processes and
runcond
for manual conduit execution.
Instances
While
/usr/lib/hypatia/conduits
is the master copy, it is not used in live script runs. Each of
incoming
,
outgoing
and
runcond
create automatically an instance copy (built bit by bit as needed) of
/usr/lib/hypatia/conduits
when they run. This is to facilitate easy testing, bootstrap on restart and testing of modifications as a non-system user prior to re-building the package and deployment.
While
runcond
being a user utility builds the instance in the current users home directory,
incoming
and
outgoing
being system utilities build their instance in temporary space (currently
/dev/shm/incoming
and
/dev/shm/outgoing
respectively to get the performance benefits on an in-memory file system).
It is possible to configure the actual instance location used to override the default by setting an environment variable prior to running. For
incoming
the environment variable containing the instance path is
INCOMING
and for
outgoing
it is
OUTGOING
. For
runcond
the default is
~/.conduits
which can be changed by setting
MY_CONDUITS
.
An instance copy has a directory structure much like the installed structure except that the
conduits
,
includes
and
supports
directories are empty at the outset. They will be populated on demand by copying files as needed. If the system installed files are upgraded (and newer) they will be re-copied when next needed. An instance directory structure may also have additional directories for generated output (
outputs
,
holding
, and
logs
for log files for example).
An instance copy can always be safely deleted in its entirety as it can always be automatically re-created on running one of the primary interface scripts.
Changing Conduits
There should be no need to alter any of the conduits at this layer. All changes would normally be made to Portal conduits.
--
TimColles - 30 Nov 2018