Conduits Incoming

This describes the automated upload of data into the database through the incoming framework. In general this is triggered by reception of email filtered by procmail. The structure of the repository for the framework is described in ConduitsFramework, described here is the usage.

Incoming Command

The incoming command script takes one mandatory argument which is the name of the specific incoming script to run. The currently ones can be found here: /usr/lib/hypatia/conduits/incoming on a machine with suitable hypatia headers.. The incoming command script is essentially just a common wrapper for individual TheonCoupler feed agents. They generally just pipe input (normally an email message) through the Python Incoming module (which handles formatting adjustments) and then straight to the appropriate feed agent for processing. Incoming data is staged and held in the holding directory in the instance copy for replay when necessary.

Incoming Mail Framework

Most feeds are initiated automatically by incoming email to the database server. Others by explicit crontab entry. At the moment this process is configured by re-direction from the Informatics mail server. This works as follows. Feeds must be sent to This email will be processed by and delivered to the local daidb user. This has a .procmailrc and uses a redirection program infdb-incoming. A copy of the mail is retained locally and then it is forwarded directly to the SMTP port on the School Database server. Logs of the process can be inspected on at /home/daidb/log-file.

On the database server there is a Postfix server running which is configured via LCFG which accepts the incoming mail and delivers it to the local postgres user. The postgres user has an LCFG generated .procmailrc (in /var/lib/pgsql/.procmailrc) which pipes the mail to the incoming command with an appropriate feed script as argument based on the source and subject of the mail. Logs of the process can be inspected at /var/lib/pgsql/procmailrc.log for procmail and /var/lcfg/log/mail for Postfix.

Testing and Manual Initiation

The incoming process runs on the database server by virtue solely of the LCFG managed procmailrc created for the postgres user. The process can be initiated manually by any user on any client with sufficient privilege by simply doing

  cat EMAIL | incoming FEED [ARG]

for example, where EMAIL is the full email message that would normally be processesed by the feed, FEED is the name of the specific feed and ARG is any optional feed argument. The command above will create and bootstrap an instance copy (into /dev/shm/incoming by default), process the feed and any associated conduits uploading the database with content from the email message.

Normally you may need to do test feeds manually when they have broken. In which case you need to use the original email message that broke the feed. Sometimes these may be left in the incoming instance copy on the database server itself (in /disk/data/incoming/holding). Alternatively a copy of all incoming email is held on the server in /home/daidb/Mail/incoming.copy.

-- TimColles - 30 Nov 2018

Topic revision: r1 - 30 Nov 2018 - 17:29:58 - TimColles
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies