Development Project 48 - Transition of Informatics staff mail services to Staffmail.ed

Summary

The goal of this project was to move all (non-student) user mail accounts from our own IMAP based mail server (mail.inf.ed.ac.uk) to the central staffmail.ed service, and eventually do away with IMAP. We didn't have to do student accounts, as we had no student provision, they have always used the central student mail service (sms.ed).

The project page: https://devproj.inf.ed.ac.uk/project/show/48 has more details.

Work Carried out

Work started in January 2007. First we confirmed what we wanted to do was possible, so most of the computing staff moved to staffmail fairly early on. Once it was confirmed things were OK, we set a date of October 2007 when all new members of staff, and some visitors, would only have a staffmail.ed account.

Then we looked at how existing staff would be migrated. It would be feasible to set a date and (with IS's help) take care of the migration for everyone, but given the volume of mail involved (some 1100 users and 120GB of mail) this would lead to significant disruption to peoples' access to mail. So we installed imapsync and produced documentation on how users could migrate themselves to staffmail, and thus avoid enforced distruption if it was left up to us to move their mail.

Towards the end of 2007, beginning of 2008, a few prominent members of staff were asked to move to staffmail ahead of a general announcement that by the end of October 2008 anyone who hadn't moved themselves, using the help provided, would have their mail moved by us and IS.

In early November 2008, the majority of people had moved themselves to staffmail, so we only had 170 accounts (40GB of mail) to move. The move took about a day, and those 170 accounts were unable to access their mail during that time. At this point all individual mailboxes were now on staffmail.

We couldn't turn off IMAP at this point, as we were providing shared group mailboxes via #imapshared on mail.inf. This was quietly forgotten about until late 2009. In the meantime, if new requests came in for shared mailboxes, we set them up on staffmail (via functional accounts in the IDMS and the use of ACLs).

From the end of 2009 to March 2010 a push was made to move those last shared IMAP folders to staffmail, and this was completed at the beginning of March 2010, and finally IMAP access to mail.inf.ed.ac.uk was turned off.

Issues

There have a been a few issues that have come to light.

In the early stages of the move to staffmail, there were a couple of users who's username at Informatics did not match the UUN that staffmail were using (which it should). Some reconciling had to be done, and at least one member of staff had to have their DICE UUN changed.

We don't have an automatic way to forward new accounts created in the Informatics DB, or with the account management tools, so that UUN@inf is forwarded to UUN@staffmailREMOVE_THIS. This isn't always appropriate anyway, as not all visitor UUNs have a staffmail account, we ask them for an existing email address and arrange to forward to that instead. This means that a new member of staff, uunfoo, might appear and be added to DB generated mailing lists, eg the staff@inf list as uunfoo!@inf.ed.ac.uk and start receiving mail. Until a suitable entry is put in the aliases-staffmail file on mail.inf, the mail will queue on the server. Providing the forward is put in within 5 days, it will be delivered, otherwise sendmail will fail the delivery and bounce it. Periodically we will spot these and/or manually check for them, but we should produce at least an automatic warning, if not an automatic forward.

Moving the shared mailboxes to staffmail is less than ideal, as previously access to those shared mailboxes was driven by our roles and capabilities system. We can't "just do this" on staffmail, and the access needs to be maintained by hand. I've done some experimenting, and it seems it would be possible for us to setup an automatic process presenting a kerberos identity, and if that identity had the ACL permission to manage other ACLs for a shared mailbox on staffmail, it could maintain those via information in our roles and capabilities database.

Another issue with moving shared folders to staffmail, is that officially a staffmail shared folder can't be shared with an SMS user, and vice versa. We had one shared mailbox which a professor and his PhD student were accessing a shared mailbox on mail.inf. I was only able to emulate this on staffmail/SMS by setting up a functional account on SMS (with IS's help - functional accounts on staffmail we can do via the IDMS), and letting the professor's kerberos identity access the mailfolder. Which means he has to use SMS web front end, or a kerberos aware mail client (which he doesn't use). Fortunately it is the PhD that accesses the mail folder the most.

Moving "root" mail to staffmail also seems to making a rod for our own back. root mail used to go to the sysman shared mailbox on mail.inf, and we rotated it daily, keeping a fortnights worth of mail. Now that the sysman shared are is on staffmail, we have no direct way to do the equivalent "logrotation". And so root mail just piles up in a single folder unless someone manually clears out mail older than 2 weeks. We could ask IS if they could setup a cron on our behalf, or again look at setting up an automated identity to do all that via a kerberos authenticated IMAP to staffmail. But it was just more convenient on mail.inf.

Not an issue as such, but we do still need to run a mail.inf service, so take care of the forwarding and mailing lists (at least for now). But there are also other system type things, eg the way we handle RT tickets and the Informatics DB account.

Effort

As this project has dragged on a bit (3+ years) I thought it might be interesting to see how much effort was spent in each of those years.

Period Effort (hours)
2007 52.4
2008 99.6
2009 10.5
2010 25.6
Total 188.1

That's about 25 days FTE spread over 3 years. The work in 2007 and 2008 includes time spend helping people migrate, not just the technical work.

Lessons

From the effort saved not having to support user mailboxes on mail.inf, this has definitely been a worthwhile project. However, I think blindly moving all the shared mailboxes from mail.inf to staffmail (so we can kill of IMAP) has been less of a gain, given the effort and hoops we have to go through to manage the access via the staffmail ACLs.

Though this project has dragged on, I don't think that shows any great problem with the project. The nature of the project, ie moving lots of email accounts while causing the minimum disruption, meant that we wanted to take our time. The delay between completing the move of the users and the shared mailboxes, was just that other things were deemed to be of a higher priority and the shared mailboxes move was not urgent.

-- NeilBrown - 31 Mar 2010

Topic revision: r3 - 31 Mar 2010 - 19:43:55 - NeilBrown
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies