This is a rough, evolving todo list for the prometheus account management system.

Timescales

(This section updated 1st June 2010)

  • Early May - aim to have test server running and dummy services being populated (DONE - test server running, dummy services populated by prometheus as part of test suite)
  • Mid June - detailed roll-out plan produced, probably meeting to discuss this - DONE
  • Late June - test server being populated with live upstream data - DONE
  • Late July - aim to have completed majority of bug fixes, refinements, support structure (web interface, password generation stuff, etc, etc) - DONE
  • Early August - aim to be in a position to create new accounts in parallel with existing system (see Rollout plan below) - DONE (rolled out 10th August)
  • August onwards - incomplete work not essential for account creation may be deferred until August, further bug fixes, refinements
  • End 2010 - main project completed, as per project completion todo below
  • 2011 - prometheus enhancements project

Project Completion TODO

27th September: This is the current TODO for the prometheus project. The aim is that we draw a line under the main project by the end of 2010, with any additional work to be spun out into a prometheus enhancements project. The horizontal line in the todo list signifies the drawing of the line.

1st October: The TODO list has had a few changes made, following today's prometheus review meeting (present: gdmr, toby, sxw, alisond, cms).

7th October: Another meeting to discuss account issues (present: gdmr, lmb, cms, alisond, timc, neilb, toby). Again, the TODO list has been edited as necessary - changes are marked with 7/10 update

  1. Test server install and bring into service (toby, 1-3 days). DONE
    • install
    • prometheus install/test
    • make switch
  2. Remaining code changes (reset-passwd, ents rework, various patches) into tree (toby/sxw, 1 day). Depends on (1) DONE
  3. Client-side rpm create and distribute (toby, 0.5 day). Depends on (2) DONE (just needs to go on all machines)
  4. Implement account lifecycle (sxw/toby, 1-2 weeks; also need to budget time for bringing into service, devproj #188) Rough plan:
    • when PopulateAccounts conduit sees that the account object should be removed, it doesn't delete it, but sets a 'lifecycle' attribute on the account to 'warning'
    • something else then goes through and sends an email to all of the accounts flagged as 'warning', and sets a flag of 'pending-archive', and the date on which the accounts should be archived (probably 28 days after the letter)
    • there's then a tool that support can run that archives accounts which are over their archive date, and changes the flag to 'pending-deletion'
    • prometheus then deletes all of the accounts that are 'pending-deletion'
    • we should provide a tool that a user can use to request an extended grace period, e.g. a URL in the warning email that if you visit lets you ask for another month. Grace periods beyond that would require database changes. We have agreed that this is not the role of support - this is an admin issue.
    • we will need to speak with support about the vast number of accounts that will be suggested for deletion and come up with a plan for dealing with these before users get email - this is the key stage in this process - we will produce lists of accounts to be deleted in advance. These should be signed off ahead of deletion (gdmr to take to CEG devproj #182)
    • we need to be able to obtain sponsor information from database toby to talk to rat-unit
    • 7/10 update The issue of a grace period is one that requires a lot more thought. We should probably mirror the policy used by the University - more details see here and here (thanks to Lindsey for the links). This makes the account lifecycle work much more complicated. Different categories of person have different lengths of grace period (e.g. 150 days for students, 30 for staff, none for "associates"). It was agreed that grace periods should be implemented within prometheus, not the database (although the database currently implements a month grace period for visitors).
  5. Roles visibility in system ldap - either remove and properly fix tools to speak to prometheus, or short-term bodge which involves replicating roles from prometheus to system ldap; current half-way house is unsustainable. Needs decision. Decision is to do it properly. sxw to mail cos with proposal for roles. (sxw/toby, 3 days)
  6. Put agents into prometheus or acl fix to run ldapBuildAmdMaps on franklin via cron (sxw or toby, 2 days former, < 1 day latter). Plan for former:
    • write tool to permit the creation of arbitrary entities in prometheus.
    • use said tool to create an agent.
    • will need to be aware of potential impact of non-person/machine entities on existing conduits
  7. systemldap group -> afs group conduit. Need to assess priority against time factors. (cms/toby, 3 days)
    • this should probably be part of the services unit prometheus work
    • in the meantime, the afs group population script should be run more often - cms
  8. CO level documentation - done (toby, 2-3 days)
  9. Proper test suite for afs (cms, 3-5 days)
  10. Component to manage server configuration, daemon, etc. (toby, 2-4 days) DONE (in testing)
  11. Mechanism for managing admin principals (sxw, 1 day)
    • Extension of the tool in (6).
  12. LCFG host import conduit creating identity objects for machines (toby, 2 days) in progress
    • Conduit needs to have a 'concept of stickiness' to guard against problems introduced by flapping spanning maps (i.e. entries disappearing and then reappearing)
  13. Audit runs and reporting - At the moment, both audit and syncronisation errors are getting put into LDAP, but nothing is reading them back out, and nothing is truncating them. The eventual plan was that the web interface would let you browse these, but we definitely need something to truncate them soon, and it might be nice to support dumping the output as text. (sxw, 2 days)
  14. Errors and logging. The output from the various conduits differs in what it reports, e.g. it would be useful if all of them logged creation/deletion. We can turn on debugging for stores, but a higher/more consistent level of default information would be useful. Ideally this should tie in with (13). The correct way to do this is for the conduits to return information to be displayed by run-conduit. (toby, 1-2 days)
  15. Data cleanup. Note that (4) has significant overlap with this. Likely to be quite a lot of work. We need to start deleting accounts from the various places. We also need to be able to rely on what the database tells us (inf/rat/us units). We have a strong preference that this be spun out as a separate project on which we have a hard dependency and that It's important that RAT/US agree that it's a priority to resolve any issues that are highlighted. The to-be-scheduled meeting to discuss account issues may be the appropriate forum for this. A few points about the nature of the issues that need to be addressed:
    • accounts in prometheus which DB says shouldn't exist
    • downstream stores inconsistency (e.g. account in afs pts, but not kdc)
    • possible issue with people disappearing from (our view of) the database (7/10 update: this shouldn't happen in normal operation - cases which have been seen may have been due to mistakes)
    • toby to start generating lists of affected accounts - in progress
  16. The infdb will be switching to a new data model some time over the next three months. This will inevitably have implications for prometheus. We need to speak to rat-unit about the rollout plan for the new model. 7/10 update: this is likely to happen in early November - initially there will be two distinct feeds into prometheus. Inf unit would like to have a feed as early as possible for tsting purposes - this should be possible. One large change will be that uuids will come from the database - currently these are generated by prometheus. (~3 days - this is a very rough estimate 1 week+) toby to speak to rat-unit devproj #187
  17. Email address - who masters it and how is it managed - the values currently in the database aren't really usable, hence prometheus is effectively current master, but with no means to update. New infdb data model may does have implications for this. We consider this to be essentially a database matter, that it provides prometheus with one valid email address. 7/10 update: This should be possible. One issue to consider is the public availability, or otherwise, of the email address. The database will have a "publically available" flag.
  18. Turn on deletions. Dependent on much of the above.

Approximations of time required to reach this point, bearing in mind piece-of-string-like aspects of some work (data cleanup, new DB data model, etc.):

  • sxw: 3-4 weeks
  • toby: 3-5 weeks
  • cms: 1-2 weeks


  1. iFriends into prometheus
  2. Other conduits (jabber, wordpress, bugzilla, various web things, rat teaching postgres server)
  3. Roles revamp. The roles we have in use are somewhat disorganised and would benefit from a complete revamp. This is quite high risk.
  4. Web interface
  5. Self service identity management - allowing users to create their own identities, and stash the password material for them into keytabs
  6. Self service group management - allowing the owners of particular roles to add and remove people from those roles, and allowing users to resign membership of a role which they have been added to.
  7. Whitepages information. Pulling office, telephone etc. information out of the database, into prometheus and then into the system LDAP. I think this is one for later. It would be a nice project for someone who wanted to gain experience with prometheus, as it should be relatively straightforwards.
  8. Reconsider need for separate Identity and Account stores.
  9. Mailing list membership management from prometheus
  10. Quotas - ideally these should be more tightly coupled with prometheus, which has volume management logic
  11. Central auth/IDMS - keep an eye on developments to see if these offer anything of use to us.
  12. Speed enhancements - we would like prometheus conduits to run faster. There are code changes and other factors we can investigate for this.
  13. 7/10 update: Be aware that EUCLID may well include applicants in the feed to the database, and that the database may, in turn, present these with the primary role of "new-student" to prometheus. This would currently result in accounts being generated for all these people, which we certainly don't want.
  14. Fix asu so users aren't prompted twice for password.
  15. Add proper, flexible debugging to conduits
  16. Consider a conduit for generating local account information (e.g. to generate localaccounts.h)

Rollout Plan

This is a rough plan of the stages involved in rolling out Prometheus (this section updated 31 May 2010).

Update 27 Sep: Rollout happened on 10th August. Subsequent work to be tracked on project completion todo above.

  • Create new accounts in LDAP, Kerberos, and AFS using prometheus. The entitlements system, quotas, netgroups, machine accounts &c would continue to be handled by the existing tool set. Deletions will not be possible.
    • Missing pieces: AFS conduits, Password management interface

During the meeting of 15/6/2010 the following deadlines/timescales were agreed
  • support training possibly the 2nd Tuesday of July
  • move existing data set to Prometheus Monday 19th July (so missing pieces above need done before this)
  • move new data set week beginning 23rd August.

  • Move machines into prometheus
    • Missing pieces: LCFG conduit (almost done)
  • Turn on deletions
    • Missing pieces: just needs enough guts to throw the switch! Needs database support for unique id in order to handle merges and renames
  • Move role management into prometheus
    • Missing pieces: None
  • Move iFriends into prometheus
  • Enable user facing web interface
  • Add conduits for new services

Framework

Framework items are those in the core Prometheus perl framework, and LDAP directory server enhancements.

Error handling

  • base class to store errors
  • directory storage format
    • audit runs
    • normal runs
  • fatal vs warnings
  • returning errors from the conduit class to the upper level

DONE

Configuration

  • configuration object format
  • file based configuration (with Config::Any)
  • directory based configuration

Status: File and command line configuration done, directory configuration pending

Directory

  • capabilities / group access control
  • default acls
  • interface for managing ACLs?

Web interface

  • Design web interface
  • Implement AJAX front end
  • Move web frontend from CVS into git
  • Use prometheus classes, rather than raw LDAP calls
  • add interface to
  • error classes
  • configuration
Status: Web frontend code and design complete. Requires backend modifications.

Conduit classes

  • syncronisation runs
  • audit runs
  • incremental runs

Documentation

This is an ongoing task. Much of the current code requires detailed class, method and attribute documentation. Having adequate documentation is now a requirement before code enters gerrit, but older code requires updating.

Object ownership

We need to know which conduit owns an object so that we can deal with sections of the tree (such as the entity space) which are created by multiple conduits.

Tests

  • Create unit test framework
  • unit tests for days
  • write framework to bring up and tear down Prometheus LDAP server

Many existing classes have few, if any, unit tests. Part of this is because many classes require a working prometheus server to test against. The plan is to create a module which will create, and teardown, a local LDAP server for each test. Once this is complete, all new code will require unit tests, and tests will be added to old code in a piecemeal fashion in the same way as with documentation.

DONE

Conduits

  • old LDAP -> Prometheus done
  • Database -> Prometheus done
  • Role translator (Prometheus -> Prometheus) done, needs to be committed
  • Tree exploder (Prometheus -> Prometheus) pretty much done, I think
  • LDAP exporter (Prometheus -> production LDAP tree) done, needs to be committed
  • KDC exporter (Prometheus -> Kerberos) done
  • AFS conduit (Prometheus -> AFS pts) in progress
  • AFS volume creator (Prometheus -> AFS fileservers)
  • iFriend (iFriend -> Prometheus)
  • Jabber (Prometheus -> Jabber)
  • LCFG (LCFG -> Prometheus) almost done

Feature Improvements

These are changes which would be nice to have, but which don't currently have any conduits requiring them

Logging datastore which is assisted by the directory

Idea here is that when an object is visited by the conduit, a marker is placed in that object containing a run timestamp, and the conduit UUID. When looking for the not visited objects, we then just have to search for all objects which don't have that timestamp recorded against the UUID. Difficulty here is how best to represent that in LDAP

The benefit of this is that it reduces the memory consumption of conduits that are performing whole directory syncronisation operations. Current testing suggests that memory usage isn't an issue with the projected size of our prometheus dataset, so this one is on the back burner for now

Attribute ownership

We haven't dealt with 'ownership' in any of the interfaces, but it seems likely that we want some way of a sync run being prevented from just blindly merging away by having some kind of concept of 'pinned' attributes that can't be altered by all comers.

One possible rule here is that if you're first, you can set an attribute to anything you like. The last conduit to modfiy an attribute can also feel free to alter it. Beyond that, you need permission from some higher source.

Questions here are:

  • Do we need this? Specifically, is it worth the complexity
  • How do we enforce it? Making it mandatory would require logic in the directory server itself. Is it worth having if it's only advisory?

Again, there isn't a clear use case for ownership beyond permitting administrators to override data sources for a given user's data. This is on the back burner for now until a real need can be demonstrated

Refactoring

This is a collection of places where code works, but isn't particularly clean or efficient, and ways we could improve it. Low priority.

Make fetch more efficient

At present, the Prometheus store's fetch method will always do a search for the primary key. If an object is found, then a search rooted at that position is started, in order to construct a tree. For those stores which have potential trees, we should override fetch so that if a primary key is provided, we will start by doing a subtree search with the DN indicated by that primary key. This removes a server round trip, and should be much faster, and neater!

Use introspection for trees

At the moment, the names of objectclasses and primaryKeys are hard coded into the prometheus datastores. We could be much tidier if we determined these by introspection of the various classes

Primary keys

Primary keys should really be a property of an object, not of a collection, otherwise we have the danger of having multiple primary keys defined for a object. The complication here is how do we handle this in the context of autogenerated objects.

Decent support for 'has' and 'clear'

Toby's use of Maybe[], and the current populatedAttributes against attributes confusion is due to the lack of decent support for checking to see if an object has a set value, and for clearing values when we merge. We should implement this, and things will become much easier.

DONE

Add asSchema & friends for classes

At the moment, asSchema can only be called on a role. Make it possible to call it on a class (in which case it will generate the schema for all roles that that class contains). Do the same for ldapPrimaryKey and ldapName (which should return the results from the role that is marked as structural)

Provide standard implementations of providerID and consumerID

Add implementations of the providerID and consumerID methods to the Conduit role which just use the primary key to go in both directions.

Clarify the division between LDAP and base objects

At the moment there's a blurred line between those features that a base Prometheus object can have, and those which it has if it's come from the PrometheusStore .

DONE

Implement private attributes

Private attributes are ones which have a name internal to a particular collection (they're designed to only be used in a single conduit). Whilst there is a 'private' attribute marker, there is no backend implementation for them.

DONE

Implement proper LDAP typing

The type of an object is currently controlled by the 'ldapType' attribute, which isn't really ideal. Implement proper typing using traits.

-- TobyBlake - 31 May 2010

Topic revision: r22 - 27 Apr 2011 - 11:33:49 - TobyBlake
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies