Item is delivered, either partially or in whole

New orders file created with rfe orders/{orderno}. Each file must adhere to the OrderFileFormat.

An order file example follows :-

date:7/11/2012
supplier:HP
ticket: rt56452
1.item:HP 8300CMT i5 3.2GHz, 8GB, 500GB
1.warranty:3yr
1.quantity:2
1.price:330.88
1.sno:CZC2457YM0
1.sno:CZC246Y8JH
1.delivered:19/11/2012
1.budget:747DIV/G40588
1.category:desktop
1.requestor:ascobie
2.item:Dell Ultrasharp 20" 2001FP
2.warranty:3 years
2.quantity:1
2.price:439.00
2.sno:
2.delivered:
2.category:monitor
2.budget:747DIV/G40588
vat:1.20

In the following, the term "orderline" will refer to each section of an orders file identified by the same starting ID. So, in the example above, we have orderline 1 which is for 2 of HP 8300 CMT desktops, and orderline 2 which is for 1 of Dell Ultrasharp monitor.

Note that item description field will be as entered by purchaser. Although it should include both manufacturer and model and may include details such as disk size and memory size, it is of no fixed format and should not be interpreted by any code.

In the example above, two desktops have been delivered. The monitor has yet to be delivered, so has a blank serial number and delivery date.

An order file is sync'd with the inventory tables as part of a periodic sweep, every 5 minutes, through all orders looking for updated files. (The periodic sweep code compares the purchaseorder->last_loaded field with the mtime of the orders file to determine whether a sync is required). The script that performs this sweep is called ordersync.

The ordersync script will :-

  • create a row in the 'purchaseorder' table, recording PO, supplier, order date, and VAT. (Note that this means that the VAT value applies to all items in a purchase order)
  • For each orderline with a valid "delivered" field
    • For each blank serial number (blank will be assumed for any missing sno lines)
      • a row in the 'item' table is created, recording description, cost, category, budget, warranty, delivered date, orderline. managed_type='unknown', owner='informatics', allocated_type='unallocated'
    • For each serial number in the orderline
      • a row will be created as above, also recording serial
      • if the item is a system (identified by 'category' field being one of server | desktop | laptop | tablet | storagearray | networkswitch | wap
        • then the 'is_a_system' field in the 'item' row is set to true

Note that some items may not have a serial number (eg a mouse), but we still want to record their purchase details so an 'item' table row is still created for that item.

Further items are delivered

The serial number and delivery date of the delivered items are manually added to the orders file using rfe orders/{orderno}, eg :-

2.delivered:02/12/2012
2.sno:HU-0C0881-46633-62M-03ML

A resync (either sync or async as before) will occur.

The ordersync script will :-

  • Remove any 'item' table rows associated with this order that have blank serial numbers. (Note that this means that you can't store state (eg allocation) for an item with a blank serial number.)
  • Perform the operations described in the previous section (ie creating rows in the 'item' table)

Note that whilst serial numbers need not be unique within the inventory, they must be unique within an individual purchase order.

Order is modified

'date', 'supplier', 'ticket', 'vat' changed
the relevant field in the appropriate 'purchase_order' row will be updated
'item', 'warranty', 'price', 'delivered', 'budget', 'category', 'requestor' changed
the relevant field for each 'item' row associated with the orderline is updated.
'sno' added
If the number of non blank serial numbers for the relevant orderline doesn't exceed the 'quantity' field for the orderline, a row in the 'item' table is created (as for 'Order is delivered'). The number of items for the orderline with blank serial numbers will be reduced.
'sno' modified
A new row in the 'item' table is created (as for 'Order is delivered'). This leaves an orphan row in the 'item' table for the serial number that is no longer listed in the orders file. This row may well have valid data (eg allocated_to, space) that was recorded against this serial number before it was discovered the serial number was incorrectly recorded. If there is any such data, or a mac address has been associated with this serial number, the row is marked an orphan ('orphaned' field set to true). (The Tartarus manager can copy across recorded information from the orphaned item row to the new correct item row, using the /usr/lib/tartarus/bin/copy_from_orphaned script).

Kit is sold on to a research grant

Splitting an item in two - eg so an item of 10 PCs can be split into two items of 5 PCs each is supported. This is the correct way to mark that a school has re-sold a desktop, bought on a central school budget, to a research grant.

Mac addresses

It is important to capture the mac addresses of all computing systems such as desktops, servers, laptops, tablets etc - as we use the mac addresses to track location of such kit, in addition to feeding DHCP.

The mac addresses of desktops bought as part of the Select PC agreement will be loaded into the inventory automatically (see below). The mac addresses of all other kit will need to be loaded manually. Multiple mac addresses can be specified per machine.

You can associate a macaddr with a system using the "ii addmacaddr" command. ( Usage: ii addmacaddr [--hostname hostname] [--serial serialno] [--iid id] macaddr1 [macaddr2 ...] )

ii addmacaddr --serial 13J06TP 4e:82:BC:98:03:3B

The long term intention is that mac addresses will no longer be specified directly in the source profiles of LCFG desktops. Instead, the mac address to system mapping will be mastered within Tartarus and exported to LCFG along with the sysinfo information.

Select PC mac addresses

As part of the Select PC agreement, the suppliers produce a periodic report (currently weekly) that lists all PCs ordered by the University under the agreement - providing serial number, cost, purchase order and, crucially, mac address.

See TartarusSupplierReports for more detail on how the the Select PC reports are loaded into the Tartarus supplier_report table.

Periodically (currently twice a day), the /usr/lib/tartarus/bin/loadmacreports script searches the supplier_report table for PCs which match on a 'serial-number:purchase_order' pair basis with items recorded in the item table. If there's a match, and we don't already have the mac address information for the matching PC, an appropriate row is created in the macaddr table.

TODO: This is always going to be a fragile solution as we have no control over the provision of the upstream spreadsheet. In any case, it doesn't work for non SelectPC kit. We should provide an alternative route for grabbing MAC addresses. One easy solution is to read MAC addresses from the BIOS, but that requires manual entry of the MAC address. A better solution would be to PXE boot devices into a system (based on the LCFG installroot). That could use a modified version of the clientreport script to populate the inventory system automatically with MAC addresses that it harvests. A step further would be to allow the user to set the hostname,allocated_to and managed_type fields so that they wouldn't have to run 'invedit' later.

Hostnames

  • Hostnames are unique across active systems in the inventory.
  • Hostnames may be re-used.
  • Self managed dynamic address machines should not normally have hostnames.
  • There is no definitive location for hostname information
    • Neither the inventory nor the DNS are to be considered definitive
    • The tool ii checkhost can be used to confirm whether a hostname is already registered in the DNS or inventory
  • Short hostnames will be postfixed with '.inf.ed.ac.uk', otherwise FQDN hostnames should be used

Logbook

Each purchased system has a logbook, with entries of the form 'date', 'changetype', 'newvalue'. Possible changetypes are creation, location, manual_location, sub_location, allocated_to, allocated_type, managed_type, hostname, comment, disposed, fault, manager, owner, barcode, parent, sync, host, macaddr, child, vmhost_change.

You can display the logbook :-

shell$ ii logbook --hostname benbecula
2010-10-29   creation  
2010-10-31   location   IF-2.09
2010-10-31   allocated ascobie
2010-10-31   hostname benbecula.inf.ed.ac.uk
2010-10-31   managed_type dice
2010-10-31   comment  provided under academic/co scheme
2012-11-04   location   IF-2.23
2012-11-05   allocated unallocated
2013-03-04   fault        motherboard failed
2013-06-04   disposed Disposed via CCL

You can add a comment to a system's logbook :-

ii edit --serial C3U88FJU --allocate mahesh ii logbook --serial C3U88FJU --comment "Temporarily lent to Mahesh for some network experiments"

NOT YET IMPLEMENTED - You can also add a fault entry to the logbook :-

ii logbook --hostname benbecula --fault "motherboard failed"

LCFG profiles

  • Self-managed static machines still require LCFG profiles to configure DHCP
  • Self-managed dynamic machines must not have an LCFG profile
  • LCFG Profile names must no longer be used as the definitive namespace for hostnames

Adding parts to a system

Items can be associated with a system. This is particularly useful for grouping together the component parts of network routers, storage arrays etc.

eg the following associates the item with serial SV02810080 with the system with serial 13K046T

ii additem --serial 13K046T --child_serial SV02810080

Note that only non system items can be associated with a system.

To remove an item from a system

ii removeitem --serial 7U8IH9

Alternative serial numbers

See TartarusMultipleSerials

Periodic maintenance

  • Deal with orphaned rows?

TODO

Various automatic processes

Location info from switches (PULL FROM)

The script /usr/lib/tartarus/bin/sync_locations runs once per hour to update the location of kit for which the macaddr is known and recorded in Tartarus. This should include all network connected kit, not just desktops and laptops. This script downloads the locations.txt file from each site netmon server (as per configuration in /etc/tartarus/synclocations), merges the results and select the most recently reported location for each macaddr. If this is the first time the kit has been seen by the switches, or the kit has moved to a new location, the new location is recorded (item->location) and the time the kit was seen by the relevant switch is recorded in item->last_seen. A logbook entry is created to record the location change. If the kit hasn't changed location, the time the kit was last seen is recorded in item->last_seen.

This will not cover kit which is only ever connected by wireless.

Client reports (LCFG only) (PUSH TO)

See TartarusClientReport for details.

TODO: We have to be careful that the script which syncs the 'clientreport' table with the 'item' table doesn't overwrite manually set information with data from an old report. For example, a DICE managed machine called 'skye' becomes a self-managed machine called 'eigg'. When converting to self-managed, the item->hostname field will be manually set to 'eigg' - if we're not careful, the next time the clientreport sync script is run, it'll overwrite the item->hostname with whatever is in the last report script. A number of solutions :-

  • delete clientreport entries when item->type transitions away from 'dice' - probably unsafe, but good housekeeping?
  • don't sync clientreport entries for items where item->type <> 'dice'
  • add an item->lastreportsync column and only update fields if the clientreport is newer than this date
  • only use clientreports from the last 12 hours (current approach, but really not good enough)

Feed into LCFG profiles (PULL FROM)

For each machine with managed_type = 'dice', create a header with the resources. TODO: Shouldn't something similar happen for 'selfstatic' machines? (We should probably do this for any piece of kit which has a macaddress with the generate_dhcp flag set.)

TODO - Expand on the following - which fields come from where

!dhclient.mac         mSET(xx.xx.xx.xx.xx.xx)    # to associated macaddr row with dhcp_generated=true
!sysinfo.node        mSET(gala)
!sysinfo.domain      mSET(inf.ed.ac.uk)
!sysinfo.sno         mSET(CZC2457YM0)
!sysinfo.location    mSET(IF-5.14)
!sysinfo.manager     mSETQ("")
!sysinfo.model       mSETQ("HP Compaq Elite 8300 CMT")
!sysinfo.owner       mSETQ("")
!sysinfo.allocated   mSET(ascobie)
!sysinfo.comment     mSETQ("")

In the old 2008 system, these headers were pulled down periodically (every 30m) from the inventory to the LCFG master. Now that the macaddr for a desktop machine comes via this route (mac -> header -> profile), we need to make sure that the info is available pretty soon after it's available in the inventory. Otherwise people have to wait for the periodical pull down before installing a new machine. We could just increase the frequency of the rmirror, but that's perhaps rather crude. Increasing the frequency to every 5m would create negligible load on either end (only takes 1s to run) and is probably quick enough for user requirements. Alternatively we could push the information from the inventory, but how does it know to do that - the trigger may just be someone setting managed_type to 'dice' for a desktop.

Select PC load (PULL TO)

The Select PC suppliers produce an excel spreadsheet (every Monday) listing details on kit that has been purchased by the University. These spreadsheets are stored on a share on an IS server, along with historical copies of the spreadsheets. The share is EASE authenticated and does require a suitably authorised EASE account to access.

In the prototype, the scripts gatherreport, run manually, pulls down these reports and combines them, using the script parsereport to extract the required information out of the excel spreadsheets. This produces a csv file listing all SelectPC equipment purchased - with a macaddr, serial-no, model-name, purchase order for each item. This file is then loaded into the 'supplierreport' table. A script 'loadmacreports' works its way through the 'supplierreport' table. If a piece of kit was purchased by the school (a row in the 'item' table with same serial number and purchase order) an entry in the macaddr table is created and associated with that 'item' table row.

The delivered version will need to be run automatically. IS has told us that it ought to be possible to access the share using a functional account, so this should be achievable.

\\df1.is.ed.ac.uk\UnManaged\SelectPC\Dell MacAddress \NewformDell.xls

Theon conduits - people, space, site (PULL TO?)

TODO Need to discuss conduits with Tim or Graham.

API

The prototype 'ii' command currently accesses the Inventory tables directly (albeit via DBIx::Class). There are two problems with this :-

  • it is in theory possible for a remote client to access the inventory tables directly and work round the rules that the 'ii' commands would normally impose. Some of this could be solved by using triggers in the database, though not sure if all could be done that way.
  • it is not easy for other systems/scripts to query the inventory as they need knowledge of the schema

It would be better if a remote API were presented (particularly for queries). An API for updates would also be useful so that we can maintain consistent behaviour across multiple edit interfaces (commandline,web,..)

Misc things

  • re schema - might be better to use tables to record possible values for 'type' fields, rather than having Enums which would necessitate schema changes for additional types.
  • MAC addresses - will be displayed in lowercase with colon separators, but accepted in lowercase/uppercase with colon or dash separators. Stored using the postgresql MACADDR type

Virtual machines

  • plan was to NOT add virtual machines to the inventory, but just leave them in the clientreport table to query
    • but this makes it difficult to make a query like - give me all servers
    • problematic if we're mastering DHCP/LCFG info from inventory
  • 10/05/16 - probably better to add virtual machines to the inventory, with an 'item' row. purchaseorder_id = NULL could represent 'this is a vm', or we could add a 'virtual' boolean column. The row could be created using 'ii', or by 'kvmtool' or triggered by a new clientreport entry.

Various Reports

  • working thrrough SelectPC reports - identify serial numbers which are missing/incorrect in our orders files (can match on order number)

Thoughts - 27/04/15

  • Should managed type be set to 'unknown' by default?
  • ? Split out DHCP/non-DHCP from managed type - eg so have type ='self' not 'selfstatic' and 'selfdynamic'. Makes it easier to add support for eg managed dynamic and mdp dynamic
  • What do about faulty kit where the serial number of the replacement item is different from the original item ?
    • ? Change the serial number in the original order and record the old serial number in the item's logbook (assuming that any item can have a logbook!). This will create an orphaned item with all the information that we might want to retain, like allocation, location etc - so we should probably create a "item replaced by" tool that does all the work behind the scenes - though that's not easy to achieve for the original order file
  • modify clientreport to report on various attached hardware (eg graphics cards, disks etc) - could then automatically link these to the machine they're part of. - can then easily, for example, list all DICE machines with particular nvidia graphics cards. Support multiple monitors
  • Should have a 'ticket' field in orders database, and orders table, to record the associated RT ticket (Now added)
  • with clientreport - what happens if the serial number of the machine can't be found in the inventory? It's probably a duff serial number?

-- AlastairScobie - 15 Nov 2013

Edit | Attach | Print version | History: r40 < r39 < r38 < r37 < r36 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r38 - 24 Sep 2019 - 11:47:37 - AlastairScobie
DICE.TartarusWorkFlow moved from DICE.InvProjectWorkFlow on 23 Sep 2019 - 13:33 by AlastairScobie - put it back
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies