(r39) TartarusWorkFlow < DICE

DICE Web>ManagedPlatformUnit>TartarusManual>TartarusWorkFlow (revision 39) (raw view)~~EditAttach~~
<!--
  This is a comment, in the HTML sense of the word, but Wiki directives still work, so you could put
  access control statements here to restrict access to your new topic, and they won't show up in the
  published page.
  Please think carefully when creating a new topic, and use a GoodWikiName.
  Remember that by default all Wiki content is world readable and editable unless you've taken steps
  to limit it.
-->
%TOC%

---+++ Item is delivered, either partially or in whole

New orders file created with =rfe orders/{orderno}=. Each file must adhere to the OrderFileFormat.

An order file example follows :-

<verbatim>
date:7/11/2012
supplier:HP
ticket: rt56452
1.item:HP 8300CMT i5 3.2GHz, 8GB, 500GB
1.warranty:3yr
1.quantity:2
1.price:330.88
1.sno:CZC2457YM0
1.sno:CZC246Y8JH
1.delivered:19/11/2012
1.budget:747DIV/G40588
1.category:desktop
1.requestor:ascobie
2.item:Dell Ultrasharp 20" 2001FP
2.warranty:3 years
2.quantity:1
2.price:439.00
2.sno:
2.delivered:
2.category:monitor
2.budget:747DIV/G40588
vat:1.20
</verbatim>

In the following, the term "orderline" will refer to each section of an orders file identified by the same starting ID. So, in the example above, we have orderline 1 which is for 2 of HP 8300 CMT desktops, and orderline 2 which is for 1 of Dell Ultrasharp monitor.

Note that =item= description field will be as entered by purchaser. Although it should include both manufacturer and model and may include details such as disk size and memory size, it is of no fixed format and should not be interpreted by any code.

In the example above, two desktops have been delivered. The monitor has yet to be delivered, so has a blank serial number and delivery date.

An order file is sync'd with the inventory tables as part of a periodic sweep, every 5 minutes,  through all orders looking for updated files.
(The periodic sweep code compares the purchaseorder->last_loaded field with the mtime of the orders file to determine whether a sync is required). The script that performs this sweep is called =ordersync=.

The =ordersync= script will :-
   * create a row in the 'purchaseorder' table, recording PO, supplier, order date, and VAT. (Note that this means that the VAT value applies to all items in a purchase order)
   * For each orderline with a valid "delivered" field
      * For each blank serial number (blank will be assumed for any missing sno lines)
         * a row in the 'item' table is created, recording description, cost, category, budget, warranty, delivered date, orderline. managed_type='unknown', owner='informatics', allocated_type='unallocated'
      * For each serial number in the orderline
         * a row will be created as above, also recording serial
         * if the item is a system (identified by 'category' field being one of server | desktop | laptop | tablet  | storagearray | networkswitch | wap
            * then the  'is_a_system' field in the 'item' row is set to true

Note that some items may not have a serial number (eg a mouse), but we still want to record their purchase details so an 'item' table row is still created for that item. 

---+++ Further items are delivered

The serial number and delivery date of the delivered items are manually added to the orders file using rfe orders/{orderno}, eg :-
<verbatim>
2.delivered:02/12/2012
2.sno:HU-0C0881-46633-62M-03ML
</verbatim>

A resync (either sync or async as before) will occur. 

The =ordersync= script will :-
   * Remove any 'item' table rows associated with this order that have blank serial numbers. (Note that this means that you can't store state (eg allocation) for an item with a blank serial number.)
   * Perform the operations described in the previous section (ie creating rows in the 'item' table)
       
Note that whilst serial numbers need not be unique within the inventory, they *must* be unique within an individual purchase order.


---+++ Order is modified 
   $ 'date', 'supplier', 'ticket', 'vat' changed: the relevant field in the appropriate 'purchase_order' row will be updated
   $ 'item', 'warranty', 'price', 'delivered', 'budget', 'category', 'requestor' changed : the relevant field for each 'item' row associated with the orderline is updated.
   $ 'sno' added : If the number of non blank serial numbers for the relevant orderline doesn't exceed the 'quantity' field for the orderline, a row in the 'item' table is created (as for 'Order is delivered'). The number of items for the orderline with blank serial numbers will be reduced.
   $ 'sno' modified : A new row in the 'item' table is created (as for 'Order is delivered'). This leaves an orphan row in the 'item' table for the serial number that is no longer listed in the orders file. This row may well have valid data (eg allocated_to, space) that was recorded against this serial number before it was discovered the serial number was incorrectly recorded. If there is any such data, or a mac address has been associated with this serial number, the row is marked an orphan ('orphaned' field set to true). (The Tartarus manager can copy across recorded information from the orphaned item row to the new correct item row, using the =/usr/lib/tartarus/bin/copy_from_orphaned= script). 

---+++ Kit is sold on to a research grant

Splitting an item in two - eg so an item of 10 PCs can be split into two items of 5 PCs each *is* supported. This is the correct way to mark that a school has re-sold a desktop, bought on a central school budget, to a research grant. 

---+++ Mac addresses

It is important to capture the mac addresses of *all* computing systems such as desktops, servers, laptops, tablets etc - as we use the mac addresses to track location of such kit, in addition to feeding DHCP.

The mac addresses of desktops bought as part of the Select PC agreement will be loaded into the inventory automatically (see below). The mac addresses of all other kit will need to be loaded manually. Multiple mac addresses can be specified per machine. 

You can associate a macaddr with a system using the "ii addmacaddr" command. ( Usage: ii addmacaddr [--hostname hostname] [--serial serialno] [--iid id] macaddr1 [macaddr2 ...] )

ii addmacaddr --serial 13J06TP  4e:82:BC:98:03:3B

The long term intention is that mac addresses will no longer be specified directly in the source profiles of LCFG desktops. Instead, the mac address to system mapping will be mastered within Tartarus and exported to LCFG along with the sysinfo information. 

---++++ Select PC mac addresses

As part of the Select PC agreement, the suppliers produce a periodic report (currently weekly) that lists all PCs ordered by the University under the agreement - providing serial number, cost, purchase order and, crucially, mac address.

See TartarusSupplierReports for more detail on how the the Select PC reports are loaded into the Tartarus =supplier_report= table.

Periodically (currently twice a day), the =/usr/lib/tartarus/bin/loadmacreports= script searches the =supplier_report= table for PCs which match on a 'serial-number:purchase_order' pair basis with items recorded in the =item= table. If there's a match, and we don't already have the mac address information for the matching PC, an appropriate row is created in the =macaddr= table. 

TODO: This is always going to be a fragile solution as we have no control over the provision of the upstream spreadsheet. In any case, it doesn't work for non !SelectPC kit. We should provide an alternative route for grabbing MAC addresses. One easy solution is to read MAC addresses from the BIOS, but that requires manual entry of the MAC address. A better solution would be to PXE boot devices into a system (based on the LCFG installroot). That could use a modified version of the clientreport script to populate the inventory system automatically with MAC addresses that it harvests. A step further would be to allow the user to set the hostname,allocated_to and managed_type fields so that they wouldn't have to run 'invedit' later. 


---+++ Hostnames
 
   * Hostnames are unique across *active* systems in the inventory. 
   * Hostnames may be re-used. 
   * Self managed dynamic address machines should not normally have hostnames. 
   * There is no definitive location for hostname information
      * Neither the inventory *nor* the DNS are to be considered definitive
      * The tool =ii checkhost= can be used to confirm whether a hostname is already registered in the DNS or inventory
   * Short hostnames will be postfixed with '.inf.ed.ac.uk', otherwise FQDN hostnames should be used

---+++ Logbook

Each purchased system has a logbook, with entries of the form 'date', 'changetype', 'newvalue'. Possible changetypes are creation, location, manual_location, sub_location, allocated_to, allocated_type, managed_type, hostname, comment, disposed, fault, manager, owner, barcode, parent, sync, host, macaddr, child, vmhost_change.

You can display the logbook :-

<verbatim>
shell$ ii logbook --hostname benbecula
2010-10-29   creation  
2010-10-31   location   IF-2.09
2010-10-31   allocated ascobie
2010-10-31   hostname benbecula.inf.ed.ac.uk
2010-10-31   managed_type dice
2010-10-31   comment  provided under academic/co scheme
2012-11-04   location   IF-2.23
2012-11-05   allocated unallocated
2013-03-04   fault        motherboard failed
2013-06-04   disposed Disposed via CCL
</verbatim>

You can add a comment to a system's logbook :-

  ii edit --serial C3U88FJU --allocate mahesh 
  ii logbook --serial C3U88FJU --comment  "Temporarily lent to Mahesh for some network experiments"

NOT YET IMPLEMENTED - You can also add a fault entry to the logbook :-

 ii logbook --hostname benbecula --fault "motherboard failed"

---+++ LCFG profiles

   * Self-managed static machines still require LCFG profiles to configure DHCP
   * Self-managed dynamic machines must *not* have an LCFG profile
   * LCFG Profile names must no longer be used as the definitive namespace for hostnames


---+++ Adding parts to a system

Items can be associated with a system. This is particularly useful for grouping together the component parts of network routers, storage arrays etc.

eg the following associates the item with serial SV02810080 with the system with serial 13K046T

ii additem --serial 13K046T --child_serial  SV02810080

Note that only non system items can be associated with a system.

To remove an item from a system

ii removeitem --serial 7U8IH9


---+++ Alternative serial numbers

See TartarusMultipleSerials

---+++ Periodic maintenance

   * Deal with orphaned rows?

*TODO*

---+++ Various automatic processes 

---++++ Location info from switches (PULL FROM)

The script =/usr/lib/tartarus/bin/sync_locations= runs once per hour to update the location of kit for which the macaddr is known and recorded in Tartarus. This should include all network connected kit, not just desktops and laptops. This script downloads the =locations.txt= file from each site netmon server (as per configuration in =/etc/tartarus/synclocations=), merges the results and select the most recently reported location for each macaddr. If this is the first time the kit has been seen by the switches, or the kit has moved to a new location, the new location is recorded (item->location) and the time the kit was seen by the relevant switch is recorded in item->last_seen. A logbook entry is created to record the location change.  If the kit hasn't changed location, the time the kit was last seen is recorded in item->last_seen. 

This will not cover kit which is only ever connected by wireless.

---++++ Client reports (LCFG only) (PUSH TO)

See TartarusClientReport for details.

TODO: 
We have to be careful that the script which syncs the 'clientreport' table with the 'item' table doesn't overwrite manually set information with data from an old report. For example, a DICE managed machine called 'skye' becomes a self-managed machine called 'eigg'. When converting to self-managed, the item->hostname field will be manually set to 'eigg' - if we're not careful, the next time the clientreport sync script is run, it'll overwrite the item->hostname with whatever is in the last report script. A number of solutions :-
   * delete clientreport entries when item->type transitions away from 'dice' - probably unsafe, but good housekeeping?
   * don't sync clientreport entries for items where item->type <> 'dice'
   * add an item->lastreportsync column and only update fields if the clientreport is newer than this date
   * only use clientreports from the last 12 hours (current approach, but really not good enough) 

---++++ Feed into LCFG profiles (PULL FROM)

The script =/usr/lib/tartarus/bin/generate_lcfg_headers= is run periodically (currently once per hour) to create stub inventory header files per 'DICE' machine.  These header files include =sysinfo= resources derived from information in the Inventory. The LCFG master downloads the files , using the rmirror component, from the Tartarus server for inclusion during profile building.
 
---+++ Virtual machines

Virtual machines (at least those on the KVM hosts) are recorded in Tartarus. The  =/usr/lib/kvmtool= tool registers new KVM guests using the [[TartarusRESTAPI][REST API]]. Virtual machines are identified in the =item= table by having a non null value for =item.vm_host=. This column points to the KVM host. Each KVM host periodically runs the =/usr/lib/tartarus/bin/sync_kvm_guest_info= script - this spots when a KVM guest has migrated from one KVM host to another. 

  

 
-- Main.AlastairScobie - 15 Nov 2013
Topic revision: r39 - 24 Sep 2019 - 13:54:47 - AlastairScobie
DICE.TartarusWorkFlow moved from DICE.InvProjectWorkFlow on 23 Sep 2019 - 13:33 by AlastairScobie - put it back
DICE
This is WebLeftBar
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies