Crazy Console AKA Brown Effect

At the 2017-02-08 Operation Meeting I recounted a recent occurrence of odd serial over LAN console behaviour, but a more extreme than previous events. See, specifically, the services-unit report. At the meeting I was to create a wiki page so we can log similar events in case a pattern could be spotted.

So here it is.

Some notes

On Monday afternoon 20/2/2017 I deliberately triggered the same scsi/device error messages on crocotta (an SL7 PE1950), to see if it would trigger the serial console problem. As of Thursday 23/2/2017 morning, the error messages were still going to the console, but the login prompt was still available and working. So I've decided to stop this attempt to trigger the problem on crocotta, and stopped the error messages.

The plan is to vacate yeti (same spec'd machine as huldra, SL7 R720), and then try the same thing. yeti vacated of active data. scsi lun mask removed 14:50 Thursday 23/2/2017, and console messages appearing. 2/3/2017 messages still going to yeti console and console working

2/3/2017 For the record, here are the outputs of "dsu -i" for huldra and yeti. The same bar the FC cards, ordering of network devices, and the uEFI diags

   YETI                                                       |    HULDRA
1. BIOS  ( Version : 2.5.4 )                                    1. BIOS  ( Version : 2.5.4 )
2. Lifecycle Controller  ( Version : 2.40.40.40 )               2. Lifecycle Controller  ( Version : 2.40.40.40 )
3. Dell 64 Bit uEFI Diagnostics  ( Version : 4247A1 )         | 3. Dell 64 Bit uEFI Diagnostics, version 4247, 4247A1, 4247.2
4. OS Collector  ( Version : 0 )                              | 4. OS COLLECTOR 1.1, OSC_1.1, X10-00  ( Version : OSC_1.1 )
5. Power Supply  ( Version : 07.2B.7D )                         5. Power Supply  ( Version : 07.2B.7D )
6. Power Supply  ( Version : 07.2B.7D )                         6. Power Supply  ( Version : 07.2B.7D )
7. CPLD  ( Version : 1.0.3 )                                    7. CPLD  ( Version : 1.0.3 )
8. PERC H710P Mini Controller 0 Firmware  ( Version : 21.3.2-   8. PERC H710P Mini Controller 0 Firmware  ( Version : 21.3.2-
9. Firmware for  - Disk 0 in Backplane 1 of PERC H710P Mini C   9. Firmware for  - Disk 0 in Backplane 1 of PERC H710P Mini C
10. Firmware for  - Disk 2 in Backplane 1 of PERC H710P Mini    10. Firmware for  - Disk 2 in Backplane 1 of PERC H710P Mini 
11. 12G Backplane Expander FW (Firmware)   ( Version : 1.03 )   11. 12G Backplane Expander FW (Firmware)   ( Version : 1.03 )
12.  iDRAC  ( Version : 2.40.40.40 )                            12.  iDRAC  ( Version : 2.40.40.40 )
13. NetXtreme BCM5720 Gigabit Ethernet PCIe (em2)  ( Version  | 13. NetXtreme BCM5720 Gigabit Ethernet PCIe (em1)  ( Version 
14. NetXtreme BCM5720 Gigabit Ethernet PCIe (em3)  ( Version    14. NetXtreme BCM5720 Gigabit Ethernet PCIe (em3)  ( Version 
15. NetXtreme BCM5720 Gigabit Ethernet PCIe (em1)  ( Version  | 15. NetXtreme BCM5720 Gigabit Ethernet PCIe (em4)  ( Version 
16. NetXtreme BCM5720 Gigabit Ethernet PCIe (em4)  ( Version  | 16. NetXtreme BCM5720 Gigabit Ethernet PCIe (em2)  ( Version 
17. QLogic QLE2462 Adapter  ( Version : 02.23.06 )            | 17. QLogic QLE2562 Adapter  ( Version : 03.22.08 )
18. QLogic QLE2462 Adapter  ( Version : 02.23.06 )            | 18. QLogic QLE2562 Adapter  ( Version : 03.22.08 )

7/3/2017 - errors still going to yeti's console. console still working

-- NeilBrown - 15 Feb 2017