Final Report for the System Security Enhancements Project (#224)

This was a fairly large wide-ranging project created in the wake of the discovery of a major compromise of our SSH service at the end of 2011. The plan was to examine the various proposals made in the review of that event. It was clear from the beginning that we would not have time to implement all possible options and that we would focus on those which were likely to provide the greatest improvements given our available effort.

There were 5 main aims of the project and each is reviewed separately below:

System Auditing

Ihe Linux audit daemon provides excellent facilities for monitoring logins and recording activity which involves important commands (e.g. setuid scripts). The aim was to enable this on all DICE machines with a basic configuration and then develop an enhanced configuration for important servers. An LCFG component was produced to manage the daemon and it has been rolled out on all DICE servers, we still need to consider enabling this on all other DICE machines. The SSH servers were configured to also watch for all user activity associated with setuid root scripts and attempts to alter kernel modules. We now have regular automated reports which help us quickly spot any attempts to compromise the SSH servers.

This work has already proved to be highly beneficial by helping us quickly analyse and resolve 3 separate incidents: a full system compromise, an attempt to compromise the system by using a local exploit and another where an account had been compromised.

The aim had been to also develop the ability to remotely log the audit information. This has been done and the LCFG auremote component has been created to help manage the configuration. Our experiments have shown this support to be a little fragile at this time, particularly regarding a restart of either system. If the support for remote logging improves we will revisit enabling this feature for important services. For now we will manually archive the logs.

There is documentation on how to use the LCFG auditd component

Enhance logging and automate the log monitoring

The original expectation was that this would be a relatively simple task involving the installation and configuration of some off-the-shelf software. In the end we decided to create our own solution, named BuzzSaw. This definitely required more effort than we had intended but it has given us a lot more control over how we check our logs and what reports we generate.

We are now doing daily monitoring for various kernel events, such as panics and Out-Of-Memory errors. We are also recording all SSH and Cosign authentication activity and an off-shoot project (#254) has created a web interface which uses this data to check their own login activity. A nice side-effect, which is entirely unrelated to security enhancements, is that we can now monitor and report on absolutely anything recorded in syslog. We are making good use of the facility to generate statistics on how much time machines spend awake and asleep for the sleep project.

A talk, titled Do bad guys work weekends?, was presented at the FLOSSUK 2013 Spring Conference. This examined some of the types of data stored in the database and looked at the possibilities for identifying suspicious activity.

All APIs have been documented and there are some high-level guides provided as part of the project source to help those who want to write their own filters and reports.

Regularly monitor the file-system

The aim was to add monitoring of the file-system of our important servers to help spot unauthorized changes. An LCFG component was developed to help configure and run the aide ((Advanced Intrusion Detection Environment) program. At this stage this has been tested but we have not fully deployed the tool as tweaking the configuration has proved slightly awkward. Also it is not clear that it gives us much additional benefit beyond the monitoring we are already doing with other tools (e.g. the audit daemon and rkhunter (rootkit hunter). If at some later date this becomes more obviously useful it would probably only be a small project to get this fully configured and deployed.

Regularly sweep for root-kits

Before starting this project we were already running the chkrootkit script on a regular basis. Experience has shown that chkrootkit gives quite a few false positives and is not really powerful or flexible enough for our requirements. As an improvement on this an LCFG component has been developed to configure and run the rkhunter (rootkit hunter) tool. This tool has much more extensive features and is very configurable. This meets our needs much better such that we avoid most false positives. Along with scanning for rootkits and other indications of attempts to compromise a system this tool monitors the important parts of our filesystem and reports on any changes. This means that the need for something like aide is greatly reduced.

There is documentation on how to use the LCFG rkhunter component.

Bootable image for investigations

Due to time constraints we decided not to examine the idea of creating a bootable image for security investigations. We may consider putting this forwards as a separate project at some point in the future.

Other Possibilities

Three other possibilities for investigation were listed in the original project plan. These were: Use SELinux, Shared database for failed logins and SSH honeypot. Due to time constraints these have not been examined as part of this project. Although the BuzzSaw database contains a log of all failed logins it is not updated frequently enough to be useful as a data source for automatically blocking access to brute force attacks across all servers with SSH external access. We may consider some of these possibilities in a separate project at a later date if any of them become more important.


This has been a large project which has required a lot more effort than originally planned. There is no doubt though that the products of this project have already proved themselves utterly invaluable when it comes to rapidly identifying system intrusions and compromised user accounts. In particular, the Linux audit daemon and the BuzzSaw log monitoring system have massively improved our ability to examine suspicious user activity. As already outlined this has helped us investigate and resolve several serious security incidents on our SSH servers.

It is clear that there is still some work to be done in terms of automating the transfer of audit logs from important servers to a central host. That should be put forwards as a project as soon as the technology has been sufficiently developed so that the support is more robust. There are also various ways in which we could enhance the data mining of our BuzzSaw database to spot suspicious activity, that is likely to be an ongoing activity as we gain a better understanding of what are the best indicators.

The MPU have now developed a good knowledge of how to use the Linux audit daemon tools and how to look for likely suspicious user activity. Also, BuzzSaw filters and reports have been written by more than one person. It is clear though that for the products of this project to be fully successful we will have to put some effort into educating other COs on how to use these tools and the possibilities that they provide.

Future Work

Following on from the work done in this project there are some tasks which will need to be done. Once the auditd remote logging becomes more reliable we should deploy that on critical servers, until then we should automate the archival of logs from auditd and also the process accounting.

There are also a number of possible subjects for future projects, which we might want to consider putting forward at some point, these are: bootable image for security investigations, use SELinux, shared database for failed logins and SSH honeypot.

Time Spent on Project

Period Hours
2012 T2 196
2012 T3 74
2013 T1 31

Total hours: 301 hours (43 days)

-- StephenQuinney - 29 Mar 2013

Topic revision: r3 - 23 Apr 2013 - 07:58:27 - StephenQuinney
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies