The DICE Hadoop cluster

What is Hadoop?

Read the Wikipedia entry or watch Isabel Drost's talk on Hadoop at FOSDEM 2010.

Access to the DICE Hadoop cluster

We run a DICE Hadoop cluster. It has two main categories of users:

1. Extreme Computing students. If you are registered for Extreme Computing you will be given access to the cluster when you need it for your assignments.
2. Researchers wishing to try out Hadoop can use it as a test cluster.

If you want to try using the Hadoop cluster, ask Computing Support (request form, contact details) to give you access.

Once you have access, read on.

Using the DICE Hadoop cluster

If you want to try out Hadoop, the Hadoop 0.20.2 documentation pages include a Quick Start Tutorial which lets you have a play with your own temporary one node Hadoop cluster. If you're ready to try a bigger cluster, try the Map/Reduce Tutorial on the DICE Hadoop cluster. In the first example of that Map/Reduce tutorial, the correct "javac" command for our cluster would be "javac -classpath /opt/hadoop/hadoop-0.20.2/hadoop-0.20.2-core.jar -d wordcount_classes WordCount.java"

The DICE Hadoop cluster runs mainly on the DICE Beowulf machines. There are currently about 70 data nodes in the cluster. It runs Hadoop 0.20.2 on Scientific Linux 6.

To get access to the web status pages of the clusters (URLs are given below) you will need to be connecting from a machine on the Informatics network. If you're connecting from outside that network (for instance if your machine is on the University's wireless network) then you may find the OpenVPN service useful.

To get access to the cluster "ssh namenode" from a machine on the Informatics network. (If this is slow you might find it quicker to "ssh 129.215.18.30".)

To examine the HDFS status see http://namenode.inf.ed.ac.uk:50070.

To examine the status of Map/Reduce jobs look at http://jobtracker.inf.ed.ac.uk:50030.

If you have a question or a problem to do with the Hadoop cluster contact Computing Support. They'll either answer your question or put you in touch with someone who can.

HadoopCareAndFeeding is a resource for computing staff. It describes how to maintain the cluster.

Hardware and Networking Details

Some users of the cluster may want to know details of the cluster's hardware and networking arrangements. If you do, read on.

The cluster contains two broad categories of machines: rack-mounted servers and spare desktop machines.

Details of Rack-Mounted Servers

The rack-mounted servers in the Hadoop cluster come in two similar configurations. You can tell them apart by their names which start with either 'bw' or 'hcrc'. All of the rack-mounted machines also form a Beowulf cluster.

There are 24 nodes with names beginning with 'bw'. The 'bw' machines are Dell PowerEdge SC1425 servers. We bought them in July 2005. Each one has one dual core 3.2GHz Intel Xeon CPU; one 80GB hard disk, 7200rpm, SATA, model number Maxtor 6Y080M0, with 22GB of space available for use by Hadoop; and 4GB of memory.

The other rack-mounted nodes, including the namenode and the jobtracker, have names beginning with 'hcrc'. These are Dell PowerEdge SC1425 servers. We bought them in January 2006. Each one has two dual core 3.2GHz Intel Xeon CPUs; one 80GB hard disk, 7200rpm, SATA, model number WDC WD800JD-75MS, with 22GB of space available for use by Hadoop; one 250GB hard disk, 7200rpm, SATA, model number Maxtor 7L250S0, with 204GB of space available for use by Hadoop; and 8GB of memory.

All of the nodes run Scientific Linux 6.x.

The rack-mounted machines' networking arrangements are explained in this diagram and its accompanying notes, courtesy of Ian Durkacz, with additions from Iain Rae (thanks to both):

1. The overall layout of the Beowulf switches is as follows:

                                                /   core2/A4
                                  3Gb/s Trunk  {     ^ core2/A5
                                                \    |  ^ core2/A6
                                                     |  |  ^
                                                     |  |  | core1/A4 (redundant
                                                     |  |  |  ^        link)
                                                     |  |  |  |
                                                     |  |  |  |
               Switch bw00 (HP ProCurve 2824)        |  |  |  |
               +-------------------------------------+--+--+--+-+
               | ... beowulf nodes ...  17 18 19 20 21 22 23 24 |
               +-------------------------+--+--+--+-------------+
                                         |  |  |  |
                             2Gb/s Trunk |  |  |  | 2Gb/s Trunk
                                  +------+  |  |  +-------------------------+
                                  |  +------+  +-------------------------+  |
                                  |  |                                   |  |
        +-------------------------+--+-+       +-------------------------+--+-+
        | ... beowulf nodes ...  49 50 |       | ... beowulf nodes ...  49 50 |
        +------------------------------+       +------------------------------+
        Switch bw01 (HP ProCurve 2650)         Switch bw02 (HP ProCurve 2650)
   

   Each individual port and link in this picture - each inter-switch connection - runs at 1Gb/s.

   So there is a trunked link giving 3Gb/s back to the core from bw00, (the other link back to the core is a redundant link back - it will normally be blocked by STP), and two trunks from bw00 - one each to bw01 and bw02 - giving 2Gb/s.

   There is no direct connection between bw01 and bw02.

2. bw00 is an HP ProCurve 2824. It has a total port count of 24, and its backplane runs at 48 Gb/s. All of its ports run at 1Gb/s.

   bw01 and bw02 are HP ProCurve 2650 switches. Each has a total port count of 50, and the backplane of each runs at 13.6 Gb/s. Their inter-switch connection ports run at 1GB/s, but the ports to which the nodes are connected run at 100Mb/s.

   Each machine has a connection to two subnets, 129.215.18 and 129.215.81. Hadoop is configured to prefer using the former.

   The backplane speeds give the total capacity of each switch.

3. A note about trunks: the capacities (3Gb/s or 2Gb/s in the above) give the total capacity of the trunked link, but each individual connection over the trunked link always uses one cable only (which one, depends on the sending and receiving MAC addresses), so each individual connection over any of the above trunks can run at a maximum of 1Gb/s.

   There are scenarios where the trunk links bw00<->bw01 and bw00<->bw02 could be swamped: it just depends on which ports are wanting to talk to which others, and at what data rates.

4. You can see usage statistics for all of the above at http://netmon.inf.ed.ac.uk/BW.html.

Details of Spare Desktop Machines

At any one time we have a variety of desktop machines which are not currently in use. They will eventually find a new home on someone's desk, but until that happens, we may as well get some use out of them by attaching them to the Hadoop cluster. The number of machines varies as machines are taken away or added regularly. The machines are a mix of all of the models of desktop machine we use. At the time of writing that would include Dell Optiplex GX620, GX745, 755 and 780, and HP dc7900. Processor specs vary; the memory is either 2GB or 4GB depending on the age of the machine. Disk size also varies. The machines are networked together like so - thanks very much to Ian Durkacz for this:

1. All of the spare desktops are connected to the single switch 'sr99', an HP ProCurve 2610-48 (the HP part number for which is J9088A.) That switch has 48 10/100Base-TX ports, 2 10/100/1000 ports, and two open GBIC slots.

   So: each desktop has a 100Mb/s link to the switch, and the switch currently has two 1Gb/s uplinks to core switch 'core2' (the same core switch to which switch bw00 on the Beowulf rack is uplinked), configured as a single 2Gb/s trunk uplink.

2. Therefore:

   • The speed of the connections between each of the spare desktops is 100Mb/s, full duplex, and that can be simultaneously sustained between all machines on the switch.
   • The speed of the connection between the switch on the spare desktops' racking and that on the Beowulf racking is 1Gb/s full duplex, and the total bandwidth of that link is 2Gb/s.

3. Visually:

                                                 /   core2/A17
                                   2Gb/s Trunk  {     ^ 
                                                 \    | core2/C17
                                                      |  ^
                                                      |  |
                                                      |  |
       +----------------------------------------------+--+-----+
       | 1 2 ... ... ... ... ... ... ... 47 48 |     49 50     |
       | <--- Hadoop nodes - all 100Mb/s ----> | 1Gb/s uplinks |
       +-------------------------------------------------------+
       Switch sr99 (HP ProCurve 2610-48)