Hadoop Cluster: Care and Feeding

If you want to use Hadoop, see this page on computing.help.

This page covers maintenance and configuration of the Hadoop EXC cluster.

Nodes

The machines are LCFG-maintained DICE servers running the current desktop version of DICE.

Machine Role Account Abbreviation
scutter01 The namenode (the master node for the HDFS filesystem). hdfs nn
scutter02 The resource manager (the master node for the YARN resource allocation system).
The job history server.
yarn
mapred
rm
jhs
scutter03
to
scutter12
The compute nodes. These run:
a datanode (stores HDFS data) and
a node manager (manages YARN and jobs on this node).

hdfs
yarn

dn
nm

The nodes are in the AT server room.

Kerberos and privilege

The cluster uses Kerberos for authentication. To get privileged access to the cluster, you'll need to authenticate. For this you'll need to know the right machine and account and abbreviation to use. Find them in the table above, then do this:

  • ssh machine
  • nsu account
  • newgrp hadoop
  • export KRB5CCNAME=/tmp/account.cc
  • kinit -k -t /etc/hadoop.abbreviation.keytab abbreviation/${HOSTNAME}

So here's how to get privileged access to the HDFS filesystem:

  • ssh scutter01
  • nsu hdfs
  • newgrp hadoop
  • export KRB5CCNAME=/tmp/hdfs.cc
  • kinit -k -t /etc/hadoop.nn.keytab nn/${HOSTNAME}

Adding users

On the exc cluster this is driven by roles and capabilities, so it's automated. A prospective user of the exc cluster needs to gain the hadoop/exc/user capability. Several roles grant that, and you can discover them with e.g.
rfe -xf roles/hadoop
Most student users of the cluster will probably gain a suitable role automatically thanks to the Informatics database and Prometheus.

Making HDFS directories

On the exc cluster this is done by a script called mkhdfs which runs nightly. It ensures that each user with hadoop/exc/user has an HDFS directory. It runs on the namenode of the cluster, and it's installed by the hadoop-cluster-master-hdfs-node.h header.

There's a companion script called rmhdfs. It runs weekly, and looks for and lists those HDFS directories which don't have the capability associated with them.

For other clusters, you could either either adapt mkhdfs or you could make an HDFS directory manually. Here's how to do that:

  1. Log in to the namenode with ssh and acquiring privileged access to HDFS.
  2. Then make the HDFS home directory:
     hdfs dfs -mkdir /user/${USER}
     hdfs dfs -chown ${USER} /user/${USER}
     exit
     exit
     logout
    

Running a test job

This is how you check that the cluster is working.

  1. If you don't yet have an HDFS directory, here's how to make one.
  2. Now ssh to the YARN master node:
     ssh scutter02
  3. Put some files into your HDFS dir. These will act as input for the test job:
     hdfs dfs -put $HADOOP_PREFIX/etc/hadoop input
  4. List your files to check that they got there:
     hdfs dfs -ls input
  5. If you have already run the job and you want to rerun it, remove the output dir which the job makes:
     hdfs dfs -rm -r output
  6. Now submit the test job:
     hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar grep input output 'dfs[a-z.]+'
    You should see lots of messages about the job's progress. The job should finish within a minute or two.
  7. Once it's finished, transfer the job's output from HDFS:
     hdfs dfs -get output
  8. ... and take a look at what the job did:
     cd output
     ls
    You should see two files - an empty file called _SUCCESS and a file with a few word counts in it called part-r-00000. If you don't see _SUCCESS then the job didn't work.

Checking the log files

Hadoop keeps comprehensive and informative log files. They're worth checking when you're doing anything with Hadoop, or when something seems to be wrong, or just to check that things are OK. Here's where to find them:

Component Log directory Host
HDFS namenode /disk/scratch/hdfsdata/hadoop/logs The namenode (the master HDFS host)
HDFS datanode /disk/scratch/hdfsdata/hadoop/logs All the compute nodes
YARN resource manager /disk/scratch/yarn/logs The resource manager (the master YARN host)
YARN node manager /disk/scratch/yarn/logs All the compute nodes
Job History Server /disk/scratch/mapred/logs The job history server host

For hostnames see #Nodes.

Removing a node from the cluster

Here's how to remove a node from the cluster. You might need to do this if it has hardware trouble, or if you want to upgrade its firmware or its software, for example. You can only do this with a compute node - not one of the two master nodes.
  1. Add this to the bottom of the node's LCFG file:
    !hadoop.excluded   mSET(true)
  2. HDFS has to decommission the node (i.e. move its share of the HDFS data to other nodes):
    • Login to the namenode.
    • Acquire hdfs privilege.
    • Tell the namenode to take a fresh look at its config files:
      hdfs dfsadmin -refreshNodes
    • Look at the namenode log. Wait for it to announce Decommissioning complete for node and the node's IP address.
    • You can also check that HDFS reports that the node is "Decommissioned":
      hdfs dfsadmin -report
  3. YARN has to decommission the node.
    • (work in progress!)
    • This part is not working. You should be able to do:
      yarn rmadmin -refreshNodes
      and then the node manager on the excluded host should stop and it should be shown here as decommissioned
      yarn node -list
      but instead it shows as "running", and the mention of the exclude procedure in the resource manager log mentions no hostnames. Broken, it seems.

Documentation

The manuals for this release are at hadoop.apache.org/docs/r2.9.2/. They're listed down the left hand side of the page. There are a lot of them, but quite a few just document one concept or one optional extension (there are a lot of these too).

systemd services

Hadoop is started and stopped using systemd services (which are configured by LCFG).
Hadoop component systemd service Host
HDFS namenode hadoop-namenode.service The namenode (the master HDFS host)
HDFS datanode hadoop-datanode.service All the compute nodes
YARN resource manager hadoop-resourcemanager.service The resource manager (the master YARN host)
YARN node manager hadoop-nodemanager.service All the compute nodes
Job History Server hadoop-mapred.service The job history server host
These can be queried, started and stopped using systemctl in the usual way. For example:
# systemctl status hadoop-nodemanager
 hadoop-nodemanager.service - The hadoop nodemanager daemon
   Loaded: loaded (/etc/systemd/system/hadoop-nodemanager.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-09-19 10:16:33 BST; 1 weeks 4 days ago
 Main PID: 4573 (java)
   CGroup: /system.slice/hadoop-nodemanager.service
           └─4573 /usr/lib/jvm/java-1.8.0-sun/bin/java -Dproc_nodemanager -Xmx4000m -Dhadoop.log.dir=/disk/scratch/yarn/logs -Dya...

How to make a new cluster

Mostly, you'll just need to copy the LCFG config which sets up the exc cluster; but there are a few manual steps too. You'll need to make a header, a namenode, a resource manager and a bunch of slave nodes. Once you've made your new cluster, and you've checked that the log files and systemd services look OK, don't forget to run a test job to check that your cluster works.

Hardware resources

Note that the YARN node manager - which runs on each slave, and matches up jobs with hardware resources - automatically determines what hardware resources are available. If you don't give the slaves enough hardware, the node managers will decide that jobs can't run! Even if you're making a little play cluster on VMs, you'll need to give each slave several CPUs. If you don't, not even a diddy test job will run. In tests, slaves with 3 VCPUs and 4GB memory were sufficient for the test job.

Make the header

Pick a one-word name for your cluster. These instructions will use the name dana. Make a new header in subversion for your cluster. In our example we'll make live/hadoop-dana-cluster.h.
  1. Check out the live SubversionRepository and cd to the include/live directory.
  2. svn copy hadoop-exc-cluster.h hadoop-dana-cluster.h
  3. Edit it appropriately - perhaps like this:
    #ifndef LIVE_HADOOP_DANA_CLUSTER
    #define LIVE_HADOOP_DANA_CLUSTER
    #define HADOOP_CLUSTER_NAME dana
    #define HADOOP_CLUSTER_HDFS_MASTER dana01.inf.ed.ac.uk
    #define HADOOP_CLUSTER_YARN_MASTER dana02.inf.ed.ac.uk
    #define HADOOP_CLUSTER_KERBEROS
    #endif /* LIVE_HADOOP_DANA_CLUSTER */
  4. Commit it with svn ci -m "Header to configure the dana Hadoop cluster" hadoop-dana-cluster.h

Make a namenode

  1. Add your cluster's header to the profile of the machine that'll be the new cluster's namenode. Our example header is live/hadoop-dana-cluster.h.
  2. Below it, add the node type header, in this case probably dice/options/hadoop-cluster-master-hdfs-node.h
  3. Let LCFG make the machine's new profile and wait for it to reach the machine.
  4. ssh onto the machine.
  5. Acquire hdfs namenode privilege as described above.
  6. hdfs namenode -format
  7. In a separate session, make a directory then format the HDFS filesystem:
    $ nsu
    # mkdir /disk/scratch/hdfsdata/hadoop/namenode
    # chown hdfs:hadoop /disk/scratch/hdfsdata/hadoop/namenode
    # systemctl restart hadoop-namenode
  8. Back in the session with hdfs namenode privilege, build the base filesystem in HDFS.
     hdfs dfs -mkdir /user
     hdfs dfs -mkdir /tmp
     hdfs dfs -chmod 1777 /tmp
     hdfs dfs -mkdir /tmp/hadoop-yarn
     hdfs dfs -mkdir /tmp/hadoop-yarn/staging
     hdfs dfs -chmod 1777 /tmp/hadoop-yarn/staging
     hdfs dfs -mkdir /tmp/hadoop-yarn/staging/history
     hdfs dfs -mkdir /tmp/hadoop-yarn/staging/history/done_intermediate
     hdfs dfs -chmod 1777 /tmp/hadoop-yarn/staging/history/done_intermediate
     hdfs dfs -chown -R mapred:hadoop /tmp/hadoop-yarn/staging
     hdfs dfs -ls /
     exit
     exit
    
  9. And to support distcp:
     nsu
     cd /disk/scratch/hdfsdata
     mkdir cache
     chown hdfs:hdfs cache
     chmod go+wt cache
     exit
    
Here are a couple of ways to test that it's successfully up and running:

Make a resource manager

  1. As for the namenode, first add the cluster's LCFG header to the profile of the machine that'll be the resource manager. Our example header is live/hadoop-dana-cluster.h.
  2. Below it, add the node type header, in this case probably dice/options/hadoop-cluster-master-yarn-node.h.
  3. Let LCFG make the machine's new profile and wait for it to reach the machine.
  4. Reboot the machine once or twice.
Here are a couple of ways to test that it's successfully up and running:

Make a slave node

The rest of the hosts in the cluster will all be slave nodes. Here's how to make one:
  1. As for the namenode, first add the cluster's LCFG header to the profile of the machine that'll be a slave node. Our example header is live/hadoop-dana-cluster.h.
  2. Below it, add the node type header, in this case probably dice/options/hadoop-cluster-slave-node.h.
  3. Let LCFG make the machine's new profile and wait for it to reach the machine.
  4. Reboot the machine once or twice.
Here are a couple of ways to test that it's successfully up and running:

Edit | Attach | Print version | History: r53 | r49 < r48 < r47 < r46 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r47 - 08 Nov 2019 - 13:32:24 - ChrisCooke
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies