Hadoop Cluster: Care and Feeding

If you just want to use Hadoop, see computing.help instead.
This page covers maintenance and configuration of the Hadoop EXC cluster.
Note that this page is out of date and is currently being revised contact iainr@infREMOVE_THIS.ed.ac.uk for more information.

Nodes

The machines are LCFG-maintained DICE servers running the current desktop version of DICE.

Machine Role Account Keytab Abbreviation
scutter01 The namenode (the master node for the HDFS filesystem). hdfs /etc/hadoop.nn.keytab nn
scutter02 The resource manager (the master node for the YARN resource allocation system).
The job history server.
yarn
mapred
/etc/hadoop.rm.keytab
/etc/hadoop.jhs.keytab
rm
jhs
scutter03
to
scutter12
A datanode (stores HDFS data).
A node manager (manages YARN and jobs on this node).
hdfs
yarn
/etc/hadoop.dn.keytab
/etc/hadoop.nm.keytab
dn
nm

The nodes are in the AT server room.

Kerberos

The cluster uses Kerberos for authentication. Before you can do any maintenance work on the cluster, you'll need to authenticate with the appropriate credentials. To do this, you'll need to know the right machine and account and keytab and keytab abbreviation to use. Find them in the above table. Once you have them, follow these general instructions:

  • ssh machine
  • nsu account
  • newgrp hadoop
  • export KRB5CCNAME /tmp/account.cc
  • kinit -k -t keytab abbreviation/${HOSTNAME}

For example, to get privileged access to the namenode you would do this:

  • ssh scutter01
  • nsu hdfs
  • newgrp hadoop
  • export KRB5CCNAME=/tmp/hdfs.cc
  • kinit -k -t /etc/hadoop.nn.keytab nn/${HOSTNAME}

Topic revision: r36 - 30 Aug 2019 - 12:54:53 - ChrisCooke
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies