Thursday, January 2, 2014

Datanode dead but pid file exists - cannot start datanode anymore

Don't know why I cannot start Cloudera CDH5 datanode with the command below:
> sudo service hadoop-hdfs-datanode start

> sudo service hadoop-hdfs-datanode status
Hadoop datanode is dead and pid file exists                [FAILED]

But I performed the following commands to fix the problem:
> for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x stop ; done
> sudo rm -rf /var/lib/hadoop-hdfs/cache/*
> sudo -u hdfs hdfs namenode -format
> for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done

and after these commands you need to re-run step 3,4, and 6  using instructions on https://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Quick-Start/cdh5qs_topic_3_3.html
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Quick-Start/cdh5qs_yarn_pseudo.html

Note: reference to http://www.manning-sandbox.com/thread.jspa?threadID=41812 Probably my problem was namenode was formatted and the namespace ID in namenode didn't get replicated to datanode.

No comments: