Multinodes Hadoop 2.4 installation and configuration

Image may be NSFW.
Clik here to view.

Prepare the environment

Setup the Hadoop user on all nodes:

root@ubuntu:~# useradd -d /home/hadoop -s /bin/bash -u 600 hadoop

add these to /etc/hosts on all 3 hosts:

192.168.47.139 nn

192.168.47.140 dn1

192.168.47.141 dn2

unzip JDK, and copy to /usr/local/jdk1.7.0_55

vi /etc/profile, add

JAVA_HOME=/usr/local/jdk1.7.0_55

PATH=$PATH:$JAVA_HOME/bin

CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib

export JAVA_HOME CLASSPATHPATH

copy ssh keys to data nodes

hadoop@nn:~/.ssh$ ssh-copy-id -i id_rsa.pub hadoop@dn2

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed — if you are prompted now it is to install the new keys

hadoop@dn2’s password:

Number of key(s) added: 1

Now try logging into the machine, with: “ssh ‘hadoop@dn2′”

and check to make sure that only the key(s) you wanted were added.

hadoop@nn:~/.ssh$ ssh-copy-id -i id_rsa.pub hadoop@dn1

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed — if you are prompted now it is to install the new keys

hadoop@dn1’s password:

Number of key(s) added: 1

Install Hadoop

Download and unzip on all nodes

hadoop@nn:~$ wget http://apache.mirror.iweb.ca/hadoop/common/hadoop-2.4.0/hadoop-2.4.0.tar.gz

hadoop@nn:~$ scp hadoop-2.4.0.tar.gz dn1:/home/hadoop/

hadoop-2.4.0.tar.gz 100% 15MB 14.6MB/s 00:00

hadoop@nn:~$ scp hadoop-2.4.0.tar.gz dn2:/home/hadoop/

hadoop-2.4.0.tar.gz 100% 15MB 14.6MB/s 00:00

change JAVA_HOME in Hadoop-env.sh

hadoop@nn:~/hadoop-2.4.0/etc/hadoop$ vi hadoop-env.sh

#export JAVA_HOME=${JAVA_HOME}

export JAVA_HOME=/usr/local/jdk1.7.0_55

hadoop@nn:~/hadoop-2.4.0/etc/hadoop$ scp hadoop-env.sh dn1:/home/hadoop/hadoop-2.4.0/etc/hadoop/

hadoop-env.sh 100% 3535 3.5KB/s 00:00

hadoop@nn:~/hadoop-2.4.0/etc/hadoop$ scp hadoop-env.sh dn2:/home/hadoop/hadoop-2.4.0/etc/hadoop/

hadoop-env.sh 100% 3535 3.5KB/s 00:00

update the .bash_profile of Hadoop user

hadoop@nn:~$ more .bash_profile

export HADOOP_HOME=/home/hadoop/hadoop-2.4.0

export PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/jdk1.7.0_55/bin:/home/hadoop/hadoop-2.4

.0/bin:/home/hadoop/hadoop-2.4.0/sbin

alias h=’cd /home/hadoop/hadoop-2.4.0/’

alias etc=’cd /home/hadoop/hadoop-2.4.0//etc/hadoop’

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME/lib/native”

export CLASSPATH=$CLASSPATH:$HADOOP_HOME/share/hadoop/common:$HADOOP_HOME/share/hadoop/common/lib:\

$HADOOP_HOME/share/hadoop/yarn:$HADOOP_HOME/share/hadoop/yarn/lib:\

$HADOOP_HOME/share/hadoop/mapreduce:$HADOOP_HOME/share/hadoop/mapreduce/lib:\

$HADOOP_HOME/share/hadoop/hdfs:$HADOOP_HOME/share/hadoop/hdfs/lib

Write configurations

Core-site.xml

<name>fs.default.name</name>

</property>

<name>hadoop.tmp.dir</name>

<value>/home/hadoop/hdfs/tmp</value>

</property>

<name>fs.checkpoint.dir</name>

<value>/home/hadoop/hdfs/checkpoint</value>

</property>

<name>hadoop.logfile.size</name>

</property>

<name>hadoop.logfile.count</name>

</property>

<name>io.file.buffer.size</name>

</property>

</configuration>

Hdfs-site.xml

<value>/home/hadoop/hdfs/name</value>

<description>Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently.</description>

</property>

<value>/home/hadoop/hdfs/data</value>

<description>Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks.</description>

</property>

<name>dfs.replication</name>

</property>

<name>dfs.datanode.du.reserved</name>

</property>

</configuration>

Mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.map.memory.mb</name>

</property>

<name>mapred.child.java.opts</name>

</property>

<name>mapreduce.map.java.opts</name>

</property>

<name>mapreduce.reduce.memory.mb</name>

</property>

<name>mapreduce.reduce.java.opts</name>

</property>

<name>mapreduce.reduce.shuffle.parallelcopies</name>

</property>

<name>mapreduce.task.io.sort.mb</name>

</property>

<name>mapreduce.task.io.sort.factor</name>

</property>

<name>mapreduce.jobhistory.address</name>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

</property>

</configuration>

Yarn-site.xml

<name>yarn.resourcemanager.address</name>

</property>

<name>yarn.resourcemanager.scheduler.address</name>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

</property>

<name>yarn.resourcemanager.webapp.address</name>

</property>

<name>yarn.resourcemanager.amin.address</name>

</property>

<name>yarn.scheduler.minimum-allocation-mb</name>

</property>

<name>yarn.scheduler.maximum-allocation-mb</name>

</property>

<name>yarn.nodemanager.resource.memory-mb</name>

</property>

<name>yarn.scheduler.minimum-allocation-vcores</name>

</property>

<name>yarn.scheduler.maximum-allocation-vcores</name>

</property>

<name>yarn.app.mapreduce.am.resource.mb</name>

</property>

<name>yarn.app.mapreduce.am.command-opts</name>

</property>

<name>yarn.resourcemanager.scheduler.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.local-dirs</name>

<value>${hadoop.tmp.dir}/nodemanager/local</value>

</property>

<name>yarn.nodemanager.address</name>

<description>the nodemanagers bind to this port</description>

</property>

<name>yarn.nodemanager.remote-app-log-dir</name>

<value>${hadoop.tmp.dir}/nodemanager/remote</value>

<description>directory on hdfs where the application logs are moved to </description>

</property>

<name>yarn.app.mapreduce.am.command-opts</name>

</property>

<name>yarn.resourcemanager.scheduler.class</name>

<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.local-dirs</name>

<value>${hadoop.tmp.dir}/nodemanager/local</value>

</property>

<name>yarn.nodemanager.address</name>

<description>the nodemanagers bind to this port</description>

</property>

<name>yarn.nodemanager.remote-app-log-dir</name>

<value>${hadoop.tmp.dir}/nodemanager/remote</value>

<description>directory on hdfs where the application logs are moved to </description>

</property>

<name>yarn.nodemanager.log-dirs</name>

<value>${hadoop.tmp.dir}/nodemanager/logs</value>

<description>the directories used by Nodemanagers as log directories</description>

</property>

<name>yarn.application.classpath</name>

<value> /home/hadoop/hadoop-2.4.0/etc/hadoop,/home/hadoop/hadoop-2.4.0/share/hadoop/common/*,/home/hadoop/hadoop-2.4.0/share/hadoop/common/lib/*,/home/hadoop/hadoop-2.4.0/share/hadoop/hdfs/*,/home/hadoop/hadoop-2.4.0/share/hadoop/hdfs/lib/*,/home/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/*,/home/hadoop/hadoop-2.4.0/share/hadoop/mapreduce/lib/*,/home/hadoop/hadoop-2.4.0/share/hadoop/yarn/*,/home/hadoop/hadoop-2.4.0/share/hadoop/yarn/lib/*

</value>

</property>

<name>yarn.log-aggregation-enable</name>

</property>

</configuration>

Setup slaves, master

hadoop@nn:~/hadoop-2.4.0/etc/hadoop$ more slaves

dn1

dn2

hadoop@nn:~/hadoop-2.4.0/etc/hadoop$ more master

copy files to other nodes

copy slaves, master to dn1, dn2

hadoop@nn:~/hadoop-2.4.0/etc/hadoop$ scp mapred-site.xml core-site.xml hdfs-site.xml yarn-site.xml dn1:/home/hadoop/hadoop-2.4.0/etc/hadoop/

mapred-site.xml 100% 2028 2.0KB/s 00:00

core-site.xml 100% 1420 1.4KB/s 00:00

hdfs-site.xml 100% 1590 1.6KB/s 00:00

yarn-site.xml 100% 5301 5.2KB/s 00:00

hadoop@nn:~/hadoop-2.4.0/etc/hadoop$ scp mapred-site.xml core-site.xml hdfs-site.xml yarn-site.xml dn2:/home/hadoop/hadoop-2.4.0/etc/hadoop/

mapred-site.xml 100% 2028 2.0KB/s 00:00

core-site.xml 100% 1420 1.4KB/s 00:00

hdfs-site.xml 100% 1590 1.6KB/s 00:00

yarn-site.xml 100% 5301 5.2KB/s 00:00

on the master, start namenode, resourcemanager and etc

136 hadoop namenode -format

137 hadoop-daemon.sh –script hdfs start namenode

138 ps -ef|grep hadoop

139 yarn-daemon.sh start resourcemanager

140 ps -ef|grep hadoop

141 yarn-daemon.sh start proxyserver

142 mr-jobhistory-daemon.sh start historyserver

On the data nodes, starts datanode, nodemanager

hadoop@dn1:~/hadoop-2.4.0/logs$ hadoop-daemon.sh –script hdfs start datanode

starting datanode, logging to /home/hadoop/hadoop-2.4.0/logs/hadoop-hadoop-datanode-dn1.out

hadoop@dn1:~/hadoop-2.4.0/logs$ yarn-daemon.sh start nodemanager

starting nodemanager, logging to /home/hadoop/hadoop-2.4.0/logs/yarn-hadoop-nodemanager-dn1.out

hadoop@dn2:~$ hadoop-daemon.sh –script hdfs start datanode

starting datanode, logging to /home/hadoop/hadoop-2.4.0/logs/hadoop-hadoop-datanode-dn2.out

hadoop@dn2:~$ yarn-daemon.sh start nodemanager

starting nodemanager, logging to /home/hadoop/hadoop-2.4.0/logs/yarn-hadoop-nodemanager-dn2.out

check status

Image may be NSFW.
Clik here to view.

hadoop@nn:~$ jps

4359 ResourceManager

6691 Jps

4275 NameNode

4652 JobHistoryServer

hadoop@dn2:~$ jps

4033 DataNode

4089 NodeManager

5392 Jps

Fix the WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform

Recompile with source codes and replace the libs

configure: error: C++ preprocessor “/lib/cpp” fails sanity check

solution: apt-get install g++

root@nn:~/Downloads/protobuf-2.5.0# protoc –version

protoc: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory

solution: add this to /etc/profile

export LD_LIBRARY_PATH=/usr/local/lib:/lib64:/usr/lib64

root@nn:~/Downloads/protobuf-2.5.0# source /etc/profile

root@nn:~/Downloads/protobuf-2.5.0# protoc –version

apt-get install zlib1g-dev libncurses5-dev

apt-get install gcc build-essential libssl-dev zlib1g-dev libglib2.0-dev cmake maven

hadoop@nn:~/hadoop-2.4.0-src$ mvn clean package -Pdist,native -DskipTests –Dtar

…

[INFO] BUILD SUCCESS

[INFO] ————————————————————————

[INFO] Total time: 17:18.833s

[INFO] Finished at: Tue May 27 07:55:58 PDT 2014

[INFO] Final Memory: 91M/239M

[INFO] ————————————————————————

hadoop@nn:~/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib$ ls -l

total 1732

-rw-rw-r– 1 hadoop hadoop 1111258 May 27 07:39 libhadoop.a

lrwxrwxrwx 1 hadoop hadoop 18 May 27 07:39 libhadoop.so -> libhadoop.so.1.0.0

-rwxrwxr-x 1 hadoop hadoop 657597 May 27 07:39 libhadoop.so.1.0.0

hadoop@nn:~/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib$ cp * /home/hadoop/hadoop-2.4.0/lib/native/

hadoop@nn:~$ cd example

hadoop@nn:~/example$ ls

file1.txt file2.txt

hadoop@nn:~/example$ hadoop fs -mkdir /data

hadoop@nn:~/example$ hadoop fs -put -f file1.txt file2.txt /data

map reduce job failed

hadoop@nn:~/hadoop-2.4.0$ hadoop jar ./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.4.0-sources.jar org.apache.hadoop.examples.WordCount /data /output

14/05/27 11:30:54 INFO client.RMProxy: Connecting to ResourceManager at nn/192.168.47.139:8132

14/05/27 11:30:55 INFO input.FileInputFormat: Total input paths to process : 2

14/05/27 11:30:55 INFO mapreduce.JobSubmitter: number of splits:2

14/05/27 11:30:56 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1401214983224_0001

14/05/27 11:30:57 INFO impl.YarnClientImpl: Submitted application application_1401214983224_0001

14/05/27 11:30:57 INFO mapreduce.Job: The url to track the job: http://nn:8188/proxy/application_1401214983224_0001/

14/05/27 11:30:57 INFO mapreduce.Job: Running job: job_1401214983224_0001

14/05/27 11:31:04 INFO mapreduce.Job: Job job_1401214983224_0001 running in uber mode : false

14/05/27 11:31:04 INFO mapreduce.Job: map 0% reduce 0%

14/05/27 11:31:04 INFO mapreduce.Job: Job job_1401214983224_0001 failed with state FAILED due to: Application application_1401214983224_0001 failed 2 times due to AM Container for appattempt_1401214983224_0001_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:

org.apache.hadoop.util.Shell$ExitCodeException:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)

at org.apache.hadoop.util.Shell.run(Shell.java:418)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

.Failing this attempt.. Failing the application.

14/05/27 11:31:04 INFO mapreduce.Job: Counters: 0

hadoop@nn:~/hadoop-2.4.0$ hadoop jar ./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.4.0-sources.jar org.apache.had oop.examples.WordCount /data /output

14/05/27 11:44:19 INFO client.RMProxy: Connecting to ResourceManager at nn/192.168.47.139:8132

14/05/27 11:44:21 INFO input.FileInputFormat: Total input paths to process : 2

14/05/27 11:44:22 INFO mapreduce.JobSubmitter: number of splits:2

14/05/27 11:44:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1401216238271_0001

14/05/27 11:44:22 INFO impl.YarnClientImpl: Submitted application application_1401216238271_0001

14/05/27 11:44:23 INFO mapreduce.Job: The url to track the job: http://nn:8188/proxy/application_1401216238271_0001/

14/05/27 11:44:23 INFO mapreduce.Job: Running job: job_1401216238271_0001

14/05/27 11:44:49 INFO mapreduce.Job: Job job_1401216238271_0001 running in uber mode : false

14/05/27 11:44:49 INFO mapreduce.Job: map 0% reduce 0%

14/05/27 11:44:58 INFO mapreduce.Job: Task Id : attempt_1401216238271_0001_m_000000_0, Status : FAILED

Container [pid=7385,containerID=container_1401216238271_0001_01_000002] is running beyond virtual memory limits. Current usage: 67.2 MB of 1.5 GB physical memory used; 4.4 GB of 3.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1401216238271_0001_01_000002 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL _CMD_LINE

|- 7396 7385 7385 7385 (java) 179 18 4726177792 16837 /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=true -D hadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcache/appli cation_1401216238271_0001/container_1401216238271_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app .container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attemp t_1401216238271_0001_m_000000_0 2

|- 7385 7224 7385 7385 (bash) 0 0 17039360 354 /bin/bash -c /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=t rue -Dhadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcach e/application_1401216238271_0001/container_1401216238271_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dy arn.app.container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_ 000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attempt_1401216238271_0001_m_000000_0 2 1>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_14012 16238271_0001_01_000002/stdout 2>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0 001_01_000002/stderr

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

14/05/27 11:45:00 INFO mapreduce.Job: Task Id : attempt_1401216238271_0001_m_000001_0, Status : FAILED

Container [pid=7384,containerID=container_1401216238271_0001_01_000003] is running beyond virtual memory limits. Current usage: 68.6 MB of 1.5 GB physical memory used; 4.4 GB of 3.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1401216238271_0001_01_000003 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL _CMD_LINE

|- 7394 7384 7384 7384 (java) 178 18 4726177792 17213 /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=true -D hadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcache/appli cation_1401216238271_0001/container_1401216238271_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app .container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attemp t_1401216238271_0001_m_000001_0 3

|- 7384 7224 7384 7384 (bash) 0 0 17039360 354 /bin/bash -c /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=t rue -Dhadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcach e/application_1401216238271_0001/container_1401216238271_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dy arn.app.container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_ 000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attempt_1401216238271_0001_m_000001_0 3 1>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_14012 16238271_0001_01_000003/stdout 2>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0 001_01_000003/stderr

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

14/05/27 11:45:08 INFO mapreduce.Job: Task Id : attempt_1401216238271_0001_m_000000_1, Status : FAILED

Container [pid=7441,containerID=container_1401216238271_0001_01_000004] is running beyond virtual memory limits. Current usage: 59.4 MB of 1.5 GB physical memory used; 4.4 GB of 3.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1401216238271_0001_01_000004 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL _CMD_LINE

|- 7441 7224 7441 7441 (bash) 0 0 17039360 354 /bin/bash -c /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=t rue -Dhadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcach e/application_1401216238271_0001/container_1401216238271_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dy arn.app.container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_ 000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attempt_1401216238271_0001_m_000000_1 4 1>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_14012 16238271_0001_01_000004/stdout 2>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0 001_01_000004/stderr

|- 7446 7441 7441 7441 (java) 178 8 4726177792 14843 /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=true -Dh adoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcache/applic ation_1401216238271_0001/container_1401216238271_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app. container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_000004 – Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attempt _1401216238271_0001_m_000000_1 4

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

14/05/27 11:45:08 INFO mapreduce.Job: Task Id : attempt_1401216238271_0001_m_000001_1, Status : FAILED

Container [pid=7456,containerID=container_1401216238271_0001_01_000005] is running beyond virtual memory limits. Current usage: 58.4 MB of 1.5 GB physical memory used; 4.4 GB of 3.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1401216238271_0001_01_000005 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL _CMD_LINE

|- 7461 7456 7456 7456 (java) 177 9 4726177792 14603 /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=true -Dh adoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcache/applic ation_1401216238271_0001/container_1401216238271_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app. container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_000005 – Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attempt _1401216238271_0001_m_000001_1 5

|- 7456 7224 7456 7456 (bash) 0 0 17039360 353 /bin/bash -c /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=t rue -Dhadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcach e/application_1401216238271_0001/container_1401216238271_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dy arn.app.container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_ 000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attempt_1401216238271_0001_m_000001_1 5 1>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_14012 16238271_0001_01_000005/stdout 2>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0 001_01_000005/stderr

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

14/05/27 11:45:17 INFO mapreduce.Job: Task Id : attempt_1401216238271_0001_m_000000_2, Status : FAILED

Container [pid=7089,containerID=container_1401216238271_0001_01_000008] is running beyond virtual memory limits. Current usage: 63.2 MB of 1.5 GB physical memory used; 4.4 GB of 3.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1401216238271_0001_01_000008 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL _CMD_LINE

|- 7089 6803 7089 7089 (bash) 0 1 17039360 353 /bin/bash -c /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=t rue -Dhadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcach e/application_1401216238271_0001/container_1401216238271_0001_01_000008/tmp -Dlog4j.configuration=container-log4j.properties -Dy arn.app.container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_ 000008 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attempt_1401216238271_0001_m_000000_2 8 1>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_14012 16238271_0001_01_000008/stdout 2>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0 001_01_000008/stderr

|- 7095 7089 7089 7089 (java) 182 25 4726177792 15832 /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=true -D hadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcache/appli cation_1401216238271_0001/container_1401216238271_0001_01_000008/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app .container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_000008 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attemp t_1401216238271_0001_m_000000_2 8

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

14/05/27 11:45:17 INFO mapreduce.Job: Task Id : attempt_1401216238271_0001_m_000001_2, Status : FAILED

Container [pid=7107,containerID=container_1401216238271_0001_01_000009] is running beyond virtual memory limits. Current usage: 54.7 MB of 1.5 GB physical memory used; 4.4 GB of 3.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_1401216238271_0001_01_000009 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL _CMD_LINE

|- 7113 7107 7107 7107 (java) 180 19 4726177792 13644 /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=true -D hadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcache/appli cation_1401216238271_0001/container_1401216238271_0001_01_000009/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app .container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_000009 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attemp t_1401216238271_0001_m_000001_2 9

|- 7107 6803 7107 7107 (bash) 0 0 17039360 353 /bin/bash -c /usr/local/jdk1.7.0_55/bin/java -Djava.net.preferIPv4Stack=t rue -Dhadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hadoop/hdfs/tmp/nodemanager/local/usercache/hadoop/appcach e/application_1401216238271_0001/container_1401216238271_0001_01_000009/tmp -Dlog4j.configuration=container-log4j.properties -Dy arn.app.container.log.dir=/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0001_01_ 000009 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.47.141 37535 attempt_1401216238271_0001_m_000001_2 9 1>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_14012 16238271_0001_01_000009/stdout 2>/home/hadoop/hdfs/tmp/nodemanager/logs/application_1401216238271_0001/container_1401216238271_0 001_01_000009/stderr

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

14/05/27 11:45:24 INFO mapreduce.Job: map 100% reduce 100%

14/05/27 11:45:24 INFO mapreduce.Job: Job job_1401216238271_0001 failed with state FAILED due to: Task failed task_1401216238271 _0001_m_000001

Job failed as tasks failed. failedMaps:1 failedReduces:0

14/05/27 11:45:24 INFO mapreduce.Job: Counters: 13

Job Counters

Failed map tasks=7

Killed map tasks=1

Launched map tasks=8

Other local map tasks=6

Data-local map tasks=2

Total time spent by all maps in occupied slots (ms)=50333

Total time spent by all reduces in occupied slots (ms)=0

Total time spent by all map tasks (ms)=50333

Total vcore-seconds taken by all map tasks=50333

Total megabyte-seconds taken by all map tasks=77311488

Map-Reduce Framework

CPU time spent (ms)=0

Physical memory (bytes) snapshot=0

Virtual memory (bytes) snapshot=0

hadoop@nn:~/hadoop-2.4.0$ hadoop jar ./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.4.0-sources.jar org.apache.hadoop.examples.WordCount /data /output4

14/05/27 13:16:57 INFO client.RMProxy: Connecting to ResourceManager at nn/192.168.47.139:8132

14/05/27 13:16:58 INFO input.FileInputFormat: Total input paths to process : 2

14/05/27 13:16:58 INFO mapreduce.JobSubmitter: number of splits:2

14/05/27 13:16:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1401221062327_0001

14/05/27 13:16:59 INFO impl.YarnClientImpl: Submitted application application_1401221062327_0001

14/05/27 13:16:59 INFO mapreduce.Job: The url to track the job: http://nn:8188/proxy/application_1401221062327_0001/

14/05/27 13:16:59 INFO mapreduce.Job: Running job: job_1401221062327_0001

14/05/27 13:17:26 INFO mapreduce.Job: Job job_1401221062327_0001 running in uber mode : false

14/05/27 13:17:26 INFO mapreduce.Job: map 0% reduce 0%

14/05/27 13:18:01 INFO mapreduce.Job: map 100% reduce 0%

14/05/27 13:18:30 INFO mapreduce.Job: map 100% reduce 100%

14/05/27 13:18:30 INFO mapreduce.Job: Job job_1401221062327_0001 completed successfully

14/05/27 13:18:30 INFO mapreduce.Job: Counters: 49

File System Counters

FILE: Number of bytes read=103

FILE: Number of bytes written=282013

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=272

HDFS: Number of bytes written=56

HDFS: Number of read operations=9

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=2

Launched reduce tasks=1

Data-local map tasks=2

Total time spent by all maps in occupied slots (ms)=65907

Total time spent by all reduces in occupied slots (ms)=25452

Total time spent by all map tasks (ms)=65907

Total time spent by all reduce tasks (ms)=25452

Total vcore-seconds taken by all map tasks=65907

Total vcore-seconds taken by all reduce tasks=25452

Total megabyte-seconds taken by all map tasks=67488768

Total megabyte-seconds taken by all reduce tasks=26062848

Map-Reduce Framework

Map input records=6

Map output records=12

Map output bytes=123

Map output materialized bytes=109

Input split bytes=188

Combine input records=12

Combine output records=8

Reduce input groups=7

Reduce shuffle bytes=109

Reduce input records=8

Reduce output records=7

Spilled Records=16

Shuffled Maps =2

Failed Shuffles=0

Merged Map outputs=2

GC time elapsed (ms)=424

CPU time spent (ms)=3410

Physical memory (bytes) snapshot=486014976

Virtual memory (bytes) snapshot=6566645760

Total committed heap usage (bytes)=256843776

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=84

File Output Format Counters

Bytes Written=56

hadoop@nn:~/hadoop-2.4.0$ hadoop fs -rm -r /output

14/05/27 16:38:41 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.

Deleted /output

The post Multinodes Hadoop 2.4 installation and configuration appeared first on Robert Chen.

Prepare the environment

Setup the Hadoop user on all nodes:

add these to /etc/hosts on all 3 hosts:

unzip JDK, and copy to /usr/local/jdk1.7.0_55

copy ssh keys to data nodes

Install Hadoop

Download and unzip on all nodes

change JAVA_HOME in Hadoop-env.sh

update the .bash_profile of Hadoop user

Write configurations

Core-site.xml

Hdfs-site.xml

Mapred-site.xml

Yarn-site.xml

Setup slaves, master

copy files to other nodes

on the master, start namenode, resourcemanager and etc

On the data nodes, starts datanode, nodemanager

check status

Fix the WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform

Recompile with source codes and replace the libs

Trending Articles