Spark On Yarn(HDFS HA)详细配置过程-spark yarn

一、服务器分布及相关说明

1、服务器角色

2、Hadoop（HDFS HA）总体架构

二、基础环境部署

1、JDK安装

http://download.oracle.com/otn-pub/java/jdk/7u45-b18/jdk-7u45-linux-x64.tar.gz

# tar xvzf jdk-7u45-linux-x64.tar.gz -C/usr/local 
# cd /usr/local 
# ln -s jdk1.7.0_45 jdk 
 
# vim /etc/profile 
export JAVA_HOME=/usr/local/jdk 
export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib 
export PATH=$PATH:$JAVA_HOME/bin 
 
# source /etc/profile

2、Scala安装

http://www.scala-lang.org/files/archive/scala-2.10.3.tgz 
# tar xvzf scala-2.10.3.tgz -C/usr/local 
# cd /usr/local 
# ln -s scala-2.10.3 scala 
  
# vim /etc/profile 
export SCALA_HOME=/usr/local/scala 
export PATH=$PATH:$SCALA_HOME/bin 
  
# source /etc/profile

3、SSH免密码登录

可参考文章：

http://blog.csdn.net/codepeak/article/details/14447627

......

4、主机名设置

# vim /etc/hosts 
172.18.35.29     namenode1 
172.18.35.30     namenode2 
172.18.34.232   datanode1 
172.18.24.136   datanode2

三、ZooKeeper集群部署

1、ZooKeeper安装

http://apache.dataguru.cn/zookeeper/stable/zookeeper-3.4.5.tar.gz 
# tar xvzf zookeeper-3.4.5.tar.gz -C/usr/local 
# cd /usr/local 
# ln -s zookeeper-3.4.5 zookeeper 
  
# vim /etc/profile 
export ZOO_HOME=/usr/local/zookeeper 
export ZOO_LOG_DIR=/data/hadoop/zookeeper/logs 
export PATH=$PATH:$ZOO_HOME/bin 
  
# source /etc/profile

2、ZooKeeper配置与启动

# mkdir -p/data/hadoop/zookeeper/{data,logs} 
  
# vim /usr/local/zookeeper/conf/zoo.cfg 
tickTime=2000 
initLimit=10 
syncLimit=5 
  
dataDir=/data/hadoop/zookeeper/data 
clientPort=2181 
  
server.1=172.18.35.29:2888:3888 
server.2=172.18.35.30:2888:3888 
server.3=172.18.34.232:2888:3888

在172.18.35.29上执行：

echo 1 > /data/hadoop/zookeeper/data/myid

在172.18.35.30 上执行：

echo 2 > /data/hadoop/zookeeper/data/myid

在172.18.34.232 上执行：

echo 3 > /data/hadoop/zookeeper/data/myid

## 启动ZooKeeper集群

# cd /usr/local/zookeeper && bin/zkServer.sh start

# ./bin/zkCli.sh -server localhost:2181

测试zookeeper集群是否建立成功，如无报错表示集群创建成功

# bin/zkServer.sh status

四、Hadoop（HDFS HA）集群部署

1、hadoop环境安装

Hadoop的源码编译部分可以参考：

http://sofar.blog.51cto.com/353572/1352713

# tar xvzf hadoop-2.2.0.tgz -C/usr/local 
# cd /usr/local 
# ln -s hadoop-2.2.0 hadoop 
  
# vim /etc/profile 
export HADOOP_HOME=/usr/local/hadoop 
export HADOOP_PID_DIR=/data/hadoop/pids 
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 
export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native" 
  
export HADOOP_MAPRED_HOME=$HADOOP_HOME 
export HADOOP_COMMON_HOME=$HADOOP_HOME 
export HADOOP_HDFS_HOME=$HADOOP_HOME 
export YARN_HOME=$HADOOP_HOME 
  
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop 
export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop 
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop 
  
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native 
  
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin 
  
# mkdir -p /data/hadoop/{pids,storage} 
# mkdir -p/data/hadoop/storage/{hdfs,tmp,journal} 
# mkdir -p/data/hadoop/storage/tmp/nodemanager/{local,remote,logs} 
# mkdir -p/data/hadoop/storage/hdfs/{name,data}

2、core.site.xml配置

# vim/usr/local/hadoop/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<configuration> 
    <property> 
        <name>fs.defaultFS</name> 
        <value>hdfs://appcluster</value> 
    </property> 
                                                                                                                                                              
    <property> 
        <name>io.file.buffer.size</name> 
        <value>131072</value> 
    </property> 
                                                                                                                                                              
    <property> 
        <name>hadoop.tmp.dir</name> 
        <value>file:/data/hadoop/storage/tmp</value> 
    </property> 
                                                                                                                                                              
    <property> 
        <name>ha.zookeeper.quorum</name> 
        <value>172.18.35.29:2181,172.18.35.30:2181,172.18.34.232:2181</value> 
    </property> 
                                                                                                                                                              
    <property> 
        <name>ha.zookeeper.session-timeout.ms</name> 
        <value>2000</value> 
    </property> 
                                                                                                                                                              
    <property> 
        <name>fs.trash.interval</name> 
        <value>4320</value> 
    </property> 
                                                                                                                                                              
    <property> 
         <name>hadoop.http.staticuser.use</name> 
         <value>root</value> 
    </property> 
                                                                                                                                                              
    <property> 
        <name>hadoop.proxyuser.hadoop.hosts</name> 
        <value>*</value> 
    </property> 
                                                                                                                                                              
    <property> 
        <name>hadoop.proxyuser.hadoop.groups</name> 
        <value>*</value> 
    </property> 
                                                                                                                                                              
    <property> 
        <name>hadoop.native.lib</name> 
        <value>true</value> 
    </property> 
</configuration>

3、hdfs-site.xml配置

# vim/usr/local/hadoop/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<configuration> 
    <property> 
        <name>dfs.namenode.name.dir</name> 
        <value>file:/data/hadoop/storage/hdfs/name</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.datanode.data.dir</name> 
        <value>file:/data/hadoop/storage/hdfs/data</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.replication</name> 
        <value>2</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.blocksize</name> 
        <value>67108864</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.datanode.du.reserved</name> 
        <value>53687091200</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.webhdfs.enabled</name> 
        <value>true</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.permissions</name> 
        <value>false</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.permissions.enabled</name> 
        <value>false</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.nameservices</name> 
        <value>appcluster</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.ha.namenodes.appcluster</name> 
        <value>nn1,nn2</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.namenode.rpc-address.appcluster.nn1</name> 
        <value>namenode1:8020</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.namenode.rpc-address.appcluster.nn2</name> 
        <value>namenode2:8020</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.namenode.servicerpc-address.appcluster.nn1</name> 
        <value>namenode1:53310</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.namenode.servicerpc-address.appcluster.nn2</name> 
        <value>namenode2:53310</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.namenode.http-address.appcluster.nn1</name> 
        <value>namenode1:8080</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.namenode.http-address.appcluster.nn2</name> 
        <value>namenode2:8080</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.datanode.http.address</name> 
        <value>0.0.0.0:8080</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.namenode.shared.edits.dir</name> 
        <value>qjournal://namenode1:8485;namenode2:8485;datanode1:8485/appcluster</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.client.failover.proxy.provider.appcluster</name> 
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.ha.fencing.methods</name> 
        <value>sshfence(root:36000)</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.ha.fencing.ssh.private-key-files</name> 
        <value>/root/.ssh/id_dsa_nn1</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.ha.fencing.ssh.connect-timeout</name> 
        <value>30000</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.journalnode.edits.dir</name> 
        <value>/data/hadoop/storage/hdfs/journal</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.ha.automatic-failover.enabled</name> 
        <value>true</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>ha.failover-controller.cli-check.rpc-timeout.ms</name> 
        <value>60000</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>ipc.client.connect.timeout</name> 
        <value>60000</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.image.transfer.bandwidthPerSec</name> 
        <value>41943040</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.namenode.accesstime.precision</name> 
        <value>3600000</value> 
    </property> 
                                                                                                                                                  
    <property> 
        <name>dfs.datanode.max.transfer.threads</name> 
        <value>4096</value> 
    </property> 
</configuration>

4、mapred-site.xml配置

# vim/usr/local/hadoop/etc/hadoop/mapred-site.xml

<?xml version="1.0"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<configuration> 
    <property> 
        <name>mapreduce.framework.name</name> 
        <value>yarn</value> 
    </property> 
                                                                                                                                       
    <property> 
        <name>mapreduce.jobhistory.address</name> 
        <value>namenode1:10020</value> 
    </property> 
                                                                                                                                       
    <property> 
        <name>mapreduce.jobhistory.webapp.address</name> 
        <value>namenode1:19888</value> 
    </property> 
</configuration>

5、yarn-site.xml配置

# vim/usr/local/hadoop/etc/hadoop/yarn-site.xml

<?xml version="1.0"?> 
<configuration> 
    <property> 
        <name>yarn.nodemanager.aux-services</name> 
        <value>mapreduce_shuffle</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> 
        <value>org.apache.hadoop.mapred.ShuffleHandler</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.resourcemanager.scheduler.address</name> 
        <value>namenode1:8030</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.resourcemanager.resource-tracker.address</name> 
        <value>namenode1:8031</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.resourcemanager.address</name> 
        <value>namenode1:8032</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.resourcemanager.admin.address</name> 
        <value>namenode1:8033</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.nodemanager.address</name> 
        <value>namenode1:8034</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.nodemanager.webapp.address</name> 
        <value>namenode1:80</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.resourcemanager.webapp.address</name> 
        <value>namenode1:80</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.nodemanager.local-dirs</name> 
        <value>${hadoop.tmp.dir}/nodemanager/local</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.nodemanager.remote-app-log-dir</name> 
        <value>${hadoop.tmp.dir}/nodemanager/remote</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.nodemanager.log-dirs</name> 
        <value>${hadoop.tmp.dir}/nodemanager/logs</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.nodemanager.log.retain-seconds</name> 
        <value>604800</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.nodemanager.resource.cpu-vcores</name> 
        <value>16</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.nodemanager.resource.memory-mb</name> 
        <value>50320</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.scheduler.minimum-allocation-mb</name> 
        <value>256</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.scheduler.maximum-allocation-mb</name> 
        <value>40960</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.scheduler.minimum-allocation-vcores</name> 
        <value>1</value> 
    </property> 
                                                                                                         
    <property> 
        <name>yarn.scheduler.maximum-allocation-vcores</name> 
        <value>8</value> 
    </property> 
</configuration>

【注意：上面的第68`96行部分，需要根据服务器的硬件配置进行修改】

6、配置hadoop-env.sh、mapred-env.sh、yarn-env.sh【在开头添加】

文件路径：

/usr/local/hadoop/etc/hadoop/hadoop-env.sh 
/usr/local/hadoop/etc/hadoop/mapred-env.sh 
/usr/local/hadoop/etc/hadoop/yarn-env.sh

添加内容：

export JAVA_HOME=/usr/local/jdk 
export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib 
  
export HADOOP_HOME=/usr/local/hadoop 
export HADOOP_PID_DIR=/data/hadoop/pids 
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 
export HADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=$HADOOP_HOME/lib/native" 
  
export HADOOP_PREFIX=$HADOOP_HOME 
  
export HADOOP_MAPRED_HOME=$HADOOP_HOME 
export HADOOP_COMMON_HOME=$HADOOP_HOME 
export HADOOP_HDFS_HOME=$HADOOP_HOME 
export YARN_HOME=$HADOOP_HOME 
  
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop 
export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop 
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop 
  
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native 
  
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

7、数据节点配置

# vim /usr/local/hadoop/etc/hadoop/slaves 
datanode1 
datanode2

8、集群启动

(1)、在namenode1上执行，创建命名空间

# hdfs zkfc -formatZK

(2)、在对应的节点上启动日志程序journalnode

# cd /usr/local/hadoop && ./sbin/hadoop-daemon.sh start journalnode

(3)、格式化主NameNode节点（namenode1）

# hdfs namenode -format

(4)、启动主NameNode节点

# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode

(5)、格式备NameNode节点（namenode2）

# hdfs namenode -bootstrapStandby

(6)、启动备NameNode节点（namenode2）

# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode

----------------------------------------------------------------------------------------------------------------------------------------------

(7)、在两个NameNode节点（namenode1、namenode2）上执行

# cd /usr/local/hadoop && sbin/hadoop-daemon.shstart zkfc

----------------------------------------------------------------------------------------------------------------------------------------------

(8)、启动所有的DataNode节点（datanode1、datanode2）

# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start datanode

----------------------------------------------------------------------------------------------------------------------------------------------

(9)、启动Yarn（namenode1）

# cd /usr/local/hadoop && sbin/start-yarn.sh

主NameNode节点上的信息：

DataNode节点上的信息：

----------------------------------------------------------------------------------------------------------------------------------------------

(10)、测试Yarn是否可用

# hdfs dfs -put/usr/local/hadoop/etc/hadoop/yarn-site.xml /tmp

# hadoop jar/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarwordcount /tmp/yarn-site.xml /mytest

-----------------------------------------------------------------------------------------------------------------------------------------------

(11)、HDFS的HA功能测试

切换前的状态：

# kill -9 11466

# cd /usr/local/hadoop && sbin/hadoop-daemon.sh start namenode

切换后的状态：

-----------------------------------------------------------------------------------------------------------------------------------------------

(12)、后续维护

HDFS的关闭与启动：

# cd /usr/local/hadoop && sbin/stop-dfs.sh

# cd /usr/local/hadoop && sbin/start-dfs.sh

YARN的关闭与启动：

# cd /usr/local/hadoop && sbin/stop-yarn.sh

# cd /usr/local/hadoop && sbin/start-yarn.sh

【注意】

需要在NameNode节点上执行。

五、Spark集群部署

1、Spark安装与配置

Spark的源码编译请参考：

http://sofar.blog.51cto.com/353572/1358457

# tar xvzf spark-0.9.0-incubating.tgz -C/usr/local

# cd /usr/local

# ln -s spark-0.9.0-incubating spark

# vim /etc/profile

export SPARK_HOME=/usr/local/spark

export PATH=$PATH:$SPARK_HOME/bin

# source /etc/profile

# cd /usr/local/spark/conf

# mkdir -p /data/spark/tmp

----------------------------------------------------------------------------------------------------------------------------------------------

# vim spark-env.sh

export JAVA_HOME=/usr/local/jdk

export SCALA_HOME=/usr/local/scala

export HADOOP_HOME=/usr/local/hadoop

SPARK_LOCAL_DIR="/data/spark/tmp"

SPARK_JAVA_OPTS="-Dspark.storage.blockManagerHeartBeatMs=60000-Dspark.local.dir=$SPARK_LOCAL_DIR -XX:+PrintGCDetails -XX:+PrintGCTi

meStamps -Xloggc:$SPARK_HOME/logs/gc.log -XX:+UseConcMarkSweepGC-XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFracti

on=60"

----------------------------------------------------------------------------------------------------------------------------------------------

# vim slaves

datanode1

datanode2

# cd /usr/local/spark && sbin/start-all.sh

=========================================================================================

2、相关测试

(1)、本地模式

# bin/run-exampleorg.apache.spark.examples.SparkPi local

----------------------------------------------------------------------------------------------------------------------------------------------

(2)、普通集群模式

# bin/run-exampleorg.apache.spark.examples.SparkPi spark://namenode1:7077

# bin/run-exampleorg.apache.spark.examples.SparkLR spark://namenode1:7077

# bin/run-exampleorg.apache.spark.examples.SparkKMeans spark://namenode1:7077file:/usr/local/spark/data/kmeans_data.txt 2 1

----------------------------------------------------------------------------------------------------------------------------------------------

(3)、结合HDFS的集群模式

# hadoop fs -put README.md .

# MASTER=spark://namenode1:7077bin/spark-shell

scala> val file =sc.textFile("hdfs://namenode1:9000/user/root/README.md")

scala> val count = file.flatMap(line=> line.split(" ")).map(word => (word, 1)).reduceByKey(_+_)

scala> count.collect()

scala> :quit

----------------------------------------------------------------------------------------------------------------------------------------------

(4)、基于YARN模式

#SPARK_JAR=assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar\

bin/spark-classorg.apache.spark.deploy.yarn.Client \

--jarexamples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar \

--classorg.apache.spark.examples.SparkPi \

--args yarn-standalone \

--num-workers 3 \

--master-memory 4g \

--worker-memory 2g \

--worker-cores 1

----------------------------------------------------------------------------------------------------------------------------------------------

(5)、最终的目录结构及相关配置

目录结构：

配置文件“/etc/profile”中的环境变量设置：