时间戳

首页 > 技术文章 > hadoop(七)集群配置同步（hadoop完全分布式四）|9

hadoop(七)集群配置同步（hadoop完全分布式四）|9

shaozhiqi 2019-09-17 16:17 原文

前置配置：rsync远程同步|xsync集群分发(hadoop完全分布式准备三）|9

1. 分布式集群分配原则

部署分配原则
部署分配原则

 

说明
Namenode和secondarynamenode占用内存较大，建议两个分开。我们的环境都是本地虚拟机，部署在一起内存不够。
Resourcemanager是资源保障，比较耗资源，所以错开前两个。

2. 修改hadoop核心配置文件core-site.xml

[shaozhiqi@hadoop102 hadoop]$ vi core-site.xml
vi core-sit.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop102:9000</value>
    </property>
   <property>
       <name>hadoop.tmp.dir</name>
       <value>/opt/module/hadoop-3.1.2/data/tmp</value>
   </property>
</configuration>

3. HDFS配置文件

修改hadoop-evn.sh 让在分布式时获取java配置
由于我们是copy的hadoop文件（在之前的伪分布式中已经配置过了所以不用配了）

[shaozhiqi@hadoop102 hadoop]$ vi hadoop-env.sh
# JAVA_HOME=/usr/java/testing hdfs dfs -ls
export JAVA_HOME=/opt/module/jdk1.8.0_211

配置hdfs-site.xml
指定 Hadoop辅助名称节点主机配置，删除副本数配置，系统默认是3，我们呢改成三配上也没啥意义，所以删掉

[shaozhiqi@hadoop102 hadoop]$ vi hdfs-site.xml
<configuration>
  <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop104:50090</value>
  </property>
</configuration>

4. 配置yarn

配置yarn-env.sh

[shaozhiqi@hadoop102 hadoop]$ vi yarn-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_211

配置yarn-site.xml
配置获取数据方式，配置resoucemanager的地址

[shaozhiqi@hadoop102 hadoop]$ vi yarn-site.xml
<configuration>

<!-- Site specific YARN configuration properties -->
 <property>
   <name> yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
 </property>
 <property>
   <name>yarn.resourcemanager.hostname</name>
   <value> hadoop103</value>
 </property>
</configuration>

5. 配置mapReduce

配置mapred-env.sh

[shaozhiqi@hadoop102 hadoop]$ vi mapred-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_211

配置mapred-site.xml

[shaozhiqi@hadoop102 hadoop]$ vi mapred-site.xml
<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>

6. 执行我们的脚本，同步我们的配置好的文件到其他机器

执行脚本

[shaozhiqi@hadoop103 hadoop]$ testxsync /opt/module/hadoop-3.1.2/

验证结果
说明：去103 104验证找一个文件看下就行
103

[shaozhiqi@hadoop103 hadoop]$ cat  hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop104:50090</value>
  </property>
</configuration>
[shaozhiqi@hadoop103 hadoop]$

104

[shaozhiqi@hadoop104 hadoop]$ cat  hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop104:50090</value>
  </property>
</configuration>
[shaozhiqi@hadoop104 hadoop]$

推荐阅读