codis是redis的分布式集群模式,由豌豆荚开源,本文简单记录一下它的集群搭建方法。
首先介绍一下我的实验环境。三台配置相同的虚拟机,
[root@test ~]# ip a|grep -w "inet"|grep -vw lo inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0 [root@test ~]# uname -r 2.6.32-754.el6.x86_64 [root@test ~]# cat /etc/redhat-release CentOS release 6.10 (Final) [root@test ~]#
安装codis之前,我们需要准备的有,JAVA环境、GO环境、Zookeeper协调服务,codis安装包
第一步:
安装java环境,三台全部安装,因为都有运行zookeeper。jdk最好去官网下载一个rpm包安装,yum源里的openjdk运行时会有点小问题,为了减少不必要的问题,我们就不使用openjdk了。rpm包自己去下载就好了,下载之后传到服务器上。
[root@test1 ~]# rz -y rz waiting to receive. zmodem trl+C ȡ 100% 166038 KB 15094 KB/s 00:00:11 0 Errors... [root@test1 ~]# ll total 166040 -rw-r--r--. 1 root root 170023183 Nov 2 2018 jdk-8u181-linux-x64.rpm [root@test1 ~]# for host in 11 12;do scp jdk-8u181-linux-x64.rpm 10.0.0.$host:/root;done root@10.0.0.11's password: jdk-8u181-linux-x64.rpm 100% 162MB 23.2MB/s 00:07 root@10.0.0.12's password: jdk-8u181-linux-x64.rpm [root@test1 ~]# for host in 10 11 12;do ssh 10.0.0.$host rpm -ivh jdk-8u181-linux-x64.rpm ;done The authenticity of host '10.0.0.10 (10.0.0.10)' can't be established. RSA key fingerprint is f9:d9:c2:cb:cc:0d:ef:9e:ba:25:32:a3:b0:15:1b:2f. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.0.0.10' (RSA) to the list of known hosts. root@10.0.0.10's password: warning: jdk-8u181-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY Preparing... ################################################## package jdk1.8-2000:1.8.0_181-fcs.x86_64 is already installed root@10.0.0.11's password: warning: jdk-8u181-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY Preparing... ################################################## jdk1.8 ################################################## Unpacking JAR files... tools.jar... plugin.jar... javaws.jar... deploy.jar... rt.jar... jsse.jar... charsets.jar... localedata.jar... root@10.0.0.12's password: warning: jdk-8u181-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY Preparing... ################################################## jdk1.8 ################################################## Unpacking JAR files... tools.jar... plugin.jar... javaws.jar... deploy.jar... rt.jar... jsse.jar... charsets.jar... localedata.jar... [root@test1 ~]#
安装完成之后验证是否成功安装。
[root@test1 ~]# java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) [root@test1 ~]#
第二步:
安装zookeeper,三台服务器全部安装。
下载zookeeper的安装包
[root@test1 ~]# wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/stable/zookeeper-3.4.14.tar.gz
[root@test1 ~]# ll
total 202836
-rw-r--r--. 1 root root 170023183 Nov 2 2018 jdk-8u181-linux-x64.rpm
-rw-r--r--. 1 root root 37676320 Apr 1 22:44 zookeeper-3.4.14.tar.gz
安装zookeeper,修改配置文件,把我们的zookeeper节点加入zookeeper集群中。
[root@test1 ~]# tar xf zookeeper-3.4.14.tar.gz -C /usr/local/ [root@test1 ~]# cd /usr/local/ [root@test1 local]# ls bin etc games include lib lib64 libexec sbin share src zookeeper-3.4.14 [root@test1 local]# mv zookeeper-3.4.14/ zookeeper [root@test1 local]# cd zookeeper/ [root@test1 zookeeper]# cp conf/zoo_sample.cfg conf/zoo.cfg 最后修改后的结果是 [root@test1 conf]# grep -Ev "^#|^$" zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zk/data #zookeeper的数据存放目录,我们要自己创建出来 clientPort=2181 server.1=10.0.0.10:2888:3888 #这三个节点我的zookeeper节点 server.2=10.0.0.11:2888:3888 server.3=10.0.0.12:2888:3888
创建数据目录和myid标识文件
[root@test1 conf]# mkdir /data/zk/data -p #这个目录在三个节点上都要创建 [root@test1 conf]# echo 1 >/data/zk/data/myid #每个节点的myid数值要不相同
启动zookeeper,这里注意一下,要注意服务器的防火墙问题,防火墙会阻止集群节点之间的通信,导致集群无法建立。建议关闭防火墙。
三个节点都要启动
[root@test3 bin]# cd /usr/local/zookeeper/bin/ [root@test3 bin]# ./zkServer.sh start
查看集群的状态
[root@test3 bin]# ./zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Mode: leader
这表明启动成功,并且该节点是集群的领导者
如果出现这种问题,说明你的服务没起来,查看一下服务器上是否启动了zookeeper的进程,如果进程存在,检查一下几台节点之间的防火墙状态,可能是防火墙阻挡了集群通信。
[root@test1 bin]# ./zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Error contacting service. It is probably not running. 查看日志信息发现是防火墙的问题 [root@test1 bin]# tailf zookeeper.out at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:958) 2019-05-10 15:46:09,991 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@185] - Resolved hostname: 10.0.0.12 to address: /10.0.0.12 2019-05-10 15:46:09,991 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@847] - Notification time out: 3200 2019-05-10 15:46:13,195 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot open channel to 2 at election address /10.0.0.11:3888 java.net.NoRouteToHostException: No route to host (Host unreachable) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:958)
第三步
安装GO环境,这个环境只要在安装codis的服务器上配置就好了,我的是10.0.0.10这台机器。
[root@test2 bin]# yum install epel-release -y [root@test2 bin]# yum install go -y
验证:
[root@test1 bin]# go version go version go1.11.5 linux/amd64
到此,我们的准备工作就完成了,限于篇幅原因,下篇写codis安装。