本文介绍如何使用play-docker-dledger.sh搭建rocketmq的集群环境

1 清理环境

这里需要删除刚才docker-compose创建的docker的bridge网络，否则会报错：Error response from daemon: Pool overlaps with other one on this address spa

1.1 查看docker网络

docker network ls

1.2 删除网络

docker network rm docker-compose_default

2 启动rocketmq的dledger集群

#进入对应目录
cd rocketmq-docker/stages/4.7.1/template

#执行play-docker-dledger.sh(如果build的rocketmq是centos，则直接执行成功；否则需要先docker tag一下，参考docker-compose搭建篇）
sh play-docker-dledger.sh

2.1 查看是否启动

执行：docker ps -a

如图，broker都退出了

看下日志报错：docker logs b0a03fa0b4cf

报错：启动broker找不到对应的配置文件broker.conf，我们看下play-docker-dledger.sh脚本。

2.2 查看play-docker-dledger.sh

# Create network
# 这里是创建一个docker的bridge网络，用于nameserver和broker直接通信
docker network create --subnet=172.18.0.0/16 dledger-br

# Start nameserver
# 启动nameserver，这里--net指定了bridge网络
docker run --net dledger-br --ip 172.18.0.11  -d -p 9876:9876 -v `pwd`/data/namesrv/logs:/home/rocketmq/logs --name rmqnamesrv  apacherocketmq/rocketmq:4.7.1 sh mqnamesrv

# Start Brokers
# 这里也同样是使用bridge网络连namserver
docker run --net dledger-br --ip 172.18.0.12 -d -p 30911:30911 -p 30909:30909 -v `pwd`/data/broker0/logs:/home/rocketmq/logs -v `pwd`/data/broker0/store:/home/rocketmq/store -v `pwd`/data/broker0/conf/dledger:/opt/rocketmq-4.7.1/conf/dledger --name rmqbroker --link rmqnamesrv:namesrv -e "NAMESRV_ADDR=namesrv:9876" apacherocketmq/rocketmq:4.7.1 sh mqbroker  -c  ../conf/dledger/broker.conf
docker run --net dledger-br --ip 172.18.0.13 -d -p 30921:30921 -p 30919:30919 -v `pwd`/data/broker1/logs:/home/rocketmq/logs -v `pwd`/data/broker1/store:/home/rocketmq/store -v `pwd`/data/broker1/conf/dledger:/opt/rocketmq-4.7.1/conf/dledger --name rmqbroker1 --link rmqnamesrv:namesrv -e "NAMESRV_ADDR=namesrv:9876" apacherocketmq/rocketmq:4.7.1 sh mqbroker  -c  ../conf/dledger/broker.conf
docker run --net dledger-br --ip 172.18.0.14 -d -p 30931:30931 -p 30929:30929 -v `pwd`/data/broker2/logs:/home/rocketmq/logs -v `pwd`/data/broker2/store:/home/rocketmq/store -v `pwd`/data/broker2/conf/dledger:/opt/rocketmq-4.7.1/conf/dledger --name rmqbroker2 --link rmqnamesrv:namesrv -e "NAMESRV_ADDR=namesrv:9876" apacherocketmq/rocketmq:4.7.1 sh mqbroker  -c  ../conf/dledger/broker.conf

2.3 修改脚本

问题应该是出在启动命令这里：

sh mqbroker -c ../conf/dledger/broker.conf

这里应该写成绝对路径（看来原脚本是有bug的）：

sh mqbroker -c /opt/rocketmq-4.7.1/conf/dledger/broker.conf

脚本内容改成如下：

docker run --net dledger-br --ip 172.18.0.12 -d -p 30911:30911 -p 30909:30909 -v `pwd`/data/broker0/logs:/home/rocketmq/logs -v `pwd`/data/broker0/store:/home/rocketmq/store -v `pwd`/data/broker0/conf/dledger:/opt/rocketmq-4.7.1/conf/dledger --name rmqbroker --link rmqnamesrv:namesrv -e "NAMESRV_ADDR=namesrv:9876" apacherocketmq/rocketmq:4.7.1 sh mqbroker  -c   /opt/rocketmq-4.7.1/conf/dledger/broker.conf
docker run --net dledger-br --ip 172.18.0.13 -d -p 30921:30921 -p 30919:30919 -v `pwd`/data/broker1/logs:/home/rocketmq/logs -v `pwd`/data/broker1/store:/home/rocketmq/store -v `pwd`/data/broker1/conf/dledger:/opt/rocketmq-4.7.1/conf/dledger --name rmqbroker1 --link rmqnamesrv:namesrv -e "NAMESRV_ADDR=namesrv:9876" apacherocketmq/rocketmq:4.7.1 sh mqbroker  -c   /opt/rocketmq-4.7.1/conf/dledger/broker.conf
docker run --net dledger-br --ip 172.18.0.14 -d -p 30931:30931 -p 30929:30929 -v `pwd`/data/broker2/logs:/home/rocketmq/logs -v `pwd`/data/broker2/store:/home/rocketmq/store -v `pwd`/data/broker2/conf/dledger:/opt/rocketmq-4.7.1/conf/dledger --name rmqbroker2 --link rmqnamesrv:namesrv -e "NAMESRV_ADDR=namesrv:9876" apacherocketmq/rocketmq:4.7.1 sh mqbroker  -c   /opt/rocketmq-4.7.1/conf/dledger/broker.conf

2.4 重新运行play-docker-dledger.sh

#进入对应目录
cd rocketmq-docker/stages/4.7.1/template

#执行play-docker-dledger.sh(如果build的rocketmq是centos，则直接执行成功；否则需要先docker tag一下，参考docker-compose搭建篇）
sh play-docker-dledger.sh

输入docker ps -a，查看是否启动

2.5 查看集群信息

docker exec -it rmqbroker1 ./mqadmin clusterList -n {宿主机ip}:9876

看到如图所示内容说明dledger模式的rocketmq单集群OK了。我们可以通过上图中的mqadmin对rocketmq的集群进行相关操作，但是在实际的使用场景里，服务A如果想要使用rocketmq，首先访问的是rocketmq的namserver，但nameserver,broker的实际ip均处于docker的bridge网络中（脚本中的dledger-br），其ip地址如上图的172.18.0.x，外部机器是无法与这个网段进行直接通信（网络隔离）。那么该如何解决这个问题呢？

我们可以使用docker的host网络共享宿主机的网络，这有就可以直接访问，所以还要在play-docker-dledger.sh中改造下脚本。

3. 改造脚本和配置

这里需要改造两个部分，一个是play-docker-dledger.sh，另外是三个broker对应的broker.conf的内容。

3.1 修改play-docker-dledger.sh

假设我的本机的ip为10.10.10.15

3.1.1 nameserver

修改脚本:

修改--net dledger-br为--net host
删除指定ip选项：--ip xxx

play-docker-dledger.sh:

#这里使用host模式
docker run --net host  -d -p 9876:9876 -v `pwd`/data//logs:/home/rocketmq/logs --name rmqnamesrv  dockerhub.qingcloud.com/apacherocketmq/rocketmq:4.7.1 sh mqnamesrv

3.1.2 broker

修改脚本

修改--net dledger-br为--net host
删除--link rmqnamesrv:namesrv
修改NAMESRV_ADDR=namesrv:9876为本机网卡IP：NAMESRV_ADDR=10.10.10.15:9876
删除指定ip选项：--ip xxx

修改broker.conf

rocketmq-docker/stages/4.7.1/template下

data/broker0/conf/dledger/broker.conf
data/broker1/conf/dledger/broker.conf
data/broker2/conf/dledger/broker.conf

修改play-docker-dledger.sh:

docker run --net host -d -p 30911:30911 -p 30909:30909 -v `pwd`/data/broker0/logs:/home/rocketmq/logs -v `pwd`/data/broker0/store:/home/rocketmq/store -v `pwd`/data/broker0/conf/dledger:/opt/rocketmq-4.7.1/conf/dledger --name rmqbroker   -e "NAMESRV_ADDR=10.10.10.15:9876" apacherocketmq/rocketmq:4.7.1 sh mqbroker  -c   /opt/rocketmq-4.7.1/conf/dledger/broker.conf
docker run --net host -d -p 30921:30921 -p 30919:30919 -v `pwd`/data/broker1/logs:/home/rocketmq/logs -v `pwd`/data/broker1/store:/home/rocketmq/store -v `pwd`/data/broker1/conf/dledger:/opt/rocketmq-4.7.1/conf/dledger --name rmqbroker1  -e "NAMESRV_ADDR=10.10.10.15:9876" apacherocketmq/rocketmq:4.7.1 sh mqbroker  -c   /opt/rocketmq-4.7.1/conf/dledger/broker.conf
docker run --net host -d -p 30931:30931 -p 30929:30929 -v `pwd`/data/broker2/logs:/home/rocketmq/logs -v `pwd`/data/broker2/store:/home/rocketmq/store -v `pwd`/data/broker2/conf/dledger:/opt/rocketmq-4.7.1/conf/dledger --name rmqbroker2  -e "NAMESRV_ADDR=10.10.10.15:9876" apacherocketmq/rocketmq:4.7.1 sh mqbroker  -c   /opt/rocketmq-4.7.1/conf/dledger/broker.conf

修改data/broker0/conf/dledger/broker.conf

brokerClusterName = RaftCluster
brokerName=RaftNode00
brokerIP1=10.10.10.15
brokerIP2=10.10.10.15
listenPort=40911
namesrvAddr=10.10.10.15:9876
storePathRootDir=/tmp/rmqstore/node00
storePathCommitLog=/tmp/rmqstore/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n0-10.10.10.15:30911;n1-10.10.10.15:30912;n2-10.10.10.15:30913
## must be unique
dLegerSelfId=n0
sendMessageThreadPoolNums=16

修改data/broker1/conf/dledger/broker.conf

brokerClusterName = RaftCluster
brokerName=RaftNode00
brokerIP1=10.10.10.15
brokerIP2=10.10.10.15
listenPort=40921
namesrvAddr=10.10.10.15:9876
storePathRootDir=/tmp/rmqstore/node00
storePathCommitLog=/tmp/rmqstore/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n0-10.10.10.15:30911;n1-10.10.10.15:30912;n2-10.10.10.15:30913
## must be unique
dLegerSelfId=n1
sendMessageThreadPoolNums=16

修改data/broker2/conf/dledger/broker.conf

brokerClusterName = RaftCluster

brokerName=RaftNode00

brokerIP1=10.10.10.15

brokerIP2=10.10.10.15

listenPort=40931

namesrvAddr=10.10.10.15:9876

storePathRootDir=/tmp/rmqstore/node00

storePathCommitLog=/tmp/rmqstore/node00/commitlog

enableDLegerCommitLog=true

dLegerGroup=RaftNode00

dLegerPeers=n0-10.10.10.15:30911;n1-10.10.10.15:30912;n2-10.10.10.15:30913

## must be unique

dLegerSelfId=n2

sendMessageThreadPoolNums=16

3.2 重新运行play-docker-dledger.sh

#进入对应目录
cd rocketmq-docker/stages/4.7.1/template

#执行play-docker-dledger.sh(如果build的rocketmq是centos，则直接执行成功；否则需要先docker tag一下，参考docker-compose搭建篇）
sh play-docker-dledger.sh