一 摘要
本文主要介绍kolla-ansible 离线方式部署 openstack ，共使用两台物理机，1台用户控制节点，另一台用于计算节点、网络、存储节点。
二 环境信息
（一）机器使用说明
2.1.1 机器规划
| 节点名称 | IP | HOSTNAME | CPU | 内存 | 硬盘 | 操作系统 |
|---|---|---|---|---|---|---|
| 控制节点 网络节点 kolla-ansible节点 | 10.3.176.10 | ussuricontroller1.cloud.chouniu.fun | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz*2 | 224G | 4.4T | CentOS Linux release 8.1.1911 (Core) |
| 计算节点 网络节点 | 10.3.176.16 | ussuricompute1.cloud.chouniu.fun | Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz*2 | 128G | 4.4T | CentOS Linux release 8.1.1911 (Core) |
kolla-ansible节点：即部署了kolla-ansible 服务的节点，由她发起安装部署指令。
（二） 软件信息
2.2.1. 操作系统
CentOS Linux release 8.1.1911 (Core)
2.2.2.ansible 版本
ansible 2.9.10
2.2.3 docker 版本
Docker version 18.03.1-ce, build 9ee9f40
2.2.4 kolla 版本
kolla -10.1.0
2.2.5 kolla-ansible 版本
kolla-ansible-10.1.0
三 安装说明
本文是基于上一篇 openstack ussuri 版本 all-in-one 离线部署 ,阅读本篇之前，请先阅读上一篇。
控制节点复用上一篇安装好的控制节点，所以需要先将原安装好的all-in-one openstack 删除，计算节点需要重新安装，不过基础安装与控制节点相同。
四 部署实施
首先 安装控制节点，然后安装计算节点，最后集群部署openstack
(一) 控制节点安装
本控制节点是基于上一遍安装好all-in-one 的控制节点，所以，只需将原先all-in-one openstack destroy 即可。
4.1.1 destroy all-in-one openstack
4.1.1.1 删除all-in-one 镜像命令
kolla-ansible destroy --yes-i-really-really-mean-it
[root@ussuricontroller1 ~]# kolla-ansible destroy
WARNING:
This will PERMANENTLY DESTROY all deployed kolla containers, volumes and host configuration.
There is no way to recover from this action. To confirm, please add the following option:
--yes-i-really-really-mean-it
[root@ussuricontroller1 ~]# kolla-ansible destroy --yes-i-really-really-mean-it
4.1.1.2 结果检查
[root@ussuricontroller1 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root@ussuricontroller1 ~]#
可见 已经没有运行的相关容器了。
如果发现仍有容器没删除掉，请使用docker rm -f 强制删除。
（二）计算节点安装
4.2.1 基础支撑安装及配置
4.2.1.1 关闭防火墙
[root@localhost ~]# systemctl stop firewalld && systemctl disable firewalld &&systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@localhost ~]#
4.2.1.2 永久关闭selinux
修改文件/etc/selinux/config SELINUX=disabled，并重启服务器
4.2.1.3 配置内网dns
参考上一篇openstack ussuri 版本 all-in-one 离线部署 对应章节。
4.2.1.4 配置内网yum 源
参考上一篇openstack ussuri 版本 all-in-one 离线部署 对应章节。
4.2.1.5 内网dns系统配置域名
在内网dns 系统配置域名 ussuricompute1.cloud.chouniu.fun，并指向该IP
4.2.1.6 设置机器名
[root@localhost ~]# hostnamectl set-hostname ussuricompute1.cloud.chouniu.fun
[root@localhost ~]# reboot
4.2.1.7 安装docker
参考上一篇openstack ussuri 版本 all-in-one 离线部署 对应章节。
安装命令
[root@ussuricompute1 ~]# yum remove podman
Modular dependency problems:
[root@ussuricompute1 ~]# yum install docker-ce
修改docker 配置文件 仓库地址改为自有仓库
设置为开机自启
[root@ussuricompute1 ~]# systemctl restart docker
[root@ussuricompute1 ~]# systemctl enable docker
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
[root@ussuricompute1 ~]#
4.2.1.8 安装pip3
[root@ussuricompute1 ~]# whereis pip3.6
pip3: /usr/bin/pip3.6 /usr/bin/pip3 /usr/share/man/man1/pip3.1.gz
[root@ussuricompute1 ~]# ln -s /usr/bin/pip3.6 /usr/bin/pip
[root@ussuricompute1 ~]# pip -V
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)
[root@ussuricompute1 ~]#
4.2.1.9 安装基础依赖
centos8 使用dnf 安装rpm 包，安装python3-devel libffi-devel gcc openssl-devel python3-libselinux git vim bash-completion net-tools 基础依赖包
安装命令
[root@ussuricompute1 ussuri]# tar -zxvf dependencies.tgz
dependencies/
dependencies/bash-completion-2.7-5.el8.noarch.rpm
dependencies/gcc-8.3.1-4.5.el8.x86_64.rpm
dependencies/git-2.18.2-2.el8_1.x86_64.rpm
dependencies/libffi-devel-3.1-21.el8.i686.rpm
dependencies/libffi-devel-3.1-21.el8.x86_64.rpm
dependencies/net-tools-2.0-0.51.20160912git.el8.x86_64.rpm
dependencies/openssl-devel-1.1.1c-2.el8_1.1.i686.rpm
dependencies/openssl-devel-1.1.1c-2.el8_1.1.x86_64.rpm
dependencies/python36-devel-3.6.8-2.module_el8.1.0+245+c39af44f.x86_64.rpm
dependencies/python3-libselinux-2.9-2.1.el8.x86_64.rpm
dependencies/vim-enhanced-8.0.1763-13.el8.x86_64.rpm
[root@ussuricompute1 ussuri]# cd dependencies/
[root@ussuricompute1 dependencies]# dnf install python3-devel libffi-devel gcc openssl-devel python3-libselinux git vim bash-completion net-tools
CentOSBase 2.6 MB/s | 2.9 kB 00:00
ansible 2.8 MB/s | 2.9 kB 00:00
docker-ce 2.8 MB/s | 2.9 kB 00:00
Modular dependency problems:
4.2.1.10 kolla-ansible 节点免密登录到计算节点
kolla-ansible 是利用ansible 做命令分发，借助免密登录更方便ansible 的管理。
4.2.1.10.1 先在kolla-ansible 生成公钥私钥
如果kolla-ansible 没有生成过公钥私钥则需要生成下。
[root@ussuricontroller1 ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:2l+4ZLgTClgtEAPNzKx06NGirbJFdSPNf2Hirx3452g root@ussuricontroller1.cloud.kxdigit.com
The key's randomart image is:
+---[RSA 3072]----+
|.B=. o |
| =B+o = . o |
|+o+o + + o . |
|o.o o . o . |
| o o . S+ |
|o o . oo.o. |
|.o ...o=+.. |
|. . o=E+. |
| .o+o. |
+----[SHA256]-----+
4.2.1.10.2 kolla-ansible 公钥拷贝到计算节点
[root@ussuricontroller1 ~]# ssh-copy-id root@10.3.176.16
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.3.176.16 (10.3.176.16)' can't be established.
ECDSA key fingerprint is SHA256:WrrAJdeLY1rhiMsI5mZ6xRq6Zx0bGtmJ2VFhAZf2X3k.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.3.176.16's password:
Permission denied, please try again.
root@10.3.176.16's password:
Permission denied, please try again.
root@10.3.176.16's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@10.3.176.16'"
and check to make sure that only the key(s) you wanted were added.
[root@ussuricontroller1 ~]#
4.2.1.11 禁用libvirt
[root@ussuricompute1 kollapip]# systemctl stop libvirtd.service && systemctl disable libvirtd.service && systemctl status libvirtd.service^C
[root@ussuricompute1 kollapip]# cd ~
[root@ussuricompute1 ~]# systemctl stop libvirtd.service && systemctl disable libvirtd.service && systemctl status libvirtd.service
Removed /etc/systemd/system/multi-user.target.wants/libvirtd.service.
Removed /etc/systemd/system/sockets.target.wants/virtlogd.socket.
Removed /etc/systemd/system/sockets.target.wants/virtlockd.socket.
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; vendor preset: enabled)
Active: inactive (dead) since Thu 2020-11-05 18:52:20 CST; 154ms ago
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 2465 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/libvirtd.service
├─3290 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leases>
└─3291 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leases>
Nov 05 14:12:33 ussuricompute1.cloud.kxdigit.com dnsmasq[3290]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFT>
Nov 05 14:12:33 ussuricompute1.cloud.kxdigit.com dnsmasq-dhcp[3290]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Nov 05 14:12:33 ussuricompute1.cloud.kxdigit.com dnsmasq-dhcp[3290]: DHCP, sockets bound exclusively to interface virbr0
Nov 05 14:12:33 ussuricompute1.cloud.kxdigit.com dnsmasq[3290]: reading /etc/resolv.conf
Nov 05 14:12:33 ussuricompute1.cloud.kxdigit.com dnsmasq[3290]: using nameserver 10.3.157.201#53
Nov 05 14:12:33 ussuricompute1.cloud.kxdigit.com dnsmasq[3290]: read /etc/hosts - 2 addresses
Nov 05 14:12:33 ussuricompute1.cloud.kxdigit.com dnsmasq[3290]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Nov 05 14:12:33 ussuricompute1.cloud.kxdigit.com dnsmasq-dhcp[3290]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Nov 05 18:52:20 ussuricompute1.cloud.kxdigit.com systemd[1]: Stopping Virtualization daemon...
Nov 05 18:52:20 ussuricompute1.cloud.kxdigit.com systemd[1]: Stopped Virtualization daemon.
lines 1-20/20 (END)
（三）openstack multi-node 一台控制节点+1台计算节点安装
4.3.1 kolla-ansible 节点 基础配置
主要涉及 修改docker配置文件、passwords.yml、/etc/kolla/globals.yml 请参考openstack ussuri 版本 all-in-one 离线部署
4.3.2 multi-node 多节点安装配置
4.3.2.1 备份多节点配置原文件
[root@ussuricontroller1 ansible]# pwd
/root/ansible
[root@ussuricontroller1 ansible]# cp /root/ansible/multinode /root/ansible/multinode.bak.orig
[root@ussuricontroller1 ansible]#
4.3.2.2 编辑 /root/ansible/multinode
主要修改点有：
[control] 配置当前控制节点域名
[network] 配置当前控制节点、计算节点域名
[compute] 配置当前计算节点 域名
[monitoring] 配置当前计算节点域名
[storage] 配置当前计算节点域名
[nova-compute-ironic:children] 配置改为compute
主要修改的配置如下
[control]
# These hostname must be resolvable from your deployment host
#control01
#control02
#control03
ussuricontroller1.cloud.kxdigit.com
# The above can also be specified as follows:
#control[01:03] ansible_user=kolla
# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]
#network01
#network02
ussuricontroller1.cloud.kxdigit.com
ussuricompute1.cloud.kxdigit.com
[compute]
#compute01
ussuricompute1.cloud.kxdigit.com
[monitoring]
#monitoring01
ussuricompute1.cloud.kxdigit.com
# When compute nodes and control nodes use different interfaces,
# you need to comment out "api_interface" and other interfaces from the globals.yml
# and specify like below:
#compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1
[storage]
#storage01
ussuricompute1.cloud.kxdigit.com
[nova-compute-ironic:children]
#nova
compute
4.3.3 kolla-ansible 预检查
4.3.3.1 检查命令
该命令需在 multinode 文件所在目录执行
[root@ussuricontroller1 ssh]# cd /root/ansible/
[root@ussuricontroller1 ansible]# ll
total 36
-rw-r--r--. 1 root root 9584 Nov 4 10:22 all-in-one
-rw-r--r-- 1 root root 10279 Nov 5 16:58 multinode
-rw-r--r-- 1 root root 10058 Nov 5 16:42 multinode.bak.orig
[root@ussuricontroller1 ansible]# kolla-ansible -i multinode prechecks
4.3.3.2 报错排查
TASK [prechecks : Checking docker SDK version] ************************************************************************************************
skipping: [localhost]
fatal: [ussuricompute1.cloud.kxdigit.com]: FAILED! => {"changed": false, "cmd": ["/usr/libexec/platform-python", "-c", "import docker; print(docker.version)"], "delta": "0:00:00.070786", "end": "2020-11-05 17:11:54.142533", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2020-11-05 17:11:54.071747", "stderr": "Traceback (most recent call last):\n File "
ok: [ussuricontroller1.cloud.kxdigit.com]
[root@ussuricompute1 kollapip]# pip install docker-4.2.1-py2.py3-none-any.whl websocket_client-0.57.0-py2.py3-none-any.whl
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Processing ./docker-4.2.1-py2.py3-none-any.whl
Processing ./websocket_client-0.57.0-py2.py3-none-any.whl
Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/lib/python3.6/site-packages (from docker==4.2.1)
Requirement already satisfied: six>=1.4.0 in /usr/lib/python3.6/site-packages (from docker==4.2.1)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python3.6/site-packages (from requests!=2.18.0,>=2.14.2->docker==4.2.1)
Requirement already satisfied: idna<2.8,>=2.5 in /usr/lib/python3.6/site-packages (from requests!=2.18.0,>=2.14.2->docker==4.2.1)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/lib/python3.6/site-packages (from requests!=2.18.0,>=2.14.2->docker==4.2.1)
Installing collected packages: websocket-client, docker
Successfully installed docker-4.2.1 websocket-client-0.57.0
[root@ussuricompute1 kollapip]# pwd
/root/software/ussuri/kollapip
[root@ussuricompute1 kollapip]#
4.3.3.3 预检查成功
PLAY RECAP ************************************************************************************************************************************
localhost : ok=16 changed=0 unreachable=0 failed=0 skipped=14 rescued=0 ignored=0
ussuricompute1.cloud.chouniu.fun : ok=55 changed=0 unreachable=0 failed=0 skipped=131 rescued=0 ignored=0
ussuricontroller1.cloud.chouniu.fun : ok=93 changed=0 unreachable=0 failed=0 skipped=190 rescued=0 ignored=0
4.3.4 多节点部署
首先执行部署命令，然后激活第二张网卡，然后生成环境变量文件
4.3.4.1 部署指令
在该目录下执行/root/ansible
输入部署指令
[root@ussuricontroller1 openstackclient]# cd /root/ansible/
[root@ussuricontroller1 ansible]# ll
total 36
-rw-r--r--. 1 root root 9584 Nov 4 10:22 all-in-one
-rw-r--r-- 1 root root 10279 Nov 5 16:58 multinode
-rw-r--r-- 1 root root 10058 Nov 5 16:42 multinode.bak.orig
[root@ussuricontroller1 ansible]# pwd
/root/ansible
[root@ussuricontroller1 ansible]#
[root@ussuricontroller1 ansible]# kolla-ansible -i multinode deploy
部署成功
PLAY RECAP ************************************************************************************************************************************
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ussuricompute1.cloud.kxdigit.com : ok=135 changed=85 unreachable=0 failed=0 skipped=120 rescued=0 ignored=0
ussuricontroller1.cloud.kxdigit.com : ok=359 changed=202 unreachable=0 failed=0 skipped=229 rescued=0 ignored=1
[root@ussuricontroller1 ansible]#
4.3.5 制作第一个实例
首先要在控制节点和计算节点安装openstackclient ，然后初始化网络、上传镜像模板，制作实例
4.3.5.1 所有节点离线安装openstackclient
所有节点都要执行安装openstack client
相关依赖包都下载了，放在这个目录 /root/software/ussuri/openstackclient
[root@ussuricontroller1 openstackclient]# pip install --no-index --find-links=/root/software/ussuri/openstackclient/ python_openstackclient-5.2.0-py3-none-any.whl
安装成功就能执行相关命令了
Successfully installed PrettyTable-0.7.2 appdirs-1.4.4 attrs-19.3.0 cliff-3.3.0 cmd2-1.1.0 colorama-0.4.3 decorator-4.4.2 dogpile.cache-0.9.2 jsonpatch-1.25 jsonpointer-2.0 keystoneauth1-4.0.0 msgpack-1.0.0 munch-2.5.0 openstacksdk-0.46.0 os-service-types-1.7.0 osc-lib-2.2.0 oslo.serialization-4.0.0 pyperclip-1.8.0 python-cinderclient-7.0.0 python-keystoneclient-4.0.0 python-novaclient-17.0.0 python-openstackclient-5.2.0 requestsexceptions-1.4.0 simplejson-3.17.0 wcwidth-0.2.4
[root@ussuricontroller1 openstackclient]# no
nohup nologin notify-send nova
[root@ussuricontroller1 openstackclient]# no
nohup nologin notify-send nova
[root@ussuricontroller1 openstackclient]# pwd
/root/software/ussuri/openstackclient
[root@ussuricontroller1 openstackclient]# open
open openssl openstack openstack-inventory openvt
[root@ussuricontroller1 openstackclient]# no
nohup nologin notify-send nova
[root@ussuricontroller1 openstackclient]# no
nohup nologin notify-send nova
[root@ussuricontroller1 openstackclient]# nova
4.3.5.2 所有部署节点生成环境变量文件 admin-openrc.sh
该文件帮助您在所有节点直接使用openstack 命令 获取云平台相关数据以及通过命令的方式制作删除实例等
4.3.5.2.1 执行命令
在 /root/ansible/ 目录下执行 kolla-ansible -i multinode post-deploy
[root@ussuricontroller1 openstackclient]# cd /root/ansible/
[root@ussuricontroller1 ansible]# ll
total 36
-rw-r--r--. 1 root root 9584 Nov 4 10:22 all-in-one
-rw-r--r-- 1 root root 10279 Nov 5 16:58 multinode
-rw-r--r-- 1 root root 10058 Nov 5 16:42 multinode.bak.orig
[root@ussuricontroller1 ansible]# pwd
/root/ansible
[root@ussuricontroller1 ansible]# kolla-ansible -i multinode post-deploy
Post-Deploying Playbooks : ansible-playbook -i multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla /usr/local/share/kolla-ansible/ansible/post-deploy.yml
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
PLAY [Creating admin openrc file on the deploy node] ******************************************************************************************
TASK [Gathering Facts] ************************************************************************************************************************
ok: [localhost]
TASK [Template out admin-openrc.sh] ***********************************************************************************************************
changed: [localhost]
PLAY RECAP ************************************************************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
[root@ussuricontroller1 ansible]#
文件生成在该目录
[root@ussuricontroller1 kolla]# pwd
/etc/kolla
[root@ussuricontroller1 kolla]# ll /etc/kolla/admin-openrc.sh
-rw-r--r-- 1 root root 517 Nov 6 09:33 /etc/kolla/admin-openrc.sh
[root@ussuricontroller1 kolla]#
4.3.5.2.2 验证
[root@ussuricontroller1 kolla]# source /etc/kolla/admin-openrc.sh
[root@ussuricontroller1 kolla]# no
nohup nologin notify-send nova
[root@ussuricontroller1 kolla]# no
nohup nologin notify-send nova
[root@ussuricontroller1 kolla]# nova hypervisor-list
+--------------------------------------+----------------------------------+-------+---------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+----------------------------------+-------+---------+
| 691248be-cefd-4c91-b00f-e73b87aa3ba3 | ussuricompute1.cloud.kxdigit.com | up | enabled |
+--------------------------------------+----------------------------------+-------+---------+
[root@ussuricontroller1 kolla]# ll
4.3.5.2.3 该文件复制到其他节点上并验证
复制到计算节点
[root@ussuricontroller1 kolla]# scp /etc/kolla/admin-openrc.sh root@10.3.176.16:/etc/kolla/
admin-openrc.sh 100% 517 1.2MB/s 00:00
[root@ussuricontroller1 kolla]#
计算节点验证
[root@ussuricompute1 ussuri]# source /etc/kolla/admin-openrc.sh
[root@ussuricompute1 ussuri]# nova hypervisor-list
+--------------------------------------+----------------------------------+-------+---------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+----------------------------------+-------+---------+
| 691248be-cefd-4c91-b00f-e73b87aa3ba3 | ussuricompute1.cloud.kxdigit.com | up | enabled |
+--------------------------------------+----------------------------------+-------+---------+
[root@ussuricompute1 ussuri]#
4.3.5.3 激活所有节点第二张网卡
我这里第一张网卡eth0,第二张网卡是eth1
4.3.5.3.1 修改所有节点配置文件/etc/sysconfig/network-scripts/ifcfg-eth1
主要是修改
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eth1
UUID=843f915c-0955-477c-ad86-72f427bfcc98
DEVICE=eth1
ONBOOT=yes
~
4.3.5.3.2 激活eth1
[root@ussuricontroller1 network-scripts]# nmcli c reload /etc/sysconfig/network-scripts/ifcfg-eth1
[root@ussuricontroller1 network-scripts]# nmcli d reapply eth1
Connection successfully reapplied to device 'eth1'.
[root@ussuricontroller1 network-scripts]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:32:fc:4a:1a txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.3.176.10 netmask 255.255.252.0 broadcast 10.3.179.255
inet6 fe80::80ae:384:1962:ba0d prefixlen 64 scopeid 0x20<link>
ether ac:74:09:c4:73:4f txqueuelen 1000 (Ethernet)
RX packets 2075054 bytes 181960331 (173.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 755059 bytes 107116034 (102.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xc7360000-c737ffff
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::915a:3963:3500:81cd prefixlen 64 scopeid 0x20<link>
ether ac:74:09:c4:73:50 txqueuelen 1000 (Ethernet)
RX packets 1817511 bytes 1046115251 (997.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 39 bytes 3343 (3.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xc7340000-c735ffff
好像是 eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
RUNNING 代表网卡激活
4.3.5.4 后台初始化网络
使用后台命令初始化云平台虚机网络，这个也可以在前端页面完成。
4.3.5.4.1 准备示例镜像模板cirros
因为我们是离线安装，自己从互联网下载cirros 镜像 放到该目录下/usr/local/share/kolla-ansible/
[root@ussuricontroller1 kolla-ansible]# ll
total 13008
drwxr-xr-x. 8 root root 4096 Nov 4 10:04 ansible
-rw-r--r-- 1 root root 13287936 Nov 6 11:05 cirros-0.3.4-x86_64-disk.img
drwxr-xr-x. 3 root root 56 Nov 4 10:04 doc
drwxr-xr-x. 3 root root 27 Nov 4 10:04 etc_examples
-rwxr-xr-x. 1 root root 5517 Jul 9 08:18 init-runonce
-rwxr-xr-x 1 root root 5517 Nov 6 11:03 init-runonce.bak.orig
-rwxr-xr-x. 1 root root 2488 Jul 9 08:18 init-vpn
-rw-r--r--. 1 root root 1560 Jul 9 08:18 setup.cfg
drwxr-xr-x. 2 root root 154 Nov 4 10:04 tools
[root@ussuricontroller1 kolla-ansible]#
4.3.5.4.1 修改初始化网络脚本init-runonce
配置文件：/usr/local/share/kolla-ansible/init-runonce
先备份该文件
[root@ussuricontroller1 kolla-ansible]# cp init-runonce init-runonce.bak.orig
修改init-runonce
主要修改三处
报错
修改cirros-0.3.4-x86_64-disk.img IMAGE 变量，改为本地IMAGE
#IMAGE_PATH=/opt/cache/files/
IMAGE_PATH=/usr/local/share/kolla-ansible/
IMAGE_URL=https://github.com/cirros-dev/cirros/releases/download/0.5.1/
#IMAGE=cirros-0.5.1-${ARCH}-disk.img
IMAGE=cirros-0.3.4-x86_64-disk.img
IMAGE_NAME=cirros
IMAGE_TYPE=linux
修改EXT_NET_CIDR ，修改为您对应的网段
# This EXT_NET_CIDR is your public network,that you want to connect to the internet via.
ENABLE_EXT_NET=${ENABLE_EXT_NET:-1}
EXT_NET_CIDR=${EXT_NET_CIDR:-'10.3.176.0/22'}
EXT_NET_RANGE=${EXT_NET_RANGE:-'start=10.3.179.2,end=10.3.179.199'}
EXT_NET_GATEWAY=${EXT_NET_GATEWAY:-'10.3.176.1'}
修改子网 网段
$KOLLA_OPENSTACK_COMMAND network create demo-net
$KOLLA_OPENSTACK_COMMAND subnet create --subnet-range 172.31.164.0/24 --network demo-net \
--gateway 172.31.164.1 --dns-nameserver 8.8.8.8 demo-subnet
这个脚本只能执行一次，主要工作是制作镜像，划分网段、子网、路由、使用镜像启动第一个实例，如果您发现脚本写错了，可以去控制台把相关配置删除了，然后重新修订脚本，重新执行。
执行命令
source /etc/kolla/admin-openrc.sh
source /usr/local/share/kolla-ansible/init-runonce
4.3.5.5 各节点安装openstack 客户端
在各节点上使用openstack 客户端命令，则需要安装相关软件
此章节是后面补充的，文件目录可能有差入，但是安装方法一样。
[root@controller1 openstackclient]# pip install --no-index --find-links=/root/ussuri/openstackclient python_openstackclient-5.2.0-py3-none-any.whl
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
Processing ./python_openstackclient-5.2.0-py3-none-any.whl
Requirement already satisfied: oslo.i18n>=3.15.3 in /usr/local/lib/python3.6/site-packages (from python-openstackclient==5.2.0)
Requirement already satisfied: Babel!=2.4.0,>=2.3.4 in /usr/lib/python3.6/site-packages (from python-openstackclient==5.2.0)
Collecting python-novaclient>=15.1.0 (from python-openstackclient==5.2.0)
安装成功示意
Successfully installed appdirs-1.4.4 attrs-19.3.0 cliff-3.3.0 cmd2-1.1.0 colorama-0.4.3 decorator-4.4.2 dogpile.cache-0.9.2 keystoneauth1-4.0.0 msgpack-1.0.0 munch-2.5.0 openstacksdk-0.46.0 os-service-types-1.7.0 osc-lib-2.2.0 oslo.serialization-4.0.0 pyperclip-1.8.0 python-cinderclient-7.0.0 python-keystoneclient-4.0.0 python-novaclient-17.0.0 python-openstackclient-5.2.0 requestsexceptions-1.4.0 simplejson-3.17.0 wcwidth-0.2.4
[root@controller1 openstackclient]# nova list
ERROR (CommandError): You must provide a user name/id (via --os-username, --os-user-id, env[OS_USERNAME] or env[OS_USER_ID]) or an auth token (via --os-token).
[root@controller1 openstackclient]# source /etc/kolla/admin-openrc.sh
[root@controller1 openstackclient]# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
[root@controller1 openstackclient]#
(三) 问题 启动实例失败
原因应该是该款cpu 不支持openstack ussuri 版本。在尝试修改了配置，均失败告终。
报错点：
[instance: 567b288b-4abb-4986-b0cd-eef13e20d458] qemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
尝试修改/etc/modprobe.d/kvm.conf
去掉该配置#options kvm_intel nested=1，然后重启服务器，然后服务器失联了。。。
# Setting modprobe kvm_intel/kvm_amd nested = 1
# only enables Nested Virtualization until the next reboot or
# module reload. Uncomment the option applicable
# to your system below to enable the feature permanently.
#
# User changes in this file are preserved across upgrades.
#
# For Intel
#options kvm_intel nested=1
#
多节点删除容器
kolla-ansible destroy --yes-i-really-really-mean-it -i /root/ansible/multinode
五 总结
(一) 多节点destroy 命令
因为目前的cpu 不支持openstack ussuri 版本，我在控制节点执行kolla-ansible destroy --yes-i-really-really-mean-it，发现该命令只删除了控制节点容器及相关配置文件，计算节点并未删除，
经研究多节点destroy 使用该命令
kolla-ansible destroy --yes-i-really-really-mean-it -i /root/ansible/multinode