docker - 港口 - 内部服务器问题 - 代码 500 - 调整磁盘空间大小后
问题描述
由于 Harbor 节点上的磁盘空间问题，我们增加它并重新启动进程，但 GUI 无法正常工作，例如复制、保留策略、CVE 扫描
我检查了服务器端的日志：
[root@ip-10-0-25-133 ec2-user]# docker container ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2abf5d05dab7 goharbor/nginx-photon:v1.9.3 "nginx -g 'daemon of…" 6 months ago Up 6 months (healthy) 0.0.0.0:4443->4443/tcp, 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp nginx
f33407aa2177 goharbor/harbor-jobservice:v1.9.3 "/harbor/harbor_jobs…" 6 months ago Up 7 weeks (healthy) harbor-jobservice
6420cc8f84ab goharbor/notary-server-photon:v0.6.1-v1.9.3 "/bin/sh -c 'migrate…" 6 months ago Up 6 months notary-server
2546c8894905 goharbor/harbor-core:v1.9.3 "/harbor/harbor_core" 6 months ago Up 6 months (healthy) harbor-core
c04e08177850 goharbor/clair-photon:v2.1.0-v1.9.3 "/docker-entrypoint.…" 6 months ago Up 6 months (healthy) 6060-6061/tcp clair
5e318f8c61d5 goharbor/notary-signer-photon:v0.6.1-v1.9.3 "/bin/sh -c 'migrate…" 6 months ago Up 6 months notary-signer
2555d6e8ff9e goharbor/chartmuseum-photon:v0.9.0-v1.9.3 "/docker-entrypoint.…" 6 months ago Up 6 months (healthy) 9999/tcp chartmuseum
599417cd4413 goharbor/harbor-registryctl:v1.9.3 "/harbor/start.sh" 6 months ago Up 6 months (healthy) registryctl
8967774fa4f2 goharbor/redis-photon:v1.9.3 "redis-server /etc/r…" 6 months ago Up 6 months (healthy) 6379/tcp redis
c4181096df06 goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.3 "/entrypoint.sh /etc…" 6 months ago Up 6 months (healthy) 5000/tcp registry
28583420d59b goharbor/harbor-db:v1.9.3 "/docker-entrypoint.…" 6 months ago Up 6 months (healthy) 5432/tcp harbor-db
f06fa3ac558d goharbor/harbor-portal:v1.9.3 "nginx -g 'daemon of…" 6 months ago Up 6 months (healthy) 8080/tcp harbor-portal
a358bc227c1c goharbor/harbor-log:v1.9.3 "/bin/sh -c /usr/loc…" 6 months ago Up 6 months (healthy) 127.0.0.1:1514->10514/tcp harbor-log
[root@ip-10-0-25-133 ec2-user]# docker logs harbor-log
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
有一个与 Harbor 相关的 rsyslogd 错误，我认为它阻止了 GUI 上的先前操作。
尝试通过 docker compose 重新启动容器无济于事，2 个容器处于不健康状态：
[root@ip-10-0-25-133 harbor]# docker-compose start
Starting log ... done
Starting registry ... done
Starting registryctl ... done
Starting postgresql ... done
Starting portal ... done
Starting redis ... done
Starting core ... done
Starting jobservice ... done
Starting proxy ... done
Starting notary-signer ... done
Starting notary-server ... done
Starting clair ... done
Starting chartmuseum ... done
[root@ip-10-0-25-133 harbor]# docker container ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2abf5d05dab7 goharbor/nginx-photon:v1.9.3 "nginx -g 'daemon of…" 7 months ago Up 3 seconds (health: starting) 0.0.0.0:4443->4443/tcp, 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp nginx
f33407aa2177 goharbor/harbor-jobservice:v1.9.3 "/harbor/harbor_jobs…" 7 months ago Up 3 seconds (health: starting) harbor-jobservice
6420cc8f84ab goharbor/notary-server-photon:v0.6.1-v1.9.3 "/bin/sh -c 'migrate…" 7 months ago Up 4 seconds notary-server
2546c8894905 goharbor/harbor-core:v1.9.3 "/harbor/harbor_core" 7 months ago Up 2 seconds (health: starting) harbor-core
c04e08177850 goharbor/clair-photon:v2.1.0-v1.9.3 "/docker-entrypoint.…" 7 months ago Up 3 seconds (health: starting) 6060-6061/tcp clair
5e318f8c61d5 goharbor/notary-signer-photon:v0.6.1-v1.9.3 "/bin/sh -c 'migrate…" 7 months ago Up 5 seconds notary-signer
2555d6e8ff9e goharbor/chartmuseum-photon:v0.9.0-v1.9.3 "/docker-entrypoint.…" 7 months ago Up 8 seconds (health: starting) 9999/tcp chartmuseum
599417cd4413 goharbor/harbor-registryctl:v1.9.3 "/harbor/start.sh" 7 months ago Up 7 seconds (health: starting) registryctl
8967774fa4f2 goharbor/redis-photon:v1.9.3 "redis-server /etc/r…" 7 months ago Up 7 seconds (health: starting) 6379/tcp redis
c4181096df06 goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.3 "/entrypoint.sh /etc…" 7 months ago Up 7 seconds (health: starting) 5000/tcp registry
28583420d59b goharbor/harbor-db:v1.9.3 "/docker-entrypoint.…" 7 months ago Up 7 seconds (health: starting) 5432/tcp harbor-db
f06fa3ac558d goharbor/harbor-portal:v1.9.3 "nginx -g 'daemon of…" 7 months ago Up 7 seconds (health: starting) 8080/tcp harbor-portal
a358bc227c1c goharbor/harbor-log:v1.9.3 "/bin/sh -c /usr/loc…" 7 months ago Up 9 seconds (health: starting) 127.0.0.1:1514->10514/tcp harbor-log
港口日志再次指向磁盘空间问题：
[root@ip-10-0-25-133 harbor]# docker logs harbor-jobservice
Error response from daemon: configured logging driver does not support reading
[root@ip-10-0-25-133 harbor]# docker logs clair
Error response from daemon: configured logging driver does not support reading
[root@ip-10-0-25-133 harbor]# docker logs harbor-log
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
docker 主机上的磁盘空间未满：
[root@ip-10-0-25-133 harbor]# df -ha
Filesystem Size Used Avail Use% Mounted on
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
devtmpfs 3.8G 0 3.8G 0% /dev
securityfs 0 0 0 - /sys/kernel/security
tmpfs 3.9G 0 3.9G 0% /dev/shm
devpts 0 0 0 - /dev/pts
tmpfs 3.9G 383M 3.5G 10% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
cgroup 0 0 0 - /sys/fs/cgroup/systemd
pstore 0 0 0 - /sys/fs/pstore
bpf 0 0 0 - /sys/fs/bpf
cgroup 0 0 0 - /sys/fs/cgroup/devices
cgroup 0 0 0 - /sys/fs/cgroup/freezer
cgroup 0 0 0 - /sys/fs/cgroup/pids
cgroup 0 0 0 - /sys/fs/cgroup/net_cls,net_prio
cgroup 0 0 0 - /sys/fs/cgroup/hugetlb
cgroup 0 0 0 - /sys/fs/cgroup/cpuset
cgroup 0 0 0 - /sys/fs/cgroup/perf_event
cgroup 0 0 0 - /sys/fs/cgroup/memory
cgroup 0 0 0 - /sys/fs/cgroup/cpu,cpuacct
cgroup 0 0 0 - /sys/fs/cgroup/blkio
cgroup 0 0 0 - /sys/fs/cgroup/rdma
configfs 0 0 0 - /sys/kernel/config
/dev/xvda2 489G 383G 107G 79% /
selinuxfs 0 0 0 - /sys/fs/selinux
systemd-1 - - - - /proc/sys/fs/binfmt_misc
hugetlbfs 0 0 0 - /dev/hugepages
debugfs 0 0 0 - /sys/kernel/debug
mqueue 0 0 0 - /dev/mqueue
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
tracefs 0 0 0 - /sys/kernel/debug/tracing
tmpfs 782M 0 782M 0% /run/user/1000
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/cac923a265f3adc2ba8a8b3f14a0fb21330db4a73562a484475cc782deb000fb/merged
shm 64M 0 64M 0% /var/lib/docker/containers/a358bc227c1cc3f986255adc03d354b773fb18f7a383182632b303e8977ddc0a/mounts/shm
nsfs 0 0 0 - /run/docker/netns/b0b97ca08f9a
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/b8f287f1437be444902922a600eb4ae1fb83c9de1be9d439f9d36a08995ff544/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/6db8129dde48fcf90260c06d287e2e02db335dc80d097d8166b92ef0c1105e4e/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/69d464cb11c9c12dd754190f734eaabbed2fe867e5cfa76d7812a068e0801602/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/b2177176bd4c93cf12ff717aabe563c8469cf6bec426ca17e8d82df0618a3b0e/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/11b8dd2688d07ad2a1b81fe11f235f7d3e648580c32c7757109d15fad8aa9030/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/8b0fe0a0b616f6d2343a919e62590b62b0bb714fbce4373936d5a3962d0cb12c/merged
shm 64M 0 64M 0% /var/lib/docker/containers/599417cd44133ac65ed8f084e1c645bf04b41548fc71cf150aa3e2ef224df7a8/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/2555d6e8ff9e3586a41f6f37fc66244d0b3496187da724787cfac175c7f999c4/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/f06fa3ac558da59cf8be1387b811e6d4e622d1ea49a4d97277eb68edf05e6388/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/8967774fa4f28c3df91ca40722dad81949b1d6c2fe60e3bbcf083cf325ebd7f2/mounts/shm
nsfs 0 0 0 - /run/docker/netns/a6281418be81
nsfs 0 0 0 - /run/docker/netns/053f162aea2a
nsfs 0 0 0 - /run/docker/netns/2c01de1d2aa5
shm 64M 4.0K 64M 1% /var/lib/docker/containers/28583420d59b3e4316b72f7a4d56d5ddf5fdf8c5932860f1f017d7b47ce2b0cf/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/c4181096df06edcddf3b2406556523ef26a47c7f441382bcd42446802cb708ec/mounts/shm
nsfs 0 0 0 - /run/docker/netns/19276a843cd7
nsfs 0 0 0 - /run/docker/netns/8d1949d48460
nsfs 0 0 0 - /run/docker/netns/85ba684c4adf
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/7faa4fa7c2fa7247d4f7a889f7d47ac97aa5002d83a03fc28bfa48e951e860ad/merged
shm 64M 0 64M 0% /var/lib/docker/containers/5e318f8c61d509346c688b4a3514597f571654620e4bd04dac536caaaa41ddf5/mounts/shm
nsfs 0 0 0 - /run/docker/netns/f8e546c2d757
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/d93e3d145924701e719d7a5fd9e93531c36b46407052e14ec578ca887678f94d/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/92736d5ae32243282bb26ebf9410f645388e5b8b1c1d498c369ad56282211dc6/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/19653cd43524e1094144b7daaeb55b3305f688c1d64bd6eacdd899f2b7c57c38/merged
shm 64M 0 64M 0% /var/lib/docker/containers/6420cc8f84ab45b145e6f142154c3ad3c09a3aa69455ded75fb33aab3415183c/mounts/shm
nsfs 0 0 0 - /run/docker/netns/c31bdc8081b5
shm 64M 0 64M 0% /var/lib/docker/containers/f33407aa21776052609f2edaec1f0c637069992279cc9d19968f2e2d77b2d965/mounts/shm
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/6f53a0994736d049eebd058f2f188e606acfa753e428e605a22943074eca7428/merged
nsfs 0 0 0 - /run/docker/netns/02436d61c648
shm 64M 0 64M 0% /var/lib/docker/containers/2abf5d05dab7d47f8fc528f78d112f7a0c3404d87b60649e897f349439e86c58/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/c04e081778502e8b8e0137512f6d9275daefa2d3641ce220f6d1f7e29a40b378/mounts/shm
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/63e4dbe4d6213f6dcc1322f3691c7ff3244fb71e6562888b955d6bc4f93c74a4/merged
shm 64M 0 64M 0% /var/lib/docker/containers/2546c88949055855eef6967c63365b97e06ca38f46c781a336e2e6d7e67226bf/mounts/shm
nsfs 0 0 0 - /run/docker/netns/9f232ba1bf77
nsfs 0 0 0 - /run/docker/netns/1f50ca91cfa6
nsfs 0 0 0 - /run/docker/netns/2a0c5ac700e3
[root@ip-10-0-25-133 harbor]#
这无济于事：https ://www.rsyslog.com/solving-rsyslog-write-errors/
解决方案
我遇到了同样的问题，以下命令为我解决了这个问题：
docker-compose down -v （if failed please reboot system）
docker-compose up -d
推荐阅读
- angular - 连接链接的mongodb问题
- node.js - IBM Watson Assistant：如何使 Db2 的查询结果可下载为 CSV 文件？
- powershell - 如果对 Array 使用 -notlike 不起作用
- javafx - Gluon Mobile (JavaFX)：无法执行目标 com.gluonhq:client-maven-plugin:0.1.38
- javascript - 为什么 Python 比 JavaScript 慢这么多？它能赶上吗？
- linux - Flutter for linux 构建失败
- jquery - 添加字段不会添加到 POST
- c - 为什么我在 xcode 上的 C 代码中有“期望表达式”？
- python - 如何将列中的数字映射到数据框中的字符串
- windows - Powershell - 在目录中创建目录表单文件