时间戳

首页 > 解决方案 > AWS EC2 磁盘已满，无法释放空间

unix - AWS EC2 磁盘已满，无法释放空间

问题描述

我正在使用 EC2 实例来运行节点应用程序。一段时间后我登录服务器才意识到服务器的磁盘空间已用完。经过调试，我意识到日志正在占用空间。我删除了 3.3Gb 日志文件。但是，即使在清理之后也没有空间。我应该怎么办？

这是我运行的命令：

ubuntu@app1:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
tmpfs           100M   11M   89M  11% /run
/dev/xvda1      7.7G  7.7G     0 100% /
tmpfs           496M  8.0K  496M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           496M     0  496M   0% /sys/fs/cgroup
tmpfs           100M     0  100M   0% /run/user/1001
tmpfs           100M     0  100M   0% /run/user/1000

ubuntu@app1:~$ sudo du -h --max-depth=1 / | sort -n
0   /proc
0   /sys
4.0K    /lib64
4.0K    /media
4.0K    /mnt
4.0K    /srv
8.0K    /dev
8.0K    /snap
16K /lost+found
24K /root
800K    /tmp
6.4M    /etc
11M /run
14M /sbin
16M /bin
246M    /boot
331M    /home
397M    /opt
429M    /var
538M    /lib
2.1G    /usr
3.7G    /data
7.7G    /

我删了一个3.3G的日志文件进去/data又跑du了

ubuntu@app1:~$ sudo du -h --max-depth=1 / | sort -h
0   /proc
0   /sys
4.0K    /lib64
4.0K    /media
4.0K    /mnt
4.0K    /srv
8.0K    /dev
8.0K    /snap
16K /lost+found
24K /root
800K    /tmp
6.4M    /etc
11M /run
14M /sbin
16M /bin
246M    /boot
331M    /home
352M    /data
397M    /opt
429M    /var
538M    /lib
2.1G    /usr
4.4G    /

ubuntu@app1:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
tmpfs           100M   11M   89M  11% /run
/dev/xvda1      7.7G  7.7G     0 100% /
tmpfs           496M  8.0K  496M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           496M     0  496M   0% /sys/fs/cgroup
tmpfs           100M     0  100M   0% /run/user/1001
tmpfs           100M     0  100M   0% /run/user/1000

尽管该/data目录现在减少到 352M，但df仍然显示 100% 的磁盘利用率。我错过了什么？

参考这个答案https://unix.stackexchange.com/a/253655/47050，这里是输出strace

ubuntu@app1:~$ strace -e statfs df /
statfs("/", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=2016361, f_bfree=4096, f_bavail=0, f_files=1024000, f_ffree=617995, f_fsid={2136106470, -680157247}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8065444 8049060         0 100% /
+++ exited with 0 +++

更新

我跑了

sudo lsof | grep deleted

并发现

node\x20/ 22318           deploy   12w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
node\x20/ 22318           deploy   13w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
node\x20/ 22318           deploy   14w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
node\x20/ 22318           deploy   15w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
node\x20/ 22318           deploy   16w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)

如何发布这些文件？

更新 2

ubuntu@app1:~$ sudo ls -l /proc/22318/fd
total 0
lrwx------ 1 deploy deploy 64 Apr  6 10:05 0 -> socket:[74749956]
lrwx------ 1 deploy deploy 64 Apr  6 10:05 1 -> socket:[74749958]
lr-x------ 1 deploy deploy 64 Apr  6 10:05 10 -> /dev/null
l-wx------ 1 deploy deploy 64 Apr  6 10:05 12 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 13 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 14 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 15 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 16 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 17 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 18 -> /data/app/shared/logs/production.log (deleted)
l-wx------ 1 deploy deploy 64 Apr  6 10:05 19 -> /data/app/shared/logs/production.log (deleted)
lrwx------ 1 deploy deploy 64 Apr  6 10:05 2 -> socket:[74749960]
l-wx------ 1 deploy deploy 64 Apr  6 10:05 20 -> /data/app/shared/logs/production.log (deleted)
lrwx------ 1 deploy deploy 64 Apr  6 10:05 21 -> socket:[74750302]
lrwx------ 1 deploy deploy 64 Apr  6 10:05 22 -> socket:[74750303]
lrwx------ 1 deploy deploy 64 Apr  6 10:05 3 -> socket:[74749962]
lrwx------ 1 deploy deploy 64 Apr  6 10:05 4 -> anon_inode:[eventpoll]
lr-x------ 1 deploy deploy 64 Apr  6 10:05 5 -> pipe:[74749978]
l-wx------ 1 deploy deploy 64 Apr  6 10:05 6 -> pipe:[74749978]
lr-x------ 1 deploy deploy 64 Apr  6 10:05 7 -> pipe:[74749979]
l-wx------ 1 deploy deploy 64 Apr  6 10:05 8 -> pipe:[74749979]
lrwx------ 1 deploy deploy 64 Apr  6 10:05 9 -> anon_inode:[eventfd]

ubuntu@app1:~$ ps aux | grep node
deploy   22318  0.0 12.7 1277192 129844 ?      Ssl   2019 173:38 node /data/app/releases/20180904094535/app.js
ubuntu   30665  0.0  0.0  12944   972 pts/0    S+   10:09   0:00 grep --color=auto node

标签： unixamazon-ec2disk

解决方案

这些文件由节点应用程序保留。确定使用：

sudo lsof | grep deleted

重新启动节点应用程序解决了我的问题。

查找节点进程 ID ps aux | grep node。然后使用 . 杀死节点服务器kill -9 <process_id>。最后重启节点服务器。在我的情况下pm2自动重新启动节点。

推荐阅读