时间戳

首页 > 解决方案 > 制作一个从日志文件中读取并删除重复消息的 bash 脚本

linux - 制作一个从日志文件中读取并删除重复消息的 bash 脚本

问题描述

让我这样说：我有一台服务器，它每天都会为特定路径创建新的日志文件，例如 /var/log/apache/file[date].log

我想创建一个 bash 脚本来读取这个文件（根据日期），然后将其消息分组，删除重复出现的消息。

所以我是这样开始的：

grep "error" file_*.log # this * is the changing date each day

问题是该文件包含每次都会重复出现的错误，例如

[datetime] PHP 消息：PHP 注意：在file

我怎样才能以一种不会出现重复错误（尽管它们有不同的时间和/或 IP）的方式做到这一点，但不会出现不同的路径。

UDPATE

例如，让我们有以下日志文​​件

[Tue May 14 16:05:33.942372 2019] [proxy_fcgi:error] [client 127.0.0.1:50427] AH01071: Got error 'PHP message: PHP Warning:  require(PHPCLASS.php): failed to open stream: No such file or directory in path/file.php on line 6\nPHP message: PHP Fatal error:  require(): Failed opening required 'PHPCLASS.php' (include_path='.:path/CLASS') in path/file.php on line 6\n', referer: anotherone.com/
[Tue May 14 16:05:50.176982 2019] [proxy_fcgi:error] [client 127.0.0.1:52448] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//example.com/file.php on line 530\n'
[Tue May 14 16:05:53.890024 2019] [auth_basic:error] [client 127.0.0.1:52862] error: client used wrong authentication scheme: /
[Tue May 14 16:06:09.702323 2019] [proxy_fcgi:error]  [client 127.0.0.2:38880] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//example.com/file.php on line 530\n'
[Tue May 14 16:06:11.563076 2019] [proxy_fcgi:error] [client 127.0.0.3:52994] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//example.com/file.php on line 530\n'
[Tue May 14 16:07:10.237942 2019] [proxy_fcgi:error]  [client 127.0.0.2:38884] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//example.com/file.php on line 530\n'
[Tue May 14 16:07:10.694641 2019] [proxy_fcgi:error] [client 127.0.0.3:53154] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//example.com/file.php on line 530\n'
[Tue May 14 16:08:09.984855 2019] [proxy_fcgi:error]  [client 127.0.0.3:53318] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//example.com/file.php on line 530\n'
[Tue May 14 16:08:10.610644 2019] [proxy_fcgi:error]  [client 127.0.0.2:38902] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//example.com/file.php on line 530\n'
[Tue May 14 16:09:10.158302 2019] [proxy_fcgi:error] [client 127.0.0.2:38906] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//example.com/file.php on line 530\n'
[Tue May 14 16:09:10.554661 2019] [proxy_fcgi:error]  [client 127.0.0.3:53478] AH01071: Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//exampletwo.com/file.php on line 230\n'
[Tue May 14 16:10:33.942372 2019] [proxy_fcgi:error]  [client 127.0.0.6:50422] AH01071: Got error 'PHP message: PHP Warning:  require(PHPCLASS.php): failed to open stream: No such file or directory in path/file.php on line 6\nPHP message: PHP Fatal error:  require(): Failed opening required 'PHPCLASS.php' (include_path='.:path/path/CLASS') in path/path/file.php on line 6\n', referer: anotherone.com/

我希望结果是这样的

- Got error 'PHP message: PHP Warning:  require(PHPCLASS.php): failed to open stream: No such file or directory in path/file.php on line 6\nPHP message: PHP Fatal error:  require(): Failed opening required 'PHPCLASS.php' (include_path='.:path/CLASS') in path/file.php on line 6\n', referer: anotherone.com/
- Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//example.com/file.php on line 530\n'
- error: client used wrong authentication scheme: /
- Got error 'PHP message: PHP Notice:  A non well formed numeric value encountered in path//exampletwo.com/file.php on line 230\n'

忽略来自不同时间/ips但不来自不同文件的所有重复错误

标签： linuxbash

解决方案

要对一个文件执行此操作，您可以执行awk -F '[][]' '!a[$7]++' input. 要重置并获取每个文件的 uniq 消息，您可以执行以下操作：

awk -F '[][]' 'FNR==1{delete a} !a[$7]++' file_*.log

推荐阅读