时间戳

首页 > 解决方案 > 如何在 fluentd tail/s3 插件中删除 unicode

amazon-web-services - 如何在 fluentd tail/s3 插件中删除 unicode

问题描述

我有 fluentd 配置，源为尾类型，目标为 aws s3。我可以将日志存储在 S3 中。

我们已经根据winston logger中的日志级别在应用程序日志中启用了着色，但是在存储在S3中时，我得到了颜色的unicode值，例如\u001b[34m debug \u001b[39m。特殊字符也会发生同样的情况（\u003c 表示 >）

Fluentd Config
--------------
    <source>
      @type tail
      path /var/log/containers/abc-*.log
      pos_file /var/log/abc.log.pos
      tag abc.**
      <parse>
        @type none
      </parse>
      read_from_head true
    </source>
    
    <match abc.**>
       @type s3
    
       aws_key_id "#{ENV['AWS_ACCESS_KEY']}"
       aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}"
       s3_bucket "#{ENV['S3_LOGS_BUCKET_NAME']}"
       s3_region "#{ENV['S3_LOGS_BUCKET_REGION']}"
       path "#{ENV['S3_LOGS_BUCKET_PREFIX']}"
       s3_object_key_format %{path}/abc/%Y%m%d/%{index}.json
    
       buffer_chunk_limit 20m
       buffer_path /var/log/fluentd-buffer
       store_as json
       flush_interval 600s
       time_slice_format %Y/%m/%d
       utc
    
       <format>
          @type single_value
       </format>
    
       <instance_profile_credentials>
         ip_address 169.254.169.254
         port       80
       </instance_profile_credentials>
    </match>

当前存储在 S3 中的日志：

{"log":"2021-04-10T12:34:51.050Z - \u001b[34mdebug\u001b[39m: \u003e\u003e\u003e\u003e testlog1 from app \n","stream":"stdout","time":"2021-04-10T12:34:51.050571552Z"}
{"log":"2021-04-10T12:34:51.067Z - \u001b[34mdebug\u001b[39m: \u003c\u003c\u003c\u003c testlog2 from app\n","stream":"stdout","time":"2021-04-10T12:34:51.068105637Z"}

预期的

{"log":"2021-04-10T12:34:51.050Z - debug: <<<< exec start from app \n","stream":"stdout","time":"2021-04-10T12:34:51.050571552Z"}
{"log":"2021-04-10T12:34:51.067Z - debug: <<<< exec end from app\n","stream":"stdout","time":"2021-04-10T12:34:51.068105637Z"}

需要有关如何打印原始值的帮助。

标签： amazon-web-servicesamazon-s3fluentfluentdwinston

解决方案

像这样尝试 fluentd record_transformer过滤器插件：

<filter abc.**>
  @type record_transformer
  enable_ruby true
  <record>
    message ${ record["message"].gsub(/(\\u\d+b\[\d+m)|(\\u\d+e)/, '') }
  </record>
</filter>

推荐阅读