时间戳

首页 > 解决方案 > Logstash JDBC 适配器：Varbinary 到 UTF-8？（mysql到弹性导入）

jdbc - Logstash JDBC 适配器：Varbinary 到 UTF-8？（mysql到弹性导入）

问题描述

我正在尝试通过logstash将mysql表导入elasticsearch。一列属于“varbinary”类型，导致以下错误：

[2018-10-10T12:35:54,922][ERROR][logstash.outputs.elasticsearch] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"\"\\xC3\" from ASCII-8BIT to UTF-8", :error_class=>"LogStash::Json::GeneratorError", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:27:in `jruby_dump'", "/usr/share/logstash/vendor/$

我的logstash配置：

input {
  jdbc { 
    jdbc_connection_string => "jdbc:mysql://localhost:3306/xyz"
    # The user we wish to execute our statement as
    jdbc_user => "test"
    jdbc_password => "test"
    # The path to our downloaded jdbc driver
    jdbc_driver_library => "/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    # our query
    statement => "SELECT * FROM x"
    }
  }
output {
  stdout { codec => json_lines }
  elasticsearch {
  "hosts" => "localhost:9200"
  "index" => "x"
  "document_type" => "data"
  }
}

如何将 varbinary 转换为 uft-8？我必须使用特殊的过滤器吗？

标签： jdbclogstashlogstash-jdbc

解决方案

尝试在连接字符串中使用可选（characterEncoding=utf8）

jdbc_connection_string => "jdbc:mysql://localhost:3306/xyz?useSSL=false&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&autoReconnect=true"

推荐阅读