时间戳

首页 > 解决方案 > 将 MariaDB 中的多个表同步到 Elasticsearch 7.5

elasticsearch - 将 MariaDB 中的多个表同步到 Elasticsearch 7.5

问题描述

我正在尝试使用 JDBC 插件将数据库同步到 Elasticsearch，但不知何故，此配置不适用于多个语句（多个表）。

input {
  jdbc {
    jdbc_driver_library => ""
    jdbc_driver_class => "Java::org.mariadb.jdbc.Driver"
    jdbc_connection_string => "jdbc:mariadb://localhost:3306/DB"
    jdbc_user => "username"
    jdbc_password => "password"
    jdbc_paging_enabled => true
    tracking_column => "column1"
    use_column_value => true
    type => "type1"
    tracking_column_type => "numeric"
    schedule => "*/5 * * * * *"
    statement => "SELECT *, UNIX_TIMESTAMP(date_column) AS unix_ts_in_secs FROM table1 WHERE (UNIX_TIMESTAMP(date_column) >= :sql_last_value) ORDER BY date_column ASC"
  }
  jdbc {
    jdbc_driver_library => ""
    jdbc_driver_class => "Java::org.mariadb.jdbc.Driver"
    jdbc_connection_string => "jdbc:mariadb://localhost:3306/DB"
    jdbc_user => "username"
    jdbc_password => "password"
    jdbc_paging_enabled => true
    tracking_column => "column1"
    use_column_value => true
    type => "type2"
    tracking_column_type => "numeric"
    schedule => "*/5 * * * * *"
    statement => "SELECT *, UNIX_TIMESTAMP(date_column) AS unix_ts_in_secs FROM table2 WHERE (UNIX_TIMESTAMP(date_column) >= :sql_last_value) ORDER BY date_column ASC"
  }
}
filter {
  mutate {
    copy => { "id" => "[@metadata][_id]"}
    remove_field => ["id", "@version", "unix_ts_in_secs"]
  }
}
 output {
  elasticsearch {
      hosts => ["localhost:9200"]
      index => "test"
      document_id => "%{[@metadata][_id]}"
  }
}

此配置对于单个语句运行得很好，但不适用于多个查询。我在这里错过了什么？我需要将此数据同步到 Elasticsearch 中的单个索引。

标签： elasticsearchmariadbkibanaelk

解决方案

通过创建唯一的 document_id 来解决它

推荐阅读