时间戳

首页 > 解决方案 > 无法将数据从mysql同步到logstash

elasticsearch - 无法将数据从mysql同步到logstash

问题描述

我的 logstash.conf 文件是

input {
  jdbc { 
    jdbc_connection_string => "jdbc:mysql://localhost:8889/optjobs"
    # The user we wish to execute our statement as
    jdbc_user => "root"
    jdbc_password => "root"
    # The path to our downloaded jdbc driver
    jdbc_driver_library => "/Users/ajoshi31/mysql-connector-java-5.1.17.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    # our query
    statement => "SELECT * FROM candidates INNER JOIN candidate_skills ON candidate_skills.candidate_id = candidates.id"
    }
  }
output {
  stdout { codec => json_lines }
  elasticsearch {
  "hosts" => "localhost:9200"
  "index" => "optjobsprd"
  "document_type" => "data"
  }
}

使用配置文件并在运行 logstash

$ logstash -f logstash.conf

我收到以下错误

  Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/local/Cellar/logstash/7.5.2/libexec/logs which is now configured via log4j2.properties
[2020-04-09T11:47:14,307][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-09T11:47:14,549][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.5.2"}
[2020-04-09T11:47:17,657][INFO ][org.reflections.Reflections] Reflections took 122 ms to scan 1 urls, producing 20 keys and 40 values 
[2020-04-09T11:47:18,565][ERROR][logstash.outputs.elasticsearch] Unknown setting '"document_type"' for elasticsearch
[2020-04-09T11:47:18,568][ERROR][logstash.outputs.elasticsearch] Unknown setting '"hosts"' for elasticsearch
[2020-04-09T11:47:18,572][ERROR][logstash.outputs.elasticsearch] Unknown setting '"index"' for elasticsearch
[2020-04-09T11:47:18,590][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:mai

一旦我删除了主机、document_type 和主机，logstash 就会运行，连接到 mysql 并执行查询，但我无法从 mysql 创建索引或更新数据。

标签： elasticsearchlogstash

解决方案

不得引用选项。删除引号，例如

output {
  stdout { codec => json_lines }

  elasticsearch {
    hosts => "localhost:9200"
    index => "optjobsprd"
    document_type => "data"
  }
}

推荐阅读