时间戳

首页 > 技术文章 > Ubuntu环境部署Logstash实战案例

Ubuntu环境部署Logstash实战案例

yinzhengjie2020 2020-05-31 23:28 原文

　　　　　　　　　　　　Ubuntu环境部署Logstash实战案例

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

 

 

 

一.准备环境

1>.部署环境说明

　　Logstash可以单独找一台机器部署，它需要安装JDK环境，我这里为了省事，就直接和一台Elasticsearch节点复用同一个节点(es103.yinzhengjie.com)。

　　博主推荐阅读:
　　　　https://www.cnblogs.com/yinzhengjie2020/p/12953504.html

2>.下载kibana软件包

　　博主推荐阅读:
　　　　https://www.cnblogs.com/yinzhengjie2020/p/12934518.html

 

二.部署Logstash实操

1>.将下载的Logstash软件包上传到es103.yinzhengjie.com节点并安装

[root@es103.yinzhengjie.com ~]# dpkg -i logstash-6.8.9.deb 
Selecting previously unselected package logstash.
(Reading database ... 103047 files and directories currently installed.)
Preparing to unpack logstash-6.8.9.deb ...
Unpacking logstash (1:6.8.9-1) ...
Setting up logstash (1:6.8.9-1) ...
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash
[root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# dpkg -i logstash-6.8.9.deb

2>.查看logstath的启动脚本

[root@es103.yinzhengjie.com ~]# find / -name logstash.service
/etc/systemd/system/logstash.service
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
[root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# cat /etc/systemd/system/logstash.service
[root@es102.yinzhengjie.com ~]# vim /etc/systemd/system/logstash.service
[root@es102.yinzhengjie.com ~]# 
[root@es102.yinzhengjie.com ~]# cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=root
Group=root
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
[root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# vim /etc/systemd/system/logstash.service　　　　　　　　#配置将logstash服务以root用户身份启动

 

三.以标准输入为数据源测试Logstash可用性

1>.输出类型为标准输出案例

[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug }}'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-04 03:01:56.765 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-04 03:01:56.776 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"}
[INFO ] 2020-06-04 03:02:01.365 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc
h.delay"=>50}[INFO ] 2020-06-04 03:02:01.473 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x19207ef3 run>"}
The stdin plugin is now waiting for input:
[INFO ] 2020-06-04 03:02:01.540 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin
es=>[]}[INFO ] 2020-06-04 03:02:01.778 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
尹正杰到此一游!　　　　　　　　　　　　　　　　　　　　　　　　　　#这是我输出的数据
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
      "@version" => "1",　　　　　　　　　　　　　　　　　　　　#事件版本号，一个事件就是一个ruby对象
    "@timestamp" => 2020-06-04T03:02:40.589Z,　　　　　　　 #当前事件的发生时间
       "message" => "尹正杰到此一游!",　　　　　　　　　　　　 #消息的具体内容
          "host" => "es103.yinzhengjie.com"　　　　　　　　#标记时间发生在哪个主机
}
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug }}'　　　　　　　　

2>.输出类型为文件案例

[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { file {path => "/tmp/log.txt"}}'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-04 03:32:00.760 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-04 03:32:00.771 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"}
[INFO ] 2020-06-04 03:32:05.126 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc
h.delay"=>50}[INFO ] 2020-06-04 03:32:05.258 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4b0b27c1 run>"}
The stdin plugin is now waiting for input:
[INFO ] 2020-06-04 03:32:05.319 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin
es=>[]}[INFO ] 2020-06-04 03:32:05.522 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
尹正杰到此一游!
[INFO ] 2020-06-04 03:33:48.228 [[main]>worker1] file - Opening file {:path=>"/tmp/log.txt"}
https://www.cnblogs.com/yinzhengjie/
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { file {path => "/tmp/log.txt"}}'
[root@es103.yinzhengjie.com ~]# tail -10f /tmp/log.txt 
{"@timestamp":"2020-06-04T03:33:47.887Z","host":"es103.yinzhengjie.com","message":"尹正杰到此一游!","@version":"1"}
{"@timestamp":"2020-06-04T03:34:28.523Z","host":"es103.yinzhengjie.com","message":"https://www.cnblogs.com/yinzhengjie/","@version":"1"}
[root@es103.yinzhengjie.com ~]# tail -10f /tmp/log.txt

3>.输出类型为Elasticsearch案例

[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch {hosts=> ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:920
0"] index=> "yinzhengjie-log-%{+YYYY.MM.dd}"}}'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-06-04 03:43:44.554 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-06-04 03:43:44.567 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"}
[INFO ] 2020-06-04 03:43:48.816 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc
h.delay"=>50}[INFO ] 2020-06-04 03:43:49.205 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102
.yinzhengjie.com:9200/]}}[WARN ] 2020-06-04 03:43:49.422 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"}
[INFO ] 2020-06-04 03:43:49.561 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2020-06-04 03:43:49.563 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version
=>6}[WARN ] 2020-06-04 03:43:49.575 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"}
[INFO ] 2020-06-04 03:43:49.626 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200",
 "http://es102.yinzhengjie.com:9200"]}[INFO ] 2020-06-04 03:43:49.651 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template
[INFO ] 2020-06-04 03:43:49.678 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refr
esh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[INFO ] 2020-06-04 03:43:49.720 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x696425bf run>"}
[INFO ] 2020-06-04 03:43:49.774 [Ruby-0-Thread-5: :1] elasticsearch - Installing elasticsearch template to _template/logstash
The stdin plugin is now waiting for input:
[INFO ] 2020-06-04 03:43:49.807 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin
es=>[]}[INFO ] 2020-06-04 03:43:50.234 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
尹正杰到此一游~
https://www.cnblogs.com/yinzhengjie/
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch {hosts=> ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:920 0"] index=> "yinzhengjie-log-%{+YYYY.MM.dd}"}}'
　　数据是否写入成功我们在Kibana无法直接查看到，我们可以通过Elasticsearch的head插件看到。但这并不说Kibana无法查看刚刚通过Logstash写入的数据，而是需要创建索引。

 

四.Kibana创建索引

1>.打开Kibana的WebUI，如下图所示，依次点击"管理" ---> "索引模式"

2>.如下图所示，点击"创建索引模式"

3>.如下图所示，输入匹配模式后，点击"下一步"

4>.如下图所示，设置筛选时间的名称为"timestamp"即可

5>.索引创建成功

6>.点击Discover，查看刚刚创建的索引内容

 

五.博主推荐阅读

 

推荐阅读