安装ELK(Elasticsearch7+Logstash7+Kibana7)并收集Nginx日志

DevOps 2020年05月31日

本篇笔记记录了在Linux的CentOS7发行版下使用RPM包安装Logstash7和Kibana7,并尝试使用Logstash7收集Nginx的访问日志输出到Elasticsearch7,使用Kibana7查看Nginx日志的过程。

工作准备

服务器
192.168.75.237已安装Elasticsearch7
192.168.75.249已安装nginx+php+mysql
系统
CentOS Linux release 7.7.1908 (Core)
selinux已关闭
firewalld已关闭
Elasticsearch 7.7
IP和端口号:192.168.75.237:9200
Nginx日志
192.168.75.249 /var/log/nginx/access.log

Elasticsearch环境搭建请参考:
PHP使用Elasticsearch实现基于IK分词搜索及多维度聚合查询实例
CentOS6.9安装ElasticSearch
Nginx+PHP+Mysql环境搭建请参考:
CentOS7源码编译安装nginx+php7.2+mysql5.7并使用systemctl管理
CentOS7yum安装nginx+php7+mysql

安装Logstash,在192.168.75.249服务器

下载并安装公共签名密钥

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

下载RPM包

sudo wget https://artifacts.elastic.co/downloads/logstash/logstash-7.7.0.rpm

安装

sudo rpm --install logstash-7.7.0.rpm

设置Logstash开机自启动

sudo systemctl daemon-reload
sudo systemctl enable logstash

测试

/usr/share/logstash/bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'

看到启动成功提示后,输入jmsite并回车

如果看到上图的json说明Logstash运行成功

收集Nginx访问日志

创建Logstash配置文件

vim /etc/logstash/conf.d/nginx_log.conf

输入以下配置

input {
    file {
        path => ["/var/log/nginx/access.log"]
        start_position => "beginning"
    }
 }

filter {
        grok {
                match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
        date {
                match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
        }
        ruby {
                code => "event.set('log_day', event.get('@timestamp').time.localtime.strftime('%Y%m%d'))"
        }
}

output {
        #elasticsearch {
        #        hosts => ["192.168.75.237:9200"]
        #        index => "nginx-access-%{log_day}"
        #}
        stdout { codec => rubydebug }
}

测试

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx_log.conf

看到启动成功提示后,浏览器请求Nginx

如果看到上图的json,说明Logstash已经成功收集和解析了Nginx日志

将Nginx日志输出到Elasticsearch

修改配置中output区块

output {
	elasticsearch {
		hosts => ["192.168.75.237:9200"]
		index => "nginx-access-%{log_day}"
	}
	#stdout { codec => rubydebug }
}

设置权限

chown -R logstash:logstash /etc/logstash
chmod -R 755 /etc/logstash

启动Logstash

systemctl start logstash

查看Logstash日志

tail -f /var/log/logstash/logstash-plain.log

如果不断的滚动出现以下错误

......
[2020-05-31T08:47:26,386][INFO ][filewatch.observingtail  ][main][a1d2e4989b4505d7e19eabe7ad28f7c5115e54c14c70a5b787a1fd14c85b2c06] START, creating Discoverer, Watch with file and sincedb collections
[2020-05-31T08:47:26,388][ERROR][logstash.javapipeline    ][main][a1d2e4989b4505d7e19eabe7ad28f7c5115e54c14c70a5b787a1fd14c85b2c06] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::File start_position=>"beginning", path=>["/var/log/nginx/access.log"], id=>"a1d2e4989b4505d7e19eabe7ad28f7c5115e54c14c70a5b787a1fd14c85b2c06", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_eeada0fd-49c6-45d0-bfb4-a2690517879a", enable_metric=>true, charset=>"UTF-8">, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc", exit_after_read=>false>
  Error: Permission denied - Permission denied
  Exception: Errno::EACCES
  Stack: org/jruby/RubyFile.java:1267:in `utime'
......

我们看到sincedbError: Permission denied - Permission denied Exception: Errno::EACCES等关键词
问题说明:
这是以root身份启动Logstash并收集Nginx日志时,sincedb文件是以root身份创建的
systemctl start logstash是以logstash用户运行,没有向root生成的sincedb文件写的权限导致。
sincedb文件是干什么的?请参考官方文档:
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-sincedb_path
解决办法:
官方文档说sincedb文件默认存放在<path.data>/plugins/inputs/file目录里,那么path.data指向的哪里呢?

cat /etc/logstash/logstash.yml | grep "path.data"
path.data: /var/lib/logstash

这个鬼在这

[root@localhost ~]# ll -a /var/lib/logstash/plugins/inputs/file/
总用量 4
drw-r--r-- 1 root root  55 5月  31 07:21 .
drw-r--r-- 1 root root  18 5月  30 20:10 ..
-rw-r--r-- 1 root root 172 5月  31 07:21 .sincedb_d883144359d3b4f516b37dba51fab2a2

在确认问题之后,设置权限

chmod -R 777 /var/lib/logstash

查看Elasticsearch中是否已生成索引

[root@localhost logstash]# curl http://192.168.75.237:9200/_cat/indices
yellow open article               pkEmM36VT9KHjVGYxfqXJA 3 2 65 2  25.9kb  25.9kb
yellow open nginx-access-20200531 --64DQD4RrSgYyO-UDGMHw 1 1 88 0 182.4kb 182.4kb

nginx-access-20200531这个索引就是我们刚才生成的

安装Kibana,在192.168.75.237服务器

下载并安装公共签名密钥

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

下载RPM包

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.7.0-x86_64.rpm

安装

sudo rpm --install kibana-7.7.0-x86_64.rpm

修改配置文件

vim /etc/kibana/kibana.yml

配置如下

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.75.237:9200"]
i18n.locale: "zh-CN"

保存,启动Kibana,启动过程很慢,不要着急

/usr/share/kibana/bin/kibana --allow-root

--allow-rootKibana不能以root身份启动,否则会失败
浏览器访问Kibana
http://192.168.75.237:5601

点击左下角红框中按钮,展开菜单
点击"management"
点击Kibana区域的"索引模式"

创建索引模式

输入nginx-access-*下一步

下拉选择@timestamp,创建索引模式
创建成功,再次展开左侧菜单,点击"Discover"

如果看到以上界面,说明我们成功了!