本篇笔记记录了使用Logstash7收集并解析Nginx的访问日志和错误日志,输出到Elasticsearch,使用Kibana展示的过程
工作准备
ELK环境搭建在上篇笔记中已经做了详细记录:CentOS7下ELK7环境搭建入门篇
日志格式
基于Nginx默认的日志格式
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log warn;
日志样例
access.log
[root@localhost ~]# tail -f /var/log/nginx/access.log
192.168.75.1 - - [03/Jun/2020:14:55:56 +0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0" "-"
192.168.75.1 - - [03/Jun/2020:14:55:56 +0800] "GET /favicon.ico HTTP/1.1" 404 168 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0" "-"
error.log
[root@localhost ~]# tail -f /var/log/nginx/error.log
2020/06/03 14:55:56 [error] 19466#0: *12 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.75.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "192.168.75.239"
2020/06/03 15:03:26 [emerg] 23649#0: unknown log format "maini" in /etc/nginx/nginx.conf:22
创建Logstash配置
vim /etc/logstash/conf.d/nginx_log.conf
配置如下
input {
file {
path => ["/var/log/nginx/access.log"]
start_position => "beginning"
type => "nginx-access"
}
file {
path => ["/var/log/nginx/error.log"]
start_position => "beginning"
type => "nginx-error"
}
}
filter {
if [type] == "nginx-access"{
grok {
match => ["message","%{IPORHOST:remote_addr} - %{HTTPDUSER:remote_user} \[%{HTTPDATE:time_local}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} (?:%{NUMBER:body_bytes}|-) %{QS:referrer} %{QS:user_agent} %{QS:x_forward_for}"]
}
}
if [type] == "nginx-error"{
grok {
match => [
"message", "(?<time_local>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:log_level}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:error_message}(?:, client: (?<client>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server}?)(?:, request: %{QS:request})?(?:, upstream: (?<upstream>\"%{URI}\"|%{QS}))?(?:, host: %{QS:request_host})?(?:, referrer: \"%{URI:referrer}\")?",
"message", "(?<time_local>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:log_level}\]\s{1,}%{GREEDYDATA:error_message}"
]
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
convert => [ "status", "integer" ]
convert => [ "body_bytes","integer" ]
}
ruby {
code => "event.set('log_day', event.get('@timestamp').time.localtime.strftime('%Y%m%d'))"
}
}
output {
if [type] == "nginx-access"{
#elasticsearch {
# hosts => ["192.168.75.238:9200"]
# index => "nginx-log-access-%{log_day}"
#}
}
if [type] == "nginx-error"{
#elasticsearch {
# hosts => ["192.168.75.238:9200"]
# index => "nginx-log-error-%{log_day}"
#}
}
stdout { codec => rubydebug }
}
检查配置
/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/nginx_log.conf
测试收集和解析日志
如果Logstash当前正在运行,先停掉
systemctl stop logstash
指定配置文件执行,测试终端输出结果
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx_log.conf
生产错误日志:把Nginx配置文件改错,重启Nginx
改回正确配置,启动Nginx
生产错误日志:浏览器访问Nginx,访问一个不存在的页面
生产访问日志:浏览器访问Nginx,访问一个存在的页面
输出到Elasticsearch
CTRL+C
- 结束Logstash当前运行
刚才在命令行是以root身份运行的,所以更改Logstash运行时目录权限
chown -R logstash:logstash /var/lib/logstash
chmod -R 755 /var/lib/logstash
设置Nginx日志logstash读取权限
chmod -R 755 /var/log/nginx/error.log
chmod -R 755 /var/log/nginx/access.log
更改output区块配置
output {
if [type] == "nginx-access"{
elasticsearch {
hosts => ["192.168.75.238:9200"]
index => "nginx-log-access-%{log_day}"
}
}
if [type] == "nginx-error"{
elasticsearch {
hosts => ["192.168.75.238:9200"]
index => "nginx-log-error-%{log_day}"
}
}
#stdout { codec => rubydebug }
}
systemd启动Logstash并查看运行状态
systemctl start logstash
systemctl status logstash
重复上面的测试步骤
刚才上面终端输出测试时,已经把日志解析的偏移量存储到了sincedb文件中,所以需要生产新的日志向Elasticsearch输出
- 生产错误日志:把Nginx配置文件改错,重启Nginx
- 改回正确配置,启动Nginx
- 生产错误日志:浏览器访问Nginx,访问一个不存在的页面
- 生产访问日志:浏览器访问Nginx,访问一个存在的页面
Kibana展示索引的日志数据
浏览器访问Kibana,创建索引模式
输入nginx-log-*
,Kibana自动匹配出了Logstash输出到Elasticsearch的两个索引
点击下一步,时间字段选择@timestamp,点击创建
展开左侧菜单,点击“Discover”,筛选我们刚才创建的索引
我们看到了Nginx的访问和错误日志信息!