ELK实践——Logstash收集Nginx的access.log和error.log

本篇笔记记录了使用Logstash7收集并解析Nginx的访问日志和错误日志，输出到Elasticsearch，使用Kibana展示的过程

工作准备

ELK环境搭建在上篇笔记中已经做了详细记录：CentOS7下ELK7环境搭建入门篇

日志格式

基于Nginx默认的日志格式

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    error_log  /var/log/nginx/error.log warn;

日志样例

access.log

[root@localhost ~]# tail -f /var/log/nginx/access.log 
192.168.75.1 - - [03/Jun/2020:14:55:56 +0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0" "-"
192.168.75.1 - - [03/Jun/2020:14:55:56 +0800] "GET /favicon.ico HTTP/1.1" 404 168 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0" "-"

error.log

[root@localhost ~]# tail -f /var/log/nginx/error.log 
2020/06/03 14:55:56 [error] 19466#0: *12 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.75.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "192.168.75.239"
2020/06/03 15:03:26 [emerg] 23649#0: unknown log format "maini" in /etc/nginx/nginx.conf:22

创建Logstash配置

vim /etc/logstash/conf.d/nginx_log.conf

配置如下

input {
    file {
        path => ["/var/log/nginx/access.log"]
        start_position => "beginning"
		type => "nginx-access"
    }
	file {
        path => ["/var/log/nginx/error.log"]
        start_position => "beginning"
		type => "nginx-error"
    }
 }

filter {
	if [type] == "nginx-access"{
		grok {
			match => ["message","%{IPORHOST:remote_addr} - %{HTTPDUSER:remote_user} \[%{HTTPDATE:time_local}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} (?:%{NUMBER:body_bytes}|-) %{QS:referrer} %{QS:user_agent} %{QS:x_forward_for}"]
		}
	}
	if [type] == "nginx-error"{
		grok {
			match => [
			    "message", "(?<time_local>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:log_level}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:error_message}(?:, client: (?<client>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server}?)(?:, request: %{QS:request})?(?:, upstream: (?<upstream>\"%{URI}\"|%{QS}))?(?:, host: %{QS:request_host})?(?:, referrer: \"%{URI:referrer}\")?",
				"message", "(?<time_local>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:log_level}\]\s{1,}%{GREEDYDATA:error_message}"
			]
		}
	}
	date {
		match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
	}
    mutate {
        convert => [ "status", "integer" ]
        convert => [ "body_bytes","integer" ]
    }
	ruby {
		code => "event.set('log_day', event.get('@timestamp').time.localtime.strftime('%Y%m%d'))"
	}
}

output {
	if [type] == "nginx-access"{
        #elasticsearch {
        #        hosts => ["192.168.75.238:9200"]
        #        index => "nginx-log-access-%{log_day}"
        #}
	}
	if [type] == "nginx-error"{
        #elasticsearch {
        #        hosts => ["192.168.75.238:9200"]
        #        index => "nginx-log-error-%{log_day}"
        #}
	}
    stdout { codec => rubydebug }
}

检查配置

/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/nginx_log.conf

测试收集和解析日志

如果Logstash当前正在运行，先停掉

systemctl stop logstash

指定配置文件执行，测试终端输出结果

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx_log.conf

生产错误日志：把Nginx配置文件改错，重启Nginx

改回正确配置，启动Nginx
生产错误日志：浏览器访问Nginx，访问一个不存在的页面

生产访问日志：浏览器访问Nginx，访问一个存在的页面

输出到Elasticsearch

CTRL+C - 结束Logstash当前运行
刚才在命令行是以root身份运行的，所以更改Logstash运行时目录权限

chown -R logstash:logstash /var/lib/logstash
chmod -R 755 /var/lib/logstash

设置Nginx日志logstash读取权限

chmod -R 755 /var/log/nginx/error.log
chmod -R 755 /var/log/nginx/access.log

更改output区块配置

output {
	if [type] == "nginx-access"{
        elasticsearch {
                hosts => ["192.168.75.238:9200"]
                index => "nginx-log-access-%{log_day}"
        }
	}
	if [type] == "nginx-error"{
        elasticsearch {
                hosts => ["192.168.75.238:9200"]
                index => "nginx-log-error-%{log_day}"
        }
	}
    #stdout { codec => rubydebug }
}

systemd启动Logstash并查看运行状态

systemctl start logstash
systemctl status logstash

重复上面的测试步骤

刚才上面终端输出测试时，已经把日志解析的偏移量存储到了sincedb文件中，所以需要生产新的日志向Elasticsearch输出

• 生产错误日志：把Nginx配置文件改错，重启Nginx
• 改回正确配置，启动Nginx
• 生产错误日志：浏览器访问Nginx，访问一个不存在的页面
• 生产访问日志：浏览器访问Nginx，访问一个存在的页面

Kibana展示索引的日志数据

浏览器访问Kibana，创建索引模式

输入nginx-log-*,Kibana自动匹配出了Logstash输出到Elasticsearch的两个索引
点击下一步，时间字段选择@timestamp，点击创建
展开左侧菜单，点击“Discover”，筛选我们刚才创建的索引

我们看到了Nginx的访问和错误日志信息！