2. Logstash实战
最后更新于：2022-04-02 07:41:55
#### 请先看看结尾的坑
### logstash注意事项
- ##### logstash可以启动多个端口接收数据
- ##### 重启logstash可能会卡住，一般由于输出到其他服务导致（Reids服务没启动，验证没通过），一般会选择kill
- ##### 执行前先检查语法，并前台测试启动
#### 检查logstash语法
```shell
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf -t
# logstash前台启动
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf
```
测试阶段可以将收集的日志输出到本地文件中，调试后发往目标
### 收集日志，存储在本地文件
```shell
#cat /etc/logstash/conf.d/system.conf
input{
file {
type => "systemlog"
path => "/var/log/messages"
start_position => "beginning"
stat_interval => "5"
}
}
output {
file {
path => "/tmp/systemlog.log"
}
}
```
### 使用logstash收集系统messages日志
文件无权限报错
```shell
[2017-08-29T16:50:00,834][INFO ][logstash.pipeline ] Pipeline main started
[2017-08-29T16:50:00,852][WARN ][logstash.inputs.file ] failed to open /var/log/messages: Permission denied - /var/log/messages
[2017-08-29T16:50:00,886][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
```
```shell
#系统日志，默认600,会报错
chmod 644 /var/log/messages
```
#### logstash中增加配置（/etc/logstash/conf.d/systemlog.conf）
```shell
#cat /etc/logstash/conf.d/system.conf
input{
file {
type => "systemlog"
path => "/var/log/messages"
start_position => "beginning"
stat_interval => "5"
}
}
output{
elasticsearch{
hosts => ["192.168.0.231:9200"]
index => "logstash-systemlog-%{+YYYY.MM.dd}"
}
}
```
### logstash收集Nginx日志
#### Nginx配置成Json格式日志
配置摘自[jack.zhang的博客](http://www.cnblogs.com/zhang-shijie/p/5384624.html "jack.zhang的博客")，感谢分享
```json
log_format logstash_json '{"@timestamp":"$time_local",'
'"remote_addr":"$remote_addr",'
'"remote_user":"$remote_user",'
'"body_bytes_sent":"$body_bytes_sent",'
'"request_time":"$request_time",'
'"status":"$status",'
'"request":"$request",'
'"request_method":"$request_method",'
'"http_referrer":"$http_referer",'
'"body_bytes_sent":"$body_bytes_sent",'
'"http_x_forwarded_for":"$http_x_forwarded_for",'
'"http_user_agent":"$http_user_agent"}';
access_log /var/log/nginx/access.log logstash_json;
```
json日志
```json
{
"@timestamp": "30/Aug/2017:10:18:59 +0800",
"remote_addr": "192.168.2.64",
"remote_user": "-",
"body_bytes_sent": "0",
"request_time": "0.000",
"status": "304",
"request": "GET /nginxweb/ HTTP/1.1",
"request_method": "GET",
"http_referrer": "-",
"http_x_forwarded_for": "-",
"http_user_agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36"
}
```
logstash中增加配置（/etc/logstash/conf.d/nginx.conf）
```json
input {
file {
path => "/var/log/nginx/access.log"
type => "nginx-accesslog"
start_position => "beginning"
}
}
output {
if [type] == "nginx-accesslog" {
elasticsearch {
hosts => ["192.168.0.231:9200"]
index => "nginx-accesslog-%{+YYYY.MM.dd}"
}
}
}
```
### logstash收集Tomcat访问日志
#### 配置Tomcat日志为Json格式
配置摘自[jack.zhang的博客](http://www.cnblogs.com/zhang-shijie/p/5384624.html "jack.zhang的博客")，感谢分享
#### 修改tomcat/conf/server.xml结尾部分的日志配置
修改日志名称，结尾格式，和pattern
```xml
';