wxin/es
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

更新 Logstash.md

Browse Source
This commit is contained in:
wxin 2025-05-04 22:47:32 +08:00
parent b911c61aba
commit 118629dc1e
1 changed files with 301 additions and 301 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

602
Logstash.md
View File

@ -1,301 +1,301 @@
<h2><center>Logstash</center></h2>
------
## 一：基本概念
### 1. Logstash 简介
​ Logstash 是 Elastic Stack 的中央数据流引擎，用于收集、丰富和统一所有数据，而不管格式或模式。
​ 相当与Elasticsearch，Kibana，及 Beats 共同使用的时候便会拥有特别强大的实时处理能力。
### 2. Logstash 原理
​ Logstash 管道中的每个输入阶段都在其自己的线程中运行。输入将事件写入位于内存（默认）或磁盘上的中央队列。每个管道工作线程从此队列中取出一批事件，通过配置的过滤器运行这批事件，然后通过任何输出运行过滤后的事件。批次的大小和管道工作线程的数量是可配置的。
​ 默认情况下，Logstash 使用管道阶段（输入 → 过滤器和过滤器 → 输出）之间的内存有界队列来缓冲事件。如果 Logstash 不安全地终止，则存储在内存中的所有事件都将丢失。为了防止数据丢失，您可以启用 Logstash 将传输中的事件保存到磁盘。
## 二：安装部署
### 1. 下载安装包
官网：[Past Releases of Elastic Stack Software | Elastic](https://www.elastic.co/downloads/past-releases#logstash)
```bash
[root@logstash ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-8.13.4-linux-x86_64.tar.gz
```
### 2. 解压安装
```bash
[root@logstash ~]# tar xf logstash-8.13.4-linux-x86_64.tar.gz -C /usr/local/
[root@logstash ~]# mv /usr/local/logstash-8.13.4/ /usr/local/logstash
```
### 3. 配置环境
```bash
[root@logstash ~]# vim /etc/profile
JAVA_HOME=/usr/local/logstash/jdk
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME PATH
[root@logstash ~]# source /etc/profile
[root@logstash ~]# java -version
openjdk version "17.0.11" 2024-04-16
OpenJDK Runtime Environment Temurin-17.0.11+9 (build 17.0.11+9)
OpenJDK 64-Bit Server VM Temurin-17.0.11+9 (build 17.0.11+9, mixed mode, sharing)
```
### 4. 数据流向
![](accents\image-202505020031.png)
### 5. 模拟运行
**输入和输出都来自于终端**
```bash
[root@logstash ~]# /usr/local/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
nihao (INPUT)
{
"@version" => "1",
"message" => "nihao (INPUT)",
"@timestamp" => 2025-05-04T08:54:48.278302130Z,
"event" => {
"original" => "nihao (INPUT)"
},
"host" => {
"hostname" => "logstash"
}
}
```
**输入来自终端，输出到ES集群**
Elasticsearch 从8.0开始， 默认开启了SSL安全验证
因此我们需要为 Logstash 配置身份验证凭据才能建立与 Elasticsearch 集群的通信
Logstash创建目录存储Elasticsearch的CA证书
```bash
[root@logstash ~]# mkdir /usr/local/logstash/config/certs
```
Elasticsearch中将证书拷贝到Logstash
```bash
[root@logstash ~]# scp 192.168.159.131:/usr/local/elasticsearch/config/certs/elasticsearch-ca.pem /usr/local/logstash/config/certs/
```
创建Logstash采集数据配置文件
```bash
[root@logstash ~]# vim /opt/stdin.conf
input {
stdin{}
}
output {
elasticsearch {
index => "stdin_logstash"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
```
执行Logstash
```bash
[root@logstash ~]# /usr/local/logstash/bin/logstash -f /opt/stdin.conf
hello world
hello wxin
nice to meet you
```
ES插件查看索引信息
![](accents\image-202505020026.png)
**输入来自日志文件，输出到ES集群**
此处只展示Logstash配置文件
```bash
[root@logstash ~]# vim /opt/nginx_access_logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
output {
elasticsearch {
index => "nginx_access_logstash"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
```
ES插件展示索引
![](accents\image-202505020027.png)
Kibana展示数据
![](accents\image-202505020028.png)
**输入来自日志多文件，输出到ES集群**
Logstash采集配置文件
```bash
[root@logstash ~]# vim /opt/files.conf
input {
file {
path => "/var/log/message"
type => "system"
start_position => "beginning"
}
}
input {
file {
path => "/var/log/yum.log"
type => "safeware"
start_position => "beginning"
}
}
output {
if [type] == 'system' {
elasticsearch {
index => "system-%{+YYYY.MM.dd}"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
if [type] == 'safeware' {
elasticsearch {
index => "safeware-%{+YYYY.MM.dd}"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
}
```
ES插件展示索引
![](accents\image-202505020029.png)
![](accents\image-202505020030.png)
## 三：数据过滤
### 1. 插件 grok 简介
- grok插件有非常强大的功能，他能匹配一切数据，但是他的性能和对资源的损耗同样让人诟病
- filter的grok是目前logstash中解析非结构化日志数据最好的方式
- grok位于正则表达式之上，所以任何正则表达式在grok中都是有效的
**官网地址：**
正则：https://github.com/logstash-plugins/logstash-patterns-core/blob/main/patterns/ecs-v1/grok-patterns
### 2. 语法格式
- Grok 模式的语法是：%{SYNTAX:SEMANTIC}
- 是`SYNTAX`将与您的文本匹配的模式的名称
- 是`SEMANTIC`您为匹配的文本片段指定的标识符
### 3. 应用案例
**针对nginx的访问日志，获取对应的IP地址**
Logstash采集数据配置文件：
```bash
[root@logstash ~]# vim /opt/grok_nginx_access_logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
filter {
grok {
match => {
"message" => "%{IP:remote_addr}"
}
}
}
output {
elasticsearch {
index => "grok_nginx_access_logstash"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
```
执行：
```bash
[root@logstash ~]# /usr/local/logstash/bin/logstash -f /opt/grok_nginx_access_logstash.conf
```
ES插件确认索引：
![](accents\image-202505020032.png)
Kibana查看新字段：
![](accents\image-202505020033.png)
**针对nginx的访问日志，对应生成新的字段**
Logstash采集数据配置文件：
```bash
[root@logstash ~]# vim /opt/grok_nginx_access_logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
filter {
grok {
match => {
"message" => "%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response_code} %{NUMBER:bytes_sent} \"%{DATA:referrer}\" \"%{DATA:user_agent}\""
}
}
date {
match => [ "timestamp", "dd/MM/YYYY:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
index => "grok_nginx_access_logstash"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
```
Kibana查看新字段：
![](accents\image-202505020034.png)
<h2><center>Logstash</center></h2>
------
## 一：基本概念
### 1. Logstash 简介
​ Logstash 是 Elastic Stack 的中央数据流引擎，用于收集、丰富和统一所有数据，而不管格式或模式。
​ 相当与Elasticsearch，Kibana，及 Beats 共同使用的时候便会拥有特别强大的实时处理能力。
### 2. Logstash 原理
​ Logstash 管道中的每个输入阶段都在其自己的线程中运行。输入将事件写入位于内存（默认）或磁盘上的中央队列。每个管道工作线程从此队列中取出一批事件，通过配置的过滤器运行这批事件，然后通过任何输出运行过滤后的事件。批次的大小和管道工作线程的数量是可配置的。
​ 默认情况下，Logstash 使用管道阶段（输入 → 过滤器和过滤器 → 输出）之间的内存有界队列来缓冲事件。如果 Logstash 不安全地终止，则存储在内存中的所有事件都将丢失。为了防止数据丢失，您可以启用 Logstash 将传输中的事件保存到磁盘。
## 二：安装部署
### 1. 下载安装包
官网：[Past Releases of Elastic Stack Software | Elastic](https://www.elastic.co/downloads/past-releases#logstash)
```bash
[root@logstash ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-8.13.4-linux-x86_64.tar.gz
```
### 2. 解压安装
```bash
[root@logstash ~]# tar xf logstash-8.13.4-linux-x86_64.tar.gz -C /usr/local/
[root@logstash ~]# mv /usr/local/logstash-8.13.4/ /usr/local/logstash
```
### 3. 配置环境
```bash
[root@logstash ~]# vim /etc/profile
JAVA_HOME=/usr/local/logstash/jdk
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME PATH
[root@logstash ~]# source /etc/profile
[root@logstash ~]# java -version
openjdk version "17.0.11" 2024-04-16
OpenJDK Runtime Environment Temurin-17.0.11+9 (build 17.0.11+9)
OpenJDK 64-Bit Server VM Temurin-17.0.11+9 (build 17.0.11+9, mixed mode, sharing)
```
### 4. 数据流向
![](http://182.92.143.66:40072/directlink/img/es/image-202505020031.png)
### 5. 模拟运行
**输入和输出都来自于终端**
```bash
[root@logstash ~]# /usr/local/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
nihao (INPUT)
{
"@version" => "1",
"message" => "nihao (INPUT)",
"@timestamp" => 2025-05-04T08:54:48.278302130Z,
"event" => {
"original" => "nihao (INPUT)"
},
"host" => {
"hostname" => "logstash"
}
}
```
**输入来自终端，输出到ES集群**
Elasticsearch 从8.0开始， 默认开启了SSL安全验证
因此我们需要为 Logstash 配置身份验证凭据才能建立与 Elasticsearch 集群的通信
Logstash创建目录存储Elasticsearch的CA证书
```bash
[root@logstash ~]# mkdir /usr/local/logstash/config/certs
```
Elasticsearch中将证书拷贝到Logstash
```bash
[root@logstash ~]# scp 192.168.159.131:/usr/local/elasticsearch/config/certs/elasticsearch-ca.pem /usr/local/logstash/config/certs/
```
创建Logstash采集数据配置文件
```bash
[root@logstash ~]# vim /opt/stdin.conf
input {
stdin{}
}
output {
elasticsearch {
index => "stdin_logstash"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
```
执行Logstash
```bash
[root@logstash ~]# /usr/local/logstash/bin/logstash -f /opt/stdin.conf
hello world
hello wxin
nice to meet you
```
ES插件查看索引信息
![](http://182.92.143.66:40072/directlink/img/es/image-202505020026.png)
**输入来自日志文件，输出到ES集群**
此处只展示Logstash配置文件
```bash
[root@logstash ~]# vim /opt/nginx_access_logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
output {
elasticsearch {
index => "nginx_access_logstash"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
```
ES插件展示索引
![](http://182.92.143.66:40072/directlink/img/es/image-202505020027.png)
Kibana展示数据
![](http://182.92.143.66:40072/directlink/img/es/image-202505020028.png)
**输入来自日志多文件，输出到ES集群**
Logstash采集配置文件
```bash
[root@logstash ~]# vim /opt/files.conf
input {
file {
path => "/var/log/message"
type => "system"
start_position => "beginning"
}
}
input {
file {
path => "/var/log/yum.log"
type => "safeware"
start_position => "beginning"
}
}
output {
if [type] == 'system' {
elasticsearch {
index => "system-%{+YYYY.MM.dd}"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
if [type] == 'safeware' {
elasticsearch {
index => "safeware-%{+YYYY.MM.dd}"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
}
```
ES插件展示索引
![](http://182.92.143.66:40072/directlink/img/es/image-202505020029.png)
![](http://182.92.143.66:40072/directlink/img/es/image-202505020030.png)
## 三：数据过滤
### 1. 插件 grok 简介
- grok插件有非常强大的功能，他能匹配一切数据，但是他的性能和对资源的损耗同样让人诟病
- filter的grok是目前logstash中解析非结构化日志数据最好的方式
- grok位于正则表达式之上，所以任何正则表达式在grok中都是有效的
**官网地址：**
正则：https://github.com/logstash-plugins/logstash-patterns-core/blob/main/patterns/ecs-v1/grok-patterns
### 2. 语法格式
- Grok 模式的语法是：%{SYNTAX:SEMANTIC}
- 是`SYNTAX`将与您的文本匹配的模式的名称
- 是`SEMANTIC`您为匹配的文本片段指定的标识符
### 3. 应用案例
**针对nginx的访问日志，获取对应的IP地址**
Logstash采集数据配置文件：
```bash
[root@logstash ~]# vim /opt/grok_nginx_access_logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
filter {
grok {
match => {
"message" => "%{IP:remote_addr}"
}
}
}
output {
elasticsearch {
index => "grok_nginx_access_logstash"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
```
执行：
```bash
[root@logstash ~]# /usr/local/logstash/bin/logstash -f /opt/grok_nginx_access_logstash.conf
```
ES插件确认索引：
![](http://182.92.143.66:40072/directlink/img/es/image-202505020032.png)
Kibana查看新字段：
![](http://182.92.143.66:40072/directlink/img/es/image-202505020033.png)
**针对nginx的访问日志，对应生成新的字段**
Logstash采集数据配置文件：
```bash
[root@logstash ~]# vim /opt/grok_nginx_access_logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
filter {
grok {
match => {
"message" => "%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response_code} %{NUMBER:bytes_sent} \"%{DATA:referrer}\" \"%{DATA:user_agent}\""
}
}
date {
match => [ "timestamp", "dd/MM/YYYY:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
index => "grok_nginx_access_logstash"
hosts => [ "https://192.168.159.131:9200" ]
cacert => "/usr/local/logstash/config/certs/elasticsearch-ca.pem"
user => "elastic"
password => "123456"
}
}
```
Kibana查看新字段：
![](http://182.92.143.66:40072/directlink/img/es/image-202505020034.png)