正文

linux12elk--＞03filebeat(代码片段)

FikL-09-19  FikL-09-19  2022-12-28  362

关键词：

filebeat

一、filebeat

轻量级日志收集组件，基本不消耗内存

主要工作：在每一台机器内收集日志，将日志同步到 > redis > logstash > Elasticsearch
  所以每台机器都要装filebeat，选择对应所需组件进入下载，这里选择filebeat > [选择版本下载](https://www.elastic.co/downloads/past-releases#filebeat)

1.下载安装

[root@elk01 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.1-x86_64.rpm
[root@elk01 ~]# yum localinstall -y filebeat-7.12.1-x86_64.rpm

2.修改配置模式

1）备份配置文件

[root@elk01 ~]# rpm -qc  filebeat | grep filebeat.yml
/etc/filebeat/filebeat.yml

# 查看除注释外的内容
[root@elk01 filebeat]# egrep -v '#' /etc/filebeat/filebeat.yml | egrep '[^\\ ]'
filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log
- type: filestream
  enabled: false
  paths:
    - /var/log/*.log
filebeat.config.modules:
  path: $path.config/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["localhost:9200"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  
 # 备份后再编辑
 [root@elk01 filebeat]# cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.bak

2）收集文件日志，输出到文件中

不同输入类型格式参考：https://www.elastic.co/guide/en/beats/filebeat/7.12/configuration-filebeat-options.html
https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html
file格式：
output.file: # filebeat输出
path: “/tmp/filebeat”
filename: filebeat.log

[root@elk01 ~]# vim /etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:   # filebeat输入
- type: log		   # 可随便定义
  paths:
    - /var/log/messages
    - /var/log/*.log
filebeat.config.modules:
  path: $path.config/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.file:		# filebeat输出
  path: "/tmp/filebeat"
  filename: filebeat.log
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

[root@elk01 filebeat]# /usr/bin/filebeat -c /etc/filebeat/filebeat.yml
重启filebeat
[root@elk01 filebeat]# systemctl restart filebeat.service
[root@elk01 ~]# tailf /tmp/filebeat/filebeat.log
5,\\"5\\":0.03,\\"norm\\":\\"1\\":0.01,\\"15\\":0.025,\\"5\\":0.015","input":"type":"log","ecs":01","type":"filebeat","version":"7.12.1"
	···

可以使用json.cn查看log文件：
json官网

3）收集日志输出到redis

https://www.elastic.co/guide/en/beats/filebeat/current/redis-output.html
redis格式：
output.redis:
hosts: [“localhost”]
password: “my_password”
key: “filebeat”
db: 0
timeout: 5

1> 配置

[root@elk01 ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  paths:
    - /var/log/messages
    - /var/log/*.log
filebeat.config.modules:
  path: $path.config/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.redis:
  hosts: ["192.168.15.13"]
  password: ""	# 对端redis的密码，若没有则为空
  key: "filebeat"
  db: 0		# 第0个数据库，redis默认有16个数据库
  timeout: 5
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# 添加此段，注意修改
output.redis:
  hosts: ["localhost"]
  password: "my_password"
  key: "filebeat"
  db: 0
  timeout: 5

2> 重启

[root@elk01 filebeat]# systemctl restart filebeat.service

3> 对端查看

# 安装redis，并修改bind监听地址为0.0.0.0
# 启动redis
# 进入查看
[root@elk01 ~]# redis-cli
127.0.0.1:6379> KEYS *   #通过filebeat自定义的key
1) "filebeat"
127.0.0.1:6379> lrange filebeat 0 -1
415) "\\"@timestamp\\":\\"2021-05-12T09:05:08.661Z\\",\\"@metadata\\":\\\\\\"1\\\\\\":0.03,\\\\\\"15\\\\\\":0.025,\\\\\\"5\\\\\\":0.025\\""
	···

4>使用logstash读取redis输出到elasticsearch中

redis 参考文档
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html

使用logstash读取redis输出到elasticsearch中
# 配置文件放在哪无所谓，运行时指定好路径即可~
[root@elk01 ~]# vim /etc/logstash/conf.d/redis-system.conf
input 
  redis 
    data_type => 'list'
    host => "192.168.15.13"
    key => "filebeat"
    port => 6379
  


output 
    elasticsearch 
    hosts => ["172.16.1.12:9200"]
    index => "filebeat-systemlog-%+YYYY.MM.dd"
  


# 运行logstash
[root@elk01 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis-system.conf 
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

5> 查看elasticsearch索引以及kibana

3. 通过 nginx 代理 kibana 并实现登录认证：

使用Nginx反向代理，配置域名访问kibana

 wget http://nginx.org/download/nginx-1.16.1.tar.gz
 tar xf nginx-1.16.1.tar.gz
 groupadd www -g 666
 useradd www -u 666 -g 666 -s /sbin/nologin -M
 cd nginx-1.16.1
 ./configure --prefix=/usr/local/nginx-1.16.1 --user=www --group=www --without-http_gzip_module
 make && make install
 ln -s /usr/local/nginx-1.16.1 /usr/local/nginx
 vim /etc/profile.d/nginx.sh
export PATH=$PATH:/usr/local/nginx/sbin
 source /etc/profile
 
system管理nginx
[root@web03 ~]# vim /etc/systemd/system/nginx.service 
[Unit]
Description=nginx - high performance web server
After=network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
ExecReload=/usr/local/nginx/sbin/nginx -s reload
ExecStop=/usr/local/nginx/sbin/nginx -s stop
[Install]
WantedBy=multi-user.target

systemctl start nginx

4.配置kibana域名访问

配置kibana域名访问
 mkdir /usr/local/nginx/conf/conf.d/
[root@elk01 conf]# vim /usr/local/nginx/conf/nginx.conf
#添加一行
include /usr/local/nginx/conf/conf.d/*.conf;

[root@elk01 conf]# vim /usr/local/nginx/conf/conf.d/kibana.conf
upstream kibana_server 
server 127.0.0.1:5601 weight=1 max_fails=3 fail_timeout=60;


server 
    listen 80;
    server_name www.kibana.com;
    location / 
        proxy_pass http://kibana_server;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
     


[root@elk01 conf]# chown www.www /usr/local/nginx/ -R
[root@elk01 conf]# systemctl restart nginx

# 启动Nginx
[root@elk01 ~]# systemctl start --now nginx

# 配置hosts解析
192.168.15.12 linux.kibana.com

# 访问测试
192.168.15.12:80
linux.kibana.com

5.实现登录认证：

yum install httpd-tools –y

[root@elk01 conf]# yum install httpd-tools –y
[root@elk01 conf]# htpasswd -bc
/usr/local/nginx/conf/htpasswd.users kibana kibana

[root@elk01 conf]# cat /usr/local/nginx/conf/htpasswd.users
zhangjie:$apr1$x7K2F2rr$xq8tIKg3JcOUyOzSVuBpz1

修改kibana配置文件===不允许外网访问
[root@elk01 ~]# grep -E '^[^#]' /etc/kibana/kibana.yml 
server.port: 5601
server.host: "172.16.1.12"
elasticsearch.hosts: ["http://172.16.1.40:9200"]

编写nginx配置文件
[root@elk01 conf]# vim /usr/local/nginx/conf/conf.d/kibana5612.conf
upstream kibana_server 
server 172.16.1.12:5601 weight=1 max_fails=3 fail_timeout=60;

server 
    listen 80;
    server_name www.kibana5612.com;
    auth_basic "Restricted Access";
    auth_basic_user_file /usr/local/nginx/conf/htpasswd.users;
    location / 
        proxy_pass http://kibana_server;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
  

[root@elk01 conf]# chown www.www /usr/local/nginx/ -R
[root@elk01 conf]# systemctl reload nginx

6.验证登录：

如果不输入密码无法登录：

除非点击取消之后提示需要认证

二、ELK架构

# 安装filebeat
[root@elk01 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.1-x86_64.rpm
[root@elk01 ~]# yum install filebeat-7.12.1-x86_64.rpm
[root@elk01 ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  paths:
    - /var/log/messages
    - /var/log/*.log
output.redis:
  hosts: ["192.168.15.13"]
  password: ""
  key: "filebeat"
  db: 0
  timeout: 5
[root@elk01 ~]# systemctl start filebeat

# 安装redis
[root@elk01 ~]# yum install redis -y
[root@elk01 ~]# systemctl start redis

# 安装logstash
[root@elk01 ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-7.12.1-x86_64.rpm
[root@elk01 ~]# yum install logstash-7.12.1-x86_64.rpm -y
[root@elk01 ~]# cat redis-system.conf
input 
  redis 

	data_type => 'list'
	host => "192.168.15.13"
	key => "filebeat"
	port => 6379
  


output 

	elasticsearch 

		hosts => ["192.168.13.12:9200"] 
		index => "filebeat-systemlog-%+YYYY.MM.dd" 

	


[root@elk01 ~]# /usr/share/logstash/bin/logstash -f redis-system.conf

# elasticsearch
[root@elk01 ~]# grep -E '^[^#]' /etc/elasticsearch/elasticsearch.yml 
cluster.name: chenyang-by-es
node.name: chenyang-node-01
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"
cluster.initial_master_nodes: node-01
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["172.16.1.12","172.16.1.40"]


# 安装kibana
[root@elk01 ~]# grep -E '^[^#]' /etc/kibana/kibana.yml 
server.port: 5601
server.host: "172.16.1.40"
elasticsearch.hosts: ["http://172.16.1.40:9200"]

# 优化
[root@elk01 ~]# yum install nginx -y
[root@elk01 ~]# vim /etc/nginx/conf.d/kibana.conf
server 
	listen 80;
	server_name kibana.default.cluster.local.com;

	auth_basic "User Authentication";
    auth_basic_user_file /etc/nginx/auth;

	location / 
		proxy_pass http://192.168.13.12:5601;
	



不允许IP访问
server 
	listen 80 default_server;
	server_name localhost;
	return 500;

三、监控ELK

本次使用普罗米修斯监控ELK

1、安装普罗米修斯

# 部署普罗米修斯
[root@elk01 /opt]# wget https://github.com/prometheus/prometheus/releases/download/v2.26.0/prometheus-2.26.0.linux-amd64.tar.gz
[root@elk01 /opt]# tar -xf prometheus-2.26.0.linux-amd64.tar.gz -C /usr/local/
[root@elk01 /opt]# vim /usr/local/prometheus/prometheus.yml
  - job_name: "ELK"
    static_configs:
      - targets: ["192.168.15.71:9114"]

# 部署elasticsearch export
[root@elk01 /opt]# docker run --rm -p 9114:9114 -e "--es.uri=http://172.16.1.12:9200/" justwatch/elasticsearch_exporter:1.1.0

[root@elk01 /opt]# wget https://dl.grafana.com/oss/release/grafana-7.5.6-1.x86_64.rpm
[root@elk01 /opt]# yum install grafana-7.5.6-1.x86_64.rpm
[root@elk01 /opt]# systemctl start grafana-server.service

参考

elk配置笔记

filebeat安装配置1下载,安装wgethttps://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.4.3-linux-x86_64.tar.gzmvfilebeat-8.4.3-linux-x86_64.tar.gz/optcd/opttar-zxvffilebeat-8.4.3-linux-x86_64.tar.gzcd 查看详情

elk-filebeat-（效果图示）

一、vimfilebeat-6.3.2-linux-x86_64/filebeat.yml-type:log#Changetotruetoenablethisinputconfiguration.#enabled:false改为true才生效enabled:true#Pathsthatshouldbecrawledandfetched.Globbasedpaths.paths:#-/var/log 查看详情

elk部署elk+filebeat日志收集分析系统(代码片段)

说明：此安装流程只适用于8.0.0以下的版本1.ElasticSearch部署1.1下载ElasticSearch的wget指令：wgethttps://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.4-linux-x86_64.tar.gz1.2解压安装包到指定目录指定解压缩到/usr/ 查看详情

Filebeat/ELK中如何为不同的日志定义单独的索引？

】Filebeat/ELK中如何为不同的日志定义单独的索引？【英文标题】：HowtodefineseperatedindexesfordifferentlogsinFilebeat/ELK?【发布时间】：2016-12-1407:33:09【问题描述】：我想知道如何为提取到logstash（后来被传递到elasticsearch）的不同日志创... 查看详情

elk笔记10--filebeat使用

elk笔记10--filebeat使用1filebeat介绍2filebeat使用案例2.1软件安装2.2采集数据到kafka2.3采集数据到es3使用技巧3.1filebeat将日志按照类别发送到不同kafkatopic3.2filebeat将日... 查看详情

elk二进制安装filebeat开机自启动设置

我的是Ubuntu20.04我的安装路径为：/usr/local/filebeat/filebeat-7.6.2-linux-x86_64系统环境：ubuntu20.04filebeat版本：7.6.2配置方法1.进入自启动目录：cd/lib/systemd/systemvimfilebeat.service2.编写filebeat.service#############事例#############[Unit]Description=filebeatW... 查看详情

elk部署详解--filebeat

filebeat.yml######################FilebeatConfigurationExample##########################Thisfileisanexampleconfigurationfilehighlightingonlythemostcommon#options.Thefilebeat.reference.ymlfilefromthesa 查看详情

在 ELK 堆栈中调试 Filebeat

】在ELK堆栈中调试Filebeat【英文标题】：DebuggingFilebeatintheELKstack【发布时间】：2018-04-1420:22:56【问题描述】：我的ELK系统出现了一些问题。客户端工作如下：Filebeat->Logstash-->Elastic-->Kibana我们的部分日志不会从特定机器到达... 查看详情

elk5.5.2分布式日志实战(代码片段)

...，分布式日志将以下图分布进行安装部署以及配置。二.Filebeat插件安装以及配置1.下载Filebeat插件5.5.2版本wgethttps://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.5.2-linux-x86_64.tar.gz2.解压fi 查看详情

elk部署步骤最后filebeat+elk

ELK结构ELK由ElasticSearch、Logstash、Kiabana组成。ElasticSearchLogstashKiabana补充Elasticsearch集群部署一、环境准备（在Node1、Node2节点上操作，只展示node1）更改node节点主机名配置域名解析安装java二、部署Elasticsearch软件安装软件... 查看详情

原版filebeat+elk(代码片段)

...日志的处理尤为重要。今天，在这里分享一下自己部署的Filebeat+ELK开源实时日志分析平台的记录过程，有不对的地方还望指出。简单介绍：日志主要包括系统日志、应用程序日志和安全日志。系统运维和开发人员可以通过日志了... 查看详情

elk_疑难杂症处理

...=>"GB2312"将GB2312的文本编码，转为UTF-8的编码。也可以在filebeat中实现编码的转换（推荐）：filebeat.prospectors:?-input_type:log?paths:?-c:UsersAdministratorDesktopperformanceTrace.txt?encoding:GB23122、删除多余日志中的多余行logstashfilter中drop删除：if... 查看详情

elk+kafka+filebeat

ELK+Kafka+Filebeat学习https://blog.csdn.net/qq_21383435/article/details/79463832https://blog.csdn.net/xiangyuan1988/article/details/78977471https://www.jianshu.com/p/f149a76ea5b5https://blog.csdn.net/qq 查看详情

初探elk-filebeat使用小结

初探ELK-filebeat使用小结2016/9/18一、安装1、下载有2种方式下载，推荐缓存rpm包到本地yum源1）直接使用rpm[[email protected] ~]# curl -L -O https://download.elastic.co/beats/filebeat/filebeat-1.3.1-x86_64.rpm2 查看详情

日志分析系统elk(elasticsearch+logstash+kibana+filebeat)

...;二、安装Logstash三、安装Kibana四、安装Filebeat五、集群模式搭建日志分析系统ELK(elasticsearch+logstash+kibana+filebeat)这里先介绍ELK的安装首先下载ELK在官网下载：https://www.elastic.co/cn/downloads/ 查看详情

ELK 堆栈中的 Logstash 和 filebeat

】ELK堆栈中的Logstash和filebeat【英文标题】：LogstashandfilebeatintheELKstack【发布时间】：2019-08-1620:14:04【问题描述】：我们正在服务器上设置elasticsearch、kibana、logstash和filebeat，以分析来自许多应用程序的日志文件。由于原因*，每个... 查看详情

elk+filebeat+kafka+zookeeper构建海量日志分析平台

原创作品，允许转载，转载时请务必以超链接形式标明文章原始出处、作者信息和本声明。否则将追究法律责任。http://tchuairen.blog.51cto.com/3848118/1861167什么要做日志分析平台？随着业务量的增长，每天业务服务器将会产... 查看详情

linuxcentos7安装elk技术栈linux系统安装elasticsearch8.1.3kibana8.1.3logstash8.1.3filebeat8.1.3(代码片段)

ElasticSearch安装1、创建目录、进入目录下载压缩包，解压文件sudomkdir-p/usr/local/ELK/escd/usr/local/ELK/essudowgethttps://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.1.3-linux-x86_64.tar.gzsudotarzxv 查看详情

正文