正文

logstash

YDDMAX  YDDMAX  2022-08-13  465

关键词：

logstash作为数据搜集器，主要分为三个部分：input->filter->output 作为pipeline的形式进行处理,支持复杂的操作，如发邮件等

input配置数据的输入和简单的数据转换

filter配置数据的提取，一般使用grok

output配置数据的输出和简单的数据转换

运行：logstash -f /etc/logstash.conf

-f 指定配置文件

-e 只在控制台运行

具体的配置见官网

https://www.elastic.co/products/logstash

Centralize, Transform & Stash Your Data

input

Plugin	Description	Github repository
beats	Receives events from the Elastic Beats framework	logstash-input-beats
couchdb_changes	Streams events from CouchDB’s `_changes` URI	logstash-input-couchdb_changes
elasticsearch	Reads query results from an Elasticsearch cluster	logstash-input-elasticsearch
file	Streams events from files	logstash-input-file
gelf	Reads GELF-format messages from Graylog2 as events	logstash-input-gelf
generator	Generates random log events for test purposes	logstash-input-generator
graphite	Reads metrics from the `graphite` tool	logstash-input-graphite
heartbeat	Generates heartbeat events for testing	logstash-input-heartbeat
http	Receives events over HTTP or HTTPS	logstash-input-http
http_poller	Decodes the output of an HTTP API into events	logstash-input-http_poller
jdbc	Creates events from JDBC data	logstash-input-jdbc
kafka	Reads events from a Kafka topic	logstash-input-kafka
log4j	Reads events over a TCP socket from a Log4j`SocketAppender` object	logstash-input-log4j
lumberjack	Receives events using the Lumberjack protocl	logstash-input-lumberjack
rabbitmq	Pulls events from a RabbitMQ exchange	logstash-input-rabbitmq
redis	Reads events from a Redis instance	logstash-input-redis
s3	Streams events from files in a S3 bucket	logstash-input-s3
sqs	Pulls events from an Amazon Web Services Simple Queue Service queue	logstash-input-sqs
stdin	Reads events from standard input	logstash-input-stdin
syslog	Reads syslog messages as events	logstash-input-syslog
tcp	Reads events from a TCP socket	logstash-input-tcp
twitter	Reads events from the Twitter Streaming API	logstash-input-twitter
udp	Reads events over UDP	logstash-input-udp

Community supported plugins

These plugins are maintained and supported by the community. These plugins have met the Logstash development & testing criteria for integration. Contributors include Community Maintainers, the Logstash core team at Elastic, and the broader community.

Plugin	Description	Github repository
cloudwatch	Pulls events from the Amazon Web Services CloudWatch API	logstash-input-cloudwatch
drupal_dblog	Retrieves watchdog log events from Drupal installations with DBLog enabled	logstash-input-drupal_dblog
eventlog	Pulls events from the Windows Event Log	logstash-input-eventlog
exec	Captures the output of a shell command as an event	logstash-input-exec
ganglia	Reads Ganglia packets over UDP	logstash-input-ganglia
gemfire	Pushes events to a GemFire region	logstash-input-gemfire
github	Reads events from a GitHub webhook	logstash-input-github
heroku	Streams events from the logs of a Heroku app	logstash-input-heroku
imap	Reads mail from an IMAP server	logstash-input-imap
irc	Reads events from an IRC server	logstash-input-irc
jmx	Retrieves metrics from remote Java applications over JMX	logstash-input-jmx
kinesis	Receives events through an AWS Kinesis stream	logstash-input-kinesis
meetup	Captures the output of command line tools as an event	logstash-input-meetup
pipe	Streams events from a long-running command pipe	logstash-input-pipe
puppet_facter	Receives facts from a Puppet server	logstash-input-puppet_facter
rackspace	Receives events from a Rackspace Cloud Queue service	logstash-input-rackspace
relp	Receives RELP events over a TCP socket	logstash-input-relp
rss	Captures the output of command line tools as an event	logstash-input-rss
salesforce	Creates events based on a Salesforce SOQL query	logstash-input-salesforce
snmptrap	Creates events based on SNMP trap messages	logstash-input-snmptrap
sqlite	Creates events based on rows in an SQLite database	logstash-input-sqlite
stomp	Creates events received with the STOMP protocol	logstash-input-stomp
unix	Reads events over a UNIX socket	logstash-input-unix
varnishlog	Reads from the `varnish` cache shared memory log	logstash-input-varnishlog
websocket	Reads events from a websocket	logstash-input-websocket
wmi	Creates events based on the results of a WMI query	logstash-input-wmi
xmpp	Receives events over the XMPP/Jabber protocol	logstash-input-xmpp
zenoss	Reads Zenoss events from the fanout exchange	logstash-input-zenoss
zeromq	Reads events from a ZeroMQ SUB socket	logstash-input-zeromq

filter

Plugin	Description	Github repository
aggregate	Aggregates information from several events originating with a single task	logstash-filter-aggregate
anonymize	Replaces field values with a consistent hash	logstash-filter-anonymize
csv	Parses comma-separated value data into individual fields	logstash-filter-csv
date	Parses dates from fields to use as the Logstash timestamp for an event	logstash-filter-date
de_dot	Computationally expensive filter that removes dots from a field name	logstash-filter-de_dot
dissect	Extracts unstructured event data into fields using delimiters	logstash-filter-dissect
dns	Performs a standard or reverse DNS lookup	logstash-filter-dns
drop	Drops all events	logstash-filter-drop
fingerprint	Fingerprints fields by replacing values with a consistent hash	logstash-filter-fingerprint
geoip	Adds geographical information about an IP address	logstash-filter-geoip
grok	Parses unstructured event data into fields	logstash-filter-grok
json	Parses JSON events	logstash-filter-json
kv	Parses key-value pairs	logstash-filter-kv
multiline	Merges multiple lines into a single event	logstash-filter-multiline
mutate	Performs mutations on fields	logstash-filter-mutate
ruby	Executes arbitrary Ruby code	logstash-filter-ruby
sleep	Sleeps for a specified time span	logstash-filter-sleep
split	Splits multi-line messages into distinct events	logstash-filter-split
syslog_pri	Parses the `PRI` (priority) field of a `syslog` message	logstash-filter-syslog_pri
throttle	Throttles the number of events	logstash-filter-throttle
translate	Replaces field contents based on a hash or YAML file	logstash-filter-translate
urldecode	Decodes URL-encoded fields	logstash-filter-urldecode
useragent	Parses user agent strings into fields	logstash-filter-useragent
uuid	Adds a UUID to events	logstash-filter-uuid
xml	Parses XML into fields	logstash-filter-xml

Community supported plugins

Plugin	Description	Github repository
alter	Performs general alterations to fields that the `mutate` filter does not handle	logstash-filter-alter
cidr	Checks IP addresses against a list of network blocks	logstash-filter-cidr
cipher	Applies or removes a cipher to an event	logstash-filter-cipher
clone	Duplicates events	logstash-filter-clone
collate	Collates events by time or count	logstash-filter-collate
elapsed	Calculates the elapsed time between a pair of events	logstash-filter-elapsed
elasticsearch	Copies fields from previous log events in Elasticsearch to current events	logstash-filter-elasticsearch
environment	Stores environment variables as metadata sub-fields	logstash-filter-environment
extractnumbers	Extracts numbers from a string	logstash-filter-extractnumbers
i18n	Removes special characters from a field	logstash-filter-i18n
json_encode	Serializes a field to JSON	logstash-filter-json_encode
metaevent	Adds arbitrary fields to an event	logstash-filter-metaevent
metricize	Takes complex events containing a number of metrics and splits these up into multiple events, each holding a single metric	logstash-filter-metricize
metrics	Aggregates metrics	logstash-filter-metrics
oui	Parse OUI data from MAC addresses	logstash-filter-oui
prune	Prunes event data based on a list of fields to blacklist or whitelist	logstash-filter-prune
punct	Strips all non-punctuation content from a field	logstash-filter-punct
range	Checks that specified fields stay within given size or length limits	logstash-filter-range
tld	Replaces the contents of the default message field with whatever you specify in the configuration	logstash-filter-tld
yaml	Takes an existing field that contains YAML and expands it into an actual data structure within the Logstash event	logstash-filter-yaml
zeromq	Sends an event to ZeroMQ	logstash-filter-zeromq

output

Elastic supported plugins

These plugins are maintained and supported by Elastic.

Plugin	Description	Github repository
csv	Writes events to disk in a delimited format	logstash-output-csv
elasticsearch	Stores logs in Elasticsearch	logstash-output-elasticsearch
email	Sends email to a specified address when output is received	logstash-output-email
file	Writes events to files on disk	logstash-output-file
graphite	Writes metrics to Graphite	logstash-output-graphite
http	Sends events to a generic HTTP or HTTPS endpoint	logstash-output-http
kafka	Writes events to a Kafka topic	logstash-output-kafka
lumberjack	Sends events using the `lumberjack` protocol	logstash-output-lumberjack
rabbitmq	Pushes events to a RabbitMQ exchange	logstash-output-rabbitmq
redis	Sends events to a Redis queue using the `RPUSH`command	logstash-output-redis
s3	Sends Logstash events to the Amazon Simple Storage Service	logstash-output-s3
stdout	Prints events to the standard output	logstash-output-stdout
tcp	Writes events over a TCP socket	logstash-output-tcp
udp	Sends events over UDP	logstash-output-udp

Community supported plugins

Plugin	Description	Github repository
boundary	Sends annotations to Boundary based on Logstash events	logstash-output-boundary
circonus	Sends annotations to Circonus based on Logstash events	logstash-output-circonus
cloudwatch	Aggregates and sends metric data to AWS CloudWatch	logstash-output-cloudwatch
datadog	Sends events to DataDogHQ based on Logstash events	logstash-output-datadog
datadog_metrics	Sends metrics to DataDogHQ based on Logstash events	logstash-output-datadog_metrics
elasticsearch_java	Stores logs in Elasticsearch using the `node` and`transport` protocols	logstash-output-elasticsearch_java
exec	Runs a command for a matching event	logstash-output-exec
ganglia	Writes metrics to Ganglia’s `gmond`	logstash-output-ganglia
gelf	Generates GELF formatted output for Graylog2	logstash-output-gelf
google_bigquery	Writes events to Google BigQuery	logstash-output-google_bigquery
google_cloud_storage	Writes events to Google Cloud Storage	logstash-output-google_cloud_storage
graphtastic	Sends metric data on Windows	logstash-output-graphtastic
hipchat	Writes events to HipChat	logstash-output-hipchat
influxdb	Writes metrics to InfluxDB	logstash-output-influxdb
irc	Writes events to IRC	logstash-output-irc
jira	Writes strutured JSON events to JIRA	logstash-output-jira
juggernaut	Pushes messages to the Juggernaut websockets server	logstash-output-juggernaut
librato	Sends metrics, annotations, and alerts to Librato based on Logstash events	logstash-output-librato
loggly	Ships logs to Loggly	logstash-output-loggly
metriccatcher	Writes metrics to MetricCatcher	logstash-output-metriccatcher
mongodb	Writes events to MongoDB	logstash-output-mongodb
nagios	Sends passive check results to Nagios	logstash-output-nagios
nagios_nsca	Sends passive check results to Nagios using the NSCA protocol	logstash-output-nagios_nsca
newrelic	Sends logstash events to New Relic Insights as custom events	logstash-output-newrelic
opentsdb	Writes metrics to OpenTSDB	logstash-output-opentsdb
pagerduty	Sends notifications based on preconfigured services and escalation policies	logstash-output-pagerduty
pipe	Pipes events to another program’s standard input	logstash-output-pipe
rackspace	Sends events to a Rackspace Cloud Queue service	logstash-output-rackspace
redmine	Creates tickets using the Redmine API	logstash-output-redmine
riak	Writes events to the Riak distributed key/value store	logstash-output-riak
riemann	Sends metrics to Riemann	logstash-output-riemann
sns	Sends events to Amazon’s Simple Notification Service	logstash-output-sns
solr_http	Stores and indexes logs in Solr	logstash-output-solr_http
sqs	Pushes events to an Amazon Web Services Simple Queue Serice queue	logstash-output-sqs
statsd	Sends metrics using the `statsd` network daemon	logstash-output-statsd
stomp	Writes events using the STOMP protocol	logstash-output-stomp
syslog	Sends events to a `syslog` server	logstash-output-syslog
webhdfs	Sends Logstash events to HDFS using the `webhdfs`REST API	logstash-output-webhdfs
websocket	Publishes messages to a websocket	logstash-output-websocket
xmpp	Posts events over XMPP	logstash-output-xmpp
zabbix	Sends events to a Zabbix server	logstash-output-zabbix
zeromq	Writes events to a ZeroMQ PUB socket	logstash-output-zeromq

logstash服务启动脚本

logstash服务启动脚本最近在弄ELK，发现logstash没有sysv类型的服务启动脚本，于是按照网上一个老外提供的模板自己进行修改#添加用户useraddlogstash-M-s/sbin/nologinmkdir/var/log/logstash/chown-Rlogstash:logstash/var/log/logstash/chown-Rlogstash:logstash/usr/ 查看详情

logstash|logstash&&logstash-input-jdbc安装

Windows系统： 1、安装Logstash 　　 1.1进入官网下载zip包　　 [1] https://artifacts.el 查看详情

logstash grok 模式来监控 logstash 本身

】logstashgrok模式来监控logstash本身【英文标题】：logstashgrokpatterntomonitorlogstashitself【发布时间】：2016-05-0316:07:48【问题描述】：我想将logstash.log日志添加到我的ELK堆栈中，但我总是遇到grokparsefailure。我的模式在http://grokconstructor.a... 查看详情

logstash学习小记

logstash学习小记标签（空格分隔）：日志收集IntroduceLogstashisatoolformanagingeventsandlogs.Youcanuseittocollectlogs,parsethem,andstorethemforlateruse(like,forsearching).–http://logstash.net自从2013年logstash被ES公司收购之后，ELK 查看详情

docker安装logstash(代码片段)

使用同版本镜像7.4.11、下载Logstash镜像dockerpulllogstash:7.4.1#查看镜像dockerimages 2、编辑logstash.yml配置文件logstash.yml配置文件放在宿主机/data/elk/logstash目录下，内容如下：path.config:/usr/share/logstash/conf.d/*.confpath.logs:/var/log/logstash ... 查看详情

logstash服务配置

配置文件：/usr/lib/systemd/system/logstash.service相关目录：logstash.conf位于/etc/logstash/conf.d中，logstash.yml位于/etc/logstash中[Unit]Description=LogstashDocumentation=http://www.elastic.coAfter=elasticsearch.service[Service]Environment=LS_HOME=/var/lib/logstashEnvironment=LS_HE... 查看详情

elk-logstash

logstash简介： logstash日志分析的配置和使用 logstash是一个数据分析软件，主要目的是分析log日志。整一套软件可以当作一个MVC模型，logstash是controller层，Elasticsearch是一个model层，kibana是view层。 ... 查看详情

了解一下logstash

　　Logstash 是一个应用程序日志、事件的传输、处理、管理和搜索的平台。你可以用它来统一对应用程序日志进行收集管理，提供 Web 接口用于查询和统计。　　Logstash配置要求　　Logstash支持Java1.7版本以上。 ... 查看详情

logstash(代码片段)

logstash文档地址：https://www.elastic.co/guide/en/logstash/index.htmllogstash命令接收输入并输出logstash-e‘inputstdinoutputstdout‘其它命令--node.nameNAME-f,--path.configCONFIG_PATH-e,--config.stringCONFIG_STRING 查看详情

Logstash: NoMethodError: 方法 `>=' for nil:NilClass Logstash

】Logstash:NoMethodError:方法`>=\\\'fornil:NilClassLogstash【英文标题】：Logstash:NoMethodError:method`>=’fornil:NilClassLogstashLogstash:NoMethodError:方法`>=\'fornil:NilClassLogstash【发布时间】：2016-06-1314:37:05【问题描述】：这是我的配置文查看详情

logstash上报数据到elasticsearch(代码片段)

logstash上报数据到elasticsearch前提是已经安装、部署完logstash和elasticsearchWindows安装启动logstash_zhangphil的博客-CSDN博客_logstashwindows安装Windows安装启动logstash(1)下载logstash，下载链接：DownloadLogstashFree|GetStartedN 查看详情

centos7安装logstash(代码片段)

下载建议到官网下载最新版https://www.elastic.co/cn/downloads/logstash本文使用logstash7.0.0https://artifacts.elastic.co/downloads/logstash/logstash-7.0.0.tar.gzwgethttps://artifacts.elastic.co/downloads/logstash/ 查看详情

logstash上报数据到elasticsearch(代码片段)

cron检测并启动logstash(代码片段)

背景线上的logstash总是莫名其妙的挂了，我打算写一个定时任务，一分钟去检查一次logstash进程，不存在时就把它启动步骤编写检测启动脚本让cron定时来调用检测启动脚本1、编写脚本第一次完成是这个样子：#!/usr/bin/envbashpid_blog=... 查看详情

logstash：运用logstash对serviceapi数据进行分析(代码片段)

我记得在之前的文章“Logstash：使用ELK堆栈进行API分析”运用Logstash对一些指标的API进行分析。在今天的练习中，我将展示如何使用Logstash来对一些日志类的ServiceAPI进行分析。我们知道在很多的时候，我们可以很快速... 查看详情

logstash：运用logstash对serviceapi数据进行分析(代码片段)

elk实验安装logstash

logstash可以理解为log的采集传输组件老样子第一步下载sudowgethttps://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz解压出来sudotar-zxvflogstash-6.2.4.tar.gz编辑一下配置配置ip和日志记录的级别vi/config/logstash.ymlhttp.host:"192. 查看详情

使用filebeat和logstash集中归档日志

方案Filebeat->Logstash->FilesFilebeat->Redis->Logstash->FilesNxlog(Rsyslog、Logstash)->Kafka->Flink(Logstash->ES-Kibana)其他方案(可根据自己需求，选择合适的架构,作者选择了第二种方案)注释: 由于Logstash无法处理输出到文查看详情