关键词:
logstash是java应用,依赖JDK,首先需要安装JDK,在安装jdk过程中,logstash-2.3.4使用JDK-1.7版本有bug,使用JDK-1.8版本正常,因此我们安装JDK-1.8版本。
安装JDK
官网地址:http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
# rpm -ivh jdk-8u101-linux-x64.rpm # echo "export JAVA_HOME=/usr/java/latest" >> /etc/profile # echo "export PATH=$PATH:$JAVA_HOME/bin" >> /etc/profile # source /etc/profile # java -version java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
安装logstash
官网地址:https://www.elastic.co/products/logstash
# tar xf logstash-2.3.4.tar.gz -C /usr/local/app/ # ln -sv /usr/local/app/logstash-2.3.4 /usr/local/logstash # cd /usr/local/logstash
# mkdir patterns
NGUSERNAME [a-zA-Z\.\@\-\+_%]+ NGUSER %{NGUSERNAME} NGINXACCESS %{IPORHOST:server_name} %{IPORHOST:server_ip} %{IPORHOST:client_ip} %{NUMBER:client_port} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:body_bytes_sent}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent} %{NUMBER:request_time} (?:%{NUMBER:upstream_response_time}|-)
SYSLOGBASE2 (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}: SYSLOGPAMSESSION %{SYSLOGBASE} (?=%{GREEDYDATA:message})%{WORD:pam_module}\(%{DATA:pam_caller}\): session %{WORD:pam_session_state} for user %{USERNAME:username}(?: by %{GREEDYDATA:pam_by})? CRON_ACTION [A-Z ]+ CRONLOG %{SYSLOGBASE} \(%{USER:user}\) %{CRON_ACTION:action} \(%{DATA:message}\) SYSLOGLINE %{SYSLOGBASE2} %{GREEDYDATA:message} # IETF 5424 syslog(8) format (see http://www.rfc-editor.org/info/rfc5424) SYSLOG5424PRI <%{NONNEGINT:syslog5424_pri}> SYSLOG5424SD \[%{DATA}\]+ SYSLOG5424BASE %{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(?:%{WORD:syslog5424_app}|-) +(?:%{WORD:syslog5424_proc}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|) SYSLOG5424LINE %{SYSLOG5424BASE} +%{GREEDYDATA:syslog5424_msg}
编写配置文件
配置文件编写是一个难点,这里有一些示例供参考:https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
input { beats { port => 5048 host => "10.80.2.181" } } filter { if [type] == "51-nginxaccesslog" { grok { patterns_dir => ["./patterns"] match => { "message" => "%{IPORHOST:server_name} %{IPORHOST:server_ip} %{IPORHOST:clientip} %{NUMBER:clientport} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} (?:%{NUMBER:upstream_time:float}|-)" } remove_field => ["message"] } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] } } else if [type] == "51-nginxerrorlog" { grok { patterns_dir => ["./patterns"] match => { "message" => "%{DATESTAMP} %{SYSLOG5424SD:nginx_error_level} %{GREEDYDATA:nginx_error_msg}"} remove_field => ["message"] } date { match => [ "timestamp", "YYYY/MMM/dd HH:mm:ss"] } } else if [type] == "51-phperrorlog" { grok { patterns_dir => ["./patterns"] match => { "message" => "%{SYSLOG5424SD} (?:%{DATA:php_error_level}\:) %{GREEDYDATA:error_msg}" } remove_field => ["message"] } date { match => [ "timestamp", "dd-MMM-YYYY HH:mm:ss Z"] } } } output { if "_grokparsefailure" in [tags] { file { path => "/var/log/logstash/grokparsefailure-%{[type]}-%{+YYYY.MM.dd}.log" } } elasticsearch { hosts => ["10.80.2.83:9200","10.80.2.84:9200"] sniffing => true manage_template => false template_overwrite => true index => "%{[type]}-%{+YYYY.MM.dd}" document_type => "%{[type]}" } }
编写启动脚本
#!/bin/sh # Init script for logstash # Maintained by Elasticsearch # Generated by pleaserun. # Implemented based on LSB Core 3.1: # * Sections: 20.2, 20.3 # ### BEGIN INIT INFO # Provides: logstash # Required-Start: $remote_fs $syslog # Required-Stop: $remote_fs $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: # Description: Starts Logstash as a daemon. ### END INIT INFO PATH=/sbin:/usr/sbin:/bin:/usr/bin export PATH if [ `id -u` -ne 0 ]; then echo "You need root privileges to run this script" exit 1 fi name=logstash pidfile="/usr/local/logstash/$name.pid" LS_USER=nobody LS_GROUP=nobody LS_HOME=/usr/local/logstash #LS_HOME=/home/logstash LS_HEAP_SIZE="12g" LS_LOG_DIR=/data/logstash/log LS_LOG_FILE="${LS_LOG_DIR}/$name.log" LS_CONF_DIR=/usr/local/logstash/conf.d LS_OPEN_FILES=65535 LS_NICE=-20 LS_THREADS=8 KILL_ON_STOP_TIMEOUT=${KILL_ON_STOP_TIMEOUT-0} #default value is zero to this variable but could be updated by user request LS_OPTS="" [ -r /etc/default/$name ] && . /etc/default/$name [ -r /etc/sysconfig/$name ] && . /etc/sysconfig/$name program=/usr/local/logstash/bin/logstash args="agent -f ${LS_CONF_DIR} -w ${LS_THREADS} -l ${LS_LOG_FILE} ${LS_OPTS}" quiet() { "$@" > /dev/null 2>&1 return $? } start() { LS_JAVA_OPTS="${LS_JAVA_OPTS} -Djava.io.tmpdir=${LS_HOME}" HOME=${LS_HOME} export PATH HOME LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING LS_GC_LOG_FILE # chown doesn't grab the suplimental groups when setting the user:group - so we have to do it for it. # Boy, I hope we're root here. SGROUPS=$(id -Gn "$LS_USER" | tr " " "," | sed 's/,$//'; echo '') if [ ! -z $SGROUPS ] then EXTRA_GROUPS="--groups $SGROUPS" fi # set ulimit as (root, presumably) first, before we drop privileges ulimit -n ${LS_OPEN_FILES} # Run the program! nice -n ${LS_NICE} chroot --userspec $LS_USER:$LS_GROUP $EXTRA_GROUPS / sh -c " cd $LS_HOME ulimit -n ${LS_OPEN_FILES} exec \"$program\" $args " > "${LS_LOG_DIR}/$name.stdout" 2> "${LS_LOG_DIR}/$name.err" & # Generate the pidfile from here. If we instead made the forked process # generate it there will be a race condition between the pidfile writing # and a process possibly asking for status. echo $! > $pidfile echo "$name started." return 0 } stop() { # Try a few times to kill TERM the program if status ; then pid=`cat "$pidfile"` echo "Killing $name (pid $pid) with SIGTERM" kill -TERM $pid # Wait for it to exit. for i in 1 2 3 4 5 6 7 8 9 ; do echo "Waiting $name (pid $pid) to die..." status || break sleep 1 done if status ; then if [ $KILL_ON_STOP_TIMEOUT -eq 1 ] ; then echo "Timeout reached. Killing $name (pid $pid) with SIGKILL. This may result in data loss." kill -KILL $pid echo "$name killed with SIGKILL." else echo "$name stop failed; still running." return 1 # stop timed out and not forced fi else echo "$name stopped." fi fi } status() { if [ -f "$pidfile" ] ; then pid=`cat "$pidfile"` if kill -0 $pid > /dev/null 2> /dev/null ; then # process by this pid is running. # It may not be our pid, but that's what you get with just pidfiles. # TODO(sissel): Check if this process seems to be the same as the one we # expect. It'd be nice to use flock here, but flock uses fork, not exec, # so it makes it quite awkward to use in this case. return 0 else return 2 # program is dead but pid file exists fi else return 3 # program is not running fi } reload() { if status ; then kill -HUP `cat "$pidfile"` fi } force_stop() { if status ; then stop status && kill -KILL `cat "$pidfile"` fi } configtest() { # Check if a config file exists if [ ! "$(ls -A ${LS_CONF_DIR}/* 2> /dev/null)" ]; then echo "There aren't any configuration files in ${LS_CONF_DIR}" return 1 fi HOME=${LS_HOME} export PATH HOME test_args="--configtest -f ${LS_CONF_DIR} ${LS_OPTS}" $program ${test_args} [ $? -eq 0 ] && return 0 # Program not configured return 6 } case "$1" in start) status code=$? if [ $code -eq 0 ]; then echo "$name is already running" else start code=$? fi exit $code ;; stop) stop ;; force-stop) force_stop ;; status) status code=$? if [ $code -eq 0 ] ; then echo "$name is running" else echo "$name is not running" fi exit $code ;; reload) reload ;; restart) quiet configtest RET=$? if [ ${RET} -ne 0 ]; then echo "Configuration error. Not restarting. Re-run with configtest parameter for details" exit ${RET} fi stop && start ;; configtest) configtest exit $? ;; *) echo "Usage: $SCRIPTNAME {start|stop|force-stop|status|reload|restart|configtest}" >&2 exit 3 ;; esac exit $?
# chomd +x /etc/init.d/logstash # chown –R nobody.nobody /usr/local/logstash # chkconfig --add logstash
日志分析系统elk(elasticsearch+logstash+kibana+filebeat)
目录 一、安装es 二、安装Logstash三、安装Kibana四、安装Filebeat五、集群模式搭建日志分析系统ELK(elasticsearch+logstash+kibana+filebeat)这里先介绍ELK的安装 首先下载ELK在官网下载... 查看详情
日志分析系统elk之logstash(代码片段)
Logstash什么是ELKLogstash简介Logstash组成1、输入2、过滤器(可选)3、输出Logstash安装与配置通过命令行运行Logstash参数-e参数-f日志输出到文件日志上传到elasticsearchLogstash伪装为日志服务器grok过滤插件分割命令行的信息输出... 查看详情
第五章作业
一、.需求分析的目的是什么,有什么作用?需求分析的目的:是要求开发人员准确地理解用户需要什么,进行细致地调查分析,将用户的需陈述转化为完整的需求定义,再由需求定义转化为相应的软件需求规格说明。需求分析... 查看详情
第五章查询处理和执行
...章查询处理和执行1.sqlserver通过四个步骤处理一个查询,分析,algebrizing,优化,执行。2.分析是分析语法错误生成分析树,绑定部分有,名字解析,类型推倒,聚合绑定,组合绑定。查询优化器,将查询树找到好的执行计划,如... 查看详情
企业运维之elk日志分析平台(logstash)(代码片段)
ELK日志分析平台--Logstash数据采集介绍与配置1.Logstash简介2.Logstash组成3.Logstash安装与配置3.1运行logstash3.2file输出插件3.3elasticsearch输出插件3.4Syslog输入插件3.5多行过滤插件3.6grok过滤插件1.Logstash简介Logstash是用于日志的搜集、分析、... 查看详情
第五章相关分析第二小组作业组长:乙佳荣
第五章相关分析第二小组成员:组长:乙佳荣组员:王洋于媛龄李天婵小组成员任务:乙佳荣:分配监督任务,进行ppt制作王洋:查询资料并做总结李天婵:回收资料,制作ppt于媛龄:总结ppt并讲解pptppt展示: 查看详情
第五章相关分析第七组作业
第五章 相关分析小组作业组号:7 一、小组成员与小组任务 1.小组成员组长:余富强组员:陈洋、单兰东、金硕、颜洋洋 小组成员任务余富强:负责制作幻灯片,全程督导组员完成任务金硕:负责幻灯片讲解以及... 查看详情
第五章
目录消息队列redis1.消息队列RabbitMQ安装安装erlang http://www.erlang.org/安装rabbitmq http://www.rabbitmq.com/download.html安装pika https://pika.readthedocs.io/en/0.10.0/1#!/usr/bin/envpython2importpika3c 查看详情
软件工程-第五章(结构化分析与设计)
一、结构化分析二、数据流图三、分层数据流图的审查四、数据字典五、描述基本加工的小说明六、结构化设计概述七、数据流图到软件体系结构的映射(信息流、数据流图的类型)八、初始结构图的改进 查看详情
第五章多重线性回归分析第四组小组作业
第四组小组成员:组长:姜锋 201550689组员: 王可欣 201552324 王凤仪 201500003 钱镜 201552661 &n 查看详情
第五章pycharm编辑器安装和使用
第五章pycharm编辑器安装和使用1、什么是python编辑器? python编辑器就是能够用来编辑Python代码的编辑器。2、Python编辑器有哪些呢? PythonIDE(集成开发环境),有JupyterNotebook、PyCharm、Atom、Redeo编辑器等... 查看详情
elk搭建实时日志分析平台之二logstash和kibana搭建(代码片段)
...时日志分析平台之一ElasticSearch》文:铁乐与猫四、安装Logstashlogstash是一个数据分析软件,主要目的是分析log日志。1)下载和解压logstash下载地址:https://www.elastic.co/cn/downloads/logstash上传到服务器/usr/ELK目录后,解压:sudotar-zxvflog... 查看详情
linuxelk日志分析系统|logstash日志收集|elasticsearch搜索引擎|kibana可视化平台|架构搭建|超详细(代码片段)
LinuxELK日志分析系统|logstash日志收集|elasticsearch搜索引擎|kibana可视化平台|架构搭建|超详细ELK日志分析系统1.日志服务器2.ELK日志分析系统3日志处理步骤一、Elasticsearch介绍1.1概述1.2核心概念二、Kibana介绍三ELK架构搭建3.1配置要求3.... 查看详情
elk(elasticsearch+logstash+kibana)开源日志分析平台搭建
环境介绍System: CentOS7.2x86_64hostname: elk-server.huangming.orgIPAddress:10.0.6.42、10.17.83.42本篇的ELK环境为单机部署方式,即将ELK所有的软件包都安装在一台服务器上,配置如下:CPU: 4cMem: 8GDisk: 50一、Elasticsearch安装1 查看详情
centos7搭建elkcluster集群日志分析平台
...台(一)已经安装完Elasticsearch5.4集群. 安装Logstash步骤 1.安装Java8 官方说明:需要安装Java8,不支持Java9... //自行安装,略过 2.安装Logstash 可以同elasticsearch一样建立repo文件通过y... 查看详情
elk日志分析平台之logstash数据采集(代码片段)
目录logstash简介数据采集三要素:输入,过滤和输出一Logstash安装与配置二Logstash的输入输出1命令行方式:标准输入到标准输出2conf文件方式:标准输入,输出到文件3conf文件方式:标准输入,输出到ES和... 查看详情
elk日志分析平台之logstash数据采集(代码片段)
目录logstash简介数据采集三要素:输入,过滤和输出一Logstash安装与配置二Logstash的输入输出1命令行方式:标准输入到标准输出2conf文件方式:标准输入,输出到文件3conf文件方式:标准输入,输出到ES和... 查看详情
第五章相关分析来自212独立团团长关育顺的小组作业
第五章 相关分析小组作业组号:1 一、小组成员与小组任务 1.小组成员组长:关育顺组员:刘嘉雯、徐小川、李泽霖、郭倚天 小组成员任务关育顺:负责幻灯片讲解以及软件操作,全程督导组员完成任务刘嘉雯:... 查看详情