关键词：
ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台
日志主要包括系统日志、应用程序日志和安全日志。系统运维和开发人员可以通过日志了解服务器软硬件信息、检查配置过程中的错误及错误发生的原因。经常分析日志可以了解服务器的负荷，性能安全性，从而及时采取措施纠正错误。
通常，日志被分散的储存不同的设备上。如果你管理数十上百台服务器，你还在使用依次登录每台机器的传统方法查阅日志。这样是不是感觉很繁琐和效率低下。当务之急我们使用集中化的日志管理，例如：开源的syslog，将所有服务器上的日志收集汇总。
集中化管理日志后，日志的统计和检索又成为一件比较麻烦的事情，一般我们使用grep、awk和wc等Linux命令能实现检索和统计，但是对于要求更高的查询、排序和统计等要求和庞大的机器数量依然使用这样的方法难免有点力不从心。
官方网站
https://www.elastic.co/
ELK中文指南 http://kibana.logstash.es/content/index.html
1、部署环境 [[email protected] ~]# cat /etc/redhat-release CentOS release 6.8 (Final) 关闭防火墙&Sellinux http://blog.csdn.net/xiegh2014/article/details/53031781 配置yum源 http://blog.csdn.net/xiegh2014/article/details/53031894 两台服务器
节点1安装部署 主机hosts文件配置 [[email protected] ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.16.8.95 elk-node1 172.16.8.96 elk-node2 JAVA安装（安装JDK需要重启操作系统） [[email protected] ~]# rpm -ivh jdk-8u111-linux-x64.rpm elasticsearch安装 [[email protected] ~]# rpm -ivh elasticsearch-5.1.1.rpm [[email protected] ~]# chkconfig --add elasticsearch elasticsearch配置 [[email protected] ~]# chkconfig --add elasticsearch [[email protected] ~]# mkdir -pv /data/elasticsearch/{data,logs} mkdir: 已创建目录 "/data" mkdir: 已创建目录 "/data/elasticsearch" mkdir: 已创建目录 "/data/elasticsearch/data" mkdir: 已创建目录 "/data/elasticsearch/logs" [[email protected] ~]# chown -R elasticsearch.elasticsearch /data/elasticsearch [[email protected] ~]# grep -n ‘^[a-z]‘ /etc/elasticsearch/elasticsearch.yml [[email protected] ~]# vi /etc/elasticsearch/elasticsearch.yml [[email protected] ~]# grep -n ‘^[a-z]‘ /etc/elasticsearch/elasticsearch.yml 17:cluster.name: app-elk 23:node.name: elk-node1 33:path.data: /data/elasticsearch/data 37:path.logs: /data/elasticsearch/logs 43:bootstrap.memory_lock: true 55:network.host: 0.0.0.0 59:http.port: 9200
修改elasticsearch的参数 # 换个集群的名字，免得跟别人的集群混在一起 cluster.name: es-5.0-test # 换个节点名字 node.name: node-101 # 修改一下ES的监听地址，这样别的机器也可以访问 network.host: 0.0.0.0 # 默认的就好 http.port: 9200 # 增加新的参数，这样head插件可以访问es http.cors.enabled: true http.cors.allow-origin: "*" [[email protected] ~]# vi /etc/security/limits.conf # allow user ‘elasticsearch‘ mlockall elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited [[email protected] ~]# vi /etc/security/limits.conf * soft nofile 65536 * hard nofile 131072 * soft nproc 2048 * hard nproc 4096 [[email protected] ~]# vi /etc/security/limits.d/90-nproc.conf 修改如下内容： * soft nproc 1024 #修改为 * soft nproc 2048 [[email protected] ~]# vi /etc/sysctl.conf 添加下面配置： vm.max_map_count=655360 [[email protected] ~]# sysctl -p [[email protected] ~]# /etc/init.d/elasticsearch restart http://172.16.8.95:9200/
节点2安装部署 主机hosts文件配置 [[email protected] ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.16.8.95 elk-node1 172.16.8.96 elk-node2 JAVA安装（安装JDK需要重启操作系统） [[email protected] ~]# rpm -ivh jdk-8u111-linux-x64.rpm elasticsearch安装 [[email protected] ~]# rpm -ivh elasticsearch-5.1.1.rpm [[email protected] ~]# chkconfig --add elasticsearch elasticsearch配置 [[email protected] ~]# mkdir -pv /data/elasticsearch/{data,logs} [[email protected] ~]# chown -R elasticsearch.elasticsearch /data/elasticsearch [[email protected] ~]# grep -n ‘^[a-z]‘ /etc/elasticsearch/elasticsearch.yml 17:cluster.name: app-elk 23:node.name: elk-node2 33:path.data: /data/elasticsearch/data 37:path.logs: /data/elasticsearch/logs 43:bootstrap.memory_lock: true 55:network.host: 0.0.0.0 59:http.port: 9200 [[email protected] ~]# /etc/init.d/elasticsearch restart 报错信息1 [[email protected] ~]# tail -f /data/elasticsearch/logs/app-elk.log [2016-09-19T18:08:11,804][INFO ][o.e.t.TransportService ] [elk-node2] publish_address {172.16.8.96:9300}, bound_addresses {[::]:9300} [2016-09-19T18:08:11,825][INFO ][o.e.b.BootstrapCheck ] [elk-node2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks [2016-09-19T18:08:11,830][ERROR][o.e.b.Bootstrap ] [elk-node2] node validation exception bootstrap checks failed memory locking requested for elasticsearch process but memory is not locked max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048] [2016-09-19T18:08:11,842][INFO ][o.e.n.Node ] [elk-node2] stopping ... [2016-09-19T18:08:11,896][INFO ][o.e.n.Node ] [elk-node2] stopped [2016-09-19T18:08:11,896][INFO ][o.e.n.Node ] [elk-node2] closing ... [2016-09-19T18:08:11,933][INFO ][o.e.n.Node ] [elk-node2] closed [[email protected] ~]# vi /etc/security/limits.conf * soft nofile 65536 * hard nofile 131072 * soft nproc 2048 * hard nproc 4096 [[email protected] ~]# vi /etc/security/limits.d/90-nproc.conf 修改如下内容： * soft nproc 1024 #修改为 * soft nproc 2048 [[email protected] ~]# vi /etc/sysctl.conf 添加下面配置： vm.max_map_count=655360 [[email protected] ~]# sysctl -p [[email protected] ~]# /etc/init.d/elasticsearch restart 报错信息2 [2016-09-19T18:18:19,270][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=无法分配内存 [2016-09-19T18:18:19,270][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out. [2016-09-19T18:18:19,270][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536 [2016-09-19T18:18:19,271][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example: # allow user ‘elasticsearch‘ mlockall elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited [2016-09-19T18:18:19,271][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect. [2016-09-19T18:18:20,000][INFO ][o.e.n.Node ] [elk-node2] initializing ... [2016-09-19T18:18:20,384][INFO ][o.e.e.NodeEnvironment ] [elk-node2] using [1] data paths, mounts [[/ (/dev/sda3)]], net usable_space [39gb], net total_space [43.9gb], spins? [possibly], types [ext4] [2016-09-19T18:18:20,385][INFO ][o.e.e.NodeEnvironment ] [elk-node2] heap size [3.9gb], compressed ordinary object pointers [true] [2016-09-19T18:18:20,391][INFO ][o.e.n.Node ] [elk-node2] node name [elk-node2], node ID [KBLSr8zERri083vvtJBQhA] [2016-09-19T18:18:20,405][INFO ][o.e.n.Node ] [elk-node2] version[5.1.1], pid[25073], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/2.6.32-642.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_111/25.111-b14] [2016-09-19T18:18:29,227][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [aggs-matrix-stats] [2016-09-19T18:18:29,228][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [ingest-common] [2016-09-19T18:18:29,228][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [lang-expression] [2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [lang-groovy] [2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [lang-mustache] [2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [lang-painless] [2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [percolator] [2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [reindex] [2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [transport-netty3] [2016-09-19T18:18:29,229][INFO ][o.e.p.PluginsService ] [elk-node2] loaded module [transport-netty4] [2016-09-19T18:18:29,231][INFO ][o.e.p.PluginsService ] [elk-node2] no plugins loaded [[email protected] ~]# vi /etc/security/limits.conf # allow user ‘elasticsearch‘ mlockall elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited [[email protected] ~]# /etc/init.d/elasticsearch restart http://172.16.8.96:9200/
elk整合安装
...，开源社区正好有相对应的开源项目：logstash（收集）、elasticsearch（存储+搜索）、kibana（展示），我们将这三个组合起来的技术称之为ELKStack，所以说ELKStack指的是Elasticsearch、Logstas 查看详情
初探elk-每天5分钟玩转docker容器技术（89）
...理方案中，最出名的莫过于ELK了。ELK是三个软件的合称：Elasticsearch、Logstash、Kibana。Elasticsearch一个近乎实时查询的全文搜索引擎。Elasticsearch的设计目标就是要能够处理和搜索巨量的日志数据。Logstash读取原始日志，并对其进行... 查看详情
企业运维之elk日志分析平台（elasticsearch）(代码片段)
ELK日志分析平台--Elasticsearch的介绍与安装1.ELK介绍2.Elasticsearch简介3.Elasticsearch的部署3.1单机模式3.2Elasticsearch集群3.3cerebro图形化3.4Elasticsearch-head图形化3.5索引、分片和副本3.6Elasticsearch节点优化1.ELK介绍ELK指的是⼀套完整的⽇志集... 查看详情
elk实验elasticsearch集群搭建
本次实验使用3台虚拟机192.168.209.168192.168.209.169192.168.209.170cp/usr/elasticsearch-6.2.3/config/elasticsearch.yml/usr/elasticsearch-6.2.3/config/elasticsearch.yml.bakvi/usr/elasticsearch-6.2.3/config/elasti 查看详情
elk日志分析平台搭建----elasticsearch
ELK日志分析平台搭建----ELASTICSEARCH介绍：ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成1、Elasticsearch是一个开源分布式的搜索引擎，特点是：分布式、零配置、自动发现、索引自动分片、索引副本机制、restful风格接口，多数据... 查看详情
elk之elasticsearch
一、下载安装包：注意版本统一wgethttps://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.zipwgethttps://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gzwgethttps://artifacts.el 查看详情
批量搞机：分布式elk平台elasticsearch介绍elasticsearch集群安装es插件的安装与使用
...软件,而是一整套解决方案,是三个软件产品的首字母缩写Elasticsearch:负责日志检索和储存Logstash:负责日志的收集和分析、处理Kiban 查看详情
java整合elk日志
...ker-compose搭建一、编写docker-compose.ymlversion:"3"services:elasticsearch:image:elasticsearch:7.5.1container_name:elasticsearchenvironment:-"cluster.name=elasticsearch"#设置集群名称为elasticsearch-"discovery.type=single-node"#以单一节点模式启动-... 查看详情
elk部署详解--elasticsearch
#Elasticsearch是一个实时的分布式搜索和分析引擎，它可以用于全文搜索，结构化搜索以及分析。它是一个建立在全文搜索引擎ApacheLucene基础上的搜索引擎，使用Java语言编写。 elasticsearch.yml#========================ElasticsearchConfigurati... 查看详情
2021年大数据elk：elasticsearch简单介绍
...;帮助大家回顾前面的知识重点。目录系列历史文章 一、Elasticsearch简介1、介绍2、创始人二、Elasticsearch可以做什么1、信息检索2、企业内部系统搜索3、数据分析引擎三、Elasticsearch特点1、海量数据处理2、开箱即用 3、作为传统... 查看详情
日志分析系统elk之elasticsearch(代码片段)
Elasticsearch什么是ELKElasticsearchElasticsearch基础模块elasticsearch应用场景Elasticsearch单节点部署Elasticsearch集群的部署集群简介elasticsearch节点角色集群部署可视化工具cerebro可视化工具elasticsearch-head插件索引、分片和副本Elasticsearch节点优... 查看详情
日志分析系统elk之elasticsearch(代码片段)
Elasticsearch什么是ELKElasticsearchElasticsearch基础模块elasticsearch应用场景Elasticsearch单节点部署Elasticsearch集群的部署集群简介elasticsearch节点角色集群部署可视化工具cerebro可视化工具elasticsearch-head插件索引、分片和副本Elasticsearch节点优... 查看详情
elasticsearch：elk架构(代码片段)
...文件结构LogstashQueueLogstash导入数据到ES同步数据库数据到Elasticsearch什么是BeatsFileBeat简介FileBeat的工作原理logstashvsFileBeatFilebeat安装ELK整合实战案例：采集tomcat服务器日志使用FileBeats将日志发送到Logstash配置Logstash接收FileBeat收... 查看详情
elk中elasticsearch安装启动报错
elasticsearch安装之后。启动报错。elasticsearch版本为5.4.1下载安装：wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.1.tar.gztar zxf elasticsearch-5.4.1.tar.gzmv elasticsea 查看详情
运维实操——日志分析系统elk（上）之elasticsearch(代码片段)
日志分析系统ELK（上）之elasticsearch1、什么是elasticsearch？2、单节点elasticsearch安装3、搭建elasticsearch集群4、elasticsearch可视化方法1——cerebro插件5、elasticsearch可视化方法2——elasticsearch-head插件6、elasticsearch节点角色E 查看详情
elk报错(代码片段)
1、elasticsearch启动失败如下：[[email protected]home]#/home/elasticsearch-6.3.0/bin/elasticsearch[2019-02-11T07:15:14,874][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler][]uncaughtexceptioninthread[main]org.elasticsearch.bootstrap.StartupException:java.lang.RuntimeException:cannotrun... 查看详情
elk实验elasticsearch集群管理
Head插件head插件是一个elasticsearch的集群管理工具，它是完全由html5编写的独立网页程序。Github：https://github.com/mobz/elasticsearch-headgitclonehttps://github.com/mobz/elasticsearch-head.git安装./elasticsearchplugin-install/usr/elast 查看详情
[elk]快速搭建简单的日志分析平台
下载ELK(Elasticsearch,Logstash,Kibana) Elasticsearch：wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.4.tar.gzLogstash：wget https://download.elasticse 查看详情