正文

flume集群安装监听测试(代码片段)

ptbx1t  ptbx1t  2023-04-11  208

关键词：

1、在已经搭建好集群基础上，配置监听机器配置，主机名 hadoop1,flume-conf.properties配置文件

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.


# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per agent, 
# in this case called ‘agent‘

agent.sources = avroSource
agent.channels = memoryChannel
agent.sinks = loggerSink

# For each one of the sources, the type is defined
agent.sources.avroSource.type = avro
agent.sources.avroSource.channels = memoryChannel
agent.sources.avroSource.bind = 0.0.0.0
agent.sources.avroSource.port = 1234

# Each channel‘s type is defined.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100

# Each sink‘s type must be defined
agent.sinks.loggerSink.type = logger
agent.sinks.loggerSink.channel = memoryChannel

被监听两台机器配置文件：flume-conf.properties

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.


# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per agent, 
# in this case called ‘agent‘

agent.sources = execSource
agent.channels = memoryChannel
agent.sinks = avroSink

# For each one of the sources, the type is defined
agent.sources.execSource.type = exec
agent.sources.execSource.channels = memoryChannel
agent.sources.execSource.command = tail -F /home/hadoop/data/flume/logs/test.log

# Each channel‘s type is defined.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100

# Each sink‘s type must be defined
agent.sinks.avroSink.type = avro
agent.sinks.avroSink.channel = memoryChannel
agent.sinks.avroSink.hostname = hadoop1
agent.sinks.avroSink.port = 1234

技术图片

ha高可用+hive+hbase+sqoop+kafka+flume+spark安装部署(代码片段)

...置文件创建logs文件夹zookeeper.propertiesserver.properties启动集群验证关闭集群flume安装部署解压安装包配置环境变量flume- 查看详情

flume环境安装(代码片段)

源码包下载：http://archive.apache.org/dist/flume/1.8.0/ 集群环境：master192.168.1.99slave1192.168.1.100slave2192.168.1.101 下载安装包：#Masterwgethttp://archive.apache.org/dist/flume/1.8.0/apache-flume 查看详情

flume的安装及使用(代码片段)

Flume的安装及使用文章目录Flume的安装及使用Flume的安装1、上传至虚拟机，并解压2、重命名目录，并配置环境变量3、查看flume版本4、测试flume5、flume的使用Flume的安装1、上传至虚拟机，并解压tar-zxvfapache-flume-1.9.0-bin.tar... 查看详情

flume的安装及使用(代码片段)

flume的安装(代码片段)

一、下载flume并解压1、手动下载flume安装包下载地址下载flume2.将安装包上传到虚拟机上，并解压tar-zxvfapache-flume-1.8.0-bin.tar.gz-C/export/software/3.重命名mvapache-flume-1.8.0-binflume4.修改环境变量5.修改配置文件cdflume/conf/mvflume-env.sh.templ... 查看详情

大数据高级开发工程师——数据采集框架flume(代码片段)

...source和sink组件Flume进阶高可用Flume-NG配置案例1.角色分配2.集群搭建3.配置flumecollection4.启动&测试数据采集框架Flume在一个完整的离线大数据处理系统中，除了hdfs+mapreduce+hive组成分析系统的核心之外，还需要数据采... 查看详情

flume安装和使用(代码片段)

概览1-flume简介2-系统要求3-安装和配置4-启动和测试一、flume的简介官网地址： http://flume.apache.org/1-概述Flume是一种分布式，可靠且可用的服务，用于高效地收集，汇总和移动大量日志数据。它具有基于流式数据流的简单... 查看详情

日志抽取框架flume简介与安装配置(代码片段)

一：flume简介与功能二：flume安装与配置与简单测试一：flume的简介与功能架构1.1flume的简介：1.1.1Flume是Cloudera提供的一个高可用的，高可靠的，分布式的海量日志采集、聚合和传输的系统，Flume支持在日志系统中定制各类数据发... 查看详情

flume整合kafka(代码片段)

一、需求利用flume采集Linux下的文件信息，并且传入到kafka集群当中。环境准备zookeeper集群和kafka集群安装好。二、配置flume官网下载flume。博主自己这里使用的是flume1.6.0。官网地址http://flume.apache.org/download.html解压缩。tar-zxvfapache-f... 查看详情

flumeng学习笔记单机与集群flume配置

下面的内容基本来自官网：http://flume.apache.org/FlumeUserGuide.html本文使用的是最新版本的apacheflume1.5，安装完Flume然后测试下Flume是否可以用，在Flume目录下用以下语句测试：bin/flume-ngagent-n$agent_name-cconf-fconf/flume-conf.properties.template结... 查看详情

flume数据采集之常见集群配置案例(代码片段)

[TOC]非集群配置这种情况非集群配置方式，比较简单，可以直接参考我整理的《Flume笔记整理》，其基本结构图如下：Flume集群之多个Agent一个source结构说明结构图如下：说明如下：即可以把我们的Agent部署在不同的节点上，上面... 查看详情

sparkstreaming+flume+kafka实时流式处理完整流程(代码片段)

...步骤1.引入依赖2.日志收集服务器3.日志接收服务器4、spark集群处理接收数据并写入数据库5、测试结果sparkstreaming+flume+kafka实时流式处理完整流程一、前期准备1、环境准备，四台测试服务器spark集群三台，hadoop02,hadoop... 查看详情

大数据flume数据流监控(代码片段)

...示gmetad所存储数据的PHP前端。在Web界面中以图表方式展现集群的运行状态下收集的多种不同指标数据。1.1安装ganglia（1）规划hadoop102:webgmetadgmodhadoop103:gmodhadoop104:gmod（2）在102103104分别安装epel-release[atguigu@hadoop102... 查看详情

hadoop(代码片段)

...脚本Flume内存优化采集通道启动/停止脚本采集日志Flume集群规划hadoop102hadoop103hadoop104 Flume(采集日志)FlumeFlume 日志采集Flume安装安装部署1.将ae-flume-1.9.0-bin.tar.gz上传到linux的/opt/software目录下2.解压apache-flume-1.9.0-bin.tar.gz到/opt/... 查看详情

flume的安装及使用(代码片段)

chd-5.3.6集群上flume的文件监控(代码片段)

收集hive的log hive的运行日志： /home/hadoop/CDH5.3.6/hive-0.13.1-cdh5.3.6/log/hive.log*memory*hdfs /user/flume/hive-log1.需要四个包：commons-configuration-1.6.jarhadoop-auth-2. 查看详情

activemq使用(代码片段)

...接口3．生产者4．消费监听5．spring配置6．单元测试六、集群环境1．集群方式2．高可用和高并发集群3．集群查看详情

flume+kafka+storm+redis大数据在线实时分析(代码片段)

...面的架构中我们可以看出，其由下面的几部分构成：Flume集群Kafka集群Storm集群从构建实时处理系统的角度出发，我们需要做的是，如何让数据在各个不同的集群系统之间打通（从上面的图示中也能很好地说明这一点），即需要做... 查看详情