ubuntu14.04离线安装cdh5.6.0(代码片段)

洽洽老大 洽洽老大     2023-02-28     556

关键词:

官方安装文档:http://www.cloudera.com/documentation/enterprise/5-6-x/topics/installation.html
相关包的下载地址:
Cloudera Manager地址:http://archive.cloudera.com/cm5/cm/5/
CDH安装包地址:http://archive.cloudera.com/cdh5/parcels/5.6.0/

由于我们的操作系统为ubuntu14.04,需要下载以下文件:

CDH-5.6.0-1.cdh5.6.0.p0.45-trusty.parcel
CDH-5.6.0-1.cdh5.6.0.p0.45-trusty.parcel.sha1 
manifest.json 

全程采用root安装

机器配置

1. 三台机器的ip和名字为

  • 192.168.10.236 hadoop-1 (内存16G)
  • 192.168.10.237 hadoop-2 (内存8G)
  • 192.168.10.238 hadoop-3 (内存8G)

我们将hadoop-1作为主节点

2. 配置/etc/hosts,使节点间通过 hadoop-X 即可访问其他节点


3. 配置主节点root免密码登录到其他节点(不需要从节点到主节点)

3.1 在hadoop-1上执行ssh-keygen -t rsa -P ''生成无密码密钥对

3.2 将公钥添加到认证文件中:cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

3.3 将认证文件拷贝到hadoop-2和hadoop-3的/root/.ssh/目录下,使主节点免密码访问从节点


4. 配置jdk

4.1 安装oracle-j2sdk1.7版本(主从都要,根据CDH版本选择对应的jdk)

$ apt-get install oracle-j2sdk1.7
$ update-alternatives --install /usr/bin/java java /usr/lib/jvm/java-7-oracle-cloudera/bin/java 300
$ update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/java-7-oracle-cloudera/bin/javac 300

4.2 $ vim /etc/profile

在末尾添加

export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
export JRE_HOME=$JAVA_HOME/jre 
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:JRE_HOME/bin:$PATH

4.3 $ vim /root/.bashrc

在末尾添加

source /etc/profile

5. 安装MariaDB-5.5(兼容性请查看官方文档,具体可参照ubuntu14.04 安装MariaDB10.0并允许远程访问)

5.1 执行$ apt-get install mariadb-server-5.5

5.2 数据库设置(官方建议设置)

$ vim /etc/mysql/my.cnf

下面是官方建议的配置

[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

重启服务,service mysql restart

5.3 创建相关数据库

进入mysql命令行:$ mysql -u root -p
进入mysql命令行后,直接复制下面的整段话并粘贴:
create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
grant all on amon.* TO 'amon'@'CDH' IDENTIFIED BY 'amon_password';
create database smon DEFAULT CHARACTER SET utf8;
grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password';
grant all on smon.* TO 'smon'@'CDH' IDENTIFIED BY 'smon_password';
create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';
grant all on rman.* TO 'rman'@'CDH' IDENTIFIED BY 'rman_password';
create database hmon DEFAULT CHARACTER SET utf8;
grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password';
grant all on hmon.* TO 'hmon'@'CDH' IDENTIFIED BY 'hmon_password';
create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';
grant all on hive.* TO 'hive'@'CDH' IDENTIFIED BY 'hive_password';
create database oozie DEFAULT CHARACTER SET utf8;
grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie_password';
grant all on oozie.* TO 'oozie'@'CDH' IDENTIFIED BY 'oozie_password';
create database metastore DEFAULT CHARACTER SET utf8;
grant all on metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';
grant all on metastore.* TO 'hive'@'CDH' IDENTIFIED BY 'hive_password';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'gaoying' WITH GRANT OPTION;
flush privileges;

5.4 安装MariaDB jdbc 驱动

$ apt-get install libmysql-java

5.5 使用cloudera脚本在mysql中进行相关配置:(先完成第6.1后再配置)

$ /opt/cloudera-manager/cm-5.6.0/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p --scm-host localhost scm scm scm_password

6. 安装Cloudera Manager Server 和 Agents

6.1 把安装包解压到主节点

$ mkdir /opt/cloudera-manager
将下载好的cloudera-manager-trusty-cm5.6.0_amd64.tar.gz解压到/opt/cloudera-manager

6.2 创建用户

$ sudo useradd --system --home=/opt/cloudera-manager/cm-5.6.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
//--home 指向你cloudera-scm-server的路径

6.3 创建cloudera manager server的本地数据存储目录(主)

$ sudo mkdir /var/log/cloudera-scm-server
$ sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-server

6.4 在每个Cloudera Manager Agent 节点配置server_host(主从都要)

$ vim /opt/cloudera-manager/cm-5.6.0/etc/cloudera-scm-agent/config.ini
//吧server_host改成主节点名称
server_host=hadoop-1

6.5 将cloudera-manager发送到各从节点对应的目录下(即/opt)

$ scp -r /opt/cloudera-manager root@hadoop-2:/opt

6.6 创建Parcel目录

6.6.1 在主节点上:

创建安装包目录 mkdir -p /opt/cloudera/parcel-repo

将CHD5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中

CDH-5.6.0-1.cdh5.6.0.p0.45-trusty.parcel
CDH-5.6.0-1.cdh5.6.0.p0.45-trusty.parcel.sha1 
manifest.json 

最后将CDH-5.6.0-1.cdh5.6.0.p0.45-trusty.parcel.sha1,重命名为CDH-5.6.0-1.cdh5.6.0.p0.45-trusty.parcel.sha,这点必须注意,否则,系统会重新下载CDH-5.6.0-1.cdh5.6.0.p0.45-trusty.parcel.sha1文件。

6.6.2 在从节点上:mkdir -p /opt/cloudera/parcels

6.7 在各节点上安装依赖

apt-get install安装以下依赖

lsb-base
psmisc
bash
libsasl2-modules
libsasl2-modules-gssapi-mit
zlib1g
libxslt1.1
libsqlite3-0
libfuse2
fuse-utils or fuse
rpcbind

6.8 启动server和agent

主节点上:

/opt/cloudera-manager/cm-5.6.0/etc/init.d/cloudera-scm-server start
/opt/cloudera-manager/cm-5.6.0/etc/init.d/cloudera-scm-agent stop

从节点上:

/opt/cloudera-manager/cm-5.6.0/etc/init.d/cloudera-scm-agent stop

若启动出错,可以查看/opt/cloudera-manager/cm-5.6.0/log里的日志
若没错,则等待几分钟后,在浏览器访问Cloudera Manager Admin Console,我的主节点ip为192.168.10.236,那么访问http://192.168.10.236:7180,默认的用户名和密码为admin


7 CDH5的安装和配置

在new hosts选项卡中用hadoop-[1-3]查找组成集群的主机名

勾选要安装的节点,点继续

出现以下包名,说明本地Parcel包配置无误,直接点继续

安装所有服务

采取默认值

数据库配置

这里也默认



等待安装

安装成功


8 简单测试

8.1 在hadoop上执行MapReduce job

在主节点终端执行

sudo -u hdfs hadoop jar \\
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \\
pi 10 100

终端会输出任务执行情况

root@hadoop-1:~# sudo -u hdfs hadoop jar  /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100
Number of Maps  = 10
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
16/05/18 21:26:58 INFO client.RMProxy: Connecting to ResourceManager at hadoop-1/192.168.10.236:8032
16/05/18 21:26:58 INFO input.FileInputFormat: Total input paths to process : 10
16/05/18 21:26:58 INFO mapreduce.JobSubmitter: number of splits:10
16/05/18 21:26:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1463558073107_0001
16/05/18 21:26:58 INFO impl.YarnClientImpl: Submitted application application_1463558073107_0001
16/05/18 21:26:59 INFO mapreduce.Job: The url to track the job: http://hadoop-1:8088/proxy/application_1463558073107_0001/
16/05/18 21:26:59 INFO mapreduce.Job: Running job: job_1463558073107_0001
16/05/18 21:27:05 INFO mapreduce.Job: Job job_1463558073107_0001 running in uber mode : false
16/05/18 21:27:05 INFO mapreduce.Job:  map 0% reduce 0%
16/05/18 21:27:10 INFO mapreduce.Job:  map 10% reduce 0%
16/05/18 21:27:14 INFO mapreduce.Job:  map 20% reduce 0%
16/05/18 21:27:15 INFO mapreduce.Job:  map 40% reduce 0%
16/05/18 21:27:18 INFO mapreduce.Job:  map 50% reduce 0%
16/05/18 21:27:20 INFO mapreduce.Job:  map 70% reduce 0%
16/05/18 21:27:22 INFO mapreduce.Job:  map 80% reduce 0%
16/05/18 21:27:24 INFO mapreduce.Job:  map 100% reduce 0%
16/05/18 21:27:27 INFO mapreduce.Job:  map 100% reduce 100%
16/05/18 21:27:27 INFO mapreduce.Job: Job job_1463558073107_0001 completed successfully
16/05/18 21:27:27 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=96
        FILE: Number of bytes written=1272025
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=2630
        HDFS: Number of bytes written=215
        HDFS: Number of read operations=43
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=3
    Job Counters 
        Launched map tasks=10
        Launched reduce tasks=1
        Data-local map tasks=10
        Total time spent by all maps in occupied slots (ms)=37617
        Total time spent by all reduces in occupied slots (ms)=2866
        Total time spent by all map tasks (ms)=37617
        Total time spent by all reduce tasks (ms)=2866
        Total vcore-seconds taken by all map tasks=37617
        Total vcore-seconds taken by all reduce tasks=2866
        Total megabyte-seconds taken by all map tasks=38519808
        Total megabyte-seconds taken by all reduce tasks=2934784
    Map-Reduce Framework
        Map input records=10
        Map output records=20
        Map output bytes=180
        Map output materialized bytes=340
        Input split bytes=1450
        Combine input records=0
        Combine output records=0
        Reduce input groups=2
        Reduce shuffle bytes=340
        Reduce input records=20
        Reduce output records=0
        Spilled Records=40
        Shuffled Maps =10
        Failed Shuffles=0
        Merged Map outputs=10
        GC time elapsed (ms)=602
        CPU time spent (ms)=12210
        Physical memory (bytes) snapshot=4803805184
        Virtual memory (bytes) snapshot=15372648448
        Total committed heap usage (bytes)=4912578560
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=1180
    File Output Format Counters 
        Bytes Written=97
Job Finished in 29.482 seconds
Estimated value of Pi is 3.14800000000000000000

在WebUI界面的Clusters > Cluster 1 > Activities > YARN Applications

参考资料:
http://blog.csdn.net/jdplus/article/details/45920733
http://itindex.net/detail/51928-cloudera-manager-cdh5

ubuntu14.04自建局域网源在线安装cdh5.6.0(代码片段)

背景:要搭建一个可以添加节点的cloudera集群,但内部网络访问不了cloudera,能访问的主机访问速度也特别慢。实现方式:搭建一个内部的软件源,采用官网第二种方式安装1.搭建CDH源服务器源服务器地址:... 查看详情

ubuntu14.04自建局域网源在线安装cdh5.6.0(代码片段)

...nager相关包(下载地址:https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/pool/contrib/e/enterprise/)cloudera-manager-agent_5.6.0-1.cm560.p0.54~trusty-cm5_amd64.debcloudera-manager-daemons_5.6.0-1.cm560.p0.54~trusty-cm5_all.debcloudera-manager-server_5.6.0-1.cm560.p0.5... 查看详情

ubuntu14.04用apt在线/离线安装cdh5.1.2[apachehadoop2.3.0]

1、CDH介绍1.1、什么是CDH和CM?  CDH一个对ApacheHadoop的集成环境的封装,可以使用ClouderaManager进行自动化安装。   Cloudera-Managerceql(本文中简称CM)是一个工具,CM能够管理一个大的Hadoopcluster并不是一只要下载tarfiles什么压... 查看详情

ubuntuapt本地源离线安装

今天一台主机(Ubuntu14.04)不知道为什么连不上外网了。只能和局域网内的其他主机相互ping通。但是上面一个正在跑的程序出了问题,需要安装两个额外的包,而且这两个包依赖还挺多的样子。这可急死我了。但... 查看详情

elasticsearch备份快照到hdfs-2.6(cdh5.6.0)(代码片段)

repository-hdfs安装包的下载地址:https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-repository-hdfs/背景说明由于业务需要将ES-2.2.1版本升级到ES-2.3.2版本,需要将index迁移过去,查了很多资料,最后决定用es... 查看详情

elasticsearch备份快照到hdfs-2.6(cdh5.6.0)(代码片段)

repository-hdfs安装包的下载地址:https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-repository-hdfs/背景说明由于业务需要将ES-2.2.1版本升级到ES-2.3.2版本,需要将index迁移过去,查了很多资料,最后决定用es... 查看详情

ubuntu14.04安装visualstudiocode

上一篇简单介绍了Ubuntu14.04上如何创建、运行helloworld程序。这篇介绍Ubuntu14.04如何安装VisualstudioCode。网上推荐的有通过UbuntuMake来进行安装,但是我一直没有成功,所以这里通过下载Zip文件,解压方式安装。1,下载地址:https://co... 查看详情

在 ubuntu 14.04 上安装 tkinter

】在ubuntu14.04上安装tkinter【英文标题】:Installingtkinteronubuntu14.04【发布时间】:2014-12-2909:26:55【问题描述】:我想在Ubuntu14.04上运行python代码,但是当我执行它时,它给了我以下错误消息Traceback(mostrecentcalllast):File"main.py",line2,in<... 查看详情

ubuntu14.04安装docker

 检查Ubuntu的内核版本#uname-r3.13.0-55-generic#要在Ubuntu14.04x64安装Docker,需要确保Ubuntu的版本是64位,而且内核版本需大于3.10版。#如果Ubuntu的版本不满足,还需升级Ubuntusudoapt-get-yupgrade更新apt包目录$sudoapt-getupdate安装https支持$sudoa... 查看详情

如何在 Ubuntu 14.04 上安装 MarkLogic 8?

】如何在Ubuntu14.04上安装MarkLogic8?【英文标题】:HowdoesoneinstallMarkLogic8onUbuntu14.04?【发布时间】:2015-09-0523:42:26【问题描述】:在Ubuntu14.04上安装MarkLogic8的步骤是什么?【问题讨论】:你看过官方文档的安装说明了吗?docs.marklogi... 查看详情

在 ubuntu 14.04 上安装 cassandra 时出错

】在ubuntu14.04上安装cassandra时出错【英文标题】:erroroncassandrainstallationonubuntu14.04【发布时间】:2019-10-1119:38:33【问题描述】:我正在尝试在ubuntu14.04上安装cassandra3.11.4。当我运行安装命令时,返回以下错误dpkg-deb:错误:存档\'/v... 查看详情

在 Ubuntu 14.04 LTS 上安装 Navicat

】在Ubuntu14.04LTS上安装Navicat【英文标题】:InstallingNavicatonUbuntu14.04LTS【发布时间】:2015-05-3113:03:45【问题描述】:有没有办法在不使用wine的情况下在Ubuntu14.04上安装Navicat?直接从deb或apt-getinstall开始。我尝试使用wine安装它,但... 查看详情

ubuntu14.04安装有道词典

  Ubuntu14.04安装有道词典之前要更新系统:1sudoapt-getupdate2sudoapt-getupgrade3sudoapt-getdist-upgrade  在有道官网下载ubuntu版本的有道词典(http://cidian.youdao.com/index-linux.html)。  使用dpkg方式安装:1sudodpkg-iyoudao-dict_1.1.0-0-ub 查看详情

为 Ubuntu 14.04 安装和配置 xampp

】为Ubuntu14.04安装和配置xampp【英文标题】:InstallandconfigurexamppforUbuntu14.04【发布时间】:2017-09-0600:50:36【问题描述】:你好!我一直试图让PHP脚本在Ubuntu14.04上运行5天。我为这些版本安装了PHP5.3、apache2和xampp。我使用Netbeans使用p... 查看详情

我无法在普通的 ubuntu 14.04 上安装 udev

】我无法在普通的ubuntu14.04上安装udev【英文标题】:Ican\'tinstalludevonplainubuntu14.04【发布时间】:2015-01-2920:01:44【问题描述】:我已经安装了普通的Ubuntu14.04服务器(64位)。使用时:sudoapt-getupdate打印此错误:....Hithttp://security.ubun... 查看详情

ubuntu14.04mysql安装及权限配置

Ubuntu14.04MySQL安装及权限配置1.MySQL安装Ubuntu14.04中MySQL安装相对简单,只需要三条命令就可以搞定1.1安装mysql-serversudoapt-getinstallmysql-server安装过程中会提示设置密码,按照内容设置密码即可,然后重复输入就可以了。1.2... 查看详情

GitLab 在 Ubuntu 14.04 上安装在哪里?

】GitLab在Ubuntu14.04上安装在哪里?【英文标题】:WhereisGitLabinstalledonUbuntu14.04?【发布时间】:2014-09-0121:16:24【问题描述】:在UbuntuServer14.04上按照https://about.gitlab.com/downloads/的说明安装了GitLab,但我找不到安装目录。通常说在/home/... 查看详情

ubuntu14.04上安装有道词典

Ubuntu14.04用户在安装前要更新系统,即update&dist-upgrade。下载地址:32/64bitshttp://codown.youdao.com/cidian/linux/youdao-dict_1.0.2~ubuntu_i386.debhttp://codown.youdao.com/cidian/linux/youdao-dict_1.0.2~ubuntu_amd6 查看详情