flume组件以及通过命令监控大数据平台转态-Toy模板网

这篇具有很好参考价值的文章主要介绍了flume组件以及通过命令监控大数据平台转态。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

实验一、Flume 组件安装配置

1、下载和解压 Flume

可以从官网下载 Flume 组件安装包，下载地址如下 URL 链接所示 https://archive.apache.org/dist/flume/1.6.0/

[root@master ~]# ls
anaconda-ks.cfg                jdk-8u152-linux-x64.tar.gz
apache-flume-1.6.0-bin.tar.gz  mysql
apache-hive-2.0.0-bin.tar.gz   mysql-connector-java-5.1.46.jar
derby.log                      sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
hadoop-2.7.1.tar.gz            zookeeper-3.4.8.tar.gz
hbase-1.2.1-bin.tar.gz
#使用 root 用户解压 Flume 安装包到“/usr/local/src”路径，并修改解压后文件夹名为 flume。
[root@master ~]# tar xf apache-flume-1.6.0-bin.tar.gz -C /usr/local/src/
[root@master ~]# cd /usr/local/src/
[root@master src]# ls
apache-flume-1.6.0-bin  hadoop  hbase  hive  jdk  sqoop  zookeeper
[root@master src]# mv apache-flume-1.6.0-bin flume
[root@master src]# ls
flume  hadoop  hbase  hive  jdk  sqoop  zookeeper

2、Flume 组件部署

步骤一：使用 root 用户设置 Flume 环境变量，并使环境变量对所有用户生效。

[root@master ~]# vim /etc/profile.d/flume.sh
export FLUME_HOME=/usr/local/src/flume
export PATH=${FLUME_HOME}/bin:$PATH

步骤二：修改 Flume 相应配置文件。

首先，切换到 hadoop 用户，并切换当前工作目录到 Flume 的配置文件夹。

[root@master ~]# chown -R hadoop.hadoop /usr/local/src/
[root@master ~]# su - hadoop
Last login: Fri Apr 14 16:31:48 CST 2023 on pts/1
[hadoop@master ~]$ ls
derby.log  input  student.java  zookeeper.out
[hadoop@master ~]$ cd /usr/local/src/flume/conf/
[hadoop@master conf]$ ls
flume-conf.properties.template  flume-env.sh.template
flume-env.ps1.template          log4j.properties

拷贝 flume-env.sh.template 文件并重命名为 flume-env.sh。

[hadoop@master conf]$ cp flume-env.sh.template flume-env.sh
[hadoop@master conf]$ ls
flume-conf.properties.template  flume-env.sh           log4j.properties
flume-env.ps1.template          flume-env.sh.template

步骤三：修改并配置 flume-env.sh 文件。

删除 JAVA_HOME 变量前的注释，修改为 JDK 的安装路径。

[hadoop@master conf]$ vi flume-env.sh
export JAVA_HOME=/usr/loocal/src/jdk
#export HBASE_CLASSPATH=/usr/local/src/hadoop/etc/hadoop

使用 flume-ng version 命令验证安装是否成功，若能够正常查询 Flume 组件版本为 1.6.0，则表示安装成功。

[hadoop@master conf]$ flume-ng version
Error: Could not find or load main class org.apache.flume.tools.GetJavaProperty
Flume 1.6.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080
Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015
From source with checksum b29e416802ce9ece3269d34233baf43f

3、使用 Flume 发送和接受信息

通过 Flume 将 Web 服务器中数据传输到 HDFS 中。

步骤一：在 Flume 安装目录中创建 xxx.conf 文件。

[hadoop@master flume]$ vi xxx.conf
a1.sources=r1
a1.sinks=k1
a1.channels=c1
a1.sources.r1.type=spooldir
a1.sources.r1.spoolDir=/usr/local/src/hadoop/logs
a1.sources.r1.fileHeader=true
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://master:9000/tmp/flume
a1.sinks.k1.hdfs.rollsize=1048760
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.rollInterval=900
a1.sinks.k1.hfds.useLocalTimeStamp=true
a1.channels.c1.type=file
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

[hadoop@master flume]$ ls /usr/local/src/hadoop/
bin  etc      lib      LICENSE.txt  NOTICE.txt  sbin   tmp
dfs  include  libexec  logs         README.txt  share
[hadoop@master flume]$ ls /usr/local/src/hadoop/logs/
hadoop-hadoop-namenode-master.example.com.log
hadoop-hadoop-namenode-master.example.com.out
hadoop-hadoop-namenode-master.example.com.out.1
hadoop-hadoop-namenode-master.example.com.out.2
hadoop-hadoop-namenode-master.example.com.out.3
hadoop-hadoop-namenode-master.example.com.out.4
hadoop-hadoop-namenode-master.example.com.out.5
hadoop-hadoop-secondarynamenode-master.example.com.log
hadoop-hadoop-secondarynamenode-master.example.com.out
hadoop-hadoop-secondarynamenode-master.example.com.out.1
hadoop-hadoop-secondarynamenode-master.example.com.out.2
hadoop-hadoop-secondarynamenode-master.example.com.out.3
hadoop-hadoop-secondarynamenode-master.example.com.out.4
hadoop-hadoop-secondarynamenode-master.example.com.out.5
SecurityAuth-hadoop.audit
yarn-hadoop-resourcemanager-master.example.com.log
yarn-hadoop-resourcemanager-master.example.com.out
yarn-hadoop-resourcemanager-master.example.com.out.1
yarn-hadoop-resourcemanager-master.example.com.out.2
yarn-hadoop-resourcemanager-master.example.com.out.3
yarn-hadoop-resourcemanager-master.example.com.out.4
yarn-hadoop-resourcemanager-master.example.com.out.5

步骤二：使用 flume-ng agent 命令加载 simple-hdfs-flume.conf 配置信息，启动 flume 传输数据。

[[hadoop@master flume]$ flume-ng agent --conf-file xxx.conf --name a1
Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hadoop libraries found via (/usr/local/src/hadoop/bin/hadoop) for HDFS access
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath
Info: Including HBASE libraries found via (/usr/local/src/hbase/bin/hbase) for HBASE access
Info: Excluding /usr/local/src/hbase/lib/slf4j-api-1.7.7.jar from classpath
Info: Excluding /usr/local/src/hbase/lib/slf4j-log4j12-1.7.5.jar from classpath
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath
Info: Including Hive libraries found via (/usr/local/src/hive) for Hive access
...
23/04/21 16:02:35 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: r1. src.append.accepted == 0
23/04/21 16:02:35 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: r1. src.append.received == 0
23/04/21 16:02:35 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: r1. src.events.accepted == 17
23/04/21 16:02:35 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: r1. src.events.received == 17
23/04/21 16:02:35 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: r1. src.open-connection.count == 0
23/04/21 16:02:35 INFO source.SpoolDirectorySource: SpoolDir source r1 stopped. Metrics: SOURCE:r1{src.events.accepted=17, src.open-connection.count=0, src.append.received=0, src.append-batch.received=1, src.append-batch.accepted=1, src.append.accepted=0, src.events.received=17}

ctrl+c 退出 flume 传输

#首先得开启所有节点
[hadoop@master flume]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out
192.168.88.201: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out
192.168.88.200: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out
192.168.88.200: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out
192.168.88.201: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out
[hadoop@master flume]$ ss -antl
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128    192.168.88.101:9000                      *:*                  
LISTEN     0      128           *:50090                     *:*                  
LISTEN     0      128           *:50070                     *:*                  
LISTEN     0      128           *:22                        *:*                  
LISTEN     0      128       ::ffff:192.168.88.101:8030                     :::*                  
LISTEN     0      128       ::ffff:192.168.88.101:8031                     :::*                  
LISTEN     0      128       ::ffff:192.168.88.101:8032                     :::*                  
LISTEN     0      128       ::ffff:192.168.88.101:8033                     :::*                  
LISTEN     0      80           :::3306                     :::*                  
LISTEN     0      128          :::22                       :::*                  
LISTEN     0      128       ::ffff:192.168.88.101:8088                     :::*

步骤三：查看 Flume 传输到 HDFS 的文件，若能查看到 HDFS 上/tmp/flume 目录有传输的数据文件，则表示数据传输成功。

[hadoop@master flume]$ hdfs dfs -ls /
Found 5 items
drwxr-xr-x   - hadoop supergroup          0 2023-04-07 15:20 /hbase
drwxr-xr-x   - hadoop supergroup          0 2023-03-17 17:33 /input
drwxr-xr-x   - hadoop supergroup          0 2023-03-17 18:45 /output
drwx------   - hadoop supergroup          0 2023-04-21 16:02 /tmp
drwxr-xr-x   - hadoop supergroup          0 2023-04-14 20:48 /user
[hadoop@master flume]$ hdfs dfs -ls /tmp/flume
Found 72 items
-rw-r--r--   2 hadoop supergroup       1560 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143693
-rw-r--r--   2 hadoop supergroup       1398 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143694
-rw-r--r--   2 hadoop supergroup       1456 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143695
-rw-r--r--   2 hadoop supergroup       1398 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143696
-rw-r--r--   2 hadoop supergroup       1403 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143697
-rw-r--r--   2 hadoop supergroup       1434 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143698
-rw-r--r--   2 hadoop supergroup       1383 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143699
...
-rw-r--r--   2 hadoop supergroup       1508 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143760
-rw-r--r--   2 hadoop supergroup       1361 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143761
-rw-r--r--   2 hadoop supergroup       1359 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143762
-rw-r--r--   2 hadoop supergroup       1502 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143763
-rw-r--r--   2 hadoop supergroup       1399 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143764

实验二、通过命令监控大数据平台运行状态

1、通过命令查看大数据平台状态

步骤一：查看 Linux 系统的信息（uname -a）

[root@master ~]# uname -a
Linux master.example.com 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

结果显示，该 Linux 节点名称为 master，内核发行号为 3.10.0-693.el7.x86_64。

步骤二：查看硬盘信息

（1）查看所有分区（fdisk -l）

[root@master ~]# fdisk -l

Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000af885

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   209715199   103808000   8e  Linux LVM

Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-swap: 8455 MB, 8455716864 bytes, 16515072 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-home: 44.1 GB, 44149243904 bytes, 86228992 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

结果显示，硬盘空间为 107.4GB。

（2）查看所有交换分区（swapon -s）

[root@master ~]# swapon -s
Filename				Type		Size	Used	Priority
/dev/dm-1                              	partition	8257532	0	-1

结果显示，交换分区为 8257532。

（3）查看文件系统占比（df -h）

[root@master ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  4.6G   46G  10% /
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G     0  1.9G   0% /dev/shm
tmpfs                    1.9G   12M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/centos-home   42G   36M   42G   1% /home
/dev/sda1               1014M  142M  873M  14% /boot
tmpfs                    378M     0  378M   0% /run/user/0

结果显示，挂载点“/”的容量为 50G，已使用 4.6G。

步骤三：查看网络 IP 地址（ip a）

[root@master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:20:cd:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.88.101/24 brd 192.168.88.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::75a3:9da2:ede2:5be7/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::1c1b:d0e3:f01a:7c11/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever

结果显示 ens33 的 IP 地址为 192.168.88.101，子网掩码为 255.255.255.0；回环地址为 127.0.0.1，子网掩码为 255.0.0.0。

步骤四：查看所有监听端口（netstat -lntp）

[root@master ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      908/sshd            
tcp6       0      0 :::3306                 :::*                    LISTEN      1053/mysqld         
tcp6       0      0 :::22                   :::*                    LISTEN      908/sshd

结果显示，在监听的端口分别为 22、3306。

步骤五：查看所有已经建立的连接（netstat -antp）

[root@master ~]# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      908/sshd            
tcp        0     36 192.168.88.101:22       192.168.88.1:61450      ESTABLISHED 1407/sshd: root@pts 
tcp6       0      0 :::3306                 :::*                    LISTEN      1053/mysqld         
tcp6       0      0 :::22                   :::*                    LISTEN      908/sshd

结果显示，已经连接上的本地端口分别为 50608、22、9000、8031 等。

步骤六：实时显示进程状态（top），该命令可以查看进程对 CPU、内存的占比等。

步骤七：查看 CPU 信息（ cat /proc/cpuinfo）

[root@master ~]# cat /proc/cpuinfo 
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 25
model		: 80
model name	: AMD Ryzen 7 5800U with Radeon Graphics
stepping	: 0
cpu MHz		: 1896.438
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 16
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl tsc_reliable nonstop_tsc extd_apicid eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext retpoline_amd vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero ibpb ibrs arat pku ospke overflow_recov succor
bogomips	: 3792.87
TLB size	: 2560 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 45 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 25
model		: 80
model name	: AMD Ryzen 7 5800U with Radeon Graphics
stepping	: 0
cpu MHz		: 1896.438
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 16
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl tsc_reliable nonstop_tsc extd_apicid eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext retpoline_amd vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero ibpb ibrs arat pku ospke overflow_recov succor
bogomips	: 3792.87
TLB size	: 2560 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 45 bits physical, 48 bits virtual

步骤八：查看内存信息（ cat /proc/meminfo），该命令可以查看总内存、空闲内存等信息。

[root@master ~]# cat /proc/meminfo 
MemTotal:        3863564 kB
MemFree:         2971196 kB
MemAvailable:    3301828 kB
Buffers:            2120 kB
Cached:           529608 kB
SwapCached:            0 kB
Active:           626584 kB
Inactive:         130828 kB
Active(anon):     226348 kB
Inactive(anon):    11152 kB
Active(file):     400236 kB
Inactive(file):   119676 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       8257532 kB
SwapFree:        8257532 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        225716 kB
Mapped:            28960 kB
Shmem:             11816 kB
Slab:              59428 kB
SReclaimable:      30680 kB
SUnreclaim:        28748 kB
KernelStack:        4432 kB
PageTables:         3664 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    10189312 kB
Committed_AS:     773684 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      185376 kB
VmallocChunk:   34359310332 kB
HardwareCorrupted:     0 kB
AnonHugePages:    180224 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       73536 kB
DirectMap2M:     3072000 kB
DirectMap1G:     3145728 kB

2、通过命令查看 Hadoop 状态

步骤一：切换到 hadoop 用户

[root@master ~]# su - hadoop
Last login: Fri Apr 21 15:26:05 CST 2023 on pts/0
Last failed login: Fri Apr 21 16:02:08 CST 2023 from slave1 on ssh:notty
There were 4 failed login attempts since the last successful login.

若当前的用户为 root，请切换到 hadoop 用户进行操作。

步骤二：切换到 Hadoop 的安装目录

[hadoop@master ~]$ cd /usr/local/src/hadoop/

步骤三：启动 Hadoop

[hadoop@master ~]$ start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out
192.168.88.201: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out
192.168.88.200: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out
192.168.88.200: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out
192.168.88.201: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out

步骤四：关闭 Hadoop

[hadoop@master hadoop]$ stop-all.sh 
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [master]
master: stopping namenode
192.168.88.201: stopping datanode
192.168.88.200: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
192.168.88.200: stopping nodemanager
192.168.88.201: stopping nodemanager
no proxyserver to stop

实验三、通过命令监控大数据平台资源状态

1、通过命令查看 YARN 状态

步骤一：确认切换到目录 /usr/local/src/hadoop

[hadoop@master hadoop]$ cd /usr/local/src/hadoop/

步骤二：返回主机界面在 Master 主机上执行 start-all.sh

#master 节点启动 zookeeper
[hadoop@master hadoop]$ zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

#slave1 节点启动 zookeeper
[root@slave1 ~]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

#slave2 节点启动 zookeeper
[root@slave2 ~]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

master节点
[hadoop@master hadoop]$ start-all.sh

步骤三：执行 JPS 命令，发现 Master 上有 NodeManager 进程和 ResourceManager 进程，则 YARN 启动完成。

[hadoop@master hadoop]$ jps

执行结果如下，说明 YARN 已启动。

[hadoop@master hadoop]$ jps
2642 QuorumPeerMain
2994 SecondaryNameNode
3154 ResourceManager
3413 Jps
2795 NameNode

2、通过命令查看 HDFS 状态

步骤一：目录操作

切换到 hadoop 目录，执行 cd /usr/local/src/hadoop 命令

[hadoop@master hadoop]$ cd /usr/local/src/hadoop/

查看 HDFS 目录

[hadoop@master hadoop]$  ./bin/hdfs dfs -ls /
Found 5 items
drwxr-xr-x   - hadoop supergroup          0 2023-04-07 15:20 /hbase
drwxr-xr-x   - hadoop supergroup          0 2023-03-17 17:33 /input
drwxr-xr-x   - hadoop supergroup          0 2023-03-17 18:45 /output
drwx------   - hadoop supergroup          0 2023-04-21 16:02 /tmp
drwxr-xr-x   - hadoop supergroup          0 2023-04-14 20:48 /user

步骤二：查看 HDSF 的报告，执行命令： bin/hdfs dfsadmin -report

[hadoop@master hadoop]$ bin/hdfs dfsadmin -report
Configured Capacity: 107321753600 (99.95 GB)
Present Capacity: 102855995392 (95.79 GB)
DFS Remaining: 102852079616 (95.79 GB)
DFS Used: 3915776 (3.73 MB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (2):

Name: 192.168.88.201:50010 (slave2)
Hostname: slave2
Decommission Status : Normal
Configured Capacity: 53660876800 (49.98 GB)
DFS Used: 1957888 (1.87 MB)
Non DFS Used: 2232373248 (2.08 GB)
DFS Remaining: 51426545664 (47.89 GB)
DFS Used%: 0.00%
DFS Remaining%: 95.84%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Apr 23 21:56:46 CST 2023


Name: 192.168.88.200:50010 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 53660876800 (49.98 GB)
DFS Used: 1957888 (1.87 MB)
Non DFS Used: 2233384960 (2.08 GB)
DFS Remaining: 51425533952 (47.89 GB)
DFS Used%: 0.00%
DFS Remaining%: 95.83%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Apr 23 21:56:46 CST 2023

步骤三：查看 HDFS 空间情况，执行命令：hdfs dfs -df

[hadoop@master hadoop]$ hdfs dfs -df /
Filesystem                  Size     Used     Available  Use%
hdfs://master:9000  107321753600  3915776  102852079616    0%

3、通过命令查看 HBase 状态

步骤一：启动运行 HBase

切换到 HBase 安装目录/usr/local/src/hbase，命令如下：

[hadoop@master hadoop]$ cd /usr/local/src/hbase/
[hadoop@master hbase]$ hbase version
HBase 1.2.1
Source code repository git://asf-dev/home/busbey/projects/hbase revision=8d8a7107dc4ccbf36a92f64675dc60392f85c015
Compiled by busbey on Wed Mar 30 11:19:21 CDT 2016
From source with checksum f4bb4a14bb4e0b72b46f729dae98a772

结果显示 HBase1.2.1，说明 HBase 正在运行，版本号为 1.2.1。

如果没有启动，则执行命令 start-hbase.sh 启动 HBase。
[hadoop@master hbase]$ start-hbase.sh 
master: starting zookeeper, logging to /usr/local/src/hbase/logs/hbase-hadoop-zookeeper-master.example.com.out
slave1: starting zookeeper, logging to /usr/local/src/hbase/logs/hbase-hadoop-zookeeper-slave1.example.com.out
slave2: starting zookeeper, logging to /usr/local/src/hbase/logs/hbase-hadoop-zookeeper-slave2.example.com.out
starting master, logging to /usr/local/src/hbase/logs/hbase-hadoop-master-master.example.com.out
slave2: starting regionserver, logging to /usr/local/src/hbase/logs/hbase-hadoop-regionserver-slave2.example.com.out
slave1: starting regionserver, logging to /usr/local/src/hbase/logs/hbase-hadoop-regionserver-slave1.example.com.out

步骤二：查看 HBase 版本信息

执行命令hbase shell，进入HBase命令交互界面

[hadoop@master hbase]$ hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.1, r8d8a7107dc4ccbf36a92f64675dc60392f85c015, Wed Mar 30 11:19:21 CDT 2016

hbase(main):001:0>

输入 version，查询 HBase 版本

hbase(main):001:0> version
1.2.1, r8d8a7107dc4ccbf36a92f64675dc60392f85c015, Wed Mar 30 11:19:21 CDT 2016

结果显示 HBase 版本为 1.2.1。

步骤三：查询 HBase 状态，在 HBase 命令交互界面，执行 status 命令

hbase(main):002:0> status
1 active master, 0 backup masters, 2 servers, 0 dead, 1.0000 average load

查询结果显示，1 台活动 master，0 台备份 masters，共 3 台服务主机，平均加载时间为 0.6667 秒。

我们还可以“简单”查询 HBase 的状态，执行命令 status 'simple'

hbase(main):003:0> status 'simple'
active master:  master:16000 1682258321433
0 backup masters
2 live servers
    slave1:16020 1682258322908
        requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=19, maxHeapMB=440, numberOfStores=2, numberOfStorefiles=0, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=7, writeRequestsCount=4, rootIndexSizeKB=0 totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[MultiRowMutationEndpoint]
    slave2:16020 1682258322896
        requestsPerSecond=0.0, numberOfOnlineRegions=0, usedHeapMB=11, maxHeapMB=440, numberOfStores=0, numberOfStorefiles=0, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
0 dead servers
Aggregate load: 0, regions: 2

显示更多的关于 Master、Slave1 和 Slave2 主机的服务端口、请求时间等详细信息。

如果需要查询更多关于 HBase 状态，执行命令 help 'status'

hbase(main):004:0> help 'status'
Show cluster status. Can be 'summary', 'simple', 'detailed', or 'replication'. The
default is 'summary'. Examples:

  hbase> status
  hbase> status 'simple'
  hbase> status 'summary'
  hbase> status 'detailed'
  hbase> status 'replication'
  hbase> status 'replication', 'source'
  hbase> status 'replication', 'sink'

hbase(main):005:0> quit
[hadoop@master hbase]$

结果显示出所有关于 status 的命令。

步骤四停止 HBase 服务

停止 HBase 服务，则执行命令 stop-hbase.sh。

[hadoop@master hbase]$ stop-hbase.sh 
stopping hbase.................
slave1: no zookeeper to stop because no pid file /tmp/hbase-hadoop-zookeeper.pid
slave2: no zookeeper to stop because no pid file /tmp/hbase-hadoop-zookeeper.pid
master: no zookeeper to stop because no pid file /tmp/hbase-hadoop-zookeeper.pid

没有错误提示，显示$提示符时，即停止了 HBase 服务。

4、通过命令查看 Hive 状态

步骤一：启动 Hive

切换到/usr/local/src/hive 目录，输入 hive，回车。

[hadoop@master hbase]$ cd /usr/local/src/hive/
[hadoop@master hive]$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/usr/local/src/hive/lib/hive-common-2.0.0.jar!/hive-log4j2.properties
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>

当显示 hive>时，表示启动成功，进入到了 Hive shell 状态。

步骤二：Hive 操作基本命令

注意：Hive 命令行语句后面一定要加分号。

（1）查看数据库

hive> show databases;
OK
default
sample
Time taken: 0.973 seconds, Fetched: 2 row(s)

显示默认的数据库 default。

（2）查看 default 数据库所有表

hive> use default;
OK
Time taken: 0.02 seconds
hive> show tables;
OK
test
Time taken: 2.201 seconds, Fetched: 1 row(s)

显示 default 数据中没有任何表。

（3）创建表 stu，表的 id 为整数型，name 为字符型

hive> create table stu(id int,name string);
OK
Time taken: 0.432 seconds

（4）为表 stu 插入一条信息，id 号为 001，name 为liuyaling

hive> insert into stu values (001,"liuyaling");
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = hadoop_20230423222915_a95e9891-fdf5-4739-a63e-fcadecc85e28
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1682258121749_0001, Tracking URL = http://master:8088/proxy/application_1682258121749_0001/
Kill Command = /usr/local/src/hadoop/bin/hadoop job  -kill job_1682258121749_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2023-04-23 22:31:36,420 Stage-1 map = 0%,  reduce = 0%
2023-04-23 22:31:43,892 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.72 sec
MapReduce Total cumulative CPU time: 2 seconds 720 msec
Ended Job = job_1682258121749_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://master:9000/user/hive/warehouse/stu/.hive-staging_hive_2023-04-23_22-30-32_985_529079703757687911-1/-ext-10000
Loading data to table default.stu
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1   Cumulative CPU: 2.72 sec   HDFS Read: 4135 HDFS Write: 79 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 720 msec
OK
Time taken: 72.401 seconds

按照以上操作，继续插入两条信息：id 和 name 分别为 1002、1003 和 yanhaoxiang、tnt。

（5）插入数据后查看表的信息

hive> show tables;
OK
stu
test
values__tmp__table__1
values__tmp__table__2
Time taken: 0.026 seconds, Fetched: 4 row(s)

（6）查看表 stu 结构

hive> desc stu;
OK
id                  	int                 	                    
name                	string              	                    
Time taken: 0.041 seconds, Fetched: 2 row(s)

（7）查看表 stu 的内容

hive> select * from stu;
OK
1	liuyaling
1002	yanhaoxiang
1002	yanhaoxiang
1003	tnt
Time taken: 0.118 seconds, Fetched: 4 row(s)

步骤三：通过 Hive 命令行界面查看文件系统和历史命令

（1）查看本地文件系统，执行命令 ! ls /usr/local/src;

hive> ! ls /usr/local/src;
flume
hadoop
hbase
hive
jdk
sqoop
zookeeper

（2）查看 HDFS 文件系统，执行命令 dfs -ls /;

hive> dfs -ls /;
Found 5 items
drwxr-xr-x   - hadoop supergroup          0 2023-04-23 21:58 /hbase
drwxr-xr-x   - hadoop supergroup          0 2023-03-17 17:33 /input
drwxr-xr-x   - hadoop supergroup          0 2023-03-17 18:45 /output
drwx------   - hadoop supergroup          0 2023-04-21 16:02 /tmp
drwxr-xr-x   - hadoop supergroup          0 2023-04-14 20:48 /user

hive> exit;

（3）查看在 Hive 中输入的所有历史命令

进入到当前用户 Hadoop 的目录/home/hadoop，查看.hivehistory 文件。

[hadoop@master hive]$ cd /home/hadoop/
[hadoop@master ~]$ cat .hivehistory 
show databases;
create database sample;
show databases;
use sample;
create table student(number string,name string);
exit
show databases;
exit;
use sample;
select * from student;
sqoop export --connect "jdbc:mysql://master:3306/sample?useUnicode=true&characterEncoding=utf-8" --username root --password Password@123! --table student --input-fields-terminated-by '|' --export-dir /user/hive/warehouse/sample.db/student/*
sqoop export --connect "jdbc:mysql://master:3306/sample?useUnicode=true&characterEncoding=utf-8" --username root --password Password@123! --table student --input-fields-terminated-by '|' --export-dir /user/hive/warehouse/sample.db/student/*;
sqoop export --connect "jdbc:mysql://master3306/sample?useUnicode=true&characterEncoding=utf-8" --username root --password Password@123! --table student --input-fields-terminated-by '|' --export-dir /user/hive/warehouse/sample.db/student/*
sqoop export --connect "jdbc:mysql://master3306/sample?useUnicode=true&characterEncoding=utf-8" --username root --password Password@123! --table student --input-fields-terminated-by '|' --export-dir /user/hive/warehouse/sample.db/student/*;
sqoop export --connect "jdbc:mysql://master3306/sample?useUnicode=true&characterEncoding=utf-8" --username root --password Password@123! --table student --input-fields-terminated-by '|' --export-dir /user/hive/warehouse/sample.db/student/*
exit
show databases;
use sample;
show tables;
use default;
show tables;
quit
'
select*from default.test;
quit;
show databases;
use default;
show tables;
insert into stu values (001,"liuyaling");
create table stu(id int,name string);
insert into stu values (001,"liuyaling");
insert into stu values (1002,"yanhaoxiang");
hive
insert into stu values (1003,"tnt");
quit
exit
exit;
quit;
show tables;
insert into stu values (1002,"yanhaoxiang");
insert into stu values (1003,"tnt");
show tables;
desc stu;
select * from stu;
! ls /usr/local/src;
dfs -ls /;
exit;

结果显示，之前在 Hive 命令行界面下运行的所有命令（含错误命令）都显示了出来，有助于维护、故障排查等工作。

实验四、通过命令监控大数据平台服务状态

1、通过命令查看 ZooKeeper 状态

步骤一: 查看 ZooKeeper 状态，执行命令 zkServer.sh status，结果显示如下

[hadoop@master ~]$ zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Mode: follower

以上结果中，Mode:follower 表示为 ZooKeeper 的跟随者。

步骤二: 查看运行进程

QuorumPeerMain：QuorumPeerMain 是 ZooKeeper 集群的启动入口类，是用来加载配置启动 QuorumPeer 线程的。

执行命令 jps 以查看进程情况。

[hadoop@master ~]$ jps
2642 QuorumPeerMain
2994 SecondaryNameNode
3154 ResourceManager
5400 Jps
2795 NameNode

此时 QuorumPeerMain 进程已启动。

步骤四: 在成功启动 ZooKeeper 服务后，输入命令 zkCli.sh，连接到 ZooKeeper 服务。

[hadoop@master ~]$ zkCli.sh 
Connecting to localhost:2181
2023-04-23 22:39:17,093 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT
2023-04-23 22:39:17,096 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=master
2023-04-23 22:39:17,096 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_152
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/local/src/jdk/jre
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/usr/local/src/zookeeper/bin/../build/classes:/usr/local/src/zookeeper/bin/../build/lib/*.jar:/usr/local/src/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/src/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/src/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/local/src/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/local/src/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/local/src/zookeeper/bin/../zookeeper-3.4.8.jar:/usr/local/src/zookeeper/bin/../src/java/lib/*.jar:/usr/local/src/zookeeper/bin/../conf:/usr/local/src/sqoop/lib:
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-862.el7.x86_64
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=hadoop
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/hadoop
2023-04-23 22:39:17,097 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/hadoop
2023-04-23 22:39:17,099 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@69d0a921
Welcome to ZooKeeper!
2023-04-23 22:39:17,122 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2023-04-23 22:39:17,208 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@876] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2023-04-23 22:39:17,223 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x187ae88b8390000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]

步骤五: 使用 Watch 监听/hbase 目录，一旦/hbase 内容有变化，将会有提示。打开监视，执行命令 get /hbase 1。

[zk: localhost:2181(CONNECTED) 0] get /hbase 1

cZxid = 0x500000002
ctime = Sun Apr 23 21:58:42 CST 2023
mZxid = 0x500000002
mtime = Sun Apr 23 21:58:42 CST 2023
pZxid = 0x500000062
cversion = 18
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 14

[zk: localhost:2181(CONNECTED) 1] quit
Quitting...
2023-04-23 22:40:16,816 [myid:] - INFO  [main:ZooKeeper@684] - Session: 0x187ae88b8390001 closed
2023-04-23 22:40:16,817 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x187ae88b8390001
[hadoop@master ~]$

结果显示，当执行命令 set /hbase value-update 后，数据版本由 0 变成 1，说明 /hbase 处于监控中。

2、通过命令查看 Sqoop 状态

步骤一: 查询 Sqoop 版本号，验证 Sqoop 是否启动成功。

首先切换到/usr/local/src/sqoop 目录，执行命令：./bin/sqoop-version

[hadoop@master ~]$ cd /usr/local/src/sqoop/
[hadoop@master sqoop]$  ./bin/sqoop-version 
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
23/04/23 22:40:59 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017

结果显示：Sqoop 1.4.7，说明 Sqoop 版本号为 1.4.7，并启动成功。

步骤二: 测试 Sqoop 是否能够成功连接数据库

切换到 Sqoop 的目录，执行命令 bin/sqoop list-databases --connect jdbc:mysql://master:3306/ --username root --password Password123$，命令中 “master:3306”为数据库主机名和端口。

[hadoop@master sqoop]$  bin/sqoop list-databases --connect jdbc:mysql://master:3306/ --username root --password Password@123!
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
23/04/23 22:42:16 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
23/04/23 22:42:16 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
23/04/23 22:42:16 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Sun Apr 23 22:42:16 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
hive
mysql
performance_schema
sample
sys

结果显示，可以连接到 MySQL，并查看到 Master 主机中 MySQL 的所有库实例，如 information_schema、hive、mysql、performance_schema 和 sys 等数据库。

步骤三: 执行命令 sqoop help，可以看到如下内容，代表 Sqoop 启动成功。

[hadoop@master sqoop]$ sqoop help
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
23/04/23 22:42:37 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.

结果显示了 Sqoop 的常用命令和功能，如下表所示。

序号	命令	功能
1	import	将数据导入到集群
2	export	将集群数据导出
3	codegen	生成与数据库记录交互的代码
4	create-hivetable	创建 Hive 表
5	eval	查看 SQL 执行结果
6	import-all-tables	导入某个数据库下所有表到 HDFS 中
7	job	生成一个 job
8	list-databases	列出所有数据库名
9	list-tables	列出某个数据库下所有的表
10	merge	将 HDFS 中不同目录下数据合在一起，并存放在指定的目录中
11	metastore	记录 Sqoop job 的元数据信息，如果不启动 metastore 实例，则默认的元数据存储目录为：~/.sqoop
12	help	打印 Sqoop 帮助信息
13	version	打印 Sqoop 版本信息

3、通过命令查看 Flume 状态

步骤一:检查 Flume 安装是否成功，执行 flume-ng version 命令，查看 Flume 的版本。

[hadoop@master sqoop]$ cd /usr/local/src/flume/
[hadoop@master flume]$ flume-ng version
Flume 1.6.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080
Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015
From source with checksum b29e416802ce9ece3269d34233baf43f
[hadoop@master flume]$

步骤二:添加 example.conf 到/usr/local/src/flume

[hadoop@master flume]$ vim /usr/local/src/flume/example.conf
a1.sources=r1
a1.sinks=k1
a1.channels=c1
a1.sources.r1.type=spooldir
a1.sources.r1.spoolDir=/usr/local/src/flume/
a1.sources.r1.fileHeader=true
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://master:9000/flume
a1.sinks.k1.hdfs.rollsize=1048760
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.rollInterval=900
a1.sinks.k1.hdfs.useLocalTimeStamp=true
a1.channels.c1.type=file
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

步骤三:启动 Flume Agent a1 日志控制台

[hadoop@master flume]$ flume-ng agent --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hadoop libraries found via (/usr/local/src/hadoop/bin/hadoop) for HDFS access
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath
Info: Including HBASE libraries found via (/usr/local/src/hbase/bin/hbase) for HBASE access
Info: Excluding /usr/local/src/hbase/lib/slf4j-api-1.7.7.jar from classpath
Info: Excluding /usr/local/src/hbase/lib/slf4j-log4j12-1.7.5.jar from classpathxxxxxxxxxx [hadoop@master flume]$ flume-ng agent --conf-file example.conf --name a1 -Dflume.root.logger=INFO,consoleWarning: No configuration directory set! Use --conf <dir> to override.Info: Including Hadoop libraries found via (/usr/local/src/hadoop/bin/hadoop) for HDFS accessInfo: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpathInfo: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpathInfo: Including HBASE libraries found via (/usr/local/src/hbase/bin/hbase) for HBASE accessInfo: Excluding /usr/local/src/hbase/lib/slf4j-api-1.7.7.jar from classpathInfo: Excluding /usr/local/src/hbase/lib/slf4j-log4j12-1.7.5.jar from classpath[hadoop@master flume]$  /usr/local/src/flume/bin/flume-ng agent --conf ./conf --conf-file ./example.conf  --name a1 -Dflume.root.logger=INFO,consoleInfo: Sourcing environment configuration script /usr/local/src/flume/conf/flume-env.shInfo: Including Hadoop libraries found via (/usr/local/src/hadoop/bin/hadoop) for HDFS access.../lib/native:/usr/local/src/hadoop/lib/native org.apache.flume.node.Application --conf-file ./example.conf --name a1/usr/local/src/flume/bin/flume-ng: line 241: /usr/loocal/src/jdk/bin/java: No such file or directory
...
23/04/23 22:52:56 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: k1. sink.connection.failed.count == 0
23/04/23 22:52:56 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: k1. sink.event.drain.attempt == 1918
23/04/23 22:52:56 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: k1. sink.event.drain.sucess == 1918
[hadoop@master flume]$ ^C

步骤四: 查看结果文章来源地址https://www.toymoban.com/news/detail-423686.html

[hadoop@master flume]$ hdfs dfs -lsr /flume
lsr: DEPRECATED: Please use 'ls -R' instead.
-rw-r--r--   2 hadoop supergroup       1499 2023-04-23 22:52 /flume/FlumeData.1682261559722
-rw-r--r--   2 hadoop supergroup       1419 2023-04-23 22:52 /flume/FlumeData.1682261559723
-rw-r--r--   2 hadoop supergroup       1468 2023-04-23 22:52 /flume/FlumeData.1682261559724
...
-rw-r--r--   2 hadoop supergroup       1795 2023-04-23 22:52 /flume/FlumeData.1682261559817
-rw-r--r--   2 hadoop supergroup       1841 2023-04-23 22:52 /flume/FlumeData.1682261559818
-rw-r--r--   2 hadoop supergroup       1665 2023-04-23 22:52 /flume/FlumeData.1682261559819
-rw-r--r--   2 hadoop supergroup       1439 2023-04-23 22:52 /flume/FlumeData.1682261559820

到了这里，关于flume组件以及通过命令监控大数据平台转态的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！