概述
目标:
- 1.实现kyuubi spark on yarn
- 2.实现 kyuubi spark on yarn 资源的动态分配
注意:版本 kyuubi 1.8.0 、 spark 3.4.2 、hadoop 3.3.6
前置准备请看如下文章
文章 | 链接 |
---|---|
hadoop一主三从安装 | 链接 |
spark on yarn | 链接 |
实践
下载
官网下载地址
配置
官方文档
修改配置文件
# 需要修改的配置文件
[root@hadoop01 conf]# ls
kyuubi-defaults.conf.template kyuubi-env.sh.template log4j2.xml.template
[root@hadoop01 conf]# pwd
/data/hadoop/soft/apache-kyuubi-1.8.0/conf
# 第一步: 修改日志文件
[root@hadoop01 conf]# mv log4j2.xml.template log4j2.xml
# 第二步: kyuubi-env.sh
# export SPARK_HOME=/data/hadoop/soft/spark-3.4.2
[root@hadoop01 conf]# ls
kyuubi-defaults.conf.template kyuubi-env.sh.template log4j2.xml
[root@hadoop01 conf]# mv kyuubi-env.sh.template kyuubi-env.sh
[root@hadoop01 conf]# vi kyuubi-env.sh
# 第三步: kyuubi-defaults.conf
[root@hadoop01 conf]# mv kyuubi-defaults.conf.template kyuubi-defaults.conf
[root@hadoop01 conf]# vi kyuubi-defaults.conf
kyuubi.engine.type SPARK_SQL
kyuubi.engine.share.level USER
# ============ Spark Config ============
spark.master=yarn
spark.submit.deployMode=cluster
spark.driver.memory 4g
spark.executor.memory 4g
spark.executor.cores 2
#Dynamic allocation
spark.dynamicAllocation.enabled=true
##false if prefer shuffle tracking than ESS
spark.shuffle.service.enabled=true
spark.dynamicAllocation.initialExecutors=2
spark.dynamicAllocation.minExecutors=6
spark.dynamicAllocation.maxExecutors=6
spark.dynamicAllocation.executorAllocationRatio=0.5
spark.dynamicAllocation.executorIdleTimeout=200s
spark.dynamicAllocation.cachedExecutorIdleTimeout=30min
# true if prefer shuffle tracking than ESS
spark.dynamicAllocation.shuffleTracking.enabled=false
spark.dynamicAllocation.shuffleTracking.timeout=30min
spark.dynamicAllocation.schedulerBacklogTimeout=1s
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=1s
spark.cleaner.periodicGC.interval=5min
# aqe
spark.sql.adaptive.enabled=true
spark.sql.adaptive.forceApply=false
spark.sql.adaptive.logLevel=info
spark.sql.adaptive.advisoryPartitionSizeInBytes=128m
spark.sql.adaptive.coalescePartitions.enabled=true
spark.sql.adaptive.coalescePartitions.minPartitionNum=30
spark.sql.adaptive.coalescePartitions.initialPartitionNum=5120
spark.sql.adaptive.fetchShuffleBlocksInBatch=true
spark.sql.adaptive.localShuffleReader.enabled=true
spark.sql.adaptive.skewJoin.enabled=true
spark.sql.adaptive.skewJoin.skewedPartitionFactor=3
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes=256m
spark.sql.adaptive.nonEmptyPartitionRatioForBroadcastJoin=0.2
spark.sql.adaptive.optimizer.excludedRules
spark.sql.autoBroadcastJoinThreshold=-1
# spark 返回数据大小设定
spark.kryoserializer.buffer.max=2047m
spark.driver.maxResultSize=4096m
# paimon
#spark.sql.catalog.hive.warehouse=hdfs:///data/paimon
spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.paimon.warehouse=hdfs:///data/paimon
spark.sql.extensions=org.apache.paimon.spark.PaimonSparkSessionExtension
# Log into Spark History Server
spark.eventLog.enabled true
spark.eventLog.dir hdfs://hadoop01:9000/spark-eventlog
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.eventLog.compress true
#将 spark 历史服务器与 yarn 历史服务器进行整合,写到 HDFS 中
spark.yarn.historyServer.address hadoop01:18080
# spark UI参数
spark.ui.retainedJobs 50
spark.ui.retainedStages 300
启动
[root@hadoop01 apache-kyuubi-1.8.0]# ./bin/kyuubi start
bin/beeline -u 'jdbc:hive2://10.32.36.142:10009/' -n root
24/02/26 15:47:30 INFO Client: Application report for application_1708505130791_0007 (state: RUNNING)
24/02/26 15:47:31 INFO Client: Application report for application_1708505130791_0007 (state: RUNNING)
24/02/26 15:47:32 INFO Client: Application report for application_1708505130791_0007 (state: RUNNING)
24/02/26 15:47:33 INFO Client: Application report for application_1708505130791_0007 (state: RUNNING)
2024-02-26 15:47:35.793 INFO KyuubiSessionManager-exec-pool: Thread-58 org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Get service instance:hadoop04:39670 engine id:application_1708505130791_0007 and version:1.8.0 under /kyuubi_1.8.0_USER_SPARK_SQL/root/default
24/02/26 15:47:35 INFO Client: Application report for application_1708505130791_0007 (state: RUNNING)
Connected to: Spark SQL (version 3.4.2)
Driver: Kyuubi Project Hive JDBC Client (version 1.8.0)
Beeline version 1.8.0 by Apache Kyuubi
0: jdbc:hive2://10.32.36.142:10009/>
0: jdbc:hive2://10.32.36.142:10009/> show databases;
2024-02-26 15:48:03.130 INFO KyuubiSessionManager-exec-pool: Thread-66 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[81c88029-474d-4ed2-98fc-21013bf10cc3]: PENDING_STATE -> RUNNING_STATE, statement:
show databases
2024-02-26 15:48:03.261 INFO KyuubiSessionManager-exec-pool: Thread-66 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[81c88029-474d-4ed2-98fc-21013bf10cc3]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.131 seconds
+------------+
| namespace |
+------------+
| default |
+------------+
1 row selected (0.34 seconds)
0: jdbc:hive2://10.32.36.142:10009/>
set spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog
set spark.sql.catalog.paimon.warehouse=hdfs:///data/paimon;
24/02/26 20:30:08 INFO ExecuteStatement: Execute in full collect mode
24/02/26 20:30:08 INFO V2ScanRelationPushDown:
Pushing operators to trace_log_refdes_hive_ro
Pushed Filters: IsNotNull(id), EqualTo(id,11C0928D5A29E048E063AA2C200ABEF3)
Post-Scan Filters: isnotnull(id#354),(id#354 = 11C0928D5A29E048E063AA2C200ABEF3)
24/02/26 20:30:08 INFO V2ScanRelationPushDown:
Output: pcbid#345, rid#346, refdes#347, bm_circuit_no#348, timestamp#349, pickupstatus#350, serial_number#351, flag#352, kitid#353, id#354, createdate#355, etl#356, opt1#357, opt2#358, opt3#359, opt4#360, opt5#361, nozzleid#362, laneno#363, componentbarcode#364, pn#365, lotcode#366, datecode#367, verdor#368, workorder#369, dt#370
24/02/26 20:30:08 INFO SparkContext: Starting job: collect at ExecuteStatement.scala:72
24/02/26 20:30:08 INFO SQLOperationListener: Query [4a52f7cf-f446-40b0-b16f-6b895ab84224]: Job 5 started with 1 stages, 1 active jobs running
24/02/26 20:30:08 INFO SQLOperationListener: Query [4a52f7cf-f446-40b0-b16f-6b895ab84224]: Stage 6.0 started with 1 tasks, 1 active stages running
三千五百万
主键id单笔获取
0: jdbc:hive2://10.32.36.142:10009/> select count(*) from trace_log_refdes_hive_ro where id='11C0928D5A26E048E063AA2C200ABEF3';
24/02/27 08:43:33 INFO AdaptiveSparkPlanExec: Final plan:
*(1) HashAggregate(keys=[], functions=[count(1)], output=[count(1)#523L])
+- *(1) HashAggregate(keys=[], functions=[partial_count(1)], output=[count#527L])
+- *(1) Project
+- *(1) Filter (isnotnull(id#505) AND (id#505 = 11C0928D5A26E048E063AA2C200ABEF3))
+- BatchScan trace_log_refdes_hive_ro[id#505] PaimonScan: [trace_log_refdes_hive_ro], PushedFilters: [IsNotNull(id),Equal(id, 11C0928D5A26E048E063AA2C200ABEF3)] RuntimeFilters: []
24/02/27 08:43:33 INFO ExecuteStatement: Processing root's query[b4889a64-d943-41ae-819e-d8a0491c60c6]: RUNNING_STATE -> FINISHED_STATE, time taken: 1.871 seconds
24/02/27 08:43:33 INFO SQLOperationListener: Query [b4889a64-d943-41ae-819e-d8a0491c60c6]: Job 7 succeeded, 0 active jobs running
2024-02-27 08:43:33.005 INFO KyuubiSessionManager-exec-pool: Thread-496 org.apache.kyuubi.operation.ExecuteStatement: Query[b4889a64-d943-41ae-819e-d8a0491c60c6] in FINISHED_STATE
2024-02-27 08:43:33.006 INFO KyuubiSessionManager-exec-pool: Thread-496 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[b4889a64-d943-41ae-819e-d8a0491c60c6]: RUNNING_STATE -> FINISHED_STATE, time taken: 1.874 seconds
+-----------+
| count(1) |
+-----------+
| 1 |
+-----------+
1 row selected (1.883 seconds)
非主键 count 测试
0: jdbc:hive2://10.32.36.142:10009/> select count(*) from trace_log_refdes_hive_ro where pcbid='E23MPM42201540';
24/02/27 08:41:56 INFO AdaptiveSparkPlanExec: Final plan:
*(2) HashAggregate(keys=[], functions=[count(1)], output=[count(1)#489L])
+- ShuffleQueryStage 0
+- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=109]
+- *(1) HashAggregate(keys=[], functions=[partial_count(1)], output=[count#493L])
+- *(1) Project
+- *(1) Filter (isnotnull(pcbid#462) AND (pcbid#462 = E23MPM42201540))
+- BatchScan trace_log_refdes_hive_ro[pcbid#462] PaimonScan: [trace_log_refdes_hive_ro], PushedFilters: [IsNotNull(pcbid),Equal(pcbid, E23MPM42201540)] RuntimeFilters: []
2024-02-27 08:41:56.274 INFO KyuubiSessionManager-exec-pool: Thread-495 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[f473dde1-ba80-4d3e-8dc7-fa0a3e32df1f]: RUNNING_STATE -> FINISHED_STATE, time taken: 5.368 seconds
+-----------+
| count(1) |
+-----------+
| 108 |
+-----------+
1 row selected (5.393 seconds)
差不多6千万文章来源:https://www.toymoban.com/news/detail-844724.html
24/02/27 16:54:54 INFO SQLOperationListener: Query [96e1d2f4-8230-4cc7-9b84-ce888387bb7d]: Job 2 succeeded, 0 active jobs running
24/02/27 16:54:54 INFO AdaptiveSparkPlanExec: Final plan:
*(2) HashAggregate(keys=[], functions=[count(1)], output=[count(1)#61L])
+- ShuffleQueryStage 0
+- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=52]
+- *(1) HashAggregate(keys=[], functions=[partial_count(1)], output=[count#64L])
+- *(1) Project
+- BatchScan trace_log_refdes_hive_ro[] PaimonScan: [trace_log_refdes_hive_ro] RuntimeFilters: []
24/02/27 16:54:54 INFO CodeGenerator: Code generated in 7.970386 ms
24/02/27 16:54:54 INFO ExecuteStatement: Processing root's query[96e1d2f4-8230-4cc7-9b84-ce888387bb7d]: RUNNING_STATE -> FINISHED_STATE, time taken: 21.921 seconds
2024-02-27 16:54:54.490 INFO KyuubiSessionManager-exec-pool: Thread-806 org.apache.kyuubi.operation.ExecuteStatement: Query[96e1d2f4-8230-4cc7-9b84-ce888387bb7d] in FINISHED_STATE
2024-02-27 16:54:54.490 INFO KyuubiSessionManager-exec-pool: Thread-806 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[96e1d2f4-8230-4cc7-9b84-ce888387bb7d]: RUNNING_STATE -> FINISHED_STATE, time taken: 21.924 seconds
+-----------+
| count(1) |
+-----------+
| 59480487 |
+-----------+
1 row selected (21.942 seconds)
0: jdbc:hive2://10.32.36.142:10009/> select * from trace_log_refdes_hive_ro limit 10;
24/02/27 16:55:27 INFO ExecuteStatement: Execute in full collect mode
24/02/27 16:55:27 INFO V2ScanRelationPushDown:
Output: pcbid#67, rid#68, refdes#69, bm_circuit_no#70, timestamp#71, pickupstatus#72, serial_number#73, flag#74, kitid#75, id#76, createdate#77, etl#78, opt1#79, opt2#80, opt3#81, opt4#82, opt5#83, nozzleid#84, laneno#85, componentbarcode#86, pn#87, lotcode#88, datecode#89, verdor#90, workorder#91, dt#92
2024-02-27 16:55:28.414 INFO KyuubiSessionManager-exec-pool: Thread-807 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[cadc193d-1343-4973-828c-23f23250b3d3]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.735 seconds
+-----------------+------------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
| pcbid | xxx | yyyyff | zzzzzxxxxxxxx | timestamp | gggggggfffff | ddddddddddddd | cccc | aaaaa | id | createdate | etl | opt1 | opt2 | opt3 | opt4 | opt5 | nozzleid | laneno | componentbarcode | pn | xyzzzzx | xzzzzzzz | txxxxx | zxxxxxxxx | dt |
+-----------------+------------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
| E23MPM42203175 | 514S00292-11420240109000060 | J0200 | 5 | 2024-02-23 18:16:42 | 0 | DLC4084004RPQVLAG | 0 | NXT | 11C0928D5A26E048E063AA2C200ABEF3 | 2024-02-23 18:14:50 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
| E23MPM42203175 | 514S00292-11420240109000060 | J0200 | 4 | 2024-02-23 18:16:42 | 0 | DLC4084004SPQVLAF | 0 | NXT | 11C0928D5A27E048E063AA2C200ABEF3 | 2024-02-23 18:14:50 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
| E23MPM42203175 | 514S00292-11420240117000057 | J0200 | 7 | 2024-02-23 18:16:42 | 0 | DLC4084004PPQVLAJ | 0 | NXT | 11C0928D5A28E048E063AA2C200ABEF3 | 2024-02-23 18:14:50 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
| E23MPM42203175 | 514S00292-11420240117000057 | J0200 | 9 | 2024-02-23 18:16:42 | 0 | DLC4084004MPQVLAL | 0 | NXT | 11C0928D5A29E048E063AA2C200ABEF3 | 2024-02-23 18:14:50 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
| E23MPM42203175 | 514S00292-11420240117000056 | J0200 | 12 | 2024-02-23 18:16:42 | 0 | DLC4084004VPQVLAC | 0 | NXT | 11C0928D5A2CE048E063AA2C200ABEF3 | 2024-02-23 18:14:50 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
| E23MPM42203176 | 514S00292-11420240117000056 | J0200 | 1 | 2024-02-23 18:16:42 | 0 | DLC4084005NPQVLAG | 0 | NXT | 11C0928D5A31E048E063AA2C200ABEF3 | 2024-02-23 18:14:50 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
| E23MPM42203176 | 514S00292-11420240117000059 | J0200 | 4 | 2024-02-23 18:16:42 | 0 | DLC4084005GPQVLAN | 0 | NXT | 11C0928D5A34E048E063AA2C200ABEF3 | 2024-02-23 18:14:50 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
| E23MPM42203176 | 514S00292-11420240116000073 | J0200 | 10 | 2024-02-23 18:16:42 | 0 | DLC4084005MPQVLAH | 0 | NXT | 11C0928D5A3AE048E063AA2C200ABEF3 | 2024-02-23 18:14:50 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
| E23MPM42201540 | 117S0158-A420240115001804 | R0514 | 7 | 2024-02-23 18:16:42 | 0 | DLC40860DTVPQVLAW | 0 | NXT | 11C0928D5A59E048E063AA2C200ABEF3 | 2024-02-23 18:14:51 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
| E23MPM42201540 | 117S0158-A420240115001804 | R0401 | 1 | 2024-02-23 18:16:42 | 0 | DLC40860DU4PQVLAJ | 0 | NXT | 11C0928D5A5DE048E063AA2C200ABEF3 | 2024-02-23 18:14:51 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
+-----------------+------------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
10 rows selected (0.777 seconds)
0: jdbc:hive2://10.32.36.142:10009/> select * from trace_log_refdes_hive_ro where id ='11C0928D5A5DE048E063AA2C200ABEF3';
24/02/27 16:55:49 INFO V2ScanRelationPushDown:
Pushing operators to trace_log_refdes_hive_ro
Pushed Filters: IsNotNull(id), EqualTo(id,11C0928D5A5DE048E063AA2C200ABEF3)
Post-Scan Filters: isnotnull(id#181),(id#181 = 11C0928D5A5DE048E063AA2C200ABEF3)
24/02/27 16:55:49 INFO V2ScanRelationPushDown:
Output: pcbid#172, rid#173, refdes#174, bm_circuit_no#175, timestamp#176, pickupstatus#177, serial_number#178, flag#179, kitid#180, id#181, createdate#182, etl#183, opt1#184, opt2#185, opt3#186, opt4#187, opt5#188, nozzleid#189, laneno#190, componentbarcode#191, pn#192, lotcode#193, datecode#194, verdor#195, workorder#196, dt#197
2024-02-27 16:55:52.191 INFO KyuubiSessionManager-exec-pool: Thread-808 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[1dd09511-efc5-4e64-ad2a-15d4b59fb553]: RUNNING_STATE -> FINISHED_STATE, time taken: 2.859 seconds
+-----------------+----------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
| pcbid | xxx | yyyyff | zzzzzxxxxxxxx | timestamp | gggggggfffff | ddddddddddddd | cccc | aaaaa | id | createdate | etl | opt1 | opt2 | opt3 | opt4 | opt5 | nozzleid | laneno | componentbarcode | pn | xyzzzzx | xzzzzzzz | txxxxx | zxxxxxxxx | dt |
+-----------------+----------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
| E23MPM42201540 | 117S0158-A420240115001804 | R0401 | 1 | 2024-02-23 18:16:42 | 0 | DLC40860DU4PQVLAJ | 0 | NXT | 11C0928D5A5DE048E063AA2C200ABEF3 | 2024-02-23 18:14:51 | N | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | 2024-02-23 |
+-----------------+----------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
1 row selected (2.875 seconds)
0: jdbc:hive2://10.32.36.142:10009/> select count(*) from trace_log_refdes_hive_ro where pcbid ='E23MPM42203176';
2024-02-27 16:57:01.984 INFO KyuubiSessionManager-exec-pool: Thread-809 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[92377a4f-2dad-4039-be2d-4762946eaea8]: PENDING_STATE -> RUNNING_STATE, statement:
select count(*) from trace_log_refdes_hive_ro where pcbid ='E23MPM42203176'
24/02/27 16:57:01 INFO ExecuteStatement: Processing root's query[92377a4f-2dad-
24/02/27 16:57:10 INFO SparkContext: Starting job: collect at ExecuteStatement.scala:72
24/02/27 16:57:10 INFO SQLOperationListener: Query [92377a4f-2dad-4039-be2d-4762946eaea8]: Job 6 started with 2 stages, 1 active jobs running
24/02/27 16:57:10 INFO SQLOperationListener: Query [92377a4f-2dad-4039-be2d-4762946eaea8]: Stage 8.0 started with 1 tasks, 1 active stages running
24/02/27 16:57:10 INFO SQLOperationListener: Finished stage: Stage(8, 0); Name: 'collect at ExecuteStatement.scala:72'; Status: succeeded; numTasks: 1; Took: 47 msec
24/02/27 16:57:10 INFO AdaptiveSparkPlanExec: Final plan:
*(2) HashAggregate(keys=[], functions=[count(1)], output=[count(1)#304L])
+- ShuffleQueryStage 0
+- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=115]
+- *(1) HashAggregate(keys=[], functions=[partial_count(1)], output=[count#308L])
+- *(1) Project
+- *(1) Filter (isnotnull(pcbid#277) AND (pcbid#277 = E23MPM42203176))
+- BatchScan trace_log_refdes_hive_ro[pcbid#277] PaimonScan: [trace_log_refdes_hive_ro], PushedFilters: [IsNotNull(pcbid),Equal(pcbid, E23MPM42203176)] RuntimeFilters: []
2024-02-27 16:57:10.578 INFO KyuubiSessionManager-exec-pool: Thread-809 org.apache.kyuubi.operation.ExecuteStatement: Query[92377a4f-2dad-4039-be2d-4762946eaea8] in FINISHED_STATE
2024-02-27 16:57:10.578 INFO KyuubiSessionManager-exec-pool: Thread-809 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[92377a4f-2dad-4039-be2d-4762946eaea8]: RUNNING_STATE -> FINISHED_STATE, time taken: 8.594 seconds
+-----------+
| count(1) |
+-----------+
| 3 |
+-----------+
1 row selected (8.604 seconds)
0: jdbc:hive2://10.32.36.142:10009/>
结束
kyuubi整合spark on yarn 至此结束。文章来源地址https://www.toymoban.com/news/detail-844724.html
到了这里,关于kyuubi整合spark on yarn的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!