seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录

这篇具有很好参考价值的文章主要介绍了seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录。希望对大家有所帮助。如果存在错误或未考虑完全的地方,请大家不吝赐教,您也可以点击"举报违法"按钮提交疑问。


seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行报错解决,解决的报错如下:
1、java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
2、java.lang.NoClassDefFoundError: org/apache/thrift/TBase
3、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
4、java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
5、java.lang.OutOfMemoryError: Java heap space


1、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
        at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
        at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.MetaException
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
        at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
        at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 11 more

seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录,seatunnel系列集合,hive,hadoop,数据仓库,seatunnel,大数据

1.1 解决办法

是缺少对应的包,去hive里面的lib将包复制到seatunnel的包下面即可

hive-metastore-2.1.1-cdh6.3.2.jar # cdh版本的记得将这个也复杂上
hive-metastore.jar

2、java.lang.NoClassDefFoundError: org/apache/thrift/TBase

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/TBase
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
        at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
        at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
        at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
        at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TBase
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录,seatunnel系列集合,hive,hadoop,数据仓库,seatunnel,大数据

2.1 解决办法

是缺少对应的包,去hive里面的lib将包复制到seatunnel的包下面即可

libthrift-0.9.3-1.jar

3、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:48)
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
        at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
        at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
        at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
        at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 13 more

seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录,seatunnel系列集合,hive,hadoop,数据仓库,seatunnel,大数据

3.1 解决办法

是缺少对应的包,去hive里面的lib将包复制到seatunnel的包下面即可

hive-common-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-common.jar
hive-exec-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-exec.jar

4、java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface

Exception in thread "main" java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
        at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
        at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 25 more

seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录,seatunnel系列集合,hive,hadoop,数据仓库,seatunnel,大数据

4.1 解决办法

是缺少对应的包,这个我的hive里面的libthrift-0.9.3-1.jar包里面没有,所以包了这个错,去下面这个地址直接下载即可
下载地址:https://repo1.maven.org/maven2/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar

seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录,seatunnel系列集合,hive,hadoop,数据仓库,seatunnel,大数据

5、java.lang.OutOfMemoryError: Java heap space

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:379)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4129)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4115)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:563)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:303)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:219)
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
        at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
        at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)

seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录,seatunnel系列集合,hive,hadoop,数据仓库,seatunnel,大数据

5.1 解决办法

去seatunnel的config里面增加对应值,截图如下:
seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录,seatunnel系列集合,hive,hadoop,数据仓库,seatunnel,大数据
将值修改大一点就可以了
seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录,seatunnel系列集合,hive,hadoop,数据仓库,seatunnel,大数据

seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录,seatunnel系列集合,hive,hadoop,数据仓库,seatunnel,大数据
seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录,seatunnel系列集合,hive,hadoop,数据仓库,seatunnel,大数据文章来源地址https://www.toymoban.com/news/detail-517380.html

到了这里,关于seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行踩坑记录的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处: 如若内容造成侵权/违法违规/事实不符,请点击违法举报进行投诉反馈,一经查实,立即删除!

领支付宝红包 赞助服务器费用

相关文章

  • SeaTunnel 海量数据同步工具的使用(连载中……)

    SeaTunnel 是一个非常易用,高性能、支持实时流式和离线批处理的海量数据处理产品,前身是 WaterDrop (中文名:水滴),自 2021年10月12日更名为 SeaTunnel 。2021年12月9日,SeaTunnel 正式通过Apache 软件基金会的投票决议,以全票通过的优秀表现正式成为 Apache 孵化器项目。 2022年

    2024年01月16日
    浏览(41)
  • CDH集群hive初始化元数据库失败

    oracle数据库操作: 报错如下:命令 (Validate Hive Metastore schema (237)) 已失败 截图如下: 后台日志部分摘录: WARNING: Use “yarn jar” to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/jars/log4j-slf4j-im

    2024年04月11日
    浏览(48)
  • 使用Apache SeaTunnel进行数据库同步(MySQL to MySQL)

    Apache SeaTunnel 起到的主要作用是什么? 目前,大数据体系里有各种各样的数据引擎,有大数据生态的 Hadoop、Hive、Kudu、Kafka、HDFS,也有泛大数据库体系的 MongoDB、Redis、ClickHouse、Doris,更有云上的 AWS S3、Redshift、BigQuery、Snowflake,还有各种各样数据生态 MySQL、PostgresSQL、IoTDB、

    2024年02月15日
    浏览(53)
  • 数据同步工具调研选型:SeaTunnel 与 DataX 、Sqoop、Flume、Flink CDC 对比

    Apache SeaTunnel 是一个非常易用的超高性能分布式数据集成产品,支持海量数据的离线及实时同步。每天可稳定高效同步万亿级数据,已应用于数百家企业生产,也是首个由国人主导贡献到 Apache 基金会的数据集成顶级项目。 SeaTunnel 主要解决数据集成领域的常见问题: * 数据源

    2024年02月04日
    浏览(51)
  • Doris:MySQL数据同步到Doris的N种方式

    目录 1.CSV文件方式 1.1 导出mysql数据 1.2 导入数据 2.JDBC 编码方式 3.JDBC Catalog 方式 3.1 上传mysql驱动包 3.2 创建mysql catalog 3.3. 插入数据 4.Binlog Load 方式         当mysql与doris服务之间无法通过网络互联时,可以通过将mysql数据导出成csv文件,然后再在doris服务器导入csv文件的方

    2024年02月04日
    浏览(49)
  • CDH6.3.2 的pyspark读取excel表格数据写入hive中的问题汇总

    需求:内网通过Excel文件将数据同步到外网的CDH服务器中,将CDH中的文件数据写入hive中。 CDH版本为:6.3.2 spark版本为:2.4 python版本:2.7.5 操作系统:CentOS Linux 7 集群方式:yarn-cluster 一、在linux中将excel文件转换成CSV文件,然后上传到hdfs中。 为何要先转csv呢?主要原因是pysp

    2024年02月07日
    浏览(45)
  • 一文教会你用Apache SeaTunnel Zeta离线把数据从MySQL同步到StarRocks

    在上一篇文章中,我们介绍了如何下载安装部署SeaTunnel Zeta服务(3分钟部署SeaTunnel Zeta单节点Standalone模式环境),接下来我们介绍一下SeaTunnel支持的第一个同步场景:离线批量同步。顾名思意,离线批量同步需要用户定义好SeaTunnel JobConfig,选择批处理模式,作业启动后开始同

    2024年02月06日
    浏览(78)
  • CDH大数据平台 24Cloudera Manager Console之hbase、hive整合配置(markdown新版)

    💖个人主页:@与自己作战 💯作者简介: CSDN@博客专家 、 CSDN@大数据领域优质创作者 、 CSDN@内容合伙人 、 阿里云@专家博主 🆘希望大佬们多多支持,携手共进 📝 如果文章对你有帮助的话,欢迎评论💬点赞👍收藏📂加关注 ⛔ 如需要支持请私信我 , 💯 必支持

    2024年02月01日
    浏览(46)
  • Flink实时同步MySQL与Doris数据

    技术解析|Doris Connector 结合 Flink CDC 实现 MySQL 分库分表 Exactly Once 精准接入-阿里云开发者社区 1. Flink环境: https://flink.apache.org/zh/ 下载flink-1.15.1 解压,修改配置 修改配置 修改rest.bind-address为 0.0.0.0 下载依赖jar包 至 flink安装目录lib下 启动flink 访问WebUI http://192.168.0.158:8081 2、

    2024年02月13日
    浏览(45)
  • Apache Doris (三十一):Doris 数据导入(九)Spark Load 4- 导入Hive数据及注意事项

    目录 1. Spark Load导入Hive非分区表数据 2. Spark Load 导入Hive分区表数据 3. 注意事项 进入正文之前,欢迎订阅专题、对博文点赞、评论、收藏,关注IT贫道,获取高质量博客内容!

    2024年02月17日
    浏览(48)

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

博客赞助

微信扫一扫打赏

请作者喝杯咖啡吧~博客赞助

支付宝扫一扫领取红包,优惠每天领

二维码1

领取红包

二维码2

领红包