KeeperErrorCode = NoAuth for /hbase/tokenauth/keys

这篇具有很好参考价值的文章主要介绍了KeeperErrorCode = NoAuth for /hbase/tokenauth/keys。希望对大家有所帮助。如果存在错误或未考虑完全的地方,请大家不吝赐教,您也可以点击"举报违法"按钮提交疑问。

kerberos配置hbase出現問題


環境如下:
KeeperErrorCode = NoAuth for /hbase/tokenauth/keys

KeeperErrorCode = NoAuth for /hbase/tokenauth/keys

问题描述

想要在hadoop ha的場景上,基於kerberos配置hbase ha,出現了如下的bug

org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/running
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:661)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:637)
        at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeCreated(ZKNodeTracker.java:199)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:460)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
2023-06-23 16:19:56,035 ERROR [main-EventThread] zookeeper.ZKWatcher: regionserver:16020-0x3029dc0d4ec0021, quorum=hadoop102:2181,hadoop103:2181,hadoop104:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/running
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:661)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:637)
        at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeCreated(ZKNodeTracker.java:199)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:460)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
2023-06-23 16:19:56,038 ERROR [main-EventThread] regionserver.HRegionServer: ***** ABORTING region server hadoop102,16020,1687508213386: Unexpected exception handling nodeCreated event *****
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/running
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:661)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:637)
        at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeCreated(ZKNodeTracker.java:199)
        at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:460)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
2023-06-23 16:19:56,041 ERROR [main-EventThread] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2023-06-23 16:19:56,060 INFO  [main-EventThread] regionserver.HRegionServer:
    "exceptions.ScannerResetException" : 0,


原因分析:

這個問題昨天卡了一天,我發現就是在只有在hadoop102幾點上啟動了master和regionserver,我通過hadop102:16010 web頁面訪問,發現是regionserver是dead,所以後面就一直卡死在這裡了
KeeperErrorCode = NoAuth for /hbase/tokenauth/keys

通過bug日誌看,可以知道是kerberos權限認證的問題,看了一下當下的配置文件

hadoop102 : vim hbase-jaas.conf

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/security/keytab/hbase.service.keytab"
  useTicketCache=false
  principal="hbase/hadoop102@EXAMPLE.COM";
};

hadoop103 : vim hbase-jaas.conf

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/security/keytab/hbase.service.keytab"
  useTicketCache=false
  principal="hbase/hadoop103@EXAMPLE.COM";
};

hadoop104 : vim hbase-jaas.conf

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/security/keytab/hbase.service.keytab"
  useTicketCache=false
  principal="hbase/hadoop104@EXAMPLE.COM";
};

我發現只有hadoop102上的master和regionserver啟動起來了,hadoop103和hadoop104沒有啟動,所以下意識就把hadoop103和hadoop104的hbase-jaas.conf配置文件改成了和hadoop102一樣的,重啟hbase,發現所有的服務是可以啟動的,但是無法執行hbase的插入語句

base(main):002:0> create 'student','info'

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2946)
	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1942)
	at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:603)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

Creates a table. Pass a table name, and a set of column family
specifications (at least one), and, optionally, table configuration.
Column specification can be a simple string (name), or a dictionary
(dictionaries are described below in main help output), necessarily
including NAME attribute.
Examples:

Create a table with namespace=ns1 and table qualifier=t1
  hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5}

Create a table with namespace=default and table qualifier=t1
  hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
  hbase> # The above in shorthand would be the following:
  hbase> create 't1', 'f1', 'f2', 'f3'
  hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true}
  hbase> create 't1', {NAME => 'f1', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}}
  hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 1000000, MOB_COMPACT_PARTITION_POLICY => 'weekly'}

Table configuration options can be put at the end.
Examples:

  hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40']
  hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40']
  hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe'
  hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' => 'myvalue' }
  hbase> # Optionally pre-split the table into NUMREGIONS, using
  hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname)
  hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'}
  hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit', REGION_REPLICATION => 2, CONFIGURATION => {'hbase.hregion.scan.loadColumnFamiliesOnDem
  hbase> create 't1', {NAME => 'f1', DFS_REPLICATION => 1}

You can also keep around a reference to the created table:

  hbase> t1 = create 't1', 'f1'

Which gives you a reference to the table named 't1', on which you can then
call methods.

Took 8.8778 seconds                                                                                                                                                 
hbase(main):003:0>  put 'student','1001','info:sex','male'

ERROR: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on hadoop102,16020,1687510685378
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3272)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3249)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2429)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

Put a cell 'value' at specified table/row/column and optionally
timestamp coordinates.  To put a cell value into table 'ns1:t1' or 't1'
at row 'r1' under column 'c1' marked with the time 'ts1', do:

  hbase> put 'ns1:t1', 'r1', 'c1', 'value'
  hbase> put 't1', 'r1', 'c1', 'value'
  hbase> put 't1', 'r1', 'c1', 'value', ts1
  hbase> put 't1', 'r1', 'c1', 'value', {ATTRIBUTES=>{'mykey'=>'myvalue'}}
  hbase> put 't1', 'r1', 'c1', 'value', ts1, {ATTRIBUTES=>{'mykey'=>'myvalue'}}
  hbase> put 't1', 'r1', 'c1', 'value', ts1, {VISIBILITY=>'PRIVATE|SECRET'}

The same commands also can be run on a table reference. Suppose you had a reference
t to table 't1', the corresponding command would be:

  hbase> t.put 'r1', 'c1', 'value', ts1, {ATTRIBUTES=>{'mykey'=>'myvalue'}}

解决方案:

提示:这里填写该问题的具体解决方案:

看到這裡我發現所有節點的regionserver都沒有正常啟動,全是dead狀態,所以我就猜測是zookeeper中的hbase數據損壞導致的,所以就想把zookeeper中的hbase信息刪除

[zk: hadoop102:2181(CONNECTED) 0] ls
ls [-s] [-w] [-R] path
[zk: hadoop102:2181(CONNECTED) 1] ls /
[dolphinscheduler, hadoop-ha, hbase, rmstore, yarn-leader-election, zookeeper]
[zk: hadoop102:2181(CONNECTED) 2] deleteall /hbase
Authentication is not valid : /hbase/replication
[zk: hadoop102:2181(CONNECTED) 3] getAcl /hbase
'sasl,'hbase/hadoop102@EXAMPLE.COM
: cdrwa

發現刪除失敗,一直再報Authentication is not valid : /hbase/replication這個bug,這個是由於zookeeper开启了ACL導致的,最後的解決方案是在zookeeper的配置文件zoo.cfg中加入一行skipACL=yes

#kerberos认证配置
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
sessionRequireClientSASLAuth=true
skipACL=yes

分發zoo.cfg到zk所有節點,重啟zookeeper,再刪除/hbase節點數據

[zk: hadoop102:2181(CONNECTED) 0] ls /
[dolphinscheduler, hadoop-ha, hbase, rmstore, yarn-leader-election, zookeeper]
[zk: hadoop102:2181(CONNECTED) 1] deleteall /hbase
[zk: hadoop102:2181(CONNECTED) 2] ls /
[dolphinscheduler, hadoop-ha, rmstore, yarn-leader-election, zookeeper]
[zk: hadoop102:2181(CONNECTED) 3] quit;
ZooKeeper -server host:port cmd args

成功刪除!!!
到這裡的時候基本上就已經解決成功了
為了保險起見,我把hdfs上的hbase所有文件也刪除了

hadoop fs -rm -r -f /hbase/*

刪除zoo.cfg中的skipACL=yes,然後重啟zk,重啟hbase,訪問hadoop102:16010 web網頁:
KeeperErrorCode = NoAuth for /hbase/tokenauth/keys
可以看到已經沒有dead regionserver了
再執行hbase 插入語句

hbase(main):001:0> create 'student','info'
Created table student
Took 2.6728 seconds                                                                                                                                                
=> Hbase::Table - student
hbase(main):002:0> put 'student','1001','info:sex','male'
Took 0.1907 seconds                                                                                                                                                
hbase(main):003:0> put 'student','1001','info:age','18'
Took 0.0055 seconds                                                                                                                                                
hbase(main):004:0>  scan 'student'
ROW                                       COLUMN+CELL                                                                                                              
 1001                                     column=info:age, timestamp=1687568561569, value=18                                                                       
 1001                                     column=info:sex, timestamp=1687568556688, value=male                                                                     
1 row(s)
Took 0.0611 seconds                                                                                                                                                
hbase(main):005:0> scan 'student',{STARTROW => '1001', STOPROW  => '1001'}
ROW                                       COLUMN+CELL                                                                                                              
 1001                                     column=info:age, timestamp=1687568561569, value=18                                                                       
 1001                                     column=info:sex, timestamp=1687568556688, value=male                                                                     
1 row(s)
Took 0.0131 seconds                                                                                                                                                
hbase(main):006:0> describe 'student'
Table student is ENABLED                                                                                                                                           
student                                                                                                                                                            
COLUMN FAMILIES DESCRIPTION                                                                                                                                        
{NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',
 DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY =
> 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}                
1 row(s)
Took 0.0590 seconds                                                                                                                                                
hbase(main):007:0> quit

至此,bug已經解決完

總結:

解決的bug一共有這麼幾個地方:

1.更改所有節點的 hbase-jaas.conf,保持和hadoop102一致

hadoop102 : vim hbase-jaas.conf

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/etc/security/keytab/hbase.service.keytab"
  useTicketCache=false
  principal="hbase/hadoop102@EXAMPLE.COM";
};

2.刪除zookeeper中的/hbase數據

在zoo.cfg中加入skipACL=yes,再重啟zk,然後刪除/hbase

3.刪除hdfs上hbase舊數據

hdfs fs -rm -r -f /hbase/*

4.重啟hbase,執行建表和數據插入語句

參考:https://zhuanlan.zhihu.com/p/396007109文章来源地址https://www.toymoban.com/news/detail-502764.html

到了这里,关于KeeperErrorCode = NoAuth for /hbase/tokenauth/keys的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处: 如若内容造成侵权/违法违规/事实不符,请点击违法举报进行投诉反馈,一经查实,立即删除!

领支付宝红包 赞助服务器费用

相关文章

  • HBase:Error:KeeperErrorCode=ConnectionLoss for /hbase/master For usage try ‘help‘ ‘disable‘

    1.我在输入查询数据库语句时出错,错误如下图: 然后我返回去检查启动的Hadoop与zookeeper,发现zookeeper的状态缺少一个follower,我是一个master,两个slave,其中一个slave的zookeeper状态是not running,其他两个状态正常,然后我就用之前的方法,删除了和myid在同一目录下生成的version

    2024年02月02日
    浏览(43)
  • ERROR: KeeperErrorCode = NoNode for /hbase/master,hbase在shell命令行中查看表时报错解决方案

    修改了hbase的配置文件hbase-site.xml伪分布式环境,会导致文件的不完整,hadoop上文件与本地文件不一致,所以进入hbase时list搜索不到之前的一整个文件。简而言之,之前时分布式存储的,3个机器,1个存储1块构成完整的文件,现去掉2个,再次查询时就会报错。 1.退出hbase shel

    2024年02月03日
    浏览(40)
  • 【zookeeper】问题解决 Authentication is not valid : /hbase/tokenauth

    最近在搭建Hbase 服务时,服务无法启动,于是决定将 hbase 服务删除,在当删除 zookeeper 的 /hbase 节点时报错,报 thentication is not valid : /hbase/tokenauth 。 看到网上大部分的文章都是使用跳过 ACL 或者 开启 super 模式 这两种方式,于是比较好奇有没有第三种解,这里整理并记录一下

    2024年02月14日
    浏览(46)
  • Kerberos安全认证-连载11-HBase Kerberos安全配置及访问_kerberos hbase(1)

    这里需要将该值设置为false,否则Hadoop HA 不能正常启动。 skipACL=yes 跳过Zookeeper 访问控制列表(ACL)验证,允许连接zookeper后进行读取和写入。这里建议跳过,否则配置HBase 启动后不能向Zookeeper中写入数据。 这里在node3节点进行zoo.cfg文件的配置,配置完成后,将zoo.cfg文件分发

    2024年04月13日
    浏览(53)
  • Kerberos安全认证-连载11-HBase Kerberos安全配置及访问

    目录 1. Zookeeper Kerberos配置 2. HBase配置Kerberos 3. HBase启动及访问验证 4. HBase Shell操作HBase ​​​​​​​5. Java API操作HBase 技术连载系列,前面内容请参考前面连载10内容:​​​​​​​​​​​​​​Kerberos安全认证-连载10-Hive Kerberos 安全配置及访问_IT贫道的博客-CSDN博客

    2024年02月07日
    浏览(53)
  • HBase鉴权设计以及Kerberos鉴权方法

    鉴权,分别由 鉴 和 权 组成 鉴 : 表示身份认证,认证相关用户是否存在以及相关的用户名和密码是否一致 权 : 完成身份的 鉴 后,还需要判断用户是否有相关操作的权限。 因此对于某一个用户来说,通常情况下,需要完成 鉴 和 权 才能够满足一个完整的业务场景,因此

    2024年01月25日
    浏览(37)
  • ZooKeeper创建节点报错:KeeperErrorCode = ConnectionLoss for

    ZooKeeper在创建节点时报错: 解决办法: 观察idea和集群的jdk版本是否一致。 集群jdk为 Java 1.8 idea下jdk为Java 12 调整idea下jdk版本 1.File - Project Structure 由于jdk向下兼容,因此Project下的SDK无需修改,仅调整Modules版本即可 2.File - Settings - Build, Execution, Deployment - Complier - Java Complier 完

    2024年02月06日
    浏览(37)
  • 解决org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for

    今天初学zookeeper遇到了下列错误: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zhiHu(连接丢失) 出错原因:设置的会话时间(sessionTimeout)太小了!!! 解决办法: 还有可能是因为没关防火墙,需要查看以下防火墙状态

    2024年02月09日
    浏览(37)
  • kafka开kerberos认证报错the client is being asked for a password

    @Kafka kerberos认证错误记录TOC kafka 开发调试,开 kerberos情况下遇到的错误。 Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user Caused by: javax.security.auth.login.LoginExcepti

    2023年04月21日
    浏览(38)
  • 对HBase集群进行持续备份和恢复的策略 Continuous Backup & Restore Strategy for Apache HBase

    作者:禅与计算机程序设计艺术 2019年初,Apache HBase项目启动了9个年头。从最初仅仅是Hadoop生态圈中的一个组件,逐渐演变成越来越多的大数据存储解决方案的一部分。在快速发展的同时,也带来了许多技术上的挑战,如一致性、性能等方面的问题。而在这样的背景下,HBa

    2024年02月09日
    浏览(41)

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

博客赞助

微信扫一扫打赏

请作者喝杯咖啡吧~博客赞助

支付宝扫一扫领取红包,优惠每天领

二维码1

领取红包

二维码2

领红包