AWS-EMR & Airflow 调度hivesql 异常thrift.transport.TTransport.TTransportException: TSocket read 0 bytes-Toy模板网

这篇具有很好参考价值的文章主要介绍了AWS-EMR & Airflow 调度hivesql 异常thrift.transport.TTransport.TTransportException: TSocket read 0 bytes。希望对大家有所帮助。如果存在错误或未考虑完全的地方，请大家不吝赐教，您也可以点击"举报违法"按钮提交疑问。

通过Airflow 调用 EMR 中的hive 执行sql，遇到错误异常

 hh = HiveServer2Hook(
        hiveserver2_conn_id="hive_connect_emr"
    )
    res = hh.get_pandas_df("show tables")

[2022-10-13, 21:46:21 ] {{taskinstance.py:1703}} ERROR - Task failed with exception
Traceback (most recent call last):
File “/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py”, line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File “/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py”, line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File “/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py”, line 1514, in _execute_task
result = execute_callable(context=context)
File “/usr/local/lib/python3.7/site-packages/airflow/operators/python.py”, line 151, in execute
return_value = self.execute_callable()
File “/usr/local/lib/python3.7/site-packages/airflow/operators/python.py”, line 162, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File “/usr/local/airflow/dags/netease_hiev_hook.py”, line 29, in execSQLDWD
res = hh.get_pandas_df(“show tables”)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/providers/apache/hive/hooks/hive.py”, line 1062, in get_pandas_df
res = self.get_results(hql, schema=schema, hive_conf=hive_conf)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/providers/apache/hive/hooks/hive.py”, line 949, in get_results
header = next(results_iter)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/providers/apache/hive/hooks/hive.py”, line 885, in _get_results
with contextlib.closing(self.get_conn(schema)) as conn, contextlib.closing(conn.cursor()) as cur:
File “/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/providers/apache/hive/hooks/hive.py”, line 870, in get_conn
database=schema or db.schema or ‘default’,
File “/usr/local/airflow/.local/lib/python3.7/site-packages/pyhive/hive.py”, line 104, in connect
return Connection(*args, **kwargs)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/pyhive/hive.py”, line 249, in init
response = self._client.OpenSession(open_session_req)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/TCLIService/TCLIService.py”, line 187, in OpenSession
return self.recv_OpenSession()
File “/usr/local/airflow/.local/lib/python3.7/site-packages/TCLIService/TCLIService.py”, line 199, in recv_OpenSession
(fname, mtype, rseqid) = iprot.readMessageBegin()
File “/usr/local/airflow/.local/lib/python3.7/site-packages/thrift/protocol/TBinaryProtocol.py”, line 148, in readMessageBegin
name = self.trans.readAll(sz)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/thrift/transport/TTransport.py”, line 62, in readAll
chunk = self.read(sz - have)
File “/usr/local/airflow/.local/lib/python3.7/site-packages/thrift/transport/TTransport.py”, line 164, in read
self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size)))
File “/usr/local/airflow/.local/lib/python3.7/site-packages/thrift/transport/TSocket.py”, line 167, in read
message=‘TSocket read 0 bytes’)
thrift.transport.TTransport.TTransportException: TSocket read 0 bytes

修改hive-site.xml

增加 hive.server2.authentication 设置为NOSASL文章来源地址https://www.toymoban.com/news/detail-803546.html

到了这里，关于AWS-EMR & Airflow 调度hivesql 异常thrift.transport.TTransport.TTransportException: TSocket read 0 bytes的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！