一、hadoop版本号确认
1. hadoop ==> 2.7.2
[root@dm46 TDH-Client]# hadoop version
Hadoop 2.7.2-transwarp-6.2.0
Subversion http://xxxx:10080/hadoop/hadoop-2.7.2-transwarp.git -r f31230971c2a36e77e4886e0f621366826cec3a3
Compiled by jenkins on 2019-07-27T11:33Z
Compiled with protoc 2.5.0
二、下载spark
注意：选择 Pre-built for Apache Hadoop 2.7
https://dlcdn.apache.org/spark/spark-3.2.3/spark-3.2.3-bin-hadoop2.7.tgz
三、准备pyspark环境
1. 创建虚拟环境
conda create -n pyspark_env python=3.8
conda activate pyspark_env
2. 安装pyspark
注意：确保spark版本与pyspark版本一致
pip install pyspark==3.2.3 -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com
3.测试
cd /home/xxx/kdh/spark/bin
./spark-submit --master local[*] /home/xxx/kdh/spark/examples/src/main/python/pi.py 30
四、常见报错
1. RuntimeError: Python in worker has different version 3.8 than that in driver 3.6
原因：local模式下，此时使用的是当前环境的python。如果代码里指定了pyspark_env，当前环境也应该是pyspark_env。
conda activate pyspark_env
cd /home/xxx/kdh/spark/bin
./spark-submit --master local[*] /home/xxx/workspace/pyspark_learn/02_pyspark_core/main/02_Wordcount_hdfs.py
五、其它
1. arrow安装文章来源：https://www.toymoban.com/news/detail-469920.html
pip install pyspark[sql] -i https://pypi.tuna.tsinghua.edu.cn/simple文章来源地址https://www.toymoban.com/news/detail-469920.html
到了这里，关于PySpark完美安装的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！
