由于https://github.com/kontext-tech/winutils
还未发布3.3.6版本,因此尝试源码编译。
本文 hadoop-3.3.6 编译后 winutils 等文件获取
GitHub 地址
环境和安装包准备,见2
- Git-2.43.0-64-bit.exe
- jdk-8u381-windows-x64.exe
- node-v21.2.0-x64.msi
- apache-ant-1.10.14-bin.zip
- apache-maven-3.9.5-bin.zip
- protoc-25.1-win64.zip
- cmake-3.28.0-rc5-windows-x86_64.zip
- GetGnuWin32-0.6.3.exet
- zlib.tar.gz
- Visual Studio 2010 Professional
- hadoop-3.3.6-src.tar.gz
zlib编译
方法一:
- VS2010打开
zlib路径\contrib\vstudio\vc10\zlibvc.sln
zlib的vs sln解决方案文件包含6个工程。其中:
(1). zlibvc生成动态链接库zlibwapi.dll和对应的导入库zlibwapi.lib
(2). zlibstat生成静态链接库zlibstat.lib
(3). testzlibdll测试生成的动态链接库zlibwapi.dll
(4). testzlib测试生成的静态链接库zlibstat.lib
-
debug模式下编译
zlibvc
和zlibstat
,在zlib路径\contrib\vstudio\vc10\x64\
的ZlibStatDebug
和ZlibDllDebug
文件夹中获取编译生成的文件 -
将这三个文件存入单独的文件夹中,将路径加入系统环境变量path
方法二:
VS命令行工具进入到zlib根目录,执行 nmake -f win32/Makefile.msc
,在根目录下生成:zlib.lib(静态库) zdll.lib(动态库的导入库) zlib1.dll(动态库)
配置文件更改
1. maven阿里云镜像
修改apache-maven-3.9.5路径\conf\settings.xml
文件镜像mirrors节点,使用阿里云仓库,添加如下代码。
<mirror>
<id>aliyunmaven</id>
<mirrorOf>*</mirrorOf>
<name>阿里云公共仓库</name>
<url>https://maven.aliyun.com/repository/public</url>
</mirror>
不更改会很慢,亲测!!!
2. Node版本
- 编辑
hadoop-3.3.6-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-catalog\hadoop-yarn-applications-catalog-webapp
的pom.xml
文件 - find
nodeVersion
,将<nodeVersion>${nodejs.version}</nodeVersion>
改为自己的node版本<nodeVersion>v21.2.0</nodeVersion>
- 如果不更改会出现错误如下。
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.11.2:yarn (yarn install) on project
hadoop-yarn-applications-catalog-webapp: Failed to run task: 'yarn ' failed.
org.apache.commons.exec.
ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
3. 越过Javadoc检查
- 在
pom.xml
中禁用严格检查<additionalparam>-Xdoclint:none</additionalparam>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<configuration>
<additionalparam>-Xdoclint:none</additionalparam>
</configuration>
</plugin>
</plugins>
特别是hadoop-3.3.6-src\hadoop-common-project\hadoop-annotations\pom.xml
需要更改,否则会出现如下错误。
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while generating Javadoc:
- mvn命令构建时跳过 Javadoc
mvn 其他参数 -Dmaven.javadoc.skip=true verify
编译Hadoop
-
VS命令行工具
cd Hadoop文件夹
下 -
执行命令
mvn clean package -Pdist,native-win -DskipTests -Dtar -Dmaven.javadoc.skip=true
-
编译成功
-
编译后文件目录为
hadoop-3.3.6-src\hadoop-dist\target
Error,其他报错见3
- RelWithDebInfo does not exist
hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo does not exist
错误是这个目录不存在,则在hadoop-hdfs-project\hadoop-hdfs-native-client\target\native
下手动创建这个目录,并根据提示在重新执行命令时添加参数-rf :hadoop-hdfs-native-client
,如下。mvn package -Pdist,native-win -DskipTests -Dtar -Dmaven.javadoc.skip=true -rf :hadoop-hdfs-native-client
- hadoop-hdfs-native-client
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.x:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="hadoop路径\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="cmake">
修改hadoop-hdfs-project\hadoop-hdfs-native-client\pom.xml
文件failonerror
的值true为false
3. convert-ms-winutils文章来源:https://www.toymoban.com/news/detail-755900.html
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (convert-ms-winutils) on project hadoop-common: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (convert-ms-winutils) on project hadoop-common: Command execution failed.`
- 添加环境变量 C:\Windows\Microsoft.NET\Framework64\v4.0.30319
- 或修改冲突文件名,确定引用
C:\Windows\Microsoft.NET\Framework64\v4.0.30319
文件
相关文章:文章来源地址https://www.toymoban.com/news/detail-755900.html
- windows上编译hadoop
- Windows10 Hadoop3.3.0源码编译
- Hadoop 3.2.1 win10 64位系统 vs2015 编译
- Hadoop源码编译打包
- hadoop源码编译
- Windows 7 64bit和Visual Studio 2010下安装及使用Zlib 1.2.5
- VS2010 编译 Zlib库-1.2.11(zip解压库)
- Windows中zlib的安装与配置
- Unable to build Maven project due to Javadoc error?
- maven项目install时候一直报错,检查Maven-javadoc-plugin声明错误(Java 8 配置Maven-javadoc-plugin)或者警告: @throws没有说明
- Windows7-64编译hadoop-3.2.0
- Hadoop on Windows Building/ Installation Error
到了这里,关于Windows11编译Hadoop3.3.6源码的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!