arrow_back Install Hadoop 3.3.0 on Windows 10 Step by Step Guide
comment Comments
11-20 of 36
articleArticles 569
imageDiagrams 58
commentComments 299
loyaltyKontext Points 6279
person Naseemuddin access_time 4 years ago
Hi Raymond,
Thanks a lot! It is working! Really appreceate it!
Best regards
Naseem
N
Naseemuddin Khan
#443
access_time 4 years ago
more_vert
Hi Raymond,
Thanks a lot! It is working! Really appreceate it!
Best regards
Naseem
person Raymond access_time 4 years ago
I've checked out Hadoop 3.3.0 source code and also manually applied the PR you mentioned to update systeminfo.c file to replace PdhAddCounter with PdhAddEnglishCounter.
The build winutils tool are published here:
winutils/hadoop-3.3.0-YARN-8246/bin at master · kontext-tech/winutils · GitHub
Please give it a try to see if it works.
articleArticles 569
imageDiagrams 58
commentComments 299
loyaltyKontext Points 6279
Hi Bikash,
Can you please confirm whether you are using Hadoop 3.3.0 installed from this guide?
I've tried the following commands in my Hadoop 3.3.0 installation on Windows 10 and it is all working well:
F:\big-data>hadoop fs -put F:\big-data\test.csv /test/csv.1 F:\big-data>hadoop fs -copyFromLocal F:\big-data\test.csv /test/csv.2 F:\big-data>hadoop fs -ls /test Found 3 items -rw-r--r-- 1 *** supergroup 29 2021-03-06 09:42 /test/csv.1 -rw-r--r-- 1 *** supergroup 29 2021-03-06 09:42 /test/csv.2 -rw-r--r-- 1 *** supergroup 29 2020-08-22 22:46 /test/test.csv
Please make sure you've followed all the steps exactly.
person Bikash Prakash access_time 4 years ago
Hi,
When ever I try to copy file from pc to hadoop, it throws error.
hadoop fs -put D:/test/file1 /test/input
-put: Can not create a Path from an empty string
also tried --copyFromLocal
hadoop fs -copyFromLocal D:/test/file1 /test/input
-copyFromLocal: Can not create a Path from an empty string
same error.
Please help
BP
Bikash Prakash Dash
#439
access_time 4 years ago
more_vert
Hi,
When ever I try to copy file from pc to hadoop, it throws error.
hadoop fs -put D:/test/file1 /test/input
-put: Can not create a Path from an empty string
also tried --copyFromLocal
hadoop fs -copyFromLocal D:/test/file1 /test/input
-copyFromLocal: Can not create a Path from an empty string
same error.
Please help
articleArticles 569
imageDiagrams 58
commentComments 299
loyaltyKontext Points 6279
I've checked out Hadoop 3.3.0 source code and also manually applied the PR you mentioned to update systeminfo.c file to replace PdhAddCounter with PdhAddEnglishCounter.
The build winutils tool are published here:
winutils/hadoop-3.3.0-YARN-8246/bin at master · kontext-tech/winutils · GitHub
Please give it a try to see if it works.
person Naseemuddin access_time 4 years ago
Hi Raymond, you have gotten farther than I have. I run into failures before the winutils is created. Could create wintuils for me with one little change in the code, described here:
https://github.com/apache/hadoop/pull/458
That is all I need, and why I am trying to do this ... whitout success.
I would really appreciate it!
N
Naseemuddin Khan
#437
access_time 4 years ago
more_vert
Hi Raymond, you have gotten farther than I have. I run into failures before the winutils is created. Could create wintuils for me with one little change in the code, described here:
https://github.com/apache/hadoop/pull/458
That is all I need, and why I am trying to do this ... whitout success.
I would really appreciate it!
person Raymond access_time 4 years ago
Hello,
I've tried my best, I could only build Hadoop 3.3.0 to the HDFS Native Client step so far (not, winutils was built in the steps before this):
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 2.203 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 3.281 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 2.297 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.375 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 1.219 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 3.360 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 4.453 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.546 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ 7.265 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 3.532 s] [INFO] Apache Hadoop Common ............................... SUCCESS [04:23 min] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 9.375 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 8.283 s] [INFO] Apache Hadoop Registry ............................. SUCCESS [ 11.093 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.515 s] [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 52.032 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [01:46 min] [INFO] Apache Hadoop HDFS Native Client ................... FAILURE [01:03 min]
This is also the reason I have not published article about building Hadoop 3.3.0 on Windows 10. The build article I published previously are for Hadoop 3.2.1 for which I can complete a successful build. I found that it becomes more and more difficult to build successfully on Windows.
Thus I would not recommend spending much more time on this unless it is really important. Build it on Linux or Windows Subsystem for Linux will be much easier. Alternatively you can build Hadoop 3.2.1 instead.
For the step where you failed, I didn't get any error:
[DEBUG] env: =C:=C:\hdp\hadoop
[DEBUG] env: =EXITCODE=00000000
[DEBUG] env: ALLUSERSPROFILE=C:\ProgramData
[DEBUG] env: APPDATA=C:\Users\kontext\AppData\Roaming
[DEBUG] env: CLASSWORLDS_JAR="C:\maven\apache-maven-3.6.3\bin\..\boot\plexus-classworlds-2.6.0.jar"
[DEBUG] env: CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher
[DEBUG] env: CMAKE_GENERATOR=Visual Studio 14 2015
[DEBUG] env: CMAKE_GENERATOR_PLATFORM=x64
[DEBUG] env: CMAKE_GENERATOR_TOOLSET=v140
[DEBUG] env: COMMONPROGRAMFILES=C:\Program Files\Common Files
[DEBUG] env: COMMONPROGRAMFILES(X86)=C:\Program Files (x86)\Common Files
[DEBUG] env: COMMONPROGRAMW6432=C:\Program Files\Common Files
[DEBUG] env: COMPUTERNAME=RAYMOND-VM
[DEBUG] env: COMSPEC=C:\WINDOWS\system32\cmd.exe
[DEBUG] env: DEVENVDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\
[DEBUG] env: DRIVERDATA=C:\Windows\System32\Drivers\DriverData
[DEBUG] env: ERROR_CODE=0
[DEBUG] env: EXEC_DIR=C:\hdp\hadoop
[DEBUG] env: FRAMEWORK40VERSION=v4.0
[DEBUG] env: FRAMEWORKDIR=C:\WINDOWS\Microsoft.NET\Framework\
[DEBUG] env: FRAMEWORKDIR32=C:\WINDOWS\Microsoft.NET\Framework\
[DEBUG] env: FRAMEWORKVERSION=v4.0.30319
[DEBUG] env: FRAMEWORKVERSION32=v4.0.30319
[DEBUG] env: GIT_HOME=C:\Program Files\Git
[DEBUG] env: HOMEDRIVE=C:
[DEBUG] env: HOMEPATH=\Users\kontext
[DEBUG] env: INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um;C:\Program Files (x86)\Windows Kits\8.1\include\\shared;C:\Program Files (x86)\Windows Kits\8.1\include\\um;C:\Program Files (x86)\Windows Kits\8.1\include\\winrt;
[DEBUG] env: JAVACMD=C:\Program Files\Java\jdk1.8.0_261\bin\java.exe
[DEBUG] env: JAVA_HOME=C:\Program Files\Java\jdk1.8.0_261
[DEBUG] env: JVMCONFIG=\.mvn\jvm.config
[DEBUG] env: LIB=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB;C:\Program Files (x86)\Windows Kits\10\lib\10.0.10240.0\ucrt\x86;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x86;C:\Program Files (x86)\Windows Kits\8.1\lib\winv6.3\um\x86;
[DEBUG] env: LIBPATH=C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB;C:\Program Files (x86)\Windows Kits\8.1\References\CommonConfiguration\Neutral;\Microsoft.VCLibs\14.0\References\CommonConfiguration\neutral;
[DEBUG] env: LOCALAPPDATA=C:\Users\kontext\AppData\Local
[DEBUG] env: LOGONSERVER=\\RAYMOND-VM
[DEBUG] env: MAVEN_CMD_LINE_ARGS=package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true -X
[DEBUG] env: MAVEN_HOME=C:\maven\apache-maven-3.6.3\bin\..
[DEBUG] env: MAVEN_PROJECTBASEDIR=C:\hdp\hadoop
[DEBUG] env: MSVS=C:\Program Files (x86)\Microsoft Visual Studio 10.0
[DEBUG] env: NETFXSDKDIR=C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\
[DEBUG] env: NUMBER_OF_PROCESSORS=2
[DEBUG] env: ONEDRIVE=C:\Users\kontext\OneDrive
[DEBUG] env: OS=Windows_NT
[DEBUG] env: PATH=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files (x86)\MSBuild\14.0\bin;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools;C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\VCPackages;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Performance Tools;C:\Program Files (x86)\Windows Kits\8.1\bin\x86;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\;C:\Python\Python38\Scripts\;C:\Python\Python38\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\CMake\bin;C:\Users\kontext\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Java\jdk1.8.0_261\bin;C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;C:\maven\apache-maven-3.6.3\bin;C:\dev\vcpkg\installed\x64-windows\tools\protobuf;
[DEBUG] env: PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW
[DEBUG] env: PLATFORM=x64
[DEBUG] env: PROCESSOR_ARCHITECTURE=AMD64
[DEBUG] env: PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
[DEBUG] env: PROCESSOR_LEVEL=6
[DEBUG] env: PROCESSOR_REVISION=5e03
[DEBUG] env: PROGRAMDATA=C:\ProgramData
[DEBUG] env: PROGRAMFILES=C:\Program Files
[DEBUG] env: PROGRAMFILES(X86)=C:\Program Files (x86)
[DEBUG] env: PROGRAMW6432=C:\Program Files
[DEBUG] env: PROMPT=$P$G
[DEBUG] env: PSMODULEPATH=C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\
[DEBUG] env: PUBLIC=C:\Users\Public
[DEBUG] env: SESSIONNAME=Console
[DEBUG] env: SYSTEMDRIVE=C:
[DEBUG] env: SYSTEMROOT=C:\WINDOWS
[DEBUG] env: TEMP=C:\Users\kontext\AppData\Local\Temp
[DEBUG] env: TMP=C:\Users\kontext\AppData\Local\Temp
[DEBUG] env: UCRTVERSION=10.0.10240.0
[DEBUG] env: UNIVERSALCRTSDKDIR=C:\Program Files (x86)\Windows Kits\10\
[DEBUG] env: USERDOMAIN=RAYMOND-VM
[DEBUG] env: USERDOMAIN_ROAMINGPROFILE=RAYMOND-VM
[DEBUG] env: USERNAME=kontext
[DEBUG] env: USERPROFILE=C:\Users\kontext
[DEBUG] env: VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\
[DEBUG] env: VCVARSPLAT=amd64
[DEBUG] env: VISUALSTUDIOVERSION=14.0
[DEBUG] env: VS100COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\
[DEBUG] env: VS110COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\
[DEBUG] env: VS120COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\
[DEBUG] env: VS140COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\
[DEBUG] env: VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\
[DEBUG] env: WDIR=C:\
[DEBUG] env: WINDIR=C:\WINDOWS
[DEBUG] env: WINDOWSLIBPATH=C:\Program Files (x86)\Windows Kits\8.1\References\CommonConfiguration\Neutral
[DEBUG] env: WINDOWSSDKDIR=C:\Program Files (x86)\Windows Kits\8.1\
[DEBUG] env: WINDOWSSDKLIBVERSION=winv6.3\
[DEBUG] env: WINDOWSSDKVERSION=\
[DEBUG] env: WINDOWSSDK_EXECUTABLEPATH_X64=C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\x64\
[DEBUG] env: WINDOWSSDK_EXECUTABLEPATH_X86=C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (tar) @ hadoop-project-dist ---
[DEBUG] Configuring mojo org.apache.maven.plugins:maven-antrun-plugin:1.7:run from plugin realm ClassRealm[plugin>org.apache.maven.plugins:maven-antrun-plugin:1.7-2093631164, parent: sun.misc.Launcher$AppClassLoader@4e25154f]
[DEBUG] Configuring mojo 'org.apache.maven.plugins:maven-antrun-plugin:1.7:run' with basic configurator -->
[DEBUG] (f) exportAntProperties = false
[DEBUG] (f) failOnError = true
[DEBUG] (f) localRepository = id: local
url: file:///C:/m-repo/
layout: default
snapshots: [enabled => true, update => always]
releases: [enabled => true, update => always]
[DEBUG] (f) pluginArtifacts = [org.apache.maven.plugins:maven-antrun-plugin:maven-plugin:1.7:, org.apache.maven:maven-plugin-api:jar:2.0.11:compile, org.apache.maven:maven-project:jar:2.0.11:compile, org.apache.maven:maven-settings:jar:2.0.11:compile, org.apache.maven:maven-profile:jar:2.0.11:compile, org.apache.maven:maven-model:jar:2.0.11:compile, org.apache.maven:maven-artifact-manager:jar:2.0.11:compile, org.apache.maven:maven-repository-metadata:jar:2.0.11:compile, org.apache.maven:maven-plugin-registry:jar:2.0.11:compile, org.codehaus.plexus:plexus-interpolation:jar:1.1:compile, org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-9-stable-1:compile, junit:junit:jar:3.8.1:compile, classworlds:classworlds:jar:1.1-alpha-2:compile, org.apache.maven:maven-artifact:jar:2.0.11:compile, org.codehaus.plexus:plexus-utils:jar:2.0.5:compile, org.apache.ant:ant:jar:1.8.2:compile, org.apache.ant:ant-launcher:jar:1.8.2:compile]
[DEBUG] (f) project = MavenProject: org.apache.hadoop:hadoop-project-dist:3.3.0 @ C:\hdp\hadoop\hadoop-project-dist\pom.xml
[DEBUG] (f) skip = false
[DEBUG] (f) target = <target if="tar"><echo file="C:\hdp\hadoop\hadoop-project-dist\target/dist-maketar.sh">cd "C:\hdp\hadoop\hadoop-project-dist\target"
tar cf - hadoop-project-dist-3.3.0 | gzip > hadoop-project-dist-3.3.0.tar.gz</echo>
<exec failonerror="true" dir="C:\hdp\hadoop\hadoop-project-dist\target" executable="bash"><arg line="./dist-maketar.sh"/>
</exec>
</target>I feel like there might be something different between your bash and mine. For example, based the following error message in your log, your bash is removing '\' from the path which lead to the problem. Unfortunately I don't know what could be the root cause:
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
My bash version is:
bash --version GNU bash, version 4.4.23(1)-release (x86_64-pc-msys) Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
articleArticles 569
imageDiagrams 58
commentComments 299
loyaltyKontext Points 6279
Hello,
I've tried my best, I could only build Hadoop 3.3.0 to the HDFS Native Client step so far (not, winutils was built in the steps before this):
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 2.203 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 3.281 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 2.297 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.375 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 1.219 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 3.360 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 4.453 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.546 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ 7.265 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 3.532 s] [INFO] Apache Hadoop Common ............................... SUCCESS [04:23 min] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 9.375 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 8.283 s] [INFO] Apache Hadoop Registry ............................. SUCCESS [ 11.093 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.515 s] [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 52.032 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [01:46 min] [INFO] Apache Hadoop HDFS Native Client ................... FAILURE [01:03 min]
This is also the reason I have not published article about building Hadoop 3.3.0 on Windows 10. The build article I published previously are for Hadoop 3.2.1 for which I can complete a successful build. I found that it becomes more and more difficult to build successfully on Windows.
Thus I would not recommend spending much more time on this unless it is really important. Build it on Linux or Windows Subsystem for Linux will be much easier. Alternatively you can build Hadoop 3.2.1 instead.
For the step where you failed, I didn't get any error:
[DEBUG] env: =C:=C:\hdp\hadoop
[DEBUG] env: =EXITCODE=00000000
[DEBUG] env: ALLUSERSPROFILE=C:\ProgramData
[DEBUG] env: APPDATA=C:\Users\kontext\AppData\Roaming
[DEBUG] env: CLASSWORLDS_JAR="C:\maven\apache-maven-3.6.3\bin\..\boot\plexus-classworlds-2.6.0.jar"
[DEBUG] env: CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher
[DEBUG] env: CMAKE_GENERATOR=Visual Studio 14 2015
[DEBUG] env: CMAKE_GENERATOR_PLATFORM=x64
[DEBUG] env: CMAKE_GENERATOR_TOOLSET=v140
[DEBUG] env: COMMONPROGRAMFILES=C:\Program Files\Common Files
[DEBUG] env: COMMONPROGRAMFILES(X86)=C:\Program Files (x86)\Common Files
[DEBUG] env: COMMONPROGRAMW6432=C:\Program Files\Common Files
[DEBUG] env: COMPUTERNAME=RAYMOND-VM
[DEBUG] env: COMSPEC=C:\WINDOWS\system32\cmd.exe
[DEBUG] env: DEVENVDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\
[DEBUG] env: DRIVERDATA=C:\Windows\System32\Drivers\DriverData
[DEBUG] env: ERROR_CODE=0
[DEBUG] env: EXEC_DIR=C:\hdp\hadoop
[DEBUG] env: FRAMEWORK40VERSION=v4.0
[DEBUG] env: FRAMEWORKDIR=C:\WINDOWS\Microsoft.NET\Framework\
[DEBUG] env: FRAMEWORKDIR32=C:\WINDOWS\Microsoft.NET\Framework\
[DEBUG] env: FRAMEWORKVERSION=v4.0.30319
[DEBUG] env: FRAMEWORKVERSION32=v4.0.30319
[DEBUG] env: GIT_HOME=C:\Program Files\Git
[DEBUG] env: HOMEDRIVE=C:
[DEBUG] env: HOMEPATH=\Users\kontext
[DEBUG] env: INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um;C:\Program Files (x86)\Windows Kits\8.1\include\\shared;C:\Program Files (x86)\Windows Kits\8.1\include\\um;C:\Program Files (x86)\Windows Kits\8.1\include\\winrt;
[DEBUG] env: JAVACMD=C:\Program Files\Java\jdk1.8.0_261\bin\java.exe
[DEBUG] env: JAVA_HOME=C:\Program Files\Java\jdk1.8.0_261
[DEBUG] env: JVMCONFIG=\.mvn\jvm.config
[DEBUG] env: LIB=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB;C:\Program Files (x86)\Windows Kits\10\lib\10.0.10240.0\ucrt\x86;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x86;C:\Program Files (x86)\Windows Kits\8.1\lib\winv6.3\um\x86;
[DEBUG] env: LIBPATH=C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB;C:\Program Files (x86)\Windows Kits\8.1\References\CommonConfiguration\Neutral;\Microsoft.VCLibs\14.0\References\CommonConfiguration\neutral;
[DEBUG] env: LOCALAPPDATA=C:\Users\kontext\AppData\Local
[DEBUG] env: LOGONSERVER=\\RAYMOND-VM
[DEBUG] env: MAVEN_CMD_LINE_ARGS=package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true -X
[DEBUG] env: MAVEN_HOME=C:\maven\apache-maven-3.6.3\bin\..
[DEBUG] env: MAVEN_PROJECTBASEDIR=C:\hdp\hadoop
[DEBUG] env: MSVS=C:\Program Files (x86)\Microsoft Visual Studio 10.0
[DEBUG] env: NETFXSDKDIR=C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\
[DEBUG] env: NUMBER_OF_PROCESSORS=2
[DEBUG] env: ONEDRIVE=C:\Users\kontext\OneDrive
[DEBUG] env: OS=Windows_NT
[DEBUG] env: PATH=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files (x86)\MSBuild\14.0\bin;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools;C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\VCPackages;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Performance Tools;C:\Program Files (x86)\Windows Kits\8.1\bin\x86;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\;C:\Python\Python38\Scripts\;C:\Python\Python38\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\CMake\bin;C:\Users\kontext\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Java\jdk1.8.0_261\bin;C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;C:\maven\apache-maven-3.6.3\bin;C:\dev\vcpkg\installed\x64-windows\tools\protobuf;
[DEBUG] env: PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW
[DEBUG] env: PLATFORM=x64
[DEBUG] env: PROCESSOR_ARCHITECTURE=AMD64
[DEBUG] env: PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
[DEBUG] env: PROCESSOR_LEVEL=6
[DEBUG] env: PROCESSOR_REVISION=5e03
[DEBUG] env: PROGRAMDATA=C:\ProgramData
[DEBUG] env: PROGRAMFILES=C:\Program Files
[DEBUG] env: PROGRAMFILES(X86)=C:\Program Files (x86)
[DEBUG] env: PROGRAMW6432=C:\Program Files
[DEBUG] env: PROMPT=$P$G
[DEBUG] env: PSMODULEPATH=C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\
[DEBUG] env: PUBLIC=C:\Users\Public
[DEBUG] env: SESSIONNAME=Console
[DEBUG] env: SYSTEMDRIVE=C:
[DEBUG] env: SYSTEMROOT=C:\WINDOWS
[DEBUG] env: TEMP=C:\Users\kontext\AppData\Local\Temp
[DEBUG] env: TMP=C:\Users\kontext\AppData\Local\Temp
[DEBUG] env: UCRTVERSION=10.0.10240.0
[DEBUG] env: UNIVERSALCRTSDKDIR=C:\Program Files (x86)\Windows Kits\10\
[DEBUG] env: USERDOMAIN=RAYMOND-VM
[DEBUG] env: USERDOMAIN_ROAMINGPROFILE=RAYMOND-VM
[DEBUG] env: USERNAME=kontext
[DEBUG] env: USERPROFILE=C:\Users\kontext
[DEBUG] env: VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\
[DEBUG] env: VCVARSPLAT=amd64
[DEBUG] env: VISUALSTUDIOVERSION=14.0
[DEBUG] env: VS100COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\
[DEBUG] env: VS110COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\
[DEBUG] env: VS120COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\
[DEBUG] env: VS140COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\
[DEBUG] env: VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\
[DEBUG] env: WDIR=C:\
[DEBUG] env: WINDIR=C:\WINDOWS
[DEBUG] env: WINDOWSLIBPATH=C:\Program Files (x86)\Windows Kits\8.1\References\CommonConfiguration\Neutral
[DEBUG] env: WINDOWSSDKDIR=C:\Program Files (x86)\Windows Kits\8.1\
[DEBUG] env: WINDOWSSDKLIBVERSION=winv6.3\
[DEBUG] env: WINDOWSSDKVERSION=\
[DEBUG] env: WINDOWSSDK_EXECUTABLEPATH_X64=C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\x64\
[DEBUG] env: WINDOWSSDK_EXECUTABLEPATH_X86=C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (tar) @ hadoop-project-dist ---
[DEBUG] Configuring mojo org.apache.maven.plugins:maven-antrun-plugin:1.7:run from plugin realm ClassRealm[plugin>org.apache.maven.plugins:maven-antrun-plugin:1.7-2093631164, parent: sun.misc.Launcher$AppClassLoader@4e25154f]
[DEBUG] Configuring mojo 'org.apache.maven.plugins:maven-antrun-plugin:1.7:run' with basic configurator -->
[DEBUG] (f) exportAntProperties = false
[DEBUG] (f) failOnError = true
[DEBUG] (f) localRepository = id: local
url: file:///C:/m-repo/
layout: default
snapshots: [enabled => true, update => always]
releases: [enabled => true, update => always]
[DEBUG] (f) pluginArtifacts = [org.apache.maven.plugins:maven-antrun-plugin:maven-plugin:1.7:, org.apache.maven:maven-plugin-api:jar:2.0.11:compile, org.apache.maven:maven-project:jar:2.0.11:compile, org.apache.maven:maven-settings:jar:2.0.11:compile, org.apache.maven:maven-profile:jar:2.0.11:compile, org.apache.maven:maven-model:jar:2.0.11:compile, org.apache.maven:maven-artifact-manager:jar:2.0.11:compile, org.apache.maven:maven-repository-metadata:jar:2.0.11:compile, org.apache.maven:maven-plugin-registry:jar:2.0.11:compile, org.codehaus.plexus:plexus-interpolation:jar:1.1:compile, org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-9-stable-1:compile, junit:junit:jar:3.8.1:compile, classworlds:classworlds:jar:1.1-alpha-2:compile, org.apache.maven:maven-artifact:jar:2.0.11:compile, org.codehaus.plexus:plexus-utils:jar:2.0.5:compile, org.apache.ant:ant:jar:1.8.2:compile, org.apache.ant:ant-launcher:jar:1.8.2:compile]
[DEBUG] (f) project = MavenProject: org.apache.hadoop:hadoop-project-dist:3.3.0 @ C:\hdp\hadoop\hadoop-project-dist\pom.xml
[DEBUG] (f) skip = false
[DEBUG] (f) target = <target if="tar"><echo file="C:\hdp\hadoop\hadoop-project-dist\target/dist-maketar.sh">cd "C:\hdp\hadoop\hadoop-project-dist\target"
tar cf - hadoop-project-dist-3.3.0 | gzip > hadoop-project-dist-3.3.0.tar.gz</echo>
<exec failonerror="true" dir="C:\hdp\hadoop\hadoop-project-dist\target" executable="bash"><arg line="./dist-maketar.sh"/>
</exec>
</target>I feel like there might be something different between your bash and mine. For example, based the following error message in your log, your bash is removing '\' from the path which lead to the problem. Unfortunately I don't know what could be the root cause:
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
My bash version is:
bash --version GNU bash, version 4.4.23(1)-release (x86_64-pc-msys) Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
person Naseemuddin access_time 4 years ago
I think a I was able to correct the path to Windws SDK 8.1. I opened the file C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\VCVarsQueryRegistry.bat and adjusted the function GetWindowsSdkDirHelper
:GetWindowsSdkDirHelper
@for /F "tokens=1,2*" %%i in ('reg query "%1\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v8.1" /v "InstallationFolder"') DO (
@if "%%i"=="InstallationFolder" (
@SET "WindowsSdkDir=%%k"
)
)
@if "%WindowsSdkDir%"=="" exit /B 1
@exit /B 0
I have just changed v7.1A to v8.1 here. In the build log it shows now:
[DEBUG] env: VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 10.0\
[DEBUG] env: WDIR=c:\
[DEBUG] env: WINDIR=C:\Windows
[DEBUG] env: WINDOWSSDKDIR=C:\Program Files (x86)\Windows Kits\8.1\
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.270 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 3.086 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.513 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.956 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.534 s]
[INFO] Apache Hadoop Project Dist POM ..................... FAILURE [ 0.812 s]
[INFO] Apache Hadoop Maven Plugins ........................ SKIPPED
[INFO] Apache Hadoop MiniKDC .............................. SKIPPED
[INFO] Apache Hadoop Auth ................................. SKIPPED
Is WINDOWSSDKDIR here correct now? This is the content of that path:
It did not change much else...
N
Naseemuddin Khan
#432
access_time 4 years ago
more_vert
I think a I was able to correct the path to Windws SDK 8.1. I opened the file C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\VCVarsQueryRegistry.bat and adjusted the function GetWindowsSdkDirHelper
:GetWindowsSdkDirHelper
@for /F "tokens=1,2*" %%i in ('reg query "%1\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v8.1" /v "InstallationFolder"') DO (
@if "%%i"=="InstallationFolder" (
@SET "WindowsSdkDir=%%k"
)
)
@if "%WindowsSdkDir%"=="" exit /B 1
@exit /B 0
I have just changed v7.1A to v8.1 here. In the build log it shows now:
[DEBUG] env: VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 10.0\
[DEBUG] env: WDIR=c:\
[DEBUG] env: WINDIR=C:\Windows
[DEBUG] env: WINDOWSSDKDIR=C:\Program Files (x86)\Windows Kits\8.1\
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.270 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 3.086 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.513 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.956 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.534 s]
[INFO] Apache Hadoop Project Dist POM ..................... FAILURE [ 0.812 s]
[INFO] Apache Hadoop Maven Plugins ........................ SKIPPED
[INFO] Apache Hadoop MiniKDC .............................. SKIPPED
[INFO] Apache Hadoop Auth ................................. SKIPPED
Is WINDOWSSDKDIR here correct now? This is the content of that path:
It did not change much else...
person Raymond access_time 4 years ago
From the logs you provided, you will at least encounter several errors:
- JAVA_HOME path has space. I recommend to install JDK in a path without space, for example: C:\Java
- C:\Users\NaseemuddinKhan\.m2\repository: Maven home path is too long; please create symbolic link as I mentioned in the other article about building Hadoop. This will lead to a very long file path for some Java libraries.
- Windows SDK version is lower than expected.
For some reason, the error message your provided still doesn't include the error messages.
Can you double confirm that you are using Command Prompt to run the maven build command?
Can you please ensure you follow exactly the steps I mentioned in the build article?
https://kontext.tech/column/hadoop/378/compile-and-build-hadoop-321-on-windows-10-guide
And also you only need to paste the detailed error messages before the summary:
Here is the debug information but doesn't seem to be the error message.
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.828 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 6.380 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 3.688 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.993 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 1.303 s]
[INFO] Apache Hadoop Project Dist POM ..................... FAILURE [ 1.884 s]
[INFO] Apache Hadoop Maven Plugins ........................ SKIPPED
N
Naseemuddin Khan
#431
access_time 4 years ago
more_vert
Hi Raymond,
thanks a lot for your help!
- I have removed and installed Java to a C:\Java
- I had created the symbolic link for Maven but forgot to add it into settings.xml. Now I have added it and rerun the command. I am not sure if the symbolic link is working correctly. At least it works with cd command:
See here for my settings.xml:
However, in the logs (below) it still appears as full path. What could be wrong?
- I have installed Windows SKD 8.1 and you can find the path C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\ in some places the logs (below). However, the problems seems to be that WINDOWSSDKDIR is pointing to the wrong path (C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\). This variable is not listed in the systems variables like the others. Do you know how to change it? I have tried this but without success: https://stackoverflow.com/questions/3599079/windowssdkdir-is-not-set-correctly-in-visual-studio-2010
- I am using the Visual Studio x64 Win64 Command Prompt
as instructed in your guide:
- I am quite sure that followed the steps in the build article exactly. I just forgot the settings.xml. The only difference is that I am using Protocol Buffers 3.7.1 for Hadoop 3.3.0
- The final debug message before the summary states that C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs is missing. It is not there in the GitHub repository. Is it required?
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.138 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 2.419 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.093 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.407 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.410 s]
[INFO] Apache Hadoop Project Dist POM ..................... FAILURE [ 0.704 s]
[INFO] Apache Hadoop Maven Plugins ........................ SKIPPED
- Here is the complete log
[DEBUG] Configuring mojo 'org.codehaus.mojo:exec-maven-plugin:1.3.1:exec' with basic configurator -->
[DEBUG] (f) arguments = [C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
[DEBUG] (f) basedir = C:\hdp\hadoop\hadoop-project-dist
[DEBUG] (f) classpathScope = runtime
[DEBUG] (f) executable = bash
[DEBUG] (f) failWithEmptyArgument = true
[DEBUG] (f) failWithNullKeyOrValueInEnvironment = true
[DEBUG] (f) longClasspath = false
[DEBUG] (f) project = MavenProject: org.apache.hadoop:hadoop-project-dist:3.3.0 @ C:\hdp\hadoop\hadoop-project-dist\pom.xml
[DEBUG] (f) skip = false
[DEBUG] (f) workingDirectory = C:\hdp\hadoop\hadoop-project-dist\target
[DEBUG] (f) session = org.apache.maven.execution.MavenSession@5edf2821
[DEBUG] -- end configuration --
[DEBUG] env: =C:=C:\hdp\hadoop
[DEBUG] env: =EXITCODE=00000000
[DEBUG] env: ALLUSERSPROFILE=C:\ProgramData
[DEBUG] env: APPDATA=C:\Users\NaseemuddinKhan\AppData\Roaming
[DEBUG] env: CLASSWORLDS_JAR="C:\apache-maven-3.6.3\bin\..\boot\plexus-classworlds-2.6.0.jar"
[DEBUG] env: CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher
[DEBUG] env: COMMANDPROMPTTYPE=Native
[DEBUG] env: COMMONPROGRAMFILES=C:\Program Files\Common Files
[DEBUG] env: COMMONPROGRAMFILES(X86)=C:\Program Files (x86)\Common Files
[DEBUG] env: COMMONPROGRAMW6432=C:\Program Files\Common Files
[DEBUG] env: COMPUTERNAME=ADV075
[DEBUG] env: COMSPEC=C:\Windows\system32\cmd.exe
[DEBUG] env: DRIVERDATA=C:\Windows\System32\Drivers\DriverData
[DEBUG] env: ERROR_CODE=0
[DEBUG] env: EXEC_DIR=C:\hdp\hadoop
[DEBUG] env: FRAMEWORK35VERSION=v3.5
[DEBUG] env: FRAMEWORKDIR=C:\Windows\Microsoft.NET\Framework64
[DEBUG] env: FRAMEWORKDIR64=C:\Windows\Microsoft.NET\Framework64
[DEBUG] env: FRAMEWORKVERSION=v4.0.30319
[DEBUG] env: FRAMEWORKVERSION64=v4.0.30319
[DEBUG] env: GIT_HOME=C:\Program Files\Git
[DEBUG] env: HADOOP_HOME=C:\hadoop-3.3.0
[DEBUG] env: HIVE_HOME=C:\apache-hive-3.1.2-bin
[DEBUG] env: HOMEDRIVE=C:
[DEBUG] env: HOMEPATH=\Users\NaseemuddinKhan
[DEBUG] env: INCLUDE=c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\include;
[DEBUG] env: JAVACMD=C:\Java\jdk1.8.0_281\bin\java.exe
[DEBUG] env: JAVA_HOME=C:\Java\jdk1.8.0_281
[DEBUG] env: JVMCONFIG=\.mvn\jvm.config
[DEBUG] env: LIB=c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\LIB\amd64;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\ATLMFC\LIB\amd64;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\lib\x64;
[DEBUG] env: LIBPATH=C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Windows\Microsoft.NET\Framework64\v3.5;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\LIB\amd64;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\ATLMFC\LIB\amd64;
[DEBUG] env: LOCALAPPDATA=C:\Users\NaseemuddinKhan\AppData\Local
[DEBUG] env: LOGONSERVER=\\ADV075
[DEBUG] env: MAVEN_CMD_LINE_ARGS=package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true -X
[DEBUG] env: MAVEN_HOME=C:\apache-maven-3.6.3\bin\..
[DEBUG] env: MAVEN_PROJECTBASEDIR=C:\hdp\hadoop
[DEBUG] env: MSVS=C:\Program Files (x86)\Microsoft Visual Studio 10.0
[DEBUG] env: NUMBER_OF_PROCESSORS=8
[DEBUG] env: ONEDRIVE=C:\Users\NaseemuddinKhan\OneDrive - ADVISORI FTC GmbH
[DEBUG] env: ONEDRIVECOMMERCIAL=C:\Users\NaseemuddinKhan\OneDrive - ADVISORI FTC GmbH
[DEBUG] env: OS=Windows_NT
[DEBUG] env: PATH=c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Windows\Microsoft.NET\Framework64\v3.5;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\VCPackages;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools\x64;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\x64;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files\Python39\Scripts\;C:\Program Files\Python39\;C:\oraclexe\app\oracle\product\11.2.0\server\bin;C:\oraclexe\app\oracle\product\11.2.0\server\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files\PuTTY\;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;c:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;c:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Python39;C:\Java\jdk1.8.0_281\bin;C:\spark-3.0.1-bin-hadoop3.2\bin;C:\hadoop-3.3.0\bin;C:\hadoop-3.3.0\sbin;C:\apache-hive-3.1.2-bin\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;C:\apache-maven-3.6.3\bin;C:\protoc-2.5.0-win32;C:\Program Files\CMake\bin;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Users\NaseemuddinKhan\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Graphviz2.38\bin
[DEBUG] env: PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW
[DEBUG] env: PLATFORM=X64
[DEBUG] env: PROCESSOR_ARCHITECTURE=AMD64
[DEBUG] env: PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
[DEBUG] env: PROCESSOR_LEVEL=6
[DEBUG] env: PROCESSOR_REVISION=8e0a
[DEBUG] env: PROGRAMDATA=C:\ProgramData
[DEBUG] env: PROGRAMFILES=C:\Program Files
[DEBUG] env: PROGRAMFILES(X86)=C:\Program Files (x86)
[DEBUG] env: PROGRAMW6432=C:\Program Files
[DEBUG] env: PROMPT=$P$G
[DEBUG] env: PSMODULEPATH=C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
[DEBUG] env: PUBLIC=C:\Users\Public
[DEBUG] env: SPARK_HOME=C:\spark-3.0.1-bin-hadoop3.2
[DEBUG] env: SPARK_LOCAL_IP=127.0.0.1
[DEBUG] env: SYSTEMDRIVE=C:
[DEBUG] env: SYSTEMROOT=C:\Windows
[DEBUG] env: TEMP=C:\Users\NASEEM~1\AppData\Local\Temp
[DEBUG] env: TMP=C:\Users\NASEEM~1\AppData\Local\Temp
[DEBUG] env: USERDOMAIN=AzureAD
[DEBUG] env: USERDOMAIN_ROAMINGPROFILE=AzureAD
[DEBUG] env: USERNAME=NaseemuddinKhan
[DEBUG] env: USERPROFILE=C:\Users\NaseemuddinKhan
[DEBUG] env: VBOX_MSI_INSTALL_PATH=C:\Program Files\Oracle\VirtualBox\
[DEBUG] env: VCINSTALLDIR=c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\
[DEBUG] env: VCVARSPLAT=amd64
[DEBUG] env: VS100COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\
[DEBUG] env: VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 10.0\
[DEBUG] env: WDIR=C:\
[DEBUG] env: WINDIR=C:\Windows
[DEBUG] env: WINDOWSSDKDIR=C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.138 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 2.419 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.093 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.407 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.410 s]
[INFO] Apache Hadoop Project Dist POM ..................... FAILURE [ 0.704 s]
person Raymond access_time 4 years ago
From the logs you provided, you will at least encounter several errors:
- JAVA_HOME path has space. I recommend to install JDK in a path without space, for example: C:\Java
- C:\Users\NaseemuddinKhan\.m2\repository: Maven home path is too long; please create symbolic link as I mentioned in the other article about building Hadoop. This will lead to a very long file path for some Java libraries.
- Windows SDK version is lower than expected.
For some reason, the error message your provided still doesn't include the error messages.
Can you double confirm that you are using Command Prompt to run the maven build command?
Can you please ensure you follow exactly the steps I mentioned in the build article?
https://kontext.tech/column/hadoop/378/compile-and-build-hadoop-321-on-windows-10-guide
And also you only need to paste the detailed error messages before the summary:
Here is the debug information but doesn't seem to be the error message.
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.828 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 6.380 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 3.688 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.993 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 1.303 s]
[INFO] Apache Hadoop Project Dist POM ..................... FAILURE [ 1.884 s]
[INFO] Apache Hadoop Maven Plugins ........................ SKIPPED
articleArticles 569
imageDiagrams 58
commentComments 299
loyaltyKontext Points 6279
From the logs you provided, you will at least encounter several errors:
- JAVA_HOME path has space. I recommend to install JDK in a path without space, for example: C:\Java
- C:\Users\NaseemuddinKhan\.m2\repository: Maven home path is too long; please create symbolic link as I mentioned in the other article about building Hadoop. This will lead to a very long file path for some Java libraries.
- Windows SDK version is lower than expected.
For some reason, the error message your provided still doesn't include the error messages.
Can you double confirm that you are using Command Prompt to run the maven build command?
Can you please ensure you follow exactly the steps I mentioned in the build article?
https://kontext.tech/column/hadoop/378/compile-and-build-hadoop-321-on-windows-10-guide
And also you only need to paste the detailed error messages before the summary:
Here is the debug information but doesn't seem to be the error message.
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.828 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 6.380 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 3.688 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.993 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 1.303 s]
[INFO] Apache Hadoop Project Dist POM ..................... FAILURE [ 1.884 s]
[INFO] Apache Hadoop Maven Plugins ........................ SKIPPED
person Naseemuddin access_time 4 years ago
Hi Raymond,
the last message before the summary is this
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
Do you think this is the main error? Why is the directory/file missing here?
Appreciate your help!
11-20 of 36
I'm glad it is working for you.