Install Hadoop 3.3.0 on Windows 10 Step by Step Guide

#428 access_time 4 years ago more_vert

Hi Raymond,

the last message before the summary is this

[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory

Do you think this is the main error? Why is the directory/file missing here?

Appreciate your help!

format_quote

person Raymond access_time 4 years ago

Hi Naseemuddin,

I need to see the detailed error before I can suggest anything.

Can you run your maven command with option -X and paste the detailed error messages here? The detailed error message shows before the summary section.

reply Reply

#426 access_time 4 years ago more_vert

Hi Raymond,

thanks a lot for your reply! Here is the complete output. There are several errors. I hope you can find my mistake

[DEBUG] Configuring mojo 'org.codehaus.mojo:exec-maven-plugin:1.3.1:exec' with basic configurator -->
[DEBUG]   (f) arguments = [C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
[DEBUG]   (f) basedir = C:\hdp\hadoop\hadoop-project-dist
[DEBUG]   (f) classpathScope = runtime
[DEBUG]   (f) executable = bash
[DEBUG]   (f) failWithEmptyArgument = true
[DEBUG]   (f) failWithNullKeyOrValueInEnvironment = true
[DEBUG]   (f) longClasspath = false
[DEBUG]   (f) project = MavenProject: org.apache.hadoop:hadoop-project-dist:3.3.0 @ C:\hdp\hadoop\hadoop-project-dist\pom.xml
[DEBUG]   (f) skip = false
[DEBUG]   (f) workingDirectory = C:\hdp\hadoop\hadoop-project-dist\target
[DEBUG]   (f) session = org.apache.maven.execution.MavenSession@5edf2821
[DEBUG] -- end configuration --
[DEBUG] env: =C:=C:\hdp\hadoop
[DEBUG] env: =EXITCODE=00000000
[DEBUG] env: ALLUSERSPROFILE=C:\ProgramData
[DEBUG] env: APPDATA=C:\Users\NaseemuddinKhan\AppData\Roaming
[DEBUG] env: CLASSWORLDS_JAR="C:\apache-maven-3.6.3\bin\..\boot\plexus-classworlds-2.6.0.jar"
[DEBUG] env: CLASSWORLDS_LAUNCHER=org.codehaus.plexus.classworlds.launcher.Launcher
[DEBUG] env: COMMANDPROMPTTYPE=Native
[DEBUG] env: COMMONPROGRAMFILES=C:\Program Files\Common Files
[DEBUG] env: COMMONPROGRAMFILES(X86)=C:\Program Files (x86)\Common Files
[DEBUG] env: COMMONPROGRAMW6432=C:\Program Files\Common Files
[DEBUG] env: COMPUTERNAME=ADV075
[DEBUG] env: COMSPEC=C:\Windows\system32\cmd.exe
[DEBUG] env: DRIVERDATA=C:\Windows\System32\Drivers\DriverData
[DEBUG] env: ERROR_CODE=0
[DEBUG] env: EXEC_DIR=C:\hdp\hadoop
[DEBUG] env: FRAMEWORK35VERSION=v3.5
[DEBUG] env: FRAMEWORKDIR=C:\Windows\Microsoft.NET\Framework64
[DEBUG] env: FRAMEWORKDIR64=C:\Windows\Microsoft.NET\Framework64
[DEBUG] env: FRAMEWORKVERSION=v4.0.30319
[DEBUG] env: FRAMEWORKVERSION64=v4.0.30319
[DEBUG] env: GIT_HOME=C:\Program Files\Git
[DEBUG] env: HADOOP_HOME=C:\hadoop-3.3.0
[DEBUG] env: HIVE_HOME=C:\apache-hive-3.1.2-bin
[DEBUG] env: HOMEDRIVE=C:
[DEBUG] env: HOMEPATH=\Users\NaseemuddinKhan
[DEBUG] env: INCLUDE=c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\include;
[DEBUG] env: JAVACMD=C:\Java\jdk1.8.0_281\bin\java.exe
[DEBUG] env: JAVA_HOME=C:\Java\jdk1.8.0_281
[DEBUG] env: JVMCONFIG=\.mvn\jvm.config
[DEBUG] env: LIB=c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\LIB\amd64;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\ATLMFC\LIB\amd64;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\lib\x64;
[DEBUG] env: LIBPATH=C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Windows\Microsoft.NET\Framework64\v3.5;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\LIB\amd64;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\ATLMFC\LIB\amd64;
[DEBUG] env: LOCALAPPDATA=C:\Users\NaseemuddinKhan\AppData\Local
[DEBUG] env: LOGONSERVER=\\ADV075
[DEBUG] env: MAVEN_CMD_LINE_ARGS=package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true -X
[DEBUG] env: MAVEN_HOME=C:\apache-maven-3.6.3\bin\..
[DEBUG] env: MAVEN_PROJECTBASEDIR=C:\hdp\hadoop
[DEBUG] env: MSVS=C:\Program Files (x86)\Microsoft Visual Studio 10.0
[DEBUG] env: NUMBER_OF_PROCESSORS=8
[DEBUG] env: ONEDRIVE=C:\Users\NaseemuddinKhan\OneDrive - ADVISORI FTC GmbH
[DEBUG] env: ONEDRIVECOMMERCIAL=C:\Users\NaseemuddinKhan\OneDrive - ADVISORI FTC GmbH
[DEBUG] env: OS=Windows_NT
[DEBUG] env: PATH=c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\BIN\amd64;C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Windows\Microsoft.NET\Framework64\v3.5;c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\VCPackages;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools;C:\Program Files (x86)\HTML Help Workshop;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools\x64;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\x64;C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files\Python39\Scripts\;C:\Program Files\Python39\;C:\oraclexe\app\oracle\product\11.2.0\server\bin;C:\oraclexe\app\oracle\product\11.2.0\server\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files\PuTTY\;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;c:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;c:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Python39;C:\Java\jdk1.8.0_281\bin;C:\spark-3.0.1-bin-hadoop3.2\bin;C:\hadoop-3.3.0\bin;C:\hadoop-3.3.0\sbin;C:\apache-hive-3.1.2-bin\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;C:\apache-maven-3.6.3\bin;C:\protoc-2.5.0-win32;C:\Program Files\CMake\bin;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Users\NaseemuddinKhan\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Graphviz2.38\bin
[DEBUG] env: PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW
[DEBUG] env: PLATFORM=X64
[DEBUG] env: PROCESSOR_ARCHITECTURE=AMD64
[DEBUG] env: PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
[DEBUG] env: PROCESSOR_LEVEL=6
[DEBUG] env: PROCESSOR_REVISION=8e0a
[DEBUG] env: PROGRAMDATA=C:\ProgramData
[DEBUG] env: PROGRAMFILES=C:\Program Files
[DEBUG] env: PROGRAMFILES(X86)=C:\Program Files (x86)
[DEBUG] env: PROGRAMW6432=C:\Program Files
[DEBUG] env: PROMPT=$P$G
[DEBUG] env: PSMODULEPATH=C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
[DEBUG] env: PUBLIC=C:\Users\Public
[DEBUG] env: SPARK_HOME=C:\spark-3.0.1-bin-hadoop3.2
[DEBUG] env: SPARK_LOCAL_IP=127.0.0.1
[DEBUG] env: SYSTEMDRIVE=C:
[DEBUG] env: SYSTEMROOT=C:\Windows
[DEBUG] env: TEMP=C:\Users\NASEEM~1\AppData\Local\Temp
[DEBUG] env: TMP=C:\Users\NASEEM~1\AppData\Local\Temp
[DEBUG] env: USERDOMAIN=AzureAD
[DEBUG] env: USERDOMAIN_ROAMINGPROFILE=AzureAD
[DEBUG] env: USERNAME=NaseemuddinKhan
[DEBUG] env: USERPROFILE=C:\Users\NaseemuddinKhan
[DEBUG] env: VBOX_MSI_INSTALL_PATH=C:\Program Files\Oracle\VirtualBox\
[DEBUG] env: VCINSTALLDIR=c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\
[DEBUG] env: VCVARSPLAT=amd64
[DEBUG] env: VS100COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\
[DEBUG] env: VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 10.0\
[DEBUG] env: WDIR=C:\
[DEBUG] env: WINDIR=C:\Windows
[DEBUG] env: WINDOWSSDKDIR=C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\
[DEBUG] Executing command line: [bash, C:\hdp\hadoop\hadoop-project/../dev-support/bin/dist-copynativelibs, --version=3.3.0, --builddir=C:\hdp\hadoop\hadoop-project-dist\target, --artifactid=hadoop-project-dist, --isalbundle=false, --isallib=, --openssllib=, --opensslbinbundle=false, --openssllibbundle=false, --snappybinbundle=false, --snappylib=, --snappylibbundle=false, --zstdbinbundle=false, --zstdlib=, --zstdlibbundle=false]
/bin/bash: C:hdphadoophadoop-project/../dev-support/bin/dist-copynativelibs: No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.138 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  2.419 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.093 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.407 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.410 s]
[INFO] Apache Hadoop Project Dist POM ..................... FAILURE [  0.704 s]

format_quote

person Raymond access_time 4 years ago

Hi Naseemuddin,

I need to see the detailed error before I can suggest anything.

Can you run your maven command with option -X and paste the detailed error messages here? The detailed error message shows before the summary section.

reply Reply

#425 access_time 4 years ago more_vert

Hi Naseemuddin,

I need to see the detailed error before I can suggest anything.

Can you run your maven command with option -X and paste the detailed error messages here? The detailed error message shows before the summary section.

format_quote

person Naseemuddin access_time 4 years ago

Hi Raymond,

I have tried to follow your advice and build Hadoop according to your instructions. All I changed for 3.3.0 is to use Protocol Buffers 3.7.1 and of course check out branch rel/release-3.3.0. Unfortunately in the build process I am getting this error:

Can you help me with this?

reply Reply

#424 access_time 4 years ago more_vert

Hi Raymond,

I have tried to follow your advice and build Hadoop according to your instructions. All I changed for 3.3.0 is to use Protocol Buffers 3.7.1 and of course check out branch rel/release-3.3.0. Unfortunately in the build process I am getting this error:

Can you help me with this?

format_quote

person Raymond access_time 4 years ago

Hi, for the winutils I built for Hadoop 3.3.0, I didn't face any issue like that.

Hadoop 3.3.0 winutils

Can you try if the ones I published work for you?

If you want to build winutils by yourself, you need to complete these steps:

You can follow the commit in the PR to apply changes to your local Hadoop 3.3.0 repo: YARN-8246 winutils - fix failure to retrieve disk and network perf co… by pgoron · Pull Request #458 · apache/hadoop (github.com)
Refer to Compile and Build Hadoop 3.2.1 on Windows 10 Guide about how to build Hadoop on Windows 10. You need to change some of the dependencies (tools/frameworks) to the Hadoop 3.3.0 ones based on the Build.txt requirements of 3.3.0 release.

reply Reply

#423 access_time 4 years ago more_vert

Namenode is one of the most important daemon services in Hadoop. If it cannot start successfully, you won't be able to perform any other actions as other services rely on it.

format_quote

person Naseemuddin access_time 4 years ago

Thanks a lot for you reply, but this is a bit to complicated for me. What disatvantages am I going to have if this error remains there?

reply Reply

#421 access_time 4 years ago more_vert

Thanks a lot for you reply, but this is a bit to complicated for me. What disatvantages am I going to have if this error remains there?

format_quote

person Raymond access_time 4 years ago

Hi, for the winutils I built for Hadoop 3.3.0, I didn't face any issue like that.

Hadoop 3.3.0 winutils

Can you try if the ones I published work for you?

If you want to build winutils by yourself, you need to complete these steps:

You can follow the commit in the PR to apply changes to your local Hadoop 3.3.0 repo: YARN-8246 winutils - fix failure to retrieve disk and network perf co… by pgoron · Pull Request #458 · apache/hadoop (github.com)
Refer to Compile and Build Hadoop 3.2.1 on Windows 10 Guide about how to build Hadoop on Windows 10. You need to change some of the dependencies (tools/frameworks) to the Hadoop 3.3.0 ones based on the Build.txt requirements of 3.3.0 release.

reply Reply

#417 access_time 4 years ago more_vert

I have not tried install Hadoop on Windows 10 Home thus I'm not sure whether that is the case or not.

Technically as long as you can run run Java applications and also if it is 64bit (instead of 32bit), the instructions I provided should work.

format_quote

person Carl access_time 4 years ago

yes I did, the issue I think is I'm on windows Home

reply Reply

#416 access_time 4 years ago more_vert

Hi, for the winutils I built for Hadoop 3.3.0, I didn't face any issue like that.

Hadoop 3.3.0 winutils

Can you try if the ones I published work for you?

If you want to build winutils by yourself, you need to complete these steps:

You can follow the commit in the PR to apply changes to your local Hadoop 3.3.0 repo: YARN-8246 winutils - fix failure to retrieve disk and network perf co… by pgoron · Pull Request #458 · apache/hadoop (github.com)
Refer to Compile and Build Hadoop 3.2.1 on Windows 10 Guide about how to build Hadoop on Windows 10. You need to change some of the dependencies (tools/frameworks) to the Hadoop 3.3.0 ones based on the Build.txt requirements of 3.3.0 release.

format_quote

person Naseemuddin access_time 4 years ago

Hi,

Thanks a lot for this detailed installation guide!

I am getting this error in the yarn nodemanager and I am not sure how to fix it:

ERROR util.SysInfoWindows: ExitCodeException exitCode=1: PdhAddCounter \Network Interface(*)\Bytes Received/Sec failed with 0xc0000bb8.
Error in GetDiskAndNetwork. Err:1

There is a suggested sultion here but I do not know how to apply it:

https://github.com/apache/hadoop/pull/458

Any help would be appreciated

reply Reply

#415 access_time 4 years ago more_vert

Hi,

Thanks a lot for this detailed installation guide!

I am getting this error in the yarn nodemanager and I am not sure how to fix it:

ERROR util.SysInfoWindows: ExitCodeException exitCode=1: PdhAddCounter \Network Interface(*)\Bytes Received/Sec failed with 0xc0000bb8.
Error in GetDiskAndNetwork. Err:1

There is a suggested sultion here but I do not know how to apply it:

https://github.com/apache/hadoop/pull/458

Any help would be appreciated

reply Reply

#414 access_time 4 years ago more_vert

yes I did, the issue I think is I'm on windows Home

format_quote

person Raymond access_time 4 years ago

Have you downloaded the winutils native libraries (Step 3 of this article)? It seems is is not there or is not the correct version for your system.

Can you run the following commands:

winutils

It should print out the following text:

Usage: winutils [command] ...
Provide basic command line utilities for Hadoop on Windows.

The available commands and their usages are:

chmod          Change file mode bits.

Usage: chmod [OPTION] OCTAL-MODE [FILE]
   or: chmod [OPTION] MODE [FILE]
Change the mode of the FILE to MODE.

   -R: change files and directories recursively

Each MODE is of the form '[ugoa]*([-+=]([rwxX]*|[ugo]))+'.


chown          Change file owner.

Usage: chown [OWNER][:[GROUP]] [FILE]
Change the owner and/or group of the FILE to OWNER and/or GROUP.

Note:
On Linux, if a colon but no group name follows the user name, the group of
the files is changed to that user's login group. Windows has no concept of
a user's login group. So we do not change the group owner in this case.


groups         List user groups.
.....

reply Reply

arrow_back Install Hadoop 3.3.0 on Windows 10 Step by Step Guide

Log in with external accounts