access_time 9 months ago languageEnglish
more_horiz

Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

visibility 3,434 comment 32
This detailed step-by-step guide shows you how to install the latest Hadoop v3.3.1 on Windows 10. It leverages Hadoop 3.3.1 winutils tool and WSL is not required. This version was released on June 15 2021.
info Last modified by Administrator 9 months ago
thumb_up 3

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

comment Comments
5 months ago link more_horiz
Raymond Raymond
web_assetArticles 586
imageDiagrams 41
forumThreads 9
commentComments 220
loyaltyKontext Points 6330
account_circleProfile
#1614 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

Can you please be more specific?

Once you’ve configured it, you can just use it as Hadoop instance. For example, you can use HDFS CLI to interact with it.

format_quote

person Super access_time 5 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

How do I use it with Hadoop?

5 months ago link more_horiz
S
Super
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 1
loyaltyKontext Points 1
5 months ago link more_horiz
Raymond Raymond
web_assetArticles 586
imageDiagrams 41
forumThreads 9
commentComments 220
loyaltyKontext Points 6330
account_circleProfile
#1604 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

It failed because the HDFS is not working probably because of the same error I mentioned earlier. Unfortunately I could not help you much as I don't have a Windows 11 system to test (my laptop CPU unfortunately is not supported). 

format_quote

person Arya access_time 5 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

Hi, 

I managed to get 64-bits winutils.exe and using this file instead you gave. 

the resource manager is on but the DFS are not. pls advise on mistake I did? thanks


STARTUP_MSG: java = 1.8.0_321 ************************************************************/ 2022-02-08 13:56:05,932 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2022-02-08 13:56:06,424 INFO checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/C:/hadoop-3.3.1/data/dfs/datanode331 2022-02-08 13:56:06,562 WARN checker.StorageLocationChecker: Exception checking StorageLocation [DISK]file:/C:/hadoop-3.3.1/data/dfs/datanode331 java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:793) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1215) at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:160) at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:142) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:239) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:52) at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:142) at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2022-02-08 13:56:06,564 ERROR datanode.DataNode: Exception in secureMain org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0 at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:233) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2841) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2754) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2798) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2942) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2966) 2022-02-08 13:56:06,568 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0 2022-02-08 13:56:06,571 INFO datanode.DataNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at ARYA-DUGI-LAPTOP/192.168.1.18 ************************************************************/ C:\hadoop-3.3.1\sbin>jps 2244 Jps 26232 ResourceManager C:\hadoop-3.3.1\sbin>

5 months ago link more_horiz
A
Arya
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 4
loyaltyKontext Points 4
#1601 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

Hi, 

I managed to get 64-bits winutils.exe and using this file instead you gave. 

the resource manager is on but the DFS are not. pls advise on mistake I did? thanks


STARTUP_MSG: java = 1.8.0_321 ************************************************************/ 2022-02-08 13:56:05,932 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2022-02-08 13:56:06,424 INFO checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/C:/hadoop-3.3.1/data/dfs/datanode331 2022-02-08 13:56:06,562 WARN checker.StorageLocationChecker: Exception checking StorageLocation [DISK]file:/C:/hadoop-3.3.1/data/dfs/datanode331 java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:793) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1215) at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:160) at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:142) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:239) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:52) at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:142) at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2022-02-08 13:56:06,564 ERROR datanode.DataNode: Exception in secureMain org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0 at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:233) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2841) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2754) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2798) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2942) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2966) 2022-02-08 13:56:06,568 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0 2022-02-08 13:56:06,571 INFO datanode.DataNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at ARYA-DUGI-LAPTOP/192.168.1.18 ************************************************************/ C:\hadoop-3.3.1\sbin>jps 2244 Jps 26232 ResourceManager C:\hadoop-3.3.1\sbin>

5 months ago link more_horiz
Raymond Raymond
web_assetArticles 586
imageDiagrams 41
forumThreads 9
commentComments 220
loyaltyKontext Points 6330
account_circleProfile
#1600 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

Looks like there is a compatible issue since the native libs were built for Windows 10. As I am not using Windows 11, I cannot really debug for you about this issue before I upgrade my system.

Can you try the following to see if it works?

Right click winutils.exe program and click Properties. Go to Compatibility tab and set Compatibility mode as Windows 10.

format_quote

person Arya access_time 5 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

tried to reinstall  the winutils from github, but still error like this



Error while running command to get file permissions : java.io.IOException: Cannot run program "C:\hadoop-3.3.1\bin\winutils.exe": CreateProcess error=216, This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher
5 months ago link more_horiz
A
Arya
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 4
loyaltyKontext Points 4
#1599 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

tried to reinstall  the winutils from github, but still error like this



Error while running command to get file permissions : java.io.IOException: Cannot run program "C:\hadoop-3.3.1\bin\winutils.exe": CreateProcess error=216, This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher
format_quote

person Raymond access_time 5 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

I have not tried installing this on Windows 11 thus wouldn't be able to provide accurate advice about this one. However, can you try winutils directly instead of winutils.exe? The path winutils.exe.exe in the error message is not right.

5 months ago link more_horiz
Raymond Raymond
web_assetArticles 586
imageDiagrams 41
forumThreads 9
commentComments 220
loyaltyKontext Points 6330
account_circleProfile
#1598 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

That usually means the installation was not successful and you will need to look into details to find out the actual error. Did you successfully complete all the steps before starting DFS and YARN?

format_quote

person Arya access_time 5 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

I always get my datanode and namenode shutdown after using comman startdfs.cmd and start-yarn.cmd. any advise how to settler?
Thanks


5 months ago link more_horiz
Raymond Raymond
web_assetArticles 586
imageDiagrams 41
forumThreads 9
commentComments 220
loyaltyKontext Points 6330
account_circleProfile
#1597 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

I have not tried installing this on Windows 11 thus wouldn't be able to provide accurate advice about this one. However, can you try winutils directly instead of winutils.exe? The path winutils.exe.exe in the error message is not right.

format_quote

person Arya access_time 5 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

Hi I am having issues with winutils.exe. pls advise

ARYA (arya.sanjaya@pradita.ac.id)

the returned on  command is: 


Program 'winutils.exe.exe' failed to run: The specified executable is not a valid application for this OS platform.At line:1 char:1 + winutils.exe +  CategoryInfo : ResourceUnavailable: (:) [], ApplicationFailedException + FullyQualifiedErrorId : NativeCommandFailed


system info: 
OS Name: Microsoft Windows 11 Pro OS Version: 10.0.22000 N/A Build 22000 OS Manufacturer: Microsoft Corporation OS Configuration: Standalone Workstation OS Build Type: Multiprocessor Free Registered Owner: arya.sanjaya@outlook.com Registered Organization: N/A Product ID: 00330-52275-85811-AAOEM Original Install Date: 11-Nov-21, 1:22:08 AM System Boot Time: 04-Feb-22, 1:54:58 PM System Manufacturer: Acer System Model: TravelMate P2410-G2-M System Type: x64-based PC Processor(s): 1 Processor(s) Installed. [01]: Intel64 Family 6 Model 142 Stepping 10 GenuineIntel ~1600 Mhz BIOS Version: Insyde Corp. V3.02, 12-Nov-18 Windows Directory: C:\WINDOWS System Directory: C:\WINDOWS\system32 Boot Device: \Device\HarddiskVolume1 System Locale: en-us;English (United States) Input Locale: en-us;English (United States) Time Zone: (UTC+07:00) Bangkok, Hanoi, Jakarta Total Physical Memory: 16,261 MB Available Physical Memory: 7,949 MB Virtual Memory: Max Size: 21,758 MB Virtual Memory: Available: 8,754 MB Virtual Memory: In Use: 13,004 MB Page File Location(s): C:\pagefile.sys Domain: WORKGROUP Logon Server: \\ARYA-DUGI-LAPTO Hotfix(s): 5 Hotfix(s) Installed. [01]: KB5008880 [02]: KB5004567 [03]: KB5008295 [04]: KB5009566 [05]: KB5007414 Network Card(s): 4 NIC(s) Installed. [01]: TAP-Windows Adapter V9 Connection Name: Ethernet 2 Status: Media disconnected [02]: Intel(R) Ethernet Connection I219-LM Connection Name: Ethernet Status: Media disconnected [03]: Intel(R) Dual Band Wireless-AC 7265 Connection Name: Wi-Fi DHCP Enabled: Yes DHCP Server: 192.168.1.1 IP address(es) [01]: 192.168.1.18 [02]: fe80::7c59:ef36:46a1:13b5 [04]: Bluetooth Device (Personal Area Network) Connection Name: Bluetooth Network Connection Status: Media disconnected Hyper-V Requirements: VM Monitor Mode Extensions: Yes Virtualization Enabled In Firmware: Yes Second Level Address Translation: Yes Data Execution Prevention Available: Yes


5 months ago link more_horiz
A
Arya
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 4
loyaltyKontext Points 4
#1596 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

I always get my datanode and namenode shutdown after using comman startdfs.cmd and start-yarn.cmd. any advise how to settler?
Thanks


5 months ago link more_horiz
A
Arya
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 4
loyaltyKontext Points 4
#1595 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide

Hi I am having issues with winutils.exe. pls advise

ARYA (arya.sanjaya@pradita.ac.id)

the returned on  command is: 


Program 'winutils.exe.exe' failed to run: The specified executable is not a valid application for this OS platform.At line:1 char:1 + winutils.exe +  CategoryInfo : ResourceUnavailable: (:) [], ApplicationFailedException + FullyQualifiedErrorId : NativeCommandFailed


system info: 
OS Name: Microsoft Windows 11 Pro OS Version: 10.0.22000 N/A Build 22000 OS Manufacturer: Microsoft Corporation OS Configuration: Standalone Workstation OS Build Type: Multiprocessor Free Registered Owner: arya.sanjaya@outlook.com Registered Organization: N/A Product ID: 00330-52275-85811-AAOEM Original Install Date: 11-Nov-21, 1:22:08 AM System Boot Time: 04-Feb-22, 1:54:58 PM System Manufacturer: Acer System Model: TravelMate P2410-G2-M System Type: x64-based PC Processor(s): 1 Processor(s) Installed. [01]: Intel64 Family 6 Model 142 Stepping 10 GenuineIntel ~1600 Mhz BIOS Version: Insyde Corp. V3.02, 12-Nov-18 Windows Directory: C:\WINDOWS System Directory: C:\WINDOWS\system32 Boot Device: \Device\HarddiskVolume1 System Locale: en-us;English (United States) Input Locale: en-us;English (United States) Time Zone: (UTC+07:00) Bangkok, Hanoi, Jakarta Total Physical Memory: 16,261 MB Available Physical Memory: 7,949 MB Virtual Memory: Max Size: 21,758 MB Virtual Memory: Available: 8,754 MB Virtual Memory: In Use: 13,004 MB Page File Location(s): C:\pagefile.sys Domain: WORKGROUP Logon Server: \\ARYA-DUGI-LAPTO Hotfix(s): 5 Hotfix(s) Installed. [01]: KB5008880 [02]: KB5004567 [03]: KB5008295 [04]: KB5009566 [05]: KB5007414 Network Card(s): 4 NIC(s) Installed. [01]: TAP-Windows Adapter V9 Connection Name: Ethernet 2 Status: Media disconnected [02]: Intel(R) Ethernet Connection I219-LM Connection Name: Ethernet Status: Media disconnected [03]: Intel(R) Dual Band Wireless-AC 7265 Connection Name: Wi-Fi DHCP Enabled: Yes DHCP Server: 192.168.1.1 IP address(es) [01]: 192.168.1.18 [02]: fe80::7c59:ef36:46a1:13b5 [04]: Bluetooth Device (Personal Area Network) Connection Name: Bluetooth Network Connection Status: Media disconnected Hyper-V Requirements: VM Monitor Mode Extensions: Yes Virtualization Enabled In Firmware: Yes Second Level Address Translation: Yes Data Execution Prevention Available: Yes


timeline Stats
Page index 14.61