Are you interested in Data Engineering Essentials course on Kontext? Learn more
web_assetArticles 583
imageDiagrams 40
forumThreads 9
commentComments 218
loyaltyKontext Points 6293
account_circle Profile
access_time 7 months ago
languageEnglish
more_vert
Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
visibility 3,131
comment 32
info Last modified by Administrator 8 months ago
thumb_up 3
Kontext Newsletter
Subscribe to Kontext Newsletter to get updates about data analytics, programming and cloud related articles. You can unsubscribe at anytime.
Log in with external accounts
comment Comments
1-10 of 32
3 months ago
link
more_vert
S
Super
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 1
loyaltyKontext Points 1
#1608 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
How do I use it with Hadoop?
4 months ago
link
more_vert
web_assetArticles 583
imageDiagrams 40
forumThreads 9
commentComments 218
loyaltyKontext Points 6293
account_circleProfile
#1604 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
It failed because the HDFS is not working probably because of the same error I mentioned earlier. Unfortunately I could not help you much as I don't have a Windows 11 system to test (my laptop CPU unfortunately is not supported).
person Arya access_time 4 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
Hi,
I managed to get 64-bits winutils.exe and using this file instead you gave.
the resource manager is on but the DFS are not. pls advise on mistake I did? thanks
STARTUP_MSG: java = 1.8.0_321
************************************************************/
2022-02-08 13:56:05,932 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-02-08 13:56:06,424 INFO checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/C:/hadoop-3.3.1/data/dfs/datanode331
2022-02-08 13:56:06,562 WARN checker.StorageLocationChecker: Exception checking StorageLocation [DISK]file:/C:/hadoop-3.3.1/data/dfs/datanode331
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:793)
at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1215)
at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:160)
at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:142)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116)
at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:239)
at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:52)
at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:142)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
2022-02-08 13:56:06,564 ERROR datanode.DataNode: Exception in secureMain
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0
at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:233)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2841)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2754)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2798)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2942)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2966)
2022-02-08 13:56:06,568 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0
2022-02-08 13:56:06,571 INFO datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at ARYA-DUGI-LAPTOP/192.168.1.18
************************************************************/
C:\hadoop-3.3.1\sbin>jps
2244 Jps
26232 ResourceManager
C:\hadoop-3.3.1\sbin>
4 months ago
link
more_vert
A
Arya
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 4
loyaltyKontext Points 4
#1601 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
Hi,
I managed to get 64-bits winutils.exe and using this file instead you gave.
the resource manager is on but the DFS are not. pls advise on mistake I did? thanks
STARTUP_MSG: java = 1.8.0_321
************************************************************/
2022-02-08 13:56:05,932 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-02-08 13:56:06,424 INFO checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/C:/hadoop-3.3.1/data/dfs/datanode331
2022-02-08 13:56:06,562 WARN checker.StorageLocationChecker: Exception checking StorageLocation [DISK]file:/C:/hadoop-3.3.1/data/dfs/datanode331
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:793)
at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1215)
at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:160)
at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:142)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116)
at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:239)
at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:52)
at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:142)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
2022-02-08 13:56:06,564 ERROR datanode.DataNode: Exception in secureMain
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0
at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:233)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2841)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2754)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2798)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2942)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2966)
2022-02-08 13:56:06,568 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0
2022-02-08 13:56:06,571 INFO datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at ARYA-DUGI-LAPTOP/192.168.1.18
************************************************************/
C:\hadoop-3.3.1\sbin>jps
2244 Jps
26232 ResourceManager
C:\hadoop-3.3.1\sbin>
4 months ago
link
more_vert
web_assetArticles 583
imageDiagrams 40
forumThreads 9
commentComments 218
loyaltyKontext Points 6293
account_circleProfile
#1600 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
Looks like there is a compatible issue since the native libs were built for Windows 10. As I am not using Windows 11, I cannot really debug for you about this issue before I upgrade my system.
Can you try the following to see if it works?
Right click winutils.exe program and click Properties. Go to Compatibility tab and set Compatibility mode as Windows 10.
person Arya access_time 4 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
tried to reinstall the winutils from github, but still error like this
Error while running command to get file permissions : java.io.IOException: Cannot run program "C:\hadoop-3.3.1\bin\winutils.exe": CreateProcess error=216, This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher
4 months ago
link
more_vert
A
Arya
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 4
loyaltyKontext Points 4
#1599 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
tried to reinstall the winutils from github, but still error like this
Error while running command to get file permissions : java.io.IOException: Cannot run program "C:\hadoop-3.3.1\bin\winutils.exe": CreateProcess error=216, This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher
person Raymond access_time 4 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
I have not tried installing this on Windows 11 thus wouldn't be able to provide accurate advice about this one. However, can you try winutils directly instead of winutils.exe? The path winutils.exe.exe in the error message is not right.
4 months ago
link
more_vert
web_assetArticles 583
imageDiagrams 40
forumThreads 9
commentComments 218
loyaltyKontext Points 6293
account_circleProfile
#1598 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
That usually means the installation was not successful and you will need to look into details to find out the actual error. Did you successfully complete all the steps before starting DFS and YARN?
person Arya access_time 4 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
I always get my datanode and namenode shutdown after using comman startdfs.cmd and start-yarn.cmd. any advise how to settler?
Thanks
4 months ago
link
more_vert
web_assetArticles 583
imageDiagrams 40
forumThreads 9
commentComments 218
loyaltyKontext Points 6293
account_circleProfile
#1597 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
I have not tried installing this on Windows 11 thus wouldn't be able to provide accurate advice about this one. However, can you try winutils directly instead of winutils.exe? The path winutils.exe.exe in the error message is not right.
person Arya access_time 4 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
Hi I am having issues with winutils.exe. pls advise
ARYA (arya.sanjaya@pradita.ac.id)
the returned on command is:
Program 'winutils.exe.exe' failed to run: The specified executable is not a valid application for this OS platform.At
line:1 char:1
+ winutils.exe
+ CategoryInfo : ResourceUnavailable: (:) [], ApplicationFailedException
+ FullyQualifiedErrorId : NativeCommandFailed
system info:
OS Name: Microsoft Windows 11 Pro
OS Version: 10.0.22000 N/A Build 22000
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Registered Owner: arya.sanjaya@outlook.com
Registered Organization: N/A
Product ID: 00330-52275-85811-AAOEM
Original Install Date: 11-Nov-21, 1:22:08 AM
System Boot Time: 04-Feb-22, 1:54:58 PM
System Manufacturer: Acer
System Model: TravelMate P2410-G2-M
System Type: x64-based PC
Processor(s): 1 Processor(s) Installed.
[01]: Intel64 Family 6 Model 142 Stepping 10 GenuineIntel ~1600 Mhz
BIOS Version: Insyde Corp. V3.02, 12-Nov-18
Windows Directory: C:\WINDOWS
System Directory: C:\WINDOWS\system32
Boot Device: \Device\HarddiskVolume1
System Locale: en-us;English (United States)
Input Locale: en-us;English (United States)
Time Zone: (UTC+07:00) Bangkok, Hanoi, Jakarta
Total Physical Memory: 16,261 MB
Available Physical Memory: 7,949 MB
Virtual Memory: Max Size: 21,758 MB
Virtual Memory: Available: 8,754 MB
Virtual Memory: In Use: 13,004 MB
Page File Location(s): C:\pagefile.sys
Domain: WORKGROUP
Logon Server: \\ARYA-DUGI-LAPTO
Hotfix(s): 5 Hotfix(s) Installed.
[01]: KB5008880
[02]: KB5004567
[03]: KB5008295
[04]: KB5009566
[05]: KB5007414
Network Card(s): 4 NIC(s) Installed.
[01]: TAP-Windows Adapter V9
Connection Name: Ethernet 2
Status: Media disconnected
[02]: Intel(R) Ethernet Connection I219-LM
Connection Name: Ethernet
Status: Media disconnected
[03]: Intel(R) Dual Band Wireless-AC 7265
Connection Name: Wi-Fi
DHCP Enabled: Yes
DHCP Server: 192.168.1.1
IP address(es)
[01]: 192.168.1.18
[02]: fe80::7c59:ef36:46a1:13b5
[04]: Bluetooth Device (Personal Area Network)
Connection Name: Bluetooth Network Connection
Status: Media disconnected
Hyper-V Requirements: VM Monitor Mode Extensions: Yes
Virtualization Enabled In Firmware: Yes
Second Level Address Translation: Yes
Data Execution Prevention Available: Yes
4 months ago
link
more_vert
A
Arya
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 4
loyaltyKontext Points 4
#1596 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
I always get my datanode and namenode shutdown after using comman startdfs.cmd and start-yarn.cmd. any advise how to settler?
Thanks
4 months ago
link
more_vert
A
Arya
web_assetArticles 0
imageDiagrams 0
forumThreads 0
commentComments 4
loyaltyKontext Points 4
#1595 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
Hi I am having issues with winutils.exe. pls advise
ARYA (arya.sanjaya@pradita.ac.id)
the returned on command is:
Program 'winutils.exe.exe' failed to run: The specified executable is not a valid application for this OS platform.At
line:1 char:1
+ winutils.exe
+ CategoryInfo : ResourceUnavailable: (:) [], ApplicationFailedException
+ FullyQualifiedErrorId : NativeCommandFailed
system info:
OS Name: Microsoft Windows 11 Pro
OS Version: 10.0.22000 N/A Build 22000
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Registered Owner: arya.sanjaya@outlook.com
Registered Organization: N/A
Product ID: 00330-52275-85811-AAOEM
Original Install Date: 11-Nov-21, 1:22:08 AM
System Boot Time: 04-Feb-22, 1:54:58 PM
System Manufacturer: Acer
System Model: TravelMate P2410-G2-M
System Type: x64-based PC
Processor(s): 1 Processor(s) Installed.
[01]: Intel64 Family 6 Model 142 Stepping 10 GenuineIntel ~1600 Mhz
BIOS Version: Insyde Corp. V3.02, 12-Nov-18
Windows Directory: C:\WINDOWS
System Directory: C:\WINDOWS\system32
Boot Device: \Device\HarddiskVolume1
System Locale: en-us;English (United States)
Input Locale: en-us;English (United States)
Time Zone: (UTC+07:00) Bangkok, Hanoi, Jakarta
Total Physical Memory: 16,261 MB
Available Physical Memory: 7,949 MB
Virtual Memory: Max Size: 21,758 MB
Virtual Memory: Available: 8,754 MB
Virtual Memory: In Use: 13,004 MB
Page File Location(s): C:\pagefile.sys
Domain: WORKGROUP
Logon Server: \\ARYA-DUGI-LAPTO
Hotfix(s): 5 Hotfix(s) Installed.
[01]: KB5008880
[02]: KB5004567
[03]: KB5008295
[04]: KB5009566
[05]: KB5007414
Network Card(s): 4 NIC(s) Installed.
[01]: TAP-Windows Adapter V9
Connection Name: Ethernet 2
Status: Media disconnected
[02]: Intel(R) Ethernet Connection I219-LM
Connection Name: Ethernet
Status: Media disconnected
[03]: Intel(R) Dual Band Wireless-AC 7265
Connection Name: Wi-Fi
DHCP Enabled: Yes
DHCP Server: 192.168.1.1
IP address(es)
[01]: 192.168.1.18
[02]: fe80::7c59:ef36:46a1:13b5
[04]: Bluetooth Device (Personal Area Network)
Connection Name: Bluetooth Network Connection
Status: Media disconnected
Hyper-V Requirements: VM Monitor Mode Extensions: Yes
Virtualization Enabled In Firmware: Yes
Second Level Address Translation: Yes
Data Execution Prevention Available: Yes
1-10 of 32
#1614 Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
Can you please be more specific?
Once you’ve configured it, you can just use it as Hadoop instance. For example, you can use HDFS CLI to interact with it.
person Super access_time 3 months ago
Re: Install Hadoop 3.3.1 on Windows 10 Step by Step Guide
How do I use it with Hadoop?