Install Hadoop 3.0.0 in Windows (Single Node)

20,028 views 21 comments about 2 years ago Raymond Tang

Re: Install Hadoop 3.0.0 in Windows (Single Node)

Muha*** about 4 months ago

Hi,
I tried your guidelines for installing Hadoop 3.2.0 on my Windows 10 but unfortunately I am unable to start it. I am stuck at the point where you stated the command hadoop namenode -format . I am getting this message at the end of the command Shutting down NameNode at {Username}/192.168.0.200.
But no descriptive error other than that. Can you help me please?

Add comment

Comments (19)

Mo*** about 3 months ago

Thanks again. I finally installed hadoop on WLS but have problem on this article. in the step 

%HADOOP_HOME%\sbin\start-dfs.cmd

I get error "datanode.DataNode: Exception in secureMain" and there is no resolution over the web :(

Raym*** about 4 months ago

Hi, this post is about installing 3.0.0 and I have not tried 3.2.0 using this approach yet. Some configuration properties have changed. 

Your error message looks like that the Username was not configured correctly due to some reason. 

However, since you are using Windows 10, I'd suggest you to follow the post below to install 3.2.0 in Windows Subsystem for Linux (WSL). 

Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

Muha*** about 4 months ago

Hi,
I tried your guidelines for installing Hadoop 3.2.0 on my Windows 10 but unfortunately I am unable to start it. I am stuck at the point where you stated the command hadoop namenode -format . I am getting this message at the end of the command Shutting down NameNode at {Username}/192.168.0.200.
But no descriptive error other than that. Can you help me please?

Muha*** about 4 months ago

Hi,
I tried your guidelines for installing Hadoop 3.2.0 on my Windows 10 but unfortunately I am unable to start it. I am stuck at the point where you stated the command hadoop namenode -format . I am getting this message at the end of the command Shutting down NameNode at {Username}/192.168.0.200.
But no descriptive error other than that. Can you help me please?

Exit*** about 4 months ago

Very well written guide. Thanks for posting this. Hadoop on Windows can be a daunting task. I have had several issues while I install Hadoop 2.9 on my machine. It encouraged me to document the working steps and ended up writing this Blog Post.

https://exitcondition.com/install-hadoop-windows/

Thanks anyway for writing this. It helps a lot of learners out there.

Raym*** about 4 months ago

Is JAVA_HOME environment variable configured correctly in your system? The value should point to your JDK location.

th*** about 4 months ago


The system cannot find the path specified.

Error: JAVA_HOME is incorrectly set.

       Please update C:xxx\hadoop-2.9.1\etc\hadoop\hadoop-env.cmd

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

The system cannot find the path specified.

Error: JAVA_HOME is incorrectly set.

       Please update C:\xxxxxxx \etc\hadoop\hadoop-env.cmd


 Please update C:\xxxxxxx\hadoop-env.cmd


please help 



th*** about 4 months ago


The system cannot find the path specified.

Error: JAVA_HOME is incorrectly set.

       Please update C:xxx\hadoop-2.9.1\etc\hadoop\hadoop-env.cmd

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

The system cannot find the path specified.

Error: JAVA_HOME is incorrectly set.

       Please update C:\xxxxxxx \etc\hadoop\hadoop-env.cmd


 Please update C:\xxxxxxx\hadoop-env.cmd


please help 



Raym*** about 2 years ago

Nw, I'm glad it worked. :)

U-n5ay*** about 2 years ago

I am sorry. It works!

U-n5ay*** about 2 years ago

I am sorry. It works!

U-n5ay*** about 2 years ago

I come again and sorry. When I run this commande %HADOOP_HOME%\bin\hdfs dfs -put file:///G:/DataAnalytics/test.txt / I get an error: put: `G:/DataAnalytics/test.txt': No such file or directory. But I followed the configuration step by step and my DataAnalytics folder is in G:.

U-n5ay*** about 2 years ago

I come again and sorry. When I run this commande %HADOOP_HOME%\bin\hdfs dfs -put file:///G:/DataAnalytics/test.txt / I get an error: put: `G:/DataAnalytics/test.txt': No such file or directory. But I followed the configuration step by step and my DataAnalytics folder is in G:.

Raym*** about 2 years ago

Did you follow all the steps in this post? For example, you need to ensure winutils tool is installed: 

Overwrite your bin folder (%HADOOP_HOME%\bin) with the files from this link:

https://github.com/steveloughran/winutils/tree/master/hadoop-3.0.0/bin

The current available version is 3.0.0 and I am not very sure whether it can fix the issue for 3.0.1 but worth giving it a try.

U-n5ay*** about 2 years ago

Hi. thank you for your answer. I found a solution for my problem. If it can help someone, the problem was related to the syntax of my system username. It contains a space. So, to fixe it, you can edit /etc/hadoop/hadoop-env.cmd, at the end of this file, you will find set HADOOP_IDENT_STRING=%USERNAME% , change this with a string that you want but without space. For example: set HADOOP_IDENT_STRING=myuser, the problem will be fixed.

Raym*** about 2 years ago

Did you follow all the steps in this post? For example, you need to ensure winutils tool is installed: 

Overwrite your bin folder (%HADOOP_HOME%\bin) with the files from this link:

https://github.com/steveloughran/winutils/tree/master/hadoop-3.0.0/bin

The current available version is 3.0.0 and I am not very sure whether it can fix the issue for 3.0.1 but worth giving it a try.

Raym*** about 2 years ago

Did you follow all the steps in this post? For example, you need to ensure winutils tool is installed: 

Overwrite your bin folder (%HADOOP_HOME%\bin) with the files from this link:

https://github.com/steveloughran/winutils/tree/master/hadoop-3.0.0/bin

The current available version is 3.0.0 and I am not very sure whether it can fix the issue for 3.0.1 but worth giving it a try.

U-n5ay*** about 2 years ago

Good morning, I am trying to install hadoop 3.0.1 in my windows but when I want to test my configuration it gives me that error: Error: Can not find or load the main class. Can someone help me please?

Thank you

U-n5ay*** about 2 years ago

Good morning, I am trying to install hadoop 3.0.1 in my windows but when I want to test my configuration it gives me that error: Error: Can not find or load the main class. Can someone help me please?

Thank you

Raym*** about 2 years ago

@Neil S

No need to apology. I don't have one VM with Windows 7 so I could not verify for you.

I was about to suggest you to create one issue in the GitHub and then I just noticed you already did that.

https://github.com/steveloughran/winutils/issues/9 

Let's see how the author would response. 

The WinUtils project source code is available here:

https://github.com/apache/hadoop/tree/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-common-project/hadoop-common/src/main/winutils 

You may want to debug it in your system (if you have Visual Studio installed). More specifically, the source code for the command you are invoking is available here:

https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-common-project/hadoop-common/src/main/winutils/systeminfo.c

If I am using Windows 7, I could actually help you to debug.

Ne*** about 2 years ago

@Raymond Tang

I have to apologize, I missed the point that you mentioned Windows 10 as the work environment at the begining of the tutorial, so the tutorial is not applied to Windows 7 64 bit.

I'm really sorry about my mistake.

Ne*** about 2 years ago

@Raymond Tang

I have to apologize, I missed the point that you mentioned Windows 10 as the work environment at the begining of the tutorial, so the tutorial is not applied to Windows 7 64 bit.

I'm really sorry about my mistake.

Ne*** about 2 years ago

@Raymond Tang

If you ran the command on Windows 10 and you didn't find any problems, then it doesn't contradict with what I wrote and the tutorial is relevant for Windows 10, but if the command "winutils systeminfo" returns the error when running on Windows7 64 bit then it may be beneficial noticing some requirenments in the tutorial that Hadoop 3.0.0 can only be installed on Windows 10 at the moment untill the issue with winutils.exe is not solved for Windows7 64 bit, due to the reason that winutils.exe for Hadoop 3.0.0 still doesn't on Windows7 64 bit.

Ne*** about 2 years ago

@Raymond Tang

If you ran the command on Windows 10 and you didn't find any problems, then it doesn't contradict with what I wrote and the tutorial is relevant for Windows 10, but if the command "winutils systeminfo" returns the error when running on Windows7 64 bit then it may be beneficial noticing some requirenments in the tutorial that Hadoop 3.0.0 can only be installed on Windows 10 at the moment untill the issue with winutils.exe is not solved for Windows7 64 bit, due to the reason that winutils.exe for Hadoop 3.0.0 still doesn't on Windows7 64 bit.

Raym*** about 2 years ago

@Neil S

I have run the command in my system. I didn't get any issue.


Raym*** about 2 years ago

@Neil S

I have run the command in my system. I didn't get any issue.


Ne*** about 2 years ago

@Raymond Tang

I followed all the steps to setup. The problem is played back on Windows7 64bit. If you try issuing a command "winutils systeminfo" in the directory where winutils.exe for Hadoop 3.0.0 is located you will receive an error responce. This does not happen if you do the same with winutils for hadoop 2.6.4 or 2.7.x for instance but unfortunately this version of the executable file is not suitable for Hadoop 3.0.

Ne*** about 2 years ago

@Raymond Tang

I followed all the steps to setup. The problem is played back on Windows7 64bit. If you try issuing a command "winutils systeminfo" in the directory where winutils.exe for Hadoop 3.0.0 is located you will receive an error responce. This does not happen if you do the same with winutils for hadoop 2.6.4 or 2.7.x for instance but unfortunately this version of the executable file is not suitable for Hadoop 3.0.

Raym*** about 2 years ago

@Neil S.

The machine I am using is Windows 10 64 bit and it is working properly all the time.

Did you follow all the steps to setup?

Raym*** about 2 years ago

@Neil S.

The machine I am using is Windows 10 64 bit and it is working properly all the time.

Did you follow all the steps to setup?

Ne*** about 2 years ago

Unfortunately the issue with the missing winutils.exe in the binary distributive of Hadoop 3.0.0 cannot be solved. The file windows.exe downloaded from the above mentioned mentioned resource causes an error when run with the parameter systeminfo on Windows7 64 bit. The text of the error is PdhAddCounter \Network Interface(*)\Bytes Received/Sec failed with 0xc0000bb8.
Error in GetDiskAndNetwork. Err:1

Ne*** about 2 years ago

Unfortunately the issue with the missing winutils.exe in the binary distributive of Hadoop 3.0.0 cannot be solved. The file windows.exe downloaded from the above mentioned mentioned resource causes an error when run with the parameter systeminfo on Windows7 64 bit. The text of the error is PdhAddCounter \Network Interface(*)\Bytes Received/Sec failed with 0xc0000bb8.
Error in GetDiskAndNetwork. Err:1