visibility 31,781 comment 18 access_time 4 years ago languageEnglish
more_vert
Raymond Raymond
articleArticles 549
imageDiagrams 49
codeCode 3
chat_bubble_outlineThreads 8
commentComments 265
loyaltyKontext Points 6055
account_circleProfile

Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

In my previous post , I showed how to configure a single node Hadoop instance on Windows 10. The steps are not too difficult to follow if you have Java programming background. However there is one step that is not very straightforward: native Hadoop executable (winutils.exe) is not included in the ...
info Last modified by Administrator 9 months ago
thumb_up 20

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

comment Comments
2 years ago link more_vert
Raymond Raymond
articleArticles 549
imageDiagrams 49
codeCode 3
chat_bubble_outlineThreads 8
commentComments 265
loyaltyKontext Points 6055
account_circleProfile
#436 Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

It seems that your config XML file encoding is not correct or file content is not complete.
Can you please make sure the binary package is downloaded successfully and also all the content is extracted properly?

You should be able to see the following content in Hadoop folder:

~/hadoop/hadoop-3.2.0$ ls
LICENSE.txt  NOTICE.txt  README.txt  bin  etc  include  lib  libexec  logs  sbin  share

BTW, from the screenshot, I can see your Hadoop version is 3.2.2 instead of 3.2.0. Technically that should be a problem but I have not tested Hadoop 3.2.2 on WSL.

format_quote

person Hemil access_time 2 years ago
Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)


bin/hdfs namenode -format

I am getting the error shown in above image when running the above command. 

I have successfully ssh to localhost. 

And the other problem is I can successfully enter into hadoop directory by typing cd hadoop but when i try to do "ls" i can not see the hadoop directory in the list.

2 years ago link more_vert
H Hemil
articleArticles 0
imageDiagrams 0
codeCode 0
chat_bubble_outlineThreads 0
commentComments 1
loyaltyKontext Points 1
#435 Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)


bin/hdfs namenode -format

I am getting the error shown in above image when running the above command. 

I have successfully ssh to localhost. 

And the other problem is I can successfully enter into hadoop directory by typing cd hadoop but when i try to do "ls" i can not see the hadoop directory in the list.

4 years ago link more_vert
Raymond Raymond
articleArticles 549
imageDiagrams 49
codeCode 3
chat_bubble_outlineThreads 8
commentComments 265
loyaltyKontext Points 6055
account_circleProfile
#239 Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

Have you tried the steps I mentioned in the post?

sudo apt-get install ssh

sudo service ssh restart

I'm not expert in network and I don't think the following solution will definitely help as they are all local traffics. I cannot reproduce this issue in my environment, so it will be hard to say where it goes wrong in your environment. 

There must be some other reasons that you cannot ssh localhost. For example, is port 22 used by your other programs? 

Can you also please try the same approach to allow ssh connections?

The websites won't start successfully until you resolve the ssh issue. So make sure you can ssh localhost first.

  • Protocol type: TCP
  • Local port: 22
  • Remote port: All Ports
  • Scope: make sure all your local IP addresses are added.
  • Profiles: Private. I'm choosing this one I will only connect to my wSL when connecting to private network. 
format_quote

person mj access_time 4 years ago
Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

ssh localhost

Connection closed by 127.0.0.1 port 22


http://localhost:9870/dfshealth.html#tab-overview not working

4 years ago link more_vert
Raymond Raymond
articleArticles 549
imageDiagrams 49
codeCode 3
chat_bubble_outlineThreads 8
commentComments 265
loyaltyKontext Points 6055
account_circleProfile
#238 Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

Have you tried the solution I mentioned in the post? I got the same issue when it is first installed but after the following commands, it work. And also make sure you stop and restart hadoop daemons. 

sudo apt-get install ssh

sudo service ssh restart

I'm not expert in network and I don't think the following solution will definitely help as they are all local traffics. There must be some other reasons that you cannot ssh localhost. For example, is port 22 used by your other programs?  Can you also use IPv4 addresses for localhost instead of the IPv6 one?

Can you try to add firewall rule to allow TCP traffic to ssh port 22?

  • Protocol type: TCP
  • Local port: 22
  • Remote port: All Ports
  • Scope: make sure all your local IP addresses are added.
  • Profiles: Private. I'm choosing this one I will only connect to my wSL when connecting to private network. 
format_quote

person joe access_time 4 years ago
Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

I installed ssh and restarted it. Now 'ssh localhost' just says 'Connection closed by ::1 port 22.'

4 years ago link more_vert
Raymond Raymond
articleArticles 549
imageDiagrams 49
codeCode 3
chat_bubble_outlineThreads 8
commentComments 265
loyaltyKontext Points 6055
account_circleProfile
#237 Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)
You are welcome! I’m glad it helped.
format_quote

person Rudy Layedra access_time 4 years ago
Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

This was a great and easy to follow post. THANK YOU!

4 years ago link more_vert
Raymond Raymond
articleArticles 549
imageDiagrams 49
codeCode 3
chat_bubble_outlineThreads 8
commentComments 265
loyaltyKontext Points 6055
account_circleProfile
#236 Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)
Hmm, sudo is actually not required if you install everything in your own folder though the better approach is to install them in other folders using root user (then sudo is required).
format_quote

person Mohammad access_time 4 years ago
Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

I get Permission Denied when trying to get hadoop binary. after research I found that I need to use sudo in front of command. So need to use 

sudo wget http://mirrors.....


Thanks for great article!


4 years ago link more_vert
Raymond Raymond
articleArticles 549
imageDiagrams 49
codeCode 3
chat_bubble_outlineThreads 8
commentComments 265
loyaltyKontext Points 6055
account_circleProfile
#233 Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

Hi,

Can you add more details about the question? I am not sure whether I understand correctly. If you want to execute that command, you can directly run it in bash/Terminal.

https://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#fsck

format_quote

person Ravinder Pal Singh Saluja access_time 4 years ago
Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

how to we execute "hdfs fsck" command, it's giving regarding the file system commands

4 years ago link more_vert
Raymond Raymond
articleArticles 549
imageDiagrams 49
codeCode 3
chat_bubble_outlineThreads 8
commentComments 265
loyaltyKontext Points 6055
account_circleProfile
#232 Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

Hi, please try this repository https://github.com/steveloughran/winutils

Unfortunately, this is only updated to Hadoop 3.0.0.

I'm not sure whether it works for 3.2.0. You can give it a try. Please pay attention to the README.md file about the author's declaration. 

format_quote

person Hadoop Newbie access_time 4 years ago
Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

You mention "native Hadoop executable (winutils.exe) is not included in the official Hadoop distribution and needs to be downloaded separately [or built locally]."  Do you happen to know where this can be downloaded for 3.2.0?

4 years ago link more_vert
Raymond Raymond
articleArticles 549
imageDiagrams 49
codeCode 3
chat_bubble_outlineThreads 8
commentComments 265
loyaltyKontext Points 6055
account_circleProfile
#231 Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

Thanks.  I also got the issue when I was writing the post. However I was able to resolve it by running the following two commands only:

sudo apt-get install ssh

sudo service ssh restart

Just in case other people cannot resolve the problem, I've updated the post to include the link to your comment so that they can follow that guide to resolve the connection issue.

format_quote

person David access_time 4 years ago
Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

This is a really cool blog.  I had the same issue with the ssh not working as others (connection closed), and was able to get past it by following the solution on this SE post: https://superuser.com/questions/1111591/how-can-i-ssh-into-bash-on-ubuntu-on-windows-10
recommendMore from Kontext