Re: Install Hadoop 3.2.0 on Windows 10 using Windows Subsystem for Linux (WSL)

Raym*** about 10 days ago

Hi,

Can you add more details about the question? I am not sure whether I understand correctly. If you want to execute that command, you can directly run it in bash/Terminal.

https://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#fsck

Ravinder*** about 14 days ago

how to we execute "hdfs fsck" command, it's giving regarding the file system commands

Add comment

Comments (15)

Raym*** about 10 days ago

Hi,

Can you add more details about the question? I am not sure whether I understand correctly. If you want to execute that command, you can directly run it in bash/Terminal.

https://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#fsck

Ravinder*** about 14 days ago

how to we execute "hdfs fsck" command, it's giving regarding the file system commands

Ravinder*** about 14 days ago

how to we execute "hdfs fsck" command, it's giving regarding the file system commands

Raym*** about 2 months ago

Hi, please try this repository https://github.com/steveloughran/winutils

Unfortunately, this is only updated to Hadoop 3.0.0.

I'm not sure whether it works for 3.2.0. You can give it a try. Please pay attention to the README.md file about the author's declaration. 

Hado*** about 2 months ago

You mention "native Hadoop executable (winutils.exe) is not included in the official Hadoop distribution and needs to be downloaded separately [or built locally]."  Do you happen to know where this can be downloaded for 3.2.0?

Hado*** about 2 months ago

You mention "native Hadoop executable (winutils.exe) is not included in the official Hadoop distribution and needs to be downloaded separately [or built locally]."  Do you happen to know where this can be downloaded for 3.2.0?

Raym*** about 3 months ago

Thanks.  I also got the issue when I was writing the post. However I was able to resolve it by running the following two commands only:

sudo apt-get install ssh

sudo service ssh restart

Just in case other people cannot resolve the problem, I've updated the post to include the link to your comment so that they can follow that guide to resolve the connection issue.

Da*** about 3 months ago

This is a really cool blog.  I had the same issue with the ssh not working as others (connection closed), and was able to get past it by following the solution on this SE post: https://superuser.com/questions/1111591/how-can-i-ssh-into-bash-on-ubuntu-on-windows-10

Da*** about 3 months ago

This is a really cool blog.  I had the same issue with the ssh not working as others (connection closed), and was able to get past it by following the solution on this SE post: https://superuser.com/questions/1111591/how-can-i-ssh-into-bash-on-ubuntu-on-windows-10

Raym*** about 3 months ago

Have you tried the steps I mentioned in the post?

sudo apt-get install ssh

sudo service ssh restart

I'm not expert in network and I don't think the following solution will definitely help as they are all local traffics. I cannot reproduce this issue in my environment, so it will be hard to say where it goes wrong in your environment. 

There must be some other reasons that you cannot ssh localhost. For example, is port 22 used by your other programs? 

Can you also please try the same approach to allow ssh connections?

The websites won't start successfully until you resolve the ssh issue. So make sure you can ssh localhost first.

  • Protocol type: TCP
  • Local port: 22
  • Remote port: All Ports
  • Scope: make sure all your local IP addresses are added.
  • Profiles: Private. I'm choosing this one I will only connect to my wSL when connecting to private network. 

*** about 3 months ago

ssh localhost

Connection closed by 127.0.0.1 port 22


http://localhost:9870/dfshealth.html#tab-overview not working

Raym*** about 3 months ago

Have you tried the solution I mentioned in the post? I got the same issue when it is first installed but after the following commands, it work. And also make sure you stop and restart hadoop daemons. 

sudo apt-get install ssh

sudo service ssh restart

I'm not expert in network and I don't think the following solution will definitely help as they are all local traffics. There must be some other reasons that you cannot ssh localhost. For example, is port 22 used by your other programs?  Can you also use IPv4 addresses for localhost instead of the IPv6 one?

Can you try to add firewall rule to allow TCP traffic to ssh port 22?

  • Protocol type: TCP
  • Local port: 22
  • Remote port: All Ports
  • Scope: make sure all your local IP addresses are added.
  • Profiles: Private. I'm choosing this one I will only connect to my wSL when connecting to private network. 

jo*** about 3 months ago

I installed ssh and restarted it. Now 'ssh localhost' just says 'Connection closed by ::1 port 22.'

*** about 3 months ago

ssh localhost

Connection closed by 127.0.0.1 port 22


http://localhost:9870/dfshealth.html#tab-overview not working

jo*** about 3 months ago

I installed ssh and restarted it. Now 'ssh localhost' just says 'Connection closed by ::1 port 22.'

Raym*** about 4 months ago

You are welcome! I’m glad it helped.

Rudy*** about 4 months ago

This was a great and easy to follow post. THANK YOU!

Rudy*** about 4 months ago

This was a great and easy to follow post. THANK YOU!

Raym*** about 4 months ago

Hmm, sudo is actually not required if you install everything in your own folder though the better approach is to install them in other folders using root user (then sudo is required).

Mo*** about 4 months ago

I get Permission Denied when trying to get hadoop binary. after research I found that I need to use sudo in front of command. So need to use 

sudo wget http://mirrors.....


Thanks for great article!


Mo*** about 4 months ago

I get Permission Denied when trying to get hadoop binary. after research I found that I need to use sudo in front of command. So need to use 

sudo wget http://mirrors.....


Thanks for great article!


be*** about 4 months ago

In my case, the command: 

sbin/start-dfs.sh 

is executed without errors, but the NameNode is not started and therefore it is not responding on http://localhost:9870.

Executing jps command I can see how running processes are:

1) SecondaryNameNode

2) DataNode

3) Jps

NameNode process is missing from the returned list.


Any idea on what can be going wrong?

I followed all the instructions in this guide to configure my WSL environment.


Thanks