Solution to fix the issue

If you have been running Hadoop on Windows machines, you may encounter issues about unhealthy data nodes.

Usually this will happen if there is no enough disk space in your local drive.

For example, if I start the HDFS and YARN demons under the context of C drive, the local temporary folders will be created in C drive.


C:\>%HADOOP_HOME%\sbin\start-dfs.cmd
C:\>%HADOOP_HOME%\sbin\start-yarn.cmd

By default, YARN will check the disk ratio and the default ratio is 90%. If your C drive has less than 10% space left (which is my case), YARN will report unhealthy nodes errors:

local-dirs have errors: [ /tmp/hadoop-fahao/nm-local-dir : Directory is not writable: mphadoop-fahao m-local-dir ]

*Your user name can be different from mine.

So to fix this problem, you can change YARN configuration to skip disk ratio check or increase the default ratio to 99%; alternatively you can also free up some space.

However, for my scenario, the issue is different as my Hadoop cluster is configured in F drive which has enough space. So if I start these daemons under the context of F drive and the issue is gone.

C:\WINDOWS\system32>cd /D F:

F:\>%HADOOP_HOME%\sbin\start-dfs.cmd

F:\>%HADOOP_HOME%\sbin\start-yarn.cmd
starting yarn daemons

I can also confirm that the temporary directories are now created in F drive:

image

For UNIX/Linux systems

Of course, you may encounter similar issues. To fix the issue, you need to ensure your local temporary folder has enough space, i.e. disk usage ratio is lower than YARN configured.

The default temporary folder is: /tmp/hadoop-{hdusername}/nm-local-dir.

*Replace {hdusername} with your user name.

info Last modified by Raymond at 2 years ago * This page is subject to Site terms.

More from Kontext

local_offer hadoop local_offer hive local_offer Java

visibility 477
thumb_up 1
access_time 3 months ago

When I was configuring Hive 3.0.0 in Hadoop 3.2.1 environment, I encountered the following error: Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V Ro...

open_in_new Hadoop

local_offer hdfs local_offer hadoop local_offer windows

visibility 398
thumb_up 0
access_time 5 months ago

Network Attached Storage are commonly used in many enterprises where files are stored remotely on those servers.  They typically provide access to files using network file sharing protocols such as  ...

open_in_new Hadoop

Fix for Hadoop 3.2.1 namenode format issue on Windows 10

local_offer windows10 local_offer hadoop local_offer hdfs

visibility 949
thumb_up 0
access_time 6 months ago

Issue When installing Hadoop 3.2.1 on Windows 10,  you may encounter the following error when trying to format HDFS  namnode: ERROR namenode.NameNode: Failed to start namenode. The error happens when running the following comm...

open_in_new Hadoop

Compile and Build Hadoop 3.2.1 on Windows 10 Guide

local_offer windows10 local_offer hadoop

visibility 1222
thumb_up 3
access_time 6 months ago

This article provides detailed steps about how to compile and build Hadoop (incl. native libs) on Windows 10. The following guide is based on Hadoop release 3.2.1. ...

open_in_new Hadoop

info About author

comment Comments (0)

comment Add comment

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts

No comments yet.

Dark theme mode

Dark theme mode is available on Kontext.

Learn more arrow_forward

Kontext Column

Created for everyone to publish data, programming and cloud related articles. Follow three steps to create your columns.


Learn more arrow_forward