By using this site, you acknowledge that you have read and understand our Cookie policy, Privacy policy and Terms .
close

Articles about Apache Hadoop installation, performance tuning and general tutorials.

rss_feed Subscribe RSS

Solution to fix the issue

If you have been running Hadoop on Windows machines, you may encounter issues about unhealthy data nodes.

Usually this will happen if there is no enough disk space in your local drive.

For example, if I start the HDFS and YARN demons under the context of C drive, the local temporary folders will be created in C drive.


C:\>%HADOOP_HOME%\sbin\start-dfs.cmd
C:\>%HADOOP_HOME%\sbin\start-yarn.cmd

By default, YARN will check the disk ratio and the default ratio is 90%. If your C drive has less than 10% space left (which is my case), YARN will report unhealthy nodes errors:

local-dirs have errors: [ /tmp/hadoop-fahao/nm-local-dir : Directory is not writable: mphadoop-fahao m-local-dir ]

*Your user name can be different from mine.

So to fix this problem, you can change YARN configuration to skip disk ratio check or increase the default ratio to 99%; alternatively you can also free up some space.

However, for my scenario, the issue is different as my Hadoop cluster is configured in F drive which has enough space. So if I start these daemons under the context of F drive and the issue is gone.

C:\WINDOWS\system32>cd /D F:

F:\>%HADOOP_HOME%\sbin\start-dfs.cmd

F:\>%HADOOP_HOME%\sbin\start-yarn.cmd
starting yarn daemons

I can also confirm that the temporary directories are now created in F drive:

image

For UNIX/Linux systems

Of course, you may encounter similar issues. To fix the issue, you need to ensure your local temporary folder has enough space, i.e. disk usage ratio is lower than YARN configured.

The default temporary folder is: /tmp/hadoop-{hdusername}/nm-local-dir.

*Replace {hdusername} with your user name.

info Last modified by Raymond at 11 months ago
info About author

info License/Terms

More from Kontext

local_offer windows10 local_offer hadoop local_offer hdfs

visibility 8
comment 0
thumb_up 0
access_time 1 day ago

Issue When installing Hadoop 3.2.1 on Windows 10,  you may encounter the following error when trying to format HDFS  namnode: ERROR namenode.NameNode: Failed to start namenode. The error happens when running the following comm...

open_in_new View

Compile and Build Hadoop 3.2.1 on Windows 10 Guide

local_offer windows10 local_offer hadoop

visibility 71
comment 0
thumb_up 1
access_time 6 days ago

This article provides detailed steps about how to compile and build Hadoop (incl. native libs) on Windows 10. The following guide is based on Hadoop release 3.2.1. ...

open_in_new View

Latest Hadoop 3.2.1 Installation on Windows 10 Step by Step Guide

local_offer windows10 local_offer hadoop local_offer yarn

visibility 71
comment 0
thumb_up 1
access_time 7 days ago

This detailed step-by-step guide shows you how to install the latest Hadoop (v3.2.1) on Windows 10. It also provides a temporary fix for bug HDFS-14084 (java.lang.UnsupportedOperationException INFO).

open_in_new View

local_offer spark local_offer hadoop local_offer pyspark local_offer oozie local_offer hue

visibility 869
comment 0
thumb_up 0
access_time 6 months ago

When submitting Spark applications to YARN cluster, two deploy modes can be used: client and cluster. For client mode (default), Spark driver runs on the machine that the Spark application was submitted while for cluster mode, the driver runs on a random node in a cluster. On this page, I am goin...

open_in_new View