Raymond Tang

Hello,

I have been able to get around my need for admin it seems so far by changing my config so the tmp-nm folder is in my Documents versus in C drive directly in tmp.

However, it seems I still have some issues.   Two of them seem to point to wrong version of winutils.exe.   I am running windows 10 64 bit and am trying to get hadoop 3.2.1 running. One symtom of the wrong version is the repeated warning in Yarn node manager window over and over

WARN util.SysInfoWindows: Expected split length of sysInfo to be 11. Got 7

Another was the failure code of a job I submitted to insert data into a table from the hive prompt.  Job details were found in the Hadoop cluster local UI 

Application application_1589548856723_0001 failed 2 times due to AM Container for appattempt_1589548856723_0001_000002 exited with exitCode: 1639

Failing this attempt.Diagnostics: [2020-05-15 09:53:23.804]Exception from container-launch.

Container id: container_1589548856723_0001_02_000001

Exit code: 1639

Exception message: Incorrect command line arguments.

Shell output: Usage: task create [TASKNAME] [COMMAND_LINE] |

task isAlive [TASKNAME] |

task kill [TASKNAME]

task processList [TASKNAME]

Creates a new task jobobject with taskname

Checks if task jobobject is alive

Kills task jobobject

Prints to stdout a list of processes in the task

along with their resource usage. One process per line

and comma separated info per process

ProcessId,VirtualMemoryCommitted(bytes),

WorkingSetSize(bytes),CpuTime(Millisec,Kernel+User)

[2020-05-15 09:53:23.831]Container exited with a non-zero exit code 1639.

Some sites have said these two issues are symtom of having the wrong winutils.exe.

I have some other issues I'll wait to post after I can get these fixed.

I have used the link in this article to get winutils.exe.    I have also tried other winutils.exe's I find out there.  However, for the other ones I've tried when trying to start yarn, in the yarn node manager window it is full of errors like

2020-05-15 10:12:16,444 ERROR util.SysInfoWindows: java.io.IOException: Cannot run program "C:\Users\XXX\Documents\Big-Data\Hadoop\hadoop-3.2.1\bin\winutils.exe": CreateProcess error=216, This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher

So those ones are worse - I can't even get yarn started with those due to that error.  

So with the version I am using now I can get YARN to start although I get the warning about "WARN util.SysInfoWindows: Expected split length of sysInfo to be 11. Got 7" but the actual hive insert fails anyway... 

Appreciate the help.  How do I find or know if a winutil.exe is meant for windows 10 64 bit and Hadoop 3.2.1?

Hi Raymond,

Ok I got my work IT helpdesk to add my ID to the Create Symbolic Links directory. That worked fine. So I am now passed the exitCode=1: CreateSymbolicLink error (1314): A required privilege is not held by the client. error.

Now, it is throwing this error:

Application application_1589579676240_0001 failed 2 times due to AM Container for appattempt_1589579676240_0001_000002 exited with exitCode: 1

Failing this attempt.Diagnostics: [2020-05-15 17:57:08.681]Exception from container-launch.

Container id: container_1589579676240_0001_02_000001

Exit code: 1

Shell output: 1 file(s) moved.

"Setting up env variables"

"Setting up job resources"

"Copying debugging information"

C:\Users\V121119\Documents\Big-Data\tmp-nm\usercache\XXX\appcache\application_1589579676240_0001\container_1589579676240_0001_02_000001>rem Creating copy of launch script

C:\Users\V121119\Documents\Big-Data\tmp-nm\usercache\XXX\appcache\application_1589579676240_0001\container_1589579676240_0001_02_000001>copy "launch_container.cmd" "C:/Users/V121119/Documents/Big-Data/Hadoop/hadoop-3.2.1/logs/userlogs/application_1589579676240_0001/container_1589579676240_0001_02_000001/launch_container.cmd"

1 file(s) copied.

C:\Users\V121119\Documents\Big-Data\tmp-nm\usercache\XXX\appcache\application_1589579676240_0001\container_1589579676240_0001_02_000001>rem Determining directory contents

C:\Users\V121119\Documents\Big-Data\tmp-nm\usercache\XXX\appcache\application_1589579676240_0001\container_1589579676240_0001_02_000001>dir 1>>"C:/Users/XXX/Documents/Big-Data/Hadoop/hadoop-3.2.1/logs/userlogs/application_1589579676240_0001/container_1589579676240_0001_02_000001/directory.info"

"Launching container"

[2020-05-15 17:57:08.696]Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :

'C:\Program' is not recognized as an internal or external command,

operable program or batch file.

[2020-05-15 17:57:08.696]Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :

'C:\Program' is not recognized as an internal or external command,

operable program or batch file.

I think I know what is happening but don't know how to fix. One of my first errors was in hadoop.config.xml where it was trying to check if not exist %JAVA_HOME%\bin\java.exe but the problem is in path : C:\program files\Java\jre8  -Please note this directory "program files" has a space in it. The result was that I had to modify hadoop.config.xml  to put double quotes around the check - making it 

if not exist "%JAVA_HOME%\bin\java.exe" .  This resolved that problem.  Then another place I had found issue was on this line

for /f "delims=" %%A in ('%JAVA% -Xmx32m %HADOOP_JAVA_PLATFORM_OPTS% -classpath "%CLASSPATH%" org.apache.hadoop.util.PlatformName') do set JAVA_PLATFORM=%%A

This was messing up too for similar reason - it was erroring with similar error since some values in the classpath list were C:\program files\...  and once it hit the space it blew up.  For this line of code I just remarked it - since my HADOOP_JAVA_PLATFORM_OPTS is empty - I am not sure what I would have done had HADOOP_JAVA_PLATFORM_OPTS been populated. In any case, these were preliminary issues and all dealt with the fact that C:\program files... path was causing issues due to space.  Therefore when I saw this latest exception, I am assuming it too is hitting this at java path or some member of classpath that has the same... but not sure where to modify - how to work around.

As of 5/15 6:10pm EDT - this is my current issue - you may disregard the prior comments if you wish since they are resolved...   Thanks