Haddop API

event 2023-03-20 visibility 112 comment 3

more_vert

Hi, I want to know if there is a hadoop API that can give me the details about the resources used by all the hadoop nodes: indeed, I want to know the total, used and available capacity for RAM, CPU and disk for all Hadoop nodes.

I tested this API "http://nm-http-address:port/ws/v1/node", but it sends me information that has no link with the real resources used by the machine and I don't know from what source this API gets this information.

Thanks for your answers

copyright This page is subject to Site terms.

More from Kontext

comment Comments

Kontext access_time 2 years ago link more_vert

Hello, can you please be more specific?

In my local instance, the information provided by the YARN web APIs are consistent with the one in the portal.

localhost:8042/ws/v1/node/info

<nodeInfo><healthReport/><totalVmemAllocatedContainersMB>17203</totalVmemAllocatedContainersMB><totalPmemAllocatedContainersMB>8192</totalPmemAllocatedContainersMB><totalVCoresAllocatedContainers>8</totalVCoresAllocatedContainers><vmemCheckEnabled>true</vmemCheckEnabled><pmemCheckEnabled>true</pmemCheckEnabled><lastNodeUpdateTime>1679444971058</lastNodeUpdateTime><resourceTypes>memory-mb (unit=Mi), vcores</resourceTypes><nodeHealthy>true</nodeHealthy><nodeManagerVersion>3.3.2</nodeManagerVersion><nodeManagerBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum df7cdcca4397b98895332a47fb6636</nodeManagerBuildVersion><nodeManagerVersionBuiltOn>2022-02-21T19:04Z</nodeManagerVersionBuiltOn><hadoopVersion>3.3.2</hadoopVersion><hadoopBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum 4b40fff8bb27201ba07b6fa5651217fb</hadoopBuildVersion><hadoopVersionBuiltOn>2022-02-21T18:39Z</hadoopVersionBuiltOn><id>localhost:33429</id><nodeHostName>localhost</nodeHostName><nmStartupTime>1679444009281</nmStartupTime></nodeInfo>

UI portal shows consistent memory allocated: