Haddop API
insights Stats
Hi, I want to know if there is a hadoop API that can give me the details about the resources used by all the hadoop nodes: indeed, I want to know the total, used and available capacity for RAM, CPU and disk for all Hadoop nodes.
I tested this API "http://nm-http-address:port/ws/v1/node", but it sends me information that has no link with the real resources used by the machine and I don't know from what source this API gets this information.
Thanks for your answers
person ameni access_time 2 years ago
Thanks for your reply.
When I tested this API, I got the same values as you: total memory= 8G and total vcores=8, while the machine only has 2G of memory and 2 vcores.
Do you have any explications.
Thanks in advance.
Thanks for your reply.
When I tested this API, I got the same values as you: total memory= 8G and total vcores=8, while the machine only has 2G of memory and 2 vcores.
Do you have any explications.
Thanks in advance.
person Kontext access_time 2 years ago
Hello, can you please be more specific?
In my local instance, the information provided by the YARN web APIs are consistent with the one in the portal.
localhost:8042/ws/v1/node/info
<nodeInfo><healthReport/><totalVmemAllocatedContainersMB>17203</totalVmemAllocatedContainersMB><totalPmemAllocatedContainersMB>8192</totalPmemAllocatedContainersMB><totalVCoresAllocatedContainers>8</totalVCoresAllocatedContainers><vmemCheckEnabled>true</vmemCheckEnabled><pmemCheckEnabled>true</pmemCheckEnabled><lastNodeUpdateTime>1679444971058</lastNodeUpdateTime><resourceTypes>memory-mb (unit=Mi), vcores</resourceTypes><nodeHealthy>true</nodeHealthy><nodeManagerVersion>3.3.2</nodeManagerVersion><nodeManagerBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum df7cdcca4397b98895332a47fb6636</nodeManagerBuildVersion><nodeManagerVersionBuiltOn>2022-02-21T19:04Z</nodeManagerVersionBuiltOn><hadoopVersion>3.3.2</hadoopVersion><hadoopBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum 4b40fff8bb27201ba07b6fa5651217fb</hadoopBuildVersion><hadoopVersionBuiltOn>2022-02-21T18:39Z</hadoopVersionBuiltOn><id>localhost:33429</id><nodeHostName>localhost</nodeHostName><nmStartupTime>1679444009281</nmStartupTime></nodeInfo>UI portal shows consistent memory allocated:
Hello, can you please be more specific?
In my local instance, the information provided by the YARN web APIs are consistent with the one in the portal.
localhost:8042/ws/v1/node/info
<nodeInfo><healthReport/><totalVmemAllocatedContainersMB>17203</totalVmemAllocatedContainersMB><totalPmemAllocatedContainersMB>8192</totalPmemAllocatedContainersMB><totalVCoresAllocatedContainers>8</totalVCoresAllocatedContainers><vmemCheckEnabled>true</vmemCheckEnabled><pmemCheckEnabled>true</pmemCheckEnabled><lastNodeUpdateTime>1679444971058</lastNodeUpdateTime><resourceTypes>memory-mb (unit=Mi), vcores</resourceTypes><nodeHealthy>true</nodeHealthy><nodeManagerVersion>3.3.2</nodeManagerVersion><nodeManagerBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum df7cdcca4397b98895332a47fb6636</nodeManagerBuildVersion><nodeManagerVersionBuiltOn>2022-02-21T19:04Z</nodeManagerVersionBuiltOn><hadoopVersion>3.3.2</hadoopVersion><hadoopBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum 4b40fff8bb27201ba07b6fa5651217fb</hadoopBuildVersion><hadoopVersionBuiltOn>2022-02-21T18:39Z</hadoopVersionBuiltOn><id>localhost:33429</id><nodeHostName>localhost</nodeHostName><nmStartupTime>1679444009281</nmStartupTime></nodeInfo>UI portal shows consistent memory allocated:
I believe it is based on the default configurations. You can play around with those configuration properties to see if they change.
Apache Hadoop 3.1.1 – Hadoop: YARN Resource Configuration