A ameni kallel | Big Data Forum

Haddop API

event 2023-03-20 visibility 90 comment 3 insights
more_vert
insights Stats

Hi, I want to know if there is a hadoop API that can give me the details about the resources used by all the hadoop nodes: indeed, I want to know the total, used and available capacity for RAM, CPU and disk for all Hadoop nodes.

I tested this API "http://nm-http-address:port/ws/v1/node", but it sends me information that has no link with the real resources used by the machine and I don't know from what source this API gets this information.

Thanks for your answers

More from Kontext
comment Comments
Raymond Raymond #1812 access_time 2 years ago more_vert

I believe it is based on the default configurations.  You can play around with those configuration properties to see if they change.


Apache Hadoop 3.1.1 – Hadoop: YARN Resource Configuration

format_quote

person ameni access_time 2 years ago

Thanks for your reply.

When I tested this API, I got the same values as you: total memory= 8G and total vcores=8, while the machine only has 2G of memory and 2 vcores.

Do you have any explications.

Thanks in advance.

A ameni kallel #1808 access_time 2 years ago more_vert

Thanks for your reply.

When I tested this API, I got the same values as you: total memory= 8G and total vcores=8, while the machine only has 2G of memory and 2 vcores.

Do you have any explications.

Thanks in advance.

format_quote

person Kontext access_time 2 years ago

Hello, can you please be more specific?

In my local instance, the information provided by the YARN web APIs are consistent with the one in the portal.

localhost:8042/ws/v1/node/info

<nodeInfo><healthReport/><totalVmemAllocatedContainersMB>17203</totalVmemAllocatedContainersMB><totalPmemAllocatedContainersMB>8192</totalPmemAllocatedContainersMB><totalVCoresAllocatedContainers>8</totalVCoresAllocatedContainers><vmemCheckEnabled>true</vmemCheckEnabled><pmemCheckEnabled>true</pmemCheckEnabled><lastNodeUpdateTime>1679444971058</lastNodeUpdateTime><resourceTypes>memory-mb (unit=Mi), vcores</resourceTypes><nodeHealthy>true</nodeHealthy><nodeManagerVersion>3.3.2</nodeManagerVersion><nodeManagerBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum df7cdcca4397b98895332a47fb6636</nodeManagerBuildVersion><nodeManagerVersionBuiltOn>2022-02-21T19:04Z</nodeManagerVersionBuiltOn><hadoopVersion>3.3.2</hadoopVersion><hadoopBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum 4b40fff8bb27201ba07b6fa5651217fb</hadoopBuildVersion><hadoopVersionBuiltOn>2022-02-21T18:39Z</hadoopVersionBuiltOn><id>localhost:33429</id><nodeHostName>localhost</nodeHostName><nmStartupTime>1679444009281</nmStartupTime></nodeInfo>

UI portal shows consistent memory allocated:

2023032204952-image.png

2023032205032-image.png

Kontext Kontext #1807 access_time 2 years ago more_vert

Hello, can you please be more specific?

In my local instance, the information provided by the YARN web APIs are consistent with the one in the portal.

localhost:8042/ws/v1/node/info

<nodeInfo><healthReport/><totalVmemAllocatedContainersMB>17203</totalVmemAllocatedContainersMB><totalPmemAllocatedContainersMB>8192</totalPmemAllocatedContainersMB><totalVCoresAllocatedContainers>8</totalVCoresAllocatedContainers><vmemCheckEnabled>true</vmemCheckEnabled><pmemCheckEnabled>true</pmemCheckEnabled><lastNodeUpdateTime>1679444971058</lastNodeUpdateTime><resourceTypes>memory-mb (unit=Mi), vcores</resourceTypes><nodeHealthy>true</nodeHealthy><nodeManagerVersion>3.3.2</nodeManagerVersion><nodeManagerBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum df7cdcca4397b98895332a47fb6636</nodeManagerBuildVersion><nodeManagerVersionBuiltOn>2022-02-21T19:04Z</nodeManagerVersionBuiltOn><hadoopVersion>3.3.2</hadoopVersion><hadoopBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum 4b40fff8bb27201ba07b6fa5651217fb</hadoopBuildVersion><hadoopVersionBuiltOn>2022-02-21T18:39Z</hadoopVersionBuiltOn><id>localhost:33429</id><nodeHostName>localhost</nodeHostName><nmStartupTime>1679444009281</nmStartupTime></nodeInfo>

UI portal shows consistent memory allocated:

2023032204952-image.png

2023032205032-image.png

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts