A ameni kallel

Haddop API

event 2023-03-20 visibility 70 comment 3 insights
more_vert
insights Stats

Hi, I want to know if there is a hadoop API that can give me the details about the resources used by all the hadoop nodes: indeed, I want to know the total, used and available capacity for RAM, CPU and disk for all Hadoop nodes.

I tested this API "http://nm-http-address:port/ws/v1/node", but it sends me information that has no link with the real resources used by the machine and I don't know from what source this API gets this information.

Thanks for your answers

More from Kontext
comment Comments
Raymond Raymond #1812 access_time 12 months ago more_vert

I believe it is based on the default configurations.  You can play around with those configuration properties to see if they change.


Apache Hadoop 3.1.1 – Hadoop: YARN Resource Configuration

format_quote

person ameni access_time 11 months ago

Thanks for your reply.

When I tested this API, I got the same values as you: total memory= 8G and total vcores=8, while the machine only has 2G of memory and 2 vcores.

Do you have any explications.

Thanks in advance.

A ameni kallel #1808 access_time 11 months ago more_vert

Thanks for your reply.

When I tested this API, I got the same values as you: total memory= 8G and total vcores=8, while the machine only has 2G of memory and 2 vcores.

Do you have any explications.

Thanks in advance.

format_quote

person Kontext access_time 12 months ago

Hello, can you please be more specific?

In my local instance, the information provided by the YARN web APIs are consistent with the one in the portal.

localhost:8042/ws/v1/node/info

<nodeInfo><healthReport/><totalVmemAllocatedContainersMB>17203</totalVmemAllocatedContainersMB><totalPmemAllocatedContainersMB>8192</totalPmemAllocatedContainersMB><totalVCoresAllocatedContainers>8</totalVCoresAllocatedContainers><vmemCheckEnabled>true</vmemCheckEnabled><pmemCheckEnabled>true</pmemCheckEnabled><lastNodeUpdateTime>1679444971058</lastNodeUpdateTime><resourceTypes>memory-mb (unit=Mi), vcores</resourceTypes><nodeHealthy>true</nodeHealthy><nodeManagerVersion>3.3.2</nodeManagerVersion><nodeManagerBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum df7cdcca4397b98895332a47fb6636</nodeManagerBuildVersion><nodeManagerVersionBuiltOn>2022-02-21T19:04Z</nodeManagerVersionBuiltOn><hadoopVersion>3.3.2</hadoopVersion><hadoopBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum 4b40fff8bb27201ba07b6fa5651217fb</hadoopBuildVersion><hadoopVersionBuiltOn>2022-02-21T18:39Z</hadoopVersionBuiltOn><id>localhost:33429</id><nodeHostName>localhost</nodeHostName><nmStartupTime>1679444009281</nmStartupTime></nodeInfo>

UI portal shows consistent memory allocated:

2023032204952-image.png

2023032205032-image.png

Kontext Kontext #1807 access_time 12 months ago more_vert

Hello, can you please be more specific?

In my local instance, the information provided by the YARN web APIs are consistent with the one in the portal.

localhost:8042/ws/v1/node/info

<nodeInfo><healthReport/><totalVmemAllocatedContainersMB>17203</totalVmemAllocatedContainersMB><totalPmemAllocatedContainersMB>8192</totalPmemAllocatedContainersMB><totalVCoresAllocatedContainers>8</totalVCoresAllocatedContainers><vmemCheckEnabled>true</vmemCheckEnabled><pmemCheckEnabled>true</pmemCheckEnabled><lastNodeUpdateTime>1679444971058</lastNodeUpdateTime><resourceTypes>memory-mb (unit=Mi), vcores</resourceTypes><nodeHealthy>true</nodeHealthy><nodeManagerVersion>3.3.2</nodeManagerVersion><nodeManagerBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum df7cdcca4397b98895332a47fb6636</nodeManagerBuildVersion><nodeManagerVersionBuiltOn>2022-02-21T19:04Z</nodeManagerVersionBuiltOn><hadoopVersion>3.3.2</hadoopVersion><hadoopBuildVersion>3.3.2 from 0bcb014209e219273cb6fd4152df7df713cbac61 by chao source checksum 4b40fff8bb27201ba07b6fa5651217fb</hadoopBuildVersion><hadoopVersionBuiltOn>2022-02-21T18:39Z</hadoopVersionBuiltOn><id>localhost:33429</id><nodeHostName>localhost</nodeHostName><nmStartupTime>1679444009281</nmStartupTime></nodeInfo>

UI portal shows consistent memory allocated:

2023032204952-image.png

2023032205032-image.png

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts