hadoop - 确定已完成作业使用的 Hadoop 集群资源

Question

如何确定已完成作业使用的 Hadoop 集群资源？

我们的集群资源管理器是yarn. 可以通过 HTTP 访问某些 yarn API 端点；例如：

curl -L http://my.hadoop.instance:8088/ws/v1/cluster/apps/application_1547448533998_502644

会返回：

{
    "app": {
        "allocatedMB": -1,
        "allocatedVCores": -1,
        "amContainerLogs": "http://someNode.hadoop.instance:8042/node/containerlogs/container_e149_1547448533998_502644_01_000001/someUser",
        "amHostHttpAddress": "someNode.hadoop.instance:8042",
        "amNodeLabelExpression": "",
        "applicationTags": "",
        "applicationType": "SPARK",
        "clusterId": 1547448533998,
        "clusterUsagePercentage": 0.0,
        "diagnostics": "",
        "elapsedTime": 583889,
        "finalStatus": "SUCCEEDED",
        "finishedTime": 1550621490747,
        "id": "application_1547448533998_502644",
        "logAggregationStatus": "TIME_OUT",
        "memorySeconds": 15821179,
        "name": "ProjectCantor",
        "numAMContainerPreempted": 0,
        "numNonAMContainerPreempted": 0,
        "preemptedResourceMB": 0,
        "preemptedResourceVCores": 0,
        "priority": 0,
        "progress": 100.0,
        "queue": "dsg",
        "queueUsagePercentage": 0.0,
        "runningContainers": -1,
        "startedTime": 1550620906858,
        "state": "FINISHED",
        "trackingUI": "History",
        "trackingUrl": "http://my.hadoop.instance:8088/proxy/application_1547448533998_502644/",
        "unmanagedApplication": false,
        "user": "someUser",
        "vcoreSeconds": 14713
    }
}

但是，有几个属性，例如allocatedMB并allocatedVCores设置为-1而不是任何有意义的值。

提前致谢。

标签： hadoophadoop-yarn