prometheus - PROMQL：没有数据返回时如何添加值？

Question

我有一个数据模型，其中一些指标按客户端、环境和部署名称命名。我有兴趣创建每个部署的摘要，其中该摘要基于每个部署存在的警报数量。

dev我可以使用以下查询获取、uat和prod环境中的部署：

group by(tenant, environment, deployment)(up{environment=~"dev|uat|prod"}) - 1

# returns the following by way of example:
{deployment="default",environment="dev",tenant="tenant1"}   0
{deployment="default",environment="prod",tenant="tenant3"}  0
{deployment="default",environment="prod",tenant="tenant2"}  0
{deployment="default",environment="uat",tenant="tenant1"}   0

所以我们可以看到租户 1 在 2 个不同的环境中有 2 个部署，而另外 2 个只有一个。 group by返回值 1，因此我们减去 1 以获得每个部署的 0，我现在希望将适用于每个部署的警报数量添加到此。

为了获得警报，我这样做：

ALERTS{severity="warning"}
# returns something like this when there is an alert, the details in the alert will vary, but will always have the `tenant`, `environment` and `deployment` labels
ALERTS{alertname="HostSystemdServiceCrashed",alertstate="firing",instance="example",job="node",deployment="default",environment="dev",tenant="tenant1",name="example.service",severity="warning",state="failed",type="oneshot"} 1

# however, when there are no alerts, I get "no data" returned

我无法弄清楚如何将警报添加到部署中，同时保留没有返回警报的部署：

(group by(tenant, environment, deployment)(up{environment=~"dev|uat|prod"}) -1)  + on(tenant, environment, deployment) (ALERTS{severity="warning"})

# returns only data for the deployment for which there is an alert
{deployment="default",environment="dev",tenant="tenant1"} 1

# if there are no alerts, I get no data returned at all

我想要的输出是这样的：

{deployment="default",environment="dev",tenant="tenant1"} 1
{deployment="default",environment="uat",tenant="tenant1"} 0
{deployment="default",environment="prod",tenant="tenant2"} 0
{deployment="default",environment="prod",tenant="tenant3"} 0

我怎样才能做到这一点？

笔记：

如果我使用sumwith or，那么我会得到这个，这取决于参数的顺序or：

(group by(tenant, environment, deployment)(up{environment=~"dev|uat|prod"}) -1)  or sum by (tenant, environment, deployment) (ALERTS{severity="warning"} )

# returns this, note the value in `tenant1|dev|default`
{deployment="default",environment="dev",tenant="tenant1"} 0
{deployment="default",environment="uat",tenant="tenant1"} 0
{deployment="default",environment="prod",tenant="tenant2"} 0
{deployment="default",environment="prod",tenant="tenant3"} 0

如果我将参数的顺序反转为or，我会得到我所追求的：

{deployment="default",environment="dev",tenant="tenant1"} 1
{deployment="default",environment="uat",tenant="tenant1"} 0
{deployment="default",environment="prod",tenant="tenant2"} 0
{deployment="default",environment="prod",tenant="tenant3"} 0

但是如果我想做一些事情，比如对不同严重级别的警报应用权重，我现在就被卡住了，例如（伪代码）：

summary = 0 + sum(warning alerts) + 2*sum(alerts(critical alerts))

这给出了相同的单值系列，如果没有警报，则没有数据。

标签： prometheuspromql

Answer 1

我确信有一种正确的方法可以做到这一点，但最后，我曾经label_replace为每个希望添加到原始值的子查询添加一个任意键值标签，然后对每个子查询应用一个or。这具有合并系列而不覆盖任何值的效果。然后，我能够对sum by结果系列执行最终结果，以将结果减少为单个结果，在此过程中删除临时标签。

sum(
  (group by(tenant, environment, deployment) (up{environment=~"dev|uat|prod"} ) -1) 

  or label_replace((sum (ALERTS{severity="warning"} ) by (tenant, environment, deployment)), "severity", "warning", "", "") or 

  2 * label_replace((sum (ALERTS{severity="critical"} )  by (tenant, environment, deployment)), "severity", "critical", "", "")
) by (tenant, environment, deployment)