时间戳

首页 > 解决方案 > 如何为作为 kubernets pod 运行的气流设置持久日志记录和 dag

kubernetes - 如何为作为 kubernets pod 运行的气流设置持久日志记录和 dag

问题描述

我使用 unbuntu wsl 安装在本地 Windows 机器上工作。

$cat pv.yml:
apiVersion: v1
kind: PersistentVolume
metadata:
  name: airflow-volume
  labels:
    type: local
spec:
  #we use local node storage here!
  #kubectl get storageclass
  storageClassName: standard
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/mnt/c/dumps"


$ cat pvc.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kube-claim
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Mi

掌舵命令：

helm upgrade $RELEASE_NAME apache-airflow/airflow --namespace $NAMESPACE     --set images.airflow.repository=airflow-dags     --set images.airflow.tag=0.2.3 --set executor=KubernetesExecutor --set logs.persistence.enabled=true --set logs.persistence.existingClaim=kube-claim

来自网络服务器的日志屏幕上的错误：

*** Trying to get logs (last 100 lines) from worker pod batchtestpodruntest.1f82d2f60f9f4766b81ed54abb28e75a ***

*** Unable to fetch logs from worker pod batchtestpodruntest.1f82d2f60f9f4766b81ed54abb28e75a ***
(403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon,

2021 年 6 月 14 日 15:21:19 GMT', 'Content-Length': '425'}) HTTP 响应正文：b'{"kind":"Status","apiVersion":"v1","metadata":{ },"status":"Failure","message":"pods \"batchtestpodruntest.1f82d2f60f9f4766b81ed54abb28e75a\" 被禁止：用户 \"system:serviceaccount:airflow:latest-airflow-webserver\" 无法获取资源 \"pods/log \" 在 API 组中 \"\" 在命名空间中 \"airflow\"","re​​ason":"Forbidden","details":{"name":"batchtestpodruntest.1f82d2f60f9f4766b81ed54abb28e75a","kind":"pods"} "代码":403}\n'

标签： kubernetesairflowkubernetes-pod

解决方案

您可以配置气流以将日志发送到 s3 或弹性搜索。

推荐阅读