google-bigquery - 在 BigQuery 中批量删除数据传输

Question

随着时间的推移，许多不同的人在我们的演示项目中创建了一个测试传输，但从未清理过它。我很想批量删除所有转移。有谁知道这样做的方法？

标签： google-bigquery

Answer 1

我在Cloud Workflows上构建了类似的东西。

从本质上讲，您需要一个工作流来为您执行从BigQuery 数据传输 API开始的步骤并发出一些命令：

• 获取传输配置列表
• 遍历它们
• 为他们发出删除 API 调用

https://cloud.google.com/bigquery-transfer/docs/reference/datatransfer/rest

我的代码是清理 Cloud Workflows（您需要使用 Transfer Service API 调用稍微调整一下这个）

#cloud workflow to cleanup 
main:
  steps:
    - initialize:
        assign:
          - project: "marton-data"
          - location: "us-central1"
    - getList:
        call: WorkflowsList
        args:
          project: ${project}
          location: ${location}
        result: items
    - loopItems:
        call: WorkflowListLoopItems
        args:
          items: ${items}
        result: res
    - final:
        return: ${res}
WorkflowsList:
  params: [project,location]
  steps:
    - list:
        call: googleapis.workflows.v1.projects.locations.workflows.list
        args:
            parent: ${"projects/"+project+"/locations/"+location}
            pageSize: 100
        result: listResult
    - documentFound:
        return: ${listResult.workflows}
WorkflowDelete:
  params: [name]
  steps:
    - delete:
        call: googleapis.workflows.v1.projects.locations.workflows.delete
        args:
            name: ${name}
        result: deleteResult
    - logStep: 
        call: sys.log
        args:
          text: ${"Calling "+name+" \r\n Results returned "+json.encode_to_string(deleteResult)}
    - deleteDone:
        return: ${deleteResult}
WorkflowListLoopItems:
  params: [items]
  steps:
    - init:
        assign:
          - i: 0
          - result: ""
    - check_condition:
        switch:
          - condition: ${items[i].name=="projects/marton-data/locations/us-central1/workflows/workflow_cleanup"}
            next: assign_loop
          - condition: ${len(items) > i}
            next: iterate
        next: exit_loop
    - iterate:
        steps:
          - process_item:
              call: WorkflowDelete
              args:
                name: ${items[i].name}
              result: result
          - assign_loop:
              assign:
                - i: ${i+1}
        next: check_condition
    - exit_loop:
        return: ${result}
logMessage:
  params: [collection]
  steps:
    - log:
        call: http.post
        args:
            url: https://logging.googleapis.com/v2/entries:write
            auth:
                type: OAuth2
            body:
                entries:
                    - logName: ${"projects/" + sys.get_env("GOOGLE_CLOUD_PROJECT_ID") + "/logs/workflow_logger"}
                      resource:
                        type: "audited_resource"
                        labels: {}
                      jsonPayload: ${collection}