google-cloud-dataflow - 无法使用 Dataflow REST API 传递作业参数
问题描述
我正在尝试通过REST API调用数据流模板,但是,当在正文中指定参数时,会引发 INVALID_ARGUMENT 错误。删除参数字段后,它可以正常工作。我已经尝试过 Javascript 和 Python,但得到了完全相同的错误。
from googleapiclient.discovery import build
from oauth2client.client import GoogleCredentials
def invoke_dataflow_job():
credentials = GoogleCredentials.get_application_default()
service = build('dataflow', 'v1b3', credentials=credentials)
TEMPLATE_LOCATION = 'gs://xxx';
PROJECT_ID = 'xxx';
JOB_NAME = 'xxx';
LOCATION = 'europe-west1';
ZONE = 'europe-west1-b';
TEMP_LOCATION = 'xxx';
BODY = {
'jobName': JOB_NAME,
'gcsPath': TEMPLATE_LOCATION,
'parameters': {
'messageToPrint': 'Blue sky'
},
'environment': {
'tempLocation': TEMP_LOCATION,
'zone': ZONE,
}
}
request = service.projects().locations().templates().create(projectId=PROJECT_ID, location=LOCATION, body=BODY)
response = request.execute()
return response
REST API 是否支持传入参数?查看示例代码,我相信它应该。
有什么建议么?
错误信息:
The workflow could not be created. Causes: (4adc3dbfd180d8a2): Found unexpected parameters: ['messageToPrint' (perhaps you meant 'userAgent')]
更新1:
样本自由度:
public static class BuildPathFn extends DoFn<String, Void> {
private static final long serialVersionUID = -2815123956194177539L;
private final ValueProvider<String> baseDir;
private final ValueProvider<String> year;
private final ValueProvider<String> month;
private final ValueProvider<String> day;
private final ValueProvider<String> hour;
private final ValueProvider<String> filePattern;
public BuildPathFn (ValueProvider<String> baseDir, ValueProvider<String> year,
ValueProvider<String> month, ValueProvider<String> day,
ValueProvider<String> hour, ValueProvider<String> filePattern) {
this.baseDir = baseDir;
this.year = year;
this.month = month;
this.day = day;
this.hour = hour;
this.filePattern = filePattern;
}
@ProcessElement
public void processElement(@Element String element, OutputReceiver<Void> receiver) {
try {
String path = baseDir.get() + "/" + year.get() + "/" + month.get() + "/" + day.get() + "/" + hour.get() + "/" + filePattern.get();
LOG.info("BuildPathFn >> Path: {}", path);
} catch (Exception e) {
LOG.error("BuildPathFn >> Exception: {}", e.getMessage());
}
}
}
运行时输出:
BuildPathFn >> Path: true/true/true/true/true/true
用于构建模板的 mvn 命令的一部分:
".... --baseDir --year --month --day --hour --filePattern"
解决方案
代码很好,它是传递参数的正确方法。请记住,该选项需要在模板中定义为ValueProvider。暂存模板时,不要传递您希望在运行时解析的任何参数。然后,在使用描述中已有的代码调用模板时(即使用parameters
内部请求正文),应评估选项值。
推荐阅读
- angular - TypeScript:在 webworker 中创建屏幕外画布
- laravel - 如何在 laravel 的邮递员中发送图像
- google-apps-script - 如何将所有 DriveApp 活动粘贴到电子表格中
- python - 使用 pandas 获取该行中第一个非零值的列名
- react-native - FCM 推送通知的负载测试
- php - 我正在使用 $_SESSION('ID') ,更新到表中不起作用并且通过 php 页面没有错误
- java - Spring Boot 应用程序属性
- azure-devops - Azure DevOps Service 已导入数据库成功但错误 404 NotFound can't access to link
- microsoft-graph-api - 是否可以通过 microsoft graph webhooks 获取更新的实时更新?
- html - 合并两个 Observable 并作为 Angular 材料表的单个 Observable 传递到 dataSource