python - 如何在没有云功能的 GCP 中安排更长的 python 脚本

Question

我有一个从 API 下载大量数据的脚本。该脚本大约需要两个小时才能运行。我想在 GCP 上运行该脚本并安排它在星期日每周运行一次，以便我们的 SQL 数据库（也在 GCP 上）在第二天之前获得最新数据。

我知道 cronjobs，但不想只为这个单个脚本运行整个服务器。我查看了云函数和云调度程序，但由于脚本执行时间过长，我无法在云函数上运行它，因为最长执行时间为 9 分钟（从这里开始）。还有其他方法可以安排python脚本运行吗？

先感谢您！

标签： pythonpython-3.xgoogle-cloud-platform

Answer 1

For running a script more than 1h, you need to use a Compute Engine. (Cloud Run can live only 1h).

However, you can use Cloud Scheduler. Here how to do

• Create a cloud scheduler with the frequency that you want
  • On this scheduler, use the Compute Engine Start API
  • In the advanced part, select a service account (create one or reuse one) which have the right to start a VM instance
  • Select OAuth token as authentication mode (not OIDC)
• Create a compute engine (that you will start with the Cloud Scheduler)
  • Add a startup script that trigger your long job
  • At the end on the script, add a line to shutdown the VM (with Gcloud for example)

Note: the startup script is run as ROOT user. Take care of the default home directory and the permission of the created files.