时间戳

首页 > 解决方案 > 使用 python 将 csv 文件从 GCS 存储桶上传到远程 sftp 位置

python - 使用 python 将 csv 文件从 GCS 存储桶上传到远程 sftp 位置

问题描述

我正在尝试使用 python 将 csv 文件从谷歌云 gcs 存储桶发送到远程 sftp 位置。

import pysftp
from google.cloud import storage
from google.cloud.storage import Blob

client = storage.Client()
bucket = client.bucket("bucket_path")
blob = bucket.blob("FILE.csv")
cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
with  pysftp.Connection(host='remote_server', username='user', password='password',
                             port=22,
                             cnopts=cnopts) as sftp:
  print("Connection succesfully established ... ")
  remote_file=sftp.open('remote_location/sample.csv', 'w+')
  blob.download_to_file(remote_file)

我收到以下错误：

Connection succesfully established ... 
Traceback (most recent call last):
  File "/dirvenv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 997, in download_to_file
    self._do_download(
  File "/dirvenv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 872, in _do_download
    response = download.consume(transport, timeout=timeout)
  File "/dirvenv/lib/python3.8/site-packages/google/resumable_media/requests/download.py", line 168, in consume
    self._process_response(result)
  File "/dirvenv/lib/python3.8/site-packages/google/resumable_media/_download.py", line 185, in _process_response
    _helpers.require_status_code(
  File "/dirvenv/lib/python3.8/site-packages/google/resumable_media/_helpers.py", line 106, in require_status_code
    raise common.InvalidResponse(
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

在处理上述异常的过程中，又出现了一个异常：

Traceback (most recent call last):
  File "/dirPycharmProjects/leanplum/file_ftp.py", line 15, in <module>
    blob.download_to_file(remote_file)
  File "/dirvenv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 1008, in download_to_file
    _raise_from_invalid_response(exc)
  File "/dirvenv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 3262, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.NotFound: 404 GET https://storage.googleapis.com/download/storage/v1/b/gs://bucket_name/o/FILE.csv?alt=media: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

Process finished with exit code 1

有什么建议吗？

标签： pythongoogle-cloud-platformsftpbucket

解决方案

上述错误“ TypeError: expected str, bytes or os.PathLike object, not SFTPFile ”表示您正在尝试下载SFTPFile类型的对象，并且方法download_to_filename()需要str, bytes 或 os.PathLike objects。

我了解您的使用案例涉及将 CSV 格式文件上传到远程 SFTP 位置，并且此 CSV 文件当前位于云存储中。

因此，我建议您首先使用以下示例将此 blob 的内容从您的 Cloud Storage 存储桶下载到类似文件的对象中：

from google.cloud import storage


def download_blob(bucket_name, source_blob_name, destination_file_name):
    """Downloads a blob from the bucket."""
    # bucket_name = "your-bucket-name"
    # source_blob_name = "storage-object-name"
    # destination_file_name = "local/path/to/file"

    storage_client = storage.Client()

    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(source_blob_name)
    blob.download_to_filename(destination_file_name)

    print(
        "Blob {} downloaded to {}.".format(
            source_blob_name, destination_file_name
        )
    )

然后，在本地下载此 blob 的内容后，您可以使用以下示例代码将其上传到远程 STFP 位置：

import pysftp

with pysftp.Connection('hostname', username='[YOUR_USERNAME]', password='[YOUR_PASSWORD]') as sftp:
  with sftp.cd('public'):             # temporarily chdir to public
     sftp.put('/my/local/filename')  # upload file to public/ on remote

有关更多示例，请参阅此Stackoverflow 问题。

推荐阅读