时间戳

首页 > 解决方案 > 从 python 到 gcs 模拟器的 IPv4 请求

python - 从 python 到 gcs 模拟器的 IPv4 请求

问题描述

我正在尝试从 python 应用程序向 docker for mac 中的 docker-compose 桥接网络中的 gcs 模拟器发出请求。当我尝试时，我发现 gcs 客户端库以某种方式尝试使用 IPv6 向 gcs 模拟器发出请求，但由于 docker for mac 不支持 IPv6 而失败。

我已经实现了以下答案来纠正 IPv4，但它似乎仍在尝试通过 IPv6 发出请求。

如何在 docker-compose 网络中从 python 向 gcs 模拟器发出成功请求？

我已经确认从本地 Python 脚本到没有 docker-compose 的 gcs 模拟器的请求是成功的。

docker-for-mac 问题：https ://github.com/docker/for-mac/issues/1432

参考答案：强制请求使用 IPv4 / IPv6

gcs 模拟器：https ://github.com/fsouza/fake-gcs-server

示例 docker-compose.yaml

version: '3'
services:
  run:
    build: .
    container_name: run
    ports:
      - 9090:8080
    env_file: 
      - ./.env
    environment:
      - PORT=8080
  gcs:
    image: fsouza/fake-gcs-server:latest
    container_name: fake-gcs-server
    ports:
      - 4443:4443
    env_file: 
      - ./.env

示例实现：

from google.cloud import storage
from google.api_core.client_options import ClientOptions
from google.auth.credentials import AnonymousCredentials
from unittest.mock import patch
from multijob_sample import variables as vs
import requests
import urllib3
import urllib3.util.connection
import traceback

import socket
orig_getaddrinfo = socket.getaddrinfo
def getaddrinfoIPv4(host, port, family=0, type=0, proto=0, flags=0):
    print(f'running patched getaddrinfo')
    return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET, type=type, proto=proto, flags=flags)
patcher = patch('socket.getaddrinfo', side_effect=getaddrinfoIPv4)
patcher.start()


# for fake-gcs-emulator
http_ssl_disabled = requests.Session()
http_ssl_disabled.verify = False
urllib3.disable_warnings(
       urllib3.exceptions.InsecureRequestWarning
)  # disable https warnings for https insecure certs

client = storage.Client(
    credentials=AnonymousCredentials(),
    project=vs.project_id,
    client_options=ClientOptions(api_endpoint='https://gcs:4443'), 
    _http=http_ssl_disabled,
)

def put_file(bucket_id: str, file, blobname: str):
    file.seek(0)
    try:
        client.get_bucket(bucket_id).blob(blob_name=blobname).upload_from_file(file)
        print(f'file {blobname} uploaded')
    except Exception as e:
        print(f'failed to put file: {blobname}')
        print(f'error: {e}')
        print(f'trace: {traceback.format_exc()}')


put_file("bucketid", file, "blobname") # do put_file

错误信息：

run              | running patched getaddrinfo
run              | failed to put file: test.csv
run              | error: HTTPSConnectionPool(host='::', port=4443): Max retries exceeded with url: /upload/resumable/efbbcde9c49cda2ff78e8da24371ea03 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8fb0765be0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))
run              | trace: Traceback (most recent call last):
run              |   File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
run              |     conn = connection.create_connection(
run              |   File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
run              |     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
run              |   File "/usr/local/lib/python3.9/unittest/mock.py", line 1093, in __call__
run              |     return self._mock_call(*args, **kwargs)
run              |   File "/usr/local/lib/python3.9/unittest/mock.py", line 1097, in _mock_call
run              |     return self._execute_mock_call(*args, **kwargs)
run              |   File "/usr/local/lib/python3.9/unittest/mock.py", line 1158, in _execute_mock_call
run              |     result = effect(*args, **kwargs)
run              |   File "/app/multijob_sample/storage.py", line 26, in getaddrinfoIPv4
run              |     return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET, type=type, proto=proto, flags=flags)
run              |   File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
run              |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
run              | socket.gaierror: [Errno -9] Address family for hostname not supported

标签： pythondockernetwork-programminggoogle-cloud-storagedocker-for-mac

解决方案

到目前为止，这是我一段时间以来不得不解决的最烦人的问题之一。-external-url http://<your docker compose service name>:<port>解决方案是使用该选项运行模拟器。

此问题仅在文件上传时发生，因为它仅在可恢复上传时发生。对于可恢复的上传，GCS 客户端首先与服务器“启动”可恢复的上传，并且在响应中服务器包含一个 URL 供将来的请求访问（不知道为什么，但它似乎是复杂 API 的一个合理部分） . 问题是模拟器不知道它自己的网址！事实上，如果你查看模拟器的日志，你会看到它打印出类似server started at http://[::]:4443. 这与您在错误中看到::的相同。::所以模拟器用它的::URL 进行响应，然后一段时间后，客户端在尝试解析该 URL 时崩溃。

我仍然不确定为什么在 docker-compose 之外运行它可以工作，我猜周围有一些特殊的外壳""，"localhost"或者“::”`。

推荐阅读