python - 无服务器 - Numpy - 找不到好的绑定路径格式
问题描述
我已经在这个问题上打了一个多星期,经历了各种各样的论坛问题和帖子,无法解决。我正在尝试将 numpy 打包在一个函数中,单独构建需求(我有多个函数,我想将多个需求分开)。
环境:
Windows 10 家庭版
适用于 Windows 的 Docker 工具箱:
Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24302
Built: Fri Mar 23 08:31:36 2018
OS/Arch: windows/amd64
Experimental: false
Orchestrator: swarm
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:52:55 2018
OS/Arch: linux/amd64
Experimental: false
无服务器版本:
serverless version 6.4.1
serverless-python-requirements version 6.4.1
目录结构:
|-test
|-env.yml
|-serverless.yml
|-Dockerfile
|-functions
|-f1
|-index.py
|-requirements.txt
|-sub_function_1.py
|-sub_function_2.py
|-f2
|-index.py
|-requirements.txt
|-sub_function_3.py
|-sub_function_4.py
无服务器.yml
service: test
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
zip: true
dockerFile: Dockerfile
dockerizePip: non-linux
provider:
name: aws
runtime: python3.6
stage: dev
environment: ${file(./env.yml):${opt:stage, self:provider.stage}.env}
region: ${file(./env.yml):${opt:stage, self:provider.stage}.aws.region}
profile: ${file(./env.yml):${opt:stage, self:provider.stage}.aws.profile}
package:
individually: true
functions:
f1:
handler:index.handler
module:functions/f1
f2:
handler:index.handleer
module:functions/f2
我在 C:\Serverless\test 中有我的项目文件。我运行npm init
,然后npm i --save serverless-python-requirements
,接受所有默认值。我得到以下内容sls deploy -v
。即使我已将 C:\ 添加到 VirtualBox 中正在运行的默认 VM 上的共享文件夹中,并选择了自动挂载和永久挂载。
如果我将 dockerizePip 和 dockerFile 都注释掉,我会根据这里和其他 SO 帖子得到以下预期:
Serverless: Invoke invoke
{
"errorMessage": "Unable to import module 'index'"
}
如果我注释掉 dockerfile 我得到:
Serverless: Docker Image: lambci/lambda:build-python3.6
Error --------------------------------------------------
error during connect: Get https://XXXXXX/v1.37/version: dial tcp
XXXXXXXXXX: connectex: A connection attempt failed because the
connected party did not properly respond after a period of time, or
established connection failed because connected host has failed to
respond.
at dockerCommand (C:\Serverless\test\node_modules\serverless-python-requirements\lib\docker.js:20:11)
at getBindPath (C:\Serverless\test\node_modules\serverless-python-requirements\lib\docker.js:100:3)
使用 Dockerfile
# AWS Lambda execution environment is based on Amazon Linux 1
FROM amazonlinux:1
# Install Python 3.6
RUN yum -y install python36 python36-pip
# Install your dependencies
RUN curl -s https://bootstrap.pypa.io/get-pip.py | python3
RUN yum -y install python3-devel mysql-devel gcc
# Set the same WORKDIR as default image
RUN mkdir /var/task
WORKDIR /var/task
.
Serverless: Building custom docker image from Dockerfile...
Serverless: Docker Image: sls-py-reqs-custom
Error --------------------------------------------------
Unable to find good bind path format
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
Stack Trace --------------------------------------------
Error: Unable to find good bind path format
at getBindPath (C:\Serverless\test\node_modules\serverless-python-requirements\lib\docker.js:142:9)
at installRequirements (C:\Serverless\test\node_modules\serverless-python-requirements\lib\pip.js:152:7)
at installRequirementsIfNeeded (C:\Serverless\test\node_modules\serverless-python-requirements\lib\pip.js:451:3)
如果我将我的项目移动到 C:\Users\,我会得到这个:
Serverless: Docker Image: sls-py-reqs-custom
Serverless: Trying bindPath /c/Users/Serverless/test/.serverless/requirements (run,--rm,-v,/c/Users/Serverless/test/.serverless/req
uirements:/test,alpine,ls,/test/requirements.txt)
Serverless: /test/requirements.txt
Error --------------------------------------------------
docker: Error response from daemon: create "/c/Users/Serverless/test/.serverless/requirements": "\"/c/Users/Serverless/test/.serv
erless/requirements\"" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you in
tended to pass a host directory, use absolute path.
See 'docker run --help'.
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
Stack Trace --------------------------------------------
Error: docker: Error response from daemon: create "/c/Users/Serverless/test/.serverless/requirements": "\"/c/Users/Serverless/test/
.serverless/requirements\"" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If y
ou intended to pass a host directory, use absolute path.
See 'docker run --help'.
at dockerCommand (C:\Users\Serverless\test\node_modules\serverless-python-requirements\lib\docker.js:20:11)
at getDockerUid (C:\Users\Serverless\test\node_modules\serverless-python-requirements\lib\docker.js:162:14)
我在这里看到了@brianz 的 Makefile 风格推荐,但我不确定如何适应这一点(Makefile 不是我的强项)。我对下一步该怎么做有点茫然,我们将不胜感激。TIA。
解决方案
我无法使插件工作,但无论如何我找到了一个更好的解决方案 - Lambda 层。这是一个好处,因为它减小了 lambda 的大小并允许代码/文件重用。您可以使用一个用于 numpy 和 scipy 的预构建 lambda 层,但我构建了自己的层以向自己展示它是如何工作的。这是我如何使它工作的:
创建图层包:
- 打开 EC2 实例或 Ubuntu 或 Linux 或其他任何东西 - 这是需要的,以便我们可以正确编译运行时二进制文件
制作依赖包 zip - 必须使用
python/lib/python3.6/site-packages
python 的目录结构才能在运行时查找mkdir -p tmpdir/python/lib/python3.6/site-packages pip install -r requirements.txt --no-deps -t tmpdir/python/lib/python3.6/site-packages cd tmpdir zip -r ../py_dependencies.zip . cd .. rm -r tmpdir
将层 zip 推送到 AWS - 需要最新的 awscli
sudo pip install awscli --upgrade --user sudo aws lambda publish-layer-version \ --layer-name py_dependencies \ --description "Python 3.6 dependencies [numpy=0.15.4]" \ --license-info "MIT" \ --compatible-runtimes python3.6 \ --zip-file fileb://py_dependencies.zip \ --profile python_dev_serverless
要在任何需要 numpy 的函数中使用,只需使用控制台中或上面上传期间显示的 arn
f1: handler: index.handler_f_use_numpy include: - functions/f_use_numpy.py layers: - arn:aws:lambda:us-west-2:XXXXX:layer:py_dependencies:1
作为额外的奖励,您也可以将常量等常用文件推送到图层。这是我在 Windows 和 lambda 上测试使用的方法:
import platform \# Set common path COMMON_PATH = "../../layers/common/" if platform.system() == "Linux": COMMON_PATH = "/opt/common/" def handler_common(event, context): # Read from a constants.json file with open(COMMON_PATH + 'constants.json') as f: return text = json.load(f)
推荐阅读
- loops - 如何在 cplex 中创建 For 循环
- puppeteer - Puppeteer Sharp - 获取包含 textContent 的元素
- scala - 如何通过有界泛型解决隐式查找?
- tabulator - 按项目数量排序组
- javascript - 在 Nuxt.js 上与 iframe 交互
- javascript - 如何使用 Javascript 将链接附加到表?
- symfony - 如何在树枝的路径url中放置一组参数
- javascript - 配置配置文件以使用 QuaggaJs 正确解码所需的条形码
- angular - 如何在 NGXS 中创建参数化选择器?
- html - 只需要使用 css 的浏览器特定链接