时间戳

首页 > 解决方案 > 在运行时构建多个轮子

python - 在运行时构建多个轮子

问题描述

我想在运行时构建包含一些脚本以及一些有效负载数据的轮子。例如，在本例中，该target文件夹包含两个简单的构建build_123并被build_124打包为一个轮子。

main_project
├── __init__.py
├── whl_util.py               # wheel building script posted below
target/
├── build_123/                # contains one build to be packaged as a whl
│   └── mypkg
|       ├── __init__.py
│       ├── data
|       |   ├── __init__.py
|       |   └── mat.json
│       └── main
|           ├── __init__.py
|           └── dumpmat.py
└── build_124/                # contains another build to be packaged as a whl
    └── mypkg
        ├── ...

在我的场景中，这个轮子被用作输出格式，包装不是主要进展。轮子打包应该被认为是一个简单的 IO 操作，它读取一个构建文件夹并输出一个轮子，除此之外没有任何副作用。为了执行这个任务，我想出了这个解决方案：

# main_project/whl_util.py
from setuptools import setup, find_packages
import sys
import shutil
import os

def bdist_wheel(build_dir=".", dist_dir=None):
    # backing up argv to restore them afterwards
    argv_bak = sys.argv[:]

    # clear args from running script with "bdist_wheel"
    file = sys.argv[0]
    sys.argv.clear()
    sys.argv.extend([file, "bdist_wheel"])

    if dist_dir is not None and "--dist-dir" not in sys.argv:
        sys.argv.extend(["--dist-dir", dist_dir])

    sys.argv.extend(["clean", "--all"])

    setup(
        name="mypkg",
        version=0.1,
        packages=find_packages(build_dir),
        install_requires=[],
        include_package_data=True,
        package_dir={'': build_dir},
        package_data={"mypkg.data": ["mat.json"]}
    )

    # restore args
    sys.argv.clear()
    sys.argv.extend(argv_bak)


def main():
    # Adding main method here for testing.
    # As mentioned in my actual scenario the wheels should be built as an output format at runtime
    print("BUILD 123")
    bdist_wheel("target/build_123", dist_dir="target/dist_123")
    print("BUILD 124")
    bdist_wheel("target/build_124", dist_dir="target/dist_124")

if __name__ == "__main__":
    main()

我也不太喜欢将参数传递给setuptoolsvia的方式，sys.argv但这似乎是唯一的方式。但是，主要问题是，第一个轮子正常构建，而bdist_wheel/的第二次调用setup引发错误：

python3 -m main_project.whl_util
BUILD 123
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/mypkg
copying target/build_123/mypkg/__init__.py -> build/lib/mypkg
creating build/lib/mypkg/data
copying target/build_123/mypkg/data/__init__.py -> build/lib/mypkg/data
creating build/lib/mypkg/main
copying target/build_123/mypkg/main/__init__.py -> build/lib/mypkg/main
copying target/build_123/mypkg/main/dumpmat.py -> build/lib/mypkg/main
running egg_info
writing target/build_123/mypkg.egg-info/PKG-INFO
writing dependency_links to target/build_123/mypkg.egg-info/dependency_links.txt
writing top-level names to target/build_123/mypkg.egg-info/top_level.txt
writing manifest file 'target/build_123/mypkg.egg-info/SOURCES.txt'
copying target/build_123/mypkg/data/mat.json -> build/lib/mypkg/data
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/mypkg
copying build/lib/mypkg/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg
creating build/bdist.linux-x86_64/wheel/mypkg/data
copying build/lib/mypkg/data/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg/data
copying build/lib/mypkg/data/mat.json -> build/bdist.linux-x86_64/wheel/mypkg/data
creating build/bdist.linux-x86_64/wheel/mypkg/main
copying build/lib/mypkg/main/__init__.py -> build/bdist.linux-x86_64/wheel/mypkg/main
copying build/lib/mypkg/main/dumpmat.py -> build/bdist.linux-x86_64/wheel/mypkg/main
running install_egg_info
Copying target/build_123/mypkg.egg-info to build/bdist.linux-x86_64/wheel/mypkg-0.1-py3.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/mypkg-0.1.dist-info/WHEEL
creating 'target/dist_123/mypkg-0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'mypkg/__init__.py'
adding 'mypkg/data/__init__.py'
adding 'mypkg/data/mat.json'
adding 'mypkg/main/__init__.py'
adding 'mypkg/main/dumpmat.py'
adding 'mypkg-0.1.dist-info/METADATA'
adding 'mypkg-0.1.dist-info/WHEEL'
adding 'mypkg-0.1.dist-info/top_level.txt'
adding 'mypkg-0.1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
BUILD 124
running bdist_wheel
running build
running build_py
copying target/build_124/mypkg/__init__.py -> build/lib/mypkg
error: could not create 'build/lib/mypkg/__init__.py': No such file or directory

Process finished with exit code 1

该no such file or directory错误表明该setuptools模块跟踪它已经创建的文件夹并假设这些文件夹仍然存在。然而，在构建第一个轮子之后，clean脚本将删除构建的文件夹（这是必要的，因为setuptools否则将重用文件夹而不清除它）

我唯一可行的解​​决方案是在申请之前分叉该过程setup：

pid = os.fork()
if pid == 0:
    setup(...)
    sys.exit(0)
os.waitpid(pid, 0)

但由于这看起来很脏，而且我的主进程非常占用内存，我宁愿避免这种方法。

所以我的主要问题是： 有没有办法制造一个没有任何副作用的轮子？或者有没有办法setuptools在应用后重置模块的状态setup？在一个最佳世界中，我想在内存中创建轮子，PyFilesystem并且只将轮子写入磁盘。

标签： pythonsetuptools

解决方案

我不确定setuptools是否打算以这种方式使用。据我所知pip and co。（wheel、setuptools等）并没有真正的公共 API，或者至少没有友好的 API。

该distlib库看起来像是一个带有实际 API 的有前途的替代方案。请参阅distlib的有关“使用 wheel API”的文档。

如果这不起作用，那么我可能会尝试其中一个：

subprocess.check_call([sys.executable, '-m', 'wheel', 'pack', 'target/build123'])

请参阅wheel pack文档。

subprocess.check_call([sys.executable, '-m', 'pip', 'wheel', 'target/build123'])

请参阅pip wheel文档。并且它不能与 API 调用一起使用的原因在 pip 文档的“在程序中使用pip ”部分中注明

有一个有点相似的问题，有一些有趣的想法：

setup.py也许在运行时动态生成一个可能会有所帮助。

推荐阅读