时间戳

首页 > 解决方案 > 在 macOS 上使用 OpenMP 使用 Cython 进行编译

python - 在 macOS 上使用 OpenMP 使用 Cython 进行编译

问题描述

我在 macOS Mojave 10.14.6 上，我正在尝试从这个存储库中用 c 和 c++ 编译一些必需的扩展模块：

python setup.py build_ext --inplace

这给了我以下错误：

No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
running build_ext
/Users/user/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py:249: UserWarning: 

                               !! WARNING !!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (g++) is not compatible with the compiler Pytorch was
built with for this platform, which is clang++ on darwin. Please
use clang++ to to compile your extension. Alternatively, you may
compile PyTorch from source using g++, and then you can also use
g++ to compile your extension.

See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help
with compiling PyTorch from source.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                              !! WARNING !!

  warnings.warn(WRONG_COMPILER_WARNING.format(

再往下：

clang: error: unknown argument: '-i'
clang: error: no such file or directory: 'sysroot'
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang: error: unsupported option '-fopenmp'
ninja: build stopped: subcommand failed.

（这应该是相关部分，最近的回溯也显示subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.最后RuntimeError: Error compiling objects for extension）

根据我的理解和到目前为止我所尝试的：

对于这个令人困惑的问题，我感到很抱歉，但是对编译器没有任何先验知识并试图解决这个问题就像陷入困境......我将非常感谢任何能阐明这种情况的评论，我很高兴提供任何缺失的信息！

更新：

完整的错误信息是：

(torch) [10:48:05] vanessamac: occupancy_networks $ python setup.py build_ext --inplace --verbose
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
running build_ext
building 'im2mesh.utils.libkdtree.pykdtree.kdtree' extension
Emitting ninja build file /Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
FAILED: /Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o 
clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang: error: unsupported option '-fopenmp'
[2/2] clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/kdtree.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
FAILED: /Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o 
clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/kdtree.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang: error: unsupported option '-fopenmp'
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1509, in _run_ninja_build
    subprocess.run(
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 112, in <module>
    setup(
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/setuptools/__init__.py", line 165, in setup
    return distutils.core.setup(**attrs)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 649, in build_extensions
    build_ext.build_extensions(self)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
    _build_ext.build_extension(self, ext)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 469, in unix_wrap_ninja_compile
    _write_ninja_file_and_compile_objects(
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1228, in _write_ninja_file_and_compile_objects
    _run_ninja_build(
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build
    raise RuntimeError(message)
RuntimeError: Error compiling objects for extension

标签： pythonc++macoscython

解决方案

这里有一些提示：

  • 使用 gcc 而不是 llvm 或 clang 在 macOS 上获得无痛的 openmp 支持。请注意，Apple 的默认 gcc 只是 Apple clang 的别名，您将在gcc --version. 您可以使用 homebrew安装真正的 gccbrew install gcc：.

  • 然后export CC='gcc-10'在同一个终端窗口中使用（最新版本应该是 gcc 10.x）临时使用 homebrew 的 gcc 作为你的 C 编译器。

  • 无需设置CXXFLAGS或CFLAGS。所需的标志由 setup.py 中的 distutils/setuptools 设置。

  • 您将无法dmc_cuda_module在 macOS 10.14.6 上进行编译。最新的 macOS 版本 nvidia 为 10.13.6 提供 cuda 驱动程序。因此，您可能会取消注释 setup.py 的这一部分，并希望您不需要这个模块......

  • 使用 numpy C-API 时，其中的一些扩展setup.py不包括 numpy 标头。在 macOS 上，必须包含每个扩展的 numpy 标头，请参阅此评论。所以你必须添加include_dirs=[numpy_include_dir]到那些扩展。

  • 编辑：正如聊天中所讨论的：错误是由于 conda env 忽略了 CC 变量。通过自制软件安装 python+pip 并通过 pip 安装所需的 python 包后，此答案的步骤适用于 OP。

总而言之，这是一个对我有用的 setup.py（macOS 10.5.7，gcc-10）：

try:
    from setuptools import setup
except ImportError:
    from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from torch.utils.cpp_extension import BuildExtension, CppExtension, CUDAExtension
import numpy


# Get the numpy include directory.
numpy_include_dir = numpy.get_include()

# Extensions
# pykdtree (kd tree)
pykdtree = Extension(
    'im2mesh.utils.libkdtree.pykdtree.kdtree',
    sources=[
        'im2mesh/utils/libkdtree/pykdtree/kdtree.c',
        'im2mesh/utils/libkdtree/pykdtree/_kdtree_core.c'
    ],
    language='c',
    extra_compile_args=['-std=c99', '-O3', '-fopenmp'],
    extra_link_args=['-lgomp'],
    include_dirs=[numpy_include_dir]
)

# mcubes (marching cubes algorithm)
mcubes_module = Extension(
    'im2mesh.utils.libmcubes.mcubes',
    sources=[
        'im2mesh/utils/libmcubes/mcubes.pyx',
        'im2mesh/utils/libmcubes/pywrapper.cpp',
        'im2mesh/utils/libmcubes/marchingcubes.cpp'
    ],
    language='c++',
    extra_compile_args=['-std=c++11'],
    include_dirs=[numpy_include_dir]
)

# triangle hash (efficient mesh intersection)
triangle_hash_module = Extension(
    'im2mesh.utils.libmesh.triangle_hash',
    sources=[
        'im2mesh/utils/libmesh/triangle_hash.pyx'
    ],
    libraries=['m'],  # Unix-like specific
    include_dirs=[numpy_include_dir]
)

# mise (efficient mesh extraction)
mise_module = Extension(
    'im2mesh.utils.libmise.mise',
    sources=[
        'im2mesh/utils/libmise/mise.pyx'
    ],
)

# simplify (efficient mesh simplification)
simplify_mesh_module = Extension(
    'im2mesh.utils.libsimplify.simplify_mesh',
    sources=[
        'im2mesh/utils/libsimplify/simplify_mesh.pyx'
    ],
    include_dirs=[numpy_include_dir]
)

# voxelization (efficient mesh voxelization)
voxelize_module = Extension(
    'im2mesh.utils.libvoxelize.voxelize',
    sources=[
        'im2mesh/utils/libvoxelize/voxelize.pyx'
    ],
    libraries=['m']  # Unix-like specific
)

# DMC extensions
dmc_pred2mesh_module = CppExtension(
    'im2mesh.dmc.ops.cpp_modules.pred2mesh',
    sources=[
        'im2mesh/dmc/ops/cpp_modules/pred_to_mesh_.cpp',
    ]   
)

# dmc_cuda_module = CUDAExtension(
#     'im2mesh.dmc.ops._cuda_ext', 
#     sources=[
#         'im2mesh/dmc/ops/src/extension.cpp',
#         'im2mesh/dmc/ops/src/curvature_constraint_kernel.cu',
#         'im2mesh/dmc/ops/src/grid_pooling_kernel.cu',
#         'im2mesh/dmc/ops/src/occupancy_to_topology_kernel.cu',
#         'im2mesh/dmc/ops/src/occupancy_connectivity_kernel.cu',
#         'im2mesh/dmc/ops/src/point_triangle_distance_kernel.cu',
#     ]
# )

# Gather all extension modules
ext_modules = [
    pykdtree,
    mcubes_module,
    triangle_hash_module,
    mise_module,
    simplify_mesh_module,
    voxelize_module,
    dmc_pred2mesh_module,
    #dmc_cuda_module,
]

setup(
    ext_modules=cythonize(ext_modules),
    cmdclass={
        'build_ext': BuildExtension
    }
)

推荐阅读