python - 我可以导入安装在全局或另一个虚拟环境中的一个虚拟环境中的包吗？

Question

我想在我的 PC 上安装两个版本的 TensorFlow (tf)。我的一些程序在 tf v1 上运行，一些在 tf v2 上运行。因此，我可以分别创建两个虚拟环境 (venv)，一个用于 tf v1，另一个用于 tf v2，并为每个 venv 使用 --use-system-packages 来满足我的代码的其他（全局）包要求。然后，我会将我的程序保存在安装了正确 tf 版本的相应 venv 中。

但是，出于某些特定原因，我也希望能够在全局环境中使用这两个版本的 TensorFlow（即无需输入/激活 venv）。

问题：

有没有一种方法可以让我在 2 个 venvs 中安装 2 个版本的 TensorFlow（和/或 numpy），然后使用 import 语句在全局环境中访问它们中的任何一个（使用指定的版本/venv 导入）？如果使用 venvs 无法做到这一点，是否还有其他方法可以实现相同的效果？

进一步扩展相同的，假设我在一个 venv 中。我想知道我是否可以导入安装在其他 venv 中的包？如果是，如何？

标签： pythontensorflowvirtualenv

Answer 1

Problem definition: we need to import two different versions of a Python package. Both versions have dependencies and provide the same namespace (tensorflow).

Why it is not possible: when Python loads a module (import tensorflow as tf), it tries to avoid redundant work and skips modules that were loaded already. So, it is not possible to load the same namespace from two different locations.

Few fun facts:

• You can manipulate paths where Python will look for a module at runtime. When loading a module for the first time, Python will go through a list of directoris in sys.path, looking for the requested namespace. Directories in sys.path are installation dependent, but usually go in this order: local directory, builtin modules, installed packages. The last one is a directory or a set of directories where your package manager (pip, easy_install, ...) stores installed packages. It is usually more than one location, including packages installed system-wide, user folder (e.g. ~/.local/lib/pythonX.Y/site-packages), and virtual environment-specific folder (e.g. xxx.venv/lib/pythonX.Y/site-packages). Depending on the setup, you can have any combination of these. The fun part is that you can insert new directories: sys.path[6:6] = ['path/to/another/venv/lib/pythonX.Y/site-packages']. However, it won't help getting two instances of the same namespace because of caching.
• In sys.path folders, Python will look for a module with matching name, which represents a namespace. Python package names are often the same as the namespace they provide, but they can also be different. For example, in pip install numpy the last portion is a package name, but in import numpy it is a namespace. Roughly 1/3 of all PyPI packages' namespaces are different from the package name, for example package intel-numpy also provides namespace numpy. If you install intel-numpy, pip will silently overwrite the folder (and so, the content of the namespace) previously provided by numpy. Yep, Python package managers keep track of installed package names, but not namespaces. So, even if two versions of a package provide different namespaces, you cannot have them both. But if you have two different packages providing the same namespace, pip will be happy to install both, but will only keep the last installed.
• If two versions of the same package provide different namespaces, you can install them different virtual environments, then manipulate sys.path to include packages from both and get both namespaces at the same time. Not our case, though.
• Another option: repackage one version of the package, renaming the namespace. In this case, it's easier to also change the package name and install them in the same place/venv. You might even reupload this repackaged version of the package to PyPI (I've seen people doing this), but it won't work with TF and any other non-trivial packages because of the diference in dependency versions. Fun fact: Python package managers don't resolve dependency versions; if package A requires C==1.0 and B requires C==2.0, after installing A and B you will get only one version of C, crippling either A or B.

Finally, why you should not do that: it is against Python principles. Explicit is better than implicit, and manipulating library paths at runtime is not the most transparent and supportable approach. Still, the problem of maintaining a compatible set of dependencies is real, and this is why we have venv, pipenv etc.

From the language design perspective, dealing with multiple versions of the same package does have its benefits and was adopted by several languages (e.g. JS). However, it makes imports a lot more expensive. Historically, this is a niche scenario for Python community, so it doesn't outweigh the performance penalty.