python - Module loading behavior in virtualenv differs depending on caller

Question

I found some very strange python behavior that breaks my virtualenv set-up:

I have set up a virtual python2 environment to run a python2 script and created a wrapper like this:

#!/usr/bin/python3
import sys
import subprocess
if __name__ == '__main__':
    sys.exit(subprocess.call('/path/to/venv/bin/python /path/to/python2script.py', shell=True))

so that calling the wrapper will call the python2 script using the virtual environment.

Calling this wrapper manually from the console (it is marked as executable) works as expected.

me@my_pc:/somewhere$ /path/to/the/wrapper
~ python2 script running as expected ~

Now I have a separate project (python3) which needs to call the python2 script, so I'd like to call the wrapper either with subprocess.check_call('/path/to/the/wrapper', shell=True) or subprocess.Popen(['/path/to/the/wrapper']).

However, when my separate python3 project does the call, it fails with:

  File "/path/to/python2script.py", line 31, in <module>
    import yaml
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 284
    class YAMLObject(metaclass=YAMLObjectMetaclass):
                              ^
SyntaxError: invalid syntax

So apparently, now the python2 in the virtual environment tries to use my global dist-packages, which are python3, hence it crashes.

But if I check again from the console:

me@my_pc:/somewhere$ /path/to/venv/bin/python
Python 2.7.15+ (default, Oct  7 2019, 17:39:04) 
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> print(yaml.__file__)
/path/to/venv/local/lib/python2.7/site-packages/yaml/__init__.pyc

If I call the wrapper from an interactive python3 shell (with the same subprocess.call()) it also works.

Why does python load 'yaml' from the virtual environment when I call it from the console or when I directly call the wrapper, but when another python script calls the wrapper it will use dist-packages? Could the other script somehow have messed with the search order in a way that affects the python installation in the virtual environment?

---

Possible work-around:

While not an explanation for the behavior, I found a hack that seems to make it work.

I found that the paths to the different module locations are in sys.path, but not in os.environ['PYTHONPATH'] and are in the wrong order when called from the script (with global paths first). Changing the order and writing the result back to os.environ['PYTHONPATH'] seems to solve the issue.

sys_path_venv = [p for p in sys.path if 'venv' in p]
sys_path_rest = [p for p in sys.path if not p in sys_path_venv]
sys.path = sys_path_venv + sys_path_rest
os.environ["PYTHONPATH"] = os.pathsep.join(sys.path)

标签： pythonpython-3.xvirtualenvpython-2.x