How to tell pydefix which python/env to use ?

[vdbma]

I am currently using pyenv virtualenv and the automatic detection of which python pydefix should use is not working properly for me. I start a discussion as it seems I am not the only struggling with virtualenvs.

The issue: it seems that the python executable/environment found by cmake for pydefix to use is not the one I want. I didn't find a way to tell cmake which to use.

Diagnostics

When configuring with cmake $IDEFIX_DIR -DIdefix_PYTHON=ON cmake finds the correct python version to use (the current environement) :

-- Found Python: /home/marc/.pyenv/versions/3.12.7/envs/default-3.12.7/bin/python3 (found suitable version "3.12.7", minimum required is "3.7") found components: Interpreter Development.Module Development.Embed 

but for some reason once running the actual python code it fails:

from pydefix import *
import sys

print(sys.executable)

returns a different path that the one detected above. The printed path in this case is the same as returned by which python3 but not the same as the one returned by python3 -c "import sys; print(sys.executable)" (the latter is the one found by cmake).

I also see that python3 -c "import sys; print(sys.path)" and print(sys.path) inside of the pydefix script return different values.

The consequences of that are that the python modules are not found.

import numpy

returns a ModuleNotFoundError.

Attempts

I tried a couple things, but to no avail:

• a very dirty

from pydefix import *
import sys

print(sys.executable)
print(sys.path)

import subprocess
import sys

subprocess.check_call([sys.executable, "-m", "pip", "install", "numpy"])
import numpy as np

inside the pydefix script returns:

...
Pydefix: ENABLED.
Pydefix: output function DISABLED.
Pydefix: initflow function ENABLED.
Main: Creating initial conditions.
/home/marc/.pyenv/shims/python3
['', '/home/marc/.pyenv/versions/3.12.7/lib/python312.zip', '/home/marc/.pyenv/versions/3.12.7/lib/python3.12', '/home/marc/.pyenv/versions/3.12.7/lib/python3.12/lib-dynload', '/home/marc/.pyenv/versions/3.12.7/lib/python3.12/site-packages']
Requirement already satisfied: numpy in /home/marc/.pyenv/versions/default-3.12.7/lib/python3.12/site-packages (2.2.2)

------------------------------------------------------------------------------
### FATAL ERROR in function CallScript file /home/marc/Documents/src/idefix_fork/src/pydefix.cpp line 260
An exception occured while calling the Python interpreter
in file "setup.py" function "init_flow":
ModuleNotFoundError: No module named 'numpy'

...

that is pip says that numpy is already installed, then python doesn't find numpy.

• modifying the root CMakeLists.txt to remove the set(PYBIND11_FINDPYTHON ON CACHE BOOL "Idefix requires python" FORCE) line and :
  • giving the location of the python executable as a cmake argument with -DPYTHON_EXECUTABLE=... (doesn't work)
  • also adding the location of the libraries with -DPYTHON_LIBRARY=... (doesn't work either)

These two raise a cmake warning because they appear to rely on the old FindPythonInterp and FindPythonLibs.

[neutrinoceros]

When configuring with cmake $IDEFIX_DIR -DIdefix_PYTHON=ON cmake finds the correct python version to use (the current environement) :

-- Found Python: /home/marc/.pyenv/versions/3.12.7/envs/default-3.12.7/bin/python3 (found suitable version "3.12.7", minimum >required is "3.7") found components: Interpreter Development.Module Development.Embed 

Note that a Python virtual environment is not necessarily uniquely attached to a Python interpreter, even though pyenv virtualenv does entertain this illusion via path shims (symbolic links); CMake may have no idea where your virtual environment lives and still find the correct Python interpreter; in fact, if you're running CMake while you're venv is active, it shouldn't be able to find any other interpreter you may have around. The issue is that knowning the path to your interpreter doesn't necessarily help with discovering a virtual environment.

I don't know off hand how idefix should learn about virtual environments.

In case this is useful: the path to an active python environment is accessed via the VIRTUAL_ENV environment variable (regardless if you're using pyenv virtualenv, venv, uv venv ...).
I think the path of the interpreter (sys.executable) looks like $VIRTUAL_ENV/bin/python (or python3, or even python3.12, or even all three), though I'm less sure about how much of this is standardized exactly.

Also, instead of

import subprocess
import sys
subprocess.check_call([sys.executable, "-m", "pip", "install", "numpy"])

you can do

import pip
pip.main(["install", "numpy"])

but that won't help much here

[neutrinoceros]

Random thought: maybe the only solution that doesn't involve renouncing isolation of your Python environments is to containerize your setup (but then I'm not convinced it's worth using pydefix at all).

[glesur]

tested on my side, if you initialise PYTHONPATH to the right location in your environment, then it seems that it picks up the right libraries. e.g.:
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.10/site-packages

Still looking for a better answer, but let me know if this works for you.

[nscepi]

This fix works for me, thanks!

[vdbma]

this also works for me, thanks!