Build fails: python interpreter not inside a virtualenv?

Rumo · October 17, 2020, 7:08pm

Hi, I’m trying to build Anki from source but I keep running into this error:

maturin failed
Caused by: Expected `python` to be a python interpreter inside a virtualenv ಠ_ಠ

The python interpreter specified in my path is inside a virtual environment I’ve created using venv. I’ve tried different environments and Python installations but the result is the same (or another error, even earlier in the building process).
Does anybody have an idea what could be causing this? What special environment set-up is Anki expecting?
I would be grateful for any input.

kelciour · October 18, 2020, 8:45pm

I can’t seem to reproduce the issue with Cygwin and the latest master. Maybe try Git for Windows instead of Cygwin.

C:\>uname -s
MSYS_NT-10.0-19041

Evandro reported and fixed it in Makefile a while ago and I’m not sure why it’d appear again.

github.com

ankitects/anki/blob/09a6a8078e1c6f2bde47f88fa3efd51c9f347d76/Makefile#L64-L67


      
          # 	https://github.com/PyO3/maturin/issues/283 - Expected `python` to be a python interpreter inside a virtualenv
          	set -eu -o pipefail ${SHELLFLAGS}; \
          	"${PYTHON_BIN}" -m venv pyenv; \
          	case "$$(uname -s)" in CYGWIN*|MINGW*|MSYS*) \

Only now I had run for the first the the make build command (on Windows), and I figure out it was not working due this problem (PyO3/maturin#283 - Expected python to be a python interpreter inside a virtualenv). Then, to fix it I had to manually edit the anki/pyenv/Scripts/activate script. Also, I am not sure why, but the Python interpreter on the GitHub Actions Windows was create the file anki/pyenv/Scripts/activate with CRLF line endings, which I had to convert to LF, other the shell script would never be sourced.
...
"${PYTHON_BIN}" -m venv pyenv && \
case "$$(uname -s)" in CYGWIN*|MINGW*|MSYS*) \
    dos2unix "${ACTIVATE_SCRIPT}" && \
    sed -i -- "s@VIRTUAL_ENV=\".*\"@VIRTUAL_ENV=\"$$(pwd)/pyenv\"@g" "${ACTIVATE_SCRIPT}"; \
    ;; esac && \
. "${ACTIVATE_SCRIPT}" && \
...

github.com/ankitects/anki

Create actions for Windows and Mac OS

ankitects:master ← evandroforks:create_actions_for_windows_macos

opened 02:21AM - 25 Mar 20 UTC

evandrocoan

+381 -90

I saw you recently created the `uses: ankitects/setup-protoc@master`, because `s…etup-protoc` was not working. I also had the same problem when I was creating the GitHub actions for Mac OS. The solution come after opening a issue on (https://github.com/actions/virtual-environments/issues/602 - API rate limit exceeded for Mac OS hosted by GitHub itself). The problem was that the README.md of the `setup-protoc` project was not updated to use the correct version of `setup-protoc`. I opened a issue there, asking for their README.md to be updated (https://github.com/arduino/actions/issues/23 - Mark the repository as deprecated). Meanwhile, on this pull request, I am already using the correct version of the `setup-protoc` actions as suggested by https://github.com/actions/virtual-environments/issues/602. 1. I created an Mac OS tests, after fixing (https://github.com/actions/virtual-environments/issues/602 - API rate limit exceeded for Mac OS hosted by GitHub itself). 1. I added to also run `build develop` because while I was building this I got a case where `make check build` was working/passing, but `make develop` not. I put the develop rule first because it should be the most important, i.e., a user can run the develop build without errors. Then, I check whether Anki can pack itself into python wheels and finally I run all unit tests. Also, by putting `develop` before `check` allows to each one of them build half of things. If `check` is the first rule, it build all rust libraries at once instead of just part of them. 1. I separated `make develop`, `make check` and `make build` in different rules just to: 1. Know more easily in which step the build failed 1. Know how much time each one of them is taking to complete 1. I created caches for everything I could, to speed up the build process. GitHub automatically deletes the caches every week and the caches are also regenerated on certain conditions: 1. For the cache of `pyenv` directory, they are rebuild when you edit a `Makefile` or some `requirements.txt` file. 1. For the cache of package manager `pacman`, they are rebuild every time you edit the `windows_checks.yml` file. 1. To build the project using all caches should take: 1. About 7 minutes on Linux and Mac (without cache about 12 to 20 minutes) 1. About 10 minutes on Windows (without cache about 22 minutes) 1. Additionally to all these, you can always for the cache to be rebuilt by editing the line `key:` and increment its the number at the end of the line or just by changing anything else on this composition. 1. I created the special environment variable `SHELLFLAGS` to allow the user to easily debug installation errors by defining the environment variable `SHELLFLAGS=-x` before running anything. The flag `-x` will cause bash to print out every command it is running. This is vital to figure out how to make the GitHub Actions to work. 1. I also added by default the `-u` flag to all shell scripts. All `Makefile`s were already using it, then, I decided to normalize it and keep everybody using it. The `-u` flags causes the shell script to fail if it attempts to use a variable which is not defined. If a variable can be undefined, we need to explicitly say it by adding this before using it: ```shell if [[ -z "${variablename+x}" ]]; then variablename=; fi ``` 1. It was difficult to use the perl `rename` script on Windows because the `cmd` terminal has a builtin command called `rename`, this it was always taking precedence over the perl script `rename`. To fix it, I directly call it with `perl rename` on windows. I also could not figure out where should I put this `rename` script, then, I just added it to `.gitignore` and used a simple `curl` to download it. 1. After [several times](https://anki.tenderapp.com/discussions/beta-testing/1866-unable-to-parse-translations-file-on-windows-due-crlf-line-endings) trying to use `Cygwin` or the `msys2` packages, the only way to make it working was using the Unix Environment already shipped with `Git for Windows`. The only problem is that it does not have a package manager. Then, I installed one (pacman, https://github.com/actions/virtual-environments/issues/596 - Update/Add Git for windows SDK). 1. I moved some declarations on the `Makefile`s as `MAKEFLAGS += --warn-undefined-variables` to the top of the `Makefile`s because they only start working after they are defined. 1. I have removed the `@set -eo pipefail &&` which the main [`Makefile`](Makefile) had because it was redundant. You were already setting the make setting `.SHELLFLAGS := -eu -o pipefail` which makes all command to be run through shell with the flags `-eu -o pipefail`. 1. Only now I had run for the first the the `make build` command (on Windows), and I figure out it was not working due this problem (https://github.com/PyO3/maturin/issues/283 - Expected `python` to be a python interpreter inside a virtualenv). Then, to fix it I had to manually edit the `anki/pyenv/Scripts/activate` script. Also, I am not sure why, but the `Python` interpreter on the GitHub Actions Windows was create the file `anki/pyenv/Scripts/activate` with `CRLF` line endings, which I had to convert to LF, other the shell script would never be sourced. ```sh ... "${PYTHON_BIN}" -m venv pyenv && \ case "$$(uname -s)" in CYGWIN*|MINGW*|MSYS*) \ dos2unix "${ACTIVATE_SCRIPT}" && \ sed -i -- "s@VIRTUAL_ENV=\".*\"@VIRTUAL_ENV=\"$$(pwd)/pyenv\"@g" "${ACTIVATE_SCRIPT}"; \ ;; esac && \ . "${ACTIVATE_SCRIPT}" && \ ... ``` 1. The cache jobs `Cache cargo rslib` and `Cache cargo rspy` are doing nothing as of now. I just put them there in case these directories are generated some day. 1. Now, you can access the checks page and download the Python Wheels generated by `make build`: https://github.com/evandroforks/anki/actions/runs/62647001 ![image](https://user-images.githubusercontent.com/5332158/77495982-b43f0e80-6e28-11ea-9c57-7e1a982722d1.png)

github.com/PyO3/maturin

Expected `python` to be a python interpreter inside a virtualenv

opened 05:06AM - 23 Mar 20 UTC

closed 07:26PM - 23 Feb 21 UTC

evandrocoan

I am setting up a GitHub Actions for [this project](https://github.com/ankitects…) however maturin is not failing with the exception: ``` 🔗 Found pyo3 bindings 🐍 Found CPython 3.8 at python.exe 💥 maturin failed Caused by: Expected `python` to be a python interpreter inside a virtualenv ಠ_ಠ ``` This is happening because: 1. From a `cmd.exe` Windows shell, the user call `make run` 1. `make` using an Unix Environment, [setup](https://github.com/ankitects/anki/blob/75b7ebb1565672fca3e4d5b4b61822fc8d5581c4/Makefile#L49-L54) a python environment 1. `maturin` check the `VIRTUAL_ENV` variable 1. If the contents of the variable are `f:\anki\pyenv`, then, everything works 100% 1. If the contents of the variable are `/cygdrive/f/anki/pyenv`, maturin breaks with: ``Expected `python` to be a python interpreter inside a virtualenv ಠ_ಠ`` This is quite hard to fix because by default, this is the `activate` script `venv` creates: ```sh File: anki/pyenv/Scripts/activate 40: VIRTUAL_ENV="f:\anki\pyenv" 41: export VIRTUAL_ENV 42: 43: _OLD_VIRTUAL_PATH="$PATH" 44: PATH="$VIRTUAL_ENV/Scripts:$PATH" 45: export PATH ``` Which is enough to `maturin` work, however, the script is wrong because set sets the Unix PATH variable to `PATH="f:\anki\pyenv:/bin:/usr/bin/:..."` breaking Unix things. The correct is to set the PATH variable to `PATH="/cygdrive/f/anki/pyenv:/bin:/usr/bin/:..."`. The only way I could find out to fix both `maturin` and unix, is by running `sed`: ```sh VIRTUAL_ENV="$(pwd)" VIRTUAL_ENV="$(cygpath -m "$${VIRTUAL_ENV}")" sed -i -- "s@VIRTUAL_ENV=\".*\"@VIRTUAL_ENV=\"$(pwd)/pyenv\"@g" pyenv/Scripts/activate sed -i -- "s@export PATH@export PATH; VIRTUAL_ENV=\"${VIRTUAL_ENV}/pyenv\";@g" pyenv/Scripts/activate ``` Making the original script `pyenv/Scripts/activate` become: ```sh File: anki/pyenv/Scripts/activate 40: VIRTUAL_ENV="/cygdrive/f/anki/pyenv" 41: export VIRTUAL_ENV 42: 43: _OLD_VIRTUAL_PATH="$PATH" 44: PATH="$VIRTUAL_ENV/Scripts:$PATH" 45: export PATH; VIRTUAL_ENV="F:/anki/pyenv"; ``` Now, the PATH variable is correctly set to `PATH="/cygdrive/f/anki/pyenv:/bin:/usr/bin/:..."` and `maturin` can find a Windows path like on `VIRTUAL_ENV="F:/anki/pyenv"`. Can `maturin` accept Unix paths like on the `VIRTUAL_ENV` variable to avoid such hacks with the `pyenv/Scripts/activate` script file?

Rumo · October 19, 2020, 7:05am

I was mainly trying MinGW and cmd, I had also given Cygwin a shot.
But what can I say… After reading your answer I tried Cygwin one more time and it worked! I must have done something else wrong when I last tried it.
Still the same error message with other consoles, though.
My bash skills are close to non-existent, but if someone else wants to investigate the issue, I can give further information.

For now, I’m just happy that I can finally build Anki. Thanks a lot, @kelciour.

bubblekraft · October 27, 2020, 3:30pm

I’m getting this error too, on Ubuntu 20.04, as well after setting up a virtual environment with venv…

bubblekraft · October 27, 2020, 4:46pm

I’ve just been spending the last few hours on this, with no luck
I tried using venv with ‘env’, also with ‘pyenv’ as I see the scripts using. Now I tried with pyenv and I still get the same error (maturin failed Caused by: Expected python to be a python interpreter inside a virtualenv ಠ_ಠ).

Any ideas?

andrewsanchez · October 27, 2020, 5:13pm

I haven’t looked into this in super great detail, but here is my impression: because of the way that Anki’s build system bootstraps its own virtual environment, trying to build Anki with an activated virtual environment seems to be problematic. Activating a virtual environment before building Anki with ./run or make build can cause problems; I think this is because it will then try to create a virtual environment within a virtual environment.

I suggest running make clean and perhaps even rm -r pyenv to remove the virtual environment created by Anki’s build scripts, then deactivate your virtual environment or start a new shell session. As long as you have the required python version installed and in your PATH (re: README.development), you should be able to build Anki with ./run. This might seem scary, but Anki will bootstrap its own python virtual environment and will not pollute your global python environment, so no worries there.

Rumo · October 27, 2020, 8:12pm

As I wrote above, the trick for me was to run make run -B in Cygwin. But that would also only work if I had deleted the pyenv folder before (after a failed build) or I would still get the error.

bubblekraft · October 28, 2020, 11:24am

Thank you for your reply, Andrew. I tried that, too, but I don’t know how to get it to work, because Ubuntu doesn’t have an executable called python, only python3. I tried to create a function and export it:

python() {
	python3 "$@"
}
export -f python

That does work correctly, because if I type python I get to the python3 interactive shell (which doesn’t happen without this function), but when running run or make run I just get back to the initial error, Expected python to be a python interpreter inside a virtualenv ಠ_ಠ).

At this point I’m a bit at a loss how to move forward…

bubblekraft · October 28, 2020, 11:44am

And if I delete the pyenv folder it first downloads pip, and then setuptools, then pyqt5, then pyqtwebengine, then PyQt5-sip, and then fails with /bin/bash: maturin: command not found.

If now I try to:

source pyenv/bin/activate
pip install maturin
deactivate

then it finally works! Gosh, that was complicated

So it seems that maturin isn’t correctly mentioned as a dependency? Also, it sounds important to mention in the build instructions to not use a virtual env for pythong doing it, and to tell Ubuntu users how to overcome the lack of a python executable.

andrewsanchez · October 30, 2020, 4:04pm

That isn’t the case, which you can verify by taking a look here and here

I think that is a good point. Maybe you can submit a PR with those additions? I think a simple note mentioning that Anki configures the virtual env for you would be helpful!

While not covered in detail, it is mentioned in README.development that one must have Python 3.7+ installed. That being said, it is a bit confusing since /usr/bin/python isn’t available on 20.04. I didn’t know that! Check this out:

Remaining packages in Ubuntu which require Python 2.7 have been updated to use /usr/bin/python2 as their interpreter, and /usr/bin/python is not present by default on any new installs. On systems upgraded from previous releases, /usr/bin/python will continue to point to python2 for compatibility. Users who require /usr/bin/python for compatibility on newly-installed systems are encouraged to install the python-is-python3 package, for a /usr/bin/python pointing to python3 instead.

bubblekraft:

I don’t know how to get it to work, because Ubuntu doesn’t have an executable called python , only python3 . I tried to create a function and export it:
python() {
	python3 "$@"
}
export -f python
That does work correctly, because if I type python I get to the python3 interactive shell (which doesn’t happen without this function), but when running run or make run I just get back to the initial error

I would strongly advise against this method of configuring Python. While I definitely commend your persistent efforts , that is not a sustainable solution. I would try following the instructions from the Ubuntu wiki I linked to above.

On Ubuntu 18.04, I had to follow instructions similar to those here.

For example:

update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1
update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 2
update-alternatives --set python3 /usr/bin/python3.7

and some symlink magic due to a weird issue described here:

ln -s /usr/lib/python3/dist-packages/apt_pkg.cpython-36m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/apt_pkg.cpython-37m-x86_64-linux-gnu.so

I don’t think it’s worth debugging this at this point, since it was in the context of not having python correctly configured.

dae · November 1, 2020, 10:33pm

I’ve mentioned that a venv is not required in the docs on Changing Anki's build system to Bazel

bubblekraft · November 28, 2020, 1:17pm

Firstly, sorry for the late reply.

I’m getting a 404 for the first link, and in the second link a search for maturin only finds the comment #https://github.com/PyO3/maturin/issues/283 - Expected python to be a python interpreter inside a virtualenv. Sorry if I am missing something obvious. I imagine it’s been updated since you wrote the post?

Many thanks for the reference, it’s super useful to know about the python-is-python3 package! I imagine, though, that others in my situation might also not know about it?

dae · November 29, 2020, 1:50am

I suggest you try the latest beta, as Anki has moved away from maturin since your original post.

Topic		Replies	Views
Anki build on mac fails at link time Help	7	1141	May 1, 2023
IProblem installing from src: maturin command not found Development	5	2314	May 1, 2023
Issue running binaries compiled using current git head on Arch linux within a venv Development	2	46	May 5, 2025
Build failing on Linux (Ubuntu) after latest update Development	7	120	February 11, 2025
Some help compiling Anki; I'm getting rustfmt errors that I don't understand Development	4	1040	May 1, 2023

Build fails: python interpreter not inside a virtualenv?

Related topics