Bundling Numpy in an add-on

[S3raph]

Is there a way to bundle numpy in an add-on so that it can be used as a dependency for another package? Currently struggling to get it to work.

[Shigeyuki]

Numpy is relatively cumbersome to bundle with add-on because it includes the C language.
Need to use the same Numpy wheel as the Python version of Anki. The latest Anki for desktop on Windows and Mac uses Python 3.9. Maybe numpy 2.0.2 is the last version that supports python 3.9. numpy·PyPI

e.g. The Numpy wheel for Python 3.9 on Windows is probably “numpy-2.0.2-pp39-pypy39_pp73-win_amd64.whl”. Put it in the add-on or download (or create your own Wheel) and unzip it then add the path to the sys.path. This explanation and code by developer Abdo will be helpful.

Add support for dependencies in addons

Vendoring is usually as simple as doing the following for me: pip install -r requirements.txt -t src/vendor. Adding src/vendor to sys.path, or maybe using a modified version of the snippet here where you don’t mess around with sys.modules/sys.path, and instead just use the object you get from importlib.util.module_from_spec to avoid conflicts. (I usually just append/insert to sys.path, and have not tested my suggestion yet) It becomes a real pain when C extensions are involved. For this case…

Note that Linux users often use the latest version of Python, not Anki version 3.9.
In this case it is necessary to identify their Python version and architecture. But Linux users are only a few percent so if it’s difficult you may want to exclude them from support.

Also in the future Anki may upgrade the Python version (I don’t know when that will be. Upgrade from Python 3.9.18 · Issue #4022 · ankitects/anki · GitHub). In this case the python 3.9 wheel will be broken so it will be necessary to update the wheel.

[dae]

If New online installer/launcher goes ahead, add-on authors could explain to users how to add numpy to their environment, skipping the complicated vendoring step, but pushing some extra complexity on your users.

[abdo]

The ability to add new dependencies and change the Python version in the new online installer is really awesome for developers! Not a fan of telling users to do that to make add-ons work though. It’s a fragile approach, just like thee sys.path solution, due to the possibility of conflicts with other add-ons or Anki’s own dependencies.

I spent a lot of time writing a script to handle most vendoring pains: ankiscripts/src/ankiscripts/vendor.py at master · abdnh/ankiscripts · GitHub

This handles downloading and bundling native modules for multiple platforms. Most recently, I was working on the idea of rewriting absolute imports in vendored packages so I can get rid of the sys.path hack. This approach appears to be working well so far (Thanks to Claude 4 for the help).

The biggest remaining issue for me is packaging both x86_64 and ARM64 binaries for macOS. I guess I have to look into universal builds or just implement an online installer.

[dae]

@abdo

The ability to add new dependencies and change the Python version in the new online installer is really awesome for developers! Not a fan of telling users to do that to make add-ons work though. It’s a fragile approach, just like thee sys.path solution, due to the possibility of conflicts with other add-ons or Anki’s own dependencies.

Agreed that it’s not very user friendly, and frequently encouraging users to modify that config file would likely lead to more breakages/support requests. I was mainly thinking of developers that want to do fancy things with cumbersome-to-vendor packages, that may not bother sharing at all otherwise.

@abdo

I spent a lot of time writing a script to handle most vendoring pains: ankiscripts/src/ankiscripts/vendor.py at master · abdnh/ankiscripts · GitHub

That’s a well-shaven yak you have there!

One potential middle ground between users modifying pyproject.toml and add-on authors bundling the third party add-ons might be to do the vendoring after add-on install. The add-on could detect a first-run scenario, and use Anki’s bundled uv to install the platform-specific requested packages into a vendor folder inside the add-on.

[abdo]

@dae

That’s a well-shaven yak you have there!

Exactly

@dae

One potential middle ground between users modifying pyproject.toml and add-on authors bundling the third party add-ons might be to do the vendoring after add-on install. The add-on could detect a first-run scenario, and use Anki’s bundled uv to install the platform-specific requested packages into a vendor folder inside the add-on.

That’s a good solution that I want to explore.

Another very common (and very annoying) issue with extension modules on Windows are permission errors thrown when the user uninstalls/updates the add-on^[1]. I’d like to look into storing vendored packages somewhere outside the add-on’s folder as a solution, but that will probably make imports complicated or force me to go back to the sys.path method.

---

1. Even having the add-on’s folder open in Explorer causes permission errors on Windows for that matter ↩︎

[dae]

It’s a shame that Python appears not to have a way to unload extension modules after they’ve been loaded. For some add-ons, perhaps a workaround would be to use the multiprocessing module to load them in a separate ephemeral process, though I imagine that would not always be practical.

[dae]

Added a PoC in Add some helpers to allow add-ons to install packages into the venv · ankitects/anki@bb1b289 · GitHub

[lucw]

This is very welcome, for HyperTTS I package a minimal set of extra python modules, and I have plans to package something larger. @dae do you encourage people to experiment with this POC ?

[lucw]

@abdo I am looking for a more rigorous way of packaging vendor modules for HyperTTS (currently very manual), do you recommend adopting your script ? I’ve been thinking that experienced addon developers could get together and sort of agree on best practices here.

[abdo]

I’ve been using that script for ~2 years. Recent additions such as import rewriting and handling of macOS’s universal libraries are less tested, but they have been working well for me in a medium-sized add-on since I added them last month. I’m also considering using the script for the AnkiHub add-on soon (it also uses a manual approach).

@lucw

I’ve been thinking that experienced addon developers could get together and sort of agree on best practices here.

That would be great. This area is a source of incompatibilities and issues in big add-ons. I’ll look into adding unit tests (at least for import rewriting) and releasing this as a package to make it easier for other add-ons to use.

[lucw]

Where should we store our discussion ? This forum, or a wiki somewhere ? for example I’d like to discuss ideas on how to add external python dependencies in light of the Python 3.13 upgrade. I’m looking at what’s needed for HyperTTS right now.

[lucw]

@dae took a closer look at the structure of Anki 25.07 with ~/.local/share/AnkiProgramFiles/.venv/lib/python3.13/site-packages. I got the impression that for stability, we might not want to encourage addon developers to directly interact with that virtual env the way it’s done in Add some helpers to allow add-ons to install packages into the venv · ankitects/anki@bb1b289 · GitHub, as that may make it difficult to rollback, and @abdo 's solution seems preferred as it provides more isolation between addons. Any thoughts ?

[abdo]

Created a wiki post here for organization and discussion:

[Wiki] Bundling Python modules with add-ons

To be converted into a wiki for dev discussions on the topic. See: Bundling Numpy in an add-on - #12 by lucw

[dae]

@lucw

I’ve been thinking that experienced addon developers could get together and sort of agree on best practices here.

That would be great.

@lucw

I got the impression that for stability, we might not want to encourage addon developers to directly interact with that virtual env the way it’s done in Add some helpers to allow add-ons to install packages into the venv · ankitects/anki@bb1b289 · GitHub, as that may make it difficult to rollback

I wonder whether it would actually be that problematic… at least if mainly used for platform-specific/large packages, and with active maintainers:

• Each time the user upgrades/downgrades Anki, any add-on additions will be overwritten, so add-ons can’t permanently bork the installation.
• The “uv add” command will use the existing lockfile and anki-release constraints when resolving versions, so the add will either succeed, or leave things untouched if the listed additions are incompatible with the existing environment.

In the example code I provided, I suggested add-on authors specify exact versions of dependencies, because otherwise they can end up as beta versions when the user has betas enabled. If we could solve that in another way (perhaps by explicitly listing aqt and anki as well as anki-release in the requirements to avoid UV_PRERELEASE=allow), then it would be less likely to conflict.

In the case where two add-ons try to install conflicting dependencies (eg numpy==2 vs numpy==3), the first one to install would win, causing the other add-on’s add to fail.

The alternative might be to use uv to install into a separate vendor folder, and put that on the path/do import rewrites. That’s going to be the most resilient to changes (or lack thereof) by other add-ons, but it does make more work for the add-on author.

@dae

It’s a shame that Python appears not to have a way to unload extension modules after they’ve been loaded.

Maybe another solution here is to put some sort of deletion marker in the add-on, and defer deletions until the next startup or when the add-on is installed again.