[Wiki] Bundling Python modules with add-ons

Anki Development
1

This post was split from Bundling Numpy in an add-on to summarize and discuss best practices.

Problem

Many add-ons rely on non-standard Python packages to work, but only standard library packages and third party packages used by Anki itself are available out of the box.

Solution

Anki doesn’t offer a ready solution for this. The solution recommended in the add-ons guide is to package required modules with the add-on in a subfolder and adjust sys.path to make them available for import.

Steps

  1. pip install -r requirements.txt -t src/vendor to download packages in your build script. A lot of popular add-ons simply include third party packages directly in their git repos, but this is not recommended.
  2. In your add-on’s __init__.py before importing any vendored package, make sure to modify sys.path:
sys.path.append(os.path.join(os.path.dirname(__file__), "vendor"))

Problems with the recommended solution

  • As noted in the add-on docs, modules that rely on C extensions such as numpy are more complicated to package, as you’ll need to bundle different C modules for each platform and Python version supported.
  • If two add-ons bundle the same package, the package imported first will win and override the other one. This can result in errors if the versions are incompatible.

Alternative solution

@abdo is working on a script that solves both problems:

  1. Packages that rely on C extensions are detected and handled by bundling required all extension files for the given platforms and Anki versions in the package’s installation folder.
  2. The package version conflict issue is solved by rewriting absolute imports in vendored packages so that the sys.path workaround is not needed.

Problems with the alternative solution

  • While import rewriting appears to be working almost perfectly, some unit tests would be nice, as the logic was mostly generated by AI.
  • The vendor.py script might need some adjustments to make it easier for other add-on developers to use, as it makes assumptions about the add-on’s structure.
  • Some packages might make runtime assumptions about the way they are imported. We’re only aware of a minor issue in structlog that has a simple workaround.
  • A weird mypy error appeared with import rewriting in the pydantic package. We’ve not got to the bottom of it yet, but it can be silenced with this in mypy.ini:
[mypy-src.vendor.pydantic.types]
follow_imports = skip

Open problems

  • On Windows, packages with C extension files notoriously cause permission errors when the add-on is update/deleted. Possible solutions include:
    • Storing vendored packages somewhere outside of the add-on’s folder. This is probably incompatible with import rewriting so it trades off one problem for the other.
    • Deferring the update/deletion process to the next Anki startup before add-ons are loaded. Possible implementations include:
      • Patching the relevant aqt.addons methods to defer the process. This requires bundling an executable to restart Anki, update/delete the add-on and launch Anki again. We have a working demo of that: see updates.py and restart_anki.py.
      • Making Anki take care of the update/deletion process in this case with minimum add-on work, maybe via an add-on manifest key, requires_restart, that makes Anki prompt the user to restart then completes the process at next startup.

References

4 Likes
2

If New online installer/launcher goes ahead, add-on authors could explain to users how to add numpy to their environment, skipping the complicated vendoring step, but pushing some extra complexity on your users.

3 Likes
3

The ability to add new dependencies and change the Python version in the new online installer is really awesome for developers! Not a fan of telling users to do that to make add-ons work though. It’s a fragile approach, just like thee sys.path solution, due to the possibility of conflicts with other add-ons or Anki’s own dependencies.

I spent a lot of time writing a script to handle most vendoring pains: ankiscripts/src/ankiscripts/vendor.py at master · abdnh/ankiscripts · GitHub

This handles downloading and bundling native modules for multiple platforms. Most recently, I was working on the idea of rewriting absolute imports in vendored packages so I can get rid of the sys.path hack. This approach appears to be working well so far (Thanks to Claude 4 for the help).

The biggest remaining issue for me is packaging both x86_64 and ARM64 binaries for macOS. I guess I have to look into universal builds or just implement an online installer.

2 Likes
4

Agreed that it’s not very user friendly, and frequently encouraging users to modify that config file would likely lead to more breakages/support requests. I was mainly thinking of developers that want to do fancy things with cumbersome-to-vendor packages, that may not bother sharing at all otherwise.

That’s a well-shaven yak you have there! :slight_smile:

One potential middle ground between users modifying pyproject.toml and add-on authors bundling the third party add-ons might be to do the vendoring after add-on install. The add-on could detect a first-run scenario, and use Anki’s bundled uv to install the platform-specific requested packages into a vendor folder inside the add-on.

2 Likes
5

Exactly :sweat_smile:

That’s a good solution that I want to explore.

Another very common (and very annoying) issue with extension modules on Windows are permission errors thrown when the user uninstalls/updates the add-on[1]. I’d like to look into storing vendored packages somewhere outside the add-on’s folder as a solution, but that will probably make imports complicated or force me to go back to the sys.path method.

  1. Even having the add-on’s folder open in Explorer causes permission errors on Windows for that matter :weary_face: ↩︎

1 Like
6

It’s a shame that Python appears not to have a way to unload extension modules after they’ve been loaded. For some add-ons, perhaps a workaround would be to use the multiprocessing module to load them in a separate ephemeral process, though I imagine that would not always be practical.

2 Likes
7

Added a PoC in Add some helpers to allow add-ons to install packages into the venv · ankitects/anki@bb1b289 · GitHub

2 Likes
8

This is very welcome, for HyperTTS I package a minimal set of extra python modules, and I have plans to package something larger. @dae do you encourage people to experiment with this POC ?

1 Like
9

@abdo I am looking for a more rigorous way of packaging vendor modules for HyperTTS (currently very manual), do you recommend adopting your script ? I’ve been thinking that experienced addon developers could get together and sort of agree on best practices here.

2 Likes
10

I’ve been using that script for ~2 years. Recent additions such as import rewriting and handling of macOS’s universal libraries are less tested, but they have been working well for me in a medium-sized add-on since I added them last month. I’m also considering using the script for the AnkiHub add-on soon (it also uses a manual approach).

That would be great. This area is a source of incompatibilities and issues in big add-ons. I’ll look into adding unit tests (at least for import rewriting) and releasing this as a package to make it easier for other add-ons to use.

1 Like
11

Where should we store our discussion ? This forum, or a wiki somewhere ? for example I’d like to discuss ideas on how to add external python dependencies in light of the Python 3.13 upgrade. I’m looking at what’s needed for HyperTTS right now.

1 Like
12

@dae took a closer look at the structure of Anki 25.07 with ~/.local/share/AnkiProgramFiles/.venv/lib/python3.13/site-packages. I got the impression that for stability, we might not want to encourage addon developers to directly interact with that virtual env the way it’s done in Add some helpers to allow add-ons to install packages into the venv · ankitects/anki@bb1b289 · GitHub, as that may make it difficult to rollback, and @abdo 's solution seems preferred as it provides more isolation between addons. Any thoughts ?

1 Like
13

That would be great.

I wonder whether it would actually be that problematic… at least if mainly used for platform-specific/large packages, and with active maintainers:

  • Each time the user upgrades/downgrades Anki, any add-on additions will be overwritten, so add-ons can’t permanently bork the installation.
  • The “uv add” command will use the existing lockfile and anki-release constraints when resolving versions, so the add will either succeed, or leave things untouched if the listed additions are incompatible with the existing environment.

In the example code I provided, I suggested add-on authors specify exact versions of dependencies, because otherwise they can end up as beta versions when the user has betas enabled. If we could solve that in another way (perhaps by explicitly listing aqt and anki as well as anki-release in the requirements to avoid UV_PRERELEASE=allow), then it would be less likely to conflict.

In the case where two add-ons try to install conflicting dependencies (eg numpy==2 vs numpy==3), the first one to install would win, causing the other add-on’s add to fail.

The alternative might be to use uv to install into a separate vendor folder, and put that on the path/do import rewrites. That’s going to be the most resilient to changes (or lack thereof) by other add-ons, but it does make more work for the add-on author.

Maybe another solution here is to put some sort of deletion marker in the add-on, and defer deletions until the next startup or when the add-on is installed again.

1 Like
14

I’ve implemented something similar recently where add-on updates/deletions are deferred to the next startup. This actually required bundling an executable to restart Anki (something similar to @Shigeyuki’s AnkiRestart add-on), delete/update the add-on and launch Anki again. I’ve been thinking of a general approach suitable for contribution to Anki.

Starting from this post would be good maybe? Bundling Numpy in an add-on - #3

1 Like
15

I have noticed this the one time I tried to bundle pydantic (and the associated rust-based pydantic-core). I rolled that release back and went back to a python-only solution.

On Windows, packages with C extension files notoriously cause permission errors when the add-on is update/deleted

I’m wondering whether packaging the modules in user_files, in a versioned folder might work.
For example in HyperTTS, I could have my external dependencies in <addon ID>/user_files/external-1.2.3/pydantic-core. Then I would have a download process which downloads a full archive containing a new package containing all my dependencies into say <addon ID>/user_files/external-2.0.0/pydantic-core This way, the new shared library could be unpacked safely and the addon could start to make use of it on the second startup. It does mean an addon update would require two restarts. But it’s perhaps already the case with all the solutions presented in this thread.

1 Like
16

This also suffers from the permission issue because Anki needs to overwrite the whole add-on folder on updates (after backing up user_files).

I think a second restart is only required if you go for the runtime download option (as opposed to the approach in my vendor.py script).

1 Like
17

The latest launcher has solved this, so explicit versions are not required.

My thinking was that Anki could do the deletion on startup, by checking for deletion records prior to loading add-ons, which should avoid any conflicts.

Updates could potentially be queued up + done on startup as well.

Ideally, we’d be checking for updates at program startup, at least after the user has changed Anki version. I imagine that might break a bunch of existing add-ons though, as some are presumably assuming they’ll already be loaded by the time Anki starts up. :frowning:

1 Like
18

(Scripts published)

1 Like
19

So far code signing has been an issue when restarting Anki from an external app, (Without code signing, Windows will detect a Trojan horse and delete the app, and Mac will probably not be able to launch it.) so I think developers who aren’t professional programmers might need a code signed app to use this method.

Recently I may have found some workarounds for this (but I haven’t checked if these work cross platform yet). It seems that starting a new Anki when AnkiApp’s QLocalServer is deleted will almost certainly avoid duplication (_srv QLocalServer identifies whether Anki is duplicated or not), like this:

mw.app._srv.destroyed.connect(run_new_anki)

Another way is to delay startup using “shell=True”, e.g.

# win
subprocess.Popen('timeout /T 5 && "C:\\Program Files\\Anki\\anki.exe"', shell=True)

# mac, linux?
subprocess.Popen('sleep 5 && "/Applications/Anki.app"', shell=True)

Other, I think a safer and simpler way might be to incorporate a bit of code into the desktop.
If Anki has not yet completely closed Anki will close due to aqt’s if not sock.waitForBytesWritten(self.TMOUT):, so it seems that it would be better to auto retry for about 10 sec at this point, rather than displaying a message and immediately exiting Anki.

anki/qt/aqt/__init__.py line 363

anki/qt/aqt/init.py line 363


    def sendMsg(self, txt: str) -> bool:
        sock = QLocalSocket(self)
        sock.connectToServer(self.KEY, QIODevice.OpenModeFlag.WriteOnly)
        if not sock.waitForConnected(self.TMOUT):
            # first instance or previous instance dead
            return False
        sock.write(txt.encode("utf8"))
        if not sock.waitForBytesWritten(self.TMOUT):
            # existing instance running but hung
            QMessageBox.warning(
                None,
                tr.qt_misc_anki_is_running(),
                tr.qt_misc_if_instance_is_not_responding(),
            )

            sys.exit(1)
        sock.disconnectFromServer()
        return True

But there are very few add-ons that require such a restart (only advanced developers use C language), and retrying may pose a risk of unexpected problems (e.g. strange errors occur when there are many duplicates?), so I’m not sure if there is a demand for incorporating it into the desktop. (I think it is safest to simply restart manually.)

So I think it’s something like this for now:

  1. Bundling an executable: Most reliable, but requires code signing and the file size is a bit large.
  2. Add-on only: Lightweight and easy, but this is run before Anki exits, so it may be unstable.
  3. Anki for desktop: Convenient if available, but general Anki users may not use this function.
1 Like
20

How common is this issue on Windows? So far my script only got testing from 1-2 real users, so I’m in the dark about this.

For the add-on update/delete case, the restart script is only needed on Windows.

1 Like