Currently, Anki warns about exact duplicates in red. However, it’s still easy to accidentally create notes that are essentially duplicates but are worded differently.
For example:
“What is the capital of France?”
“France’s capital is?”
“Which city is the capital of France?”
My idea is to make this an optional feature that, when adding a note, searches for similar existing notes and highlights the card in yellow if similarity check exceeds a certain number (say 80/90). We can also configure this number in settings
Interesting, but how could you possibly test that? Are there algorithms that could do that? I mean, I did see you wrote
But looking at the web results, they all seem to be plagiarism checks. Even if that could be used for anki, wouldn’t that mean we’d have to train the plagiarism checks with the contents of all of our cards so that it could find duplicates (as an aside, do plagiarism checks actually flag similar things? Or just straight up “stolen” content due to missing credit?)
So essentially local computations would be quite high, especially for users with many cards or big cards.
there are some really fast fuzzy search algorithms out there which can allow us to find similarities between already added cards and the cards we are adding. One downside is that we may have to load all the contents to ram which might be problematic for users who have many cards.
Since anki uses sqlite, i propose using FTS5 extension rather than ‘LIKE’ operator since its much faster and efficient. We can gather the top 10 or 20 similar texts and compare each on of them with the card we are adding, in the backend with algos such as Levenshtein . If it exceeds a certain threshold (we can configure the number in the backend), we can warn the user of possible similarities.
I don’t think it would be of any help here. The distance between “What is the capital of France?” and “France’s capital is?” in your example is huge (the similarity score is < 50%), so it would either not detect it or, if the threshold is lowered enough, would falsely highlight half of the database as matches in every search. Levenshtein distance in general is suited for handling things such as minor typos and alternative spellings, not for measuring semantic proximity.
Cosine similarity seems like a better fit for the task, and there are even extensions for SQLite (which Anki uses) that supposedly can detect matches in log N time. They rely on precomputed vector embeddings, however, which in case of Anki would pose multiple demanding requirements on a model used, as it will have to
be able to run locally and on any OS and hardware Anki exists for
work with multiple languages
produce embeddings for whole sentences, not just individual words
The closest thing I could find is the multilingual version of MiniLM (which, surprisingly, runs on CPU in almost real time). But even this small model takes as much space as whole Anki itself, and uses considerably more RAM than I have ever seen Anki using.
So this clearly is not just a minor side feature overall, and probably shouldn’t be distributed with the app itself until language models become significantly more compact and efficient (not sure if this can be implemented as an add-on, considering the changes to the database it will require).
It would be a nice thing to have without doubt, though. Not because of duplicate detection alone, but also because of siblings and database handling in general, as well as a deeper understanding manually traversing a relation graph can add to learning.