updating existing notes won't work - all notes are added as if they were new

[Spanisch5000]

Hi! I am the author of “Spanisch 5000”, a big stack (7500 notes) that is being published on Anki (8700 downloads), Flashcards Deluxe and Memorion. It is listed as the highest rated deck in the category “Spanish”. I am a language guy, not a software guy.

I am experiencing problems with the update of the stack (screenshot). Instead of updating the existing notes, all notes are being added as if they were new. I am not sure when this started. I tried to add a new field to the deck, and at this point the problems definitely emerged. I gave up on the new field and restored the deck to the state before that. But even so, existing notes are not being updated with new versions, but instead the whole deck is being doubled.

The notes have 13 fields, the first of which is the ID field (sceenshot).

I noted that strange stuff turns up in the “note types” field. The deck consists of notes of the type “B+R+hint on side 2”. I upload the deck. I share the deck in Ankiweb and then download it from there. I switch profile from my spanisch-5000@online.de profile to my s.gruben@online profile (that should simulate a user). There I import an older version of the deck. The type is also “B+R+hint on side 2”. I import the downloaded “Spanisch_5000.apkg”. That is were it happenes again: All the imported notes are being added to the deck (screenshot), and all the pre-existing notes are being renamed to a type like “B+R+hint on side 2-a4f68.”

I honestly don’t know what to do. My users have worked with the deck for years, putting hundreds of hours of learning into it. They can’t be made to lose their statistics.

I am happy to let someone use teamviever, or whatever helps.

[dae]

For Anki to be able to update previous notes, the note type’s fields must not have been changed in either the source or destination collection.

[Spanisch5000]

Not have been changed or be the same? I create a deck from an apkg file, and then update it from a second apkg file, and the note type fields match completely, and they have 90% of number in the ID field in common, and still in the update process all the notes are added instead of being mostly updated or left unchanged.

How can I make two decks (or files of a deck) behave in a way so the second one updates the first one, if the ID fields match? I am really really lost here.

[dae]

The note id and notetype id must be identical in both collections. The note can have its field content modified, but if you create new notes, they will not be associated with the previous content.

[Spanisch5000]

Hi Damien,

thank you for the hint. We already had contact last year via a user of “Spanisch 5000” named Werner Krag who knew you from some time ago because both of you had Japanese wives. Funny story. I pasted your answer below.

“Side 3” without audio, I guess that is still true?

As you may have read, “Spanisch 5000” has become quite successful, and it is now complete, after 4 years of hard work.

Can we move on via email, or does it have to be via the forum? If email is possible, here are my questions:

What is the note ID and where do I find it? Same for note type ID.

The problem seems to be, that I did an update where maybe the note type had changed for reasons that I don’t understand.

I have to find a way to offer an update for users who use previous versions of the deck, so their learning statistics are not lost. Or if there is no way they can simply update the deck with the standard procedure, I should be able to offer some solution so they cane update their decks and be safe for future standard updates.
Of course, they use different versions of the stack. And I don’t know where it began to go wrong.

For myself, I can solve the update problem like this:

I install a previous version. E.g. I open the backup file from 2021-03-05 (*.colpkg). Here I already find strange names in “Note Types”:

R+B+hint on side 2 (0 cards)
R+B+hint on side 2-42126 (12758 cards)
R+B+hint on side 2-d53d1 (0 cards)
B+R+hint on side 2-f8d07 (0 cards)

Now I import my latest *.tsf file, containing the whole deck as it is right now. When importing, the field “type” “R+B+hint on side 2-42126” appears (why?). I import this whith the options “Update existing notes when first filed matches” and “Allow HTML in fields”.
I get: Importing complete. 1150 notes added, 6348 notes updated, 2 notes unchanged. The deck is as it should be.

The same thing happens if I import the old package and first delete “R+B+hint on side 2” and then rename “R+B+hint on side 2-42126” to “R+B+hint on side 2”. In that case, while importing my tsf file, in the field “type” “R+B+hint on side 2” appears. Does this have to do with the note ID?

But as soon as I import not the tsf file but the apkg file that I downloaded after updating the deck, the update is completely added, nothing is being updated.

Best regards

Sebastian

[dae]

text imports and apkg imports are not handled the same way. Text imports detect duplicates based on the first field. apkg imports rely on an internal id. If you have two files that have differing internal ids, they will need to be made to match first, such as by starting with a copy of the target collection, and then modifying it with a text import. If you then export the resulting files, the internal ID should match the target collection.

[Spanisch5000]

Thanks for the hint. Things are getting clearer little by little. I understand my apkg files contain the information about their internal ID. This internal ID has to match for one apkg file to be updated properly by another.

Next question (I looked in google and in the maunal and did not find the answer): Where do I find the internal note ID of a deck? And how can I edit it?

The internal ID must have been modified somewhere along the way. Different users may have decks with different versions. I don’t know what internal ID they contain. They may or may not match with the apkg file people can download frotm anki web. What solution is there to the problem? I can make the users who experience difficulties updating contact me (I put a message in the description of the deck). But what do I tell them?

[ijgnd]

@Spanisch5000: I’m just an Anki user with limited technical/coding skills. But I’ve made some add-ons over the years and done some free support and next weekend I’ll have some time to look into this …

I looked at the current version you have on ankiweb. The note creation time is 2020-04-13. I guess you have a master file in some spreadsheet software and then import this into the different flash card software you support. I guess you used different ways of importing/updating your anki deck from this text file over the years which should explain why you have different note creation times.

Anki internally doesn’t identify notes, cards, note types, decks by their names but by unique ids that are usually hidden from the user. The creation time (in “unix time” in milliseconds since 1970) of the note, card, note type, deck is often used internally. When it comes to updating notes my understanding is that the deck ids don’t matter.

In Anki these ids are not meant to be changed (though you can do this with additional code with add-ons).

you can see the note ids and card ids in a browser table column if you use the add-on Advanced Browser - AnkiWeb and have changed its config so that internal fields are shown.

 

You had older versions of your deck e.g. in 2019 on ankiweb. My understanding of the importing/updating code is limited but this means that someone who downloaded from ankiweb with the note ids/note creation times before 2020-04-13 will run into problems when updating and we need to figure out a good solution for your old users.

 

I wonder if you have a field that has always been completely unchanged over the years? Do you still have all the versions of the apkg you uploaded to ankiweb over the years?

 

you could make the warning about not updating at the moment bold (surround it with <b> and </b>) and put it on top of your deck description.

[Spanisch5000]

Hi ijgnd, thank you for helping. Yes indeed, the deck is being developed in a giant google spreadsheet and then imported to Anki, Memorion and Flashcards Deluxe. Now, yesterday just before falling to sleep an idea came to my mind. Would it be possilbe to simple offer my latest *tsv file to the users that experience difficulties with the update? The one that I use myself for updating the deck? I could put a hint into the descricption text, and then make the file downloadable in the course website, togehter with a description of how to open it in Anki (program) and thus update the deck.

Is this a strike of genius or just dumb?

[ijgnd]

about the weird numbers in the note type names you originally asked about: Note types with strange names - Frequently Asked Questions

A recently downloaded deck that you uploaded to ankiweb on 2021-08-10 where the notes were created on 2020-04-13 has these fields

ID
Deutsch
Spanisch
Beispiel
AudioS
Einheit
Thematische Einheit
BildD
BildS
Bild3
Bild4
Frequency Dictionary Ranking
Nummer

I found an old version of your deck that you uploaded to ankiweb on 2019-06-14 and where the notes were created on 2019-05-30. They have has these fields

Frequency Dictionary Ranking
Deutsch
Spanisch
Beispiel
AudioD
AudioS
Einheit
Thematische Einheit
BildD
BildS
Bild3
Bild4
Nummer

 

The fields that are in both versions and that seem to be ones that are most likely to be unchanged should be “Frequency Dictionary Ranking” and “Nummer”. E.g. if “Frequency Dictionary Ranking” were a field that’s the same across all versions of your deck since 2017 and “Frequency Dictionary Ranking” was the first field in the latest tsv version of your deck (and the rest of the fields in this tsv were in the old order and the number of fields in a note match) updating should work.

 

But I see a different problem. The contents of “Frequency Dictionary Ranking” and “Nummer” are not persistent. For many notes they seem to have changed and you have improved = changed most of the other fields over the years. So I guess that in effect fully matching one field between the any old and the latest version of your deck and then updating the rest of the fields won’t work.

e.g. have a look at the note for “den Unschuldsengel spielen”.

In the uploaded to ankiweb on 2021-08-10 the note (note-id/nid: 1586782689172) has these contents:

7787
den Unschuldsengel spielen<br>(Ausdruck mit "Fliege")
hacerse la mosquita muerta
wörtl.: sich zum toten Flieglein machen, wobei «mosquita muerta» allein stehend auch etwa «die graue Maus, der Duckmäuser» bedeutet<br><br>ähnlich: actuar como si fuera un angelito (sich wie ein Engelein benehmen)
[sound:tts_hacerse la mosquita muert_5519_2.mp3]
E39

Rang 3432C
5519

In the deck you uploaded to ankiweb on 2019-06-14 the note with the nid 1559220807132 has these field contents:

2956C
den Unschuldsengel spielen
hacerse la mosquita muerta
wörtl.: sich zum toten Flieglein machen, wobei «mosquita muerta» alleinstehend auch etwa «die graue Maus, der Duckmäuser» bedeutet<br><br>ähnlich: actuar como si fuera un ángelito (sich wie ein Engelein benehmen)
[sound:tts_den Unschuldsengel spiele_3171_1.mp3]
[sound:tts_hacerse la mosquita muert_3171_2.mp3]
E34

3233

Manually adusting the “Nummer” is too time consuming and likely error prone.

One solution is to be more lenient when try to match an identifying field so that you consider den Unschuldsengel spielen<br>(Ausdruck mit "Fliege") and den Unschuldsengel spielen as matching but this might give you false matches.

Another solution might be to go back to dealing with apkg anki deck files. I assume you updated the ankiweb listing regularly, e.g. on average every two months with incremental updates. Take e.g. the version that you uploaded to ankiweb on 2021-08-10 with note creation times from 2020-04-13. Maybe you still have the version you uploaded to ankiweb shortly before 2020-04-13. This immediately preceding version with different nids likely is very similar to the old version so that e.g. most of the fields “Frequency Dictionary Ranking” or “Nummer” should match. This way you could generate a list of matching note ids. Then an add-on could be used that reads a list of nid pairings new<->old and overwrites the old note id with the new note id. But this approach works best if we had access to as many prior versions you uploaded to ankiweb as possible.

This latter solution seems time consuming. We’d better make sure that it actually works and that there are no alternatives like that tsv is no option.

I wonder about the meaning of the fields “Frequency Dictionary Ranking” and “Nummer”. How did the updating work for the other two apps?

[ijgnd]

Another idea: Google Sheets offers a file/version history. If you could share that document (only reading permission needed) it might be helpful.