How best to implement randomisation into Anki

[stabicool]

Hi all,

I’m one of the developers behind the Malleus Clinical Medicine — think of it like AnKing, but tailored for Australian medical students. Our team has been working on incorporating randomisation into our Anki note types, and we’ve had a lot of internal discussion about how best to implement this while still respecting how the Anki algorithm works.

I’m making this post to gather opinions and feedback from the wider Anki community.

Here is the randomisation in action:

We’ve uploaded a brief showcase on YouTube showing how our new note type enables randomised, dynamic cards (e.g. for VBGs, GCS, etc.): https://www.youtube.com/watch?v=xpgUj3s4hKw

Each time the card is viewed, various elements are randomised and can be attributed a score to which can change the answer accordingly (this is quite helpful for medical cards where scoring is needed). Simple randomisation can also be implemented so that it can just pick a random set of text or images from a selection or a random number from a number range.

Can Anki be used to test knowledge Application?

Generally these flashcards are used in the context of testing a user’s ability to apply the knowledge that they know as opposed to simple fact recall.

Our main concerns about randomisation were two fold:

• Anki is a flashcard application. Flashcards are inherently static always having the same front of the card and users are expected to recall the same information for the back side of the card. Randomisation defeats this purpose of using flashcards for fact recall.
• Can the Anki FSRS or similar algorithm be used to test a user’s ability to apply knowledge rather than recall knowledge? And in effect, would the algorithm be confused by users rating the card differently because its inherently a different card that appears on the subsequent reviews vs in the long term would the Anki algorithm be able to efficiently calculate intervals for knowledge application once the card matures?
• Randomisation of cards also means that in the short term when the card is shown again in the learning phase (often in the same review session), it remains as a slightly different card so perhaps it offers some advantage in this domain?

Our team had some discussion about the above concerns in our committee meetings and we could not really reach a true agreement.

We also had some discussion about how to best implement randomised cards in the context of having one note vs several notes. This is best shown with an example:

Example #1

We’ve been debating two main options for how to create cards that test interpretation of BMD T-scores (e.g. normal, osteopenia, osteoporosis).

Users are expected to remember the cut-off values for a T-score and that spcifically a T-score below -2.5 means osteoporosis and a T-score below -1.0 means osteopenia and above that would be normal. We intend to keep cards that test the actual cut-off values in our deck (e.g. A T-score value {{c1::≤-2.5}} indicates {{c2::osteoporosis}})

However there are two main ways that the application of the knowledge can be tested, that being in a single note vs several notes.

Single note implementation

A single note would look like:

What does the bone density of a T-score of [scorednumber:-3.5,-2.5,-1.0,0:1:0,1,2] indicate?
{{c1::[answerbyscore:0:osteoporosis:1:osteopenia:2:normal bone mineral density]}}

The card would randomise a T-score between 0 and -3.5. The user would then determine the interpretation of the randomised T-score
(normal, osteopenia, or osteoporosis). Issues raised with this implementation would be:

• Could confuse Anki’s algorithm, since the card technically tests three different facts
• Might result in inconsistent scheduling. For example, you know that a T-score of -0.6 is probably normal so rate it good, the next time it shows -1.3 but you thought osteopenia cut-off was -1.5 so you rate it again. This can also be seen as a positive as Anki realises that you failed to apply the knowledge so it shortens the interval

Multiple notes implementation

Each note tests one diagnosis category.
Within each note, the T-score randomises only within the appropriate range.

One of the three notes might look like:

What does the bone density of a T-score of [random:-2.5,-1.0,1] indicate?
{{c1::Osteopenia}}

The disadvantage of this being that it is less efficient and involves making multiple cards. It’s also not possible to make randomised cards with multiple variables that are scored like the CURB-65 example card above or a GCS card shown below:

What is the GCS score for a patient with the following findings
Eyes: [scoredlist:Opens spontaneously:4, Opens to speech:3, Opens to pain:2, No response:1]
Verbal: [scoredlist:Oriented to person place and time:5, Confused:4, Inappropriate words:3, Incomprehensible sounds:2, No response:1]
Motor: [scoredlist:Obeys commands:6, Localises pain:5, Withdraws:4, Abnormal flexion:3, Extension:2, None:1]

GCS: {{c1::[showscore]}}

Example #2

Another example might just be randomised images of what a certain skin disorder might look like. It can be implemented as just several different notes where the front of each note is just a different image of the same skin condition. Alternatively using the randomisation, it can be made to be just the one note and the note would show a random image of the skin condition from a selection of images on each review. Would the latter be better and effective in terms of knowledge retention and application and would Anki effectively calculate the intervals given the varying nature of the front of the card?

Concluding Remarks

We are curious what the wider community thought

• Can Anki effectively test knowledge application using randomisation or should it stay with static flaschards for simple fact recall?
• If it can be used to test application, what is the best way to use randomisation in note types together with Anki?

Looking forward to hearing your thoughts

[dae]

Personally, at least in your example, I feel like it’s a suboptimal approach. Even if the user answers the test questions correctly, you don’t know if their internal idea of the cutoff points is exactly correct. And if the card queried those cutoff points directly, you can get by with a single card, instead of needing multiple examples.

[stabicool]

@dae good point. Perhaps the T-score osteoporosis cards are not well suited for randomisation. I suppose we can put the cut off values into the extra field but it again may not appropriately test a user.

I suppose the GCS card and the CURB-65 card example are better suited in that case since they ask the user to actually calculate the score for the patient.

[safely]

I don’t think there is anything inherently wrong with your approach at all. While normally, Anki cards are a 100%-system, asking you to recall everything the card contains, randomization can work as well if you ask me. I only use it for different audio examples at this point, which is more clearly simple and without downsides.

In my opinion, consolidating the knowledge is important and your approach to integrating this is a great idea in itself. Another advantage is that the users may learn incrementally, not being required to learn exact values immediately, but learn by practice and narrow down the thresholds over time.

Single note implementation: With this, you risk that a user who can correctly answer all examples but one will pass on Anki and not be shown this particular example in time, leaving them thinking they don’t have a gap in their knowledge. This is a question of review intervals and numbers of combinations: The higher, the more likely this is. But since review intervals grow exponentially, you’ll likely run into this risk, especially if you have more than just 2 or 3 different cases. You’ll have to adjust the parameters so that one card can cover that much content. Users may also become frustrated if they know 99% of cases and the other 1% keep on requiring the card to be repeated. The cases may have different Ease, after all. But maybe this is also beneficial for you, because certain conditions are much less likely than others. It may make sense to look at the likelihood of each card and set them systematically.

Multiple notes implementation: This is, of course, less elegant and efficient to implement, but more accurate and rigorous in its results. For multiple variables, I wouldn’t want to use this. Also, you lose the advantage of being able to control the likelihood of each case, if you want that. On the other hand, it will give each example a different Ease, which is more appropriate than handling them as one in my eyes.

I think both approaches are promising and you should test it. Personally, I’m the type of person who would want the multi-note version. Also, what you are trying to achieve reminds me of Cloze cards, in which the number of cards is dynamic and dependent on the number of keywords in the note. It seems to me that you want a similar, but different thing: From a set of n examples, you want to make n cards. And n may be a combination of examples taken from two different dimensions: n = a * b. Maybe developing or searching for an add-on that adds such a card type would solve your problem (and those of other people!), then you’d have the multiple notes implementation without having to create multiple notes. This most closely aligns with the spirit of Anki, I believe. I assume here that you want the user to learn all examples, else this makes no sense.

[stabicool]

@safely thanks for your input, it is quite helpful. We will try make a randomisation guideline for our users.

[dae]

@stabicool

I suppose the GCS card and the CURB-65 card example are better suited in that case since they ask the user to actually calculate the score for the patient.

Different people will have different opinions, but personally, I’d argue even the GCS example is better implemented by a few cards that cover the eyes/verbal/motor scores separately, and perhaps a card that covers the different categories that comprise the GCS score. Study in Anki is most efficient when cards are small, discrete pieces of information that can be reviewed in isolation and don’t require much thought, and when you start adding randomization and multiple steps to get an answer, you make it more likely the cards will be failed due to a mistake in one step, and increase the time it takes to get through the daily reviews. I’d argue that fluency training, where you practice applying the knowledge you have, is best done separately if required, and often will be a natural byproduct of using that knowledge in the real world.