Update simulator for FSRS to compare with Anki's built-in scheduler

[L.M.Sherlock]

Continuing the discussion from New progress in implement the custom algorithm:

I updated the simulator based on the DSR memory model and compared Anki and FSRS.

The setting of experiment is:

# parameters for FSRS
w = [1.014, 2.2933, 4.9588, -1.1608, -0.9954, 0.0234, 1.3923, -0.0484, 0.7363, 1.6937, -0.4708, 0.6032, 0.9762]
requestRetention = 0.9  # recommended setting: 0.8 ~ 0.9

# parameters for Anki
graduatingInterval = 1
easyInterval = 4
intervalModifier = 1
newInterval = 0
minimumInterval = 1

# common parameters
maximumInterval = 36500
easyBonus = 1.3
hardInterval = 1.2

new_cards_limits = 20
review_limits = 1000
learn_days = 450
deck_size = 6000

# get the true repetitions from review logs
filename = "ALL__Learning.apkg"

# smooth curves
moving_average_period = 14

• The parameters for FSRS are generated from my collection.
• The parameters for Anki is the default setting.
• New cards/day: 20
• Maximum reviews/day: 1000
• Days to simulate: 450
• Deck size: 6000

Note: The simulator hasn’t supported (re)learning steps. And I won’t implement it because steps bigger than one day are complicated to handle.

Here are some interesting figures I want to introduce:

In the number of due cards per day, FSRS reduced by 10% compared with Anki.

In the number of repetitions per day, FSRS reduced by 8% compared with Anki.

The retention of Anki is too higher in early stage.

---

You can try it at Google Colab:

Google Colaboratory

[biodecus]

Nice comparison, thanks for putting this together.

Would be really interesting to be able to get SM-18 into the comparison, but obviously not possible since it’s closed source.

It’s not possible to get much hard data on SM-18, but Guillem Palau, who was a heavy Anki user for 11+ years has mentioned that he’s ended up with slightly higher retention from SM-18 but with roughly half the repetitions of Anki SM-2.

[L42]

This simulator is a great idea. I’ve been using a custom scheduler based on a speculation of how a certain app works, but could never get the retention above 90%. This simulator has helped me work out some quirks. Thanks.

FSRS looks promising. However, it requires the right weights for each study subject. This means more tuning and customization from the user’s side which has always been a headache for those new to SRS. I would not advice users to go above 20% FI (below 89.62% retention). This has proven to harm learning leading to problems down the line. It was written extensively in SM blogs.

[quote]
Setting the forgetting index above 20% would be like giving up SuperMemo altogether and coming back to remembering only that what is easy to remember…Nevertheless, if you want to maximize the speed of learning with little control over what actually stays in your memory, set the forgetting index to 20%[/quote]

I think if we have more shared profiles, we could analysis these in depth. Maybe create a tool to wipe the card data and export them for anonymous sharing? I didn’t want to take up the task myself, any brave souls? I’ll leave you with some screenshots comparing different algos, it took forever to generate these. (Seems like I’m limited to 5 images.)

[kuroahna]

I’ve been using a custom scheduler based on a speculation of how a certain app works, but could never get the retention above 90%

Are you able to share the details of the custom scheduler that you’re referring to here?

These graphs that you shared are interesting. I never heard of the “kensho” algorithm before (I can’t find anything on google either). It seems strange that it’s able to maintain a near 95% retention with the lowest amount of repetitions compared to all algorithms. Generally, the higher the retention rate, the more reviews you need to do (smaller intervals), so how is this possible?

[L.M.Sherlock]

The latest version of FSRS optimizer can calculate the optimal retention, and you can try it with the simulator.

[L.M.Sherlock]

It only counts the retention on reviewed cards per day. If an algorithm only schedule the review on a small set of cards, the retention could be higher. To avoid this cheating, I will add a figure to show the retention on all cards.

[L.M.Sherlock]

I update the simulator in v3.8.0 with counting the expected memorization.

Release v3.8.0 · open-spaced-repetition/fsrs4anki

[L42]

@kuroahna

Sorry for the misunderstanding. Kensho is a code name for a scheduler I made for personal use. Nothing published. It’s a no brainer which app I was referring to.

@L.M.Sherlock

Your link above still points to the old version using google colaboratory.

Calling it cheating is a bit rude don’t you think, the goal is to get to the cause of the problem. It would be nice if we can use a common apkg file for these tests. Can you provide a link to an apkg that I can use? I don’t really trust this collection. It’s full of old scheduling data, so it might be throwing off your simulator.

Full disclaimer: For the images below, I took the changes for “expected_memorization_per_day” and patched it to the old version v3.3.2 for this test. It would be too much work otherwise.

[L.M.Sherlock]

Sorry for that, I meant the indicator is misguiding. It is interesting to do some research on your scheduler.

[L.M.Sherlock]

Can you use the collection file in this issue?

Converting from Anki SM-2 to FSRS: Rescheduling gives a lot of cards?

[L.M.Sherlock]

I don’t know the details of your own scheduler. So I have some problems to these figures:

[L42]

That’s probably because I was messing around with different startup intervals and forgot to change it back for this test?

The result is less pronounced on a regular user’s collection, so it’s definitely something in my collection that’s giving the algo an advantage. I’ll dig into it later. This is addicting, but also mentally distracting.

[kuroahna]

Are you interested in making your scheduler algorithm open source for others to see and provide feedback? There’s not going to be much discussion here if all we can see is just the graphs

[Vitorvlv]

Hi! I was just using the simulator and could use some help interpreting the results.

I’ve tried to simulate the anki vs fsrs using the deck I used during 2022 with the options I was using. These were the changes I made:

W = 0.286, 1.01, 5.143, -0.6442, -0.597, 0.0331, 1.3379, -0.2, 0.7318, 1.6617, -0.5291, 0.7266, 0.5825 (value the optimizer gave me for this deck)

requestRetention = 0.9
graduatingInterval = 3
newInterval = 0.2
easyBonus = 1.5
leechSuspend = False
new_cards_limits = 100
review_limits = 300
learn_days = 364
deck_size = 3835

and I added my deck to the simulator.

To my surprise I didn’t find the same results you did on your simulation. FSRS did managed to stick to the target retention rate (90%), but it gave so much more repetitions that I’m not sure if it was worth it.

Here are the results with the 0.9 retention:

Settings the target retention to 85% did result in a better performance from FSRS. But what does this means? If I’m targeting 90% retention should I just stick with default anki?

Results with 0.85:

[L.M.Sherlock]

@Vitorvlv

Settings the target retention to 85% did result in a better performance from FSRS. But what does this means? If I’m targeting 90% retention should I just stick with default anki?

If your target retention is 90%, using default Anki will not achieve the retention.

[Vitorvlv]

That’s right, forgot about that, sorry.

I played around with the interval modifier until SM-2 was giving a similar retention to FSRS and FSRS did achieve it in less repetitions. Sorry for the confusion!

[krstoevan]

hi,
i seldom hear people talk about priority, but i heard from your supermemo lecture that priority is a great point in supermemo.

so, will frsr bring priority to anki? or something like that?
thanks