Update simulator for FSRS to compare with Anki's built-in scheduler

Continuing the discussion from New progress in implement the custom algorithm:

I updated the simulator based on the DSR memory model and compared Anki and FSRS.

The setting of experiment is:

# parameters for FSRS
w = [1.014, 2.2933, 4.9588, -1.1608, -0.9954, 0.0234, 1.3923, -0.0484, 0.7363, 1.6937, -0.4708, 0.6032, 0.9762]
requestRetention = 0.9  # recommended setting: 0.8 ~ 0.9

# parameters for Anki
graduatingInterval = 1
easyInterval = 4
intervalModifier = 1
newInterval = 0
minimumInterval = 1

# common parameters
maximumInterval = 36500
easyBonus = 1.3
hardInterval = 1.2

new_cards_limits = 20
review_limits = 1000
learn_days = 450
deck_size = 6000

# get the true repetitions from review logs
filename = "ALL__Learning.apkg"

# smooth curves
moving_average_period = 14
  • The parameters for FSRS are generated from my collection.
  • The parameters for Anki is the default setting.
  • New cards/day: 20
  • Maximum reviews/day: 1000
  • Days to simulate: 450
  • Deck size: 6000

Note: The simulator hasn’t supported (re)learning steps. And I won’t implement it because steps bigger than one day are complicated to handle.

Here are some interesting figures I want to introduce:

image

In the number of due cards per day, FSRS reduced by 10% compared with Anki.

image

In the number of repetitions per day, FSRS reduced by 8% compared with Anki.

image

The retention of Anki is too higher in early stage.


You can try it at Google Colab:

6 Likes

Nice comparison, thanks for putting this together.

Would be really interesting to be able to get SM-18 into the comparison, but obviously not possible since it’s closed source.

It’s not possible to get much hard data on SM-18, but Guillem Palau, who was a heavy Anki user for 11+ years has mentioned that he’s ended up with slightly higher retention from SM-18 but with roughly half the repetitions of Anki SM-2.

4 Likes

This simulator is a great idea. I’ve been using a custom scheduler based on a speculation of how a certain app works, but could never get the retention above 90%. This simulator has helped me work out some quirks. Thanks.

FSRS looks promising. However, it requires the right weights for each study subject. This means more tuning and customization from the user’s side which has always been a headache for those new to SRS. I would not advice users to go above 20% FI (below 89.62% retention). This has proven to harm learning leading to problems down the line. It was written extensively in SM blogs.

[quote]
Setting the forgetting index above 20% would be like giving up SuperMemo altogether and coming back to remembering only that what is easy to remember…Nevertheless, if you want to maximize the speed of learning with little control over what actually stays in your memory, set the forgetting index to 20%[/quote]

I think if we have more shared profiles, we could analysis these in depth. Maybe create a tool to wipe the card data and export them for anonymous sharing? I didn’t want to take up the task myself, any brave souls? I’ll leave you with some screenshots comparing different algos, it took forever to generate these. (Seems like I’m limited to 5 images.)





3 Likes

I’ve been using a custom scheduler based on a speculation of how a certain app works, but could never get the retention above 90%

Are you able to share the details of the custom scheduler that you’re referring to here?

These graphs that you shared are interesting. I never heard of the “kensho” algorithm before (I can’t find anything on google either). It seems strange that it’s able to maintain a near 95% retention with the lowest amount of repetitions compared to all algorithms. Generally, the higher the retention rate, the more reviews you need to do (smaller intervals), so how is this possible?

2 Likes

The latest version of FSRS optimizer can calculate the optimal retention, and you can try it with the simulator.

1 Like

It only counts the retention on reviewed cards per day. If an algorithm only schedule the review on a small set of cards, the retention could be higher. To avoid this cheating, I will add a figure to show the retention on all cards.

1 Like

I update the simulator in v3.8.0 with counting the expected memorization.

@kuroahna

Sorry for the misunderstanding. Kensho is a code name for a scheduler I made for personal use. Nothing published. It’s a no brainer which app I was referring to.

@L.M.Sherlock

Your link above still points to the old version using google colaboratory.

Calling it cheating is a bit rude don’t you think, the goal is to get to the cause of the problem. It would be nice if we can use a common apkg file for these tests. Can you provide a link to an apkg that I can use? I don’t really trust this collection. It’s full of old scheduling data, so it might be throwing off your simulator.

Full disclaimer: For the images below, I took the changes for “expected_memorization_per_day” and patched it to the old version v3.3.2 for this test. It would be too much work otherwise.

1 Like

Sorry for that, I meant the indicator is misguiding. It is interesting to do some research on your scheduler.

Can you use the collection file in this issue?

I don’t know the details of your own scheduler. So I have some problems to these figures:

1 Like

That’s probably because I was messing around with different startup intervals and forgot to change it back for this test?

The result is less pronounced on a regular user’s collection, so it’s definitely something in my collection that’s giving the algo an advantage. I’ll dig into it later. This is addicting, but also mentally distracting.

Are you interested in making your scheduler algorithm open source for others to see and provide feedback? There’s not going to be much discussion here if all we can see is just the graphs

4 Likes