Replace CMRR with workload-vs-DR graph (+more)

Hmmm. This problem is beyond me. If there’s data in that region of the parameter space (seems to be the case), if the user has since hit optimize parameters (seems to be the case), and if the user isn’t abusing hard (seems to be the case), and if 28 days is indeed inaccurate (seems to be the case), then this is beyond the realm of the mathematics and statistics of FSRS or using low DR values like 0.7… It’s probably some technical issue unrelated to DR=0.7, but for which the low DR exacerbates the underlying issue.

Actually 2.8 months, so about 84 days.

Edit

For reference, I made a new deck with a completely new preset set to default values, and added a new notes. With 90% DR, the good interval is 2 days. With 70% DR the good interval is 21 days; a lot more reasonable.

If I were to guess, it’s probably because I have empty learning/relearning steps, so I end up pressing again way less than most Anki users.

In fact, I have pressed again exactly 72 times out of the 9,098 reviews for my kanji deck (the one I was referring to earlier). Nine of them are from the time when I was still using (re)learning steps.

One possible explanation is that FSRS tends to underestimate the value of decay (Decay tends to be lower than true value when the retention range is narrow by L-M-Sherlock · Pull Request #1 · open-spaced-repetition/fsrs-trained-on-simulated-data · GitHub). So, if you ask FSRS to make a prediction in the R range which is not present in the actual data, the prediction can be quite wrong.

In other words, if most of your reviews are in the 90% R range, FSRS will give a unreasonably high value of interval at DR=70%.

3 Likes

Does that mean that people who have used a certain DR for a long time can’t change it too much without getting too long/short intervals?

Yes, DR should generally be changed by small amounts, gradually moving towards your target value.

2 Likes

With so many data points in one given region of the parameter space, it can very easily overfit if you try to ask it to make predictions about another part of the parameter space.

That is, if you feed it data for 10k reps at DR=0.9, but never any data for R=0.7, it will not necessarily be accurate at predicting when R=0.7, only in places near/around R=0.9.

I did the math in my head the other day for how to solve this problem, but… the solution is not exactly easy to implement, so I haven’t even bothered with posting the details.

The easiest solution is for the user to slowly lower it over time.

Lower it by 5%, wait until you get some reviews from failed cards. Then repeat until you get to DR=0.7.

Or like, create a new deck with no reviews and start it at DR=0.7, get some reviews in there, and then over time merge it with your old deck.

1 Like

Can you give some details of that solution? Possibly someone would be able to implement that (or a simplified version of that)?

I am interested not only because of its implications in changing the DR but also because a wrong predicted value of decay also makes the graph requested in this post (and included in latest Anki) less realistic.