From what I understand, parameters should be optimized approximately every month or 2n reviews, optimizing with a low # of reviews is more accurate than using the default parameters, and optimizing with a high # of reviews is more accurate than with a lower #.
But I’m confused on a few things: #1 How does the optimizer decide whether to fully optimize, pretrain, or keep the default parameters? #2 If the second thing I understand is the case, how much more accurate is it? #3 How many reviews are necessary to reach benchmark RMSE levels?
I’m guessing all of this varies from person to person but I’m trying to see if there’s any data or averages that might clarify this. I’m also trying to understand how the optimizer makes that decision because I haven’t been able to find any info on this in the docs. I also think in the future, it would help if this was explained on the docs if it isn’t already and I’m misinterpreting everything.
This isn’t laid out in the regular docs, but that’s mostly because the changes have been rapid, and in reality, aside from just being curious about it – you don’t need to worry about this. You’ll move very quickly through those stages, and you shouldn’t worry about which one you’re in.
The full-optimize threshold started at 1000 reviews, but then that came down to 400 reviews, and pre-train was introduced, which I think starts at 100 reviews. I should just let Expertium rattle off the correct numbers.
No individual user is guaranteed they will reach some certain ideal RMSE. It’s a measure of how well FSRS is doing matching your memory curve.
Thanks for your help. Only thing is, which is my fault for not being specific enough, is that in my 2nd question I was asking if there was any data on how fsrs becomes more accurate as the # of reviews increase.
If pretraining starts at 100 reviews and full optimization starts at 400 that helps. But from what I understand from the table, the full fsrs optimization could include people who optimized at 1000 reviews, 5000, 10000, etc. If there was a way to view these specifics, that would help me even more. I’m also not just looking at how likely it is to perform better, but how much it’s actually better on average, but this is already on github for the data in the table above at least.
This is all for a science fair project specifically for improving fsrs in decks with low card or review amounts. If I knew how review # impacts accuracy and when fsrs exactly starts to optimize, it would let me know which decks need improving or not, and if so to what degree.
Oooh, a science fair! That’s an excellent question for @Expertium and @L.M.Sherlock – they will know best whether that sort of differential is already available, or if it can be teased out of the data!
This is based on data from 20k collections of Anki users. I had an image with a trendline somewhere, but can’t find it atm
EDIT: I made one for FSRS-5, on a new dataset (10k collections instead of 20k), with a trendline, and I cut the X axis off somewhat because of outliers with more than a million reviews:
RMSE is strongly correlated with the number of reviews, logloss is strongly correlated with retention. Ideally, we would like to have a metric that isn’t correlated with either, but alas.
Regarding pretrain, IIRC it starts at 8 reviews and full optimization starts at 64 reviews.
That’s an excellent question for 
