@A_Blokee I don’t think there is a way to fix the issue with a negative amount of cards memorized?
See this.
Negative values aren’t a problem if my suggestions in Replace CMRR with workload-vs-DR graph (+more) - #46 by vaibhav are implemented. Knowledge gain can be negative as that implies a loss of total knowledge.
You mean using (t_end - t_start) / (R_end - R_start) for the Time/Memorized (Gain) Ratio, but using (t_end - t_start) / R_end for Memorized? I’m not sure if that’s a good idea, it creates more confusion. We could add a little tip to explain it, but that’s more visual clutter, so idk.
All of that seems perfectly reasonable.
Just to be clear, I think fixed 90% DR will always be the default, and SSP-MMC will be something the user has to consciously choose to enable. It won’t be called SSP-MMC, of course, just “Adaptive”. Anyway, I think the default will be 90% DR.
Also just to point out, memory loss isn’t some rare edge-case. In the case of a user who has gone through all new cards in a deck and is no longer adding new cards, he should be losing ΣR unless he increases his DR. It’s a rather common situation.
(R_end - R_start) / (t_end - t_start) for the Ratio (knowledge gain per time). I prefer knowledge gain per review, though.
Just R_end for memorized. No factor of time or reviews here.
@A_Blokee graphs with the change that vaibhav proposed above would be appreciated, just so we can see whether those graphs make sense
Either way, this is very exciting stuff. Is this something that we can expect as an experimental Anki feature anytime soon?
Not any time soon. I’m working on FSRS-7, that will have to be finished first. Then there are several things that must be done to make the scheduling benchmark more realistic. If I’m being optimistic, the scheduling benchmark will be fully done by December 2025 - January 2026.
Could you please share a link to those posts?
This but per review?
Yes, but also
- The Memorized graph should use
sum(R)_end(and notsum(R)_end - sum(R)_start) - add a dashed horizontal line to the Memorized graph representing the
sum(R)_start. - In the Ratio graph, use total reviews over simulation span in the denominator instead of average reviews. Also, multiply the result by 100 so that we get “Knowledge gain per 100 reviews”
Here is one I found (thanks to OP liking my reply here, couldn’t have found otherwise):
The mathematics behind calculating the statistical error between the experimentally measured values and their true averages given an infinite data set, and then propagating that error through the curve-fitting process, (let alone the changes in the user’s grading habits between the time the data was taken and the time when you calculate R(t) based off his button press), to the point that you can put actual statistically accurate error bars on the actual predicted value of R at an arbitrary point in time t after the last review, is extremely complicated and beyond the scope of a single forum thread. It would probably take an entire PhD thesis.
Even if the model were 100% accurate (it isn’t), and even if you fit the parameters perfectly (probably very close to theoretical maximum), there is still going to be statistical error due to the fact that the number of times a user hits a certain button in a given state is going to follow a Poisson distribution, and thus it is impossible to calculate R_true for past data with 0% error. There’s additional error in the fact that the user’s habits are virtually guaranteed to change over time between what he did for the reviews and when the parameters are fitted and when the prediction for R(t) is done for when the parameters are used. There’s always going to be these statistical and error fluctuations. It’s inevitable.
However, because the user sets some DR, and all of the parameters are fit on data with past calculated versions of R(t)≈DR, the forgetting curve is going to be, relatively speaking, very accurate when predicting R(t)=DR. However, because all of the calibration data (i.e. the curve-fitting data) is extremely biased towards certain R(t) values (i.e. what the user has set to DR), this means that it will not be nearly as accurate for calculating R(t) for values when R(t) is very different from DR.
Because of this, the graph you have posted in the above link is a best-case scenario in which the FSRS parameters were fit with data appropriately sitting around that bar, for each bar. This does not, however mean, that if you have a bunch of data and parameters fit for R(t)=95% that you are capable of predicting when R(t)=70% with much accuracy at all.
If you would like to see this phenomenon in action, create a new deck, set DR to 98%, and memorize ~100 random facts in it. Then after doing that for a week or 2, change the DR to 10%. You’ll probably get year+ long intervals. However, when they do come up, I find it extremely unlikely that you will actually get anywhere close to 10% of information retrievable after said year. I predict that it will probably be closer to either 0% or 20%.
When the DR is low (e.g. 70%), the amount of time required to gather additional data in-around R(t)=DRcan take months or years before the graph can become appropriately calibrated, and this effect may become profound. (It can be alleviated through slowly decreasing DR over time, getting more calibration data along the way down, frequently fitting the FSRS parameters and frequently rescheduling cards based on the latest calibrations.)
In the end, the amount of the difference is…. probably negligible and will not negatively affect the user, as long as they don’t make sudden drastic changes to their DR. Especially because FSRS tends to overestimate in the lower range, we are kind of saved by that, so users aren’t going to be screwed out of forgetting all their knowledge that they spent all that time and effort memorizing. However, as a data scientist, I have extreme reservations about relying on a system that happens to always be incorrect on the safe side for unknown reasons, as opposed to using the correct formulae, or having something which is mathematically proven to be on the safe side. It is, however, probably fine and not where the team needs to focus their efforts at this time.
I gave it a bit more thought and came up with a quick and dirty process for reliably doing this if anybody were so inclined.
I’ll just put it in pseudocode because that’s easier to explain than in words:
dataset = [(all the data for calculating FSRS normally)]
fsrs_params_list = []
for i in range(20): # 20 should be good enough. Theoretically infty is necessary, but 20 works.
this_dataset = [datum for datum in dataset if random.random() < 0.5]
this_fsrs_params = fit_fsrs_params(this_dataset)
fsrs_params_list.append(this_fsrs_params)
best_fsrs_params = fit_fsrs_parms(dataset)
def R_with_error(t):
R_mu = R(t, *best_fsrs_params)
sample_R = (R(t *fsrs_params) for fsrs_params in fsrs_params_list)
R_sigma = np.stdev(sample_R)
return R_mu, R_sigma
The reasons why this works “well enough” are long and complicated, and why 0.5 is the correct value there (unless I made a mistake, which I probably did) is complicated, but if you were to do this you would be able to put error bars on just how accurate the given calculation of R(t) is for a given dataset from which the FSRS parameters are fitted. I’m not sure there is any particular need for this at this moment in time.
Also, it only applies to propagating statistical error, not systemic error, which also exists and is probably larger, anyway.
But you would be able to know just how accurate it is to calculate t s.t. R(t)=0.10when all of your draining data is from t s.t. R(t)=0.90, in so far as the model itself is accurate.
The transformation for calculating the error associated with t from the calculation of the error associated with R(t) is left as an exercise for the reader.
My personal experience of dropping a deck from 90 to 70% DR has felt pretty smooth. The deck is around 8 years old and I continue adding new cards to it. My young card retention pretty quickly stabilised close to 70% after a month or two with periodic refitting FSRS parameters (and during that time it wasn’t miles off 70% either). Anecdotally my mature card retention feels fine too (although I avoided rescheduling when I changed DR so the mature stats are warped by all the previously learnt cards with 90% DR intervals).
A random thought would for anki to occasionally intentionally schedule cards based on a different DR, just to give FSRS a more diverse dataset. This would be harmful for users who don’t want to change DR much though, and SSP-MMC is probably a better way to give FSRS this diverse type of dataset anyway.
Would this cause problems when people make multiple plots? It’s an edge case to be sure but since decay is a parameter now, this value could be different through different simulations. It would probably be difficult to implement regardless.
I don’t want to make per review the only option. I think the impact of failed reviews taking longer is a key part of what the graph is trying to display.
I don’t think so. The value will be the same for all simulations unless the FSRS parameters are also modified. Changing the FSRS parameters requires closing the simulator page. So, it should not be a problem.
Though I am not sure about the exact implementation steps, I don’t think it should be difficult to implement. The value is simply equal to the sum of current R values of all the cards.
I thought failures were causing longer study times mainly by increasing the review count (relearning steps + subsequent intervals are shorter) rather than by increasing time per review.
About a month ago I changed my DR to ~70% for all of my decks and changed the learning and relearning steps to ““. I’ve been refitting the FSRS parameter just about every day and frequently rescheduling cards based upon the latest fitted parameters. For most of them, the transition was relatively smooth with no issues, and such frequent FSRS fitting and rescheduling was likely unnecessary.
However, for 2 of the decks, immediately after the change, it gave an initial fail interval of about 18 days. It has since re-calibrated itself back down to about 1 day for an initial fail, which is roughly the correct value.
I have the feeling the issue is more with FSRS overfitting the curve in this region, due to there being, effectively, no data for this type of review until I made the changes, but lots of data on every other type of review. (My previous post above about calculating the statistical error associated with the fitted parameters could theoretically be employed to fix this sort of issue.)
I’m kind of torn between recommending 70% DR (or possibly even lower…. would love to be able to simulate and/or set even lower values…) to everyone and worrying about new users having similar calibration issues that I had and doing 18 days worth of new cards with improperly calibrated FSRS.