Can we compare RMSE in FSRS 4.5 and FSRS 5?

Anki FSRS
1

It makes sense to compare two RMSE values only when they are calculated on exactly the same data.

FSRS 4.5 discards the same-day reviews but FSRS 5 uses them. So, does it make sense to compare the RMSE values obtained by clicking Evaluate on exactly the same collection in Anki 24.10 and Anki 24.06.3?

The scenario in which comparing them would make sense (and I advice implementing Evaluate in this way if it doesn’t already work like this):

When the user clicks Evaluate, FSRS should use ALL the reviews (including the same-day ones) to calculate the DSR at each review. Then, it should compare the predicted R and actual R ONLY for the first review of a day when calculating the RMSE.

2

Do you mean Evaluate or Optimise?

I don’t get why should RMSE be calculated that way. You can use the latest version to compare parameters generated in 24.06.03 with parameters generated in 24.10.

3

Evaluate

You are right. But, what if I want to evaluate the same parameters in 24.06.3 and 24.10? The formulas have changed slightly. So, the same parameters will produce slightly different DSR values in FSRS 4.5 and FSRS 5.

Apart from facilitating proper comparison with older Anki versions, calculating RMSE this way is important because the predicted R for same-day reviews is always 100% and comparing the actual R with a constant doesn’t really make sense.

4

Come next version (or later), we’ll have to change it again though. And it creates extra work for Evaluate. I’m still fine if you or anyone else wants to implement this in a way that DSR is only calculated the first time around (when you still have old params).

It should change with short term version maybe? If not, this does make sense.

(btw this post should be in suggestions category imo)

5

This is not in the suggestions category because I am 90% sure that Evaluate already works in the way I am describing. I just wanted to confirm that.

6

The RMSE in FSRS 5 is higher than in FSRS 4.5, can these indicators be directly compared. Or has something changed in the RMSE calculation?

7

As of beta 2, FSRS has a “bug” in how it uses FSRS 4.5 parameters. It can be one reason why the RMSE is higher. This issue should be fixed in the next beta.

However, I am not 100% sure if the RMSE is directly comparable, which is why I have made this post.

1 Like
8

That’s how it works. Metrics are calculated only based on the first review.

1 Like
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.