In Anki desktop 25.02, optimizer chooses parameters with greater (worse) log loss and RMSE

Anki FSRS
1

( I’m not sure if I should post here in Anki > FSRS, in Anki > Help, or inside an existing post. Am I in the right place? )

Hello. I’m considering moving from Anki 24.11 to Anki 25.02 because it’s about time to optimize my parameters again.

However, when I ran “check database” in Anki 24.11, uninstalled Anki 24.11, installed Anki 25.02, ran “check database” again, and ran the optimizer on individual decks:

  1. The optimizer automatically ran a second time on some decks. It went to 100% and then started again.
  2. On those decks that it automatically ran a second time on, the log loss and RMSE appeared to get worse (increased in value). On the decks that it ran only once on, the log loss and RMSE either stayed the same, because parameters were already optimal, or got better (decreased in value).
  3. On those decks that had an increase in log loss and RMSE values, the last two parameters were set to 0.0000 (18th and 19th values, w17 and w18 from FSRS 5’s formula).
  4. Here are the results. I omitted some for not being relevant. In each section, Anki 24.11 and Anki 25.02 both have the same “before” parameters. The “before” log loss and RMSE only appear different because of the different evaluation results between versions:
    2025-03-09

--- kanji a (57,530 reviews)

Anki 24.11
optimizer runs ONCE
before: 0.3078 4.36% (ALREADY OPTIMAL)
before: 0.3590, 0.5282, 0.9424, 1.3864, 7.7088, 0.4937, 2.3647, 0.0230, 1.1735, 0.1263, 0.5781, 1.8452, 0.1751, 0.3855, 0.9342, 0.0079, 3.9109, 0.5301, 0.5805

Anki 25.02
optimizer runs TWICE (automatic second run!)
before: 0.2784 3.74%
 after: 0.2801 4.20% (WORSE, LAST TWO PARAMETERS SET TO 0.0000)
before: 0.3590, 0.5282, 0.9424, 1.3864, 7.7088, 0.4937, 2.3647, 0.0230, 1.1735, 0.1263, 0.5781, 1.8452, 0.1751, 0.3855, 0.9342, 0.0079, 3.9109, 0.5301, 0.5805
 after: 0.5533, 0.7786, 0.9872, 1.2084, 7.4332, 0.3943, 2.1594, 0.0274, 1.1543, 0.1721, 0.6164, 1.9848, 0.0938, 0.5042, 1.4209, 0.0000, 3.4684, 0.0000, 0.0000


--- kanji b (43,646 reviews)

Anki 24.11
optimizer runs ONCE
before: 0.2833 3.91%
 after: 0.2830 3.90% (BETTER)
before: 0.5893, 1.2798, 4.0960, 24.1313, 7.3079, 0.1973, 2.3113, 0.0194, 1.3269, 0.0965, 0.7964, 1.7688, 0.2125, 0.2988, 1.6172, 0.0118, 4.8276, 0.5757, 0.8310
 after: 0.5598, 1.2615, 4.2674, 24.2626, 7.3330, 0.4326, 2.4665, 0.0226, 1.3028, 0.0797, 0.7430, 1.8164, 0.2238, 0.3036, 1.4658, 0.0073, 4.3935, 0.5132, 0.8183

Anki 25.02
optimizer runs TWICE (automatic second run!)
before: 0.2578 3.62%
 after: 0.2601 4.19% (WORSE, LAST TWO PARAMETERS SET TO 0.0000)
before: 0.5893, 1.2798, 4.0960, 24.1313, 7.3079, 0.1973, 2.3113, 0.0194, 1.3269, 0.0965, 0.7964, 1.7688, 0.2125, 0.2988, 1.6172, 0.0118, 4.8276, 0.5757, 0.8310
 after: 0.7790, 1.9671, 3.7444, 23.4165, 7.1470, 0.4380, 2.3402, 0.0174, 1.4060, 0.1139, 0.8506, 1.9492, 0.1259, 0.4012, 1.5203, 0.0170, 3.5602, 0.0000, 0.0000


--- vocab a (74,848 reviews)

Anki 24.11
optimizer runs ONCE
before: 0.3568 4.46%
 after: 0.3556 4.19% (BETTER)
before: 0.6185, 0.8989, 1.5747, 44.2150, 7.0304, 0.4272, 2.2767, 0.0129, 1.5717, 0.2753, 0.9731, 2.0137, 0.0179, 0.3050, 0.7159, 0.0266, 4.8286, 0.3252, 0.1962
 after: 0.6957, 0.9843, 1.6565, 48.9230, 6.9911, 0.5258, 2.7288, 0.0071, 1.6408, 0.2830, 0.9951, 2.1167, 0.0161, 0.3620, 0.1724, 0.0254, 5.1615, 0.2858, 0.1151

Anki 25.02
optimizer runs ONCE
before: 0.3173 3.82%
 after: 0.3163 3.56% (BETTER)
before: 0.6185, 0.8989, 1.5747, 44.2150, 7.0304, 0.4272, 2.2767, 0.0129, 1.5717, 0.2753, 0.9731, 2.0137, 0.0179, 0.3050, 0.7159, 0.0266, 4.8286, 0.3252, 0.1962
 after: 0.6191, 1.0413, 1.4945, 48.7713, 7.0307, 0.6117, 2.3952, 0.0053, 1.6310, 0.3060, 0.9964, 1.9954, 0.0568, 0.3488, 0.6312, 0.0218, 3.3989, 0.3629, 0.2057


--- vocab b (59,458 reviews)

Anki 24.11
optimizer runs ONCE
before: 0.3303 4.51% (ALREADY OPTIMAL)
before: 0.4403, 6.8131, 16.4443, 100.0000, 7.2472, 0.4877, 1.5236, 0.0146, 1.2928, 0.0000, 0.7570, 1.6987, 0.2186, 0.1348, 0.5454, 0.0214, 5.1275, 0.9048, 0.7033

Anki 25.02
optimizer runs TWICE (automatic second run!)
before: 0.2952 4.59%
 after: 0.3031 4.92% (WORSE, LAST TWO PARAMETERS SET TO 0.0000)
before: 0.4403, 6.8131, 16.4443, 100.0000, 7.2472, 0.4877, 1.5236, 0.0146, 1.2928, 0.0000, 0.7570, 1.6987, 0.2186, 0.1348, 0.5454, 0.0214, 5.1275, 0.9048, 0.7033
 after: 0.7892, 7.8490, 26.6964, 100.0000, 7.1073, 0.3950, 1.9624, 0.0187, 1.3732, 0.0000, 0.8216, 1.9310, 0.0986, 0.3034, 0.5492, 0.0006, 4.0268, 0.0000, 0.0000


--- vocab c (49,279 reviews)

Anki 24.11
optimizer runs ONCE
before: 0.3189 5.01%
 after: 0.3171 4.78% (BETTER)
before: 0.3116, 1.0480, 6.4631, 100.0000, 6.8019, 0.2849, 2.1378, 0.0075, 1.7005, 0.0004, 1.1392, 1.9142, 0.0814, 0.1911, 1.5837, 0.0000, 4.4897, 0.5012, 0.4408
 after: 0.3550, 1.2864, 8.8747, 99.9999, 6.8671, 0.5838, 2.2402, 0.0031, 1.6890, 0.0369, 1.1443, 1.9644, 0.0296, 0.1991, 1.4039, 0.0000, 4.6054, 0.4230, 0.3997

Anki 25.02
optimizer runs ONCE
before: 0.2922 4.94% (ALREADY OPTIMAL)
before: 0.3116, 1.0480, 6.4631, 100.0000, 6.8019, 0.2849, 2.1378, 0.0075, 1.7005, 0.0004, 1.1392, 1.9142, 0.0814, 0.1911, 1.5837, 0.0000, 4.4897, 0.5012, 0.4408


--- vocab d (43,965 reviews)

Anki 24.11
optimizer runs ONCE
before: 0.3153 6.10% (ALREADY OPTIMAL)
before: 0.6954, 2.1809, 2.1809, 98.5326, 7.0345, 0.9863, 2.5210, 0.0022, 1.8257, 0.0860, 1.2461, 1.9353, 0.0040, 0.2541, 1.8685, 0.0007, 3.7187, 0.3759, 0.1265

Anki 25.02
optimizer runs ONCE
before: 0.3021 5.98%
 after: 0.3018 5.97% (BETTER)
before: 0.6954, 2.1809, 2.1809, 98.5326, 7.0345, 0.9863, 2.5210, 0.0022, 1.8257, 0.0860, 1.2461, 1.9353, 0.0040, 0.2541, 1.8685, 0.0007, 3.7187, 0.3759, 0.1265
 after: 0.6228, 2.4299, 2.4299, 97.3825, 6.9264, 0.8577, 2.4740, 0.0010, 1.8663, 0.0795, 1.2817, 2.0056, 0.0618, 0.2537, 1.9052, 0.0089, 3.1609, 0.4408, 0.2293


--- vocab e (5,308 reviews)

Anki 24.11
optimizer runs ONCE
before: 0.2289 7.04%
 after: 0.2298 6.97% (BETTER)
before: 0.6146, 1.9837, 11.5027, 57.4142, 7.1394, 0.6637, 1.6189, 0.0010, 1.5890, 0.0578, 1.0589, 1.8717, 0.1991, 0.2597, 2.1564, 0.0000, 3.2597, 0.5024, 0.6668
 after: 0.5414, 2.4369, 23.2735, 58.3397, 7.1479, 0.5046, 1.6477, 0.0010, 1.5312, 0.1054, 1.0089, 1.8551, 0.1978, 0.2253, 2.1788, 0.0185, 3.3407, 0.4260, 0.5532

Anki 25.02
optimizer runs TWICE (automatic second run!)
before: 0.2176 6.74%
 after: 0.2202 6.77% (WORSE, LAST TWO PARAMETERS SET TO 0.0000)
before: 0.6146, 1.9837, 11.5027, 57.4142, 7.1394, 0.6637, 1.6189, 0.0010, 1.5890, 0.0578, 1.0589, 1.8717, 0.1991, 0.2597, 2.1564, 0.0000, 3.2597, 0.5024, 0.6668
 after: 0.3576, 4.1873, 23.1463, 58.0611, 7.1141, 0.5395, 1.6885, 0.0010, 1.5612, 0.0618, 1.0363, 1.9709, 0.0926, 0.3613, 2.3176, 0.0673, 3.0746, 0.0000, 0.0000

My questions:

  1. Is there a reason that the optimizer would run to 100% more than once?
  2. Is there a reason to expect an increase in log loss and RMSE when optimizing? For example, is this something caused by the recency weighting mentioned by @Expertium ?
  3. Is there a reason to expect the optimizer to set the last two parameters to 0.0000 despite all 19 parameters already being used?
  4. Are these “worse” results in Anki 25.02 actually worse, or should I trust the optimizer and save?
2

Sometimes this happens:

  1. Your last interval was, say, 10 days
  2. You pressed Again
  3. You went through your re-learning steps
  4. Your next interval is now 15 days

To prevent this from happening, the optimizer runs a check, and if it detects that this can happen, the last two parameters will be “frozen” and the rest will be re-optimized.
Jarrett is working on a better version that doesn’t set the last two parameters to zero, just to some small values.

2 Likes
3

Understood. Thank you for the explanation!

Also, thank you for your other comment on reddit which saved me the trouble of uncertainty / asking:

Without rescheduling on change, <…>, new parameters/desired retention will affect the length of the NEXT interval.

Some things happened in the past when I (a) optimized too frequently and (b) still didn’t know to watch both the card info and previous card info windows during reviews after a change, so I had some trouble believing any testing I’d do.

Extra note, especially for if someone gets here by a search:
I was going to ask if this is a permanent or temporary symptom, but this is happening on decks with around 50,000 reviews. Each deck has only one card type from a parent note type. On my “kanji a” deck from above, when I set a date for the option “Ignore cards reviewed before” to bring the review count to around 1,000 before optimization, the last two parameters w17 and w18 still get set to 0.0000. So, at least in my case, it’s very likely a symptom of my behavior / ability that’ll follow card types no matter how much time passes or how many new cards get introduced.