Ordering Request: Reverse Relative Overdueness

Alright, here are the results

Default FSRS-5 parameters

desired_retention = 0.9
deck_size = 20000
learn_limit_perday = 20
review_limit_perday = 200

learn_span = int(deck_size / learn_limit_perday)



As you can see, without a backlog there isn’t much of a difference

2 Likes

@sorata Here is with DR = 0.9, and 20/200 review limits.

The other graphs didn’t show much difference among them at all except this one.

@Expertium Can you explain what True Retention per Day is telling us?

Desired retention is what you want. It’s your way of telling the algorithm “I want to successfully recall x% of cards when they are due” (that’s an important nuance).

True retention (download the Helper add-on and Shift + Left Mouse Click on Stats) is measured from your review history. Ideally, it should be close to the desired retention. If it deviates from desired retention a lot, there isn’t much you can do about it.

Basically, desired retention is what you want, and true retention is what you get. The closer they are, the better.

https://www.reddit.com/r/Anki/comments/1anfmcw/you_dont_understand_retention_in_fsrs/

Ok, so PRL does seem to maintain better proximity to the DR on the True Retention per Day, but it’s not much better on the other metrics.

What Keks said. However, in Jarrett’s code it’s a bit different. Actually, very different. It’s the average retrievability of cards reviewed today. It’s not based on binary 0-1 pass-fail outcomes, like the True Retention in the Helper add-on and now in the Anki beta. It’s based on retrievability.

Well, in this simulation it doesn’t matter, since in this simulation retrievability cannot possibly be inaccurate, but in real life it matters, since in real life retrievability can be inaccurate and the user can recall 90% of cards even when average retrievability was 70% today.

In fact, the current approach is better for the simulation, because if the number reviews is low, average retrievability can be quite different from average of binary outcomes.
Example: 80% average retrievability. Both cards have been recalled, so both outcomes are equal to 1. The average is (1+1)/2=1.

1 Like

This is what was confusing me. I read the code and thought it was dividing the entire sum, for the whole deck, retrievability by the reviews for today. Makes much more sense now. Thanks.

Running this now with my historical study habits and I probably wouldn’t get through 20k cards in a lifetime :joy:

With difficulty_asc it’d take me 36 years to get through them

Screen Shot 2024-10-01 at 9.56.01 AM

I can run someone else’s if you want. Just need your percent of days studied and your average reviews for days studied.

I just got a really cool idea - iterating over different review card limit/new card limit ratios at different desired retentions to find out the ratio that doesn’t cause a backlog at every level of DR. Though, of course, this will be based on default parameters, but still. It will be better than the current rule of thumb “You will have 10x more reviews than the number of new cards”.

Will take forever, though.

EDIT: wait, I’m dumb. I don’t need to iterate over ratios, I just need to uncap review limit and see how many reviews the simulated user gets, on average

1 Like

The results after inputting my study habits in for reviews/day:

As annoying as it is because the formula isn’t really intuitive, I might just have to admit difficulty_asc is the best and start using that. Minimizing reviews is what I want to do, and it’s been winning on that front pretty much every time.

To make it accurate you would need

  1. Your own FSRS-5 parameters, which you can’t get right now because the optimizer in Anki 24.10 beta isn’t working properly
  2. This stuff
    image
    Which you can’t access, Anki calculates these in the background when you run CMRR

Yeah, I’m ok with not getting it that exact. Not sure it would make a huge difference with which perform better. Seems like no matter what variables I change, the rank of sorts stays pretty consistent.

The biggest takeaway for me is that random is actually performing really well without a backlog. It starts performing really bad only when you have a backlog. Assuming this simulation captures at least some of reality, I see no reason why random cannot become the default sort order. It’s not as good with maintaining a constant retention though, that I see.

Or if not, what about difficulty_desc? For some reason it seems we’re instead suggesting dae to make least overdue first the default. Any reason why?

Another thing that I dont get is why difficulty_asc suddenly performs really bad when we stop learning any new cards in the previous sim. I would like to see what a simulation looks like when youre not learning any new material and just maintaining previously learnt stuff.

Done

1 Like

Retrievability descending performs well both with and without a backlog, random performs well only without a backlog, and even then R desc. performs better. Why would random be the default?

For some reason it seems we’re instead suggesting dae to make least overdue first the default. Any reason why?

Retrievability is less arbitrary and more interpretable, and will likely work better in edge cases.

1 Like

I tried to switch it up and put a 500 day gap at the 2500 day point to see if the consistency was affecting anything. It’s not rare for me to take a year off. difficulty_asc is still basically the best :roll_eyes:

I just don’t want to base my sorting on something that isn’t dynamically changing with each passing day. Difficulty only changes when you answer a card, and it’s only affected by what answer you give it, it doesn’t factor Retrievability at all.

Even though most or your cards have very high D values, possibly with a lot of similar numbers? Its surprising how is that happening.

1 Like

The simulation isn’t using my actual deck, so I have no idea what the D distribution is during the sim. Maybe it’s reflecting my deck somewhat because I changed the daily reviews, but that’s not guaranteed. But yeah, I am surprised.