New Sort Oder: PSG

[sorata]

(results from the new simulation)

Suggestion

Why change to PSG?

---

In the latest simulation, a new sort order PSG_desc performs better than retrievability_desc in seconds_per_remembered_card. But it’s worse than difficulty_asc albeit slightly. Unsurprisingly, maintaining a constant desired retention is still the forte of retrievability_desc but that’s not a very valuable metric.

Now, one of the issues with difficulty_asc was mentioned that the sorting doesn’t change with time as difficulty only changes when grading cards. Even for very old backlogs, thus sorting stays the same as before. On the other hand, the sorting with retrievability_desc is more dynamic but comes with a slightly worse number for seconds_per_remembered_card. PSG_desc combines the strong points of both of them:

• PSG value changes with time.
• It performs really well in seconds_per_remembered_card almost as good as difficulty_desc.

There have been some aspersions cast about seconds_per_remembered_card or the calculation of total_time so I’d also point out that when it comes total_remembered, PSG_desc is performing better than all other sort orders. (Note: total_remembered is amount you remember when the simulation ends i.e. when you’ve finished the twenty thousand is:new cards)

What is PSG?

---

PSG stands for “Potential Stability
Gain” and is calculated something like this:

PSG = (S_recall ÷ S) × R

Or in other words,

PSG = change in S after recall × probability of recall

Purely from intuition, I expect PSG_desc to perform better in edge cases as the formula for PSG includes both difficulty and retrievability. Say, when retrievability_desc creates an order very much reversed of what difficulty_asc does.

Now, it will probably be slow but I wanted to inquire if we implement this instead of what was suggested in Improving sort orders.

Tangent

---

We tried something like PSG = (S_recall × R – S_forget × (1 – R) ) ÷ S but it didn’t work out too well.

Discussion

---

For discussion, see Ordering Request: Reverse Relative Overdueness.

[Keks]

@sorata

We tried something like PSG = (S_recall × R – S_forget × (1 – R) ) ÷ S but it didn’t work out too well.

Why is it not so
PSG = (S_recall × R + S_forget × (1 – R) ) ÷ S

[sorata]

There are two things going on in a backlog:

• R is constantly dropping.
• 1 — R is constantly increasing.

If S_recall is higher, you should do the card sooner assuming S_recall × R is decreasing.

If S_forget is higher, you should do the card later assuming S_forget × (1 – R) is increasing.

I felt this had a negative relation with what should come first.

---

Your suggestion was what Jarrett initially said but it was never tested. Do you expect this to work well?

Also, I am not sure this will work but I still asked Jarrett to try once,

PLSG = PSG(today) – PSG(tomorrow)

If you’re wondering, PLSG is “Potential Loss in Stability Gain”.

[Keks]

I see the point in: S_recall × R + S_forget × (1 – R)
You will get something like the expected stability.
By analogy with Expected value - Wikipedia

(S_recall × R + S_forget × (1 – R) ) ÷ S
The expected increase or average increase…

[sorata]

@L.M.Sherlock Can you also try doing (S_recall × R + S_forget × (1 – R)) ÷ S as the formula for PSG today? (apart from trying out PLSG.)

[L.M.Sherlock]

Fine. Will do it.

Update:

        if review_sorting_order == "PSG_desc":
            card["S(recall)"] = card.apply(
                lambda row: (
                    float(student.next_states(row["states"], row["delta_t"], 3)[0])
                    if row["stability"] != 0
                    else 0
                ),
                axis=1,
            )
            card["S(forget)"] = card.apply(
                lambda row: (
                    float(student.next_states(row["states"], row["delta_t"], 1)[0])
                    if row["stability"] != 0
                    else 0
                ),
                axis=1,
            )
            card["PSG"] = (
                card["S(recall)"] * card["retrievability"]
                + card["S(forget)"] * (1 - card["retrievability"])
            ) / card["stability"].map(lambda x: x if x != 0 else 1)

[sorata]

Okay, so this is clearly worse. Thanks! The formula stays the same.

By the way, will you try PLSG? I’m not hoping much but the data will be interesting.

[L.M.Sherlock]

@sorata

PLSG

PLSG ASC or PLSG DESC?

[sorata]

Ascending.

PLSG = PSG(today) - PSG(tomorrow)

[zZH2z]

Hi, what is the image with table of order list? Is this is some addon? How to interpret this data? Are those statistics how order of the reviews impact learning? Very interesting

[sorata]

@zZH2z

Hi, what is the image with table of order list? Is this is some addon?

It’s not an add-on. It’s a simulator developed by Jarrett Ye, the creator of FSRS. See the discussion I linked in OP. We’ll have too much clutter if we discuss it here again.

[rich70521]

@sorata

I’d also point out that when it comes total_remembered, PSG_desc is performing better than all other sort orders. (Note: total_remembered is amount you remember when the simulation ends i.e. when you’ve finished the twenty thousand is:new cards)

I want to reiterate my aspersions about total_remembered here (which filter down to seconds_per_remembered_card ). I don’t think we should be using these as the objective function.

https://forums.ankiweb.net/t/ordering-request-reverse-relative-overdueness/50051/358?u=rich70521

[rich70521]

@sorata

PSG = (S_recall ÷ S) × R

Or in other words,

PSG = change in S after recall × probability of recall

I think you’re calculating percent change in S, not absolute change in S. That may be intentional, but I think absolute change in S would be more useful for PSG purposes.

PSG = (S_Recall - S) x R

With percent change in S, it’s basically the exact same sort as difficulty_asc because that’s exactly the function of D in the algorithm (the sim results show it’s not the exact same, but pretty damn close).

Taken from https://github.com/open-spaced-repetition/fsrs4anki/wiki/The-Algorithm:

1. The larger the value of D, the smaller the SInc value. This means that the increase in memory stability for difficult material is smaller than for easy material.

With absolute change in S, you’d be maximizing the actual time between reviews. That makes more sense to me.

[sorata]

I recommend this discussion be done at the associated forum thread. I want to keep it clean.

[rich70521]

Was this in response to my first comment, second, or both?

[sorata]

No, I am talking about all discussions. I saw Keks typing so was meant for them too.

[L.M.Sherlock]

            card["S(recall)"] = card.apply(
                lambda row: (
                    float(student.next_states(row["states"], row["delta_t"], 3)[0])
                    if row["stability"] != 0
                    else 0
                ),
                axis=1,
            )
            card["S(forget)"] = card.apply(
                lambda row: (
                    float(student.next_states(row["states"], row["delta_t"], 1)[0])
                    if row["stability"] != 0
                    else 0
                ),
                axis=1,
            )
            card["PSG"] = (
                card["S(recall)"] * card["retrievability"]
                + card["S(forget)"] * (1 - card["retrievability"])
            ) / card["stability"].map(lambda x: x if x != 0 else 1)
            card["S(recall_tomorrow)"] = card.apply(
                lambda row: (
                    float(student.next_states(row["states"], row["delta_t"] + 1, 3)[0])
                    if row["stability"] != 0
                    else 0
                ),
                axis=1,
            )
            card["S(forget_tomorrow)"] = card.apply(
                lambda row: (
                    float(student.next_states(row["states"], row["delta_t"] + 1, 1)[0])
                    if row["stability"] != 0
                    else 0
                ),
                axis=1,
            )
            card["PSG_tomorrow"] = (
                card["S(recall_tomorrow)"] * card["retrievability"]
                + card["S(forget)"] * (1 - card["retrievability"])
            ) / card["stability"].map(lambda x: x if x != 0 else 1)
            card["PLSG"] = card["PSG"] - card["PSG_tomorrow"]

[sorata]

@keks It doesn’t work apparently. It’s worse.

Btw, would you try PLSG_desc Sherlock?

Okay, I don’t get it. What did you try here? Did you try Keks’ suggestion or my suggestion or you tried both?

If you tried both, now we don’t know what worked and what didn’t.

[L.M.Sherlock]

I thought I replied your ask.

@sorata

Btw, would you try PLSG_desc Sherlock?

OK

[sorata]

Use this formula though (for calculating PSG):

PSG = S_recall/S × R

As the other one with S_forget didn’t work.