Agreed this seems counterintuitive to me.
“Counterintuitive” is putting it very nicely. I think there’s a flaw somewhere in the system.
Simulation Results (DR=0.01)
Metric Value Reviews per day (average) 0.0 Minutes per day (average) 9.8 Memorized cards (average, all days) 3207 Memorized/hours spent (average, all days) 107.3
Reviews per day are 0.0; there were 9999 cards learned, but somehow 3207 of those are still remembered at the end of a year? So basically there were never any reviews, but somehow the user still recalls 1/3 of all of the information at the end of the year?
Something is wrong with the forgetting curve if a single review gives R=0.30 a year later. (Obv. how many cards are studied on which day will have some impact, some would have been studied later near the end of the year, but still.) I have an extremely hard time believing that someone is going to remember ~1/3 of the material that they saw a single time one year prior.
Maybe this just shows us most users know some of their new cards already and it’s better to speed run all new cards to find those known already, marking them as good/easy to “artificially” boost retention
This seems like a very likely possibility given the results. (That or there’s a flaw in the simulator itself.)
You could theoretically learn X new cards, go over a ~15 year interval, and then see what percent of X cards are learned at the end of that, and then subtract that from the gain in ΣR over the course of the simulation, as that R was not truly 0 at the start
The reason why we look at average knowledge (number of memorized cards, aka sum of probabilities of recall of each card) across all days rather than only looking at knowledge at the end of the simulation is because the latter can be cheated. Imagine the following algorithm (duration of the simulation=5 years): first, it assigns an interval equal to 4 years and 11 months to every card. Then, during the last month, it assigns an interval equal to a few days to every card. The number of cards memorized at the end will be very high despite the fact that on most days knowledge was extremely low.
This seems to be a… shortsighted approach to solving the issue of the ability to game the system with strategies that are designed to maximize knowledge on a given day, and a possible source of bias for the current issue at hand, as it’s overly valuing information remembered on a given day which is then promptly forgotten afterwards.
Some more appropriate solutions to the gameability problem are to A) calculate ΣR at the end of the simulation, but have the end of the simulation be a randomly chosen point in time (or an average of randomly chosen points in time), or B) calculate ∫(t_end,inf)ΣR(t’)dt at the end and use that as well (to more strongly weight long-lived knowledge at the end of the simulation), and to ensure that the strategies which maximize ΣR_end are the same strategies that maximize ∫(t_end,inf)ΣR(t’)dt. (You could cheat and make it easier to calculate by just using Σ(SR) which is probably easier to do and approximately proportional to ∫(t_end,inf)ΣR(t’)dt, but I read somewhere that they can integrate the forgetting curve without too much issue, so that would be preferable if easily doable.)
Edit: Now that I think about it a bit more, Calculating the average ΣR over all days probably is fine, as that is fundamentally the same as calculating the average of the simulation ending on a random day therein (i.e. the second half of what I said for A above). So I no longer think that that is a relevant issue or source of possible bias.
I do, however, not see how 3000 is the average number of cards remembered throughout the length of the simulation. That… doesn’t make sense.