Display “You already have optimal parameters” if the previous parameters are better
Do not display any message if the previous parameters are better
0voters
The disadvantage of the latter is that the user will be confused if he presses “Optimize” and nothing changes. The disadvantage of the former is that it may sound too much like an error message, depending on the wording. And we don’t want it to sound like an error message because it isn’t.
I think the purpose of pressing Optimize button is to get optimal parameters, I don’t care about the details of parameters as an average user. And I don’t think getting nothing after pressing Optimize button is confusing. So I vote the second choice.
I’d always lean towards making it clear and explicit what is happening. I voted for the first message to indicate that I support any kind of message, but I feel it isn’t entirely clear in its current phrasing. Maybe something like “Your parameters were not changed, as the old parameters are already optimal”. Could add an option to use new params anyway with a “Use new parameters anyway” button combined with a simple “OK” button.
Tangent: I’m unclear on how parameters trained on a specific review history can end up worse in contrast to parameters trained on a broader dataset when tested on the specific dataset. Shouldn’t the specific optimizing run always be at least as good as the general parameters (assuming it reaches some estimate of the global minimum)?
I guess what I am asking is what makes the algorithm not optimize to the same parameters that are clearly more optimal? Is it getting stuck in a local minimum? Without knowing much about how it is currently implemented, might be worthwhile looking into some stochastic optimization.
There is no reason to do that. The whole point of this change is to prevent parameters from getting worse.
I’m unclear on how parameters trained on a specific review history can end up worse in contrast to parameters trained on a broader dataset when tested on the specific dataset.
Ask LMSherlock, I don’t really get it either. Also, it’s not quite what you described. Old parameters were trained on n reviews, new parameters were trained on n+m reviews. Theoretically, the new parameters should provide a better fit for the dataset that consists of n+m reviews, and the old parameters should provide a better fit for the dataset that consists of n reviews, but that’s not always the case with FSRS.
There is no reason to do that. The whole point of this change is to prevent parameters from getting worse.
I agree assuming the optimization process is perfect. But if there are artifacts in the optimization step that cause misleading results the user should be able to override the decision, because RMSE is then no longer the sole indicator of “badness”.
Here is an example:
When I first reached 1000 reviews and optimized my parameters it resulted in worse parameters (according to RMSE) than the default parameters. In this case there is, IMO, no reason to believe that those parameters were actually worse when applied to my review history and predicting my memory state, especially since the reverse was the case once a couple hundred additional reviews were in the review history. Add a couple hundred more, and it may be the reverse yet again. The fluctuation here was probably in the ~0.2 percentage points so I’d wager there is a very low signal to noise ratio here, and that keeping the new parameters is still better because they are trained on more of my review history.
Most users don’t understand the algorithm very well, and the whole point of FSRS is to do as much math behind the scenes as possible to free the user’s mind from worrying about tweaking (granted, in practice, it’s bloated with stuff like “SM-2 retention” and whatnot). So making the user decide which parameters to keep would not only confuse them but would also go against the spirit of FSRS.
I agree with displaying something in this scenario (otherwise I’d assume the optimization isn’t actually executed and that would be a bug). However, I don’t think it’s correct to say that the parameters are “already optimal”. It simply means the optimizer cannot find a parameter set that reduces log loss / rmse. It doesn’t indicate either of the two things below:
no parameter sets could reduce log loss / rmse further;
no parameter sets could “fit the users’ need” better (depending on what the user want, not necessarily minimizing log loss / rmse).
I think a better message would be something like “Anki cannot find a better parameter set than the current one.”. Which clearly says why parameters are not changed, and avoids the implications above.
The message “You already have optimal parameters” suggests that no further improvement is possible throughout the entire learning lifecycle, whereas in reality, as more reviews are accumulated, the parameters calculated in the future might actually be more optimal. Perhaps a better statement might be “Parameters for the current review history are already optimal”.
There is an old proverb that says, “The first one to plead his cause seems right, until his neighbor comes and examines him.” Whatever the merits of this suggestion may be, the way this poll is presented strikes me as one-sided and sets up a bias before the reader gets a chance to vote.
Actually, I see the original post has been edited since I read it earlier today when I voted. I appreciate that it attempts to be little more neutral in its current form.
EDIT: I know that 10 isn’t a large sample size by any means, but I don’t think we will get significantly more responses than this, unless we somehow bring the attention of the entire forum to this problem.
EDIT 2: I’ll make a survey instead.
will doing it on reddit be a problem? we can get a much larger number of people. should add another option in the poll that says “no idea what you’re talking about”
Honestly I do not understand how people can be against your suggestion. This sounds like a very good idea. I know the pain of optimising twice a day and having no way to return to previous parameters. I wish this message was there!
I personally feel it will not cause anyone to think that they’ve caused an error. Isn’t it quite intuitive for people to believe that some actions on digital platforms cannot be performed again and again? Product designers seem to design stuff with this in mind. Right now I can think of the example of closing recent apps. In my phone at least, doing it multiple times causes to show a toast with the message “your phone is in the best condition” or something similar to that effect. I think we’re trying to do something very similar.
edit: I find vaibhav’s suggestions for the message to very accurately capture the issue. In any case, users must know somehow that pressing optimize to frequently is not optimal. I wish I knew that but no use talking about it now.
It’s not really all that technical from a user perspective though. It seems to me the problem is that it’s a bit of a push-poll with a deliberately limited field of choice. The question of what should happen if the parameters are not going to be updated (message or no-message), and what the message should be (if there’s going to be one) are really separate issues.
It also doesn’t seem like a decision that is appropriate for democracy, because the individual user can really only offer their own hot-take. In that situation, a focus-group style survey [Ex. if you saw X, would you understand it to mean 1, 2, 3, or “I wouldn’t understand”] could get you closer to a useful response.
I decided to make a survey now: https://forms.gle/c8wHJQHjFbzUJHGw8
I will post it in Discord and on Reddit, but first I want to know whether you have any suggestions regarding the survey itself
I doubt you get a better spread of responses on Reddit, and most likely you will get very strange responses.
I think keep it as it is, this is not a bug or something that breaks the app or makes it unusable – developers should focus of those rather than something like this. In a SW company I would classify this as a P4 bug – i.e. if someone ever has some time to look at it, go for it.
