FSRS 5: Optimizer

L.M.Sherlock · October 27, 2024, 1:16pm

Finally, I located the bug. The Adam optimizer doesn’t work expectedly after applying parameter clipper in FSRS-rs.

It costs me 8 hours… And I cannot fix it by myself. This bug requires a patch from the upstream lib:

[Bug] Parameter Clipping Causes Loss of AdamState in Optimizer

opened 01:13PM - 27 Oct 24 UTC

### Description When implementing parameter clipping after the optimization ste…p, I encountered a bug that causes the loss of AdamState in the optimizer. This occurs because the clipping operation creates a new `Param`, changing the `id` of `model.w`. Since the optimizer's `AdamState` is stored in `records` with the parameter `id` as the key, this clipping step leads to the loss of the `AdamState`, preventing the optimizer from working as expected. ### Code Example ```rust model = optim.step(lr, model, grads); model.w = Param::from_tensor(parameter_clipper(model.w.val())); ``` ### Expected Behavior The parameter clipping should maintain the optimizer state (AdamState) for the clipped parameters. ### Actual Behavior The optimizer state is lost after clipping, causing the optimization process to behave incorrectly in subsequent steps. ### Environment - burn version: 0.13.1 ### Possible Solution We need a way to clip parameters without creating a new `Param` instance, or a method to update the optimizer's state when parameter IDs change. ### Related Issues This issue is related to the feature request for a Weight Clipper: - #662 ### Additional Context This bug was discovered while implementing a weight clipping feature similar to PyTorch's approach, where weights are clamped after the optimization step.

Topic		Replies	Views
Anticipated Features in 24.10 Help	40	982	October 27, 2024
When/How to separate presets for FSRS FSRS	27	1982	December 12, 2024
Measures to prevent new FSRS parameters from being worse than the old ones Suggestions	28	3064	January 28, 2024
25.02 possible bug - minimum recommended retention 0.70 FSRS	19	677	February 27, 2025
FSRS 5: Ideas and Sims FSRS	9	331	October 29, 2024

FSRS 5: Optimizer

Related topics