Right now I’m just gonna wait for like 30 hours to test 1 vs 10 seeds. That stuff takes forever even on a relatively small dataset
EDIT: update, I finished doing it with 2 seeds on 200 collections, some with (relatively) few reviews, and some with a lot of reviews. Note that I deliberately selected collections close to the extreme ends - either a lot of reviews or not a lot.
Here is the graph of the differences as a function of the number of reviews:
For each collection, RMSE(2 seeds) is calculated simply as min(RMSE(seed 1), RMSE(seed 2)).
Average difference = -0.0005, meaning that using 2 seeds to run the optimizer twice and then selecting the minimum improves RMSE by 0.05% (absolutely, not relatively).
Max difference = -0.0086 (-0.86%) in favor of 2 seeds.
Average difference for collections with <5k reviews = -0.0007 (-0.07%) in favor of 2 seeds.
Average difference for collections with <1k reviews = -0.0008 (-0.08%) in favor of 2 seeds.
Here’s the distribution of differences:
It looks like an exponential distribution to me.
Of course, this is with just 2 seeds. I will keep running it with more seeds. More seeds = the minimum will be lower = the difference will be bigger.
Also, this is only for 200 collections, but I don’t want to run it on 20k collections, it will take me months.