# Suggestion: run several simulations and use their average + confidence intervals

**URL:** <https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476>\
**Category:** FSRS\
**Created:** [December 22, 2024, 2:13pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476 "2024-12-22T14:13:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Expertium](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/expertium/32/24176_2.png) [@Expertium](https://forums.ankiweb.net/u/Expertium)\
**Post date:** [December 22, 2024, 2:13pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/1 "2024-12-22T14:13:14Z")

</div>

Currently, when using the simulator, only one graph absed on oen simulation is plotted. However, I think it would be beneficial to use 5-10 simulations and make the final graph an average of those, plus confidence add intervals.  
Kinda like this:

 ![image](https://us1.discourse-cdn.com/flex002/uploads/anki2/original/3X/6/f/6f5b1f43c77c004df65ec526b33f8d1621f2d5e2.png)

Use different seeds for each simulation.  
This also means that the simulation will be 5-10 times slower, however. I think we should try it first and see how bad the slowdown is.  
@sorata pinging because this was originally your idea  
@L.M.Sherlock pinging because you are The FSRS-Man  
@A_Blokee pinging because you are the only one who knows how to plot stuff, or at least the only one who puts that knowledge to use for the sake of Anki

EDIT: you can _kind of_ do that by running several simulations with a different value of “Days to simulate”

 ![image](https://us1.discourse-cdn.com/flex002/uploads/anki2/original/3X/3/8/38d476c07a62302a5c89689328bf531f81a0ab2e.png)

Shorter timeframe:

 ![image](https://us1.discourse-cdn.com/flex002/uploads/anki2/original/3X/e/7/e7c1d22e58f76b8fca924f56eae7f6f59024ab6b.png)

---

<div class="post-metadata">

**Author:** ![A\_Blokee](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/a_blokee/32/17955_2.png) [@A\_Blokee](https://forums.ankiweb.net/u/A_Blokee)\
**Post date:** [December 22, 2024, 2:51pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/2 "2024-12-22T14:51:57Z")

</div>

Based on that graph the variance looks almost negligible

---

<div class="post-metadata">

**Author:** ![Expertium](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/expertium/32/24176_2.png) [@Expertium](https://forums.ankiweb.net/u/Expertium)\
**Post date:** [December 22, 2024, 3:01pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/3 "2024-12-22T15:01:36Z")

</div>

Yeah, but I wonder if there might be something I’m missing. I still would like to see a proper graph where everything is the same, only the seed is different.  
If the variance is small even then, then we can ditch this idea.

---

<div class="post-metadata">

**Author:** ![A\_Blokee](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/a_blokee/32/17955_2.png) [@A\_Blokee](https://forums.ankiweb.net/u/A_Blokee)\
**Post date:** [December 22, 2024, 3:26pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/4 "2024-12-22T15:26:47Z")

</div>

There.

 ![image](https://us1.discourse-cdn.com/flex002/uploads/anki2/original/3X/c/2/c2608bb418cc74e41a2ab53f646962b5247971b6.png)

Maybe my decks just weird but that is significantly more.

Still don’t think its worth it personally though.

If you want to try it:

> **[GitHub - Luc-Mcgrady/anki at random-simulator-seed](https://github.com/Luc-Mcgrady/anki/tree/random-simulator-seed)**
>
> random-simulator-seed

---

<div class="post-metadata">

**Author:** ![sorata](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/sorata/32/34115_2.png) [@sorata](https://forums.ankiweb.net/u/sorata)\
**Post date:** [December 22, 2024, 3:42pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/5 "2024-12-22T15:42:11Z")

</div>

Well, at least you got the idea out from the deep abyss of discord. The concerns you had are still valid, no way to do this without making the compute time longer. Or we only do this conditionally in some cases thereby confusing users.

---

<div class="post-metadata">

**Author:** ![Expertium](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/expertium/32/24176_2.png) [@Expertium](https://forums.ankiweb.net/u/Expertium)\
**Post date:** [December 22, 2024, 3:47pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/6 "2024-12-22T15:47:15Z")

</div>

I don’t know how to build from source code 😅  
Also, I’ve tried this (using different “Days to simulate”) on another preset, and it seems like there’s more variance

 ![image](https://us1.discourse-cdn.com/flex002/uploads/anki2/original/3X/e/7/e7c1d22e58f76b8fca924f56eae7f6f59024ab6b.png)

Maybe variance is harder to see on longer timeframes (the very first graph spans around 10 years, this one around 1 year)

---

<div class="post-metadata">

**Author:** ![rossgb](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/rossgb/32/23908_2.png) [@rossgb](https://forums.ankiweb.net/u/rossgb)\
**Post date:** [December 22, 2024, 3:51pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/7 "2024-12-22T15:51:45Z")

</div>

I would have expected that the simulations would diverge more over time.  
It’s weird that they are all so close.

---

<div class="post-metadata">

**Author:** ![sorata](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/sorata/32/34115_2.png) [@sorata](https://forums.ankiweb.net/u/sorata)\
**Post date:** [December 22, 2024, 4:12pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/8 "2024-12-22T16:12:44Z")

</div>

Interesting results with a low timeframe and a cap on review count.

_Review graph_

 ![1000240359](https://us1.discourse-cdn.com/flex002/uploads/anki2/original/3X/e/3/e3d3dacd4663ca76e1602c54cfdb9c91b1c60deb.jpeg)

_Time graph_

 ![1000240361](https://us1.discourse-cdn.com/flex002/uploads/anki2/original/3X/d/f/df1fc86cec9cc147083a84623ff4cc0ac8902b7c.jpeg)

No difference in review count but there is with time.

---

<div class="post-metadata">

**Author:** ![Anon\_0000](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@Anon\_0000](https://forums.ankiweb.net/u/Anon_0000)\
**Post date:** [December 22, 2024, 4:13pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/9 "2024-12-22T16:13:30Z")

</div>

> [@sorata](#):
>
> The concerns you had are still valid, no way to do this without making the compute time longer. Or we only do this conditionally in some cases thereby confusing users.

Maybe a checkbox saying something like “Compute confidence intervals” could be helpful? Or maybe something like “Compute confidence intervals (might take a long time)” to emphasize that it might take longer?

---

<div class="post-metadata">

**Author:** ![sorata](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/sorata/32/34115_2.png) [@sorata](https://forums.ankiweb.net/u/sorata)\
**Post date:** [December 22, 2024, 4:14pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/10 "2024-12-22T16:14:36Z")

</div>

Yeah, but there are two many options already and this one wouldn’t even make sense to a normal human. I bet you’ll see a lot of questions on the forums about it.

---

<div class="post-metadata">

**Author:** ![Anon\_0000](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@Anon\_0000](https://forums.ankiweb.net/u/Anon_0000)\
**Post date:** [December 22, 2024, 4:21pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/11 "2024-12-22T16:21:43Z")

</div>

> [@sorata](#):
>
> this one wouldn’t even make sense to a normal human

What if we hide it behind a ui thingy called “Expert” or similar?

---

<div class="post-metadata">

**Author:** ![sorata](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/sorata/32/34115_2.png) [@sorata](https://forums.ankiweb.net/u/sorata)\
**Post date:** [December 22, 2024, 4:21pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/12 "2024-12-22T16:21:46Z")

</div>

Ah wait, @rossgb do you remember that issue with number of reviews you have for a particular day changing when you change `days to simulate`? Isn’t that it? I guess I solved the issue? (Or all of you knew it before)?

This might need some work otherwise we’re gonna get bug reports from confused users who’ve got no idea what’s happening.

---

<div class="post-metadata">

**Author:** ![sorata](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/sorata/32/34115_2.png) [@sorata](https://forums.ankiweb.net/u/sorata)\
**Post date:** [December 22, 2024, 4:24pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/13 "2024-12-22T16:24:31Z")

</div>

I’ll bet that it’s never happening.

If I may brainstorm too: a pop-up that allows you to cancel midway?

---

<div class="post-metadata">

**Author:** ![Anon\_0000](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@Anon\_0000](https://forums.ankiweb.net/u/Anon_0000)\
**Post date:** [December 22, 2024, 4:30pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/14 "2024-12-22T16:30:18Z")

</div>

> [@Expertium](#):
>
> I don’t know how to build from source code 😅

You basically do this in a terminal:

```auto
git clone --single-branch --branch=random-simulator-seed https://github.com/Luc-Mcgrady/anki.git random-simulator-seed-folder

```

The above downloads the repo from Luc-Mcgrady, but only the branch _random-simulator-seed_, which is downloaded into the folder _random-simulator-seed-folder_.

```auto
cd random-simulator-seed-folder/

```

The above enters the folder.

```auto
./run

```

This command builds the entire branch and then opens the newly compiled anki. It might take quite a long time depending on your hardware.

For more info, see:

> <https://github.com/Luc-Mcgrady/anki/blob/random-simulator-seed/docs/development.md>

---

<div class="post-metadata">

**Author:** ![rossgb](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/rossgb/32/23908_2.png) [@rossgb](https://forums.ankiweb.net/u/rossgb)\
**Post date:** [December 22, 2024, 4:31pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/15 "2024-12-22T16:31:29Z")

</div>

I’m confused. I’m not sure what you are talking about.

It turned out that the issue I had earlier was because Jarrett was smoothing the data with a moving average. I don’t see how this applies to what we are discussing now.

---

<div class="post-metadata">

**Author:** ![sorata](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/sorata/32/34115_2.png) [@sorata](https://forums.ankiweb.net/u/sorata)\
**Post date:** [December 22, 2024, 4:34pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/16 "2024-12-22T16:34:47Z")

</div>

That’s the other one where tooltip data was one day off. You also had the data for same day appearing different when you put `days to simulate` 30 versus 60. And I don’t remember Jarrett talking about that.

---

<div class="post-metadata">

**Author:** ![Expertium](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/expertium/32/24176_2.png) [@Expertium](https://forums.ankiweb.net/u/Expertium)\
**Post date:** [December 22, 2024, 4:35pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/17 "2024-12-22T16:35:09Z")

</div>

The first two commands worked, the last one (`./run`) didn’t

---

<div class="post-metadata">

**Author:** ![Anon\_0000](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@Anon\_0000](https://forums.ankiweb.net/u/Anon_0000)\
**Post date:** [December 22, 2024, 4:36pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/18 "2024-12-22T16:36:04Z")

</div>

It’s

```auto
.\run

```

if you’re on windows.

---

<div class="post-metadata">

**Author:** ![rossgb](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/rossgb/32/23908_2.png) [@rossgb](https://forums.ankiweb.net/u/rossgb)\
**Post date:** [December 22, 2024, 4:37pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/19 "2024-12-22T16:37:44Z")

</div>

You’re probably going to need `ninja`/`n2` and I think also `rsync` installed to get that to actually work.

edit:  
`rustup` too

---

<div class="post-metadata">

**Author:** ![Expertium](https://sea2.discourse-cdn.com/flex002/user_avatar/forums.ankiweb.net/expertium/32/24176_2.png) [@Expertium](https://forums.ankiweb.net/u/Expertium)\
**Post date:** [December 22, 2024, 4:38pm UTC](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476/20 "2024-12-22T16:38:17Z")

</div>

That is not clear to an average user (how many people know what’s a “confidence interval”?), and splitting the UI into Beginner/Advanced mode is something I’ve proposed a long time ago and the responses (especially from Dae) were lukewarm at best.

[Next page](https://forums.ankiweb.net/t/suggestion-run-several-simulations-and-use-their-average-confidence-intervals/53476.md?page=2)
