What's a good way to fake time to simulate reviews in tests?

Anki Development
1

I’m trying to test my addon with Anki and I want to simulate some reviews over time. Since my addon is triggered by a card being shown in reviewer, and messes with the queue, I want to simulate this by, well, doing actual reviews in the reviewer. For this to work, I need to change Anki’s clock. (I could just use time.sleep(), but GitHub Actions doesn’t let their workers run for a months.)

Apparently Anki doesn’t use its own clock object and calls system time functions directly? I tried a dumb way of monkey-patching time.time() and it kind of worked… for the v2 scheduler. But not so much for the v3 scheduler, which as I understand is done mostly in Rust and hence doesn’t use time.time(). I tried looking for v3 scheduler tests but I’m not sure there are tests that change time? I am not sure how this is tested. Probably I missed some tests somewhere?

So, what’s a good way to simulate some reviews in the past?

2

I once successfully used RunAsDate to demonstrate intervals for an introductory article about Anki. This is a manual process, and you may want to find a way to do what RunAsDate does programmatically (Not sure if this is possible from Python in a way that affects the whole application).

3

So I looked around a found a not quite perfect but suitable for my case workaround, libfaketime. It fakes time in a way that also works for Rust.

The problem is the way Anki handles time in Rust. It uses elapsed(), which does not like time going backwards considering the start of the application or something. And then Anki has some uncommented logic to prevent today going backwards. So, if you only ever forward time to the future, it will work okay. If you don’t, expect some discrepancies.

Anyway, this makes my tests pass. But I think Anki should support using a custom clock?

# this changes clock for both Python and Rust by doing some C magic.
# libfaketime needs to be preloaded by the dynamic linker;
# this can be done by running the following before the tests:
#   $ eval $(python-libfaketime)
# clock can only be set forward due to the way Anki's Rust backend handles deck time:
# it calls `elapsed()`, which returns 0 if clock went backwards.
#
# EXTREME CAUTION: Rust backend usually regenerates scheduler “today” via current time,
# but it stores the last result and will return it instead if is more “recent”.
# this means that “today”, at least as seen from python, can't go backwards!
# on the other hand, it seems that this does not affect answering cards in reviewer.
# see rslib/src/scheduler/mod.rs -> `impl Collection` -> `fn scheduler_info`
@contextmanager
def clock_set_forward_by(**kwargs):
    delta = timedelta(**kwargs)

    if delta < timedelta(0):
        raise Exception(f"Can't set clock backwards ({delta=})")

    with libfaketime.fake_time(datetime.now() + delta):
        yield
4

The v3 scheduler has its own unit tests in Rust, some of which don’t rely on the current time. The API exposed to Python does depend on the current time, and was not really intended to be used in testing. If you need historical data to test with, you could just inject it directly into the collection.

5

By simulating actual reviews I can be sure as to what is actually happening in the reviewer. If I am injecting review data, I am testing against that data, not against the actual reviewers and changes in Anki. And changes in Anki is the very thing I want to test against.

Also, If I do review simulation, I know that this test is a good test, since it’s as close to the real thing as it can be. I don’t have to think about whether or not my injected data is “real” enough. Perhaps in an actual Anki the test that works with injected data would fail?

Besides, my addon is triggered by reviewer action and touches the reviewer anyway.

(By the way, Qt is quite easy to test, but in some cases Anki makes testing very hard. I think this can be easily fixed. Perhaps Anki could even provide an official framework for testing Anki? Only a few addons use tests; having an official framework with neat examples would serve as a major incentive for people to write tests.

What I currently use for testing Anki takes some 450 lines of code along with all the helper methods, plus a little bit from pytest-anki, and it could be reduced by quite a bit if I could remove stuff patching Anki. So it wouldn’t be a very big job.)

6

I think it’s great that you’re trying to be thorough, but I suspect you’re get most of the way there with something simpler, such as a combination of type checking and a simple test that doesn’t require modifying the clock. But I’m running on assumptions here, and perhaps you’r doing something I didn’t expect.

Regarding changes to the core code to facilitate testing, do you have some specific examples? Small changes that are minimally invasive might be possible, but an official framework is less likely in the near future.

7

It’s not that I’m being thorough, it’s just that what my addon is doing is pretty dangerous and I don’t want it to screw things up.

Either way, the reason I want to test via the most human-like route is because I think addons need different tests than Anki itself needs.

See, I don’t want unit tests. Unit tests are of little value to me. Unit tests are useful when you change your units; I made this addon years ago and I haven’t really added any new functionality to it since. (Although I’m going to.) Because it works fine. It does a very simple—dangerous, but rather simple—job of slightly delaying siblings of cards, so that they don’t appear too close to each other. (By the way, my users strongly thing that this should be core Anki functionality :p).

Anyway, you may ask, why do you want tests at all, then? Well, it’s because I can’t pin my dependencies, or rather my dependncy, which is Anki. You can ship Anki with Python 2 if you want to, but if my addon does not support the latest version of Anki, people are going to be a little sad.

So what I want is to make sure my addon doesn’t blow up with a new Anki version. That is, I want integration test, the most integrationey ones there are. And it’d be awesome if I can put them into GitHub actions, and make those fetch the latest prerelease version of Anki and run tests against that. So instead of learning of any issues 6 months after users report them on my addon page (I don’t think I get emails about addon comments?), GitHub emails me about failing CI jobs on Dependabot’s PR.

I have kind of done this already, actually, and it works pretty well. For now. But since Anki is sometimes a bit mean with tests, I had to invent various clutches to make tests work. It’s likely that upcoming versions of Anki are going to be breaking these clutches. Maybe more so than the addon itself. So it would be nice if I didn’t need these clutches.

I just think making Anki a tiny bit testing-friendlier, and making a little official testing framework, would greatly benefit Anki ecosystem, at very little cost.

1 Like