Test has different results when run singly vs run with all tests

EDIT

running the single test with bazel test //pylib:pytest also fails, so I presume it’s a error/difference with how pytest runs the single test?

col.media.render_all_latex() is failing with

(1659401324352, 'Error executing \u2068latex\u2069.<br>Generated file: \u2068/tmp/anki_temp/tmp.tex\u2069<br><small><pre>latex: Environment variable HOME is not set.\n\nSorry, but latex did not succeed.\nterminate called after throwing an instance of &#x27;MiKTeX::Core::MiKTeXException&#x27;\n  what():  /miktex/source/Libraries/MiKTeX/App/app.cpp:1138: internal error\n</pre></small>')

I’ve got MikTex installed on my machine, could this be an issue with the renderer that gets called?

ORIGINAL

I’ve just downloaded the source code and went to run the tests on my machine, and I’ve found that one of the tests will fail if run as part of the whole test suite, but will pass if run on its own. The test that’s failing is test_latex.py. I’m running both sets of tests for the root anki directory.

I’m running the one test with PYTEST=test_latex bazel run //pylib:pytest --cache_test_results=no. This passes.

I’m running the tests globally with bazel test ..., and this test fails with

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pylib:pytest
-----------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.9.10, pytest-7.1.1, pluggy-1.0.0 -- /home/fabian/.cache/bazel/_bazel_fabian/8538c16ff032da4d6faec17a1623dac5/external/python/bin/python3
cachedir: .pytest_cache
rootdir: /home/fabian/.cache/bazel/_bazel_fabian/8538c16ff032da4d6faec17a1623dac5/sandbox/linux-sandbox/1057/execroot/ankidesktop/bazel-out/k8-fastbuild/bin/pylib/pytest.runfiles/ankidesktop
collecting ... collected 129 items

pylib/tests/test_cards.py::test_delete PASSED
pylib/tests/test_cards.py::test_misc PASSED
pylib/tests/test_cards.py::test_genrem PASSED
pylib/tests/test_cards.py::test_gendeck PASSED
pylib/tests/test_collection.py::test_create_open PASSED
pylib/tests/test_collection.py::test_noteAddDelete PASSED
pylib/tests/test_collection.py::test_fieldChecksum PASSED
pylib/tests/test_collection.py::test_addDelTags PASSED
pylib/tests/test_collection.py::test_timestamps PASSED
pylib/tests/test_collection.py::test_furigana PASSED
pylib/tests/test_collection.py::test_translate PASSED
pylib/tests/test_collection.py::test_db_named_args PASSED
pylib/tests/test_decks.py::test_basic PASSED
pylib/tests/test_decks.py::test_remove PASSED
pylib/tests/test_decks.py::test_rename PASSED
pylib/tests/test_exporting.py::test_export_anki PASSED
pylib/tests/test_exporting.py::test_export_ankipkg PASSED
pylib/tests/test_exporting.py::test_export_anki_due pylib/tests/test_exporting.py:133:getCard is deprecated: please use 'get_card'
PASSED
pylib/tests/test_exporting.py::test_export_textnote PASSED
pylib/tests/test_exporting.py::test_exporters PASSED
pylib/tests/test_find.py::test_find_cards PASSED
pylib/tests/test_find.py::test_findReplace PASSED
pylib/tests/test_find.py::test_findDupes PASSED
pylib/tests/test_flags.py::test_flags PASSED
pylib/tests/test_importing.py::test_anki2_mediadupes PASSED
pylib/tests/test_importing.py::test_apkg PASSED
pylib/tests/test_importing.py::test_anki2_diffmodel_templates pylib/tests/test_importing.py:121:getCard is deprecated: please use 'get_card'
PASSED
pylib/tests/test_importing.py::test_anki2_updates PASSED
pylib/tests/test_importing.py::test_csv PASSED
pylib/tests/test_importing.py::test_csv2 PASSED
pylib/tests/test_importing.py::test_tsv_tag_modified PASSED
pylib/tests/test_importing.py::test_tsv_tag_multiple_tags PASSED
pylib/tests/test_importing.py::test_csv_tag_only_if_modified PASSED
pylib/tests/test_importing.py::test_supermemo_xml_01_unicode PASSED
pylib/tests/test_importing.py::test_mnemo PASSED
pylib/tests/test_latex.py::test_latex <anki.collection.Collection object at 0x7fb45fc3e790>
FAILED
pylib/tests/test_media.py::test_add PASSED
pylib/tests/test_media.py::test_strings PASSED
pylib/tests/test_media.py::test_deckIntegration PASSED
pylib/tests/test_models.py::test_modelDelete PASSED
pylib/tests/test_models.py::test_modelCopy PASSED
pylib/tests/test_models.py::test_fields PASSED
pylib/tests/test_models.py::test_templates PASSED
pylib/tests/test_models.py::test_cloze_ordinals PASSED
pylib/tests/test_models.py::test_text PASSED
pylib/tests/test_models.py::test_cloze PASSED
pylib/tests/test_models.py::test_cloze_mathjax PASSED
pylib/tests/test_models.py::test_typecloze PASSED
pylib/tests/test_models.py::test_chained_mods PASSED
pylib/tests/test_models.py::test_modelChange PASSED
pylib/tests/test_models.py::test_req PASSED
pylib/tests/test_sched2021.py::test_clock PASSED
pylib/tests/test_sched2021.py::test_basics PASSED
pylib/tests/test_sched2021.py::test_new PASSED
pylib/tests/test_sched2021.py::test_newLimits PASSED
pylib/tests/test_sched2021.py::test_newBoxes PASSED
pylib/tests/test_sched2021.py::test_learn PASSED
pylib/tests/test_sched2021.py::test_relearn PASSED
pylib/tests/test_sched2021.py::test_relearn_no_steps PASSED
pylib/tests/test_sched2021.py::test_learn_collapsed PASSED
pylib/tests/test_sched2021.py::test_learn_day PASSED
pylib/tests/test_sched2021.py::test_reviews PASSED
pylib/tests/test_sched2021.py::test_review_limits PASSED
pylib/tests/test_sched2021.py::test_button_spacing PASSED
pylib/tests/test_sched2021.py::test_overdue_lapse PASSED
pylib/tests/test_sched2021.py::test_nextIvl PASSED
pylib/tests/test_sched2021.py::test_bury PASSED
pylib/tests/test_sched2021.py::test_suspend PASSED
pylib/tests/test_sched2021.py::test_filt_reviewing_early_normal PASSED
pylib/tests/test_sched2021.py::test_filt_keep_lrn_state PASSED
pylib/tests/test_sched2021.py::test_preview PASSED
pylib/tests/test_sched2021.py::test_ordcycle PASSED
pylib/tests/test_sched2021.py::test_counts_idx SKIPPED (old sched only)
pylib/tests/test_sched2021.py::test_counts_idx_new PASSED
pylib/tests/test_sched2021.py::test_repCounts PASSED
pylib/tests/test_sched2021.py::test_timing PASSED
pylib/tests/test_sched2021.py::test_collapse PASSED
pylib/tests/test_sched2021.py::test_deckDue PASSED
pylib/tests/test_sched2021.py::test_deckTree PASSED
pylib/tests/test_sched2021.py::test_deckFlow PASSED
pylib/tests/test_sched2021.py::test_reorder PASSED
pylib/tests/test_sched2021.py::test_forget PASSED
pylib/tests/test_sched2021.py::test_resched PASSED
pylib/tests/test_sched2021.py::test_norelearn PASSED
pylib/tests/test_sched2021.py::test_failmult PASSED
pylib/tests/test_sched2021.py::test_negativeDueFilter PASSED
pylib/tests/test_sched2021.py::test_initial_repeat PASSED
pylib/tests/test_schedv2.py::test_clock PASSED
pylib/tests/test_schedv2.py::test_basics PASSED
pylib/tests/test_schedv2.py::test_new PASSED
pylib/tests/test_schedv2.py::test_newLimits PASSED
pylib/tests/test_schedv2.py::test_newBoxes PASSED
pylib/tests/test_schedv2.py::test_learn PASSED
pylib/tests/test_schedv2.py::test_relearn PASSED
pylib/tests/test_schedv2.py::test_relearn_no_steps PASSED
pylib/tests/test_schedv2.py::test_learn_collapsed PASSED
pylib/tests/test_schedv2.py::test_learn_day PASSED
pylib/tests/test_schedv2.py::test_reviews PASSED
pylib/tests/test_schedv2.py::test_review_limits PASSED
pylib/tests/test_schedv2.py::test_button_spacing PASSED
pylib/tests/test_schedv2.py::test_overdue_lapse PASSED
pylib/tests/test_schedv2.py::test_nextIvl PASSED
pylib/tests/test_schedv2.py::test_bury PASSED
pylib/tests/test_schedv2.py::test_suspend PASSED
pylib/tests/test_schedv2.py::test_filt_reviewing_early_normal PASSED
pylib/tests/test_schedv2.py::test_filt_keep_lrn_state PASSED
pylib/tests/test_schedv2.py::test_preview PASSED
pylib/tests/test_schedv2.py::test_ordcycle PASSED
pylib/tests/test_schedv2.py::test_counts_idx PASSED
pylib/tests/test_schedv2.py::test_counts_idx_new SKIPPED (new sched ...)
pylib/tests/test_schedv2.py::test_repCounts PASSED
pylib/tests/test_schedv2.py::test_timing PASSED
pylib/tests/test_schedv2.py::test_collapse PASSED
pylib/tests/test_schedv2.py::test_deckDue PASSED
pylib/tests/test_schedv2.py::test_deckTree PASSED
pylib/tests/test_schedv2.py::test_deckFlow PASSED
pylib/tests/test_schedv2.py::test_reorder PASSED
pylib/tests/test_schedv2.py::test_forget PASSED
pylib/tests/test_schedv2.py::test_resched PASSED
pylib/tests/test_schedv2.py::test_norelearn PASSED
pylib/tests/test_schedv2.py::test_failmult PASSED
pylib/tests/test_schedv2.py::test_negativeDueFilter PASSED
pylib/tests/test_schedv2.py::test_initial_repeat PASSED
pylib/tests/test_stats.py::test_stats PASSED
pylib/tests/test_stats.py::test_graphs_empty PASSED
pylib/tests/test_stats.py::test_graphs PASSED
pylib/tests/test_template.py::test_deferred_frontside PASSED
pylib/tests/test_undo.py::test_op PASSED
pylib/tests/test_undo.py::test_review PASSED

=================================== FAILURES ===================================
__________________________________ test_latex __________________________________

    def test_latex():
        col = getEmptyCol()
        print(col)
        # change latex cmd to simulate broken build
        import anki.latex
    
        anki.latex.pngCommands[0][0] = "nolatex"
        # add a note with latex
        note = col.newNote()
        note["Front"] = "[latex]hello[/latex]"
        col.addNote(note)
        # but since latex couldn't run, there's nothing there
        assert len(os.listdir(col.media.dir())) == 0
        # check the error message
        msg = note.cards()[0].question()
        assert "executing nolatex" in without_unicode_isolation(msg)
        assert "installed" in msg
        # check if we have latex installed, and abort test if we don't
        if not shutil.which("latex") or not shutil.which("dvipng"):
            print("aborting test; latex or dvipng is not installed")
            return
        # fix path
        anki.latex.pngCommands[0][0] = "latex"
        # check media db should cause latex to be generated
        col.media.render_all_latex()
>       assert len(os.listdir(col.media.dir())) == 1
E       AssertionError: assert 0 == 1
E        +  where 0 = len([])
E        +    where [] = <built-in function listdir>('/tmp/tmprzy9_ibt.media')
E        +      where <built-in function listdir> = os.listdir
E        +      and   '/tmp/tmprzy9_ibt.media' = <bound method MediaManager.dir of <anki.media.MediaManager object at 0x7fb45fc3e880> {'_dir': '/tmp/tmprzy9_ibt.media'}>()
E        +        where <bound method MediaManager.dir of <anki.media.MediaManager object at 0x7fb45fc3e880> {'_dir': '/tmp/tmprzy9_ibt.media'}> = <anki.media.MediaManager object at 0x7fb45fc3e880> {'_dir': '/tmp/tmprzy9_ibt.media'}.dir
E        +          where <anki.media.MediaManager object at 0x7fb45fc3e880> {'_dir': '/tmp/tmprzy9_ibt.media'} = <anki.collection.Collection object at 0x7fb45fc3e790>.media

pylib/tests/test_latex.py:38: AssertionError
=============================== warnings summary ===============================
pylib/tests/test_models.py:243
  /home/fabian/.cache/bazel/_bazel_fabian/8538c16ff032da4d6faec17a1623dac5/sandbox/linux-sandbox/1057/execroot/ankidesktop/bazel-out/k8-fastbuild/bin/pylib/pytest.runfiles/ankidesktop/pylib/tests/test_models.py:243: DeprecationWarning: invalid escape sequence \(
    q5 = "text with \(x^2\) jax"

pylib/tests/test_models.py:246
  /home/fabian/.cache/bazel/_bazel_fabian/8538c16ff032da4d6faec17a1623dac5/sandbox/linux-sandbox/1057/execroot/ankidesktop/bazel-out/k8-fastbuild/bin/pylib/pytest.runfiles/ankidesktop/pylib/tests/test_models.py:246: DeprecationWarning: invalid escape sequence \(
    ] = "{{{{c1::{}}}}} \(2^2\) {{{{c2::{}}}}} \(2^{{{{c3::{}}}}}\) \(x^3\) {{{{c4::{}}}}} {{{{c5::{}}}}}".format(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED pylib/tests/test_latex.py::test_latex - AssertionError: assert 0 == 1
============ 1 failed, 126 passed, 2 skipped, 2 warnings in 10.18s =============

I’m not able to reproduce this. Is HOME defined in the shell you’re running from?

Hmm, that’ll do it. I passed in HOME as --test_env=HOME=$HOME and all tests pass, both when running all tests and when running just the one.

Strangely, HOME is defined in my usual shells, as well as in my usual python interpreter, and the interpreter under tools. However it’s not defined when I run either the single test or all the tests, which should mean that both fail. Very strange. From here, I figure that whatever is executing the latex renderer isn’t being passed that value of the TEST_TMPDIR properly, and looks for the original home? I’m really not sure.

Not sure what the issue is - for me, a normal ‘bazel test …’ works.

1 Like