Incremental reading in long term future?

Anki Suggestions
1

Background:

I am an avid learner. I focused on Q&A for language learning in terms of quantity of flashcards. I tried incremental reading several times across my whole Anki 10y experience. There is a huge difference now, I do learn by pleasure, no academic degree, no tests, no deadlines, just following my learn drive. That eventually leads incremental reading.

After a decade of hearing about Supermemo and ignoring it for being proprietary software, I finally give it a change for incremental reading (IR). Recently I have seen trying out SuperMemo 15 freeware version for 10+ hours, watching videos of its capabilities and reading a lot about it. I am convinced, although there are someinherently limitations as being propietary software. I’d use that software for IR if it was open source i.e. free as in freedom.

Today I came across an open source Project called Polar for incremental reading which has a simple Anki synchronization. The bad news is that two weeks ago the development team (I think just 2 guys) decided to go cloud-only, so even if the project is open-source, I cannot use it for my needs.

Personal source of frustration: Semantic learning is getting more cumbersome as the collection grows.

Most of my collection has been Q&A cards generated from electronic sources and then I added mnemonics manually in some cases. For IR i do mostly clozes by copy-paste-adapt from electronic documents and is time consuming to get the metadata in the flashcards which creation I semi-automate.

For the time being I use Firefox to keep track of what I do read, what I am reading, and what I want to read. This limited approach forces to read whole documents before moving to another one for the sake simplicity.

So I read more linearly than I’d like in order to create flashcards on not have a chaos of so many incomplete read documents in parallel since I cannot keep track of the sections I read. So strictly speaking, I do no incremental read that much.

Goal:

This rises a question if any point in the future there is an opportunity to have incremental reading included in Anki. Managing documents, importing websites, pdf files, extracting text from them to finally create flashcards out of them and have everything linked (that is not the case with Polar connect add-on) so I can track back the source text where the flashcard was created, manage semantic redundancy, etc.

I do strongly believe in the Anki project. I think an external app (and team, community) with different aims, focuses, priorities, convictions, etc won’t materialize a healthily functional IR-fied Anki implementation, at least following the Piotr Wasniak definition of IR.

I think in some years time that idea could be feasible once the backend is fully migrated and all priority stuff has been done.

What are your thoughts?

14 Likes
2

There is this: https://ankiweb.net/shared/info/1781298089
Unfortunately, it isn’t included in the queue, so I decided developing my own add-on, which is currently not in a usable state for even myself though:
https://github.com/p4nix/true_incrementalreading

I personally believe that incremental reading is one of the reasons why starting intervals/exponential growth in Supermemo actually can be so huge: it gives extremely coherent knowledge.

Please note that one issue of my implementation will be that IR ideally requires prioritisation of topics, and prioritising topics without prioritising cards doesn’t make sense. However, with the current algorithm which does not allow early or late repetition, this would be a recipe for disaster. This is why I try to convince all of the mathematicians/statistics wizards I can find into trying to develop an algorithm. As soon as I am done with this add-on (while writing a bachelor thesis, lol) - I will also have a look at SM17/18. Based on my understanding, the algorithm by itself isn’t that hard, just acquiring the needed repetition data and finding the right parameters for weighing/matrix smoothing.

1 Like
3

have you tried incremental reading add-on?
i have not used this add-on that much. i just tried it, it can import webpages into one card, then you can highlight/edit that card on the reviewer page.

i honestly haven’t tried incremental reading and don’t know how this add-on will help you (you might even know about it already). but as you said:

So I read more linearly than I’d like in order to create flashcards on not have a chaos of so many incomplete read documents in parallel since I cannot keep track of the sections I read.

the highlighting/editing the card on reviewer page, might help you solve that.

this add-on obviously has more features than just the simple stuff i mentioned, but as i have not used it, i can’t really say what it does.

4

I’m afraid it’s unlikely to happen in the near future - there are a number of things that need to change before it’s even practical. Anki was not designed to handle large amounts of content in fields, and supporting that requires changes to syncing, the database format, full text search, etc, all of which will take time.

4 Likes
5

The Incremental Reading Add-On hasn’t been updated for ages…

Incremental reading as it is implemented in Supermemo is a truly amazing experience, and I hope that my add-on comes close to it - aside from the mentioned difficulties.

@dae just how large can data in fields get? I did not notice any issues with my add-on and a few wikipedia pages yet. However, I expect most data to be pdf anyway which will be stored externally.

2 Likes
6

130712 bytes. That was the reason I made this add-on among others.

7

Yes. Although briefly becouse the approach is more or less what I do. I don’t want the whole text to be as a flashcard, etc since searching becomes caothic with so many unrelated hits, etc. It is a good apprach for an add-on since you cannot change the Anki structure.

8

I am awar eof this, that why I said long-term future although this term is subjective. What I had in mind was arround year 2022 or so. Is that too optimistic?

9

I do beliebethat Supermemo saves the text in external html files. At least when the content is not marked as “done”. I think that is actually a good aproach. So you can separete searching on flashcards and searching on the source file(s).

What I am most keen is the file management and progress tracking that Polar can do. (this pic for example https://getpolarized.io/assets/screenshots/2019-11-document-view.webp).

I am motivated enough to start a project like Polar with some Zotero behaviour fully local and customizable and in the long run that could be merged with Anki.

3 Likes
10

Actually I tried as well. I need to find someone who believe in the SRS, usually they have not the slightest idea hance the difficulty to get them involved. Other than that, just having all the implementation for IR with Anki’s current algorithm would be a huge game changer already.

3 Likes
11

Large fields increase the size of the collection, and can result in it growing above the 250MB AnkiWeb supports. It also slows down every search, as more content has to be scanned through.

I can’t make long term commitments at this point - as mentioned above it depends on numerous other changes being done first, and it’s not a topic I’ve even had a chance to properly explore yet.

12

3 posts were split to a new topic: Large collections

13

Why? Can you describe your current approach and its limitations in more detail and explain why you have to read the whole document before moving on? Have you thought about using something like Pocket or Hypothesis? I’m very interest in Hypothesis and plan to make an add-on that will consume annotations from its API to generate cards.

Why try to force everything into Anki when there are other tools that will likely always do some tasks related to the IR workflow better? There are entire companies, like Pocket, that are focused on curating libraries of “save-for-later,” material. What are the essential components of this aspect of IR?

  • Save pages/documents for later
  • Organization tools like tags
  • Searchable database
  • Tracking reading progress
  • Inline highlights and annotations
  • REST API for consuming highlights and annotations for card creation

I think it would be better to leverage those external tools and let Anki do what it’s best at rather than trying to force everything into Anki.

You could have the dream IR work flow now by glueing together few external tools in a unix-like manner. let outside tools handle the collecting, organizing, tracking progress, annotating, and let Anki handle scheduling.

Also, check out readwise.io if you haven’t already. It facilitates a really nice, low-effort workflow, though it leaves much to be desired for those of us who want a more sophisticated approach.

5 Likes
14

Hi Andrew, thank your the links, I wasn’t aware of readwise.io

Limitations:

  • The process of reading is not scheduled. That should be mandatory. Secondly, priority management, dependancies, etc should be done manually which gets harder as more documents are read in parellel.
  • Relying on cloud-based solutions at some extend.
  • External apps may track what I have read once. IR involves rereading to break down information into smaller chunks at each iteration. Also, those iterations are not done linearly. Taking annotations or highlighting are different cognitive processes.
  • Usually cards generated are content only i.e. clozes or Q&A missing metadata. This can be solved but it’s time-consuming. If the creation load is light there is no issue, if I create 3K clozes in two weeks (which in fact I did) the process using external apps becomes too cumbersome to handle hence I prefer to select a whole document or a bunch of files to avoid all non-tracked and/or non-scheduled (re)reading left.

If I cannot understand a text with my previous knowledge, I move on and learn what I need to be able to do so, perhaps doing so several times back and forth. That leads to a decent ammount of documents read in parallel i.e. without finishing to read documents even once.

I mean, Wozniak’s Article about IR mentions the possibility to read incrementally +2K documents. I though that was as crazy as unnecesary when I was introduced to IR. Now I do understand that is failry common for non-casual users. I am eager to learn and I end up with lots of unfinished documents in parallel, as descrived in the previous paragraph. I am the kind of guy who reads a scientific paper about the sleeping patern of sea lions in the antarctic at 2 am and feels excitement to piece things together with other stuff I read about.

So, I am aware what my vision is for so niche use and involves a great deal of maintainance. For the time being, I will be moving on SuperMemo, dealing with the cognitive dissonance of being propietary has been painfull to accept. I will keep using Anki for everthing else than IR, which itself is a broad range of learning stuff.

4 Likes
15

This can be handled by Anki by generating a card for an article which has a special note type and perhaps a special scheduling algorithm.

This is where Hypothesis comes in. I think the workflow I have in mind with Hypothesis would satisfy your needs. The primary requirement of IR is an iterative workflow in which a chunk of info reaches its final form as an SR card after several steps of processing. Does that sound right to you?

The highlights/annotations serve as the first step of this process. That first step is simply to get the chunks into Anki. You can do whatever you want with those chunks once you pull them into Anki. You might need some special add-ons to handle splitting the chunks, etc., but the workflow could be accomplished quite easily. This workflow eliminates the technical overhead of implementing a bunch of stuff inside of Anki that might be better left outside of Anki.

Side note: I actually have a related but distinct notion of a “Smart Cloze Note Type,” which I just realized is, in a way, the reverse of the iterative process we’re discussing. Instead of breaking chunks down into smaller and smaller chunks, a smart cloze note type would merge multiple cloze cards generated from the note into a single cloze card (i.e., all cloze numbers get changed to 1), once the separate cloze cards reach a state of maturation. E.g., a cloze note that generates 5 different cards would eventually suspend those 5 cards and merge them together into a single review card. This would require extra care when creating such cards.

Have you read Michael Nielsen’s article about how he uses Anki to understand scientific papers?

I predict you will be pleasantly surprised about what can be accomplished by stitching together a few different tools to implement a fully featured IR workflow. I will keep you posted as I start working on this. The Hypothesis add-on is first on the list.

2 Likes
16

I honestly think the fundamental ideas of incremental reading are quite simple. It’s the nuts and bolts involved with the technical logical implementation that becomes confusing. Far more complicated problems have been solved with very simple design patterns. I suspect that an Anki-centric implementation of IR is actually closely within reach by weaving together a couple of external tools and a few simple add-ons. and I suspect it will be far more flexible and powerful than other approaches. That’s my hypothesis, which I intend to test :nerd_face: :bulb:

2 Likes
17

I am very looking forward about these key add-ons. I may have a rather strong personal bias against cloud solutions and I don’t know if I will keep using SM once this day comes. What I can state is that I will try it out for sure and potentially get involved.

Btw, this topic is the 15h most visited ever here. I guess we are having some signs of interest from users or google hits.

5 Likes
18

Hi, how’s the add-on projects going now?

2 Likes
19

didn’t read ur post, TL;DR

but recently someone reboot-ed the anki’s IR addon (unofficial, clone one).

somehow it works like SM’s IR,
but much less sophisticated.

have a look

20

What is SM’s IR?

Thank-you