Could We Improve the AnkiWeb Add-on Rating System?

Anki Suggestions
1

I’d like to hear some opinions on the current rating system for AnkiWeb add-ons, especially from other add-on developers.

Currently, users can either like or dislike an add-on and are required to leave a comment. In theory, I think requiring some feedback is a good idea. In practice, though, many comments are just things like “good”, “nice”, a few words, or even effectively empty comments. So I’m not sure the requirement actually achieves much.

My bigger issue is the binary like/dislike system itself.

As an add-on developer, getting a very visible dislike can be surprisingly demotivating, especially when the reason is a minor issue, a missing feature, a misunderstanding, or something that could easily have been resolved through the add-on’s support page or GitHub.

Of course, negative feedback is important. If an add-on is broken, harmful, misleading, or simply doesn’t work, users absolutely need a way to express that. Negative ratings should also affect rankings so that genuinely problematic add-ons don’t stay at the top.

But I don’t think every criticism needs to translate into the same big red thumbs-down.

There is a huge difference between:

“This add-on destroyed my collection and doesn’t work.”

and:

“It works, but I wish it had feature X.”

Yet under the current system, both can result in exactly the same rating.

It can become particularly frustrating when a dislike contains only two or three vague words. You reply, offer support, ask for more information or a proper bug report, and then never hear from that person again. At that point, there is often nothing actionable for the developer, but the negative rating remains permanently visible.

I’ve also occasionally seen several dislikes appear within a short period after weeks or months without any, which makes the system feel particularly vulnerable to rating manipulation. I obviously can’t know whether that is actually happening, but a binary system makes a small number of negative votes disproportionately noticeable.

I think there are several ways the system could be improved:

  1. Bring back a star rating system.
    A 1 to 5 star system would allow much more nuance. Someone who generally likes an add-on but has a few problems could give it 3 or 4 stars instead of having to choose between “good” and “bad”.

  2. Keep likes and dislikes, but only display likes publicly.
    Dislikes could still affect ranking and remain visible to the add-on developer, but the public add-on page would only show the number of likes. This is somewhat similar to how YouTube handles dislikes today. The information would still be useful for ranking without putting such a strong visual emphasis on individual negative votes.

  3. Separate ratings from comments.
    Let users like or dislike an add-on without forcing them to write something, while providing a normal comment section for bug reports, suggestions, questions, and general feedback. The current requirement to comment doesn’t seem particularly useful when people can satisfy it with one word anyway.

Personally, I think a star rating combined with a proper comment section would be the best solution. I also understand that migrating the existing like/dislike data to a star system could be difficult, since there is obviously no sensible way to determine whether an old dislike should become one, two, or three stars.

So even something simpler, such as keeping the current system but hiding the public dislike count, could already be an improvement.

I’m bringing this up because the current system has genuinely affected my motivation to continue developing free add-ons at times. When you spend a lot of time building and maintaining something for free, an unexplained thumbs-down with no actionable feedback can be disproportionately frustrating.

I’ve also seen similar situations on other add-on pages, where developers receive very vague negative feedback and still respond patiently, but there is simply not enough information to understand what actually needs to be fixed.

I’d be particularly interested in hearing from other add-on developers: How do you feel about the current system? Do you prefer having likes and dislikes publicly visible, or would you rather have something like star ratings or a different feedback system altogether?

Thank you for reading.

2

I agree in parts, I guess we can shape your suggestion into something cleverer to something that can be well-used in the future.

  1. The binary way of thinking is fast and easy for most users, but stars help developers know with a bit more accuracy what they think. In terms of this, I believe we should prioritize what is best and easier to the user, even if stars are cool and more specific.
  2. Criticism must be displayed, not debatable. When you create an app, or an add-on, or anything and you put it out there, this is the price. True respect and recognition from the community comes from overcoming those hating comments. However, a way to filter down what is hate and what is actual feedback would be interesting, it’s a start, not a definitive solution.
  3. Now this is interesting and it’s something that is default for other stores such as App Store or Play Store. The user can either write it’s comment and have it displayed or just give a rating. Would make it easier for users to comment and stimulate who wants to contribute. An option to pin comments would also be cool.

When it comes to the current system: not very helpful to display add-ons, not very useful to evaluate comments and help users around, not very pretty (and tbh, very minor and nitpicky issue).

Here’s my take: AnkiWeb Add-on platform is simple and easy to maintain, the more features you add (stars, hide comments, report hate, etc etc), more bugs will show up and more maintenance will be required, so it needs to be decided either to enhance a few things, make it a little better, keeping the core of the website untouched, or building another one from ground-up like Unlucky suggested and did.

Interesting discussion, but low priority, I believe AnkiHub has other things on the list first before starting to improve the Add-on system. Personally, I hope for the best, but wouldn’t pressure them, even though it would be appreciated, and I would love to have a few new things too (such as pinning comments or having a built-in gallery to display previews), but we need to go step by step, be grateful to have a platform to show our creations and, when the time comes, give this part of the system a little more care.

Appreciate you bringing that up, Glace, and asking for opinions of the devs!

PS: An interesting side note, a little while ago I wrote this comment here, about a similar topic.

3

There’s a lot of slop on AnkiWeb nowadays and some addons could be dangerous, so it’s important that the negative feedback (dislike counts and negative comments) stays clearly visible to everyone.

4

I agree this addon rating system has remained the same since… 2013! No wonder that it feels primitive.

However I would stand against a star ranking system, even though it is a fine way to rank, simply because it is not backward compatible. All addons would have to restart from 0 ratings.

The number 1 problem I have is why are we forced into either a thumb up or a thumb down? Why is there not a lukewarm reaction available? Options like “rather positive” and “rather negative” are needed.*

both as an add-on / public deck user and as an addon developer, I’ve seen people and have myself have downvoted add-ons or decks that were very good simply because they had some specific issues and there was no way to communicate to the author/warn other users of the existing problem(s).


Simply adding another alternative to the thumb up and the thumb down, and relying offering a percent of approval instead, where the number of users that voted would be much more fair:

rating_addon
rating_addon1281×251 33 KB



You could even do something like what Steam does, where they offer two ratings: one for the entire lifetime of a specific game, and another that is calculated only from the recent reviews of the last 30 days. Because some add-ons may have some problems during the initial cycle of development, but later become more mature and worth using:

rating_addon2
rating_addon21284×257 32.2 KB

…And surely the people running anki can pay a guy / claude to make svgs so that the color/shape of the icon changes smoothly from a red thumb down to a green thumb up at specific levels, say 25% liked, 50%, 70%, 80%, 90%.

5

The official Anki team has recently been conducting a survey for add-on developers, so it might be helpful to send requests regarding the add-on system directly to them.

6

Sometimes a user runs into trouble and leaves a dislike, even though they don’t actually know how to use Anki properly. That’s been my experience—sadly, I’ve received exactly one dislike across all my add-ons. People do leave likes, though—sometimes even when there are errors (some of which aren’t my fault)—but it’s good that they took the time to leave a positive rating at all. If you make a perfect add-on, people often won’t even bother rating it positively; they just can’t be bothered. You can look at download counts, but that’s not always a reliable indicator. However, when you update a version and see at least 100 downloads over a week or two, that serves as a decent benchmark for popularity. On the other hand, there are plenty of add-ons that don’t even reach 100 downloads over their entire lifespan—they might only be useful to ten people.

From a developer’s perspective, it’s useful to know the number of positive and negative reviews—and not just the all-time download count, since it’s unclear whether a negative review is from five years ago or the current year. So, in addition to the total number of reviews, it would be great to show the annual figures in parentheses—something like this:
:+1: 20 (+1); :-1: 5 (+1); :down_arrow: 1000 (+150)

7

I have to point out that percentages could be cheated by malicious authors.

There was a deck with (I think) +7000 fake likes. The percentage would drown the negative reviews even if there were a 100 users warning others about the deck.

8

and? I don’t see how that is being “exploited”. A deck with a 10 to 1 like/dislike ratio under current rating and credible looking reviews would still be downloaded by most people. Bot control is a separate issue.

9

98% likes sounds good. 7000 likes and 100 dislikes sounds suspicious.

Just changing the way the info is displayed can have an effect on users’s choices.

Until there’s a solution to the bot issue, this is still important to consider.

10

We are working on improving the add-ons listing as we speak. Improving how rating system will is displayed is on the list of requirements for the new listing.

11

Before thinking if the the rating system should be changed, I think it’s worth looking into giving authors a better way to respond to negative feedback, like some integration with the forums so that comments are posted here and authors/users are notified when someone replies.

12

Still hoping to keep this discussion from being split up, but adding a link to the parallel Reddit thread for reference – Reddit .

13

Just a small follow-up after also discussing this on Reddit.

One argument that came up there was that dislikes and negative reviews need to remain clearly visible because some add-ons may be completely broken or even malicious, and users need to be warned about them.

I completely agree with the goal, but I don’t think a binary :+1:/:-1: rating is a particularly good safety mechanism for that.

A :-1: can currently mean anything from “this add-on is dangerous and caused data loss” to “I don’t like this feature”, “I had a minor problem”, or simply a misunderstanding of how the add-on works. All of those have exactly the same effect on the rating.

Negative reviews themselves should absolutely remain visible. My point is more that ratings, criticism, and actual safety reports are three different things and ideally shouldn’t all be represented by the same thumbs-down.

For genuinely harmful or malicious add-ons, I think a dedicated reporting system would actually protect users much better. If an add-on is suspected of causing data loss, doing something malicious, or otherwise being dangerous, users should be able to report it so that it can potentially receive a prominent warning or be removed after review. That is a much stronger safety mechanism than relying on ordinary dislikes.

There is also the developer side of negative reviews. We can reply and the reviewer gets notified, but quite often there is no response afterwards. You can offer support, ask for more information, explain a misunderstanding, or fix the reported problem, but if the reviewer never comes back, there isn’t really a way to continue that conversation. Making it easier for reviewers and developers to communicate around negative feedback could therefore also be useful.

And since it was mentioned above that the add-on listing is already being improved: that’s really great to hear.

If the rating display is already being reconsidered, I think the most useful ideas from this discussion so far would be:

  • More nuance than a strict :+1:/:-1:, without necessarily requiring a complete migration to stars.

  • Keeping actual negative reviews visible while separating them more clearly from general ratings.

  • A dedicated reporting mechanism for genuinely harmful or malicious add-ons.

  • Better interaction between developers and reviewers when someone reports a problem.

  • Potentially showing recent rating trends separately from lifetime ratings, so an issue from years ago doesn’t necessarily represent the current state of an add-on.

I understand that not all of this is necessarily practical or worth the additional complexity. But I wanted to clarify that my suggestion isn’t really about shielding developers from criticism. It’s about making the feedback system more informative for users while also making negative feedback more actionable for developers.

Really glad to hear that the listing is already being worked on, and I’m looking forward to seeing where it goes.

14

Ideally, the add-on author would receive an email notification for every comment received—similar to how it works on forums or Reddit, where a notification count (e.g., “3 notifications”) appears at the top.

Furthermore, regarding replies, it would be great if users could reply directly beneath the original comment—like on Facebook—and continue the conversation right there. As it stands, the author has to delete and repost their reply to update it, and the commenter has to do the same.

When a user comments on an add-on saying “it doesn’t work anymore,” the author often doesn’t know what to do because they are unaware of the user’s Anki version or operating system. I believe making it mandatory to specify the operating system and Anki version in comments would be a huge help to add-on authors.

Many of my add-ons receive the same type of comment: “There’s no English version,” “there’s no Chinese version,” “there’s no Arabic version.” Sometimes the add-on is in English, but the person says “I don’t understand this language” and leaves a negative review anyway. I feel this type of comment just hinders add-on creators and is completely irrelevant, since anyone can easily translate the code. Ideally, such comments would be reviewed before publication, as they only confuse people checking the add-on’s ratings. For example, an add-on might get five negative reviews simply because someone didn’t like the language; another person sees only the rating and loses interest in the add-on entirely.

Another issue: someone says something stopped working, leaves a negative review, lists details, etc.; the author fixes all the problems, but months later, that negative review is still there. I’m not even sure what can be done about that. It would be cool if the person posting the comments could be identified too, since they can do whatever they want there—like swearing and such—and get away with it, given that there’s no report button.

15

It’s never a good move to make stuff mandatory. If it’s within the privacy policies, maybe the addon author can get access to debug info directly through some custom UI or maybe recieve error reports like AnkiDroid.

As for the stupid reviews and bugs that later get fixed:

  • Leave a reply to let other users know that is not an issue/fixed.
  • Someone above suggested showing ratings in the past 6 months in addition to the overall ratings.
16

If the person commenting doesn’t state which version of Anki and which operating system they are using, how is the add-on author supposed to figure out where the bug is? Anki has many versions, and a bug might occur in just one specific version.

They would simply need to select—or type in—the Anki version and operating system they are using.

I have several versions of Anki installed on one of my PCs myself, but even so, errors sometimes pop up for versions of Anki that I don’t have installed.

17

Well I gave two alternatives in my comment.

A middle ground would be to provide a template that asks the user to do such and such like on this forum.

18

I agree with you that display language isn’t a valid criticism – but it’s incorrect to assume that your users would be able to “easily” translate an add-on to a different language. Most users would have no idea how to do that.

We consider that space to be an extension of the Forums, so you should feel free to report user comments that violate our Community Guidelines or our Terms of Service. You can do that by opening a private ticket, or by DM here to @ moderators.

That wouldn’t be something we could share without users specifically opting in. But it’s also unlikely the browser where the comment is being entered could access those details. I think voluntary is the only way there.