I develop and fix Add-ons. When I was creating a GUI to pop up a changelog, I came up with the idea of asking for a rating along with it.
Press “RateThis” to activate the new feature. (The original purpose was to ask users if they would like to enable new features, because new features are likely to be bugs)
It looks like this.
[ Problem ]
I incorporated this into popular Add-ons, then Pokemanki seemed to come out on top in the Ratings ranking.(This feature is currently incorporated in 5 add-ons)
So I am concerned that this may have a negative impact on AnkiWeb’s rating system.
Thanks @Shigeyuki, I appreciate your offering to discuss this openly!
For context: I reached out to Shige because I noticed that some add-ons seemed to be asking for ratings in exchange for activating features, and I wanted to open up a dialogue on what effects this could have on the add-on ecosystem at large.
I don’t think any practices in this regard are banned, and my main concern here would be more towards transactional rating encouragements (rating in exchange for something), rather than prompting for a rating in an unobtrusive way.
But perhaps this discussion can help us come up with some guidelines that tackle these questions and at least discourage some patterns which might have a negative impact on the community. It might also help shed some light on some of the shortcomings of the current rating system that might motivate add-on authors towards practices like this.
Finally, @Shigeyuki, let me also reiterate that I really appreciate all the hard work you’ve been putting into your add-ons and ports, and that I don’t for a moment think that there was any ill intent here at all. I just think that these are patterns that, if more add-on/deck authors were to adopt them, would make things worse for everyone.
OK, to recap my thoughts, this is how I see the situation:
Users installing or updating these add-ons are currently likely under the impression that they need to leave a rating on AnkiWeb in order to access the full feature-set of the add-on. I think the implication for a lot of users probably also is that they need to leave a positive rating:
Among them are likely many users who have not actually used the add-on, yet:
As a result, these add-ons have been receiving more upvotes per day than I have ever seen, with most of them being single-word ratings:
For instance, within the last 23 days (when I assume this change went live based on the wording and frequency of the ratings), Pokemanki Gold managed to accumulate 192 upvotes.
To put this into perspective, 200-ish upvotes are a huge milestone in add-on terms. This is how long it has taken other popular add-ons to achieve it:
AnkiHub: 1.5 years
All previous Pokemanki add-ons combined: 4 years
Button Colours: 9 years
Japanese Support: 12 years
My concerns with this situation are:
From a user perspective, it’s not particularly great to feel like you have to rate something in order to use it, in particular if you haven’t even used it at all yet. This is an even worse experience when you install or update multiple add-ons at once and are then inundated with pop-ups.
From a developer perspective, it’s creating a situation where the add-ons in question are receiving a disproportionate amount of reviews compared to those of other authors, which will have a negative impact on the visibility of other people’s work, and will probably also feel quite discouraging to a lot of authors.
I am also concerned about the potential ripple effects of this:
We would no longer have a (reasonably) even playing field for ratings, so ratings would essentially become meaningless at some point.
Other add-on authors might feel like they have to adopt these patterns in order to compete for visibility. So we could end up in a situation where users would have to deal with dozens of these rating prompts any time they install or update their add-ons.
This was before my time, but from what I understand, there’s precedent for this when AnkiWeb had to disable download counts on add-ons because some users would redownload their own items to boost their ranking.
So I think this is a fairly realistic scenario, and one that would be a fairly terrible outcome for the add-on ecosystem, alienating both users and developers alike.
Also a couple of thoughts on the limitations of the current shared items system that might be contributing here:
As AnkiWeb lives separate from Anki, and many users might only be logged into Anki on their desktop and mobile devices, there likely is a fairly large barrier of entry towards leaving a rating. I think the review to download ratios demonstrate this well. In my experience they are around an order of magnitude lower than what people report for app store ratings, for instance.
There really is no good way for users to discover new add-ons or decks within the platform itself. They can scour through recently updated items, but as this is a mix of updates and newly shared things, it’s not a particularly good path. Right now, discoverability seems to mostly be driven by word of mouth and social media. This is always the case for any kind of project you want to share of course, but it seems particularly pronounced here.
I think there are some extension/plugin marketplaces that do organic discoverability quite well (e.g. VS Code Marketplace, Chrome Web Store). They also have a lot more resources at hand, of course. Still, perhaps we can take some more realistically implementable things away from them, like having a section for trending add-ons and decks.
I realize of course that there are a lot of different fronts to improve in the ecosystem, and this is probably not the one with the highest priority right now, but I still thought it would be a good opportunity to mention these issues.
Since extension marketplace was mentioned, Plugins - Obsidian has a great extension discoverability, they basically incorporated this page, in an easy accessible tab built into Obisidian
@glutanimate’s answer was comprehensive, and I agree with all the points he made - please don’t make users rate in order to gain access to something.
To be honest, I find even emoji usage in add-on titles to be a little unfair, as it makes the titles stand out more than other add-ons. I’m not sure how other marketplaces handle this though.
Thank you for the reply.
To clarify, how far is permissible?
[ Pop-up, rating ]
Pop-up asking for evaluation on first launch of Add-ons (no reward)
Pop up a changelog when Add-ons is launched for the first time (no rating button)
Add-ons has its own option with a button and description asking for ratings (no popup)
Add-on’s own option with button for rating (no description)
2, there are already some Add-ons on AnkiWeb that pop up a changelog.
1, there are rare add-ons that pop up a dialog asking for rating (other than me).
And if I (or add-ons developer) violate these, what will be done?
Account deletion
Add-on deletion
Not appear higher in the rankings (Same as a low rating)
I think this would be good to clarify, so that there is some guidance for future purposes. But in the meanwhile: I think it would be important to swiftly resolve the more straightforward issue of ratings in exchange for something:
Since we started this discussion three days ago, the add-ons in question received close to another 100 votes in total, which - again - is something that took other popular add-ons with just as many users years to achieve. So let’s please not allow the ratings to become even more skewed.
If the official Anki decision is “1. Account deletion” or “2.Add-on deletion”, I plan to delete all 5 add-ons that added this feature. In this case, I think the currently increased ratings are irrelevant.
If “3.Not appear higher in the rankings (Same as a lower rating)” or “4.Only a warning”, meaning it is my discretion, I would like to take glutanimate’s opinion on how this issue should be handled. In my personal opinion, if Dae does nothing, the Add-ons should be deleted and re-uploaded, or I will need to process the ratings intentionally lowered.
For the Emoji issue, the purpose of this is to use it as a tag.
I currently have 52 Add-ons uploaded to AnkiWeb, so it’s hard to read if they are all text. Some of the Add-ons I have forgotten that I uploaded.If I set Emojis by genre, it will be easier to recognize them and reply to users’ comments. This was inspired by the idea of using emojis in the names of Anki decks.
Emojis are not available on Anki for desktop, so if the same functionality is added to AnkiWeb, Emojis would not be available.
If the sole purpose of the emoji is as a visual tag for yourself so you can check the pages and respond to comments – wouldn’t it be much simpler to use your items-shared-by-author page to filter the list down for yourself?
I would even advocate for the deletion of all the ratings since this “pay”-for-play scheme was introduced on these add-ons. Users rely on the ratings and reviews to mean something – both in their quantity and in their assessment of the add-on. If it’s just a count of everyone who downloaded it (ever) and wanted to try the update – it is meaningless and useless (bordering on deceptive).
Honestly, I would prefer to remove or hide the ratings, but that would be too much work for the official Anki, so I think just deleting the add-ons would be fine.
Since Dae and Glutanimate agree, can I proceed on the assumption that Glutanimate’s opinion is almost equivalent to the official Anki’s opinion? I would like to leave the decision to both of you.
I don’t think any of that is even close to being warranted. In my view, this would only be relevant when talking about gross transgressions and people willfully harming the community and greater ecosystem. I don’t think that’s the case here at all @Shigeyuki. Maybe if someone else did this repeatedly in the future after being asked no to, but definitely not in this case.
In terms of the other options you listed, it’s tricky.
As @Danika_Dakika says, from an even playing field perspective, removing all ratings since the vote-to-unlock changes were implemented would be the fairest move towards other add-on authors. It would also make sense in order to set a precedent and discourage future add-on authors from playing around with the same idea.
From a user perspective, I see two different viewpoints:
I’m sure a lot of the votes were actually left in earnest by users who were happy with these add-ons, and taking those votes away is not ideal.
However, these users would likely have done the same for other add-ons they love, if they had been nudged more strongly in that direction. These are votes missing from those add-ons, and this has real ramifications on add-on discoverability. E.g. users picking “Pokemanki” over “Ankimon”, because the latter stands no hope to ever organically reach the same like to download ratio. As Danika said, in order for ratings to be useful to users they have to be meaningful and comparable, and the current status would have a long term impact on the entire rating system.
Similarly, I see two viewpoints from an add-on ecosystem perspective:
@Shigeyuki you’ve been doing a fantastic and important job in fixing and porting broken add-ons to Anki 23+, and the community is much better off for it. One could therefore argue that it’s to the benefit of the ecosystem at large for some of these add-ons to be signal boosted.
At the same time, other authors have also done a lot of work both over the years and specifically for 23/24 to port their add-ons, keep them updated, and push out new features. So watching their own add-ons see much less attention, and knowing that they will likely never be able to reach the same success naturally is quite discouraging, which is not healthy for the dev ecosystem long-term.
So my thinking would be this (though I want to stress that I’m just one out of many add-on authors contributing to the ecosystem, and this is just one opinion):
I think it’s in the community’s best interest to be generous, while still fair, as long as we say that these are special allowances we are only willing to make once.
My recommendation would therefore be to do the following:
Let’s split the five add-ons into two categories:
Rapid growth: Pokemanki and Progress Bar
Moderate growth: Anki Killstreaks, Zoom, and Always on Top
For the add-ons in rapid growth, assume that had they grown organically, they would have grown twice as fast as the fastest growing add-ons.
For the add-ons in moderate growth, assume that had they grown organically, they would have grown as fast as the fastest growing add-ons.
Review Heatmap has historically been one of the fastest growing add-ons. Right now it receives slightly less than a vote per day, so if we round that up, we get a nice number to work with.
Assuming that this is actually feasible and something you would be willing to do @dae, in practical terms, I would propose to:
Determine when the change went live for each add-on (e.g. probably ≈2024-03-27
for Pokemanki and ≈2024-04-06 for Progress Bar - if you still know the exact dates @Shigeyuki that would be perfect)
Keep all votes before the change went live
For Pokemanki and Progress Bar: Keep 2 votes per day since the change went live
For Anki Killstreaks, Zoom, and Always on Top: Keep 1 vote per day since the change went live
Set today as a cut-off for the vote compression, dropping any additional votes that might come in until then and/or until the add-ons have been updated
To filter out empty or single letter reviews, ideally we’d pick the longest one(s) on each day
To make sure all votes are encompassed in this and that we don’t protract this any longer it would then indeed be important to disable the rating prompt quickly @Shigeyuki
Thank you very much, I will stop all prompts as soon as possible.
However, I personally think they need to be removed, I think if official Anki Dae does that such work, it will slow down the development of Anki as a whole, and I do not think these add-ons are important enough to protect them.
Edit: Popups disabled. For confirmation, please use this code. 1045980020 1562475180 1677779223 1708250053 1923741581
For emoji, I removed only the 5 Add-ons emoji that are currently problematic, I did not remove emoji from the other add-ons. If they are clearly prohibited, I will remove them, but if possible, I would like to request a tag function exclusively for the author’s screen.
In my opinion, this is how I would like to process it.
If Glutanimate or official Anki’s Dae is OK, I think it can be processed within 1-3 days.
[ Processing the 5 problem add-ons ]
Upload an empty dummy add-on and disable it.
Set the add-on title to “Deleted”
Describe that there was a problem with inappropriate rating on the add-on’s page.
Dummy add-on pop-up notifying users of the problem and directing them to the new Add-on.
After 1-2 weeks, completely remove the problematic add-ons page.
And the popup looks like this. I figure it is just fine because I can notify the developers who downloaded the add-ons. If the icon is inappropriate, it will be removed.
to a new feature/option, thus the reviews will be overwhelming positive and will be on top
