Handle Mediocre Beta Kata

[Kacarott]

As of the writing of this post, there are 2141 existing beta kata. However I would guess that the number of these kata which actually have any chance of ever being approved, is less than 200. One of the reasons for this is that the beta system currently only handles very bad kata (with retirement) or good kata (with eventual approval). Any kata which "settles" into a mediocre level of satisfaction is fated to just stay in beta indefinitely.

My suggestion is a way to deal with these mediocre kata which have settled at a low enough satisfaction rate that they have little hope of ever reaching 80%. The measuring metric would be the number of 'Very satisfied' reviews in a row, required to bring the overall satisfaction up to 80%. The higher this number is, the less chance the kata has of ever being approvable, so an appropriate cut off point can be used to automatically retire kata whose chances are too low.

Some examples:
Doubly Linked List currently has 60% satisfaction, out of 10 reviews. It would need exactly 10 positive reviews in a row to bring it to 80% satisfaction, making it barely approvable. But considering its current review history, this seems incredibly unlikely.

Moving element in an array sits higher at a 70% satisfaction rating, which might make it seem better, however it already has 40 reviews. It would require 20 positive reviews in a row to make it barely approvable.

Decimal to binary converter with a very high 78% satisfaction, has however already 269 reviews! Gaining 2% satisfaction to make it approvable, would require another 26 positive reviews in a row!

My suggestion for a final metric would be, any beta with >5 reviews should be automatically retired if the number of positive reviews required in a row to reach 80% is >=10. This would mean a kata with 5 reviews should have >40% satisfaction, a kata with 10 reviews should have >60% satisfaction, with 20 reviews >70% etc. asymptotically approaching 80%.

[kazk]

So, retire beta kata with more than 5 votes if getting n straight up votes is not enough to make it approvable.

I tried some n with the following definition:

# `true` if even getting `n` straight up votes is not enough
# `required` depends on average assessed rank
def hopeless?(n)
  n_votes > 5 && (n_up_votes + 0.5*n_ok_votes + n) / (n_votes + n) < required/100.0
end

Checked 2176 beta kata:

n	retires	% retired
1	816	37.5%
2	757	34.8%
3	707	32.5%
4	659	30.3%
5	618	28.4%
8	501	23.0%
10	427	19.6%
12	347	15.9%
16	249	11.4%
20	157	7.2%
30	70	3.2%
50	27	1.2%
100	11	0.5%
200	4	0.2%
250	1	0.0%

[kazk]

50 straight up votes are not enough for these:

1. National Bath Safety Month by PG1 (75%)
2. All Star Code Challenge #4 by shaddyjr (70%)
3. Holiday I - Temperature in Bali by PG1 (73%)
4. All Star Code Challenge #5 by shaddyjr (76%)
5. Holy cats by aweleshetu (76%)
6. Grasshopper - if/else review by danleavitt0 (75%)
7. SQL Basics: Biggest in Each Column by PG1 (75%)
8. Greetings by sam-karis (74%)
9. No Loops 2 - You only need one - Shortest Code Version by PG1 (74%)
10. Stop worrying! by Samatar26 (72%)
11. All Star Code Challenge #26 by shaddyjr (70%)
12. How bigga ma pizza ? by dinglemouse (74%)
13. Climb Time - Can you send it? by hollyewhite (65%)
14. Is it Golden? by DiegoSalazar (72%)
15. Counter (React) by Catherine1988 (53%)
16. NOP by dinglemouse (64%)
17. Custom hook (React) by Catherine1988 (70%)
18. Beam me up Statey using ReactJS by richkotze (53%)
19. Uncurrying by surtich (66%)
20. Find twice of a number's value by setkyar (59%)
21. Find a number increase by setkyar (57%)
22. Basic Statement Coverage in Unit Testing by drchangliu (56%)
23. Class extension model with mixin ability. by .blue (55%)
24. The Kingdom of Squares by Jari Jarvis (53%)
25. [Code Golf] Sum of Minimums by mubashir1 (54%)
26. Random string by sunboshan (47%)
27. NHA JS1 Variables - Declaration & Assignment by lpaglione (42%)

[kazk]

n = 10 will retire all kata listed on https://github.com/codewars/content-issues/wiki/List-of-Mediocre-Beta-Kata

What do you think?

[Kacarott]

I think that n = 10 would be a good threshold. With a quick scan over the list, I don't see any particular kata standing out as "underrated" or worth saving.

[hobovsky]

I like the general idea, but I am not sure it covers all necessary/useful areas:

• How would the retirement be triggered? Would there be some mechanism introduced, or is it one-off thing, or we ask admins to run a query periodically?
• Do we need to satisfy the hoarder's mindset many of us seem to display? I think one problem with removal of betas is a hope that "it could be fixed one day", because the idea is nice and execution is bad. Or that I might not like it, but others apparently do". After getting unpublished or retired, kata becomes virtually inaccessible. Is it a problem? It is not for me, but I think it could be for some.
• What about approved kata of low satisfaction? Do we want to handle these somehow? It's probably out of scope of this issue, but maybe it would be nice to have some way to handle also approved kata of low satisfaction, with many issues?

[Kacarott]

My intent with starting this discussion was for it to be an automatic process, essentially just an addition to the existing auto-retire thresholds. I think it is more valuable as an automatic process rather than a one-off thing, since in the future it will mean that it is always worthwhile solving and rating beta katas, since it will drive them closer to approval or retirement.

I don't think that we should be trying to satisfy hoarders mindset. The initial batch of retired kata could be kept (for a while) in the list for people's reference, and in the future if people are aware of the new retirement rule, then they can keep track of kata they want to save (via collections or bookmarks)

Your third point I think is outside the scope of this issue, as you said.

[kazk]

How would the retirement be triggered?

Same as existing auto-retire.

Do we need to satisfy the hoarder's mindset many of us seem to display? I think one problem with removal of betas is a hope that "it could be fixed one day", because the idea is nice and execution is bad.

This is why I tried different n. n = 50 is pretty much impossible to get it approved unless we reset the votes.
n = 10 may include some that can be salvaged by fixing issues and removing some outdated votes.

What about approved kata of low satisfaction?

We should start a separate discussion.

[hobovsky]

This is why I tried different n. n = 50 is pretty much impossible to get it approved unless we reset the votes.
n = 10 may include some that can be salvaged by fixing issues and removing some outdated votes.

I think this is the case with lolisa's HAskell kata, which got recycled at some point. But still, even if somewhat strict, I like the threshold of 10. With current amount of betas, I prefer to cut off more rather than few. I am fine with 10.

[hobovsky]

Another potential problem: currently, it's difficult to establish how far from approval/retirement a kata is. Voter's voting power is a hidden stat which impacts satisfaction rating, but is not exposed anywhere. I wanted to propose (or create with a user script) some kind of an indicator which shows that "This kata can be approved after receiving X green votes, and retired after Y red votes" but it's not possible to get the data anywhere.
Would such indicator be useful?

[Kacarott]

Might be nice to have this, perhaps as a popup when hovering over the satisfaction rate?

[ejini6969]

I'm not sure if 10 is the right threshold, maybe abit too strict. I prefer 15, but that's just my opinion. I suggest we not do this auto-retire thing first and glance through the list 1-by-1, starting from the 60% threshold. Some of those kata are worth saving, like myjinxin's thinking & testing series, as well as bubbler's katas. But for those beyond 60%, they can be retired or sent back to draft imo due to being exact / mixture of multiple tasks directly or indirectly.

[hobovsky]

I would propose to add another filter to kata search, to be able to search for retired kata. Maybe additional status, aside of "Approved" and "Beta"?
Potentially coupled with subcriterion of "Retirement reason" (retired by admin, or by reaching low satisfaction level, or by being "hopeless"), just like "Beta status" now, it would be possible for whoever enjoys this to search though retired, bad kata. Adding this to search mechanism has an additional advantage of being able to apply other filters and sorts too: filter by author, sort by satisfaction, rank (if assigned), etc. If someone finds a kata worth redeeming, they can fix it and ask privileged users to restore it.

I'd hope that making retired kata discoverable would possibly fix the problem of hoarding.

[Blind4Basics]

maybe restrict the search to menders/mods/admins?

[ejini6969]

I agree with approach proposed by hobs (considering the fact that most of the katas have issues raised and authors were alr long gone or unwilling to fix them !) And all this should be limited to content maintainers only as there will be abundant of copied solutions here and there if it's available to public! But imo we need more opinions from other PUs so this list should be kept as such until then, when retirement takes place!

[kazk]

I wouldn't want retired kata to be accessible too easily, but will allow menders/mods/admins to list them. I don't want average users to see them because retired kata can't be completed, and it might not even work at all because it hasn't been updated.

We currently have 2106 retired kata.

[hobovsky]

I'm not sure if 10 is the right threshold, maybe abit too strict. I prefer 15, but that's just my opinion.

"10 upvotes away" might seem restrictive, but it doesnt necessarily have to be: note that this formula gives kata which need 10 green votes straight and nothing else, what, I believe, would effectively translate into a need for significantly many many more solutions and voters: if a kata does not meet the threshold already, it's highly probable that some users will vote yellow or red, effectively increasing the requirement.

[kazk]

Added the condition with n = 10, so it should auto-retire when the vote is updated by someone. I can later run a script to retire all beta matching the condition as well.

https://www.codewars.com/kata/retired can be used to search retired kata. This page is only for mods and menders.