Don't Revalidate Old Kata Solutions

[Kacarott]

An often heard concern when updating or fixing a kata is "but this will invalidate many/all solutions!". Deliberating on this issue usually slows down the whole process of getting the kata fixed, and even then there is no good outcome. Either:

• The fix is abandoned, leaving the kata with issues
• The fix is implemented, and many valuable solutions are lost
• A hacky workaround is implemented to try and "save" the solutions, resulting in difficult to understand test code

My idea is simple. To eliminate this problem, only solutions which are younger than some threshold (eg. 3 months) are ever revalidated when a kata is updated. All other kata simply stay visible. To make this behaviour more clear, a small alert could be present on all old solutions saying something like "This solution is old and may not pass the kata in its current state". But this alert may not be necessary at all. I would also make an exception for beta kata, which should revalidate all solutions on update/approval.

A simple idea, but the benefits are great. Contributors can freely fix issues without fear of invalidating thousands of solutions. Similarly language versions can be updated without much loss. It's probably also easier for the server, not having to validate as many solutions.

[hobovsky]

As I stated in other discussions, I will also say here for the record:

Your idea assumes that the invalidated solutions are valuable. And I agree that your, or any other, idea would be nice to protect valuable solutions from invalidation.

But still, I've seen only one occurrence of someone giving an example of a valuable solution being invalidated. Everything else are attempts to save worthless solutions to 7 kyus which are never going to be looked back at, are top-voted golfed one-liners with O(n!) complexity, and will be replenished in hundreds right after the kata gets bumped up in the search list.

The most common problem is not with attempts of saving potentially valuable solutions, which I would fully support. The problem is with attempts to save worthless solutions, which drives me crazy. I think the real stress is not at the point you put it.

[Kacarott]

All it takes is to look through any decently old kata and see how many solutions have votes and comments. These solutions wouldn't have these votes and comments if people weren't actively looking through the solutions page to see how other people have solved it. These solutions in question may not be valuable to you, however there is clear evidence that other people have found them worthwhile to look at. The very fact that people worry about invalidating solutions also shows that they think that solutions in general have value.

And regardless, I don't really see what you are arguing here. I don't see any downsides to my idea, besides perhaps some effort in implementation. So even if we were to assume that there are no solutions out there worth saving, and that the worry of collaborators about invalidating solutions is completely irrational, then I still think the idea is worth implementing, if only to remove the irrational concern I just mentioned and making the fixing process easier.

Unless you think my idea has issues I haven't realised?

[Kacarott]

In writing my last response I also realised another reason its worth saving solutions, is that attached to solutions are often discussions. For example users explaining how to achieve the optimal time complexity, or how the elegant one liner is actually terrible in performance, how certain built-ins work, the advantages of certain data types over another, etc etc. If you invalidate these solutions you are essentially deleting these conversations too.

[hobovsky]

Unless you think my idea has issues I haven't realised?

No, absolutely not, it's not what I think. I am not arguing the idea, but I am challenging its premise. I am not sure if it's worth to put real efforts into implementation of an idea which addresses unreasonable concerns.

I think your idea attempts to present a solution to a different problem. Saving old solutions is good where it's worth it, but we already know how to do this in most cases. There are some languages which do not allow this, but we already have backward compatibility recipes for Python, JS, Java, C#. What I still can't understand is why concerns about invalidation of specific kinds of solutions (i.e. interesting ones) impedes fixing kata in general. If your idea converts into kata being fixed by removal of idiotic rounding, changing of unidiomatic interfaces, improvements of consistency between languages, adding tests for edge cases, then hell yeah, I am all for it. But this has a cost, and what I don't understand is why do we even need this idea, and pay this cost, to fix this kind of issues.

[Kacarott]

Do we really know how to save solutions in most cases? I know that in some languages we can save solutions in the specific case that a name needs to be updated, but can we also save them if tests are altered, or language version is changed in a breaking way, etc?

It may have a cost, but we also don't know how much that cost is. I know that for some very old kata, there is already rules to not check every solution because there are too many. For all I know, adding this idea might be as simple as adding one line of code somewhere.

But as further motivation since you said you lacked good examples, consider smaller languages. For example, in Factor there are not many regular solvers, but it is always very interesting to see how everyone else solved a kata. Coming up sometime this year (hopefully) is a big update which will very likely break almost all kata solutions to most Factor translations. Since the regular factor solvers aren't going to go back and re-solve the kata, it means that any person solving in factor after the update will be seeing a completely empty solutions screen.

[Blind4Basics]

sounds good to me.

I'd just move the "This solution is old and may not pass the kata in its current state" part to the docs, maybe. Or just something "less massive"? (a watermark of any kind? Not sure it's actually needed either)

[Kacarott]

Yeah, in my mind its simply a small information-icon which might show the text on hover or something. I wouldnt want something huge and distracting. As said it might not be needed at all.

[hobovsky]

Maybe use something similar to what's already there: there is this "Invalidated" filter on solutions page which toggles between valid and invalid solutions. Maybe change the filter to Valid/Invalid/All, and make "All" the default. This way it would be possible to see all solutions, invalid ones would be marked with a red gutter as they are now, the list could be sorted and filtered as it is now even when showing all solutions (valid and not valid), and maybe the solutions could be even voted on. Visibility of invalidated solutions would not be a concern anymore because it wouldn't be affected by invalidation, and invalidation mechanism wouldn't have to be changed and its exact logic would become irrelevant.

[mauro-1]

There are cases where the old tests are too weak and the poupose of the fix is invalidate wrong solutions.
E.g. https://www.codewars.com/kata/589d1e88e8afb7a85e00004e/solutions/python?filter=all&sort=best_practice&invalids=true

Another idea: the choice between "normal" (keep old solutions) and "revalidating" update in fork/translation editor.

[hobovsky]

This will fail in a scenario when a not-revalidating update (for example, a change of a testing framework or whatever) is followed by a revalidating update (for example, strenghtened tests): it's possible that there would be solutions which are perfectly able to pass the strenghtened tests but will not pass the revalidation because they do not fit the new testing framework.

I still don't understand why a solution being valid or not is such a big deal. To me, the problem seems to be visibility, and not validity itself: to me, everyone seems to be worried by solutions becoming not immediately visible, rather by receiving a red bar. And the problem of visibility would be easily solved by having a filter which shows all solutions, withtout changing the logic of revalidation.

[Kacarott]

The issue is that there are really 2 kinds of invalid solutions: Solutions which are bad/weak/fail edge cases/abuse tests, And then solutions which merely have the wrong name, or wrong number of args, or produce the wrong output format, or use an older version of a language. Ideally, the first group would be hidden by default, but the second group would ideally be shown by default even if it is with a red bar. Since the current system makes no distinction between these, it means you either lose solutions from the second group into essential non-existence (since no one ever looks at the invalid solutions), or you keep them, but also keep all the junk that the filter was designed to keep us from seeing in the first place. This is essentially the point of my idea, to keep these categories separate by allowing (as much as possible) the latter category to stay in the valid kata, keeping the invalidated kata section just for the first kind.