Fix flaky clipboard on Windows #8361

matthiasblaesing · 2025-03-26T21:54:45Z

It was observed, that copying from NetBeans on Windows sometimes was flaky. Flaky in this case means, that the copy operation was executed in NetBeans, but the content of the system clipboard was not updated.

Based on the discussion in #7051 this is causes by an implementation problem inside the JDK. While an upstream fix is preferred, the problem is a very real problem and an alternative was found in changing the JDK classes at runtime using an javaagent.

Closes: #3962

platform/o.n.agent/src/org/netbeans/agent/NetBeansAgent.java

platform/o.n.bootstrap/launcher/unix/nbexec

platform/o.n.agent/src/org/netbeans/agent/TransferHandlerActionsNonFinalTransformer.java

platform/o.n.agent/src/org/netbeans/agent/NetBeansAgent.java

lahodaj

After an initial round of review:

Some comments inline, adding some general here:

previous attempt to add a Java agent to NetBeans (which, among other things, did roughly what TransferHandlerActionsNonFinalTransformer.java is doing) was shut down, on the principle that agents are too complex. I don't object to the agent here, but I would like to hear what changed in the past 5-6 months so that an agent is OK now, and wasn't 5-6 months ago.
I don't particularly like that this is bound to ASM only. This means the IDE won't run as well on newer JDKs, until ASM is upgraded. Given we have ClassFile API now, I suspect there should be a ClassFile backend for the transformers. Whether there should be an ASM backend is (to me, personally), less important. I.e. I think it is important to run well with minimal maintenance on newer JDKs.

I am requesting changes here - that applies specifically to the license of the jar with the shaded ASM.

lahodaj · 2025-03-27T06:40:16Z

platform/o.n.agent/build.xml

+    <import file="../../nbbuild/templates/projectized.xml"/>
+    <target name="-post-jar">
+        <jar destfile="${cluster}/${module.jar}" update="true">
+            <zipfileset src="../libs.asm/external/asm-commons-9.7.1.jar">


Shading is generally frown upon, although may be more or less necessary here.

But - the LICENSE (and, generally NOTICE) file in the resulting jar that includes multiple libraries need to include licenses (and, when appropriate notices) for all the libraries. I may be missing something here, but this does not seem to include the ASM license in the resulting jar.

I agree, that the shading is ugly, but the alternative would be to add the ASM too the boot classloader and this would mean further fiddling with the command line arguments.

The LICENSE problem was a problem and I added the required notice modification.

lahodaj · 2025-03-27T07:26:32Z

platform/o.n.agent/src/org/netbeans/agent/NetBeansAgent.java

+        return Logger.getLogger(NetBeansAgent.class.getName());
+    }
+
+    public static void premain(String arg, Instrumentation instrumentation) throws IOException, URISyntaxException {


I may be wrong, but one of the things I don't like here is that:
a) the transfomers run permanently, for all classes, forever. The transformers transform only one class, it seems wasteful to keep them running (and, potentially, complaining, see below) forever. Even if the transformers "don't do anything" after transforming the classes. I think I would suggest to either 1) uninstall the transformer after it transforms the class it is interested in, and/or 2) load the proactively class here or soon after start (and uninstall the transformer), so that the transformers run for least amount of classes.
b) the transformers are installed even if disabled. If a transformer is disabled, it should not be installed/added at all.

The overhead is an if and and string comparison. I have some hope, that if this gets as hot as you imply, hotspot will optimize this, so that the unlikely matching path is mostly optimized away.

Anyway, modified the only currently retained transformer to remove itself after it is done and only be registered if they are not disabled.

lahodaj · 2025-03-27T07:32:51Z

platform/o.n.agent/manifest.mf

+OpenIDE-Module-Specification-Version: 1.0
+Premain-Class: org.netbeans.agent.NetBeansAgent
+Can-Redefine-Classes: true
+Can-Retransform-Classes: true


Can-Retransform-Classes means the agent itself is requesting ability to retransform classes. Does this agent really initiate retransforming classes? If not, then I think this ability should be removed from the manifest.

The current state neither redefines, nor retransforms, so removed both.

lahodaj · 2025-03-27T07:36:35Z

platform/o.n.agent/src/org/netbeans/agent/TransferHandlerActionsNonFinalTransformer.java

+        // We assume, that we are loaded early enough before AWT is initialzed,
+        // so we don't expect to need to retransform
+        if (classBeingRedefined != null) {
+            getLog().log(Level.WARNING, "Attempted retransform for: {0} (Ignored)", new Object[]{className});


I don't understand this at all, sorry. What is the point? As I look at the patch, the agent is not initiating retransforming of classes. It does not even check if javax.swing.TransferHandler was already loaded before the transformer was installed.

If some other agent will request some retransformations, then this transformer will start to complain (I think, at least!), but I don't understand why.

This whole if seems weird to me, unnecessary, and maybe harmful.

Dtto for the other transformer.

Agreed (codepath can't be hit as the agent declares itself as non retransforming and the registration with instrumentation declares the same).

lahodaj · 2025-03-27T07:37:20Z

platform/o.n.bootstrap/launcher/unix/nbexec

+then
+    jargs_without_clusters="$jargs -javaagent:${plathome}/agent/org-netbeans-agent.jar";
+else
+    jargs_without_clusters="$jargs";


I like that when the agent is excluded from the application, then nothing happens - definitely a +1 for that.

lahodaj · 2025-03-27T08:11:58Z

platform/o.n.agent/src/org/netbeans/agent/TransferHandlerActionsNonFinalTransformer.java

+                            case "cutAction":
+                            case "copyAction":
+                            case "pasteAction":
+                                return super.visitField(access & (~ Opcodes.ACC_FINAL), name, descriptor, signature, value);


Sorry, I don't like this. Removing the final modifier will at least make sure that deep reflection will work on the field, but it is still adding (new) deep reflection against JDK class, which not very good. And removing the final modifier also feels like a highly suspicious thing to me.

Given we have a transforming agent, is there a reason why not change TransferAction.getClipboard() to do whatever is needed? Or something else, that would be less problematic than using deep reflection on a field from the JDK, and removing the final modifier to make the reflection work.

Removed the Transformer, so does not apply anymore.

platform/o.n.agent/src/org/netbeans/agent/NetBeansAgent.java

neilcsmith-net · 2025-03-27T09:42:32Z

+1 to the Windows launcher changes for JAVA_HOME and PATH. Another change that would be good to consider (which is relevant to this) is allowing relative paths in the conf. eg. the shell launcher sources the conf file so can handle things like ${plathome}. That might make agent registration, as well as arg files, easier to handle.

-0.9 to the agent as is ...

I don't object to the agent here, but I would like to hear what changed in the past 5-6 months so that an agent is OK now, and wasn't 5-6 months ago.

Well, what was discussed was an optional (ideally opt-in), lightweight and temporary solution via patch or agent for the Windows clipboard issue. That optionality has to include in my opinion configuration via the conf file and cli (shortcut) args.

I agree with Jan. This feels far more akin to revisiting the agent discussion from 5-6 months ago. If that discussion needs reopening, then let's reopen it. But bear in mind that given revisions and launcher vote required here, this will probably not land in master until a week before branch (if we stick to the date).

I also share concerns about the dependence on ASM.

-1 to registering NBClipboard by default or without easy override. Its removal fixed as much as it broke. My suggestion if taking the agent route was to bring back as an option for regression issues and testing, not to bring it back permanently and therefore embed the need for an agent in the IDE.

It was suggested this Windows issue only occurs with flavor listeners registered. I would still like anyone who can reproduce the clipboard issue (my Windows machine stubbornly refuses to have the bug!) to check whether the removal of flavor listener registration does stop this issue occurring. Because it might be that we can work around the issue that way for now, which would potentially be a lot simpler!

mbien · 2025-03-27T09:59:37Z

I would also like to see this reduced to the clipboard bug fix. I agreed to not get in the way of temporary agent proposals to resolve the issue until it can be hopefully fixed in the JDK. #7051 (reply in thread)

Regarding ASM: This PR here will log a exception on JDK 25 and NB will continue to launch. I think this is good enough for a temporary agent driven patch. (speaking only for WClipboard patch). I have a ClassFile API version here - but I wrote this mostly as learning experience. This would require a) build on JDK 24 + multi release jar, or b) release agent separately. For temporary solution it is likely a bit too much overhead.

I haven't looked at the code gen bits yet. But it would be good to proof via javap that the generated code is identical to javac emitted code when fed with the patched source file - i think this is probably the fastest way to ensure correctness (see linked repo where I tried that approach)

matthiasblaesing · 2025-03-27T16:44:21Z

I pushed an update the reduces the scope of the agent, fixes issues not caught on the first try and addresses review concerns.

For the general comments:

previous attempt to add a Java agent to NetBeans (which, among other things, did roughly what TransferHandlerActionsNonFinalTransformer.java is doing) was shut down, on the principle that agents are too complex. I don't object to the agent here, but I would like to hear what changed in the past 5-6 months so that an agent is OK now, and wasn't 5-6 months ago.

-0.9 to the agent as is ...

I don't object to the agent here, but I would like to hear what changed in the past 5-6 months so that an agent is OK now, and wasn't 5-6 months ago.

I think I never ruled out an agent, so I can't answer this. What is a reality is, that I did not realize that #7928 and #8169 would lead to this. I learned about copy/paste history from a recent comment by @neilcsmith-net (I think) and thus this lead me down the rabbit hole.

I also don't think, that the agent code is hard, it is the classfile manipulation that is. What makes it further hard is, that this (WClipboard) is not just any code, but code, that is called from native, so it is the uglier/harder to debug side of the JVM.

Well, what was discussed was an optional (ideally opt-in), lightweight and temporary solution via patch or agent for the Windows clipboard issue. That optionality has to include in my opinion configuration via the conf file and cli (shortcut) args.

You can disable the code via config (see the system properties) and remove the module. If you insist, I can add a "--disable-nb-agent" option to the launchers.

I agree with Jan. This feels far more akin to revisiting the agent discussion from 5-6 months ago. If that discussion needs reopening, then let's reopen it. But bear in mind that given revisions and launcher vote required here, this will probably not land in master until a week before branch (if we stick to the date).

As said above, maybe not all problems were on the table at time of discussion.

I don't particularly like that this is bound to ASM only. This means the IDE won't run as well on newer JDKs, until ASM is upgraded. Given we have ClassFile API now, I suspect there should be a ClassFile backend for the transformers. Whether there should be an ASM backend is (to me, personally), less important. I.e. I think it is important to run well with minimal maintenance on newer JDKs.

I also share concerns about the dependence on ASM.

It is ASM or manully coding the class modifications. I'm willing to see if the Class-File API can be sensibly invoked via reflection and implement as corresponding code path, but the current state of discussion does not motivate me to waste energy on this. If we agree to get this in, I'll do it.

-1 to registering NBClipboard by default or without easy override. Its removal fixed as much as it broke. My suggestion if taking the agent route was to bring back as an option for regression issues and testing, not to bring it back permanently and therefore embed the need for an agent in the IDE.

I don't remember where I got that that from, but I think I remember, you asking for it. Anyway, I'm not tied to this, not sure, why the security manager approach was more acceptable or how you intent to solve it differently.

It was suggested this Windows issue only occurs with flavor listeners registered. I would still like anyone who can reproduce the clipboard issue (my Windows machine stubbornly refuses to have the bug!) to check whether the removal of flavor listener registration does stop this issue occurring. Because it might be that we can work around the issue that way for now, which would potentially be a lot simpler!

What do you have in mind? I now have a virtual machine that can reliably reproduce the problem. I still don't know what changed in that machine, but I used it to verify the change (problem visible without agent, problem gone with it).

neilcsmith-net · 2025-03-27T17:33:03Z

Thanks for the detailed responses.

You can disable the code via config (see the system properties) and remove the module. If you insist, I can add a "--disable-nb-agent" option to the launchers.

The system property disables the transformation, not the agent? Module removal requires the IDE location to be modifiable, which is not always the case. I think we need the ability for people to switch this on or off easily in case of issues, either with it or to rule it out. I'm also not sure yet that we want it enabled by default in the cross-platform zip. Launcher updates are something of a PITA, so we should consider how the launcher changes can cover various eventualities.

Ideally in my opinion we'd have something like an --agent option that could perhaps take none, default or a full path as an argument. Path could be useful for end user testing at a later date. Like --userdir, etc. latest specified wins - setting on CLI or shortcut overrides conf.

Default in the launcher should be off in my opinion, and controlled by the default options. Could be other way around. Decision on enabling by default in the zip should happen after we've had some rc testing, but would be done via default options.

-1 to registering NBClipboard by default or without easy override. Its removal fixed as much as it broke ...
I don't remember where I got that that from, but I think I remember, you asking for it. Anyway, I'm not tied to this, not sure, why the security manager approach was more acceptable or how you intent to solve it differently.

Yes, I suggested it as an option for testing, because we know it has side effects. The security manager approach was not more acceptable. When we talked about its removal, there was various discussion about the clipboard. What was interesting is that removing it fixed other things, like copy&paste in the file trees.

What do you have in mind? I now have a virtual machine that can reliably reproduce the problem. I still don't know what changed in that machine, but I used it to verify the change (problem visible without agent, problem gone with it).

When xxDark mentioned the JDK issue he said it was only triggered with registered flavor listeners. We're not using them in many places https://github.com/search?q=repo%3Aapache%2Fnetbeans+flavorlistener&type=code I think registering just in NbClipboard and ExplorerActionsImpl? If commenting out adding them to the system clipboard stops the bug occurring, it's possible we might look at something different? Particularly given NbClipboard is already partially disconnected.

mbien · 2025-03-27T22:39:32Z

I also share concerns about the dependence on ASM.

It is ASM or manully coding the class modifications. I'm willing to see if the Class-File API can be sensibly invoked via reflection >and implement as corresponding code path, but the current state of discussion does not motivate me to waste energy on this. If >we agree to get this in, I'll do it.

IMO: ASM is fine for this usecase since:

I checked using ea JDK 25 and the exception handling successfully disables the agent and allows regular NB startup for testing purposes for example. I am not super worried that we forget to update asm there, but if we wanted we could add a tripwire unit test which will remind us to update ASM once we move the upper bound to JDK 25.
this is meant as temporary patch, so we hopefully don't have to do this for all eternity - it could be still converted to classfile api if needed in future - easiest way to achieve this would be to bump the build requirement for the release build to JDK 24+ and put it into a multi release jar
please don't use it via reflection, this would be a lot of reflection - bytecode manipulation is already unreadable enough without it

Bytecode transformations which are meant to be maintained permanently, should use the classfile API IMO. Beside it being a quite nice API, it being part of the JDK solves all version related compatibility problems. I hope we can avoid those agents though.

mbien · 2025-04-06T17:56:28Z

Ideally in my opinion we'd have something like an --agent option that could perhaps take none, default or a full path as an argument. Path could be useful for end user testing at a later date. Like --userdir, etc. latest specified wins - setting on CLI or shortcut overrides conf.

As mentioned on the mailing list, I wouldn't touch the launchers at all and put the -J-javaagent... into the config file so that it can be easily changed or removed without having to release the launcher again.

The windows launcher improvements could be split into a separate PR - not sure when the branching happens, getting them in for NB 26 might be close.

neilcsmith-net · 2025-04-06T18:37:40Z

@mbien it was a deliberate suggestion because I don't think there's any other way to handle a path relative to the base dir here? If I've missed something and I'm wrong with that, then I agree with you. Otherwise, we'd need a launcher update for this as well as JDK env vars. Not much time to get one in before branch now! Assuming on time, will be middle of next week.

matthiasblaesing · 2025-04-06T18:43:45Z

I don't know where you want to go, so this is what happened to the launcher in the meantime:

https://github.com/matthiasblaesing/netbeans/commits/enhance_launcher/

mbien · 2025-04-06T18:59:54Z

@matthiasblaesing sounds good to me. (typo in commit msg "substitution")

neilcsmith-net · 2025-04-08T15:13:37Z

@matthiasblaesing if that basedir option can handle spaces in paths in JDK options without getting in to quoting hell, that would be a great addition for this and other things! I'd considered something similar myself a while back, but was concerned about handling for paths with whitespace.

EDIT: the other concern might be alternative path separators on different OS though?

matthiasblaesing · 2025-04-08T20:59:49Z

Pushed another update:

launcher code got another update to support replacement of more variables and thus now allows to refer to the NB installation from the netbeans_default_options
o.n.agent moved from platform to nb cluster. Without integration it is useless in the platform.
The modules o.n.agent is now just a plain module, that "holds" the additional file nb/netbeans-javaagent.jar. The latter is the agent.
with the new option in the launcher, the agent can be referenced from configuration and is integrated that way
rebased onto current master
tested path with spaces on Linux and Windows by running the IDE from the directory netbeans with spaces

ide/o.n.agent/src-agent/org/netbeans/agent/NetBeansAgent.java

neilcsmith-net · 2025-04-09T10:02:17Z

Many thanks @matthiasblaesing !

o.n.agent moved from platform to nb cluster. Without integration it is useless in the platform.

I do not understand this one. Firstly, it seems to be in ide and not nb? And as mentioned on the other (launcher) PR, the platform Windows launcher should hopefully be picking up the same change already. Unix launcher may need a tweak? I think this needs to be possible to include in the platform as well, so could probably remain in the platform cluster?

Putting do not merge on this as and until we've been through the launcher release process.

matthiasblaesing · 2025-04-09T15:44:52Z

o.n.agent moved from platform to nb cluster. Without integration it is useless in the platform.

I do not understand this one. Firstly, it seems to be in ide and not nb?

Yeah, had to move from nb to ide as the feature on demand tests failed. The core theme is: The agent is IDE only.

And as mentioned on the other (launcher) PR, the platform Windows launcher should hopefully be picking up the same change already. Unix launcher may need a tweak? I think this needs to be possible to include in the platform as well, so could probably remain in the platform cluster?

The platform launcher will not get the agent. The platform launcher has no configuration file, so there is no basis to add it. I had it in the platform, because I envision a serious integration of instrumentation into NB. That was a hard reject, so I went down the minimal version.

neilcsmith-net · 2025-04-09T16:15:39Z

The platform launcher has no configuration file, so there is no basis to add it.

Huh? The platform uses the same etc/app.conf as the IDE. And given how the Windows launcher is extended, should be working there already. This has to work across platform and IDE.

matthiasblaesing · 2025-04-09T19:07:45Z

@lahodaj could you please recheck?

@ALL for comments about the launcher changes, please direct comments to #8408, comments about the agent approach should be here.

It was observed, that clipboard on windows sometimes reacts flaky. Investigation hints that missing synchronization in the AWT implementation/integration. Testers reported, that the changes implemented here: - the change handling from the system clipboard is dispatched into the event loop and - the change handling is modified to be called synchronized It is expected, that fixing this upstream in the JDK will take longer and so runtime patching must be considered. Runtime patching in this case is accomplished using the javaagent mechanism. The IDE launcher is modified to load an agent, that will intercept loading classes and modify WClipboard while it is being loaded. The transformer is written, so that it will only transform the target class and will act transparantly for all other classes.

matthiasblaesing · 2025-04-14T21:32:29Z

Launcher release 3-282bbc032bcd is out and integrated into this PR. Released version migrated to archive.apache.org and integration commit was update to that location. For the agent the license was moved to the right location/name META-INF/LICENSE.

This survived a smoketest on Windows and Linux with spaces in directory name.

mbien

agent integration code looks good to me. Haven't really looked at the emitted bytecode (or platform integration).

with JDK-8353950 filed and the fix available this all will be hopefully only needed temporary

neilcsmith-net · 2025-04-16T08:19:33Z

Removed the do not merge label that was pending launcher release. This needs to go in now to be included in the branch for NB26. It should be in rc1 for greatest testing.

There's a minor formatting issue in clusters file, and I would still prefer this was in the platform cluster, but not reason not to merge now. Can be addressed later if decided.

mbien · 2025-04-16T08:44:56Z

Haven't really looked at the emitted bytecode

this all will be hopefully only needed temporary

to elaborate a bit: since we essentially need static bytecode injection for the purpose of patching (to swap out a method essentially), verification would be conceptually easy, since all it would have to proof is that the emitted bytecode matches javac bytecode. I tested this here (mail). But since this is hopefully only needed until the JDK is patched and the agent would also disable itself on failure (+ option to edit the launch config), I think this is good to go. (thats why I approved - there is also no time left for changes)

lahodaj

Looks OK to me.

LouisCollet · 2025-04-22T05:07:18Z

Seems fixed in nb 26-rc1, after hours work ...
thanks to everybody !

matthiasblaesing added Platform ci:all-tests ci:dev-build labels Mar 26, 2025

matthiasblaesing added this to the NB26 milestone Mar 26, 2025

mbien reviewed Mar 27, 2025

View reviewed changes

lahodaj requested changes Mar 27, 2025

View reviewed changes

mbien reviewed Mar 27, 2025

View reviewed changes

platform/o.n.agent/src/org/netbeans/agent/NetBeansAgent.java Outdated Show resolved Hide resolved

matthiasblaesing force-pushed the clipboard-agent branch from 16b9958 to 3ada442 Compare March 27, 2025 12:14

matthiasblaesing removed the ci:all-tests label Mar 27, 2025

matthiasblaesing force-pushed the clipboard-agent branch from 3ada442 to a4161ca Compare March 27, 2025 15:35

matthiasblaesing force-pushed the clipboard-agent branch 3 times, most recently from 1af6b9c to 551bb04 Compare April 8, 2025 20:55

matthiasblaesing changed the title ~~Clipboard / Launcher fixes~~ Fix flaky clipboard on Windows Apr 8, 2025

xxDark reviewed Apr 8, 2025

View reviewed changes

ide/o.n.agent/src-agent/org/netbeans/agent/NetBeansAgent.java Outdated Show resolved Hide resolved

neilcsmith-net added the do not merge label Apr 9, 2025

neilcsmith-net mentioned this pull request Apr 10, 2025

Enhance launcher #8408

Merged

matthiasblaesing force-pushed the clipboard-agent branch from 551bb04 to 8413480 Compare April 10, 2025 21:57

Use updates launchers (version 3-282bbc032bcd)

2e97532

matthiasblaesing force-pushed the clipboard-agent branch from 8413480 to 4783f6d Compare April 14, 2025 21:05

matthiasblaesing force-pushed the clipboard-agent branch from 4783f6d to 9b6f8a0 Compare April 14, 2025 21:16

mbien approved these changes Apr 16, 2025

View reviewed changes

neilcsmith-net removed the do not merge label Apr 16, 2025

neilcsmith-net requested a review from lahodaj April 16, 2025 08:19

lahodaj approved these changes Apr 16, 2025

View reviewed changes

mbien added the os:windows label Apr 16, 2025

ebarboni merged commit 27b43e7 into apache:master Apr 16, 2025
32 checks passed

matthiasblaesing deleted the clipboard-agent branch April 21, 2025 12:41

This comment was marked as outdated.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix flaky clipboard on Windows #8361

Fix flaky clipboard on Windows #8361

matthiasblaesing commented Mar 26, 2025 •

edited

Loading

lahodaj left a comment

lahodaj Mar 27, 2025

matthiasblaesing Mar 27, 2025

lahodaj Mar 27, 2025

matthiasblaesing Mar 27, 2025

lahodaj Mar 27, 2025

matthiasblaesing Mar 27, 2025

lahodaj Mar 27, 2025

matthiasblaesing Mar 27, 2025

lahodaj Mar 27, 2025

lahodaj Mar 27, 2025

matthiasblaesing Mar 27, 2025

neilcsmith-net commented Mar 27, 2025

mbien commented Mar 27, 2025

matthiasblaesing commented Mar 27, 2025

neilcsmith-net commented Mar 27, 2025

mbien commented Mar 27, 2025

mbien commented Apr 6, 2025

neilcsmith-net commented Apr 6, 2025

matthiasblaesing commented Apr 6, 2025

mbien commented Apr 6, 2025

neilcsmith-net commented Apr 8, 2025 •

edited

Loading

matthiasblaesing commented Apr 8, 2025

neilcsmith-net commented Apr 9, 2025

matthiasblaesing commented Apr 9, 2025

neilcsmith-net commented Apr 9, 2025

matthiasblaesing commented Apr 9, 2025

matthiasblaesing commented Apr 14, 2025

mbien left a comment •

edited

Loading

neilcsmith-net commented Apr 16, 2025

mbien commented Apr 16, 2025

lahodaj left a comment

LouisCollet commented Apr 22, 2025

This comment was marked as outdated.

Fix flaky clipboard on Windows #8361

Fix flaky clipboard on Windows #8361

Conversation

matthiasblaesing commented Mar 26, 2025 • edited Loading

lahodaj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neilcsmith-net commented Mar 27, 2025

mbien commented Mar 27, 2025

matthiasblaesing commented Mar 27, 2025

neilcsmith-net commented Mar 27, 2025

mbien commented Mar 27, 2025

mbien commented Apr 6, 2025

neilcsmith-net commented Apr 6, 2025

matthiasblaesing commented Apr 6, 2025

mbien commented Apr 6, 2025

neilcsmith-net commented Apr 8, 2025 • edited Loading

matthiasblaesing commented Apr 8, 2025

neilcsmith-net commented Apr 9, 2025

matthiasblaesing commented Apr 9, 2025

neilcsmith-net commented Apr 9, 2025

matthiasblaesing commented Apr 9, 2025

matthiasblaesing commented Apr 14, 2025

mbien left a comment • edited Loading

Choose a reason for hiding this comment

neilcsmith-net commented Apr 16, 2025

mbien commented Apr 16, 2025

lahodaj left a comment

Choose a reason for hiding this comment

LouisCollet commented Apr 22, 2025

This comment was marked as outdated.

matthiasblaesing commented Mar 26, 2025 •

edited

Loading

neilcsmith-net commented Apr 8, 2025 •

edited

Loading

mbien left a comment •

edited

Loading