Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JavaScript-runtime] Export ATN transitions with types #4805

Open
wants to merge 4 commits into
base: dev
Choose a base branch
from

Conversation

MatthiasHarzer
Copy link

@MatthiasHarzer MatthiasHarzer commented Mar 24, 2025

The current JS runtime does not export ATN transitions, which makes it hard to walk the ATN from outside.

This PR adds types for the ATN transitions (src/antlr4/transition) as well as exporting them so they can be imported from outside.

closes #4810

@ericvergnaud
Copy link
Contributor

@MatthiasHarzer thanks for this, but this is internal stuff. Exporting it would create an obligation to maintain it. What problem are you trying to solve ?

@MatthiasHarzer
Copy link
Author

Hi there @ericvergnaud,
thanks for your response. I'm sorry to hear that, but let me further explain the issue:

As I mentioned in my issue #4810, I'm trying to build a suggestion system to recommend tokens, that could follow a partial input.

I really like the idea of leveraging the existing ANTLR ATN for this, as ANTLR is already a direct dependency, and I want to keep external dependencies to a minimum. Using the ATN also ensures that any changes made to the grammar (.g4 file) are automatically reflected in the suggestion system, avoiding the need for additional manual adjustments.
While the ATN is already exported and partially typed in the current version, it is not very usable in its current state because its transitions cannot be distinguished without access to the transition classes.

Projects like antlr4-c3 use the ATN to predict following tokens and rules from an ANTLR-grammar and a partial input. However, I hesitate to use such tool as it depends on an alternative ANTLR-generator antlr4ng. My goal is to build on the official ANTLR target and runtime as they appear to be supported the best. Additionaly, JavaScript isn't the only target in my custom language and I strongly prefer using the official ANTLR generator for all targets.

Exporting and typing these classes would not impact the rest of the runtime, but it would enable features like the one I’m trying to build. In my opinion, the benefits of making them available outweigh the maintenance overhead.

I hope this clarifies my reasoning, and I welcome any further questions or feedback.

@kaby76
Copy link
Contributor

kaby76 commented Apr 4, 2025

Another application of the ATN data structures is to enumerate all the ambiguous parse trees at decision states. The code I wrote to do this, in C# and part of a generated driver app for testing grammars in Trash, is used instead of the erroneous one currently in the CSharp runtime. I could not do this without access to the ATN data structures. To do this tree enumeration, you also need the DecisionInfo data structure, aka "profiling", to know which decision states are ambiguous. Unfortunately, that's not implemented in the JS/TS runtime. If profiling and ambiguous parse tree enumeration are not part of the runtime (and they should not), then give us the tools so we can implement them. Once enumerated, Trash has about a half-dozen different ways to display the parse trees, which are much faster and more intuitive than generating .dot files and trying to find the decision state by eyeball. Ambiguity in grammars-v4 is pervasive: out of 363 grammars, 215 are ambiguous.

@ericvergnaud
Copy link
Contributor

@MatthiasHarzer Some time ago I started looking at implementing a code-completion core. The code is here https://github.com/ericvergnaud/antlr4/tree/suggestion-helper. The intent is precisely to provide a built-in tool for suggestions without exhibiting all internals of ANTLR (which create stickiness). It's still early days but maybe it's worth glancing at it ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

JavaScript-Runtime makes it hard to work with the ATN
3 participants