Add support for TableFunction [WITH OFFSET|ORDINALITY] #2219

davyboyhayes · 2025-04-07T17:49:10Z

This PR fixes #2218, by adding a new withClause to a TableFunction, which allows for support of SQL-99's UNNEST(...) WITH ORDINALITY or GoogleSQL's UNNEST(...) WITH OFFSET.

This PR fixes JSQLParser#2218, by adding a new `withClause` to a TableFunction, which allows for support of SQL-99's `UNNEST(...) WITH ORDINALITY` or GoogleSQL's `UNNEST(...) WITH OFFSET`. Currently, this is a feeler PR, as I'm getting a test failure, as the output of the parsing differs in my tests: ``` org.opentest4j.AssertionFailedError: Output from Deparser does not match. ==> Expected :select*from unnest(array[1,2,3])with offset as t(a,b) Actual :select*from unnest(array[1,2,3])as t(a,b) <Click to see difference> at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197) at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182) at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1156) at [email protected]/net.sf.jsqlparser.test.TestUtils.assertStatementCanBeDeparsedAs(TestUtils.java:126) at [email protected]/net.sf.jsqlparser.test.TestUtils.assertSqlCanBeParsedAndDeparsed(TestUtils.java:99) at [email protected]/net.sf.jsqlparser.test.TestUtils.assertSqlCanBeParsedAndDeparsed(TestUtils.java:85) at [email protected]/net.sf.jsqlparser.statement.select.TableFunctionTest.testTableFunctionWithSupportedWithClauses(TableFunctionTest.java:60) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ```

…ithClause where present

manticore-projects

Good stuff and thank you for your work and contribution.
Please change the token definition and then I will merge.

manticore-projects · 2025-04-07T22:04:39Z

src/main/jjtree/net/sf/jsqlparser/parser/JSqlParserCC.jjt

@@ -545,6 +545,8 @@ TOKEN: /* SQL Keywords. prefixed with K_ to avoid name clashes */
 |   <K_WHERE:"WHERE">
 |   <K_WINDOW:"WINDOW">
 |   <K_WITH:"WITH">
+|   <K_WITH_OFFSET:"WITH OFFSET">


We don't need the compound token. Just WITH and ORDINALITY/ The goal must be to define as few tokens as possible.

manticore-projects · 2025-04-07T22:10:09Z

src/main/jjtree/net/sf/jsqlparser/parser/JSqlParserCC.jjt

 }
 {
    [ prefix = <K_LATERAL> ]
    function=Function()
+    [ withClause = <K_WITH_OFFSET> | withClause = <K_WITH_ORDINALITY> ]


[ <K_WITH> ( withClause = <K_OFFSET> | withClause = <K_ORDINALITY> ) ]

I've tried changing to this, but there is a new test failing:
KeywordsTest.testRelObjectNameWithoutValue(String)

Keyword ORDINALITY net.sf.jsqlparser.JSQLParserException: net.sf.jsqlparser.parser.ParseException: Encountered unexpected token: "SELECT" <K_SELECT> at line 1, column 1. Was expecting one of: "WITH" at [email protected]/net.sf.jsqlparser.parser.CCJSqlParserUtil.parseStatement(CCJSqlParserUtil.java:352) at [email protected]/net.sf.jsqlparser.parser.CCJSqlParserUtil.parse(CCJSqlParserUtil.java:125) at [email protected]/net.sf.jsqlparser.parser.CCJSqlParserUtil.parse(CCJSqlParserUtil.java:91) at [email protected]/net.sf.jsqlparser.test.TestUtils.assertSqlCanBeParsedAndDeparsed(TestUtils.java:98) at [email protected]/net.sf.jsqlparser.test.TestUtils.assertSqlCanBeParsedAndDeparsed(TestUtils.java:85) at [email protected]/net.sf.jsqlparser.statement.KeywordsTest.testRelObjectNameWithoutValue(KeywordsTest.java:53)

Found the issue, I needed to add ORDINALITY to the restricted SQL2016 keywords

I've tried changing to this, but there is a new test failing:
KeywordsTest.testRelObjectNameWithoutValue(String)

Please see here for an explanation: https://manticore-projects.com/JSQLParser/contribution.html#manage-reserved-keywords

I don't think, we need to reserve those keywords but only whitelist them properly by running gradle updateKeywords.

I've made those changes, and pushed up. Thank you

manticore-projects · 2025-04-07T22:11:33Z

src/test/java/net/sf/jsqlparser/statement/select/TableFunctionTest.java

@@ -47,4 +49,14 @@ void testTableFunctionWithNamedParameterWhereNameIsOuterKeyword() throws JSQLPar
                        "     lateral flatten(input => i.DATA_VALUE:Friends, outer => true) AS f1;";
        TestUtils.assertSqlCanBeParsedAndDeparsed(sqlStr, true);
    }
+
+    @ParameterizedTest


…s, adding ORDINALITY keyword.

davyboyhayes · 2025-04-08T08:10:57Z

Ready for re-review.

…Utils, and updating pushing the result of `gradle updateKeywords`

manticore-projects · 2025-04-08T09:19:55Z

Thank you a lot for your work and contribution. Merged.

David Hayes added 3 commits April 7, 2025 18:48

rename clause -> withClause

78dc700

Resolving test failure, and modifying the SelectDeParser to add the w…

bf1892e

…ithClause where present

davyboyhayes marked this pull request as ready for review April 7, 2025 18:08

manticore-projects requested changes Apr 7, 2025

View reviewed changes

Addressing PR Review comments. Using existing WITH and OFFSET keyword…

6549bcb

…s, adding ORDINALITY keyword.

davyboyhayes requested a review from manticore-projects April 8, 2025 08:10

Removing ORDINALITY from the reserved keywords list in ParserKeywords…

a066a96

…Utils, and updating pushing the result of `gradle updateKeywords`

manticore-projects approved these changes Apr 8, 2025

View reviewed changes

manticore-projects merged commit e5fa004 into JSQLParser:master Apr 8, 2025
3 checks passed

davyboyhayes deleted the fix/2218 branch April 8, 2025 09:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for TableFunction [WITH OFFSET|ORDINALITY] #2219

Add support for TableFunction [WITH OFFSET|ORDINALITY] #2219

davyboyhayes commented Apr 7, 2025 •

edited

Loading

manticore-projects left a comment

manticore-projects Apr 7, 2025

manticore-projects Apr 7, 2025

davyboyhayes Apr 8, 2025

davyboyhayes Apr 8, 2025

manticore-projects Apr 8, 2025

davyboyhayes Apr 8, 2025

manticore-projects Apr 7, 2025

davyboyhayes commented Apr 8, 2025

manticore-projects commented Apr 8, 2025

Add support for TableFunction [WITH OFFSET|ORDINALITY] #2219

Add support for TableFunction [WITH OFFSET|ORDINALITY] #2219

Conversation

davyboyhayes commented Apr 7, 2025 • edited Loading

manticore-projects left a comment

Choose a reason for hiding this comment

manticore-projects Apr 7, 2025

Choose a reason for hiding this comment

manticore-projects Apr 7, 2025

Choose a reason for hiding this comment

davyboyhayes Apr 8, 2025

Choose a reason for hiding this comment

davyboyhayes Apr 8, 2025

Choose a reason for hiding this comment

manticore-projects Apr 8, 2025

Choose a reason for hiding this comment

davyboyhayes Apr 8, 2025

Choose a reason for hiding this comment

manticore-projects Apr 7, 2025

Choose a reason for hiding this comment

davyboyhayes commented Apr 8, 2025

manticore-projects commented Apr 8, 2025

davyboyhayes commented Apr 7, 2025 •

edited

Loading