build: switch to Java 17 for all modules

[nielspardon]

BREAKING CHANGE: this might break consumers running older Java versions

We've been seeing occasional compilation errors where the Gradle toolchain JDK auto detection picked up a Java 11 JDK which does not work with some of the newer syntax being used especially in unit tests.

The behavior was a bit random, sometimes it picked up Java 17 and the build went through, other times it picked up Java 11 and failed to compile.

I would propose to switch all modules to Java 17. This made the compilation more reliable for me locally.

I had to configure one JVM arg for the Spark tests to run successfully since something in those tests tries to access a protected module.

[vbarua]

(not a Java build expert)

My understanding based on

• Downgrade core to java 8 #95
• Downgrade core to java 8 via Jabel #97

is that we explicilty want to support Java 8 for Spark. If we want to change that, we should check with the broader community.

That being said, I've also run into annoying build issues locally because of this, especially with Jabel (#195) and have considered ripping that out entirely to see if that helps the issue.

[nielspardon]

yeah, I tried to dig into that and found the following page from Cloudera saying that Java 17 is supported since Spark 3.3:

https://community.cloudera.com/t5/Community-Articles/Spark-and-Java-versions-Supportability-Matrix/ta-p/383669

I consulted my colleague @andrew-coleman who has been working on the Spark mappings for us and he told me he's only running with Java 17.

And I found this line in substrait-spark/build.gradle which makes be believe we are already compiling Scala to the Java 17 release level:

substrait-java/spark/build.gradle.kts

Line 79 in 7a689e9

scalaCompileOptions.additionalParameters = listOf("-release:17")

[nielspardon]

I can not find any dependency on substrait-java in the Apache Gluten project. it rather looks like they have their own implementation of the Java classes:
https://github.com/apache/incubator-gluten/blob/main/gluten-substrait/pom.xml
https://github.com/apache/incubator-gluten/tree/main/gluten-substrait/src/main/java/org/apache/gluten/substrait

[EpsilonPrime]

Gluten has their own fork of the Substrait protos. Ideally we can unfork their efforts at some point but they aren't willing to put in any work towards that end despite being an Apache project.