Skip to content

chore: add spark 3.5.6 #1142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ All notable changes to this project will be documented in this file.
- kafka: Add `4.0.0` ([#1117])
- Include `.tar.gz` snapshots of the product source code in container images ([#1126])
- airflow: OPA authorizer for Airflow 3.x ([#1127])
- spark-k8s: Add `3.5.6` ([#1142])
- spark-connect-client: Add `3.5.6` ([#1142])
- git-sync: Bump version to 4.4.1 ([#1151]).

### Changed
Expand Down Expand Up @@ -111,6 +113,7 @@ All notable changes to this project will be documented in this file.
- druid: Remove `30.0.0` ([#1110]).
- nifi: Remove `2.2.0` ([#1114]).
- kafka: Remove `3.7.1` and `3.8.0` ([#1117])
- spark-connect-client: Remove `3.5.5` ([#1142])
Copy link
Member

@NickLarsenNZ NickLarsenNZ Jun 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is deprecated, not removed in this release

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, the client it being removed as it is only used in demos.

If it is being used in tests, it will need to stay in for the release.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark connect support is experimental has not been released yet. The tests are performed with the version of our choosing which has been 3.5.5 until now but we update it to 3.5.6.
There is no requirement to keep the connect client for 3.5.5 around, unless we discover major bugs with 3.5.6.


[nifi-iceberg-bundle]: https://github.com/stackabletech/nifi-iceberg-bundle
[#1025]: https://github.com/stackabletech/docker-images/pull/1025
Expand Down Expand Up @@ -165,6 +168,7 @@ All notable changes to this project will be documented in this file.
[#1137]: https://github.com/stackabletech/docker-images/pull/1137
[#1138]: https://github.com/stackabletech/docker-images/pull/1138
[#1139]: https://github.com/stackabletech/docker-images/pull/1139
[#1142]: https://github.com/stackabletech/docker-images/pull/1142
[#1151]: https://github.com/stackabletech/docker-images/pull/1151
[#1152]: https://github.com/stackabletech/docker-images/pull/1152

Expand Down
2 changes: 1 addition & 1 deletion spark-connect-client/versions.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
versions = [
{
"product": "3.5.5",
"product": "3.5.6",
"spark-k8s": "3.5.5",
"java-base": "17",
"python": "3.11",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
From 94ccf32b4d0eb7c3191b4e5a646605e7386c39ff Mon Sep 17 00:00:00 2001
From: Razvan-Daniel Mihai <[email protected]>
Date: Tue, 11 Mar 2025 17:29:39 +0200
Subject: Update CycloneDX plugin

---
dev/make-distribution.sh | 1 -
pom.xml | 7 ++++++-
2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/dev/make-distribution.sh b/dev/make-distribution.sh
index ef7c010e93..0f4c1c74e4 100755
--- a/dev/make-distribution.sh
+++ b/dev/make-distribution.sh
@@ -171,7 +171,6 @@ BUILD_COMMAND=("$MVN" clean package \
-Dmaven.javadoc.skip=true \
-Dmaven.scaladoc.skip=true \
-Dmaven.source.skip \
- -Dcyclonedx.skip=true \
$@)

# Actually build the jar
diff --git a/pom.xml b/pom.xml
index 68e2c422a2..6216ebb08e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -3534,7 +3534,12 @@
<plugin>
<groupId>org.cyclonedx</groupId>
<artifactId>cyclonedx-maven-plugin</artifactId>
- <version>2.7.9</version>
+ <version>2.8.0</version>
+ <configuration>
+ <projectType>application</projectType>
+ <schemaVersion>1.5</schemaVersion>
+ <skipNotDeployed>false</skipNotDeployed>
+ </configuration>
<executions>
<execution>
<phase>package</phase>
1 change: 1 addition & 0 deletions spark-k8s/stackable/patches/3.5.6/patchable.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
base = "303c18c74664f161b9b969ac343784c088b47593"
18 changes: 18 additions & 0 deletions spark-k8s/versions.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,22 @@
"tini": "0.19.0",
"hbase_connector": "1.0.1",
},
{
"product": "3.5.6",
"java-base": "17",
"java-devel": "17",
"python": "3.11",
"hadoop": "3.4.1", # Current Stackable LTS version. Source of the AWS and Azure artifacts to Spark's classpath.
"hbase": "2.6.2", # Current Stackable LTS version. Used to build the HBase connector.
"aws_java_sdk_bundle": "2.24.6", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/3.4.1
"azure_storage": "7.0.1", # https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.4.1
"azure_keyvault_core": "1.0.0", # https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage/7.0.1
"jackson_dataformat_xml": "2.15.2", # https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.13/3.5.2
"stax2_api": "4.2.1", # https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.15.2
"woodstox_core": "6.5.1", # https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.15.2
"vector": "0.46.1",
"jmx_exporter": "1.2.0",
"tini": "0.19.0",
"hbase_connector": "1.0.1",
},
]