[SPARK-51733][BUILD] make-distribution.sh supports sbt
#50529
Conversation
| | grep -v "INFO"\ | ||
| | grep -v "WARNING"\ | ||
| | tail -n 1) | ||
| SPARK_HIVE=$("$MVN" help:evaluate -Dexpression=project.activeProfiles -pl sql/hive $@ \ |
|
I believe this is an excellent job, but what I'm curious about is whether it's feasible to convert it into an sbt-only script, or under what circumstances it could be transformed into one. Supporting both tools at the same time always makes us inclined to "align" their outputs. |
|
also cc @cloud-fan ,although we may not be able to use sbt to make-distribution when version 4.0 is released. |
@LuciferYang I think this depends on whether the community wants to migrate to a single SBT building tool, or just keep SBT as a faster alternative for dev only. For the former, we must make SBT support packaging and publishing first(may take a few weeks), and ensure the SBT outputs as the same as Maven(may take a few months), then Maven can be dropped after finishing a release using SBT. For the latter, we just leave it, and improve when something affects the daily developing circles |
|
LGTM |
What changes were proposed in this pull request?
Make
make-distribution.shsupports sbt, developers could switch the building tool from Maven to SBT by using--sbt-enabled.Note the packaging outputs are NOT exactly the same(e.g. shading rules have some differences), which should be improved gradually later.
Why are the changes needed?
Reduce gaps between SBT and Maven. And SBT is generally much faster than Maven, so the change would save much time for developers' daily development for cases that require packaging and testing on a real cluster.
Does this PR introduce any user-facing change?
No, it's a dev-only change.
How was this patch tested?
Manually tested on M1 Max
Maven
SBT
Was this patch authored or co-authored using generative AI tooling?
No.