What feature you would like to be added?

Define an interface to allow customizing Spark application submit mechanism.

Why is this needed?

• Improve performance of the Spark operator when submitting large amount of Spark application at the same time.
• Reduce cpu/memory usage.

Describe the solution you would like

1. Define a new interface (e.g. SparkApplicationSubmitter ):

// SparkApplicationSubmitter is the interface for submitting a SparkApplication.
type SparkApplicationSubmitter interface {
	Submit(ctx context.Context, app *v1beta2.SparkApplication) error
}

2. Move current spark-submit code into a new class SparkSubmitter:

// SparkSubmitter submits a SparkApplication by calling spark-submit.
type SparkSubmitter struct {
}

// SparkSubmitter implements SparkApplicationSubmitter interface.
var _ SparkApplicationSubmitter = &SparkSubmitter{}

// Submit implements SparkApplicationSubmitter interface.
func (*SparkSubmitter) Submit(ctx context.Context, app *v1beta2.SparkApplication) error {
   // TODO: Move code related to spark-submit here
}

3. If one wants to extend the Spark application submit mechanism, for example, use golang to implement spark-submit logic, then he can create a new class which implements the SparkApplicationSubmitter interface.

Describe alternatives you have considered

No response

Additional context

No response

Love this feature?

Give it a 👍 We prioritize the features with most 👍