Implement integration with hdfs

[adwk67]

As a user of the spark-k8s-operator I want to be able to upload job dependencies to a hdfs path

• include namenode in CRD, making this conditional
• include necessary jars in the job image, so that the are available for each executor

[zaultooz]

We were able to enable Spark to access the HDFS cluster in our setup by creating two configmaps and mouting them into the driver and executors:

One configmap containing a simple script that Spark executes on startup for loading env:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "spark-map-process.name" . }}-spark-env
data:
  spark-env.sh: |
    #!/bin/bash
    export HADOOP_CONF_DIR=/etc/hadoop/conf

The configmap is mounted to /stackable/spark/conf in the Spark pods.

Inside the HADOOP_CONF_DIR we then mount the Stackable HDFS discovery configmap:

volumes:
    - name: hadoop-env
      configMap:
        name: example-hdfs-cluster-name
...
#In each driver and executor create the volume mount
volumeMounts:
  - name: hadoop-env
    mountPath: /etc/hadoop/conf

Full example can be found here: https://github.com/gbif/stackable/tree/master/helm/charts/spark-map-process