Make the History Server storage configuration shareable/reusable/discoverable

[Jimvin]

Spark jobs should be able to discover the location to store logs in the Spark History Server. Configuring this for each job is fragile and likely to lead to logs not being persisted after the job has finished. Jobs should be able to discover the location, access keys etc. for log storage.

[sbernauer]

Just for the record: This is how you share the the log configuration between two SparkApplications currently:

kind: SparkApplication
metadata:
  name: spark-1
spec:
  logFileDirectory:
    s3:
      prefix: eventlogs/
      bucket: # S3BucketDef
        reference: eventlogs-bucket
---
kind: SparkApplication
metadata:
  name: spark-2
spec:
  logFileDirectory:
    s3:
      prefix: eventlogs/
      bucket: # S3BucketDef
        reference: eventlogs-bucket

IIRC in the ADR we decided against SparkApplication simply linking to SparkHistoryServer objects, because then you have problems such as how the application get's credentials to write to the bucket (and probably others).