You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Spark jobs should be able to discover the location to store logs in the Spark History Server. Configuring this for each job is fragile and likely to lead to logs not being persisted after the job has finished. Jobs should be able to discover the location, access keys etc. for log storage.
The text was updated successfully, but these errors were encountered:
IIRC in the ADR we decided against SparkApplication simply linking to SparkHistoryServer objects, because then you have problems such as how the application get's credentials to write to the bucket (and probably others).
Spark jobs should be able to discover the location to store logs in the Spark History Server. Configuring this for each job is fragile and likely to lead to logs not being persisted after the job has finished. Jobs should be able to discover the location, access keys etc. for log storage.
The text was updated successfully, but these errors were encountered: