Tolerations and node affinity support

[Lemmons]

Taints, tolerations, and node affinities seem to be the way to provide fine-grain scheduling support in a kubernetes cluster. Maybe I'm blind, but I can't seem to find any way to provide pod spec level properties when running spark. The closest I can get is spark.kubernetes.node.selector., but this doesn't provide the level of required control.

I can spin up another dedicated kube cluster to only host spark, but I would prefer to instead just have a dedicated node pool, which adding support for tolerations would do.

[dawnbreaks]

+1
Yes, we need this feature too. Is there a plan to support tolerations?

[rvesse]

NB - Per the README all development work and discussion has moved to the main apache/spark repo

You can probably do this with the new Pod Templates feature which is currently only available in master and is targeted for release with Spark 3.0 sometime later in the year