Add check for jira_ticket parameter being passed via spark-run for adhoc spark jobs

[sidsatyelp]

What this change does

In order to more accurately attribute the costs of feature development to infrastructure costs we are making a change to how we submit adhoc spark jobs. Users will need to pass an additional parameter jira_ticket in spark-args. The value of this parameter should be the top level jira ticket being used to track your project. This will allow us to aggregate the entire project's spark development costs over time.

Will work only after the flag spark.yelp.jira_ticket.enabled is set to true in srv-configs and the new version with these changes are included in paasta and spark-tools.

Testing

# Shouldn't launch because we don't provide jira_ticket
paasta spark-run --aws-profile=dev-cloud-economics --cmd "spark-submit /code/integration_tests/s3.py"
# output: https://fluffy.yelpcorp.com/i/XDdDpJB7vrXLvcqVNkLBxvBdtTFl44RX.html

# Should launch because we provide jira_ticket
paasta spark-run --aws-profile=dev-cloud-economics --jira-ticket=ABC-123 --cmd "spark-submit /code/integration_tests/s3.py"
# output: https://fluffy.yelpcorp.com/i/1VCTvzjk0wK6mbC97KF8JPRqt2WmsXTd.html

jira_ticket label is being added to executors
https://app.cloudzero.com/explorer?activeCostType=real_cost&granularity=hourly&partitions=k8slabel%3Aspark_yelp_com_jira_ticket&dateRange=Last%2024%20Hours&k8slabel%3Aspark_yelp_com_user=sids&showRightFlyout=filters

Tech Spec
Jira Ticket

[chi-yelp]

How about letting user to use a separate argument for the jira ticket?
It's easy to miss (or be copied from other jobs along with other spark configs) when the option is part of the job's --spark-args, and we normally use a prefix spark.yelp.* for Yelp-specific Spark configs for easier tracking.

I.e.

# Ad-hoc CLI
paasta spark-run --aws-profile=dev-core-ml --jira-ticket="ABC-123" --spark-args "..."

# Jupyter
spark = paasta.create_spark_session(
    {
        'spark.executor.cores': '4',
        ...
    },
    profile_name="dev-core-ml",
    jira_ticket="ABC-123",
)

[jhereth]

How about letting user to use a separate argument for the jira ticket? It's easy to miss (or be copied from other jobs along with other spark configs) when the option is part of the job's --spark-args, and we normally use a prefix spark.yelp.* for Yelp-specific Spark configs for easier tracking.

I.e.

# Ad-hoc CLI
paasta spark-run --aws-profile=dev-core-ml --jira-ticket="ABC-123" --spark-args "..."

# Jupyter
spark = paasta.create_spark_session(
    {
        'spark.executor.cores': '4',
        ...
    },
    profile_name="dev-core-ml",
    jira_ticket="ABC-123",
)

This is a cleaner solution I think. Would this work out of the box? Or do users need to update spark-tools to be able to use this additional kwarg?

[chi-yelp]

How about letting user to use a separate argument for the jira ticket? It's easy to miss (or be copied from other jobs along with other spark configs) when the option is part of the job's --spark-args, and we normally use a prefix spark.yelp.* for Yelp-specific Spark configs for easier tracking.
I.e.

# Ad-hoc CLI
paasta spark-run --aws-profile=dev-core-ml --jira-ticket="ABC-123" --spark-args "..."

# Jupyter
spark = paasta.create_spark_session(
    {
        'spark.executor.cores': '4',
        ...
    },
    profile_name="dev-core-ml",
    jira_ticket="ABC-123",
)

This is a cleaner solution I think. Would this work out of the box? Or do users need to update spark-tools to be able to use this additional kwarg?

yeah this will need updating spark_run in paasta and spark-tools, but both approaches require bumping servcie_configuration_lib in paasta and spark-tools anyway

[jhereth]

This is a cleaner solution I think. Would this work out of the box? Or do users need to update spark-tools to be able to use this additional kwarg?

yeah this will need updating spark_run in paasta and spark-tools, but both approaches require bumping servcie_configuration_lib in paasta and spark-tools anyway

Then I'd prefer the cleaner way.

[sidsatyelp]

@chi-yelp and @jhereth Can you please review these changes?

[chi-yelp]

Commented for a nit, lgtm otherwise

[chi-yelp]

nit: can be simplified as follows, and add a comment

Suggested change

	needs_jira_check = os.environ.get('USER', '') not in ['batch', 'TRON', '']
	# Check Jira ticket if the run is not a Tron run
	needs_jira_check = os.environ.get('USER') not in ['batch', 'TRON']