-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Setting spark.app.id to a more intuitive name #124
Conversation
|
||
if aws_creds[2] is not None: | ||
spark_conf['spark.hadoop.fs.s3a.aws.credentials.provider'] = AWS_ENV_CREDENTIALS_PROVIDER | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just for moving the following lines down a bit for better code readability:
# app_name from env is already appended port and time to make it unique
app_name = (spark_opts_from_env or {}).get('spark.app.name')
if not app_name:
# We want to make the app name more unique so that we can search it
# from history server.
app_name = f'{app_base_name}_{ui_port}_{int(time.time())}'
ee699e8
to
dbc8301
Compare
a05259a
to
c0ef079
Compare
# in all places for metric systems: | ||
# - since in the Promehteus metrics endpoint those will be converted to '_' | ||
# - while the 'spark-app-selector' executor pod label will keep the original app id | ||
app_id = re.sub(r'[\.,-]', '_', app_name) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry I missed it earlier.
There is a character limit of 63 for app id and I think 253 for app name. app id needs to be trimmed to make sure it is within the limit.
Secondly, would likely be useful to have app name as: service__instance_timestamp or service__job__id__action_name_timestamp or adhoc/tron jobs.
-
,.
,,
characters to_
for app idTest: test starting a new spark session and call the Spark's API at
http://<driver_ip>:<ui_port>/api/v1/applications
: