How to set Spark configuration in ProcessingStep?

[neilmcguigan]

It's easy to set Spark configuration when you RUN a PySparkProcessor:

processor = PySparkProcessor(
    role=role,
    instance_type="ml.m5.large",
    instance_count=1,
    framework_version="3.0"
)

processor.run(
    submit_app="processor.py",
    arguments=[
        "--output",
        "s3://bucket/prefix/"
    ],
    configuration=[{
        "Classification": "spark-defaults",
        "Properties":{
            "spark.hadoop.mapreduce.fileoutputcommitter.marksuccessfuljobs": "false" # avoid creating _SUCCESS file in output:
        }
    }]
)

However, I do not see a way to do so in a SageMaker pipeline, via

processing_step = ProcessingStep(
    name="process",
    processor=processor,
    code="processor.py",
    job_arguments = [
        "--output",
        "s3://bucket/prefix/"
    ]    
)

How can I do so? TIA

[neilmcguigan]

Figured it out. The answer is to use get_run_args():

# this uploads `configuration` to s3 and converts it to a ProcessingInput:
run_args = processor.get_run_args(
    "processor.py",
    arguments=[
        "--output",
        "s3://bucket/prefix/"
    ],
    configuration=[{
        "Classification": "spark-defaults",
        "Properties":{
            "spark.hadoop.mapreduce.fileoutputcommitter.marksuccessfuljobs": "false" # avoid creating _SUCCESS file in output:
        }
    }]
)

processing_step = ProcessingStep(
    name="process",
    processor=processor,
    code=run_args.code,
    job_arguments = run_args.arguments,
    inputs=run_args.inputs    
)