Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to parse pipeline definition. Unknown Argument member 'AdditionalModelDataSources' #4994

Open
fulcus opened this issue Jan 15, 2025 · 0 comments
Assignees
Labels
component: pipelines Relates to the SageMaker Pipeline Platform type: bug

Comments

@fulcus
Copy link

fulcus commented Jan 15, 2025

Describe the bug
A clear and concise description of what the bug is.

Setting the additional_model_data_sources parameter of sagemaker.model.Model throws the error:

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreatePipeline operation: Unable to parse pipeline definition. Unknown Argument member 'AdditionalModelDataSources'

This is probably due to the fact that the sdk assumes that CreatePipeline has the request parameter AdditionalModelDataSources that does not exist (as per docs).

additional_model_data_sources: Optional[Dict[str, Any]] = None,

To reproduce

import boto3
from sagemaker.model import Model
from sagemaker.session import Session


Model(
    name="model_name",
    image_uri="763104351884.dkr.ecr.us-east-1.amazonaws.com/autogluon-training:1.1.1-cpu-py311-ubuntu20.04",
    model_data="<s3 uri>",
    role="SageMakerExecutionRole",
    sagemaker_session=Session(boto_session=boto3.Session(region_name="us-east-1")),
    additional_model_data_sources={
        "train_raw_data_uri": ParameterString(name="TrainRawDataUri"),
        "test_raw_data_uri": ParameterString(name="TestRawDataUri"),
        "config": ParameterString(name="ConfigDataUri"),
    },
)

Expected behavior
The info should be saved somewhere in the registered model version, or the parameter should be removed since it doesn't seem to be supported by the CreatePipeline API.

Screenshots or logs

Traceback (most recent call last):
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/.venv/bin/invoke", line 8, in <module>
    sys.exit(program.run())
             ^^^^^^^^^^^^^
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/.venv/lib/python3.11/site-packages/invoke/program.py", line 398, in run
    self.execute()
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/.venv/lib/python3.11/site-packages/invoke/program.py", line 583, in execute
    executor.execute(*self.tasks)
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/.venv/lib/python3.11/site-packages/invoke/executor.py", line 140, in execute
    result = call.task(*args, **call.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/.venv/lib/python3.11/site-packages/invoke/tasks.py", line 138, in __call__
    result = self.body(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/tasks.py", line 15, in pipelines
    pipelines = create_pipeline_definitions(update, execute, local)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/src/amzn_dgcs_sage_maker_pipeline/create_pipelines.py", line 24, in create_pipeline_definitions
    pipeline = _local_pipeline_definition(update, execute)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/src/amzn_dgcs_sage_maker_pipeline/create_pipelines.py", line 60, in _local_pipeline_definition
    pipeline.upsert(
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/.venv/lib/python3.11/site-packages/sagemaker/workflow/pipeline.py", line 297, in upsert
    raise ce
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/.venv/lib/python3.11/site-packages/sagemaker/workflow/pipeline.py", line 292, in upsert
    response = self.create(role_arn, description, tags, parallelism_config)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/.venv/lib/python3.11/site-packages/sagemaker/workflow/pipeline.py", line 169, in create
    return self.sagemaker_session.sagemaker_client.create_pipeline(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/.venv/lib/python3.11/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workplace/ffgonz/DgcsML/src/DgcsSageMakerPipeline/.venv/lib/python3.11/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreatePipeline operation: Unable to parse pipeline definition. Unknown Argument member 'AdditionalModelDataSources'.

System information
A description of your system. Please provide:

  • SageMaker Python SDK version:2.232.2
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): NA
  • Framework version: NA
  • Python version: 3.11
  • CPU or GPU: CPU
  • Custom Docker image (Y/N): Y
@rohangujarathi rohangujarathi added the component: pipelines Relates to the SageMaker Pipeline Platform label Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: pipelines Relates to the SageMaker Pipeline Platform type: bug
Projects
None yet
Development

No branches or pull requests

3 participants