Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Master ci py312 new #5072

Open
wants to merge 54 commits into
base: master-ci-py312-pr-test
Choose a base branch
from
Open

Master ci py312 new #5072

wants to merge 54 commits into from

Conversation

mollyheamazon
Copy link

Issue #, if available:

Description of changes:

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)
  • If adding any dependency in requirements.txt files, I have spell checked and ensured they exist in PyPi

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@mollyheamazon
Test py312 ci support
4aa6c7c
@mollyheamazon
Test py312 ci support
280424d
@mollyheamazon
first commit
1374184
@mollyheamazon
test to point to personal stack
49de844
@mollyheamazon
changed tox.ini
696cb47
@mollyheamazon
Revert "changed tox.ini"
5d35bd0 
This reverts commit 696cb47.
@mollyheamazon
Revert "Revert "changed tox.ini""
2ab95fb 
This reverts commit 5d35bd0.
@mollyheamazon
Revert "Revert "Revert "changed tox.ini"""
dc2fcd5 
This reverts commit 2ab95fb.
@mollyheamazon
Revert "Revert "changed tox.ini""
f143ca7 
This reverts commit 5d35bd0.
@mollyheamazon
Revert "changed tox.ini"
31d7478 
This reverts commit 696cb47.
@mollyheamazon
Revert "test to point to personal stack"
f78febe 
This reverts commit 49de844.
@mollyheamazon
Revert "first commit"
7669550 
This reverts commit 1374184.
@mollyheamazon
add pyproject.toml
598447f
@mollyheamazon
Merge branch 'master-ci-py312-pr-test' into master-ci-py312
1f0f608
@mollyheamazon
bump numpy version
332389c
@mollyheamazon
numpy version change
041c739
@mollyheamazon
add py312
88fbe0c
@mollyheamazon
upgrade pip version
5c20eee
@mollyheamazon
add setuptools wheel to tox.ini
3feadc5
@mollyheamazon
add pyyaml version constraint, remove py312 from docstring because th…
a27527d 
…ere is no py312 image yet
@mollyheamazon
update pyyaml version constraint
340ab3b
@mollyheamazon
update pyyaml version constraint
5bfbb55
@mollyheamazon
deprecate py38
5084e6f
@mollyheamazon
bump scipy
f679f2c
@mollyheamazon
bump tensorflow and tensorboard
6655760
@mollyheamazon
bump dill
4116094
@mollyheamazon
bump apache-airflow to ensure dill and greenlet version
1a23c2a
@mollyheamazon
remove constraint for apache-airflow
9b9eb32
@mollyheamazon
remove constraint for apache-airflow
b5da915
@mollyheamazon
bump torch version
e0caf76
benieric
benieric reviewed Mar 12, 2025
View reviewed changes
pintaoz-aws and others added 18 commits March 13, 2025 17:10
@pintaoz-aws @mollyheamazon
Fix key error in _send_metrics() (aws#5068)
8af122b 
Co-authored-by: pintaoz <[email protected]>
@keshav-chandak @mollyheamazon
fix: Added check for the presence of model package group before creat…
c6c6b64 
…ing one (aws#5063)

Co-authored-by: Keshav Chandak <[email protected]>
@pintaoz-aws @mollyheamazon
Use sagemaker session's s3_resource in download_folder (aws#5064)
09f0285 
Co-authored-by: pintaoz <[email protected]>
@pintaoz-aws @mollyheamazon
Fix error when there is no session to call _create_model_request() (a…
604fae7 
…ws#5062)

* Fix error when there is no session to call _create_model_request()

* Fix codestyle

---------

Co-authored-by: pintaoz <[email protected]>
@pintaoz-aws @mollyheamazon
Ensure Model.is_repack() returns a boolean (aws#5060)
7c29b96 
* Ensure Model.is_repack() returns a boolean

* update test

---------

Co-authored-by: pintaoz <[email protected]>
@benieric @mollyheamazon
feat: Allow ModelTrainer to accept hyperparameters file (aws#5059)
ed2c7e7 
* Allow ModelTrainer to accept hyperparameter file and create Hyperparameter class

* pylint

* Detect hyperparameters from contents rather than file extension

* pylint

* change: add integs

* change: add integs

* change: remove custom hyperparameter tooling

* Add tests for hp contracts

* change: add unit tests and remove unreachable condition

* fix integs

* doc check fix

* fix tests

* fix tox.ini

* add unit test
@Narrohag @sagemaker-bot
@benieric @varunmoris @gwang111 @sage-maker @rsareddy0329 @brunopistone @mollyheamazon
feature: support training for JumpStart model references as part of C…
3f1d2de 
…urated Hub Phase 2 (aws#5070)

* change: update image_uri_configs  01-27-2025 06:18:13 PST

* fix: skip TF tests for unsupported versions (aws#5007)

* fix: skip TF tests for unsupported versions

* flake8

* change: update image_uri_configs  01-29-2025 06:18:08 PST

* chore: add new images for HF TGI (aws#5005)

* feat: add pytorch-tgi-inference 2.4.0

* add tgi 3.0.1 image

* skip faulty test

* formatting

* formatting

* add hf pytorch training 4.46

* update version alias

* add py311 to training version

* update tests with pyversion 311

* formatting

---------

Co-authored-by: Erick Benitez-Ramos <[email protected]>

* feat: use jumpstart deployment config image as default optimization image (aws#4992)

Co-authored-by: Erick Benitez-Ramos <[email protected]>

* prepare release v2.238.0

* update development version to v2.238.1.dev0

* Fix ssh host policy (aws#4966)

* Fix ssh host policy

* Filter policy by algo-

* Add docstring

* Fix pylint

* Fix docstyle summary

* Unit test

* Fix unit test

* Change to unit test

* Fix unit tests

* Test comment out flaky tests

* Readd the flaky tests

* Remove flaky asserts

* Remove flaky asserts

---------

Co-authored-by: Erick Benitez-Ramos <[email protected]>

* change: Allow telemetry only in supported regions (aws#5009)

* change: Allow telemetry only in supported regions

* change: Allow telemetry only in supported regions

* change: Allow telemetry only in supported regions

* change: Allow telemetry only in supported regions

* change: Allow telemetry only in supported regions

---------

Co-authored-by: Roja Reddy Sareddy <[email protected]>

* mpirun protocol - distributed training with @Remote decorator (aws#4998)

* implemented multi-node distribution with @Remote function

* completed unit tests

* added distributed training with CPU and torchrun

* backwards compatibility nproc_per_node

* fixing code: permissions for non-root users, integration tests

* fixed docstyle

* refactor nproc_per_node for backwards compatibility

* refactor nproc_per_node for backwards compatibility

* pylint fix, newlines

* added unit tests for bootstrap_environment remote

* added  mpirun protocol for distributed training with @Remote decorator

* aligned mpi_utils_remote.py to mpi_utils.py for estimator

* updated docstring for sagemaker sdk doc

---------

Co-authored-by: Erick Benitez-Ramos <[email protected]>

* feat: Add support for deepseek recipes (aws#5011)

* feat: Add support for deeepseek recipes

* pylint

* add unit test

* feat: [JumpStart] Add access configs and training instance type variants artifact uri handling for Curated Hub Phase 2 training integration (aws#1653)

* Add access config to training input for Curated Hub Training Integration

* Add support to retrieve instance specific training artifact keys

* Fix some typos and naming issues

* Fix more typos

* fix formatting issues with black

* modify access config logic so accept_eula is passed into fit

* update black formatting

* Add more unit tests for passing access configs

* fix style errors

* fix for failing integ test

* fix styles and integ test error

* skip blocking integ test

* fix formatting

* remove env vars when access configs are being used

* fix docstyle issue

* update usage of access configs, remove conversion of training artifact key to uri

* fix styling issues

* fix styling issues

* fix unit tests

* fix adding hubaccessconfig only if hubcontentarn exists

* move logic to JumpStartEstimator from Job

* Fix styling issues

* Remove unused code

* fix styling issues

* fix unit test failure

* fix some formatting, add comments

* remove typing for estimator in get_access_configs function

* fix circular import dependency

* fix styling issues

---------

Co-authored-by: Erick Benitez-Ramos <[email protected]>

* Always add code channel, regardless of network isolation (aws#1657)

* fix formatting issue

* fix formatting issue

* fix formatting issue

* fix tensorflow file

---------

Co-authored-by: sagemaker-bot <[email protected]>
Co-authored-by: Erick Benitez-Ramos <[email protected]>
Co-authored-by: varunmoris <[email protected]>
Co-authored-by: Gary Wang <[email protected]>
Co-authored-by: ci <ci>
Co-authored-by: parknate@ <[email protected]>
Co-authored-by: rsareddy0329 <[email protected]>
Co-authored-by: Roja Reddy Sareddy <[email protected]>
Co-authored-by: Bruno Pistone <[email protected]>
@benieric @pintaoz-aws
@keshav-chandak @mollyheamazon
feat: Make DistributedConfig Extensible (aws#5039)
cb2f1b2 
* feat: Make DistributedConfig Extensible

* pylint

* Include none types when creating config jsons for safer reference

* fix: update test to account for changes

* format

* Add integ test

* pylint

* prepare release v2.240.0

* update development version to v2.240.1.dev0

* Fix key error in _send_metrics() (aws#5068)

Co-authored-by: pintaoz <[email protected]>

* fix: Added check for the presence of model package group before creating one (aws#5063)

Co-authored-by: Keshav Chandak <[email protected]>

* Use sagemaker session's s3_resource in download_folder (aws#5064)

Co-authored-by: pintaoz <[email protected]>

* remove union

* fix merge artifact

* Change dir path to distributed_drivers

* update paths

---------

Co-authored-by: ci <ci>
Co-authored-by: pintaoz-aws <[email protected]>
Co-authored-by: pintaoz <[email protected]>
Co-authored-by: Keshav Chandak <[email protected]>
Co-authored-by: Keshav Chandak <[email protected]>
@pintaoz-aws @mollyheamazon
Skip tests with deprecated instance type (aws#5077)
f186104 
Co-authored-by: pintaoz <[email protected]>
@mollyheamazon
prepare release v2.241.0
1df4f38
@mollyheamazon
update development version to v2.241.1.dev0
7b9013b
@rohangujarathi @mollyheamazon
pipeline definition function doc update (aws#5074)
23674fe 
Co-authored-by: Rohan Gujarathi <[email protected]>
@Narrohag @mollyheamazon
feat: add integ tests for training JumpStart models in private hub (a…
e266baa 
…ws#5076)

* feat: add integ tests for training JumpStart models in private hub

* fixed formatting

* remove unused imports

* fix unused imports

* fix unit test failure and fix bug around versioning

* fix formatting

* fix unit tests

* fix model_uri usage issue

* fix some formatting

* separate private hub setup code

* add try catch block

* fix flake8 issue so except clause is not bare

* black formatting
@Julfried @mollyheamazon
fix: resolve infinite loop in _find_config on Windows systems (aws#4970)
3717e4d 
* fix: resolve Windows path handling in _find_config

* Replace Path.match("/") with Path.anchor comparison
* Fix infinite loop in _studio.py path traversal

* test: Add tests for the new root path exploration

* Fix formatting style

* Fixed line to long

* Fix docstyle by running black manually

* Fix testcase with \\ when running on non-windows machines

* Fix formatting style

* cleanup unused import
@sagemaker-bot @mollyheamazon
change: update image_uri_configs 03-11-2025 07:18:09 PST
11aac41
@nargokul @mollyheamazon
Fixing Pytorch training python version in tests (aws#5084)
b872f3e 
* Fixing Pytorch training python version in tests

* Updating Inference test handling
@bencrabtree @nargokul @mollyheamazon
remove s3 output location requirement from hub class init (aws#5081)
5e8e894 
* remove s3 output location requirement from hub class init

* fix integ test hub

* lint

* fix test

---------

Co-authored-by: Gokul Anantha Narayanan <[email protected]>
@rrrkharse @nargokul @mollyheamazon
fix: Prevent RunContext overlap between test_run tests (aws#5083)
a7458b9 
Co-authored-by: Gokul Anantha Narayanan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@benieric benieric benieric left review comments

@pintaoz-aws pintaoz-aws pintaoz-aws left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.