Reduce job history dir length

[joel-wong-aws]

Related to: aws-deadline/deadline-cloud-for-cinema-4d#105

What was the problem/requirement? (What/Why)

When submitting jobs from DCCs which don't have long path support, the job history bundle will not be able to be created if the path is too long due to a long job name.

What was the solution? (How)

Truncated the job history dir for the submitted bundle so that the job history bundle, including asset manifest, can be created.

What is the impact of this change?

Previously failing submissions caused by long paths in the job history dir will succeed for DCCs which don't support long paths.

How was this change tested?

hatch build && hatch shell

Confirmed that paths were truncated when submitting a job with a long name but the job as still able to submit successfully.

Edited my settings.job_history dir to a very long directory and got the expected error

Was this change documented?

No, N/A

Does this PR introduce new dependencies?

• This PR adds one or more new dependency Python packages. I acknowledge I have reviewed the considerations for adding dependencies in DEVELOPMENT.md.
• This PR does not add any new dependencies.

Is this a breaking change?

No

Does this change impact security?

No

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[leongdl]

the path of \manifests\d2b2c3102af5a862db950a2e30255429_input may become longer in the near future. Can we have more reserved characters so the structure does not change? Say 160 or 180 instead of "207" ?

[joel-wong-aws]

We can always shorten this more later if it becomes an issue :)

[leongdl]

Discussion on how to handle long paths?

[epmog]

nit: MAX_PATH is actually 260: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry

D:\some 256-character path string<NUL>

[joel-wong-aws]

While the official MAX_PATH is 260, there may be some DCC constraints that limit this to 256 characters, so I decided to go with that. This also gives us a few characters of wiggle room.

[epmog]

Have we considered what happens if this ends up being 0 or a negative number?

[joel-wong-aws]

Added error handling for this case!

[epmog]

Do we have a stack trace or anything of the experience when being unable to create a history directory? Was it the job bundle history creation that failed? Perhaps for a quick fix, we could just be catching/warning on that.

[joel-wong-aws]

Do we have a stack trace or anything of the experience when being unable to create a history directory? Was it the job bundle history creation that failed? Perhaps for a quick fix, we could just be catching/warning on that.

The error message:

Error occurred: [Errno 2] No such file or directory: 'C:\\Users\\test-user\\.deadline\\job_history\\test-user-gamma-package-build\\2025-01\\2025-01-23-01-Cinema4D-GSG_REDSHIFT_StarterScene_April_2024 sdkjfhsakjdfghhagwefkjhawvefkjhavwfjkhavwefjhawvfjhvasdjkhfvsdjfvajwfjkhwevfjhwevfjhawevfjh\\manifests\\7512c012abaae0a151ca32f96e07ace5_input'

In Cinema 4D, the console has the stack trace

[Errno 2] No such file or directory: 'C:\\Users\\test-user\\.deadline\\job_history\\test-user-gamma-package-build\\2025-01\\2025-01-23-01-Cinema4D-GSG_REDSHIFT_StarterScene_April_2024 sdkjfhsakjdfghhagwefkjhawvefkjhavwfjkhavwefjhawvfjhvasdjkhfvsdjfvajwfjkhwevfjhwevfjhawevfjh\\manifests\\7512c012abaae0a151ca32f96e07ace5_input'
Traceback (most recent call last):
  File "C:\Users\test-user\AppData\Roaming\DeadlineCloudSubmitter\deadline\client\ui\dialogs\submit_job_progress_dialog.py", line 386, in _upload_background_thread
    ).upload_assets(
      ^^^^^^^^^^^^^^
  File "C:\Users\test-user\AppData\Roaming\DeadlineCloudSubmitter\deadline\job_attachments\upload.py", line 1288, in upload_assets
    (partial_manifest_key, asset_manifest_hash) = self.asset_uploader.upload_assets(
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\test-user\AppData\Roaming\DeadlineCloudSubmitter\deadline\job_attachments\upload.py", line 195, in upload_assets
    self._write_local_manifest(
  File "C:\Users\test-user\AppData\Roaming\DeadlineCloudSubmitter\deadline\job_attachments\upload.py", line 253, in _write_local_manifest
    self._write_local_input_manifest(manifest_write_dir, manifest_name, manifest, root_dir_name)
  File "C:\Users\test-user\AppData\Roaming\DeadlineCloudSubmitter\deadline\job_attachments\upload.py", line 274, in _write_local_input_manifest
    with open(local_manifest_file, "w") as file:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\test-user\\.deadline\\job_history\\test-user-gamma-package-build\\2025-01\\2025-01-23-01-Cinema4D-GSG_REDSHIFT_StarterScene_April_2024 sdkjfhsakjdfghhagwefkjhawvefkjhavwfjkhavwefjhawvfjhvasdjkhfvsdjfvajwfjkhwevfjhwevfjhawevfjh\\manifests\\7512c012abaae0a151ca32f96e07ace5_input'

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[epmog]

Sorry for the delay, unfortunately due to the customer experience impact we can't do this change.

A valid windows submission that works with long paths will now fail to submit because we won't create the job history dir (to be used as input to the create job bundle callback for where to create the submission files).

In my mind, there's 2 part of this at play:

1. job bundle history directory right now is a required portion of submission or export
   • in my opinion, it is an internal implementation detail that we must create a history entry to submit, rather than populate it after submission with best effort.
2. using the library in a runtime that doesn't support long paths will have submission issues based off the job bundle directory location + job name length

I think short-term we can make the second issue less likely and do the truncation but only if it would fail otherwise. We can't have previously working setups failing due to this. If we can successfully create the history directory and a file whose path matches the longest path therein, then no truncation should happen.

Longer-term I think we should absolutely remove the truncation and:

1. use a local temp folder to allow submission and then move it to the history folder after submission, and/or
2. leverage \\?\ for this specific case of the internal implementation detail of having a history directory and how we populate it.

Let me know your thoughts