Enabled path normalization when enable_best_effort_determinism set to…

[praneetnadella]

… True for cloudpickle

This change fixes a source of non-deterministic serialization in the Beam Python SDK by ensuring that file paths associated with code objects (co_filename) are normalized before being pickled by cloudpickle.

This PR implements path normalization by tying it to the existing enable_best_effort_determinism flag in cloudpickle_pickler.py. When this flag is enabled, a filepath_interceptor is used to replace the absolute path with a deterministic, relative path.

Tested:

• Added test_code_object_path_is_normalized, a new unit test that specifically verifies that a function's co_filename is correctly normalized when the enable_best_effort_determinism flag is used.
• Updated test_best_effort_determinism_not_implemented (renamed to test_best_effort_determinism_is_partially_supported) to check for the new, more informative log message that is emitted when the flag is enabled.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[gemini-code-assist]

Summary of Changes

Hello @praneetnadella, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a crucial enhancement to the Apache Beam Python SDK by enabling path normalization for code objects during cloudpickle serialization. By ensuring that file paths (co_filename) are consistently represented as relative paths when the enable_best_effort_determinism flag is active, this change effectively eliminates a significant source of non-deterministic behavior in the serialization process, leading to more reliable and reproducible Beam pipelines.

Highlights

• Path Normalization for Deterministic Pickling: Enabled path normalization for co_filename in cloudpickle when enable_best_effort_determinism is set to True, resolving a source of non-deterministic serialization in the Beam Python SDK.
• Implementation Detail: The change utilizes a filepath_interceptor to replace absolute file paths with deterministic relative paths during the pickling process.
• Test Coverage: A new unit test, test_code_object_path_is_normalized, was added to specifically verify the correct normalization of function co_filename.
• Updated Logging and Test: The existing test test_best_effort_determinism_not_implemented was renamed to test_best_effort_determinism_is_partially_supported and updated to check for a new, more informative log message indicating partial support for determinism.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[praneetnadella]

R: @claudevdm

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[claudevdm]

Can you define this in cloudpickle_pickler_test instead of creating a new file?

[claudevdm]

I'm a bit confused about the difference between these options.

Is enable_best_effort_determinism meant to include the stable identifier change too? Or is that not part of best effort determinisn?

[claudevdm]

Can we rather explicitly define a BEST_EFFORT_DETERMINISM config at the top of this file and use that? Its easier to see exact settings BEST_EFFORT_DETERMINISM requires and no copying configs required