Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

export_workspace: skip writing to local storage #883

Open
jdries opened this issue Sep 26, 2024 · 2 comments · May be fixed by #903 or Open-EO/openeo-python-driver#320
Open

export_workspace: skip writing to local storage #883

jdries opened this issue Sep 26, 2024 · 2 comments · May be fixed by #903 or Open-EO/openeo-python-driver#320
Assignees
@bossie

Comments

@jdries
Copy link
Contributor

jdries commented Sep 26, 2024 •
edited
Loading

When doing save_result => export_workspace,
the data is stored once in default location and once in workspace.
This is problematic for huge processing jobs, because we will have to pay for the storage.
@bossie
Copy link
Collaborator

bossie commented Oct 1, 2024

@jdries What are the implications from the user's point of view? Does this mean that the STAC Collection of batch job results has no items then (also because OpenEO won't be able to serve the assets anymore)?

@jdries
Copy link
Contributor Author

jdries commented Oct 1, 2024

or can we make those items point to the location where we exported the results to?
If complex, the main thing to achieve here is simply that the file is gone, potentially triggered by a feature flag or job option if the impact is too unpredictable.

bossie added a commit that referenced this issue Oct 11, 2024
@bossie
update tests
254a97e 
#883
@bossie bossie linked a pull request Oct 11, 2024 that will close this issue
avoid local storage #903
Draft
@bossie bossie linked a pull request Oct 11, 2024 that will close this issue
Draft
bossie added a commit that referenced this issue Oct 17, 2024
@bossie
refactor: invert gdal flag
e1ad944 
#883
bossie added a commit that referenced this issue Oct 17, 2024
@bossie
add gdalinfo statistics to exported STAC
51fc71f 
#883
bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 17, 2024
@bossie
export_workspace: support remove_original
547dbb1 
Open-EO/openeo-geopyspark-driver#883
@bossie bossie linked a pull request Oct 17, 2024 that will close this issue
Draft
@bossie bossie linked a pull request Oct 17, 2024 that will close this issue
export_workspace: support remove_original Open-EO/openeo-python-driver#320
Draft
bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 17, 2024
@bossie
fix call to shutil#move on Python 3.8
d476753 
tmp_path = PosixPath('/var/lib/jenkins/workspace/enEO_openeo-python-driver_PR-320/pytest-tmp/pytest-of-jenkins/pytest-0/test_disk_workspace_remove_ori1')
remove_original = True

    @pytest.mark.parametrize("remove_original", [False, True])
    def test_disk_workspace_remove_original(tmp_path, remove_original):
        source_directory = tmp_path / "src"
        source_directory.mkdir()
        source_file = source_directory / "file"
        source_file.touch()

        merge = "."
        target_directory = tmp_path / merge

        workspace = DiskWorkspace(root_directory=tmp_path)
>       workspace.import_file(source_file, merge=merge, remove_original=remove_original)

tests/test_workspace.py:43:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
openeo_driver/workspace.py:35: in import_file
    operation(file, target_directory)
/usr/lib64/python3.8/shutil.py:776: in move
    real_dst = os.path.join(dst, _basename(src))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = PosixPath('/var/lib/jenkins/workspace/enEO_openeo-python-driver_PR-320/pytest-tmp/pytest-of-jenkins/pytest-0/test_disk_workspace_remove_ori1/src/file')

    def _basename(path):
        # A basename() variant which first strips the trailing slash, if present.
        # Thus we always get the last component of the path, even for directories.
        sep = os.path.sep + (os.path.altsep or '')
>       return os.path.basename(path.rstrip(sep))
E       AttributeError: 'PosixPath' object has no attribute 'rstrip'

/usr/lib64/python3.8/shutil.py:740: AttributeError

Open-EO/openeo-geopyspark-driver#883
bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 17, 2024
@bossie
remove unused import
83a3d6d 
Open-EO/openeo-geopyspark-driver#883
bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 17, 2024
@bossie
clean up test
ed553fa 
Open-EO/openeo-geopyspark-driver#883
bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Oct 17, 2024
@bossie
clean up test
75984a1 
Open-EO/openeo-geopyspark-driver#883
bossie added a commit that referenced this issue Oct 17, 2024
@bossie
remove files/objects exported to workspace
bbe2ed9 
#883
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bossie bossie

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@bossie @EmileSonneveld @jdries