Skip to content

Conversation

ninsbl
Copy link
Member

@ninsbl ninsbl commented Aug 4, 2025

This PR follows up a post on Discourse:
https://discourse.osgeo.org/t/r-out-gdal-performance-with-known-data-type-and-range/148619

The code changes make r.out.gdal skip data range and nodata checks if the f-flag (force export) as well as the nodata, and data type option are given by the user. In that case the assumption is the user has made a deliberate choice about this parameters. Skipping checks cuts export time almost by half...

In a workflow I have to regularly export data to multi-band GeoTiff (COG actually) where I know the data range and the suitable GDAL data type across all bands to export in advance. Even though I use the -f flag and set the nodata option, a full data type and nodata check is performed on every band. To me that feels like unnecessary processing and seems inconsistent with the documentation of the -f flag.

Any feedback is welcome...

@ninsbl ninsbl added this to the 8.5.0 milestone Aug 4, 2025
@ninsbl ninsbl added raster Related to raster data processing C Related code is in C labels Aug 4, 2025
@github-actions github-actions bot added the module label Aug 4, 2025
ninsbl and others added 2 commits August 5, 2025 08:20
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@ninsbl
Copy link
Member Author

ninsbl commented Aug 5, 2025

I see that r.out.gdal lacks a test. Will add one. [Done.]

@ninsbl ninsbl requested review from metzm, pesekon2 and wenzeslaus August 6, 2025 21:10
@github-actions github-actions bot added Python Related code is in Python tests Related to Test Suite labels Aug 6, 2025
@ninsbl
Copy link
Member Author

ninsbl commented Aug 8, 2025

Another question regarding the introduction of pytest:
I notice, many test include code for the creation of simple_vector_map fixues. Would not it be preferable to collect such fixures in a pytest fixures module and reuse existing fixures, where appropriate?

@github-actions github-actions bot added HTML Related code is in HTML docs labels Aug 8, 2025
@wenzeslaus
Copy link
Member

Another question regarding the introduction of pytest: I notice, many test include code for the creation of simple_vector_map fixues. Would not it be preferable to collect such fixures in a pytest fixures module and reuse existing fixures, where appropriate?

Definitely yes, I just didn't make any steps towards that like deciding where to put them in the library, but grass.tests, grass.fixtures, or grass.test_fixtures seems like a good start.

@echoix
Copy link
Member

echoix commented Aug 20, 2025

Another question regarding the introduction of pytest: I notice, many test include code for the creation of simple_vector_map fixues. Would not it be preferable to collect such fixures in a pytest fixures module and reuse existing fixures, where appropriate?

Definitely yes, I just didn't make any steps towards that like deciding where to put them in the library, but grass.tests, grass.fixtures, or grass.test_fixtures seems like a good start.

That's what the conftest.py files do. They apply to all subfolders when placed at one level. (So, there could be one in the vector/ folder, in the vector/v.Xxxx.yy folder etc).

We just can't have one in the root of the repo, as autoconf overwrites or deletes them (they delete conf* files during configure). Both autotools and pytest won't budge on their positions. Since we don't have a src/ folder, we can't have a conftest file for pytest tests for the whole repo.

@wenzeslaus
Copy link
Member

...we can't have a conftest file for pytest tests for the whole repo.

This is becoming offtopic here, but we also have grass-addons repo where the folder-level conftest.py files from the main repo would not work. Anyway, we can definitely add a file to vector/.

Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this in general.

Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for missing this one earlier. I checked my "Awaiting review from you" list and this one didn't have "review requested" so it was not on the list.

At this point, I'm confused about the changes. Is the PR title and description still relevant or is this just about the tests and the verbosity/warnings?

@github-actions github-actions bot added the markdown Related to markdown, markdown files label Oct 1, 2025
@ninsbl
Copy link
Member Author

ninsbl commented Oct 1, 2025

At this point, I'm confused about the changes. Is the PR title and description still relevant or is this just about the tests and the verbosity/warnings?

The main point of the PR is (still) to allow skipping nodata and data type checks to speed up export when users have made deliberate choices. For larger imagery groups that can save quite some time....

@ninsbl ninsbl requested a review from wenzeslaus October 5, 2025 12:09
@ninsbl ninsbl enabled auto-merge (squash) October 6, 2025 10:23
Copy link
Member

@wenzeslaus wenzeslaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for bringing this back to my attention. It looks great.

@ninsbl ninsbl merged commit 74505e2 into OSGeo:main Oct 7, 2025
30 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C Related code is in C docs HTML Related code is in HTML markdown Related to markdown, markdown files module Python Related code is in Python raster Related to raster data processing tests Related to Test Suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants