Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

C stack usage is too close to the limit when loading a LAScatalog #6

Open
Jean-Romain opened this issue Feb 22, 2024 · 0 comments
Open
Assignees
Labels
bug Something isn't working devel Development branch multithreading Parallel processing

Comments

@Jean-Romain
Copy link
Collaborator

Jean-Romain commented Feb 22, 2024

In the multithreading development branch there is one unit test that is failling for very weird reasons. If a LAScatalog from lidR is loaded it crashes the session with Erreur : C stack usage is too close to the limit. This typically comes from conccurent call to R's C API but here no call to R's C API is used.

library(lasR)

set_lasr_strategy(concurrent_files(2))

LASfile <- system.file("extdata", "Example.las", package="lasR")
las = lidR::readLAScatalog(LASfile)

f <- system.file("extdata", "bcts/", package="lasR")
f = list.files(f, full.names = TRUE, pattern = "\\.laz")
f = f[1:2]

read = reader(f)
bound = hulls()
pipeline = read+bound
pipeline
processor(pipeline, verbose = T)
# > Erreur : C stack usage  17652210908 is too close to the limit

Removing las = lidR::readLAScatalog(LASfile) it works.

After investigations, it fails because lidR::is.overlapping() is called internally. Using a modified version that does not call lidR::is.overlapping() it works. After more investigation and using a modified version without is.overlapping() it can still fail using

las = lidR::readLAScatalog(LASfile)
sfdf = las@data

But

las = lidR::readLAScatalog(LASfile)
las@data

does work !!

After even more investigations the error is called into the vector writer by polygon.IsValid(). Removing this line it works despite it has 0 relationship with a LAScatalog not even used.

if (!polygon.IsValid())

I was also able to reproduce loading testthat but it was inconsistent.

@Jean-Romain Jean-Romain self-assigned this Feb 22, 2024
@Jean-Romain Jean-Romain added multithreading Parallel processing bug Something isn't working labels Feb 22, 2024
@Jean-Romain Jean-Romain added the devel Development branch label Feb 27, 2024
@Jean-Romain Jean-Romain changed the title C stack usage is too close to the limit with multithreading when loading a LAScatalog C stack usage is too close to the limit when loading a LAScatalog Feb 27, 2024
@Jean-Romain Jean-Romain reopened this Mar 4, 2024
Jean-Romain added a commit that referenced this issue Mar 5, 2024
Jean-Romain added a commit that referenced this issue Mar 5, 2024
Jean-Romain added a commit that referenced this issue Mar 22, 2024
* encapsulation and const

* Separate implementation progress bar

* multi file parallelism

* add clone method

* Create raster file before to process the point cloud

* Create vector file before to process the point cloud

* shared pointer to gdaldataset

* Thread safe error handling

* mutlithread raster

* connected algorithms stored in a map

* A lot of things are working...

* Fix clone of LASfilter

* precompute nmetrics

* A lot of change

* Critical aggregate

* parallel noise

* Thread safe print

* thread safe stages

* hread safe progress

* --

* Callback merge

* writelas

* virtual is_parallelizable

* Thread safe copy constructor

* Fix

* finalize multi-thread

* Documentation

* passes test removed in #6

* Change user API

* Thread safe progress bar

* error context

* print error context

* omp set max active level

* Thread safe call to R C API

* Fully thread safe callback

* Ready for nested parallism

* Fix data race

* is_parallized

* change multithreading api + nested

* Fix unit tests strategy applied

* Fix callback merging

* Temporarily ignore test

* Order preserving multi-threading

* Temporary workaround for #6

* --

* Fix nmetrics evaluation

* Get rid of useless warning

* set_parallel_strategy

* Benchmarks

* Benchmarks

* Multi-threading article.

* Add clone method

* OpenMP stuff

* Note about macos

* New benchmarks

* Update benchmarks

* man

* Update benchmarks

* Remove messages

* vendor json parser

* coverage

* v0.4.0

* Rinterface.h is missing on windows

* test multithreading with gh-actions

* R_CStackLimit + tests

* Import R_CStackLimit from DLL for windows

Test

* Try with STRICT_R_HEADERS

* Remove windows.h

* fix test on mac ?

* Skip test on mac OS

* Fixed conditionnal jump on unitialized value

* dump

* Wrap query in critical section

* Dynamic scheduling

* Fix #8

* Order preserving output file

* remove order preserving code for static scheduling

* Fix ordering

* Order callback

* No critical section

* set_exec_options

* skip test if no omp support

* remove print

* Fix no omp support

* -

* Fix verbose

* new
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working devel Development branch multithreading Parallel processing
Projects
None yet
Development

No branches or pull requests

1 participant