Omniscape fails on specific row / column: "median of empty array is undefined" #122
Comments
|
Sorry for the delay in getting back to you on this. Is this still giving you problems? It does make sense that the timing of the error being thrown is random since the order of the moving window solves is randomized for more accurate ETA estimates in the progress bar. If possible, would you mind sharing your input files (and the associate ini file)? I'll try to take a look in the coming couple of weeks. |
|
@vlandau, and my apologies for taking even longer to get back. Yes, this is still giving me issues, across several different input files. Here's the input data and ini file: https://drive.google.com/drive/folders/1kC4FVzw2HjlXVbLLot4a6LU0EPu-E-7s?usp=sharing. |
|
By the way, here's the problem cells from two failed runs. The value of all of these cells are NA. |
|
Hi guys, I'm having the exact same error launching a Europe-wide Omniscape scan. The error appeared after adding a conditional layer in the .ini file. I've already checked that the conditional layer has the same size and resolution as the resistance layer. I tried to change some parameters (radius and block size) but the error keeps appearing. Omniscape failed on the moving window centered on row 3323 column 2228 Stacktrace: |
|
My best guess is at the column/row of one of your inputs, the resulting slice/chunks of the array is just empty (all NaN values). Would you be able to clip out a section (square, with side lengths equal to 2×radius + 1) of each of your input rasters centered on the failing row/column and share it with me? |
|
Hi @vlandau and thanks for replying, you're probably right. I share the clipped input rasters with your options. However, if this is the problem, I don't know why using only the resistance raster doesn't cause problems, while this error only occurs when using a second layer as a condition.
|
|
Hi there! I'm a PhD student and I'm building a tool to help developers automatically track down where NaNs come from in computations, and I'd like to try it out on the problem you're having here to see if I can help. Would someone mind showing me how to fire it all off to reproduce the problems that you're having here? I think I have the |
|
Hi @ashton314! If you already have .ini and .tiff file, I think you just need to donwload Omniscape on Julia and then run the .ini file. You can get information on this page https://docs.circuitscape.org/Omniscape.jl/stable/usage/#Running-Omniscape |
|
@vlandau I also thought there could be NaNs on the failing pixels, but after investigating the failures I found no NaNs. Does it make sense that identical jobs fail at different locations/durations? Sometimes the job will near completion (>90%), but others will fail early--all failing on different pixels. Unlike @Daviser95, some jobs will fail whether or not I have conditional criteria (it's not very predictable). Here's an output from the last job I ran without conditional criteria: |
|
It does make sense that it would occur at different times because the order in which the windows are solved is random (this helps the ETA on the progress bar to be more accurate). I assume in your case there are multiple moving window locations that will cause failure, but which one you run into first is random due to the random solve order. |
|
I regularly get the same issue as @slamander and @Daviser95. Omniscape will fail on specific row / columns. These are not No Data or Nan pixels however. I have clipped out a small area centered on these pixels and run OS on the subset and it finishes successfully. What are the potential reasons for this type of error? |
|
@geomenke this is a different error than the empty array error mentioned above (but I see now one of the stack traces had an AssertionError too), coming directly from Circuitscape, but fortunately, we just made a patch release to Circuitscape that may fix that issue. Would you update to Circuitscape 5.13.2 (you'll need Julia 1.9) and try again? |
|
@vlandau I am already at Julia 1.9.2 so what is the best way to update to Circuitscape 5.13.2? Do I upgrade Omniscape? Can that be done via the command line? |
|
Oh great! To update, in a Julia terminal, enter the package interface by typing the |
|
It did not specifically list Circuitscape as being updated, just things like updating registry etc. |
|
In the same package prompt/interface you should be able to run |
|
I see that I have an outdated and restricted by compatibility contraints version of Omniscape and that Circuitscape did not update after running update. Running update Omniscape did not update Omniscape. Do I need to run |
|
You shouldn't need to do that. There are some solutions in this thread that might work for you: https://discourse.julialang.org/t/how-to-force-an-update/73071/12 |
|
I updated to Circuitscape v5.13.2 and Omniscape v0.6.1, but it failed again with a similar stacktrace as before: Any recommendations? |
|
@slamander or @vlandau - we are also running into this issue using conditional files to incorporate climate connectivity into our models. We've exported all of our input layers at the same resolution and CRS but we're running into the same error with the latest version of Omniscape. Was this issue ever resolved? |
|
@ptfreeman-csp hmm, no I'm not sure why this is happening, but I guess either something is wrong with the input, or Omniscape's processing of the input. Do you have an exact column and row that it failed on? Taking a look around that area/coordinate of all of the inputs might shed some light about what could be going on. |
|
I think we figured out what the issue was. I had too many threads going and
it was burning through the memory of our VM and unable to actually complete
the solving calculations. We ultimately had to reduce our moving window and
block sizes and reduce the total number of threads to stay under the
necessary memory limit. Updating to the latest version of Julia and
Omniscape also seemed to help.
…On Mon, Aug 26, 2024 at 5:57 PM Vincent Landau ***@***.***> wrote:
@ptfreeman-csp <https://github.com/ptfreeman-csp> hmm, no I'm not sure
why this is happening, but I guess either something is wrong with the
input, or Omniscape's processing of the input. Do you have an exact column
and row that it failed on? Taking a look around that area/coordinate of all
of the inputs might shed some light about what could be going on.
—
Reply to this email directly, view it on GitHub
<#122 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AUE56Z47SM34O5JY37E4JELZTPFGXAVCNFSM6AAAAAAQY5S6T2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGM3DONRSGQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Also - for whatever reason we had to only allow the values of the
conditions to be rounded to the nearest tenth. Anything that had two
decimal points in the conditions refused to run but as soon as we knocked
it down to a single decimal point then Omniscape decided to cooperate.
…On Mon, Aug 26, 2024 at 6:10 PM Patrick Freeman ***@***.***> wrote:
I think we figured out what the issue was. I had too many threads going
and it was burning through the memory of our VM and unable to actually
complete the solving calculations. We ultimately had to reduce our moving
window and block sizes and reduce the total number of threads to stay under
the necessary memory limit. Updating to the latest version of Julia and
Omniscape also seemed to help.
On Mon, Aug 26, 2024 at 5:57 PM Vincent Landau ***@***.***>
wrote:
> @ptfreeman-csp <https://github.com/ptfreeman-csp> hmm, no I'm not sure
> why this is happening, but I guess either something is wrong with the
> input, or Omniscape's processing of the input. Do you have an exact column
> and row that it failed on? Taking a look around that area/coordinate of all
> of the inputs might shed some light about what could be going on.
>
> —
> Reply to this email directly, view it on GitHub
> <#122 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AUE56Z47SM34O5JY37E4JELZTPFGXAVCNFSM6AAAAAAQY5S6T2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGM3DONRSGQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
|
Oh that second point is very strange. Glad it's working now though! |
Cross-posting from gitter: there have two mentions of Omniscape failing on a specific row / column.
In my case, I receive this error:
ERROR: ArgumentError: median of an empty array is undefined, Float64[]. The run will fail at different stages--sometimes quite far into a run, other times immediately--but always at the same row-column combination. I've tried to diagnose the error by changing the raster inputs or adjusting model parameters (e.g., solver, block size, r_cutoff [for source = resistance], and running conditional and non-conditional criteria).Here's my error print-out. Let me know if additional information is needed.
The text was updated successfully, but these errors were encountered: