Building ISCE3 0.4.0 from source code w/ CUDA capability

[seongsujeong]

Hi, @vbrancat , @rtburns-jpl , and @gmgunter ,

Following the previous posting regarding ISCE3 conda package (Issue #7 ), here I would like to report the issue that I have experienced from building the software from source code (for CUDA capability).

I am using ISCE3 source code from public GitHub in branch "develop"

I have tried building the software on two linux-based systems as below:

Docker
Environment: Docker container from the image "nvidia/cuda:latest"
kernel: 5.10.47-linuxkit
gcc version 9.3.0
nvcc version 11.0.194
Host OS: MacOS ver 11.6 (Big Sur)
Docker version: 20.10.8, build 3967b7d

AWS EC2
kernel: 5.4.0-1045-aws
Instance type: p3.2xlarge
nvcc version: 11.5.50
gcc version 9.3.0
AMI: Ubuntu 20.04

What I did and saw:

• I installed the dependency using conda as in the instruction page below:
  
https://isce-framework.github.io/isce3/install_linux.html



• cmake error message has helped me to identify additional packages not listed on the instruction page.
  Few examples are eigen, pyre, gmock, etc.

• I had to use version-down few packages to make the building attempt successful.
  Few of them are hdf5 (to 1.10.6), pybind11 (to 2.6.2), and pyre (to 1.9.9).

• ctest: Bunch of tests were failed as below:

The following tests FAILED:
	126 - test.python.pybind..config (Failed)
	127 - test.python.pybind.core.attitude (Failed)
	128 - test.python.pybind.core.basis (Failed)
	129 - test.python.pybind.core.constants (Failed)
	130 - test.python.pybind.core.datetime_ (Failed)
	131 - test.python.pybind.core.ellipsoid (Failed)
	132 - test.python.pybind.core.euler (Failed)
	133 - test.python.pybind.core.interp1d (Failed)
	134 - test.python.pybind.core.kernels (Failed)
	135 - test.python.pybind.core.linspace (Failed)
	136 - test.python.pybind.core.LUT1d (Failed)
	137 - test.python.pybind.core.LUT2d (Failed)
	138 - test.python.pybind.core.orbit (Failed)
	139 - test.python.pybind.core.projections (Failed)
	140 - test.python.pybind.core.quaternion (Failed)
	141 - test.python.pybind.core.statevector (Failed)
	142 - test.python.pybind.core.timedelta (Failed)
	143 - test.python.pybind.focus.backproject (Failed)
	144 - test.python.pybind.focus.chirp (Failed)
	145 - test.python.pybind.focus.presum (Failed)
	146 - test.python.pybind.focus.rangecomp (Failed)
	147 - test.python.pybind.geocode.geocodeSlc (Failed)
	148 - test.python.pybind.geocode.geocodeCov (Failed)
	149 - test.python.pybind.geocode.radar_grid_cube (Failed)
	150 - test.python.pybind.geometry.bbox (Failed)
	151 - test.python.pybind.geometry.dem (Failed)
	152 - test.python.pybind.geometry.geo2rdr (Failed)
	153 - test.python.pybind.geometry.rdr2geo (Failed)
	154 - test.python.pybind.geometry.rtc (Failed)
	155 - test.python.pybind.geometry.metadata_cubes (Failed)
	156 - test.python.pybind.image.resamp (Failed)
	157 - test.python.pybind.io.gdal.dataset (Failed)
	158 - test.python.pybind.io.gdal.raster (Failed)
	159 - test.python.pybind.io.raster (Failed)
	160 - test.python.pybind.polsar.symmetrize (Failed)
	161 - test.python.pybind.signal.convolve2D (Failed)
	162 - test.python.pybind.signal.covariance (Failed)
	163 - test.python.pybind.signal.crossmul (Failed)
	164 - test.python.pybind.signal.crossmultiply (Failed)
	165 - test.python.pybind.signal.filter2D (Failed)
	166 - test.python.pybind.product.generic_product (Failed)
	167 - test.python.pybind.product.radargridparameters (Failed)
	168 - test.python.pybind.product.swath (Failed)
	169 - test.python.pybind.unwrap.icu (Failed)
	170 - test.python.pybind.unwrap.phass (Failed)
	171 - test.python.pybind.geometry.ltpcoordinates (Failed)
	172 - test.python.pybind.geometry.pntintersect (Failed)
	173 - test.python.pybind.antenna.frame (Failed)
	174 - test.python.pybind.antenna.geometryfunc (Failed)
	175 - test.python.pybind.cuda.core.compute_capability (Failed)
	176 - test.python.pybind.cuda.core.device (Failed)
	177 - test.python.pybind.cuda.focus.backproject (Failed)
	178 - test.python.pybind.cuda.geocode.geocode (Failed)
	179 - test.python.pybind.cuda.geometry.geo2rdr (Failed)
	180 - test.python.pybind.cuda.geometry.rdr2geo (Failed)
	181 - test.python.pybind.cuda.image.resamp (Failed)
	182 - test.python.pybind.cuda.signal.crossmul (Failed)
	183 - test.python.pkg.pybind_nisar.products.readers.attitude (Failed)
	184 - test.python.pkg.pybind_nisar.products.readers.orbit (Failed)
	185 - test.python.pkg.pybind_nisar.products.readers.raw (Failed)
	186 - test.python.pkg.pybind_nisar.workflows.crossmul (Failed)
	187 - test.python.pkg.pybind_nisar.workflows.focus (Failed)
	188 - test.python.pkg.pybind_nisar.workflows.stage_dem (Failed)
	189 - test.python.pkg.pybind_nisar.workflows.gcov (Failed)
	190 - test.python.pkg.pybind_nisar.workflows.geo2rdr (Failed)
	191 - test.python.pkg.pybind_nisar.workflows.gslc (Failed)
	192 - test.python.pkg.pybind_nisar.workflows.gpu_check (Failed)
	193 - test.python.pkg.pybind_nisar.workflows.geocode_insar (Failed)
	194 - test.python.pkg.pybind_nisar.workflows.insar (Failed)
	196 - test.python.pkg.pybind_nisar.workflows.point_target_info (Failed)
	197 - test.python.pkg.pybind_nisar.workflows.rdr2geo (Failed)
	198 - test.python.pkg.pybind_nisar.workflows.resample_slc (Failed)
	199 - test.python.pkg.pybind_nisar.workflows.rubbersheet (Failed)
	200 - test.python.pkg.pybind_nisar.workflows.unwrap (Failed)
	201 - test.python.pkg.pybind_nisar.workflows.antenna_parser (Failed)
	202 - test.python.pkg.pybind_nisar.workflows.dense_offsets (Failed)
	203 - test.python.pkg.pybind_nisar.workflows.cuda_insar (Failed)

But interestingly manually importing some of the failed modules are actually working on python. For example, the code below was working;

from isce3.io import Raster as raster # failed ctest no. 159

Also I was able to process some test data using CUDA.

So Two things I suspect:

• Building ISCE3 0.4.0 from source code is sensitive to the version of certain dependent packages
• Some changes in the object names have prevented ctest to run smoothly, even though the build was successful. For example, I noticed that the tutorial package imports raster from isce3.io, but it actually seems like Raster in this version of ISCE3.

Thanks again for the opportunity to try out this software.
Best,

[gmgunter]

Hi @seongsujeong.

Sorry, the public installation instructions for isce3 have gotten somewhat outdated and are a bit misleading.

In particular, I suspect the reason all of the python unit tests are failing is because the instructions tell you to run ctest before installing the software with make install. In fact, the software must be installed before running the test suite. Otherwise, all of the python unit tests fail due to ImportErrors.

Let me look into updating the public docs. That should hopefully address the other issues like unlisted dependencies as well. @rtburns-jpl we should consider adding this as an additional step in our release procedure.

[rtburns-jpl]

Thanks for the report, I'm looking into this and think we can resolve most of these issues.

One part that sticks out to me is that you needed to downgrade pybind11 to 2.6.2 - can you elaborate on the issue you encountered? I wasn't aware of any incompatibility with newer versions, so if there is one, I want to get that sorted out quickly.

[seongsujeong]

Hi @gmgunter, Thanks for the feedback regarding the issue. Running ctest after make install did make them pass except for stage_dem, which I am considering as "okay" as mentioned in issue #5.

My apologies for the false alarm of the pybind11 version. I confirmed the installation goes okay with the recent version of pybind11 (i.e. 2.8.1). I think I was confused after a few times of trials and errors to troubleshoot the procedure.