phase_linking stage takes too long

[gjustin40]

I am performing analysis using miaplpy.
Attempting to run full-resolution without applying Multi-looking in ISCE2.

The configuration used is as follows.

If there is a trade-off between accuracy and processing time, what option can I modify to reduce the time?
Are there any options other than reducing the resolution?

Thank you.

# vim: set filetype=cfg:
##------------------------------- ISCE/topsStack ----------------------##
#ssaraopt = --platform=SENTINEL-1A,SENTINEL-1B -r 128 -f 589,590,591,592,593  -e 2017-07-01
#sentinelStack.boundingBox      = '36.33 36.45 125.59 125.7'
#sentinelStack.subswath         = 3  # comment 
#sentinelStack.numConnections   = 2   # comment
#sentinelStack.azimuthLooks     = 1   # comment
#sentinelStack.rangeLooks       = 1  # comment
#sentinelStack.filtStrength     = 0.5  # comment
#sentinelStack.unwMethod        = snaphu  # comment
#sentinelStack.coregistration   = auto(NESD)  # comment

################### paper miaplpy #############################
miaplpy.load.processor      = isce  #[isce,snap,gamma,roipac], auto for isceTops
miaplpy.load.updateMode     = auto  #[yes / no], auto for yes, skip re-loading if HDF5 files are complete
miaplpy.load.compression    = auto  #[gzip / lzf / no], auto for no.
miaplpy.load.autoPath       = yes    # [yes, no] auto for no
miaplpy.multiprocessing.numProcessor = 32

miaplpy.load.slcFile        = ../merged/SLC/*/*.slc.full  #[path2slc_file]
##---------for ISCE only:
miaplpy.load.metaFile       = ../reference/IW*.xml
miaplpy.load.baselineDir    = ../baselines
##---------geometry datasets:
miaplpy.load.demFile          = ../merged/geom_reference/hgt.rdr.full
miaplpy.load.lookupYFile      = ../merged/geom_reference/lat.rdr.full
miaplpy.load.lookupXFile      = ../merged/geom_reference/lon.rdr.full
miaplpy.load.incAngleFile     = ../merged/geom_reference/los.rdr.full
miaplpy.load.azAngleFile      = ../merged/geom_reference/los.rdr.full
miaplpy.load.shadowMaskFile   = ../merged/geom_reference/shadowMask.rdr.full
miaplpy.load.waterMaskFile    = None
##---------interferogram datasets:
miaplpy.load.unwFile        = ./inverted/interferograms_single_reference/*/*fine*.unw
miaplpy.load.corFile        = ./inverted/interferograms_single_reference/*/*fine*.cor
miaplpy.load.connCompFile   = ./inverted/interferograms_single_reference/*/*.unw.conncomp

########## 2. modify_network
## 1) Network modification based on temporal/perpendicular baselines, date, num of connections etc.
mintpy.network.tempBaseMax     = 400  #[1-inf, no], auto for no, max temporal baseline in days
mintpy.network.perpBaseMax     = 200  #[1-inf, no], auto for no, max perpendicular spatial baseline in meter

########## 6. correct_troposphere (optional but recommended)
mintpy.troposphericDelay.method = no #[pyaps / height_correlation / gacos / no], auto for pyaps

[mirzaees]

Hi @gjustin40
I believe the wrapped-phase estimation is quick enough with your config, but unwrapping takes time especially if your area is decorrelated and you process in full resolution without any filtering. Unfortunately I don't have any suggestions to reduce the time other that what you are doing.

[Jesse-kearse]

Hi, I would like to ask how best to optimize the phase-linking process for my machine. My CPU is 4.5 GHz with 16 cores 32 threads. However, some of the patches (200 x 200, 135 slc images in stack) take >40 hours to run (see below example of job 1). With another 31 jobs still in the queue, this stack may take >1 month to complete. My AOI is 4000x104000 pixels. Do you have any advice for improving efficiency, or are the default parameters the best I can hope for with a large stack of SLC images (135)?

Thank you for any help you can provide.

20240928:180440 * phase_linking.py --work_dir /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy --range_window 15 --azimuth_window 15 --patch_size 200 --method sequential_EMI --test ks --num_worker 32 --mini_stack_size 10 --time_lag 10 --ps_num_shp 10 --mask /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/mask_hills.h5 --index 0 --slc_stack /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/inputs/slcStack.h5
Total number of PATCHES/tasks for job 0 : 32
Remaining number of PATCHES/tasks: 32
Number of parallel tasks: 32
Reading SLC data from /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/inputs/slcStack.h5 and inverting patches in parallel ...
[======================= 86% ==============> ] 34500/40000 pixels, patch 28 0s / 0s Phase inversion of PATCH_0026 is Completed in 00 mins 41 secs

[======================= 72% =======> ] 29000/40000 pixels, patch 27 0s / 0s Phase inversion of PATCH_0024 is Completed in 00 mins 41 secs

[======================= 90% ================> ] 36000/40000 pixels, patch 27 0s / 0s Phase inversion of PATCH_0025 is Completed in 00 mins 41 secs

[======================= 98% ====================> ] 39000/40000 pixels, patch 30 0s / 0s Phase inversion of PATCH_0023 is Completed in 00 mins 42 secs

[==================================================] 40000/40000 pixels, patch 30 Phase inversion of PATCH_0028 is Completed in 00 mins 42 secs
Phase inversion of PATCH_0029 is Completed in 00 mins 42 secs

[==================================================] 40000/40000 pixels, patch 27
Phase inversion of PATCH_0031 is Completed in 00 mins 42 secs

Phase inversion of PATCH_0022 is Completed in 00 mins 42 secs

Phase inversion of PATCH_0027 is Completed in 00 mins 42 secs

Phase inversion of PATCH_0030 is Completed in 00 mins 42 secs

[==================================================] 40000/40000 pixels, patch 21 Phase inversion of PATCH_0021 is Completed in 03 mins 37 secs

[==================================================] 40000/40000 pixels, patch 20 Phase inversion of PATCH_0020 is Completed in 186 mins 37 secs

[==================================================] 40000/40000 pixels, patch 18 Phase inversion of PATCH_0018 is Completed in 452 mins 47 secs

[==================================================] 40000/40000 pixels, patch 19 Phase inversion of PATCH_0019 is Completed in 459 mins 26 secs

[==================================================] 40000/40000 pixels, patch 0 Phase inversion of PATCH_0000 is Completed in 489 mins 44 secs

[==================================================] 40000/40000 pixels, patch 17 Phase inversion of PATCH_0017 is Completed in 768 mins 15 secs

[==================================================] 40000/40000 pixels, patch 16 Phase inversion of PATCH_0016 is Completed in 1140 mins 35 secs

[==================================================] 40000/40000 pixels, patch 15 Phase inversion of PATCH_0015 is Completed in 1169 mins 03 secs

[==================================================] 40000/40000 pixels, patch 14 Phase inversion of PATCH_0014 is Completed in 1206 mins 00 secs

[==================================================] 40000/40000 pixels, patch 9 Phase inversion of PATCH_0009 is Completed in 1852 mins 08 secs

[==================================================] 40000/40000 pixels, patch 10 Phase inversion of PATCH_0010 is Completed in 1990 mins 24 secs

[==================================================] 40000/40000 pixels, patch 8 Phase inversion of PATCH_0008 is Completed in 2015 mins 18 secs

[==================================================] 40000/40000 pixels, patch 1 Phase inversion of PATCH_0001 is Completed in 2097 mins 30 secs

[==================================================] 40000/40000 pixels, patch 5 Phase inversion of PATCH_0005 is Completed in 2110 mins 36 secs

[==================================================] 40000/40000 pixels, patch 4 Phase inversion of PATCH_0004 is Completed in 2140 mins 56 secs

[==================================================] 40000/40000 pixels, patch 11 Phase inversion of PATCH_0011 is Completed in 2221 mins 22 secs

[==================================================] 40000/40000 pixels, patch 2 Phase inversion of PATCH_0002 is Completed in 2223 mins 13 secs

[==================================================] 40000/40000 pixels, patch 13 Phase inversion of PATCH_0013 is Completed in 2338 mins 40 secs

[==================================================] 40000/40000 pixels, patch 6 Phase inversion of PATCH_0006 is Completed in 2360 mins 34 secs

[==================================================] 40000/40000 pixels, patch 3 Phase inversion of PATCH_0003 is Completed in 2363 mins 04 secs

[==================================================] 40000/40000 pixels, patch 12 Phase inversion of PATCH_0012 is Completed in 2500 mins 21 secs

[==================================================] 40000/40000 pixels, patch 7 Phase inversion of PATCH_0007 is Completed in 2500 mins 51 secs

Using default MintPy Path: /home/seis/anaconda3/envs/insar/lib/python3.10/site-packages
20240930:114451 * phase_linking.py --work_dir /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy --range_window 15 --azimuth_window 15 --patch_size 200 --method sequential_EMI --test ks --num_worker 32 --mini_stack_size 10 --time_lag 10 --ps_num_shp 10 --mask /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/mask_hills.h5 --index 1 --slc_stack /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/inputs/slcStack.h5
Total number of PATCHES/tasks for job 1 : 32
Remaining number of PATCHES/tasks: 32
Number of parallel tasks: 32
Reading SLC data from /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/inputs/slcStack.h5 and inverting patches in parallel ...
[======================= 49% ] 19500/40000 pixels, patch 48 0s / 0s Phase inversion of PATCH_0041 is Completed in 00 mins 36 secs

[=============> 28% ] 11000/40000 pixels, patch 37 0s / 0s Phase inversion of PATCH_0036 is Completed in 00 mins 36 secs

[Jesse-kearse]

a typo in the above dimensions of my AOI, it is actually 4000x10400 pixels.

[mirzaees]

@Jesse-kearse
the long times might be related to the new versions of scipy as others experienced it too
I did not have time to upgrade yet but you might want to try using older versions

[Jesse-kearse]

Thanks for your reply, @mirzaees . Which version of scipy do you suggest I downgrade to? are "new versions" 1.0 and newer?

[Jesse-kearse]

I have the following version installed in my miaplpy conda environment: scipy==1.10.1; numpy==1.23.1

I am still experiencing extremely slow phase_linking. Are the version listed above perhaps still to new?

I have used "conda install scipy==1.10.1" to downgrade to these versions, but do I need to re-install Miaplpy as well, I wouldn't think so?

Thanks for any help you can provide.

[ZGHHGZ]

I have the following version installed in my miaplpy conda environment: scipy==1.10.1; numpy==1.23.1

I am still experiencing extremely slow phase_linking. Are the version listed above perhaps still to new?

I have used "conda install scipy==1.10.1" to downgrade to these versions, but do I need to re-install Miaplpy as well, I wouldn't think so?

Thanks for any help you can provide.

Phase linking takes a lot of time to compute because DS-InSAR is very time consuming. After reducing numpy and scipy to the version you describe, if it is still very slow at this point, that is normal. If you don't downgrade numpy and scipy to the current version, it will take 10 times as long as it does now. DS-InSAR is suitable for very small study areas, and large areas can take months to years to process.

[Jesse-kearse]

Thanks for your comment.

I do understand that the DS-InSAR is computationally expensive. My confusion is that after Miaplpy separates the study area into patches of 200x200 elements, a single patch is taking up to 50 hours to run - which I thought would be independent of the study area size, as it still needs to loop through all the other patches with the subsequent jobs. Is this correct. Maybe I am not understanding something more fundamental about the computations. Thanks again for any further help (and please ignore my comment on the other thread)

[ZGHHGZ]

Thanks for your comment.

I do understand that the DS-InSAR is computationally expensive. My confusion is that after Miaplpy separates the study area into patches of 200x200 elements, a single patch is taking up to 50 hours to run - which I thought would be independent of the study area size, as it still needs to loop through all the other patches with the subsequent jobs. Is this correct. Maybe I am not understanding something more fundamental about the computations. Thanks again for any further help (and please ignore my comment on the other thread)

Does it still take around 50 hours for a single patch after lowering the scipy version and numpy version, normally it takes around 5 hours for a single patch.

[ZGHHGZ]

Thanks for your comment.
I do understand that the DS-InSAR is computationally expensive. My confusion is that after Miaplpy separates the study area into patches of 200x200 elements, a single patch is taking up to 50 hours to run - which I thought would be independent of the study area size, as it still needs to loop through all the other patches with the subsequent jobs. Is this correct. Maybe I am not understanding something more fundamental about the computations. Thanks again for any further help (and please ignore my comment on the other thread)

Does it still take around 50 hours for a single patch after lowering the scipy version and numpy version, normally it takes around 5 hours for a single patch.

Please output the following code in miaplpy environment
print(numpy.version.version)
print(scipy.version.version)

[Jesse-kearse]

Here is an example of job 4 (32 patches processed in parallel) from start to finish. you can see that it took 4720 minutes for the final patch to be processed, before beginning job 5 (which I terminated sometime after 5600 minutes).

20241009:200325 * phase_linking.py --work_dir /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy --range_window 15 --azimuth_window 15 --patch_size 200 --method sequential_EMI --test ks --num_worker 32 --mini_stack_size 10 --time_lag 10 --ps_num_shp 10 --mask /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/mask_hills.h5 --index 4 --slc_stack /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/inputs/slcStack.h5
Total number of PATCHES/tasks for job 4 : 32
Remaining number of PATCHES/tasks: 32
Number of parallel tasks: 32
Reading SLC data from /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/inputs/slcStack.h5 and inverting patches in parallel ...
[======================= 90% ================> ] 36000/40000 pixels, patch 148 0s / 0s Phase inversion of PATCH_0145 is Completed in -1 mins 59 secs

[======================= 81% ===========> ] 32500/40000 pixels, patch 149 0s / 0s Phase inversion of PATCH_0154 is Completed in -1 mins 59 secs

[==================================================] 40000/40000 pixels, patch 155 Phase inversion of PATCH_0151 is Completed in -1 mins 59 secs
[======================= 96% ===================> ] 38500/40000 pixels, patch 147 0s / 0s Phase inversion of PATCH_0148 is Completed in -1 mins 59 secs
[==================================================] 40000/40000 pixels, patch 150
[======================= 88% ===============> ] 35000/40000 pixels, patch 159 0s / 0s
Phase inversion of PATCH_0156 is Completed in -1 mins 59 secs

[======================= 98% ====================> ] 39000/40000 pixels, patch 149 0s / 0s Phase inversion of PATCH_0157 is Completed in -1 mins 59 secs

[======================= 96% ===================> ] 38500/40000 pixels, patch 159 0s / 0s Phase inversion of PATCH_0146 is Completed in -1 mins 59 secs

[==================================================] 40000/40000 pixels, patch 159 Phase inversion of PATCH_0152 is Completed in -1 mins 59 secs

[==================================================] 40000/40000 pixels, patch 149 Phase inversion of PATCH_0150 is Completed in -1 mins 59 secs

Phase inversion of PATCH_0159 is Completed in -1 mins 59 secs

Phase inversion of PATCH_0147 is Completed in -1 mins 59 secs

Phase inversion of PATCH_0153 is Completed in -1 mins 59 secs
Phase inversion of PATCH_0149 is Completed in -1 mins 59 secs


Phase inversion of PATCH_0155 is Completed in -1 mins 59 secs

Phase inversion of PATCH_0158 is Completed in -1 mins 59 secs

[==================================================] 40000/40000 pixels, patch 144 Phase inversion of PATCH_0144 is Completed in 1556 mins 40 secs

[==================================================] 40000/40000 pixels, patch 137 Phase inversion of PATCH_0137 is Completed in 2192 mins 40 secs

[==================================================] 40000/40000 pixels, patch 135 Phase inversion of PATCH_0135 is Completed in 2206 mins 49 secs

[==================================================] 40000/40000 pixels, patch 138 Phase inversion of PATCH_0138 is Completed in 2285 mins 20 secs

[==================================================] 40000/40000 pixels, patch 140 Phase inversion of PATCH_0140 is Completed in 2413 mins 60 secs

[==================================================] 40000/40000 pixels, patch 136 Phase inversion of PATCH_0136 is Completed in 2747 mins 28 secs

[==================================================] 40000/40000 pixels, patch 141 Phase inversion of PATCH_0141 is Completed in 2883 mins 44 secs

[==================================================] 40000/40000 pixels, patch 139 Phase inversion of PATCH_0139 is Completed in 2885 mins 07 secs

[==================================================] 40000/40000 pixels, patch 143 Phase inversion of PATCH_0143 is Completed in 2906 mins 15 secs

[==================================================] 40000/40000 pixels, patch 142 Phase inversion of PATCH_0142 is Completed in 3135 mins 05 secs

[==================================================] 40000/40000 pixels, patch 134 Phase inversion of PATCH_0134 is Completed in 3518 mins 49 secs

[==================================================] 40000/40000 pixels, patch 132 Phase inversion of PATCH_0132 is Completed in 3721 mins 16 secs

[==================================================] 40000/40000 pixels, patch 131 Phase inversion of PATCH_0131 is Completed in 4114 mins 37 secs

[==================================================] 40000/40000 pixels, patch 133 Phase inversion of PATCH_0133 is Completed in 4456 mins 39 secs

[==================================================] 40000/40000 pixels, patch 130 Phase inversion of PATCH_0130 is Completed in 4690 mins 04 secs

[==================================================] 40000/40000 pixels, patch 129 Phase inversion of PATCH_0129 is Completed in 4719 mins 57 secs

[==================================================] 40000/40000 pixels, patch 128 Phase inversion of PATCH_0128 is Completed in 4720 mins 15 secs

Using default MintPy Path: /home/seis/anaconda3/envs/insar/lib/python3.10/site-packages
20241013:024343 * phase_linking.py --work_dir /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy --range_window 15 --azimuth_window 15 --patch_size 200 --method sequential_EMI --test ks --num_worker 32 --mini_stack_size 10 --time_lag 10 --ps_num_shp 10 --mask /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/mask_hills.h5 --index 5 --slc_stack /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/inputs/slcStack.h5
Total number of PATCHES/tasks for job 5 : 32
Remaining number of PATCHES/tasks: 32
Number of parallel tasks: 32
Reading SLC data from /home/seis/data/insar_work/osaka/AT_10/sb_stack/miaplpy/inputs/slcStack.h5 and inverting patches in parallel ...
[=================> 36% ] 14500/40000 pixels, patch 169 0s / 0s Phase inversion of PATCH_0170 is Completed in -1 mins 12 secs

[=================> 35% ] 14000/40000 pixels, patch 171 0s / 0s Phase inversion of PATCH_0161 is Completed in -1 mins 12 secs
[=====================> 44% ] 17500/40000 pixels, patch 165 0s / 0s
[======================= 71% =======> ] 28500/40000 pixels, patch 166 0s / 0s Phase inversion of PATCH_0160 is Completed in -1 mins 12 secs
[===================> 39% ] 15500/40000 pixels, patch 167 0s / 0s
[======================= 81% ===========> ] 32500/40000 pixels, patch 167 0s / 0s Phase inversion of PATCH_0163 is Completed in -1 mins 13 secs

[======================= 91% =================> ] 36500/40000 pixels, patch 167 0s / 0s Phase inversion of PATCH_0166 is Completed in -1 mins 13 secs
[======================= 84% =============> ] 33500/40000 pixels, patch 171 0s / 0s
[======================= 85% =============> ] 34000/40000 pixels, patch 171 0s / 0s Phase inversion of PATCH_0169 is Completed in -1 mins 13 secs
[======================= 92% =================> ] 37000/40000 pixels, patch 167 0s / 0s
[======================= 94% ==================> ] 37500/40000 pixels, patch 171 0s / 0s Phase inversion of PATCH_0168 is Completed in -1 mins 13 secs
[======================= 95% ===================> ] 38000/40000 pixels, patch 171 0s / 0s
[======================= 98% ====================> ] 39000/40000 pixels, patch 171 0s / 0s Phase inversion of PATCH_0164 is Completed in -1 mins 13 secs
Phase inversion of PATCH_0162 is Completed in -1 mins 13 secs
[==================================================] 40000/40000 pixels, patch 171

Phase inversion of PATCH_0165 is Completed in -1 mins 13 secs

Phase inversion of PATCH_0167 is Completed in -1 mins 13 secs

Phase inversion of PATCH_0171 is Completed in -1 mins 13 secs

[==================================================] 40000/40000 pixels, patch 172 Phase inversion of PATCH_0172 is Completed in 333 mins 39 secs

[==================================================] 40000/40000 pixels, patch 173 Phase inversion of PATCH_0173 is Completed in 2518 mins 14 secs

[==================================================] 40000/40000 pixels, patch 191 Phase inversion of PATCH_0191 is Completed in 4533 mins 06 secs

[==================================================] 40000/40000 pixels, patch 190 Phase inversion of PATCH_0190 is Completed in 4558 mins 10 secs

[==================================================] 40000/40000 pixels, patch 189 Phase inversion of PATCH_0189 is Completed in 5258 mins 04 secs

[==================================================] 40000/40000 pixels, patch 187 Phase inversion of PATCH_0187 is Completed in 5565 mins 12 secs

[==================================================] 40000/40000 pixels, patch 185 Phase inversion of PATCH_0185 is Completed in 5665 mins 23 secs

[======================= 57% ] 23000/40000 pixels, patch 181354929s / 267753s^C
Caught KeyboardInterrupt, terminating workers

[Jesse-kearse]

this was after I had downgraded to the versions listed above

[Jesse-kearse]

import numpy
print(numpy.version.version)
1.23.1
import scipy
print(scipy.version.version)
1.10.1

[ZGHHGZ]

import numpy
print(numpy.version.version)
1.23.1
import scipy
print(scipy.version.version)
1.10.1

You can try installing the following version.
print(numpy.version.version)
1.26.4
print(scipy.version.version)
1.12.0
When entering phase linking, the command line gives an estimate of the processing time and compares it to see if the speed has improved.

[Jesse-kearse]

Thanks for your suggestion. I will install these versions and retry phase linking.

"When entering phase linking, the command line gives an estimate of the processing time and compares it to see if the speed has improved." I am not familiar with this command line output? The example that I gave above is the direct command line print out.