autoadaloc and maxsing musings

[wkitlasten]

I am an unapologetic fan boi of autoadaloc. But man, it can take some time for big problems! I'm also pretty bad at "seeing through the maths" which results in some misguided musings, so feel free to disregard!

It appears the autoadaloc calcs are done for all pars. But we don't really care about spurious correlations if the sensitivities don't even meet subspace reg constraints (other than to define the distribution of correlation coefficients), right? I wonder if we could assume the same std criteria holds for the distribution of correlation coefficients calculated using a smaller number of samples (or scale it somehow)?

I'm imagining something like doing the calcs in blocks, obvious late night psuedo-code:

while unsolved:
    calc correlations using the most sensitive N*MAXSING pars (assuming N*MAXSING pars is enough to establish spurious correlations) or alternatively N*MAXSING plus M*MINSING or random... or maybe it is all or nothing!
    if mapped_pars > MAXSING:
        initiate solve
        unsolved = False
    else:
        add another chunk of pars

[wkitlasten]

Are the correlation coefficients finished by 20:00:19, or is that just the start of "shuffling" process? The autoadaloc process quickly picks out a few "fully localized" pars. Most of these are mapped to a single obs in my absurdly granular "manual" localizer. I assume the empirical sensitivity of this par to the single obs it is meant to influence is below the "background" correlation?

19:56:58,7.99e-05,autoadaloc: calculating correlation coefficients
20:00:19,200.126,launching autoadaloc threads
20:00:19,0.0216553,waiting to join threads
20:00:19,0.0753556,autoadaloc warning: parameter PNAME:BORE_ELEV_INST:1_PTYPE:GR_USECOL:BORE_ELEV_PSTYLE:A_ID_DEPTH:HEAD_MEASURED_1666_31 is completely localized -it maps to no observations

A bit further along and it starts slowing down, taking about 50 sec (per thread?), presumably because there are more obs to consider for each par? (Note, the counter maybe backwards as fewer get "done" with each iteration.) Based on my limited understanding, this is where I would think we don't need to do all 156994 parameters, but maybe instead a subset based on singular values. Isn't the empirical Jacobian already calculated at this point?

20:03:58,48.7335,autoadaloc iter 1 progress: 136000 of 156994 parameters done
20:04:48,49.8333,autoadaloc iter 1 progress: 135000 of 156994 parameters done
20:05:38,50.0008,autoadaloc iter 1 progress: 134000 of 156994 parameters done
20:06:27,49.0639,autoadaloc iter 1 progress: 133000 of 156994 parameters done

Another big chunk of time comes during the (very granular) upgrade. What determines the number of parts (presumably mapped par/obs relationships, number of threads, other)? (Note, more parts getting "done" with each iteration)

21:13:47,12.9343,starting calcs for glm factor 0.1
21:13:47,3.33e-05,see .log file for more details
21:13:48,1.13391,launching threads
21:13:48,0.105454,waiting to join threads
21:13:48,0.0860875,upgrade thread progress: 0 of 114478 parts done
21:17:39,230.517,upgrade thread progress: 1000 of 114478 parts done
21:21:29,230.128,upgrade thread progress: 2000 of 114478 parts done

[jtwhite79]

Hey Wes - so whats happening algorithmically is the corr coef between every par and every obs is first calculated. Thats the first multithreaded part. Using these CCs, we can decide if a par-obs relation is noise or not and use that to make a localizer. Then we can feed this localizer to the solution process (thats the second multithreaded part). The upgrade is where the relations between pars and obs are put together to form upgrade. But we have to know what relations are allowed before we can form this upgrade. Im trying to think thru what you are putting down - Am I missing it?

[wkitlasten]

Thanks for taking the time to pick up what I'm putting down!

The empirical jacobian can be calculated before the CCs are calculated, right? If so, could you do a 2 step partial evaluation of the CCs to speed up step 1 a bit?

1. Calc CCs for random sample of par-obs to cycle through all realizations to approximate the distribution used to determine noise. No idea how many would be required, or if I'm missing something fundamental here... but I would think with 100,000+ obs and 100,000+ pars you wouldn't need every pair?
2. Calc CCs for any remaining par-obs combos that fulfill the MAXSING requirement, add them to the localizer if they look like noise. The rest get 0 in the localizer for that iteration.

Solve... likely always slow with granular localization.