Why should we train NetL separately in train_L_step_w

[itapty-ily]

In the function train_L_step_w, line 192, in train.py, we the gradients of loss are firstly propagated to NetL with Doc3D data set. Then in the gradients are propagated to NetL with DIW. People update the NetL twice in one step (optimizer_L.step()x2).
I have a question. We compute the spvLoss.lloss and warp_diff_loss with the corresponding data set and a supervision mask (using the torch.where function to remove unrelative gradients) , so that we can update the NetL in one step.
I read your paper, did not read the issues, and train the NetL with my idea. It does not converge.
According to the optimization theory, these two training tricks will lead to different NetLs. Therefore, I want to know the result of my training trick. Or let us check the reason.

[wkema]

Sry my bad. I saw the email and I must reply in mind...

IIUC your idea is correct. It is only a different implementation.

I am not sure about it does not converge. When you train the Tnet in the weakly supervised manner, you should see the loss on the data from doc3d fluctuates (as the network has already converged when it was trained on doc3d only), while the loss on the data from diw drops. Since there is a 0.1 multiplier, if you just monitor the whole loss, you may not see significant changes.

Also I found the weakly sup training is tricky. The loss might not drop in the first several epochs.

Below might be (did not document the exp well so I am not 100% sure lol) part of my training log for your reference:

100,0.005063588265329599,0.02370934202136986
200,0.004935386124998331,0.022192930404885902
300,0.005119685549288988,0.021928391292158716
400,0.004981142934411764,0.02187072460517174
500,0.0050409529358148575,0.020735550189557334
600,0.0049947695806622505,0.0211542390116961
700,0.004915028810501099,0.020587545698645834
800,0.004990944638848305,0.020549847342187944
900,0.004903901368379593,0.02045272541466805
1000,0.005064345896244049,0.02088677820217645
1100,0.0049436925910413265,0.020250055080146698
1200,0.004958824720233679,0.020161601953728493
1300,0.004986349493265152,0.02039550368504056
1400,0.00499705970287323,0.019787318414702548
1500,0.005011213012039661,0.01999400551726301
1600,0.004894820973277092,0.019615766742502306
1700,0.005025567952543497,0.01998966940524982
1800,0.005114749539643526,0.01963189327273391
1900,0.005022508092224598,0.01974157745019287
2000,0.005051152780652046,0.019586424166879952
2100,0.0048703644424676895,0.01913408813931006
2200,0.004957091063261032,0.018964561364001764
2300,0.005040901713073254,0.019552078878607746
2400,0.00502112926915288,0.019812215049959395
2500,0.0051467400044202805,0.019536561044368718
2600,0.0051370542496442795,0.01931596776574929
2700,0.0049997540190815926,0.019503309363752386
2800,0.00503464974462986,0.01917101903851312
2900,0.004995362833142281,0.0189093729876982
3000,0.005060319788753986,0.019391178179471037
3100,0.0050315591506659985,0.018922625512100506
3200,0.0051877968944609165,0.019159920200847214
3300,0.00510516669601202,0.019096903228442975
3400,0.005153291393071413,0.01920557122347776
3500,0.005053812172263861,0.018502684357541974
3600,0.0050436812452971935,0.019163471748832067
3700,0.005045927129685879,0.018949978152939296
3800,0.005036205053329468,0.018755136477167353
3900,0.005005819257348776,0.018657028925450434
4000,0.005015464965254068,0.0186833193359192
4100,0.005152611993253231,0.0187683873301277
4200,0.005056890659034252,0.018977879092028797
4300,0.005157520994544029,0.01858032355662087
4400,0.005055396351963282,0.01836480664451265
4500,0.005114920437335968,0.018911168594058655
4600,0.005029668565839529,0.01864923090862618
4700,0.005187106318771839,0.01893219656656726
4800,0.005118091590702534,0.018685637584060184
4900,0.005208715330809355,0.01891479855790604
5000,0.005045102909207344,0.018463619530186146
5100,0.005198843311518431,0.018528455137144
5200,0.005084374453872442,0.01821221580195834
5300,0.005017156712710857,0.018156634556730264
5400,0.005164026748389006,0.018550131587547935
5500,0.005054773297160864,0.018286904431760346
5600,0.005188659764826298,0.01867897027372242
5700,0.005099665839225054,0.018122819249187076
5800,0.004954097326844931,0.01821176953402006
5900,0.004983177874237299,0.018063550908384268
6000,0.0050760190933942795,0.01838901475107358
6100,0.0050915866158902645,0.018429808222736543
6200,0.005066524725407362,0.018086263409570882
6300,0.005140822846442461,0.018022173762230892
6400,0.004976288881152868,0.017803625915408766
6500,0.005096477456390858,0.018190466708287462
6600,0.005129553377628326,0.018321200010965367
6700,0.005067517980933189,0.01847482156500423
6800,0.0051100836135447025,0.018117362248123194
6900,0.004947172477841377,0.017991205841614632
7000,0.00500882463529706,0.01808356638170746
7100,0.005111531354486942,0.018221665996114314
7200,0.005158735904842615,0.017969297821906868
7300,0.005106328055262566,0.018003088725714884
7400,0.005048552993685007,0.01797017322638618
7500,0.00505921570584178,0.01773643644663264
7600,0.005083793308585882,0.017740979126475025
7700,0.0050888885743916035,0.017991992946363743
7800,0.005112549755722284,0.017688068847847762
7900,0.0050511048175394535,0.018229370963517653
8000,0.0049722096882760525,0.017501540532057193
8100,0.005142809357494116,0.017890013834209795
8200,0.005066135432571173,0.01744315579839847
8300,0.00511603057384491,0.01748281598038487
8400,0.005207541864365339,0.0179856979743903
8500,0.005039854906499386,0.018147616880004595
8600,0.005058737006038427,0.01751584804817577
8700,0.0050256880931556225,0.017666095496568404
8800,0.0051104240119457245,0.01791217255691693
8900,0.005057178437709808,0.018108111947888146
9000,0.005073448643088341,0.01751806254175639
9100,0.005069409497082233,0.01763443953852933
9200,0.004967368673533201,0.01781840996920026
9300,0.005104700569063425,0.017298269710773778
9400,0.005121214780956507,0.018104995522937768
9500,0.005068340804427862,0.01769434156151156
9600,0.00506182387471199,0.0177548410635703
9700,0.005070047918707132,0.01768864319672391
9800,0.005082407034933567,0.017681782471694288
9900,0.005042869132012129,0.017680984781758932
10000,0.005144722759723663,0.017649849083728693
10100,0.005149520467966795,0.01782877853451249
10200,0.005029845517128706,0.017689869467588756
10300,0.005004498641937971,0.01770577735083724
10400,0.005142222158610821,0.017733169813551455
10500,0.005035723093897104,0.017489189457483074
10600,0.005146523471921682,0.017886761965415422
10700,0.005066465120762587,0.017881848862513334
10800,0.00519703421741724,0.017751254158660176
10900,0.0052534244023263454,0.01766847537581638
11000,0.005095356609672308,0.017325492920294554
11100,0.005109227262437344,0.017326535374238408
11200,0.005051949992775917,0.01724445531971433
11300,0.004993387497961521,0.01717610032537176
11400,0.005004818085581064,0.017540716627468658
11500,0.005011163651943207,0.01731159731750203
11600,0.0050667268224060535,0.0176553093727354
11700,0.005025411956012249,0.01742547087709361
11800,0.005094574298709631,0.017385293080985235
11900,0.004999820608645678,0.01711685024227235
12000,0.005150946322828531,0.017528203680422204
12100,0.004913340322673321,0.01708819018085543
12200,0.00516160111874342,0.01752402819259588
12300,0.005033752880990505,0.017194476921818344
12400,0.005021130666136742,0.01742381763959208
12500,0.004977911710739136,0.017267174064356915
12600,0.005074720364063978,0.017081996364153303
12700,0.005014317575842142,0.01724405743882404
12800,0.00513584865257144,0.017619379132109272
12900,0.005105980206280947,0.017341053219106187
13000,0.004963347222656012,0.017015018766546954
13100,0.005080872680991888,0.01708559109772523
13200,0.005049297120422125,0.017389978788253914
13300,0.0049912030808627605,0.017269668142476043
13400,0.005138266831636429,0.017159466563777034
13500,0.005093121435493231,0.017481376159964333
13600,0.005029590800404549,0.01702950325446383
13700,0.00511751975864172,0.017336320683297865
13800,0.004985549487173557,0.01731252500695391
13900,0.004978805314749479,0.016822183279029793
14000,0.0050791045650839806,0.017147356183233323
14100,0.005115289706736803,0.01718729724977572
14200,0.005044733174145222,0.016885606143606388
14300,0.005097928922623396,0.016765603508410516
14400,0.004988375585526228,0.017009330714409302
14500,0.005115858744829893,0.01718717492041414
14600,0.005077785812318325,0.017138921134050442
14700,0.005027669947594404,0.017026507674188014
14800,0.005161136854439974,0.017272212045865952
14900,0.005096090957522392,0.017131204053052896
15000,0.005074597895145416,0.017328664095173592
15100,0.0050181495025753975,0.016989927437741192
15200,0.005027234088629484,0.016949108675530625
15300,0.005086027085781097,0.016982037915826736
15400,0.0051588015630841255,0.016979691761674635
15500,0.005010928027331829,0.016853461790167275
15600,0.005075820256024599,0.01677895493904782
15700,0.005030659027397633,0.017015818410753693
15800,0.0051097567193210125,0.016914397613342062
15900,0.005015738774091005,0.017168930507203133
16000,0.00513323163613677,0.01704400165707704
16100,0.005079986061900854,0.017215396960201895
16200,0.005077979993075132,0.017389186217122224
16300,0.005066171288490295,0.01703552727312825
16400,0.005185272078961134,0.017332382197871605
16500,0.005027707666158676,0.017063749420024525
16600,0.005190769676119089,0.01736109989003955
16700,0.005173949524760246,0.017408522330081975
16800,0.005009079817682505,0.01707219486207405
16900,0.005081778857856989,0.017054250524635762
17000,0.005094040185213089,0.017123837941243718
17100,0.005071504041552544,0.017141971764489547
17200,0.005277260672301054,0.017408842515744122
17300,0.005172219593077898,0.016957586480792358
17400,0.005042460281401873,0.01743945400296821
17500,0.005064051598310471,0.016741323842592103
17600,0.005149027798324823,0.01723910541509017
17700,0.004989778622984886,0.017045945484915833
17800,0.005025830119848251,0.016978719986969776
17900,0.005147910676896572,0.017312073484943385
18000,0.005073569715023041,0.01695158000239283
18100,0.005074741318821907,0.017127971701446174
18200,0.00505696889013052,0.016980369910154607
18300,0.00505310483276844,0.017111300210977513
18400,0.005096033681184053,0.01731119412993352
18500,0.004993887152522802,0.017083545935357436
18600,0.0052564446814358234,0.017154309001907175
18700,0.005086237099021673,0.017093555657946506
18800,0.0050272406078875065,0.01698923420892316
18900,0.005038172472268343,0.01731060073459176
19000,0.004986036103218794,0.016878104985226294
19100,0.00506561016663909,0.017287441577607452
19200,0.0050743576139211655,0.01709481608641651
19300,0.005089832469820976,0.017113640417450386
19400,0.0050940606743097305,0.017131619414273318
19500,0.004956440534442663,0.016614284556741632

the first col might steps (sry cannot remember all the details) and the 2nd col is the sup loss and 3rd col is w sup loss. You can see the 2nd col just fluctuated while the 3rd col dropped from 0.024 to 0.016