updated MOOSE and it doesnt converge when PorousFlowMassTimeDerivative is active

[Traiwit]

Hi guys,

I have just updated MOOSE to the latest version and for some reason when have PorousFlowMassTimeDerivative in my input file (to make simulation transient) it won't converge at all, the steady-state run works fine:

  [PorousFlowMassTimeDerivative]
    type = PorousFlowMassTimeDerivative
    ### see https://mooseframework.inl.gov/source/kernels/PorousFlowMassTimeDerivative.html
    variable = porepressure_L1
    fluid_component = 0
    multiply_by_density = true
  []

Here is the input: https://github.com/Traiwit/public_moose_beh/blob/main/input_file_transient_doesnt_converge.i

Not sure if this is related to New initial residual evaluation behaviour
or the node number system has been updated (since we have nodal-constraints action) [this shouldnt be the case since SS run works fine], etc

I can confirm the only difference between these 2 runs are the moose-submodule (current vs 8c776d8)

old moose 8c776d8

Time Step 1, time = 1.833, dt = 0.083

  Executing Controls.                                                                    [ 10.28 s] [52180 MB]



    Computing Initial Residual
      Computing Residual..                                                               [ 19.04 s] [52364 MB]
    Finished Computing Initial Residual                                                  [ 19.04 s] [52364 MB]
    Computing Residual..                                                                 [ 18.69 s] [52365 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 7.38048e+13

 0 Nonlinear |R| = 7.380481e+13

    Computing Jacobian
      Executing Controls.                                                                [ 10.19 s] [52365 MB]
    Still Computing Jacobian...                                                          [ 33.42 s] [52365 MB]

      0 Linear |R| = 7.380481e+13
      1 Linear |R| = 3.720675e+11
      2 Linear |R| = 1.254415e+11
      3 Linear |R| = 4.925024e+10
      4 Linear |R| = 1.800967e+10
      5 Linear |R| = 6.676370e+09
      6 Linear |R| = 6.396018e+09
      7 Linear |R| = 2.179829e+09
      8 Linear |R| = 7.906879e+08
      9 Linear |R| = 3.987515e+08
     10 Linear |R| = 3.036119e+08
     11 Linear |R| = 2.503080e+08
     12 Linear |R| = 1.728642e+08
     13 Linear |R| = 1.051660e+08
     14 Linear |R| = 7.462272e+07
     15 Linear |R| = 5.744832e+07
     16 Linear |R| = 3.984994e+07
     17 Linear |R| = 2.520098e+07
     18 Linear |R| = 1.940504e+07
     19 Linear |R| = 1.580070e+07
     20 Linear |R| = 1.343324e+07
     21 Linear |R| = 1.171208e+07
     22 Linear |R| = 1.042212e+07
     23 Linear |R| = 9.357395e+06
     24 Linear |R| = 7.844318e+06
     25 Linear |R| = 5.975674e+06


    Computing Residual..                                                                 [ 18.90 s] [52427 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 5.59173e+12

 1 Nonlinear |R| = 5.591733e+12

    Computing Jacobian
      Executing Controls.                                                                [ 10.25 s] [52427 MB]
    Still Computing Jacobian..                                                           [ 27.18 s] [52427 MB]

      0 Linear |R| = 5.591733e+12
      1 Linear |R| = 2.817171e+10
      2 Linear |R| = 9.503947e+09
      3 Linear |R| = 3.756188e+09
      4 Linear |R| = 1.379314e+09
      5 Linear |R| = 5.133807e+08
      6 Linear |R| = 4.920184e+08
      7 Linear |R| = 1.665441e+08
      8 Linear |R| = 5.878113e+07
      9 Linear |R| = 2.709168e+07
     10 Linear |R| = 1.963950e+07
     11 Linear |R| = 1.666981e+07
     12 Linear |R| = 1.305705e+07
     13 Linear |R| = 9.731031e+06
     14 Linear |R| = 8.192687e+06
     15 Linear |R| = 7.326149e+06
     16 Linear |R| = 6.166595e+06
     17 Linear |R| = 5.082912e+06
     18 Linear |R| = 4.561948e+06
     19 Linear |R| = 3.949898e+06
     20 Linear |R| = 3.315544e+06
     21 Linear |R| = 2.682667e+06
     22 Linear |R| = 2.105884e+06
     23 Linear |R| = 1.629972e+06
     24 Linear |R| = 1.090884e+06
     25 Linear |R| = 7.046941e+05
     26 Linear |R| = 5.332410e+05


    Computing Residual.                                                                  [ 18.72 s] [52429 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 1.35934e+08

 2 Nonlinear |R| = 1.359343e+08

    Computing Jacobian
      Executing Controls.                                                                [ 10.29 s] [52429 MB]
    Still Computing Jacobian..                                                           [ 27.06 s] [52429 MB]

      0 Linear |R| = 1.359343e+08
      1 Linear |R| = 2.827777e+07
      2 Linear |R| = 4.020453e+06
      3 Linear |R| = 1.296405e+06
      4 Linear |R| = 6.605339e+05
      5 Linear |R| = 5.785215e+05
      6 Linear |R| = 5.690869e+05
      7 Linear |R| = 5.125083e+05
      8 Linear |R| = 3.985427e+05
      9 Linear |R| = 2.934734e+05
     10 Linear |R| = 2.456820e+05
     11 Linear |R| = 2.184374e+05
     12 Linear |R| = 1.821157e+05
     13 Linear |R| = 1.597602e+05
     14 Linear |R| = 1.465883e+05
     15 Linear |R| = 1.274646e+05
     16 Linear |R| = 1.068457e+05
     17 Linear |R| = 9.477471e+04
     18 Linear |R| = 8.117203e+04
     19 Linear |R| = 6.081105e+04
     20 Linear |R| = 5.248531e+04
     21 Linear |R| = 5.027332e+04
     22 Linear |R| = 4.894493e+04
     23 Linear |R| = 4.779100e+04
     24 Linear |R| = 4.736635e+04
     25 Linear |R| = 4.736106e+04
     26 Linear |R| = 4.640053e+04
     27 Linear |R| = 4.391674e+04
     28 Linear |R| = 4.152090e+04
     29 Linear |R| = 3.880270e+04
     30 Linear |R| = 3.563013e+04
     31 Linear |R| = 3.402201e+04
     32 Linear |R| = 3.273588e+04
     33 Linear |R| = 3.184190e+04
     34 Linear |R| = 3.097622e+04
     35 Linear |R| = 3.061087e+04
     36 Linear |R| = 3.055569e+04
     37 Linear |R| = 3.044372e+04
     38 Linear |R| = 3.043495e+04
     39 Linear |R| = 3.041891e+04
     40 Linear |R| = 3.030292e+04
     41 Linear |R| = 2.995032e+04
     42 Linear |R| = 2.935002e+04
     43 Linear |R| = 2.849810e+04
     44 Linear |R| = 2.753399e+04
     45 Linear |R| = 2.655116e+04
     46 Linear |R| = 2.602818e+04
     47 Linear |R| = 2.588978e+04
     48 Linear |R| = 2.581278e+04
     49 Linear |R| = 2.566896e+04
     50 Linear |R| = 2.556381e+04
     51 Linear |R| = 2.554928e+04
     52 Linear |R| = 2.554926e+04
     53 Linear |R| = 2.554362e+04
     54 Linear |R| = 2.553974e+04
     55 Linear |R| = 2.553580e+04
     56 Linear |R| = 2.550884e+04
     57 Linear |R| = 2.540196e+04
     58 Linear |R| = 2.527670e+04
     59 Linear |R| = 2.517381e+04
     60 Linear |R| = 2.511611e+04
     61 Linear |R| = 2.510296e+04
     62 Linear |R| = 2.509814e+04
     63 Linear |R| = 2.501215e+04
     64 Linear |R| = 2.495319e+04
     65 Linear |R| = 2.493811e+04
     66 Linear |R| = 2.493588e+04
     67 Linear |R| = 2.492640e+04
     68 Linear |R| = 2.492364e+04
     69 Linear |R| = 2.492111e+04
     70 Linear |R| = 2.491984e+04
     71 Linear |R| = 2.491982e+04
     72 Linear |R| = 2.491821e+04
     73 Linear |R| = 2.491074e+04
     74 Linear |R| = 2.489401e+04
     75 Linear |R| = 2.487410e+04
     76 Linear |R| = 2.485212e+04
     77 Linear |R| = 2.483084e+04
     78 Linear |R| = 2.480848e+04
     79 Linear |R| = 2.479260e+04
     80 Linear |R| = 2.478150e+04
     81 Linear |R| = 2.477475e+04
     82 Linear |R| = 2.477202e+04
     83 Linear |R| = 2.477006e+04
     84 Linear |R| = 2.476879e+04
     85 Linear |R| = 2.476818e+04
     86 Linear |R| = 2.476797e+04
     87 Linear |R| = 2.476795e+04
     88 Linear |R| = 2.476791e+04
     89 Linear |R| = 2.476761e+04
     90 Linear |R| = 2.476662e+04
     91 Linear |R| = 2.476600e+04
     92 Linear |R| = 2.476569e+04
     93 Linear |R| = 2.476560e+04
     94 Linear |R| = 2.476560e+04
     95 Linear |R| = 2.476557e+04
     96 Linear |R| = 2.476557e+04
     97 Linear |R| = 2.476539e+04
     98 Linear |R| = 2.476505e+04
     99 Linear |R| = 2.476452e+04
    100 Linear |R| = 2.476369e+04
  Linear solve did not converge due to DIVERGED_ITS iterations 100


    Computing Residual..                                                                 [ 18.70 s] [52433 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 1.44824e+07

 3 Nonlinear |R| = 1.448238e+07
Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 3

 Solve Converged!

  Finished Solving                                                                       [261.26 s] [52433 MB]

current moose

Time Step 1, time = 1.833, dt = 0.083

  Executing Controls                                                                     [  9.73 s] [50506 MB]



    Computing Pre-SMO Residual
      Computing Residual..                                                               [ 18.68 s] [50693 MB]

Pre-SMO residual: 7.38819e+13

    Finished Computing Pre-SMO Residual                                                  [ 18.68 s] [50693 MB]
    Computing Residual..                                                                 [ 18.76 s] [50693 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 7.38817e+13

 0 Nonlinear |R| = 7.388166e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.60 s] [50693 MB]
    Still Computing Jacobian...                                                          [ 32.91 s] [50693 MB]

      0 Linear |R| = 7.388166e+13
      1 Linear |R| = 3.733504e+11
      2 Linear |R| = 1.260424e+11
      3 Linear |R| = 4.926991e+10
      4 Linear |R| = 2.267720e+10
      5 Linear |R| = 2.057737e+10
      6 Linear |R| = 6.759040e+09
      7 Linear |R| = 2.566590e+09
      8 Linear |R| = 1.444516e+09
      9 Linear |R| = 1.132049e+09
     10 Linear |R| = 9.045942e+08
     11 Linear |R| = 5.844577e+08
     12 Linear |R| = 3.965591e+08
     13 Linear |R| = 3.287753e+08
     14 Linear |R| = 2.822538e+08
     15 Linear |R| = 2.378113e+08
     16 Linear |R| = 2.113608e+08
     17 Linear |R| = 1.973534e+08
     18 Linear |R| = 1.887242e+08
     19 Linear |R| = 1.823290e+08
     20 Linear |R| = 1.769259e+08
     21 Linear |R| = 1.713424e+08
     22 Linear |R| = 1.675969e+08
     23 Linear |R| = 1.651754e+08
     24 Linear |R| = 1.629609e+08
     25 Linear |R| = 1.612301e+08
     26 Linear |R| = 1.596335e+08
     27 Linear |R| = 1.583508e+08
     28 Linear |R| = 1.572229e+08
     29 Linear |R| = 1.555715e+08
     30 Linear |R| = 1.538618e+08
     31 Linear |R| = 1.528400e+08
     32 Linear |R| = 1.518562e+08
     33 Linear |R| = 1.511202e+08
     34 Linear |R| = 1.500224e+08
     35 Linear |R| = 1.487995e+08
     36 Linear |R| = 1.487269e+08
     37 Linear |R| = 1.480599e+08
     38 Linear |R| = 1.472862e+08
     39 Linear |R| = 1.462387e+08
     40 Linear |R| = 1.444704e+08
     41 Linear |R| = 1.422823e+08
     42 Linear |R| = 1.407045e+08
     43 Linear |R| = 1.385644e+08
     44 Linear |R| = 1.348543e+08
     45 Linear |R| = 1.311320e+08
     46 Linear |R| = 1.277760e+08
     47 Linear |R| = 1.222609e+08
     48 Linear |R| = 1.136556e+08
     49 Linear |R| = 1.060376e+08
     50 Linear |R| = 9.879759e+07
     51 Linear |R| = 9.275633e+07
     52 Linear |R| = 8.905395e+07
     53 Linear |R| = 8.635530e+07
     54 Linear |R| = 8.394804e+07
     55 Linear |R| = 8.191572e+07
     56 Linear |R| = 7.972789e+07
     57 Linear |R| = 7.660939e+07
     58 Linear |R| = 7.330479e+07
     59 Linear |R| = 6.986137e+07
     60 Linear |R| = 6.453021e+07
     61 Linear |R| = 6.167031e+07
     62 Linear |R| = 5.946889e+07
     63 Linear |R| = 5.747792e+07
     64 Linear |R| = 5.617373e+07
     65 Linear |R| = 5.533845e+07
     66 Linear |R| = 5.522277e+07
     67 Linear |R| = 5.446259e+07
     68 Linear |R| = 5.359906e+07
     69 Linear |R| = 5.249143e+07
     70 Linear |R| = 5.143509e+07
     71 Linear |R| = 5.025167e+07
     72 Linear |R| = 4.886324e+07
     73 Linear |R| = 4.731705e+07
     74 Linear |R| = 4.606945e+07
     75 Linear |R| = 4.516937e+07
     76 Linear |R| = 4.457003e+07
     77 Linear |R| = 4.397629e+07
     78 Linear |R| = 4.324198e+07
     79 Linear |R| = 4.268887e+07
     80 Linear |R| = 4.217426e+07
     81 Linear |R| = 4.165707e+07
     82 Linear |R| = 4.122138e+07
     83 Linear |R| = 4.074918e+07
     84 Linear |R| = 4.034092e+07
     85 Linear |R| = 3.999418e+07
     86 Linear |R| = 3.953260e+07
     87 Linear |R| = 3.912674e+07
     88 Linear |R| = 3.886832e+07
     89 Linear |R| = 3.859983e+07
     90 Linear |R| = 3.835384e+07
     91 Linear |R| = 3.819630e+07
     92 Linear |R| = 3.803917e+07
     93 Linear |R| = 3.785614e+07
     94 Linear |R| = 3.769001e+07
     95 Linear |R| = 3.766517e+07
     96 Linear |R| = 3.755430e+07
     97 Linear |R| = 3.743751e+07
     98 Linear |R| = 3.732686e+07
     99 Linear |R| = 3.722430e+07
    100 Linear |R| = 3.714092e+07
  Linear solve did not converge due to DIVERGED_ITS iterations 100


    Computing Residual..                                                                 [ 18.20 s] [50756 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 7.24332e+13

 1 Nonlinear |R| = 7.243324e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.68 s] [50756 MB]
    Still Computing Jacobian..                                                           [ 25.82 s] [50756 MB]

      0 Linear |R| = 7.243324e+13
      1 Linear |R| = 3.654422e+11
      2 Linear |R| = 1.236631e+11
      3 Linear |R| = 4.898308e+10
      4 Linear |R| = 2.257844e+10
      5 Linear |R| = 2.049187e+10
      6 Linear |R| = 6.793798e+09
      7 Linear |R| = 2.558853e+09
      8 Linear |R| = 1.413870e+09
      9 Linear |R| = 1.096966e+09
     10 Linear |R| = 8.745039e+08
     11 Linear |R| = 5.561600e+08
     12 Linear |R| = 3.591150e+08
     13 Linear |R| = 2.859984e+08
     14 Linear |R| = 2.308822e+08
     15 Linear |R| = 1.719023e+08
     16 Linear |R| = 1.310853e+08
     17 Linear |R| = 1.052678e+08
     18 Linear |R| = 8.684565e+07
     19 Linear |R| = 7.352416e+07
     20 Linear |R| = 6.265816e+07
     21 Linear |R| = 4.994757e+07
     22 Linear |R| = 4.077404e+07
     23 Linear |R| = 3.483568e+07
     24 Linear |R| = 2.939342e+07
     25 Linear |R| = 2.464322e+07
     26 Linear |R| = 1.976382e+07
     27 Linear |R| = 1.610597e+07
     28 Linear |R| = 1.430543e+07
     29 Linear |R| = 1.283520e+07
     30 Linear |R| = 1.182885e+07
     31 Linear |R| = 1.139516e+07
     32 Linear |R| = 1.098219e+07
     33 Linear |R| = 1.077064e+07
     34 Linear |R| = 1.059577e+07
     35 Linear |R| = 1.045275e+07
     36 Linear |R| = 1.044920e+07
     37 Linear |R| = 1.035980e+07
     38 Linear |R| = 1.028193e+07
     39 Linear |R| = 1.023212e+07
     40 Linear |R| = 1.018970e+07
     41 Linear |R| = 1.014876e+07
     42 Linear |R| = 1.011777e+07
     43 Linear |R| = 1.009521e+07
     44 Linear |R| = 1.006856e+07
     45 Linear |R| = 1.003647e+07
     46 Linear |R| = 1.001605e+07
     47 Linear |R| = 9.995897e+06
     48 Linear |R| = 9.968740e+06
     49 Linear |R| = 9.945233e+06
     50 Linear |R| = 9.917240e+06
     51 Linear |R| = 9.878895e+06
     52 Linear |R| = 9.833167e+06
     53 Linear |R| = 9.767643e+06
     54 Linear |R| = 9.663246e+06
     55 Linear |R| = 9.543895e+06
     56 Linear |R| = 9.405934e+06
     57 Linear |R| = 9.129376e+06
     58 Linear |R| = 8.747407e+06
     59 Linear |R| = 8.387055e+06
     60 Linear |R| = 8.091923e+06
     61 Linear |R| = 7.952405e+06
     62 Linear |R| = 7.804966e+06
     63 Linear |R| = 7.637440e+06
     64 Linear |R| = 7.501918e+06
     65 Linear |R| = 7.475204e+06
     66 Linear |R| = 7.408377e+06
     67 Linear |R| = 7.305041e+06
     68 Linear |R| = 7.190163e+06


    Computing Residual..                                                                 [ 18.77 s] [50759 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 6.98022e+13

 2 Nonlinear |R| = 6.980222e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.65 s] [50759 MB]
    Still Computing Jacobian..                                                           [ 26.46 s] [50759 MB]

      0 Linear |R| = 6.980222e+13
      1 Linear |R| = 3.523360e+11
      2 Linear |R| = 1.187674e+11
      3 Linear |R| = 4.641056e+10
      4 Linear |R| = 2.063926e+10
      5 Linear |R| = 1.931454e+10
      6 Linear |R| = 6.399603e+09
      7 Linear |R| = 2.429474e+09
      8 Linear |R| = 1.354337e+09
      9 Linear |R| = 1.054395e+09
     10 Linear |R| = 8.387192e+08
     11 Linear |R| = 5.331993e+08
     12 Linear |R| = 3.459104e+08
     13 Linear |R| = 2.724231e+08
     14 Linear |R| = 2.161811e+08
     15 Linear |R| = 1.582752e+08
     16 Linear |R| = 1.207155e+08
     17 Linear |R| = 9.847684e+07
     18 Linear |R| = 8.300487e+07
     19 Linear |R| = 6.952539e+07
     20 Linear |R| = 5.737831e+07
     21 Linear |R| = 4.456699e+07
     22 Linear |R| = 3.531113e+07
     23 Linear |R| = 2.937306e+07
     24 Linear |R| = 2.428257e+07
     25 Linear |R| = 2.005064e+07
     26 Linear |R| = 1.566085e+07
     27 Linear |R| = 1.257040e+07
     28 Linear |R| = 1.087213e+07
     29 Linear |R| = 9.516468e+06
     30 Linear |R| = 8.617314e+06
     31 Linear |R| = 8.176587e+06
     32 Linear |R| = 7.735984e+06
     33 Linear |R| = 7.516513e+06
     34 Linear |R| = 7.344059e+06
     35 Linear |R| = 7.315355e+06
     36 Linear |R| = 7.220488e+06
     37 Linear |R| = 7.122507e+06
     38 Linear |R| = 7.039379e+06
     39 Linear |R| = 6.982167e+06
     40 Linear |R| = 6.928032e+06


    Computing Residual..                                                                 [ 18.46 s] [50760 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 6.716e+13

 3 Nonlinear |R| = 6.716001e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.72 s] [50760 MB]
    Still Computing Jacobian..                                                           [ 26.36 s] [50760 MB]

      0 Linear |R| = 6.716001e+13
      1 Linear |R| = 3.390190e+11
      2 Linear |R| = 1.140532e+11
      3 Linear |R| = 4.465943e+10
      4 Linear |R| = 1.980442e+10
      5 Linear |R| = 1.858126e+10
      6 Linear |R| = 6.166643e+09
      7 Linear |R| = 2.344652e+09
      8 Linear |R| = 1.307791e+09
      9 Linear |R| = 1.017537e+09
     10 Linear |R| = 8.055736e+08
     11 Linear |R| = 5.118278e+08
     12 Linear |R| = 3.359680e+08
     13 Linear |R| = 2.659011e+08
     14 Linear |R| = 2.072922e+08
     15 Linear |R| = 1.470151e+08
     16 Linear |R| = 1.106445e+08
     17 Linear |R| = 9.089508e+07
     18 Linear |R| = 7.785710e+07
     19 Linear |R| = 6.623703e+07
     20 Linear |R| = 5.475839e+07
     21 Linear |R| = 4.218830e+07
     22 Linear |R| = 3.348988e+07
     23 Linear |R| = 2.813884e+07
     24 Linear |R| = 2.337734e+07
     25 Linear |R| = 1.914206e+07
     26 Linear |R| = 1.476005e+07
     27 Linear |R| = 1.211351e+07
     28 Linear |R| = 1.102093e+07
     29 Linear |R| = 1.032689e+07
     30 Linear |R| = 9.896076e+06
     31 Linear |R| = 9.675687e+06
     32 Linear |R| = 9.395628e+06
     33 Linear |R| = 9.204130e+06
     34 Linear |R| = 9.024043e+06
     35 Linear |R| = 8.972721e+06
     36 Linear |R| = 8.894957e+06
     37 Linear |R| = 8.800796e+06
     38 Linear |R| = 8.732640e+06
     39 Linear |R| = 8.694064e+06
     40 Linear |R| = 8.660222e+06
     41 Linear |R| = 8.633437e+06
     42 Linear |R| = 8.622732e+06
     43 Linear |R| = 8.616945e+06
     44 Linear |R| = 8.612579e+06
     45 Linear |R| = 8.609495e+06
     46 Linear |R| = 8.608607e+06
     47 Linear |R| = 8.608599e+06
     48 Linear |R| = 8.608420e+06
     49 Linear |R| = 8.607592e+06
     50 Linear |R| = 8.604176e+06
     51 Linear |R| = 8.592342e+06
     52 Linear |R| = 8.558983e+06
     53 Linear |R| = 8.488054e+06
     54 Linear |R| = 8.368367e+06
     55 Linear |R| = 8.226839e+06
     56 Linear |R| = 8.044048e+06
     57 Linear |R| = 7.561557e+06
     58 Linear |R| = 6.753114e+06
     59 Linear |R| = 5.816972e+06


    Computing Residual.                                                                  [ 18.21 s] [50762 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 6.45144e+13

 4 Nonlinear |R| = 6.451436e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.75 s] [50762 MB]
    Still Computing Jacobian..                                                           [ 26.18 s] [50762 MB]

      0 Linear |R| = 6.451436e+13
      1 Linear |R| = 3.256762e+11
      2 Linear |R| = 1.093830e+11
      3 Linear |R| = 4.278308e+10
      4 Linear |R| = 1.906654e+10
      5 Linear |R| = 1.798952e+10
      6 Linear |R| = 5.994491e+09
      7 Linear |R| = 2.275242e+09
      8 Linear |R| = 1.270941e+09
      9 Linear |R| = 9.881745e+08
     10 Linear |R| = 7.873919e+08
     11 Linear |R| = 5.030104e+08
     12 Linear |R| = 3.316261e+08
     13 Linear |R| = 2.631256e+08
     14 Linear |R| = 2.087163e+08
     15 Linear |R| = 1.506486e+08
     16 Linear |R| = 1.138164e+08
     17 Linear |R| = 9.376965e+07
     18 Linear |R| = 8.172976e+07
     19 Linear |R| = 7.112791e+07
     20 Linear |R| = 5.961880e+07
     21 Linear |R| = 4.714547e+07
     22 Linear |R| = 3.867749e+07
     23 Linear |R| = 3.321782e+07
     24 Linear |R| = 2.783046e+07
     25 Linear |R| = 2.213689e+07
     26 Linear |R| = 1.608261e+07
     27 Linear |R| = 1.217810e+07
     28 Linear |R| = 1.002160e+07
     29 Linear |R| = 8.105572e+06
     30 Linear |R| = 6.832895e+06
     31 Linear |R| = 6.156868e+06


    Computing Residual..                                                                 [ 18.48 s] [50764 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 6.18675e+13

 5 Nonlinear |R| = 6.186752e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.62 s] [50764 MB]
    Still Computing Jacobian..                                                           [ 26.54 s] [50764 MB]

      0 Linear |R| = 6.186752e+13
      1 Linear |R| = 3.121567e+11
      2 Linear |R| = 1.053992e+11
      3 Linear |R| = 4.146173e+10
      4 Linear |R| = 1.830122e+10
      5 Linear |R| = 1.731596e+10
      6 Linear |R| = 5.781862e+09
      7 Linear |R| = 2.193993e+09
      8 Linear |R| = 1.220005e+09
      9 Linear |R| = 9.472894e+08
     10 Linear |R| = 7.487798e+08
     11 Linear |R| = 4.688921e+08
     12 Linear |R| = 3.040223e+08
     13 Linear |R| = 2.423569e+08
     14 Linear |R| = 1.932007e+08
     15 Linear |R| = 1.402663e+08
     16 Linear |R| = 1.056736e+08
     17 Linear |R| = 8.533132e+07
     18 Linear |R| = 7.253285e+07
     19 Linear |R| = 6.303904e+07
     20 Linear |R| = 5.368170e+07
     21 Linear |R| = 4.175618e+07
     22 Linear |R| = 3.306897e+07
     23 Linear |R| = 2.783436e+07
     24 Linear |R| = 2.281090e+07
     25 Linear |R| = 1.779244e+07
     26 Linear |R| = 1.269714e+07
     27 Linear |R| = 9.307828e+06
     28 Linear |R| = 7.685094e+06
     29 Linear |R| = 6.157873e+06


    Computing Residual.                                                                  [ 18.65 s] [50765 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 5.92214e+13

 6 Nonlinear |R| = 5.922141e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.62 s] [50765 MB]
    Still Computing Jacobian..                                                           [ 26.02 s] [50765 MB]

      0 Linear |R| = 5.922141e+13
      1 Linear |R| = 2.988746e+11
      2 Linear |R| = 1.011788e+11
      3 Linear |R| = 3.971373e+10
      4 Linear |R| = 1.731784e+10
      5 Linear |R| = 1.649016e+10
      6 Linear |R| = 5.505707e+09
      7 Linear |R| = 2.080908e+09
      8 Linear |R| = 1.162502e+09
      9 Linear |R| = 9.029988e+08
     10 Linear |R| = 7.145095e+08
     11 Linear |R| = 4.498287e+08
     12 Linear |R| = 2.944696e+08
     13 Linear |R| = 2.346165e+08
     14 Linear |R| = 1.883611e+08
     15 Linear |R| = 1.389579e+08
     16 Linear |R| = 1.052216e+08
     17 Linear |R| = 8.342113e+07
     18 Linear |R| = 6.927081e+07
     19 Linear |R| = 5.945320e+07
     20 Linear |R| = 5.097309e+07
     21 Linear |R| = 4.056180e+07
     22 Linear |R| = 3.266478e+07
     23 Linear |R| = 2.752025e+07
     24 Linear |R| = 2.250288e+07
     25 Linear |R| = 1.796434e+07
     26 Linear |R| = 1.269930e+07
     27 Linear |R| = 8.887491e+06
     28 Linear |R| = 7.050097e+06
     29 Linear |R| = 5.634193e+06


    Computing Residual..                                                                 [ 18.83 s] [50765 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 5.65763e+13

 7 Nonlinear |R| = 5.657626e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.73 s] [50765 MB]
    Still Computing Jacobian..                                                           [ 26.50 s] [50765 MB]

      0 Linear |R| = 5.657626e+13
      1 Linear |R| = 2.858632e+11
      2 Linear |R| = 9.671258e+10
      3 Linear |R| = 3.800687e+10
      4 Linear |R| = 1.646923e+10
      5 Linear |R| = 1.577033e+10
      6 Linear |R| = 5.270549e+09
      7 Linear |R| = 1.990804e+09
      8 Linear |R| = 1.101216e+09
      9 Linear |R| = 8.523076e+08
     10 Linear |R| = 6.756843e+08
     11 Linear |R| = 4.300816e+08
     12 Linear |R| = 2.803099e+08
     13 Linear |R| = 2.230871e+08
     14 Linear |R| = 1.768939e+08
     15 Linear |R| = 1.265775e+08
     16 Linear |R| = 9.443118e+07
     17 Linear |R| = 7.614234e+07
     18 Linear |R| = 6.418230e+07
     19 Linear |R| = 5.503160e+07
     20 Linear |R| = 4.683723e+07
     21 Linear |R| = 3.699839e+07
     22 Linear |R| = 3.003246e+07
     23 Linear |R| = 2.565558e+07
     24 Linear |R| = 2.094222e+07
     25 Linear |R| = 1.652051e+07
     26 Linear |R| = 1.182082e+07
     27 Linear |R| = 8.326162e+06
     28 Linear |R| = 6.466828e+06
     29 Linear |R| = 4.983476e+06


    Computing Residual..                                                                 [ 19.18 s] [50766 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 5.39316e+13

 8 Nonlinear |R| = 5.393161e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.79 s] [50766 MB]
    Still Computing Jacobian..                                                           [ 26.10 s] [50766 MB]

      0 Linear |R| = 5.393161e+13
      1 Linear |R| = 2.721092e+11
      2 Linear |R| = 9.159563e+10
      3 Linear |R| = 3.587168e+10
      4 Linear |R| = 1.324932e+10
      5 Linear |R| = 4.271664e+09
      6 Linear |R| = 1.759343e+09
      7 Linear |R| = 1.282957e+09
      8 Linear |R| = 1.274304e+09
      9 Linear |R| = 1.006418e+09
     10 Linear |R| = 8.243501e+08
     11 Linear |R| = 6.019760e+08
     12 Linear |R| = 3.690396e+08
     13 Linear |R| = 2.606965e+08
     14 Linear |R| = 2.106334e+08
     15 Linear |R| = 1.628711e+08
     16 Linear |R| = 1.203183e+08
     17 Linear |R| = 9.253857e+07
     18 Linear |R| = 7.536836e+07
     19 Linear |R| = 6.163290e+07
     20 Linear |R| = 5.261479e+07
     21 Linear |R| = 4.251854e+07
     22 Linear |R| = 3.182510e+07
     23 Linear |R| = 2.524098e+07
     24 Linear |R| = 2.062385e+07
     25 Linear |R| = 1.673889e+07
     26 Linear |R| = 1.332470e+07
     27 Linear |R| = 9.811176e+06
     28 Linear |R| = 7.799290e+06
     29 Linear |R| = 6.242724e+06
     30 Linear |R| = 4.961267e+06


    Computing Residual..                                                                 [ 18.51 s] [50767 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 5.1285e+13

 9 Nonlinear |R| = 5.128504e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.60 s] [50767 MB]
    Still Computing Jacobian..                                                           [ 26.28 s] [50767 MB]

      0 Linear |R| = 5.128504e+13
      1 Linear |R| = 2.592503e+11
      2 Linear |R| = 8.723042e+10
      3 Linear |R| = 3.416652e+10
      4 Linear |R| = 1.259428e+10
      5 Linear |R| = 4.079403e+09
      6 Linear |R| = 1.674010e+09
      7 Linear |R| = 1.196978e+09
      8 Linear |R| = 1.194240e+09
      9 Linear |R| = 9.585337e+08
     10 Linear |R| = 7.888072e+08
     11 Linear |R| = 5.819862e+08
     12 Linear |R| = 3.658361e+08
     13 Linear |R| = 2.637431e+08
     14 Linear |R| = 2.131924e+08
     15 Linear |R| = 1.628966e+08
     16 Linear |R| = 1.183153e+08
     17 Linear |R| = 9.154716e+07
     18 Linear |R| = 7.724322e+07
     19 Linear |R| = 6.587665e+07
     20 Linear |R| = 5.693803e+07
     21 Linear |R| = 4.701185e+07
     22 Linear |R| = 3.752199e+07
     23 Linear |R| = 3.139946e+07
     24 Linear |R| = 2.647978e+07
     25 Linear |R| = 2.140018e+07
     26 Linear |R| = 1.581000e+07
     27 Linear |R| = 1.113488e+07
     28 Linear |R| = 8.973949e+06
     29 Linear |R| = 7.136740e+06
     30 Linear |R| = 5.654792e+06
     31 Linear |R| = 4.894344e+06


    Computing Residual.                                                                  [ 18.32 s] [50767 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 4.8637e+13

10 Nonlinear |R| = 4.863702e+13

    Computing Jacobian
      Executing Controls                                                                 [  9.69 s] [50767 MB]
    Still Computing Jacobian..                                                           [ 26.29 s] [50767 MB]

      0 Linear |R| = 4.863702e+13
      1 Linear |R| = 2.455052e+11
      2 Linear |R| = 8.290248e+10
      3 Linear |R| = 3.245398e+10
      4 Linear |R| = 1.194027e+10
      5 Linear |R| = 3.863013e+09
      6 Linear |R| = 1.582437e+09
      7 Linear |R| = 1.125791e+09
      8 Linear |R| = 1.124910e+09
      9 Linear |R| = 9.138065e+08
     10 Linear |R| = 7.526897e+08
     11 Linear |R| = 5.506308e+08
     12 Linear |R| = 3.354266e+08
     13 Linear |R| = 2.354881e+08
     14 Linear |R| = 1.896384e+08
     15 Linear |R| = 1.472313e+08
     16 Linear |R| = 1.101864e+08
     17 Linear |R| = 8.457774e+07
     18 Linear |R| = 6.844279e+07
     19 Linear |R| = 5.599294e+07
     20 Linear |R| = 4.756799e+07
     21 Linear |R| = 3.826863e+07
     22 Linear |R| = 2.909173e+07
     23 Linear |R| = 2.307805e+07
     24 Linear |R| = 1.845484e+07
     25 Linear |R| = 1.487552e+07
     26 Linear |R| = 1.189793e+07
     27 Linear |R| = 9.044232e+06
     28 Linear |R| = 7.253943e+06
     29 Linear |R| = 5.666078e+06
     30 Linear |R| = 4.588173e+06


    Computing Residual..                                                                 [ 18.66 s] [50767 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 4.59913e+13

Thank you guys!
Traiwit

[GiudGiud]

Hello

You can check whether the new initial residual behavior is causing the problem by reverting to the old one

It s a parameter in the Problem. Something like use_smo_residual

[oanaoana]

@GiudGiud you mean the lu option? That should be full LU, there's also an ilu option, but I doubt it may help if the solver diverges. Also SuperLU may help iteration count but does no miracles if the solver diverges. It may help if the divergence is due to error accumulation, but not if there's some underlying ill-posedness, and getting to 1e+13 is rarely accumulation of errors.

[GiudGiud]

Yes full LU. I thought we had found that one of the packages took some liberties with LU so it would not converge always in 1 iteration.

I ll fact check myself when I get back

[Traiwit]

Hi, I also tried mumps last night

  # petsc_options_iname = '-pc_type -pc_factor_mat_solver_package'
  # petsc_options_value = ' lu       mumps'

below is convergence for the old MOOSE (the one that works fine).

At this point I dont think the solver isn't the problem

maybe the way that matrix is constructed or something in FE.problem has changed?

Time Step 1, time = 1.833, dt = 0.083

  Executing Controls.                                                                    [ 10.21 s] [51927 MB]
Still Executing


    Computing Initial Residual
      Computing Residual..                                                               [ 18.49 s] [52127 MB]
    Finished Computing Initial Residual                                                  [ 18.49 s] [52127 MB]
    Computing Residual..                                                                 [ 18.19 s] [52128 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 7.38517e+13

 0 Nonlinear |R| = 7.385167e+13

    Computing Jacobian
      Executing Controls.                                                                [ 10.16 s] [52128 MB]
    Still Computing Jacobian...                                                          [ 33.19 s] [52129 MB]

      0 Linear |R| = 7.385167e+13
      1 Linear |R| = 1.560608e+00


    Computing Residual..                                                                 [ 18.40 s] [57196 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 5.66489e+12

 1 Nonlinear |R| = 5.664893e+12

    Computing Jacobian
      Executing Controls.                                                                [ 10.15 s] [57196 MB]
    Still Computing Jacobian..                                                           [ 26.18 s] [57196 MB]

      0 Linear |R| = 5.664893e+12
      1 Linear |R| = 1.577210e-01


    Computing Residual..                                                                 [ 18.62 s] [57196 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 1.43303e+08

 2 Nonlinear |R| = 1.433029e+08

    Computing Jacobian
      Executing Controls.                                                                [ 10.15 s] [57196 MB]
    Still Computing Jacobian..                                                           [ 26.09 s] [57196 MB]

      0 Linear |R| = 1.433029e+08
      1 Linear |R| = 8.028432e-04


    Computing Residual..                                                                 [ 18.45 s] [57195 MB]

    |residual|_2 of individual variables:
            porepressure_L1: 1.44819e+07

 3 Nonlinear |R| = 1.448193e+07
Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 3

 Solve Converged!

[lindsayad]

With a direct solve we expect a single solve iteration for the linear solver

This isn't generally true if using PJFNK.

I thought we had found that one of the packages took some liberties with LU so it would not converge always in 1 iteration.

I think I have sometimes seen PETSc's native LU (the default when running in serial) take multiple iterations even when running with NEWTON. I think I may have even seen it for superlu_dist but only when the problem's in a very "tough" state, e.g. unlikely to converge with mumps either (with mumps you might see an out of memory error in this case ... even though you probably didn't run out of memory). However, you can test all the different factorization packages (mumps, superlu_dist, strumpack, petsc) on something like simple_diffusion.i with NEWTON and see the 15 order of magnitude residual drop you expect.

[lindsayad]

MUMPS is the most accurate as you are seeing in your solver history. That being said, the fact that the linear solve is doing so poorly with an LU preconditioner with factorization packages other than MUMPS is a sign that your problem might be poorly conditioned.

@Traiwit did you try -snes_test_jacobian like I suggested on a smaller problem? You might also try running with -ksp_monitor_singular_values on your full-size problem to get a sense for the condition number. And finally you should really consider running a bisect like @GiudGiud suggested