Questions about parallelization

[NikoYura]

Dear JUDI community,

1. What is the global concept of parallelization in JUDI? Can we make shot-by-shot parallelization using JUDI and Distributed.jl? Now I see that when using Workers, the task seems to be split sequentially without the ability to control individual shots, which may cause some stability issues for large industrial tasks when using multiple nodes (and performance degradation when exchanging data between nodes if our parallelization is dependent, i.e. one shot is distributed to multiple nodes)?

Examples from COFII or SLIMoptim are not suitable, because we lose the ability to use built-in JUDI tools (judiModelling, fwi_objective). That is, we would have to explicitly write out the calculation of all inversion steps. Or is there another way to optimize parallelization for large tasks, using both judimodeling and fwi_objective, and make shot-by-shot parallelization?

2. I also have a question on the topic of using RAM in the FWI process. I notice that iterations are not completely independent and RAM utilization increases during iterations. Is this normal behavior and why?

I'm testing an FWI example from SlimOptim on my data now, where the judiModeling structure is used, but the fwi_objective is not used (I want to use it to reduce the number of forward modeling). For some reason, 6 iterations count normally, I get a good result, but at 7 I get the following error. And I can stop at 6 iteration and additionally, after that calculate some more iterations separately.

TaskFailedException

    nested task error: On worker 3:
    On worker 9:
    InexactError: trunc(Int64, NaN)
    Stacktrace:
      [1] trunc
        @ ./float.jl:905 [inlined]
      [2] round
        @ ./float.jl:385 [inlined]
      [3] Colon
        @ ./twiceprecision.jl:427
      [4] time_resample
        @ ~/.julia/packages/JUDI/BYqzP/src/TimeModeling/Utils/time_utilities.jl:16
      [5] devito_interface
        @ ~/.julia/packages/JUDI/BYqzP/src/TimeModeling/Modeling/python_interface.jl:58
      [6] macro expansion
        @ ~/.julia/packages/JUDI/BYqzP/src/TimeModeling/Modeling/time_modeling_serial.jl:42 [inlined]
      [7] macro expansion
        @ ./timing.jl:395 [inlined]
      [8] macro expansion
        @ ~/.julia/packages/JUDI/BYqzP/src/JUDI.jl:148 [inlined]
      [9] _time_modeling
        @ ~/.julia/packages/JUDI/BYqzP/src/TimeModeling/Modeling/time_modeling_serial.jl:41
     [10] #96#97
        @ ./error.jl:308
     [11] #96
        @ ./error.jl:291
     [12] propagate
        @ ~/.julia/packages/JUDI/BYqzP/src/TimeModeling/Modeling/propagation.jl:9
     [13] #invokelatest#2
        @ ./essentials.jl:892
     [14] invokelatest
        @ ./essentials.jl:889
     [15] #107
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:283
     [16] run_work_thunk
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:70
     [17] run_work_thunk
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:79
     [18] #100
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:88
    Stacktrace:
      [1] #remotecall_fetch#159
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/remotecall.jl:465
      [2] remotecall_fetch
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/remotecall.jl:454
      [3] remotecall_fetch
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/remotecall.jl:492 [inlined]
      [4] local_reduce!
        @ ~/.julia/packages/JUDI/BYqzP/src/TimeModeling/Modeling/distributed.jl:32
      [5] #invokelatest#2
        @ ./essentials.jl:892
      [6] invokelatest
        @ ./essentials.jl:889
      [7] #114
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:303
      [8] run_work_thunk
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:70
      [9] run_work_thunk
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:79
     [10] #100
        @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:88
    Stacktrace:
     [1] remotecall_wait(::Function, ::Distributed.Worker, ::Future, ::Vararg{Future}; kwargs::@Kwargs{})
       @ Distributed /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/remotecall.jl:507
     [2] remotecall_wait
       @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/remotecall.jl:497 [inlined]
     [3] remotecall_wait
       @ /snap/julia/118/share/julia/stdlib/v1.10/Distributed/src/remotecall.jl:519 [inlined]
     [4] (::JUDI.var"#287#288"{Vector{Future}, Int64})()
       @ JUDI ~/.julia/packages/JUDI/BYqzP/src/TimeModeling/Modeling/distributed.jl:48

...and 3 more exceptions.


Stacktrace:
 [1] linesearch(ls::BackTracking{Float32, Int64}, sol::result{Float32}, d::Vector{Float32}, f::SlimOptim.var"#obj#24"{SlimOptim.var"#objgrad!#23"{typeof(g), result{Float32}}, Vector{Float32}}, g!::SlimOptim.var"#grad!#22"{typeof(g), result{Float32}}, fg!::SlimOptim.var"#objgrad!#23"{typeof(g), result{Float32}}, t::Float32, funRef::Float32, gtd::Float32, gvec::Vector{Float32})
   @ SlimOptim ~/devito_mpi/burservis/SlimOptim.jl-master/src/linesearches.jl:51
 [2] _pqn(obj::Function, grad!::SlimOptim.var"#grad!#22"{typeof(g), result{Float32}}, objgrad!::SlimOptim.var"#objgrad!#23"{typeof(g), result{Float32}}, projection::SlimOptim.var"#projection#21"{typeof(prj), result{Float32}}, x::Vector{Float32}, g::Vector{Float32}, sol::result{Float32}, ls::Nothing, options::SlimOptim.PQN_params; callback::typeof(SlimOptim.noop_callback))
   @ SlimOptim ~/devito_mpi/burservis/SlimOptim.jl-master/src/PQNSlim.jl:262
 [3] pqn(funObj::typeof(g), x::Vector{Float32}, funProj::typeof(prj), options::SlimOptim.PQN_params; ls::Nothing, callback::Function)
   @ SlimOptim ~/devito_mpi/burservis/SlimOptim.jl-master/src/PQNSlim.jl:95
 [4] pqn(funObj::Function, x::Vector{Float32}, funProj::Function, options::SlimOptim.PQN_params)
   @ SlimOptim ~/devito_mpi/burservis/SlimOptim.jl-master/src/PQNSlim.jl:83
 [5] top-level scope
   @ In[46]:1

[mloubout]

the task seems to be split sequentially without the ability to control individual shots which may cause some stability issues for large industrial tasks

I am not sure what you mean by that. JUDI run shot distribution (one worker per shot) so I'm not sure what is "sequential" here or why it would be unstable

, i.e. one shot is distributed to multiple nodes

This is not supported by JUDI at the time

Examples from COFII or SLIMoptim are not suitable, because we lose the ability to use built-in JUDI tools (judiModelling, fwi_objective)

I don't think that's true, there is plenty of examples in JUDI using SlimOptim with both judiModelling and fwi_objective. This is what it's designed for. So if you have a specific example that doesn't work could you share it.

I notice that iterations are not completely independent and RAM utilization increases during iterations. Is this normal behavior and why?

By how much, and which algorithm are you using?

I get the following error

This seems to hint that there is some invalid values in your velocity leading to NaN dt, Are you using any bound constraint?

[NikoYura]

Hello and thank you for your reply! I apologize for being inaccurate in my question, I will try to correct myself.

1. I wanted to clarify how top-level parallelization in JUDI is performed through the Distributed.jl package workers. I am focusing on the FWI spg example from the SlimOptim package. I want to find out how the task is divided in this case: one independent shot is completely counted on one worker or is it somehow divided between workers, so who is responsible for this? I've seen approaches from Chevron and tried to figure out for what purpose they designed their forward modeling operators and fully explicit inversion solution. In addition to managing the elastic scalability of a cluster, they explicitly count each shot independently on separate workers. However, I want to use all the advantages of JUDI, including fwi_objective, to optimize the count for a large task and not to write out the inversion steps explicitly (as it is done in the SlimOptim example), while counting each shot entirely on the workers.

2. I think that my RAM usage increases several times during the inversion process (I'm talking about spg/pqn FWI approaches in JUDI examples). Do you think something can stay in RAM from iteration to iteration and thus accumulate? I am also trying to estimate the amount of RAM needed for my task. I took Overthrust 3D as an example, I saw in discussions that you used 500 nodes of 128GB RAM. I take it this was not a minimum requirement, but rather for the fastest possible count? At what time did this task count in?

3. This seems to hint that there is some invalid values in your velocity leading to NaN dt, Are you using any bound constraint?

Yes, I use the constraints from this example, substituting my true model instead of the m used in the example. 6 iterations go well, and the result is adequate, on the 7th iteration the above-mentioned error occurs.

[kerim371]

@NikoYura hi,

Just wanted to say regarding your second point.

I used to experience something like memory leak #262 when doing computations on small cluster with older version of JUDI.

But now with versin v4.0.2 there is no such problem. I do the computations on cluster for a regular survey without memory growth.

[mloubout]

1. JUDI computes one shot per worker and schedules them as resources become available. Still, as an example, if you have three Julia workers (1,2, 3) and three shots (1,2,3), worker 1 computes shot 1, worker 2 shot 2, and worker 3 shot 3, in parallel, then gathers the result. This is full, So if you have a bunch of shots and Julia workers, it will automatically schedule and compute them in parallel with the resources available. This is the case for both the linear operator and fwi_objective, see for example https://github.com/slimgroup/JUDI.jl/blob/master/examples/scripts/fwi_example_minConf.jl

2 Memory is cleared after each shot computation, so there should not be any significant memory increase, just a few Kb, at most Mb, from some Python symbolic cache. Depending on the option you use for SPG/PQN it may store the history of the inversion ,i.e the model at every iteration, so you will get an increase of memory of the model size at each iteration.

3. WHich algorithm leads to that error? It might be that the line-search on the 7th iteration goes wrong and leads to invalid velocity.

[NikoYura]

@kerim371 hi!

Thank you for your reply. Yeah, I had a pretty old version of JUDI. Using the new version, I do not see such a problem anymore.

@mloubout 1) Thank you for the clarification, I got it. I did tests and it works as discussed and expected.

2. Now I see that the problem was with the old version of JUDI.

3. It's definitely linesearch, yes. What is the best way to track such errors? I see that the result of the iteration before the error is pretty good; it's nothing unusual.

4. What is the approximate counting time of the 3D overthrust from your example-article on a supercomputer, I couldn't find the information?