Unhandled Task ERROR: IOError #233

tjk9501 · 2024-03-13T02:49:21Z

tjk9501
Mar 13, 2024

Hello!

Simulation is ran two times, with the following error message reported:
First run:

Worker 8 terminated.
Worker 5 terminated.Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:949
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:955
  [4] unsafe_read
    @ ./io.jl:761 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:760
  [6] read!
    @ ./io.jl:762 [inlined]
  [7] deserialize_hdr_raw
    @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed ./task.jl:514

Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:949
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:955
  [4] unsafe_read
    @ ./io.jl:761 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:760
  [6] read!
    @ ./io.jl:762 [inlined]
  [7] deserialize_hdr_raw
    @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed ./task.jl:514
      From worker 10:	Operator `forward` ran in 24.71 s
      From worker 12:	Operator `forward` ran in 26.16 s
      From worker 12:	Operator `gradient` ran in 37.97 s
      From worker 10:	Operator `gradient` ran in 42.44 s
ERROR: LoadError: TaskFailedException

    nested task error: On worker 12:
    ProcessExitedException(8)
    Stacktrace:
      [1] worker_from_id
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/cluster.jl:1093
      [2] worker_from_id
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/cluster.jl:1090 [inlined]
      [3] #remotecall_fetch#162
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:492 [inlined]
      [4] remotecall_fetch
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:492 [inlined]
      [5] local_reduce!
        @ ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/distributed.jl:38
      [6] #invokelatest#2
        @ ./essentials.jl:819
      [7] invokelatest
        @ ./essentials.jl:816
      [8] #114
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:302
      [9] run_work_thunk
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
     [10] run_work_thunk
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:79
     [11] #100
        @ ./task.jl:514
    Stacktrace:
     [1] remotecall_wait(::Function, ::Distributed.Worker, ::Future, ::Vararg{Future}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
       @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:507
     [2] remotecall_wait
       @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:497 [inlined]
     [3] #remotecall_wait#167
       @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:519 [inlined]
     [4] remotecall_wait
       @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:519 [inlined]
     [5] (::JUDI.var"#237#238"{Vector{Future}, Int64})()
       @ JUDI ./task.jl:514

...and 1 more exception.

Stacktrace:
  [1] sync_end(c::Channel{Any})
    @ Base ./task.jl:445
  [2] macro expansion
    @ ./task.jl:477 [inlined]
  [3] reduce_level!(futures::Vector{Future}, nleaf::Int64)
    @ JUDI ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/distributed.jl:53
  [4] reduce!(futures::Vector{Future})
    @ JUDI ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/distributed.jl:79
  [5] run_and_reduce(func::Function, pool::WorkerPool, nsrc::Int64, arg_func::JUDI.var"#273#274"{judiJacobian{Float32, :adjoint_born, judiDataSourceModeling{Float32, :forward}}, judiVector{Float32, Matrix{Float32}}})
    @ JUDI ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/propagation.jl:30
  [6] multi_src_propagate(F::judiJacobian{Float32, :adjoint_born, judiDataSourceModeling{Float32, :forward}}, q::judiVector{Float32, Matrix{Float32}})
    @ JUDI ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/propagation.jl:85
  [7] *
    @ ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/LinearOperators/operators.jl:173 [inlined]
  [8] mul!(out::PhysicalParameter{Float32, 2}, F::judiJacobian{Float32, :adjoint_born, judiDataSourceModeling{Float32, :forward}}, q::judiVector{Float32, Matrix{Float32}})
    @ JUDI ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/LinearOperators/operators.jl:178
  [9] lsqr_method!(log::ConvergenceHistory{false, Nothing}, x::PhysicalParameter{Float32, 2}, A::judiJacobian{Float32, :born, judiDataSourceModeling{Float32, :forward}}, b::judiVector{Float32, Matrix{Float32}}; damp::Int64, atol::Float32, btol::Float32, conlim::Float32, maxiter::Int64, verbose::Bool)
    @ IterativeSolvers ~/.julia/packages/IterativeSolvers/bjiMt/src/lsqr.jl:170
 [10] lsqr!(x::PhysicalParameter{Float32, 2}, A::judiJacobian{Float32, :born, judiDataSourceModeling{Float32, :forward}}, b::judiVector{Float32, Matrix{Float32}}; maxiter::Int64, log::Bool, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Bool}}})
    @ IterativeSolvers ~/.julia/packages/IterativeSolvers/bjiMt/src/lsqr.jl:74
 [11] top-level scope
    @ /media/tju/0FD709D40FD709D4/jktong/Pipe/PipeFWIInversion/Pipe_2D_GN_v2.jl:101
in expression starting at /media/tju/0FD709D40FD709D4/jktong/Pipe/PipeFWIInversion/Pipe_2D_GN_v2.jl:94
┌ Warning: Forcibly interrupting busy workers
│   exception = rmprocs: pids [2, 3, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17] not terminated after 5.0 seconds.
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/cluster.jl:1253
┌ Warning: Failed to gracefully kill worker 2, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 3, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 4, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 6, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 7, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 9, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 10, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 11, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 12, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 13, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 14, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 15, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 16, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 17, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: rmprocs: process 1 not removed
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/cluster.jl:1049
      From worker 6:

Second run:

      From worker 17:	Trying to allocate more memory for symbol u than available on physical device, this will start swapping
      From worker 5:	Trying to allocate more memory for symbol u than available on physical device, this will start swapping
Worker 17 terminated.      From worker 6:	Operator `forward` ran in 20.01 s

Worker 5 terminated.Unhandled Task ERROR: IOError: read: connection reset by peer (ECONNRESET)
Stacktrace:
  [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
    @ Base ./stream.jl:410
  [2] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
    @ Base ./stream.jl:949
  [3] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
    @ Base ./stream.jl:955
  [4] unsafe_read
    @ ./io.jl:761 [inlined]
  [5] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
    @ Base ./io.jl:760
  [6] read!
    @ ./io.jl:762 [inlined]
  [7] deserialize_hdr_raw
    @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
  [8] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
  [9] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
    @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [10] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
    @ Distributed ./task.jl:514
      From worker 12:	Operator `forward` ran in 21.21 s

Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#715")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:947
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:955
 [3] unsafe_read
   @ ./io.jl:761 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:760
 [5] read!
   @ ./io.jl:762 [inlined]
 [6] deserialize_hdr_raw
   @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:514
ERROR:       From worker 6:	Operator `gradient` ran in 29.38 sLoadError: 
TaskFailedException
      From worker 12:	Operator `gradient` ran in 28.21 s
    nested task error: 
On worker 6:
    ProcessExitedException(17)
    Stacktrace:
      [1] worker_from_id
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/cluster.jl:1093
      [2] worker_from_id
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/cluster.jl:1090 [inlined]
      [3] #remotecall_fetch#162
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:492 [inlined]
      [4] remotecall_fetch
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:492 [inlined]
      [5] local_reduce!
        @ ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/distributed.jl:38
      [6] #invokelatest#2
        @ ./essentials.jl:819
      [7] invokelatest
        @ ./essentials.jl:816
      [8] #114
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:302
      [9] run_work_thunk
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
     [10] run_work_thunk
        @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:79
     [11] #100
        @ ./task.jl:514
    Stacktrace:
     [1] remotecall_wait(::Function, ::Distributed.Worker, ::Future, ::Vararg{Future}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
       @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:507
     [2] remotecall_wait
       @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:497 [inlined]
     [3] #remotecall_wait#167
       @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:519 [inlined]
     [4] remotecall_wait
       @ ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/remotecall.jl:519 [inlined]
     [5] (::JUDI.var"#237#238"{Vector{Future}, Int64})()
       @ JUDI ./task.jl:514

...and 1 more exception.

Stacktrace:
  [1] sync_end(c::Channel{Any})
    @ Base ./task.jl:445
  [2] macro expansion
    @ ./task.jl:477 [inlined]
  [3] reduce_level!(futures::Vector{Future}, nleaf::Int64)
    @ JUDI ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/distributed.jl:53
  [4] reduce!(futures::Vector{Future})
    @ JUDI ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/distributed.jl:79
  [5] run_and_reduce(func::Function, pool::WorkerPool, nsrc::Int64, arg_func::JUDI.var"#273#274"{judiJacobian{Float32, :adjoint_born, judiDataSourceModeling{Float32, :forward}}, judiVector{Float32, Matrix{Float32}}})
    @ JUDI ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/propagation.jl:30
  [6] multi_src_propagate(F::judiJacobian{Float32, :adjoint_born, judiDataSourceModeling{Float32, :forward}}, q::judiVector{Float32, Matrix{Float32}})
    @ JUDI ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/Modeling/propagation.jl:85
  [7] *
    @ ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/LinearOperators/operators.jl:173 [inlined]
  [8] mul!(out::PhysicalParameter{Float32, 2}, F::judiJacobian{Float32, :adjoint_born, judiDataSourceModeling{Float32, :forward}}, q::judiVector{Float32, Matrix{Float32}})
    @ JUDI ~/.julia/packages/JUDI/JEsVr/src/TimeModeling/LinearOperators/operators.jl:178
  [9] lsqr_method!(log::ConvergenceHistory{false, Nothing}, x::PhysicalParameter{Float32, 2}, A::judiJacobian{Float32, :born, judiDataSourceModeling{Float32, :forward}}, b::judiVector{Float32, Matrix{Float32}}; damp::Int64, atol::Float32, btol::Float32, conlim::Float32, maxiter::Int64, verbose::Bool)
    @ IterativeSolvers ~/.julia/packages/IterativeSolvers/bjiMt/src/lsqr.jl:170
 [10] lsqr!(x::PhysicalParameter{Float32, 2}, A::judiJacobian{Float32, :born, judiDataSourceModeling{Float32, :forward}}, b::judiVector{Float32, Matrix{Float32}}; maxiter::Int64, log::Bool, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Bool}}})
    @ IterativeSolvers ~/.julia/packages/IterativeSolvers/bjiMt/src/lsqr.jl:74
 [11] top-level scope
    @ /media/tju/0FD709D40FD709D4/jktong/Pipe/PipeFWIInversion/Pipe_2D_GN_v2.jl:101
in expression starting at /media/tju/0FD709D40FD709D4/jktong/Pipe/PipeFWIInversion/Pipe_2D_GN_v2.jl:94
┌ Warning: Forcibly interrupting busy workers
│   exception = rmprocs: pids [2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] not terminated after 5.0 seconds.
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/cluster.jl:1253
┌ Warning: Failed to gracefully kill worker 2, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 3, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 4, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 6, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 7, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 8, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 9, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 10, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 11, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 12, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 13, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 14, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 15, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: Failed to gracefully kill worker 16, sending SIGTERM
└ @ Distributed ~/Program/julia-1.9.4/share/julia/stdlib/v1.9/Distributed/src/managers.jl:745
┌ Warning: rmprocs: process 1 not removed

Answered by mloubout

Mar 13, 2024

You can use time subbsampling (subsampling_factor in Options), checkpointing (optimal_checkpointing=true in Options) or on-the fly fourier transform (need to supply frequencies in Options)

View full answer

tjk9501 · 2024-03-13T02:52:19Z

tjk9501
Mar 13, 2024
Author

Using Gauss-Newton 2D FWI, and allocate 16 process

4 replies

mloubout Mar 13, 2024
Maintainer

From the look of it you are running out of memory

Trying to allocate more memory for symbol u than available on physical device

tjk9501 Mar 13, 2024
Author

I want to use a large model to maintain small grid spacing, what can I do to reduce memory consumption.

mloubout Mar 13, 2024
Maintainer

You can use time subbsampling (subsampling_factor in Options), checkpointing (optimal_checkpointing=true in Options) or on-the fly fourier transform (need to supply frequencies in Options)

Answer selected by tjk9501

tjk9501 Mar 13, 2024
Author

Ok, I will try it.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unhandled Task ERROR: IOError #233

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Unhandled Task ERROR: IOError #233

tjk9501 Mar 13, 2024

Replies: 1 comment · 4 replies

tjk9501 Mar 13, 2024 Author

mloubout Mar 13, 2024 Maintainer

tjk9501 Mar 13, 2024 Author

mloubout Mar 13, 2024 Maintainer

tjk9501 Mar 13, 2024 Author

tjk9501
Mar 13, 2024

Replies: 1 comment 4 replies

tjk9501
Mar 13, 2024
Author

mloubout Mar 13, 2024
Maintainer

tjk9501 Mar 13, 2024
Author

mloubout Mar 13, 2024
Maintainer

tjk9501 Mar 13, 2024
Author