Segfault in heavily nested function call

[serenity4]

I get a segfault on nightly for the SymbolicGA.jl package:

(@v1.12) pkg> activate --temp
  Activating new project at `/tmp/jl_Q2CEQp`

(jl_Q2CEQp) pkg> add SymbolicGA#segfault-reproducer
... [truncated]

julia> using SymbolicGA: @ga

julia> @ga 2 (1, (1, 2))::(0, 1)

[97611] signal 11 (1): Segmentation fault
in expression starting at REPL[4]:1
_jl_invoke at /cache/build/builder-amdci5-3/julialang/julia-master/src/gf.c:3284 [inlined]
ijl_invoke at /cache/build/builder-amdci5-3/julialang/julia-master/src/gf.c:3313
traverse at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:816
unknown function (ip: 0x7f18a7b98927) at (unknown file)
#traverse#51 at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:828
unknown function (ip: 0x7f18a7b987d9) at (unknown file)
traverse at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:816 [inlined]
#traverse#51 at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:821 [inlined]
traverse at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:816
unknown function (ip: 0x7f18a7b987a0) at (unknown file)
#traverse#51 at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:828
traverse at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:816
#traverse#51 at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:828
traverse at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:816
#traverse#51 at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:828
traverse at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:816
#traverse#51 at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:828
traverse at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:816
#traverse#51 at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:828
traverse at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:816 [inlined]
traverse at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/expressions.jl:816 [inlined]
gather_scalar_expressions! at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/optimization.jl:61 [inlined]
gather_scalar_expressions at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/optimization.jl:28 [inlined]
IterativeRefinement at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/optimization.jl:26 [inlined]
optimize! at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/optimization.jl:53 [inlined]
#generate_expression#65 at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/macro.jl:190
generate_expression at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/macro.jl:186 [inlined]
#codegen_expression#67 at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/macro.jl:211
codegen_expression at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/macro.jl:209
#codegen_expression#66 at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/macro.jl:206
unknown function (ip: 0x7f18a7b48d1a) at (unknown file)
codegen_expression at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/macro.jl:206
unknown function (ip: 0x7f18a7b45cda) at (unknown file)
@ga at /home/serenity4/.julia/packages/SymbolicGA/bCHM9/src/macro.jl:49
jl_invoke_julia_macro at /cache/build/builder-amdci5-3/julialang/julia-master/src/ast.c:1173
jl_expand_macros at /cache/build/builder-amdci5-3/julialang/julia-master/src/ast.c:1235
[...] (truncated, C lines)

Allocations: 33948422 (Pool: 33933839; Big: 14583); GC: 22
fish: Job 1, '$jlpath --color=yes -tauto $argv' terminated by signal SIGSEGV (Address boundary error)

If uncommenting the line that was commented here: serenity4/SymbolicGA.jl@3fe2ccc, with the same reproducer (setting it up with pkg> add SymbolicGA#lookup-error instead), I instead get an undefined symbol:

(@v1.12) pkg> activate --temp
  Activating new project at `/tmp/jl_Q2CEQp`

(jl_Q2CEQp) pkg> add SymbolicGA#lookup-error
... [truncated]

julia> using SymbolicGA: @ga

julia> @ga 2 (1, (1, 2))::(0, 1)
/home/serenity4/.julia/juliaup/julia-nightly/bin/julia: symbol lookup error: /home/serenity4/.julia/compiled/v1.12/SymbolicGA/krY9u_Z0AFz.so: undefined symbol: jl_fptr_sparam

Version info

julia> versioninfo()
Julia Version 1.12.0-DEV.1690
Commit 81568a6dc1b (2024-11-25 13:17 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 32 × 13th Gen Intel(R) Core(TM) i9-13900F
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, alderlake)
Threads: 32 default, 0 interactive, 32 GC (on 32 virtual cores)
Environment:
  JULIA_INFO_COLOR = green

Everything works fine on 1.11.

[serenity4]

git bisect indicates that the first bad commit for the segfault is: c31710a (#54899)

The lookup error is older, will open a separate issue for it.

EDIT: #56701

[giordano]

git bisect indicates that the first bad commit for the segfault is: c31710a (#54899)

CC: @vtjnash.