Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation Fault (turing) #457

Closed
devmotion opened this issue Sep 28, 2022 · 21 comments
Closed

Segmentation Fault (turing) #457

devmotion opened this issue Sep 28, 2022 · 21 comments
Labels
segfault

Comments

@devmotion
Copy link
Contributor

I tried a simple Turing model again (TuringLang/Turing.jl#1887) with Enzyme#main.

It didn't work a month ago and but since I'm not familiar with Enzyme and its stacktraces I did not investigate it further.
I just tried and the example still fails with a Julia segfault:

...
Illegal address space propagation
UNREACHABLE executed at /workspace/srcdir/Enzyme/enzyme/Enzyme/FunctionUtils.cpp:366!
...

A longer output is available here: https://gist.github.com/devmotion/1352197f2354c6fecddd7b778ec4bcf7#file-log-txt (there are many more pointer lookup warnings, it filled my whole terminal and therefore unfortunately the gist is incomplete)

Do you have any idea what the problem is?
Just pointing me to some possible issues or some tools for debugging would be helpful.
Currently basically I am only able to run this example from time to time but I neither know what Enzyme does not like here nor how to debug it.
@wsmoses
Copy link
Member

wsmoses commented Sep 28, 2022

Would you be able to minimize this and post the minimal example? The pointer warnings can be ignored (though we should clean that up), but the latter part requires attention.

@devmotion
Copy link
Contributor Author

A bit simpler, with an empty model and without sample, NUTS etc. and with Turing#master (or latest release, should be the same):

using Turing
using DynamicPPL
using LogDensityProblems
using Enzyme

@model function demo() end
f = Turing.LogDensityFunction(VarInfo(demo()), demo(), SampleFromPrior(), DefaultContext())
g = ADgradient(Val(:Enzyme), f)

LogDensityProblems.logdensity(g, Float64[]) # correctly returns 0.0
LogDensityProblems.logdensity_and_gradient(g, Float64[]) # segfaults

The output of the last call is

 rep:   %9 = bitcast {} addrspace(10)* %8 to { { {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)* } addrspace(10)*, !enzyme_caststack !4 prev:   %10 = alloca { { {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)* }, align 8 inst:   call fastcc void @julia_ThreadSafeVarInfo_3887({ { {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)* }* noalias nocapture nofree noundef nonnull writeonly sret({ { {} addrspace(10)*, {} addrspace(10)* }, {} addrspace(10)* }) align 8 dereferenceable(24) %10, {} addrspace(10)* noalias nocapture nofree nonnull readnone align 16 undef, { {} addrspace(10)*, {} addrspace(10)* } addrspace(11)* nocapture nofree noundef nonnull readonly align 8 dereferenceable(16) %1) #14, !dbg !26
Illegal address space propagation
UNREACHABLE executed at /workspace/srcdir/Enzyme/enzyme/Enzyme/FunctionUtils.cpp:366!

signal (6): Aborted
in expression starting at REPL[12]:1
__pthread_kill_implementation at /lib64/libc.so.6 (unknown line)
raise at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
_ZN4llvm25llvm_unreachable_internalEPKcS1_j at /home/david/.julia/juliaup/julia-1.8.1+0.x64/bin/../lib/julia/libLLVM-13jl.so (unknown line)
RecursivelyReplaceAddressSpace at /workspace/srcdir/Enzyme/enzyme/Enzyme/FunctionUtils.cpp:366
UpgradeAllocasToMallocs at /workspace/srcdir/Enzyme/enzyme/Enzyme/FunctionUtils.cpp:437
preprocessForClone at /workspace/srcdir/Enzyme/enzyme/Enzyme/FunctionUtils.cpp:1694
CloneFunctionWithReturns at /workspace/srcdir/Enzyme/enzyme/Enzyme/FunctionUtils.cpp:1982
CreateFromClone at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:3399
CreateAugmentedPrimal at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:2058
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:11768
delegateCallInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:302 [inlined]
visitCall at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209 [inlined]
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:112 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:3916
EnzymeCreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:473
EnzymeCreatePrimalAndGradient at /home/david/.julia/packages/Enzyme/2K2vv/src/api.jl:118
enzyme! at /home/david/.julia/packages/Enzyme/2K2vv/src/compiler.jl:4292
unknown function (ip: 0x7f5ab0b17002)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#codegen#112 at /home/david/.julia/packages/Enzyme/2K2vv/src/compiler.jl:5278
codegen##kw at /home/david/.julia/packages/Enzyme/2K2vv/src/compiler.jl:4948 [inlined]
_thunk at /home/david/.julia/packages/Enzyme/2K2vv/src/compiler.jl:5751 [inlined]
_thunk at /home/david/.julia/packages/Enzyme/2K2vv/src/compiler.jl:5745
unknown function (ip: 0x7f5ab0a8f4dd)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
cached_compilation at /home/david/.julia/packages/Enzyme/2K2vv/src/compiler.jl:5789
unknown function (ip: 0x7f5ab0a6dc23)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#s814#139 at /home/david/.julia/packages/Enzyme/2K2vv/src/compiler.jl:5849 [inlined]
#s814#139 at ./none:0
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
GeneratedFunctionStub at ./boot.jl:582
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
jl_call_staged at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/method.c:520
ijl_code_for_staged at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/method.c:571
get_staged at ./compiler/utilities.jl:114
retrieve_code_info at ./compiler/utilities.jl:126 [inlined]
InferenceState at ./compiler/inferencestate.jl:284
typeinf_edge at ./compiler/typeinfer.jl:868
abstract_call_method at ./compiler/abstractinterpretation.jl:641
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1696
abstract_call at ./compiler/abstractinterpretation.jl:1766
abstract_call at ./compiler/abstractinterpretation.jl:1733
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1890
typeinf_local at ./compiler/abstractinterpretation.jl:2366
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2462
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:641
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1696
abstract_call at ./compiler/abstractinterpretation.jl:1766
abstract_call at ./compiler/abstractinterpretation.jl:1733
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1890
typeinf_local at ./compiler/abstractinterpretation.jl:2366
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2462
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:641
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1696
abstract_call at ./compiler/abstractinterpretation.jl:1766
abstract_call at ./compiler/abstractinterpretation.jl:1733
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1890
typeinf_local at ./compiler/abstractinterpretation.jl:2366
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2462
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:641
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1696
abstract_call at ./compiler/abstractinterpretation.jl:1766
abstract_call at ./compiler/abstractinterpretation.jl:1733
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1890
typeinf_local at ./compiler/abstractinterpretation.jl:2340
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2462
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:641
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1696
abstract_call at ./compiler/abstractinterpretation.jl:1766
abstract_call at ./compiler/abstractinterpretation.jl:1733
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1890
typeinf_local at ./compiler/abstractinterpretation.jl:2366
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2462
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_ext at ./compiler/typeinfer.jl:967
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1000
typeinf_ext_toplevel at ./compiler/typeinfer.jl:996
jfptr_typeinf_ext_toplevel_18807.clone_1 at /home/david/.julia/juliaup/julia-1.8.1+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
jl_type_infer at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:319
jl_generate_fptr_impl at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jitlayers.cpp:319
jl_compile_method_internal at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2081 [inlined]
jl_compile_method_internal at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2025
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2359 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
do_call at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:126
eval_value at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:215
eval_stmt_value at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:166 [inlined]
eval_body at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:612
jl_interpret_toplevel_thunk at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:750
jl_toplevel_eval_flex at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:906
jl_toplevel_eval_flex at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:850
jl_toplevel_eval_flex at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:850
ijl_toplevel_eval_in at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:965
eval at ./boot.jl:368 [inlined]
eval_user_input at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:151
repl_backend_loop at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:247
start_repl_backend at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:232
#run_repl#47 at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:369
run_repl at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:355
jfptr_run_repl_66557.clone_1 at /home/david/.julia/juliaup/julia-1.8.1+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#967 at ./client.jl:419
jfptr_YY.967_49700.clone_1 at /home/david/.julia/juliaup/julia-1.8.1+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:729 [inlined]
invokelatest at ./essentials.jl:726 [inlined]
run_main_repl at ./client.jl:404
exec_options at ./client.jl:318
_start at ./client.jl:522
jfptr__start_61720.clone_1 at /home/david/.julia/juliaup/julia-1.8.1+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
true_main at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jlapi.c:575
jl_repl_entrypoint at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jlapi.c:719
main at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/cli/loader_exe.c:59
__libc_start_call_main at /lib64/libc.so.6 (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 85016387 (Pool: 84950857; Big: 65530); GC: 57

@devmotion
Copy link
Contributor Author

BTW Julia segfaults also when using forward mode:

...
g = ADgradient(Val(:Enzyme), f; mode=Enzyme.Forward)

LogDensityProblems.logdensity(g, Float64[]) # correctly returns 0.0
LogDensityProblems.logdensity_and_gradient(g, Float64[]) # segfaults

Output is

signal (11): Segmentation fault
in expression starting at REPL[10]:1
llvm_type_rewrite at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/ccall.cpp:359
emit_llvmcall at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/ccall.cpp:792
emit_intrinsic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/intrinsics.cpp:1067
emit_call at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:3817
emit_expr at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:4741
emit_ssaval_assign at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:4315
emit_stmtpos at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:4562 [inlined]
emit_function at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:7416
jl_emit_code at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:7778
jl_emit_codeinst at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:7823
_jl_compile_codeinst at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jitlayers.cpp:119
jl_generate_fptr_impl at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jitlayers.cpp:332
jl_compile_method_internal at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2081 [inlined]
jl_compile_method_internal at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2025
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2359 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
logdensity_and_gradient at /home/david/.julia/packages/LogDensityProblems/oAYeE/src/AD_Enzyme.jl:46
unknown function (ip: 0x7f3045f2f0d1)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
do_call at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:126
eval_value at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:215
eval_stmt_value at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:166 [inlined]
eval_body at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:612
jl_interpret_toplevel_thunk at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:750
jl_toplevel_eval_flex at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:906
jl_toplevel_eval_flex at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:850
jl_toplevel_eval_flex at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:850
ijl_toplevel_eval_in at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:965
eval at ./boot.jl:368 [inlined]
eval_user_input at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:151
repl_backend_loop at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:247
start_repl_backend at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:232
#run_repl#47 at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:369
run_repl at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:355
jfptr_run_repl_66557.clone_1 at /home/david/.julia/juliaup/julia-1.8.1+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#967 at ./client.jl:419
jfptr_YY.967_49700.clone_1 at /home/david/.julia/juliaup/julia-1.8.1+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:729 [inlined]
invokelatest at ./essentials.jl:726 [inlined]
run_main_repl at ./client.jl:404
exec_options at ./client.jl:318
_start at ./client.jl:522
jfptr__start_61720.clone_1 at /home/david/.julia/juliaup/julia-1.8.1+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
true_main at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jlapi.c:575
jl_repl_entrypoint at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jlapi.c:719
main at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/cli/loader_exe.c:59
__libc_start_call_main at /lib64/libc.so.6 (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 84804096 (Pool: 84739048; Big: 65048); GC: 58

@wsmoses
Copy link
Member

wsmoses commented Sep 30, 2022

For all of these can you make a minimal reproducer that directly calls Enzyme autodiff (and ideally has no package dependencies)?

@wsmoses
Copy link
Member

wsmoses commented Oct 4, 2022

This now appears to work on main, closing:

julia> LogDensityProblems.logdensity(g, Float64[])
0.0

julia> LogDensityProblems.logdensity_and_gradient(g, Float64[])
(0.0, Float64[])

@wsmoses wsmoses closed this as completed Oct 4, 2022
@wsmoses wsmoses reopened this Oct 4, 2022
@wsmoses
Copy link
Member

wsmoses commented Oct 4, 2022

The forward mode bug, however persists and is weird so actually leaving this open

@devmotion if you can minimize this (e.g. into a direct call to Enzyme.autodiff ideally with no dependencies), we can get the forward mode bug fixed.

julia> LogDensityProblems.logdensity_and_gradient(g, Float64[]) # segfaults

signal (11): Segmentation fault
in expression starting at REPL[11]:1
llvm_type_rewrite at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/ccall.cpp:359
emit_llvmcall at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/ccall.cpp:792
emit_intrinsic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/intrinsics.cpp:1067
emit_call at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:3817
emit_expr at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:4741
emit_ssaval_assign at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:4315
emit_stmtpos at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:4562 [inlined]
emit_function at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:7416
jl_emit_code at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:7778
jl_emit_codeinst at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/codegen.cpp:7823
_jl_compile_codeinst at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jitlayers.cpp:119
jl_generate_fptr_impl at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jitlayers.cpp:332
jl_compile_method_internal at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2081 [inlined]
jl_compile_method_internal at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2025
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2359 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
logdensity_and_gradient at /home/wmoses/.julia/packages/LogDensityProblems/oAYeE/src/AD_Enzyme.jl:46
unknown function (ip: 0x7f38b75ab951)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
do_call at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:126
eval_value at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:215
eval_stmt_value at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:166 [inlined]
eval_body at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:612
jl_interpret_toplevel_thunk at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/interpreter.c:750
jl_toplevel_eval_flex at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:906
jl_toplevel_eval_flex at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:850
ijl_toplevel_eval_in at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/toplevel.c:965
eval at ./boot.jl:368 [inlined]
eval_user_input at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:151
repl_backend_loop at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:247
start_repl_backend at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:232
#run_repl#47 at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:369
run_repl at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:355
jfptr_run_repl_66557.clone_1 at /home/wmoses/git/Enzyme.jl/julia-1.8.1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#967 at ./client.jl:419
jfptr_YY.967_49700.clone_1 at /home/wmoses/git/Enzyme.jl/julia-1.8.1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:729 [inlined]
invokelatest at ./essentials.jl:726 [inlined]
run_main_repl at ./client.jl:404
exec_options at ./client.jl:318
_start at ./client.jl:522
jfptr__start_61720.clone_1 at /home/wmoses/git/Enzyme.jl/julia-1.8.1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
true_main at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jlapi.c:575
jl_repl_entrypoint at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jlapi.c:719
main at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/cli/loader_exe.c:59
unknown function (ip: 0x7f3aea6e0d8f)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 75539437 (Pool: 75485508; Big: 53929); GC: 56
Segmentation fault (core dumped)

@wsmoses
Copy link
Member

wsmoses commented Oct 5, 2022

Turns out the error here is using a batch size of 0, which is unsupported. Differentiating with batch size of 1, succeeds in forward mode.

LogDensityProblems.logdensity_and_gradient(g, Float64[1.0])
LogDensityProblems.logdensity_and_gradient(g, Float64[1.0]) = (0.0, [0.0])
(0.0, [0.0])

However, we should emit a nicer error message, which I am adding:

ERROR: LoadError: Cannot differentiate with a batch size of 0
Stacktrace:
 [1] autodiff(#unused#::Enzyme.ForwardMode, f::Base.Fix1{typeof(LogDensityProblems.logdensity), Turing.LogDensityFunction{TypedVarInfo{NamedTuple{(), Tuple{}}, Float64}, Model{typeof(demo), (), (), (), Tuple{}, Tuple{}, DefaultContext}, SampleFromPrior, DefaultContext}}, #unused#::Type{BatchDuplicated}, args::BatchDuplicated{Vector{Float64}, 0})
   @ Enzyme ~/git/Enzyme.jl/src/Enzyme.jl:417
 [2] logdensity_and_gradient(∇ℓ::LogDensityProblems.EnzymeGradientLogDensity{Turing.LogDensityFunction{TypedVarInfo{NamedTuple{(), Tuple{}}, Float64}, Model{typeof(demo), (), (), (), Tuple{}, Tuple{}, DefaultContext}, SampleFromPrior, DefaultContext}, Enzyme.ForwardMode, Nothing}, x::Vector{Float64})
   @ LogDensityProblems ~/.julia/packages/LogDensityProblems/oAYeE/src/AD_Enzyme.jl:46
 [3] top-level scope

@wsmoses wsmoses closed this as completed Oct 5, 2022
@devmotion
Copy link
Contributor Author

The example in TuringLang/Turing.jl#1887 (comment) still causes a segfault on my machine it seems 😢 It's a different log now though so it might be a different issue:

[...omitting pointer warnings...]

signal (11): Segmentation fault
in expression starting at REPL[5]:1
unsafe_store! at ./pointer.jl:118 [inlined]
unsafe_store! at ./pointer.jl:118
unknown function (ip: 0x7f8303b3c8cc)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
julia_allocator at /home/david/.julia/packages/Enzyme/wYJeS/src/compiler.jl:3676
unknown function (ip: 0x7f8303b3b966)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
julia_allocator at /home/david/.julia/packages/Enzyme/wYJeS/src/compiler.jl:3535
unknown function (ip: 0x7f83473e825c)
CreateAllocation at /workspace/srcdir/Enzyme/enzyme/Enzyme/Utils.cpp:258
UpgradeAllocasToMallocs at /workspace/srcdir/Enzyme/enzyme/Enzyme/FunctionUtils.cpp:431
preprocessForClone at /workspace/srcdir/Enzyme/enzyme/Enzyme/FunctionUtils.cpp:1710
CloneFunctionWithReturns at /workspace/srcdir/Enzyme/enzyme/Enzyme/FunctionUtils.cpp:1998
CreateFromClone at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:3447
CreateAugmentedPrimal at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:2097
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:12114
delegateCallInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:302 [inlined]
visitCall at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209 [inlined]
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:112 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:3947
EnzymeCreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:473
EnzymeCreatePrimalAndGradient at /home/david/.julia/packages/Enzyme/wYJeS/src/api.jl:118
enzyme! at /home/david/.julia/packages/Enzyme/wYJeS/src/compiler.jl:4563
unknown function (ip: 0x7f8303b26492)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
#codegen#120 at /home/david/.julia/packages/Enzyme/wYJeS/src/compiler.jl:5655
codegen##kw at /home/david/.julia/packages/Enzyme/wYJeS/src/compiler.jl:5325 [inlined]
_thunk at /home/david/.julia/packages/Enzyme/wYJeS/src/compiler.jl:6122 [inlined]
_thunk at /home/david/.julia/packages/Enzyme/wYJeS/src/compiler.jl:6116
unknown function (ip: 0x7f8303f7d6fd)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
cached_compilation at /home/david/.julia/packages/Enzyme/wYJeS/src/compiler.jl:6160
unknown function (ip: 0x7f8303f5c363)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
#s766#147 at /home/david/.julia/packages/Enzyme/wYJeS/src/compiler.jl:6220 [inlined]
#s766#147 at ./none:0
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
GeneratedFunctionStub at ./boot.jl:582
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/julia.h:1839 [inlined]
jl_call_staged at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/method.c:520
ijl_code_for_staged at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/method.c:571
get_staged at ./compiler/utilities.jl:114
retrieve_code_info at ./compiler/utilities.jl:126 [inlined]
InferenceState at ./compiler/inferencestate.jl:284
typeinf_edge at ./compiler/typeinfer.jl:868
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call_known at ./compiler/abstractinterpretation.jl:1716
abstract_call at ./compiler/abstractinterpretation.jl:1786
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2386
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call_known at ./compiler/abstractinterpretation.jl:1716
abstract_call at ./compiler/abstractinterpretation.jl:1786
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2386
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call_known at ./compiler/abstractinterpretation.jl:1716
abstract_call at ./compiler/abstractinterpretation.jl:1786
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2386
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call_known at ./compiler/abstractinterpretation.jl:1716
abstract_call at ./compiler/abstractinterpretation.jl:1786
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2360
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call_known at ./compiler/abstractinterpretation.jl:1716
abstract_call at ./compiler/abstractinterpretation.jl:1786
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2386
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call_known at ./compiler/abstractinterpretation.jl:1716
abstract_call at ./compiler/abstractinterpretation.jl:1786
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2386
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call at ./compiler/abstractinterpretation.jl:1784
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2360
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call_known at ./compiler/abstractinterpretation.jl:1716
abstract_call at ./compiler/abstractinterpretation.jl:1786
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2386
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call_known at ./compiler/abstractinterpretation.jl:1716
abstract_call at ./compiler/abstractinterpretation.jl:1786
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2386
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call_known at ./compiler/abstractinterpretation.jl:1716
abstract_call at ./compiler/abstractinterpretation.jl:1786
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2360
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_edge at ./compiler/typeinfer.jl:877
abstract_call_method at ./compiler/abstractinterpretation.jl:647
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:139
abstract_call_known at ./compiler/abstractinterpretation.jl:1716
abstract_call at ./compiler/abstractinterpretation.jl:1786
abstract_call at ./compiler/abstractinterpretation.jl:1753
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1910
typeinf_local at ./compiler/abstractinterpretation.jl:2386
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2482
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
typeinf_ext at ./compiler/typeinfer.jl:967
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1000
typeinf_ext_toplevel at ./compiler/typeinfer.jl:996
jfptr_typeinf_ext_toplevel_13386.clone_1 at /home/david/.julia/juliaup/julia-1.8.2+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/julia.h:1839 [inlined]
jl_type_infer at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:319
jl_generate_fptr_impl at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/jitlayers.cpp:319
jl_compile_method_internal at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2081 [inlined]
jl_compile_method_internal at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2025
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2359 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
#step#23 at /home/david/.julia/packages/DynamicPPL/zPOYL/src/sampler.jl:104
step##kw at /home/david/.julia/packages/DynamicPPL/zPOYL/src/sampler.jl:71 [inlined]
macro expansion at /home/david/.julia/packages/AbstractMCMC/fnRmh/src/sample.jl:120 [inlined]
macro expansion at /home/david/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:328 [inlined]
macro expansion at /home/david/.julia/packages/AbstractMCMC/fnRmh/src/logging.jl:9 [inlined]
#mcmcsample#20 at /home/david/.julia/packages/AbstractMCMC/fnRmh/src/sample.jl:111
unknown function (ip: 0x7f8303f2dd4e)
mcmcsample##kw at /home/david/.julia/packages/AbstractMCMC/fnRmh/src/sample.jl:90
unknown function (ip: 0x7f8303f15555)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
#sample#42 at /home/david/.julia/packages/Turing/zBIeZ/src/inference/hmc.jl:133
unknown function (ip: 0x7f8303f15099)
sample at /home/david/.julia/packages/Turing/zBIeZ/src/inference/hmc.jl:103 [inlined]
#sample#2 at /home/david/.julia/packages/Turing/zBIeZ/src/inference/Inference.jl:145 [inlined]
sample at /home/david/.julia/packages/Turing/zBIeZ/src/inference/Inference.jl:138 [inlined]
#sample#1 at /home/david/.julia/packages/Turing/zBIeZ/src/inference/Inference.jl:135 [inlined]
sample at /home/david/.julia/packages/Turing/zBIeZ/src/inference/Inference.jl:129
unknown function (ip: 0x7f8303f14dc2)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/julia.h:1839 [inlined]
do_call at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/interpreter.c:126
eval_value at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/interpreter.c:215
eval_stmt_value at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/interpreter.c:166 [inlined]
eval_body at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/interpreter.c:612
jl_interpret_toplevel_thunk at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/interpreter.c:750
jl_toplevel_eval_flex at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/toplevel.c:906
jl_toplevel_eval_flex at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/toplevel.c:850
jl_toplevel_eval_flex at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/toplevel.c:850
ijl_toplevel_eval_in at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/toplevel.c:965
eval at ./boot.jl:368 [inlined]
eval_user_input at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:151
repl_backend_loop at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:247
start_repl_backend at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:232
#run_repl#47 at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:369
run_repl at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:355
jfptr_run_repl_64841.clone_1 at /home/david/.julia/juliaup/julia-1.8.2+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
#967 at ./client.jl:419
jfptr_YY.967_30403.clone_1 at /home/david/.julia/juliaup/julia-1.8.2+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/julia.h:1839 [inlined]
jl_f__call_latest at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:729 [inlined]
invokelatest at ./essentials.jl:726 [inlined]
run_main_repl at ./client.jl:404
exec_options at ./client.jl:318
_start at ./client.jl:522
jfptr__start_56736.clone_1 at /home/david/.julia/juliaup/julia-1.8.2+0.x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/julia.h:1839 [inlined]
true_main at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/jlapi.c:575
jl_repl_entrypoint at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/src/jlapi.c:719
main at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-8/cli/loader_exe.c:59
__libc_start_call_main at /lib64/libc.so.6 (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 134908418 (Pool: 134826746; Big: 81672); GC: 91

@devmotion
Copy link
Contributor Author

When I try to run the simplified example above with Enzyme#main, I run into the following error:

julia> LogDensityProblems.logdensity_and_gradient(g, Float64[])
ERROR: could not load symbol "EnzymeSetForMemSet":
/home/david/.julia/artifacts/8395259abf52b24216c9c9cbe21f94b15ce30a2e/lib/libEnzyme-13.so: undefined symbol: EnzymeSetForMemSet
Stacktrace:
  [1] SetForMemSet!(i1::LLVM.BitCastInst)
    @ Enzyme.API ~/.julia/packages/Enzyme/wYJeS/src/api.jl:446
  [2] julia_allocator(B::LLVM.Builder, LLVMType::LLVM.StructType, Count::LLVM.ConstantInt, AlignedSize::LLVM.ConstantInt, IsDefault::UInt8, ZI::Ptr{Ptr{LLVM.API.LLVMOpaqueValue}})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/wYJeS/src/compiler.jl:3664
  [3] julia_allocator(B::Ptr{LLVM.API.LLVMOpaqueBuilder}, LLVMType::Ptr{LLVM.API.LLVMOpaqueType}, Count::Ptr{LLVM.API.LLVMOpaqueValue}, AlignedSize::Ptr{LLVM.API.LLVMOpaqueValue}, IsDefault::UInt8, ZI::Ptr{Ptr{LLVM.API.LLVMOpaqueValue}})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/wYJeS/src/compiler.jl:3535
  [4] EnzymeCreatePrimalAndGradient(logic::Enzyme.Logic, todiff::LLVM.Function, retType::Enzyme.API.CDIFFE_TYPE, constant_args::Vector{Enzyme.API.CDIFFE_TYPE}, TA::Enzyme.TypeAnalysis, returnValue::Bool, dretUsed::Bool, mode::Enzyme.API.CDerivativeMode, width::Int64, additionalArg::Ptr{Nothing}, typeInfo::Enzyme.FnTypeInfo, uncacheable_args::Vector{Bool}, augmented::Ptr{Nothing}, atomicAdd::Bool)
    @ Enzyme.API ~/.julia/packages/Enzyme/wYJeS/src/api.jl:118
  [5] enzyme!(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams, GPUCompiler.FunctionSpec{Base.Fix1{typeof(LogDensityProblems.logdensity), Turing.LogDensityFunction{TypedVarInfo{NamedTuple{(), Tuple{}}, Float64}, Model{typeof(demo), (), (), (), Tuple{}, Tuple{}, DefaultContext}, SampleFromPrior, DefaultContext}}, Tuple{Vector{Float64}}}}, mod::LLVM.Module, primalf::LLVM.Function, adjoint::GPUCompiler.FunctionSpec{Base.Fix1{typeof(LogDensityProblems.logdensity), Turing.LogDensityFunction{TypedVarInfo{NamedTuple{(), Tuple{}}, Float64}, Model{typeof(demo), (), (), (), Tuple{}, Tuple{}, DefaultContext}, SampleFromPrior, DefaultContext}}, Tuple{Duplicated{Vector{Float64}}}}, mode::Enzyme.API.CDerivativeMode, width::Int64, parallel::Bool, actualRetType::Type, dupClosure::Bool, wrap::Bool, modifiedBetween::Bool, returnPrimal::Bool, jlrules::Vector{String})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/wYJeS/src/compiler.jl:4563
  [6] codegen(output::Symbol, job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams, GPUCompiler.FunctionSpec{Base.Fix1{typeof(LogDensityProblems.logdensity), Turing.LogDensityFunction{TypedVarInfo{NamedTuple{(), Tuple{}}, Float64}, Model{typeof(demo), (), (), (), Tuple{}, Tuple{}, DefaultContext}, SampleFromPrior, DefaultContext}}, Tuple{Vector{Float64}}}}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, ctx::LLVM.Context, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/wYJeS/src/compiler.jl:5655
  [7] _thunk
    @ ~/.julia/packages/Enzyme/wYJeS/src/compiler.jl:6122 [inlined]
  [8] _thunk(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams, GPUCompiler.FunctionSpec{Base.Fix1{typeof(LogDensityProblems.logdensity), Turing.LogDensityFunction{TypedVarInfo{NamedTuple{(), Tuple{}}, Float64}, Model{typeof(demo), (), (), (), Tuple{}, Tuple{}, DefaultContext}, SampleFromPrior, DefaultContext}}, Tuple{Vector{Float64}}}})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/wYJeS/src/compiler.jl:6116
  [9] cached_compilation(job::GPUCompiler.CompilerJob, key::UInt64, specid::UInt64)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/wYJeS/src/compiler.jl:6160
 [10] #s766#147
    @ ~/.julia/packages/Enzyme/wYJeS/src/compiler.jl:6220 [inlined]
 [11] var"#s766#147"(F::Any, Fn::Any, DF::Any, A::Any, TT::Any, Mode::Any, ModifiedBetween::Any, width::Any, specid::Any, ReturnPrimal::Any, ::Any, #unused#::Type, f::Any, df::Any, #unused#::Type, tt::Any, #unused#::Type, #unused#::Type, #unused#::Type, #unused#::Type, #unused#::Any)
    @ Enzyme.Compiler ./none:0
 [12] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any})
    @ Core ./boot.jl:582
 [13] thunk
    @ ~/.julia/packages/Enzyme/wYJeS/src/compiler.jl:6253 [inlined]
 [14] thunk (repeats 2 times)
    @ ~/.julia/packages/Enzyme/wYJeS/src/compiler.jl:6246 [inlined]
 [15] autodiff
    @ ~/.julia/packages/Enzyme/wYJeS/src/Enzyme.jl:311 [inlined]
 [16] logdensity_and_gradient(∇ℓ::LogDensityProblems.EnzymeGradientLogDensity{Turing.LogDensityFunction{TypedVarInfo{NamedTuple{(), Tuple{}}, Float64}, Model{typeof(demo), (), (), (), Tuple{}, Tuple{}, DefaultContext}, SampleFromPrior, DefaultContext}, Enzyme.ReverseMode, Nothing}, x::Vector{Float64})
    @ LogDensityProblems ~/.julia/packages/LogDensityProblems/oAYeE/src/AD_Enzyme.jl:58
 [17] top-level scope
    @ REPL[10]:1

@wsmoses
Copy link
Member

wsmoses commented Oct 6, 2022

The jll bump landed, try again?

@wsmoses
Copy link
Member

wsmoses commented Oct 6, 2022

We also put out a release today which should include the fix.

@devmotion
Copy link
Contributor Author

devmotion commented Oct 6, 2022 •
edited
Loading

I tried both the latest release and Enzyme#main. The EnzymeSetForMemSet issue is fixed but Julia segfaults in both cases when running the simple reverse-mode example above:

julia> LogDensityProblems.logdensity_and_gradient(g, Float64[])

signal (11): Segmentation fault
in expression starting at REPL[10]:1
unknown function (ip: 0x7f93f0a1ce75)
Allocations: 88193666 (Pool: 88130884; Big: 62782); GC: 67

@wsmoses wsmoses reopened this Oct 6, 2022
@vchuravy
Copy link
Member

I get:

 pp:   %_replacementA2 = phi i64 , !dbg !94 of   %11 = ptrtoint i32* %3 to i64, !dbg !95
julia: /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.h:1078: void GradientUtils::era
seFictiousPHIs(): Assertion `pp->getNumUses() == 0' failed.

signal (6): Aborted
in expression starting at /home/vchuravy/src/Enzyme/turing.jl:10
unknown function (ip: 0x7f56b4c3449c)

For:

using Turing
using Enzyme

@model function model()
    m ~ Normal(0, 1)
    s ~ InverseGamma()
    x ~ Normal(m, s)
end

sample(model() | (; x=0.5), NUTS{Turing.EnzymeAD}(), 10)

with

(turing) pkg> st
Status `~/src/Enzyme/turing/Project.toml`
  [7da242da] Enzyme v0.10.10 `~/src/Enzyme`
  [fce5fe82] Turing v0.21.12 `https://github.com/TuringLang/Turing.jl.git#dw/enzyme`

@vchuravy vchuravy removed their assignment Oct 11, 2022
@wsmoses
Copy link
Member

wsmoses commented Oct 12, 2022

https://fwd.gymni.ch/kNekaA Looks like an inactive lgamma_r issue.

@wsmoses
Copy link
Member

wsmoses commented Oct 12, 2022

@vchuravy on your full turing test with both the fix from (#509) and (EnzymeAD/Enzyme#892), I get the segfault:

signal (11): Segmentation fault
in expression starting at REPL[3]:1
unknown function (ip: 0x7f5248ec298f)
Allocations: 143533752 (Pool: 143454142; Big: 79610); GC: 109
Segmentation fault (core dumped)

@wsmoses wsmoses changed the title Illegal address space propagation Segmentation Fault (turing) Oct 13, 2022
@wsmoses
Copy link
Member

wsmoses commented Dec 24, 2022

Marginally more information:


Thread 1 "julia" received signal SIGSEGV, Segmentation fault.
0x00007fffe0efa237 in model () at REPL[4]:4
4	REPL[4]: No such file or directory.
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) bt
#0  0x00007fffe0efa237 in model () at REPL[4]:4
#1  0x00007fffe0ef9c49 in macro expansion; ()
    at /home/wmoses/.julia/packages/DynamicPPL/xKo8W/src/model.jl:593
#2  _evaluate!! () at /home/wmoses/.julia/packages/DynamicPPL/xKo8W/src/model.jl:576
#3  0x00007fffe0efd94e in diffejulia_Fix1_3867_inner_1wrap ()
#4  0x00007ffdc5dcdae9 in macro expansion () at /home/wmoses/git/Enzyme.jl/src/compiler.jl:6676
#5  enzyme_call () at /home/wmoses/git/Enzyme.jl/src/compiler.jl:6401
#6  CombinedAdjointThunk () at /home/wmoses/git/Enzyme.jl/src/compiler.jl:6364
#7  autodiff () at /home/wmoses/git/Enzyme.jl/src/Enzyme.jl:217
#8  julia_logdensity_and_gradient_8335 ()
    at /home/wmoses/.julia/packages/LogDensityProblemsAD/dPwnh/src/AD_Enzyme.jl:58
#9  0x00007ffdc5dcf00d in ∂logπ∂θ ()
    at /home/wmoses/.julia/packages/Turing/cwg98/src/inference/hmc.jl:165
#10 ∂H∂θ () at /home/wmoses/.julia/packages/AdvancedHMC/iWHPQ/src/hamiltonian.jl:31
#11 phasepoint () at /home/wmoses/.julia/packages/AdvancedHMC/iWHPQ/src/hamiltonian.jl:76
#12 julia_phasepoint_8532 ()
    at /home/wmoses/.julia/packages/AdvancedHMC/iWHPQ/src/hamiltonian.jl:153
#13 0x00007ffdc5de1227 in julia_#initialstep#43_8253 ()
    at /home/wmoses/.julia/packages/Turing/cwg98/src/inference/hmc.jl:169
#14 0x00007ffdc5de4653 in julia_initialstep##kw_8250 ()
    at /home/wmoses/.julia/packages/Turing/cwg98/src/inference/hmc.jl:143
#15 0x00007ffdc5de46d5 in jfptr_initialstep##kw_8251 ()
#16 0x00007ffff7485b3e in _jl_invoke (world=<optimized out>, mfunc=0x7ffdc145b7b0, nargs=6, 
    args=0x7fffffff4b50, F=0x7ffdb49ac4f0)
    at /cache/build/default-amdci5-6/julialang/julia-release-1-dot-8/src/gf.c:2365
#17 ijl_apply_generic (F=<optimized out>, args=0x7fffffff4b50, nargs=<optimized out>)
    at /cache/build/default-amdci5-6/julialang/julia-release-1-dot-8/src/gf.c:2547
#18 0x00007fffe0fb77d2 in julia_#step#27_2182 ()
    at /home/wmoses/.julia/packages/DynamicPPL/xKo8W/src/sampler.jl:111
#19 0x00007fffe0fc92b9 in step##kw ()
    at /home/wmoses/.julia/packages/DynamicPPL/xKo8W/src/sampler.jl:84
#20 macro expansion () at /home/wmoses/.julia/packages/AbstractMCMC/E5r4y/src/sample.jl:120
#21 macro expansion ()
    at /home/wmoses/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:328
#22 julia_#21_2435 () at /home/wmoses/.julia/packages/AbstractMCMC/E5r4y/src/logging.jl:12
#23 0x00007fffe0fd3f70 in jfptr_#21_2436 ()
#24 0x00007ffff7485b3e in _jl_invoke (world=<optimized out>, mfunc=0x7ffdc04e8240, nargs=0, 
    args=0x0, F=0x7ffdb671f610)
    at /cache/build/default-amdci5-6/julialang/julia-release-1-dot-8/src/gf.c:2365
#25 ijl_apply_generic (F=<optimized out>, args=0x0, nargs=<optimized out>)
    at /cache/build/default-amdci5-6/julialang/julia-release-1-dot-8/src/gf.c:2547
#26 0x00007fffe2bd1df8 in julia_with_logstate_43206 () at logging.jl:511
#27 0x00007fffe0fc7f4d in julia_with_logger_2433 () at logging.jl:623
#28 0x00007fffe0fc7fc2 in jfptr_with_logger_2434 ()
#29 0x00007ffff7485b3e in _jl_invoke (world=<optimized out>, mfunc=0x7ffdbfa54510, nargs=2, 
    args=0x7fffffff7d50, F=0x7fffe492f3a0 <jl_system_image_data+15368416>)
    at /cache/build/default-amdci5-6/julialang/julia-release-1-dot-8/src/gf.c:2365
#30 ijl_apply_generic (F=<optimized out>, args=0x7fffffff7d50, nargs=<optimized out>)
    at /cache/build/default-amdci5-6/julialang/julia-release-1-dot-8/src/gf.c:2547
#31 0x00007fffe0fc7560 in julia_with_progresslogger_2400 ()
    at /home/wmoses/.julia/packages/AbstractMCMC/E5r4y/src/logging.jl:36
#32 0x00007fffe0fc75e9 in jfptr_with_progresslogger_2401 ()
#33 0x00007ffff7485b3e in _jl_invoke (world=<optimized out>, mfunc=0x7ffdbc7709c0, nargs=3, 
--Type <RET> for more, q to quit, c to continue without paging--q
Quit
(gdb) disassemble
Dump of assembler code for function model:
   0x00007fffe0efa110 <+0>:	push   %rbp
   0x00007fffe0efa111 <+1>:	push   %r15
   0x00007fffe0efa113 <+3>:	push   %r14
   0x00007fffe0efa115 <+5>:	push   %r13
   0x00007fffe0efa117 <+7>:	push   %r12
   0x00007fffe0efa119 <+9>:	push   %rbx
   0x00007fffe0efa11a <+10>:	sub    $0x198,%rsp
   0x00007fffe0efa121 <+17>:	movaps %xmm3,%xmm5
   0x00007fffe0efa124 <+20>:	movsd  %xmm2,0x38(%rsp)
   0x00007fffe0efa12a <+26>:	movsd  %xmm1,0x30(%rsp)
   0x00007fffe0efa130 <+32>:	movsd  %xmm0,0x28(%rsp)
   0x00007fffe0efa136 <+38>:	mov    %r9,0x48(%rsp)
   0x00007fffe0efa13b <+43>:	mov    0x6f0(%rsp),%rbx
   0x00007fffe0efa143 <+51>:	xorps  %xmm0,%xmm0
   0x00007fffe0efa146 <+54>:	movaps %xmm0,(%rsp)
   0x00007fffe0efa14a <+58>:	movq   $0x0,0x10(%rsp)
   0x00007fffe0efa153 <+67>:	mov    %fs:0x0,%rax
   0x00007fffe0efa15c <+76>:	mov    -0x8(%rax),%rcx
   0x00007fffe0efa160 <+80>:	movq   $0x4,(%rsp)
   0x00007fffe0efa168 <+88>:	mov    (%rcx),%rax
   0x00007fffe0efa16b <+91>:	mov    %rax,0x8(%rsp)
   0x00007fffe0efa170 <+96>:	mov    %rsp,%rax
   0x00007fffe0efa173 <+99>:	mov    %rcx,0x40(%rsp)
   0x00007fffe0efa178 <+104>:	mov    %rax,(%rcx)
   0x00007fffe0efa17b <+107>:	movaps %xmm0,0x180(%rsp)
   0x00007fffe0efa183 <+115>:	movaps %xmm0,0x170(%rsp)
   0x00007fffe0efa18b <+123>:	movaps %xmm0,0x160(%rsp)
   0x00007fffe0efa193 <+131>:	movaps %xmm0,0x150(%rsp)
   0x00007fffe0efa19b <+139>:	movaps %xmm0,0x140(%rsp)
   0x00007fffe0efa1a3 <+147>:	movaps %xmm0,0x130(%rsp)
   0x00007fffe0efa1ab <+155>:	movaps %xmm0,0x120(%rsp)
   0x00007fffe0efa1b3 <+163>:	movaps %xmm0,0x110(%rsp)
   0x00007fffe0efa1bb <+171>:	movaps %xmm0,0x100(%rsp)
   0x00007fffe0efa1c3 <+179>:	movaps %xmm0,0xf0(%rsp)
   0x00007fffe0efa1cb <+187>:	movaps %xmm0,0x50(%rsp)
   0x00007fffe0efa1d0 <+192>:	movaps %xmm0,0x60(%rsp)
   0x00007fffe0efa1d5 <+197>:	movaps %xmm0,0x70(%rsp)
   0x00007fffe0efa1da <+202>:	movaps %xmm0,0x80(%rsp)
   0x00007fffe0efa1e2 <+210>:	movaps %xmm0,0x90(%rsp)
   0x00007fffe0efa1ea <+218>:	movaps %xmm0,0xa0(%rsp)
   0x00007fffe0efa1f2 <+226>:	movaps %xmm0,0xb0(%rsp)
   0x00007fffe0efa1fa <+234>:	movaps %xmm0,0xc0(%rsp)
   0x00007fffe0efa202 <+242>:	movaps %xmm0,0xd0(%rsp)
   0x00007fffe0efa20a <+250>:	movaps %xmm0,0xe0(%rsp)
   0x00007fffe0efa212 <+258>:	mov    0xe0(%rsp),%rcx
   0x00007fffe0efa21a <+266>:	movsd  (%rbx),%xmm0
   0x00007fffe0efa21e <+270>:	mov    0x180(%rsp),%rax
   0x00007fffe0efa226 <+278>:	movsd  (%rsi),%xmm1
   0x00007fffe0efa22a <+282>:	movsd  %xmm1,0x18(%rsp)
   0x00007fffe0efa230 <+288>:	movq   $0x0,(%rsi)
=> 0x00007fffe0efa237 <+295>:	movsd  (%rax),%xmm3
   0x00007fffe0efa23b <+299>:	movq   $0x0,(%rax)
--Type <RET> for more, q to quit, c to continue without paging--q
Quit
(gdb) p $rax
$1 = 0

@wsmoses
Copy link
Member

wsmoses commented Dec 24, 2022

@devmotion can you eliminate the syntactic sugar and lower this into the base functions being called and we can attempt to determine the source and then fix the segfault.

@wsmoses
Copy link
Member

wsmoses commented Dec 24, 2022

You will need current Enzyme#main

@wsmoses
Copy link
Member

wsmoses commented Dec 25, 2022

Reducing:

using Turing
using Enzyme
using Random
using LogDensityProblems
using AbstractPPL
Enzyme.API.printall!(true)

function modelf(_, vi::DynamicPPL.AbstractVarInfo, __context__::AbstractPPL.AbstractContext)
		right = InverseGamma()
		vn = (AbstractPPL.VarName){:s}()
    	value, logp, vi = DynamicPPL.tilde_assume(__context__, right, vn, vi)
		value
		# s = first(tilde_assume!!(__context__, var"##dist#340", var"##vn#337", __varinfo__))
end

model = (DynamicPPL.Model)(modelf, NamedTuple(), NamedTuple()) | (; x = 0.5)
alg = NUTS{Turing.EnzymeAD}()
        
spl = Turing.Sampler(alg, model)::DynamicPPL.AbstractSampler
rng = Random.GLOBAL_RNG

varinfo = DynamicPPL.VarInfo()
model(rng, varinfo, DynamicPPL.initialsampler(spl), DynamicPPL.DefaultContext())
# md0 = (s=varinfo.metadata,)
md0 = DynamicPPL.TypedVarInfo(varinfo).metadata

# md0 = DynamicPPL.Metadata{Dict{VarName, Int64}, Vector{Distribution}, Vector{VarName}, Vector{Real}, Vector{Set{DynamicPPL.Selector}}}(Dict{VarName, Int64}(s => 1), VarName[s], UnitRange{Int64}[1:1], Real[1.3478497124090778], Distribution[InverseGamma{Float64}(
# invd: Gamma{Float64}(α=1.0, θ=1.0)
# θ: 1.0
# )
# ], Set{DynamicPPL.Selector}[Set()], [0], Dict{String, BitVector}("del" => [0], "trans" => [0]))

    # Extract parameters.
    # theta = vi[spl]
    theta = Float64[1.8470301477692637]
 
function hack(md0, theta)
	md = DynamicPPL.newmetadata(md0, Val(()), theta)

    vi_new = DynamicPPL.VarInfo(
        md, Base.RefValue{Float64}(1.6664252371660568), Ref(0)
    )
	
	right = InverseGamma()
	vn = (AbstractPPL.VarName){:s}()
	r = vi_new[vn, right]
	logpdf(InverseGamma(), r[1])
	# @assert haskey(vi_new, vn) "Dynamic PPL"
	# r = DynamicPPL.getindex_raw(vi_new, vn, right)
	# r = DynamicPPL.maybe_invlink(vi_new, vn, right, r)
	# map(x -> logpdf(InverseGamma(), x), r) #- logabsdetjac(bijector(right), r)
	return nothing 
end

hack(md0, theta)

	Enzyme.autodiff(Enzyme.Reverse, hack, Enzyme.Const, Const(md0),
                    Enzyme.Duplicated(theta, zero(theta)))
                    ```

@wsmoses
Copy link
Member

wsmoses commented Dec 25, 2022 

using Enzyme
using SpecialFunctions
Enzyme.API.printall!(true)

theta = Float64[2.0]

@noinline function vew(vals)
	view(vals, 1:1)
end

@noinline function gi(vals)
	r = vew(vals)[1]
	return r
end

function logpd(x::Real)
    (α, θ) = (1.0, 1.0)
    - loggamma(α) - log(x) - θ / x
end
 
function hack(theta)
	r = gi(theta)
	logpd(r[1])
	return nothing 
end

hack(theta)

Enzyme.autodiff(Enzyme.Reverse, hack, Enzyme.Const, Enzyme.Duplicated(theta, zero(theta)))

@wsmoses
Copy link
Member

wsmoses commented Dec 25, 2022

Once this (EnzymeAD/Enzyme#959) lands and there's a jll bump this successfully runs. There's separate work to be done to reduce compile time, warnings, etc -- but one step at a time.

│ 
│ Stacktrace:
│  [1] Fix1
│    @ ./operators.jl:0
└ @ Enzyme.Compiler ~/.julia/packages/GPUCompiler/hi5Wg/src/utils.jl:35
┌ Info: Found initial step size
└   ϵ = 3.2
Sampling 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|

@wsmoses wsmoses closed this as completed Dec 29, 2022
@sethaxen sethaxen mentioned this issue Mar 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
segfault
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@devmotion @vchuravy @wsmoses