Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyCall on thread != main: process will not exit #186

Open
snickell opened this issue Aug 8, 2024 · 2 comments · May be fixed by #187
Open

PyCall on thread != main: process will not exit #186

snickell opened this issue Aug 8, 2024 · 2 comments · May be fixed by #187

Comments

@snickell
Copy link

snickell commented Aug 8, 2024

If you use PyCall from only one thread , but that thread is NOT the main thread, the process will not exit when the main thread exits.

This is not the same issue as: "Is PyCall Thread Safe" #96, as we are only using PyCall from one thread:

  1. PyCall is only called from one thread: "side_thread"
  2. side_thread exits
  3. Process still does not exit

Example:

#!/usr/bin/env ruby

def run(do_pycall_import:)
  if do_pycall_import
    puts "Running with do_pycall_import=true, process will not exit when main thread is done"
  else
    puts "Running with do_pycall_import=false, process will exit when main thread is done"
  end

  puts "Main thread ID: #{Thread.current.object_id}\n"

  def print_threads
    puts
    puts "Threads:"
    Thread.list.each do |thread|
      puts "\tThread ID: #{thread.object_id}, Status: #{thread.status}, Name: #{thread.name}"
    end
    puts
  end

  Thread.new do
    Thread.current.name = "side_thread"
    Thread.current.abort_on_exception = true

    # Demonstrate that PyCall has not been used:
    raise "Only load pycall in this thread" if defined?(PyCall)

    require 'pycall'

    if do_pycall_import
      puts "side_thread: import sys"
      # This will initialize libpython, if this happens, the process wil not exit:
      PyCall.import_module('sys')
    end

    sleep 2
    puts "side_thread: exiting"
  end

  sleep 1
  print_threads() #=> Two threads: main and side_thread

  sleep 4
  print_threads() #=> One thread: main
end

if __FILE__ == $0
  run(do_pycall_import: true) #=> Process does NOT exit after printing "End of main thread"
  # run(do_pycall_import: false) #=> Process exits after printing "End of main thread"
end

at_exit { puts "at_exit called"}
puts "End of main thread"

Output when do_pycall_import: true

zsh >> ./pycall_hangs_main_thread.rb
Running with do_pycall_import=true, process will not exit when main thread is done
Main thread ID: 60
side_thread: import sys

Threads:
        Thread ID: 60, Status: run, Name: 
        Thread ID: 80, Status: sleep, Name: side_thread

side_thread: exiting

Threads:
        Thread ID: 60, Status: run, Name: 

End of main thread
at_exit called

#=> control never returns to the shell

Output when do_pycall_import: false

zsh >> ./pycall_hangs_main_thread.rb
Running with do_pycall_import=false, process will exit when main thread is done
Main thread ID: 60

Threads:
        Thread ID: 60, Status: run, Name: 
        Thread ID: 80, Status: sleep, Name: side_thread

side_thread: exiting

Threads:
        Thread ID: 60, Status: run, Name: 

End of main thread
at_exit called

zsh >> # notice, process exited
@snickell snickell changed the title If you use PyCall on a thread != main: process will not exit If you use PyCall on a single thread, but not the main thread: process will not exit Aug 8, 2024
@snickell snickell changed the title If you use PyCall on a single thread, but not the main thread: process will not exit If you use PyCall on one thread, but thread != main: process will not exit Aug 8, 2024
@snickell snickell changed the title If you use PyCall on one thread, but thread != main: process will not exit PyCall on thread != main: process will not exit Aug 8, 2024
@snickell
Copy link
Author

snickell commented Aug 8, 2024

I noticed we do not call Py_FinalizeEx(). If I call Py_FinalizeEx() at the end of my thread: the process exits! 🥳

Changes in my repro code:

  Thread.new do

    # CUT TO MAKE SHORT

    # I have modified pycall.c to add this method, which calls Py_API(Py_FinalizeEx)()
    PyCall.finalize
  end

With these changes, now the main process exits.

@snickell snickell linked a pull request Aug 8, 2024 that will close this issue
@snickell
Copy link
Author

snickell commented Aug 8, 2024

I have started a fix for this here: #187, but its not complete yet. It permits the process to exit, but sometimes it segfaults at exit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant