FFITypes for non-base classes are never removed from typeInfoMap, causing the Cleaner thread to loop forever

[emarteca]

The FFIType class has a static WeakHashMap of type information, the typeInfoMap. Since it is a weak map, key-value pairs would be removed if the key was garbage collected. However, since the keys in this map are Classes, they are extremely unlikely to ever be collected.

For classes that are Structures, the FFIType stored in the typeInfoMap wraps the structure object itself, here. However, when the FFIType is a Structure, there is underlying Memory in it, which has a MemoryDisposer registered on it.

Since the Class is never unloaded, the FFType cannot be garbage collected either -- however, the Cleaner thread expects to be able to clean up its Memory, and so it will loop forever waiting to be able to clean it.

---

Potential solutions: I am not sure the best way to go about fixing this problem, but there are two approaches I think of first:

1. Change the way FFITypes for Structures are made, so they don't have any underlying Memory.
2. Use a different, special kind of Memory for FFITypes, that has no cleaner registration. Since this memory corresponds to a Class, I think it could be safe to assume there are a limited number of classes being loaded.

---

Let me know what you think. I''m trying to hack around this problem right now in my project, and will update this issue when I have a temporary solution too.

[matthiasblaesing]

Could you please provide a minimal reproducer, that shows the problem?

[emarteca]

Yes, for sure! Here is a simple Kotlin program that reproduces the issue:

import com.sun.jna.Structure
import com.sun.jna.internal.Cleaner

// simple struct
internal class NativeWrapper: Structure(), Structure.ByValue {
    @JvmField
    internal var field: Boolean = false;
  
    // Define the fields of the struct
    override fun getFieldOrder(): List<String> {
        return listOf("field")
    }
}

// function to get the cleaner thread
fun get_cleaner_thread(): Thread? {
    val threadName = "JNA Cleaner"
    for (t: Thread in Thread.getAllStackTraces().keys) {
        if (t.getName().equals(threadName)) return t;
    }
    return null;
}

object Main {
    @JvmStatic
    fun main(args: Array<String>) {
        if(true) {
               var w: NativeWrapper = NativeWrapper() // create a struct that immediately goes out of scope
        }
        Thread.sleep(1000)
        System.gc()
        Thread.sleep(30000) // current waiting timeout on the Cleaner thread
        val cleaner_thread = get_cleaner_thread()
        // at this point, the Cleaner thread should be done, but it is not
        if (cleaner_thread != null)
            System.out.println(cleaner_thread.getState())
    }
}

This program just makes a minimal Structure. Internally, there is an FFIType created for the custom Structure, to represent my type NativeWrapper.

The struct value w immediately goes out of scope, and so it should be collected by the garbage collector. I add a sleep to allow the Cleaner to wake up, and also call the GC. I then add a sleep that's the same length as the Cleaner thread "linger" time, to make sure the Cleaner has time to wake up and start running. At this point, the struct and its sub-components should all be cleaned up, and the Cleaner thread should be in state TERMINATED (see here how the Cleaner thread exits its run function when the cleanup queue is empty). However, the state printed is either RUNNABLE or TIMED_WAITING, which means the thread is not done.

---

Internally, what's going on is that the FFIType for the NativeWrapper custom Structure is never getting unloaded from the typeInfoMap in FFIType, since the NativeWrapper class is not getting unloaded. This means that the FFIType value cannot be cleaned (and it has been registered with the Cleaner because of its underlying Memory), since it is still alive.

[matthiasblaesing]

I had to adjust the sample (after translating to java), as the FFIType is only allocated, when the structure is actually allocated. My test case:

import com.sun.jna.Structure;
import com.sun.jna.Structure.FieldOrder;

public class JNACollection {

    @FieldOrder({"field"})
    public static class NativeWrapper extends Structure {
        public boolean field = false;
    }

    private static Thread getCleanerThread() {
        for (Thread t : Thread.getAllStackTraces().keySet()) {
            if ("JNA Cleaner".equals(t.getName())) {
                return t;
            }
        }
        return null;
    }

    public static void main(String[] artv) throws InterruptedException {
        {
               NativeWrapper nw = new NativeWrapper();
               nw.write();
               System.out.println(nw);
        }
        Thread.sleep(1000);
        System.gc();
        Thread.sleep(60000);
        Thread cleanerThread = getCleanerThread();
        System.out.println(cleanerThread);
    }
}

I see the problem, but I raise the question, how bad this is? This is only one of several hard to close things in JNA, that will keep the cleaner alive.

The reason for the existence of the cleaner is, that there is no defined lifecylce for objects tied to native. They are created when needed and are kept alive until their java anchor goes away. I see only a way to fix this by making all objects closeable and use them with try-with-resource, but that has major usage/api impact.