Why not use godot-cpp? / Using a specific Python environment

[mzwaal]

Hi Mai,

Great effort because I am looking to use some Python libraries in project I made.

I have created an cpp extension myself once and used the godotcpp repo as an interface to the Godot objects. I am curious about why you would want link to the actual Godot binary? I believe you can do the same with just using godotcpp .

An example can be found here btw (including some github actions to build):

https://github.com/nathanfranke/gdextension

I am also curious about using a specific Conda environment and copying it to the game project. Would it be as simple as just making a copy and pointing to it?

Cheers!

[maiself]

This project links with neither Godot or godot-cpp, nor does it need to. It uses the APIs defined in gdextension_interface.h and generates the remaining bindings dynamicly at runtime. This simplifies the bindings, reduces build times and sizes, and allows for greater flexibility.

As for your other question, I haven't gotten around to figuring how integration with any sort of existing package management should work.

The current situation is any pure Python code found within the project directory should just work. There is a project setting to specify res:// paths as additional module search paths if you need to organize things a certain way.

For Python extension modules (numpy for example), the situation is that no specific handing yet exists. For an exported project there should exist the usual site-packages subdirectory somewhere beside the export. Non-exported may work from within the project directory, but there isn't any export code yet, so you'd have to copy manually.

[GeorgeS2019]

the usual site-packages subdirectory

Will there be possibility to use e.g. pip to install python packages into site-packages?

[maiself]

Will there be possibility to use e.g. pip to install python packages into site-packages?

I'd really like to see something like that working, and I imagine its possible, but investigation would need to be done on how to make it work, including considerations such as multiple platforms and editor vs exported projects.

Does anyone have any insight or suggestions on the matter? At the moment I'm working on finishing up other features and don't have the time to spare to dig into it.

[hemebond]

What does it mean to use pip in this context? Is it not already possible if you create a virtual environment in the project directory?

[GeorgeS2019]

I do not understand why the file structure is arranged in this way. Why this is distributed as Zip

There are some inconsistencies of the documentation
This is the file structure I get from the Github Action Windows artifact

entry_symbol = "python_extension_init"

``` python_extension_init libgodot-python.windows.x86_64.dll ```

[configuration]

compatibility_minimum = 4.2
entry_symbol = "python_extension_init"

[libraries]

macos = "res://bin/macos/libgodot-python.macos.framework"

windows.x86_32 = "res://bin/windows-x86_32/libgodot-python.windows.x86_32.dll"
windows.x86_64 = "res://bin/windows-x86_64/libgodot-python.windows.x86_64.dll"

linux.x86_64 = "res://bin/linux-x86_64/libgodot-python.linux.x86_64.so"
linux.arm64 = "res://bin/linux-arm64/libgodot-python.linux.arm64.so"
linux.rv64 = "res://bin/linux-rv64/libgodot-python.linux.rv64.so"

android.x86_64 = "res://bin/android-x86_64/libgodot-python.android.x86_64.so"
android.arm64 = "res://bin/android-arm64/libgodot-python.android.arm64.so"

Properties and Methods registered AND initialization called by "python_extension_init"

https://github.com/maiself/godot-python-extension/blob/master/src/extension/extension.cpp

extern "C" PYBIND11_EXPORT GDExtensionBool python_extension_init(
	GDExtensionInterfaceGetProcAddress get_proc_address,
	GDExtensionClassLibraryPtr library,
	GDExtensionInitialization* initialization)
{
	using namespace pygodot;

	auto* raw_interface = reinterpret_cast<uint32_t*>(get_proc_address);
	if(raw_interface[0] == 4 && raw_interface[1] == 0) {
		printf("ERROR: Cannot load a GDExtension built for Godot 4.1+ in legacy Godot 4.0 mode.\n");
		return false;
	}

	extension_interface::print_error =
		reinterpret_cast<GDExtensionInterfacePrintError>(get_proc_address("print_error"));

	if(!extension_interface::print_error) {
		printf("ERROR: Unable to load GDExtension interface function print_error()\n");
		return false;
	}

	extension_interface::library = library;
	extension_interface::token = library;


	// TODO: work this into the build system

	// grep -r -h -o --color=no -E '\bextension_interface::\w+' ./src/ | sort | uniq | sed -E 's/\w+::(\w+)/\t\t"\1",/g'

	std::vector<std::string> referenced_extension_apis = {
		"callable_custom_create",
		"callable_custom_get_userdata",
		"classdb_construct_object",
		"classdb_get_method_bind",
		"classdb_register_extension_class",
		"classdb_register_extension_class_integer_constant",
		"classdb_register_extension_class_method",
		"classdb_register_extension_class_property",
		"classdb_register_extension_class_property_group",
		"classdb_register_extension_class_property_subgroup",
		"classdb_register_extension_class_signal",
		"get_godot_version",
		"get_library_path",
		"get_variant_from_type_constructor",
		"get_variant_to_type_constructor",
		"global_get_singleton",
		"godot_version",
		"library",
		"name",
		"object_destroy",
		"object_get_class_name",
		"object_get_instance_binding",
		"object_get_instance_id",
		"object_method_bind_ptrcall",
		"object_set_instance",
		"object_set_instance_binding",
		"packed_byte_array_operator_index_const",
		"placeholder_script_instance_create",
		"placeholder_script_instance_update",
		"print_error",
		"print_warning",
		"ref_set_object",
		"script_instance_create",
		"string_new_with_utf8_chars",
		"string_new_with_utf8_chars_and_len",
		"string_to_utf8_chars",
		"token",
		"variant_call",
		"variant_destroy",
		"variant_get_ptr_builtin_method",
		"variant_get_ptr_constructor",
		"variant_get_ptr_destructor",
		"variant_get_ptr_getter",
		"variant_get_ptr_indexed_getter",
		"variant_get_ptr_indexed_setter",
		"variant_get_ptr_keyed_getter",
		"variant_get_ptr_keyed_setter",
		"variant_get_ptr_operator_evaluator",
		"variant_get_ptr_setter",
		"variant_get_ptr_utility_function",
		"variant_get_type",
		"variant_new_nil",
		"variant_stringify"
	};


#define GDEXTENSION_API(name, type) \
	if(std::find(referenced_extension_apis.begin(), referenced_extension_apis.end(), #name) \
		!= referenced_extension_apis.end()) \
	{ \
		extension_interface::name = reinterpret_cast<type>(get_proc_address(#name)); \
		if(!extension_interface::name) { \
			extension_interface::print_error("Unable to load GDExtension interface function " #name "()", \
				__FUNCTION__, __FILE__, __LINE__, false); \
			return false; \
		} \
	} \
	else { \
		extension_interface::name = reinterpret_cast<type>(static_cast<void(*)()>([]() { \
			extension_interface::print_error("Cannot call unloaded GDExtension interface function " #name "()", \
				__FUNCTION__, __FILE__, __LINE__, false); \
			std::fflush(stdout); \
			std::abort(); \
		})); \
	}

	GDEXTENSION_APIS

#undef GDEXTENSION_API

	extension_interface::get_godot_version(&extension_interface::godot_version);

	GDExtensionGodotVersion minium_version = {.major = 4, .minor = 2, .patch = 0, .string = nullptr};

	if(version_to_uint(extension_interface::godot_version) < version_to_uint(minium_version)) {
		char error_msg[256];
		snprintf(error_msg, sizeof(error_msg),
			"Minimum required version for Python extension not met. Need v%d.%d.%d or newer, have v%d.%d.%d\n",
			minium_version.major, minium_version.minor, minium_version.patch,
			extension_interface::godot_version.major,
			extension_interface::godot_version.minor,
			extension_interface::godot_version.patch
		);

		extension_interface::print_error(error_msg, __FUNCTION__, __FILE__, __LINE__, false);

		return false;
	}

	initialization->initialize = initialize_python_module;
	initialization->deinitialize = uninitialize_python_module;
	initialization->minimum_initialization_level = GDEXTENSION_INITIALIZATION_CORE;

	return true;
}

	initialization->initialize = initialize_python_module;
	initialization->deinitialize = uninitialize_python_module;
	initialization->minimum_initialization_level = GDEXTENSION_INITIALIZATION_CORE;

void initialize_python_module(void* userdata, GDExtensionInitializationLevel level) {
	if(level == GDEXTENSION_INITIALIZATION_CORE) {
		try {
			//printf("initializing interpreter...\n");

			init_python_isolated();

			//printf("interpreter initialized\n");

			released_gil = std::make_unique<py::gil_scoped_release>();
		}
		CATCH_FATAL_EXCEPTIONS_PRINT_ERRORS_AND_ABORT("During godot python module initialization")

		py::gil_scoped_acquire gil;

		try {
			initialization_level = level;

			const char* lib_dir = std::getenv("GODOT_PYTHON_MODULE_LIB_DIR");
			if(lib_dir && strlen(lib_dir) > 0) {
				auto sys = py::module_::import("sys");
				auto pathlib = py::module_::import("pathlib");

				if(!pathlib.attr("Path")(lib_dir).attr("joinpath")("godot").attr("is_dir")().cast<bool>()) {
					throw std::runtime_error(std::string("if set 'GODOT_PYTHON_MODULE_LIB_DIR' must be a directory containing the 'godot' python module, path '") + lib_dir + "' does not have a 'godot' subdirectory");
				}

				sys.attr("path").attr("insert")(0, lib_dir);
			}
			else {
				// embedded module
				if(!_init_godot_module()) {
					throw std::runtime_error("the 'godot' python module was not embedded, the environment variable 'GODOT_PYTHON_MODULE_LIB_DIR' must be set to a directory containing the 'godot' module");
				}
			}

			py::module_::import("_gdextension");
			py::module_::import("godot._internal");
		}
		CATCH_FATAL_EXCEPTIONS_PRINT_ERRORS_AND_ABORT("During godot python module initialization")
	}

	if(level >= GDEXTENSION_INITIALIZATION_CORE) {
		initialization_level = level;

		py::gil_scoped_acquire gil;
		try {
			py::module_::import("godot._internal").attr("initialize")(level);
		}
		CATCH_FATAL_EXCEPTIONS_PRINT_ERRORS_AND_ABORT()
	}
}

File Structures (bright background images)

[GeorgeS2019]

Mechanism

Setting the GODOT_PYTHON_MODULE_LIB_DIR environment variable will change where the godot module is loaded from.

When set to the path of the source lib directory, the godot module will be loaded from that directory instead of the embedded module archive.

file `python312.zip` (bright background image)

For example, if the project's repository is checked out to ~/src/godot-python-extension then starting the Godot editor may look like this:

GODOT_PYTHON_MODULE_LIB_DIR=~/src/godot-python-extension/lib godot -e

godot-python-extension/src/extension/extension.cpp

Line 247 in 3eb39bd

const char* lib_dir = std::getenv("GODOT_PYTHON_MODULE_LIB_DIR");

https://github.com/maiself/godot-python-extension/blob/master/lib/godot/__init__.py

file listing (bright background image)

[GeorgeS2019]

[WIP]

I think during Github Action, the embedded module archive. needs to have the proper file structures which include a typical e.g. conda distributed structure that provide e.g. python.exe and pip.exe for package management in the case of Windows

This godot python extension just need to know where to look for site-package folder

The site package folder is independently managed by pip.exe using command line.

Which mean, the user need to unzip the embedded module archive or pip is done through method instead of manual command line pip.exe

OR

https://github.com/touilleMan/godot-python/blob/godot4-meson/scripts/python_distrib.py

Install Windows
https://github.com/touilleMan/godot-python/blob/c2f25e451a861e6ea7545fc3e96c9da99dbdb2a9/scripts/python_distrib.py#L215

[maiself]

@GeorgeS2019, please don't copy large portions of code into comments, its a bit distracting from the discussion and doesn't communicate clearly. Smaller snippets and direct links to potions of code, and a description of why you're referencing it will be much more effective. Right now I'm not really sure what you are trying to say.

(Also please be careful, copying code like that could violate terms of the projects you coped from and potentially cause some trouble for you...)

I'm having trouble with the screenshots, as the bright background hurts my eyes (I'm a dark mode only user). If, in the future, you could post such listings as copyable text that would help a lot. The extra formatting is also distracting, but I want to thank you for using collapsible details.

I hope you don't mind, but I made some quick edits to your comments so that I can try to understand better. I've run out of time today, but I'll likely have another look and reply tomorrow.

[maiself]

@hemebond

What does it mean to use pip in this context? Is it not already possible if you create a virtual environment in the project directory?

It likely is (haven't tried it tho), but the issue is the difference between editor and exported versions of projects. In an exported project the entire project directory is bundled into the pck file. For pure python modules that's fine, but any shared libraries wont be loadable by the operating system when bundled that way.

There's also the issue of distributing for multiple platforms. It would be nice if it were possible to list some required packages, and have things pulled into the project automatically, and also selectively for each export platform. I haven't had time to investigate any of this, so I'm not really sure how things will go.