On shared library handling

[flysand7]

Context

The issue regarding distribution of code that links to shared libraries using foreign imports has been brought up a few times now in various discussions and issues. Linking with shared objects is necessary in certain cases:

• When Odin's vendor library doesn't provide a static version of the library (e.g. SDL2)
• When you library is literally shared between two separate modules (most commonly seen in code that does hotloading), and this library contains a global state
• When a library literally cannot be linked statically (see vdso man page. We'll eventually want to link to it for the long-term goal of fully supporting -no-crt)

Summary of the current situation:

Shared objects are imported with foreign import:

when ODIN_OS == "windows" {
  foreign import lib "mydll.lib"
} else when ODIN_OS == "linux" {
  foreign import lib "libmy.so.5"
} else when ODIN_OS == "darwin" {
  foreign import lib "my.dylib"
}

This will put that shared library into executable's dependencies, and it will be searched for in runtime. The paths searched depend on platform as well as special entries in the executable image.

On Linux (and OSX?) the search will only occur in directories specified by DT_RPATH entry in the elf file, $LD_LIBRARY_PATH, the directories specified by config files in /etc/ld.so.conf.d, /usr, /usr/lib (in that order). None of these have the executable's directory by default.

Foreign imports have a special system: collection, however specifying this makes no difference in terms of behavior.

On linux there's a bug that hasn't been brought up, where if a library doesn't have a .so suffix (for example if it's versioned like in the example above) it will be treated as a system library (Similarly to system:c or system:pthread) and will use incorrect linker syntax to import itself.

The proposal

1. Allow specifying search paths for the linker

The users need to distribute code in such a way where the accompanying shared objects are bundled together with the executable (when distributing an pre-built archive). By default on Linux and OSX rpath doesn't contain the path to the current executable.

A -relative-path flag has been proposed in #894. The proposed version with embedding search paths into the import declaration is weird, easily misused, unreliable and doesn't work with vendor: collection (which currently import shared libraries on their behalf). The command-line option is what probably the user wants.

The reason for the flag as opposed to hardcoding it to be the path of the executable is because maybe the user wants the libraries stashed away in lib or bin or whatever folder relative to the executable.

2. Auto copy the shared libraries

The vendor collection can import shared libraries that are pre-compiled and shipped with Odin, but these libraries have no way of finding themselves in a place where dynamic linker can find them. Currently the way you would make a binary that ships with the DLLs is:

• Manually copy the shared objects from the vendor library
• Specify -rpath=lib with -extra-linker-args

This solution has a drawback is that if you're cloning a repository that depends on shared objects in vendor package, the person who has cloned the repository has to do that manually. The developer cannot put that copying operation into the build script because the path where Odin is installed is unknown to the shell environment.

Having dynamic objects directly in the repository is wrong for multiple reasons, but most importantly - current vendor bindings and the dll in the repo may become out of sync.

Being able to just clone any repository and run the code is really convenient and I don't want that convenience to be broken by random-ass linker issues.

Rather than trying to come up with roundabout ways to allow this copying I propose that Odin compiler should do the job of copying. The library shall be cloned from the location relative to the package importing it library to the location specified in -lib-path compiler option.

(Note: -lib-path is chosen over -relative-path because it is more clear what it's used for. The name is up for debate)

The sensible default is: if -lib-path is not specified the libraries will be copied to the same directory where the executable is built in.

In order to prevent the compiler from being confused, this option is allowed to be specified only once.

Note that shared libraries don't always need to be copied. In particular, when it's a system: library. Guess what, now system: pseudo-collection has actual meaning attached to it!

Discussing the usecases

1. Distributing bindings

The bindings would have a directory structure similar to this:

libxxx/
|- bin/
|  |- libxxx-windows.lib
|  |- libxxx-linux.a
|  |- libxxx-windows.dll
|  |- libxxx-windows-dll.lib
|  +- libxxx-linux.so
+- libxxx.odin

The part of libxxx that does the shared foreign imports will look like this (the same as before):

when ODIN_OS == "windows" {
  foreign import "bin/libxxx-windows.dll"
} else when ODIN_OS == "linux" {
  foreign import "bin/libxxx-linux.so"
}

The user will clone the download these bindings on their machine and copy them to the project directory

./
|- assets/*
|- game/
|  +- libxxx/*
|- bin/
|- build.sh 

The game will use the following Odin compiler flags when building:

odin build game -lib-path:bin

This will copy libraries to the bin folder. Anyone would be able to clone, build and run that game from its repository if it's shipped like that, when the developer decides to ship the game they only need to archive the assets, bin and game.exe, and the link paths are guaranteed to be resolved.

[flysand7]

One issue come up when discussing this proposal in discord, is how does the compiler know which library to copy. On linux and OSX the situation is trivial you just copy the file imported, on windows the situation is different. The file imported is going to be .lib, however the .dll is what actually needs to be copied.

One way to solve this without modifying foreign import syntax is to enforce a policy on naming of dll libraries. (libxxxdll.lib must contain the symbols for libxxx.dll)

The other way is to add an attribute, which increases the amount of syntax in the language, but that's also an option.