Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nov 5th 2024 update Linux not working w/Valheim : Unable to launch game via a mod manager with mods. #708

Open
AngelQC opened this issue Nov 13, 2024 · 25 comments

Comments

@AngelQC
Copy link

AngelQC commented Nov 13, 2024

Your system information

  • Steam Runtime Version: 0.20240916.0+srt1
  • Distribution (e.g. Ubuntu 18.04): Arcolinux, rolling release current.
  • Link to your full system information (Help -> Steam Runtime Diagnostics) in a Gist:
  • Have you checked for system updates?: [Yes/No] Yes, system up to date.
  • What compatibility tool are you using?: [None / Steam Linux Runtime / Proton 5.13+ / older Proton] None, native game.
  • What versions are listed in steamapps/common/SteamLinuxRuntime/VERSIONS.txt?
    • depot 0.20240806.0
    • scripts 0.20240806.0
  • What versions are listed in steamapps/common/SteamLinuxRuntime_soldier/VERSIONS.txt?
    • depot 0.20241111.107902
    • pressure-vessel 0.20241111.0
    • scripts 0.20241111.0
    • soldier 0.20241111.107902
    • soldier_platform_0.20241111.107902
  • What versions are listed in steamapps/common/SteamLinuxRuntime_sniper/VERSIONS.txt?
    • depot 0.20241008.104210
    • pressure-vessel 0.20240916.0
    • scripts 0.20240916.0
    • sniper 0.20241008.104210
    • sniper_platform_0.20241008.104210

Please describe your issue in as much detail as possible:

I have been paying Valheim exclusively on linux, various distributions, since march 2021. I have been using Arcolinux now for close to a year without a hitch. Since the November 5th patch, I have been unable to launch the game with mods through my usual method, r2modman. This is a mod manager for launching Unity-engine games with, or without, mods.

At first the game wasn't launching at all, even vanilla, until this tip was given into another thread, using a newer version of the steam soldier. With this version the game launches without mods, not having PlayFab-related problems anymore.

Then launching the game with any amount of mods will not make it to recreating the player.log file, so I basically do not have a working log to work with.

Being the tech that I am, I digged into the two scripts used by r2modman to launch the game, and saw two different methods/executables

  • /home/jay/.local/share/Steam/ubuntu12_32/steam-launch-wrapper
  • /home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim.x86_64

The former crashes without generating the expected Valheim log file,
The later goes further, so I tricked the script into starting the bin file instead of the wrapper and I get the log, which ends with :

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
0x758c9611565a - /home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so : 
0x758c960be049 - /home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so : 
0x758c960434d0 - /home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so : 
0x758cbcb2f1d0 - /usr/lib/libc.so.6 : 
0x758c1037001e - /usr/lib/libvulkan_radeon.so : 
0x758c103741af - /usr/lib/libvulkan_radeon.so : 
0x758c103721af - /usr/lib/libvulkan_radeon.so :   

=================================================================
	Telemetry Dumper:
=================================================================
Thread 0x758bfce006c0 may have been prematurely finalized* Assertion at mono-threads.c:702, condition `info' not met, function:mono_thread_info_current, 

An error has occured in the native fault reporting. Some diagnostic information will be unavailable.
...
=================================================================
	External Debugger Dump:
=================================================================
ERROR: ld.so: object '/home/jay/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
/usr/bin/gdb: /home/jay/.local/share/Steam/ubuntu12_32/steam-runtime/pinned_libs_64/libcurl.so.4: version `CURL_OPENSSL_4' not found (required by /usr/lib/libdebuginfod.so.1)

Exiting early due to double fault.

..
Here is the full log

Normally, this execution ends with the game screen being shows.

I am still browsing through ~/.steam/steam/log files, trying to find some hints as to what Valheim could leave behind when exiting, no luck so far..

Steps for reproducing this issue:

  1. Install Valheim
  2. Install r2modman from your distribution, aur or thunderstore.io
  3. Modify the game's launch string as directed by the tool
  4. Install some mods
  5. Start the game "modded". If Steam is open in the Library view, you will see the game's states:
  • Launching
  • Running
  • None
  1. To verify the proper execution, from the tool also start the game "Vanilla", without mods, which will work.
  2. To verify further more, setting the game to the previous version "default_old", starting "modded" -will- work
@AngelQC
Copy link
Author

AngelQC commented Nov 13, 2024

@smcv As suggested, new issue opened.

@smcv
Copy link
Contributor

smcv commented Nov 13, 2024

Arcolinux, rolling release current

For easy reference: this appears to be an Arch derivative.

Modify the game's launch string as directed by the tool

Please assume that I don't know anything about Valheim or this mod manager (I don't currently have access to a copy of Valheim). What modifications does it direct you to make?

We cannot really support modded games and third-party tools, especially if they insert arbitrary code into the critical path of launching a particular game. Obviously it would be nice if we can make this work transparently, but we can't guarantee anything.

From your system report, it seems that the LD_LIBRARY_PATH has had this inserted into it, before launching Steam:

/tmp/.mount_r2modmE4mtZH/usr/lib

but according to the diagnostic tool, the container runtime ends up removing this directory from the search path (because the purpose of the container runtime is to present the game with a more predictable runtime environment, without application-level libraries from the host OS).

What libraries exist in that directory? Depending on what's here, this could be overriding a system/container library in ways that could interfere with correct functioning of various components. Or, if it contains libraries that are required by a component that is in LD_PRELOAD or a similar code-injection mechanism, the fact that it is not picked up from the host system could be what's breaking things.

@smcv
Copy link
Contributor

smcv commented Nov 13, 2024

[steam-launch-wrapper] crashes without generating the expected Valheim log file

steam-launch-wrapper is necessary for correct functioning of some parts of the Steam launch path, so we should treat this crash as being a problem in its own right.

As a first diagnostic step, please try this:

  • revert whatever workarounds you have been using, e.g. to bypass steam-launch-wrapper
  • set the game's Properties → Launch Options to STEAM_LINUX_RUNTIME_VERBOSE=1 %command%
    • if they are already set non-empty, instead prepend STEAM_LINUX_RUNTIME_VERBOSE=1
  • try to launch the game

There are several items of diagnostic information you will be able to get from this.

One item is that ~/.steam/steam/logs/console_log.txt should have a long command-line in it, similar to this one. In the real file it's one very long line, but for this example I've broken it into shorter lines to be easier to read:

[2024-11-11 12:05:45] /bin/sh\0-c\0
/home/you/.local/share/Steam/ubuntu12_32/steam-launch-wrapper --
/home/you/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=something --
'/home/you/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point --verb=waitforexitandrun --
'/home/you/.local/share/Steam/steamapps/common/SteamLinuxRuntime'/scout-on-soldier-entry-point-v2 --
'/home/you/.local/share/Steam/steamapps/common/Something'/something\0

Please copy/paste that log line here.

And, in ~/.steam/steam/logs/console-linux.txt, there will be some output from launching the game - as a minimum there will be some messages mentioning the same app-ID (which I think should be 892970 for Valheim), but there might also be a crash message of some sort. Please copy/paste that whole section, or put it in a Gist.

If the mod tool redirects stdout/stderr from the actual game to some file, then please look there for output instead. I don't know any specifics of how this mod tool behaves, so I'll have to leave it up to you to find what redirections it does (if any).

Another important piece of debugging information is whatever information you can get about the crash, for example a backtrace. Usually the easiest way to collect this is to enable a system-wide core dump collection facility like systemd-coredump, corekeeper, apport or abrt, and let that collect and analyze the crash dump. Please see your distribution's documentation for how to collect backtraces, or if your distribution does not document this, Arch might.

If the launch procedure gets far enough to be starting the container, then STEAM_LINUX_RUNTIME_VERBOSE=1 will result in it recording a lot of debugging output (multiple pages' worth of text), which I'll also need to see as a Gist or attachment. The first few lines will look something like this:

_v2-entry-point[2244]: argv (unescaped): xterm
pressure-vessel-wrap[2244]: D: Enabled profiling
pressure-vessel-wrap[2244]: D: Hypervisor Present bit not set in CPUID 0x1
pressure-vessel-wrap[2244]: D: Original argv:
pressure-vessel-wrap[2244]: D:  0: '/home/desktop/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier/...
...

and I'll need to see all output up to and including the crash, which might look something like this:

...
pressure-vessel-adverb[2318]: D: Child 2343 exited with wait status 134
pressure-vessel-adverb[2318]: I: Command killed by signal 6
pressure-vessel-adverb[2318]: D: Child 2345 exited with wait status 1
pressure-vessel-adverb[2318]: D: No more child processes
pressure-vessel-adverb[2318]: D: Exiting with status 134
pressure-vessel-adverb[2318]: D: Releasing lock 13

@smcv
Copy link
Contributor

smcv commented Nov 13, 2024

As a secondary (less important) thing, after getting details of the original crash, please describe what was involved in your workaround to "trick the script into starting the bin file instead of the wrapper", and then show me the same debug output that I asked for in the previous comment, but with that workaround re-applied.

I think that your workaround is actually having a side-effect of making the game not run in the container environment, because this stack trace does not look like the container:

	0x758cbcb2f1d0 - /usr/lib/libc.so.6 : 
	0x758c1037001e - /usr/lib/libvulkan_radeon.so : 
	0x758c103741af - /usr/lib/libvulkan_radeon.so : 

If it was running in the container, on a fast-moving rolling-release OS distribution, I would have expected to see /run/host/usr/lib/libc.so.6 and /run/host/usr/lib/libvulkan_radeon.so here.

@AngelQC
Copy link
Author

AngelQC commented Nov 15, 2024

Arcolinux, rolling release current

For easy reference: this appears to be an Arch derivative.

You are absolutely right, a SteamoOS cousin :).

What modifications does it direct you to make?

Basically, when the game is launched from steam, in order to invoke the mod manager a script file is executed, as per:
Launch string:
"/home/jay/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" %command% -console

This wrapper analyse the command line and either starts the server or the game with or without mods. linux_wrapper.sh
I added logging lined in it so I could get an idea of the variables, to make sense of it.

/tmp/.mount_r2modmE4mtZH/usr/lib

What libraries exist in that directory?

r2modman, the mod manager, is installed as a flatpak and not a native arch package. All flatpaks are archives mounted as such, and this specific folder contains libraries used by it.
I would assume the content has nothing the game would use, it's quite limited to 6 files:

libappindicator.so.1
libgconf-2.so.4
libindicator.so.7
libnotify.so.4
libXss.so.1
libXtst.so.6

I take this moment to thank you for the reply, or should I say replies. I am genuinely surprised, and very happy, by the questions. I will reply diligently to each of them.

@AngelQC
Copy link
Author

AngelQC commented Nov 15, 2024

steam-launch-wrapper is necessary for correct functioning of some parts of the Steam launch path, so we should treat this crash as being a problem in its own right.

Right, so I have modified the previously mentioned launch options as:

STEAM_LINUX_RUNTIME_VERBOSE=1 "/home/jay/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" %command% -console

Note

I still use the beta version of Steam Soldier to fix the PlayFab problem

Here is the long command line, from the console_log.txt file:

[2024-11-15 07:38:08] /bin/sh\0-c\0STEAM_LINUX_RUNTIME_VERBOSE=1 "/home/jay/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" /home/jay/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- /home/jay/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=892970 -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point --verb=waitforexitandrun -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime'/scout-on-soldier-entry-point-v2 --  '/home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim.x86_64' '--doorstop-enable' 'true' '--doorstop-target' '/home/jay/.config/r2modmanPlus-local/Valheim/profiles/Takers_Underworld/BepInEx/core/BepInEx.Preloader.dll' '--r2profile' 'Takers_Underworld' -console\0

BepInEx is the key mod that allows for mods in the game, which seems to be refered to as "doorstop". Is is not present when the game is launched as vanilla, even by r2modman. The "profile" is the folder into which there are mods. In R2modman we can define different profiles depending on what mods we want for specifig map/server. Here "Takers_Underworld" server has 93 mods, from which for the sake of speed I only enabled two, BepInExPack_Valheim and MoreGatesExtended, just so there are mods and it does not take minutes to validation before showing the main screen of the game. In both situations, it has the same beharior of crashing at start.

I am afraid there is not much content leading to the crash here. This is the full remainder of the log file:

[2024-11-15 07:38:08] Game process added : AppID 892970 "STEAM_LINUX_RUNTIME_VERBOSE=1 "/home/jay/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" /home/jay/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- /home/jay/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=892970 -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point --verb=waitforexitandrun -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime'/scout-on-soldier-entry-point-v2 --  '/home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim.x86_64' '--doorstop-enable' 'true' '--doorstop-target' '/home/jay/.config/r2modmanPlus-local/Valheim/profiles/Takers_Underworld/BepInEx/core/BepInEx.Preloader.dll' '--r2profile' 'Takers_Underworld' -console", ProcID 808442, IP 0.0.0.0:0
[2024-11-15 07:38:08] Controller slots reset
[2024-11-15 07:38:08] GameAction [AppID 892970, ActionID 74] : LaunchApp changed task to WaitingGameWindow with ""
[2024-11-15 07:38:08] GameAction [AppID 892970, ActionID 74] : LaunchApp changed task to Completed with ""
[2024-11-15 07:38:08] Game process removed: AppID 892970 "STEAM_LINUX_RUNTIME_VERBOSE=1 "/home/jay/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" /home/jay/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- /home/jay/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=892970 -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point --verb=waitforexitandrun -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime'/scout-on-soldier-entry-point-v2 --  '/home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim.x86_64' '--doorstop-enable' 'true' '--doorstop-target' '/home/jay/.config/r2modmanPlus-local/Valheim/profiles/Takers_Underworld/BepInEx/core/BepInEx.Preloader.dll' '--r2profile' 'Takers_Underworld' -console", ProcID 808442
[2024-11-15 07:38:08] ThreadGetProcessExitCode: no such process 808452
[2024-11-15 07:38:08] ThreadGetProcessExitCode: no such process 808451
[2024-11-15 07:38:08] ThreadGetProcessExitCode: no such process 808450
[2024-11-15 07:38:08] ThreadGetProcessExitCode: no such process 808445
[2024-11-15 07:38:08] ThreadGetProcessExitCode: no such process 808443
[2024-11-15 07:39:46] CAPIJobRequestUserStats - Server response failed 2

If the mod tool redirects stdout/stderr from the actual game to some file, then please look there for output instead. I don't know any specifics of how this mod tool behaves, so I'll have to leave it up to you to find what redirections it does (if any).

I am not aware of any logs by r2modman, given the fact that scripts actually starts the game and I don't see any redirects.

coredumps

Hmm I can't find any information about core dumps on my computer. When the game actually launches, there is a crash file in the Valheim folder, multiple if I start and game multiple times, but there are none right now as the process seems to not have reach the game binary being launched, or long enough to create any log. The Player.log file is the previous, vanilla one, it created in my first test before launching with mods, and it has no errors in it.

[Update] I am reading about core dump and backtrace right now, will follow through.
[Update 2] The core dumps I find are from Nov 9th, nothing recent, and relates to the Valheim executable, which in the next post, we agreed this was not the best course of research. I see nothing linked to the wrapper/steam directly.

@AngelQC
Copy link
Author

AngelQC commented Nov 15, 2024

please describe what was involved in your workaround to "trick the script into starting the bin file instead of the wrapper", and then show me the same debug output that I asked for in the previous comment, but with that workaround re-applied.

It was simple enough to do. The secondary launch script start_game_bepinex.sh which starts the bepinex by prepending the library tests is the steam wrapper variable is set or not:

if [ -n "$launch" ]; then
    echo Launching launch...>> ~/R2Valheim.log
    exec "$launch" $rest
else
    echo Launching exec...>> ~/R2Valheim.log
    exec "$exec"
fi

To trick it, I simple negated the check with a ! like:
if [ ! -n "$launch" ]; then
which let the script use the straight Valheim binary.
Here is my latest home log with the variables.

Now with the log output. The command line and rest of the log as it's more or less the same output:

[2024-11-15 08:15:59] /bin/sh\0-c\0STEAM_LINUX_RUNTIME_VERBOSE=1 "/home/jay/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" /home/jay/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- /home/jay/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=892970 -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point --verb=waitforexitandrun -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime'/scout-on-soldier-entry-point-v2 --  '/home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim.x86_64' '--doorstop-enable' 'true' '--doorstop-target' '/home/jay/.config/r2modmanPlus-local/Valheim/profiles/Takers_Underworld/BepInEx/core/BepInEx.Preloader.dll' '--r2profile' 'Takers_Underworld' -console\0
[2024-11-15 08:15:59] Game process added : AppID 892970 "STEAM_LINUX_RUNTIME_VERBOSE=1 "/home/jay/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" /home/jay/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- /home/jay/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=892970 -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point --verb=waitforexitandrun -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime'/scout-on-soldier-entry-point-v2 --  '/home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim.x86_64' '--doorstop-enable' 'true' '--doorstop-target' '/home/jay/.config/r2modmanPlus-local/Valheim/profiles/Takers_Underworld/BepInEx/core/BepInEx.Preloader.dll' '--r2profile' 'Takers_Underworld' -console", ProcID 814551, IP 0.0.0.0:0
[2024-11-15 08:15:59] Controller slots reset
[2024-11-15 08:15:59] GameAction [AppID 892970, ActionID 76] : LaunchApp changed task to WaitingGameWindow with ""
[2024-11-15 08:15:59] GameAction [AppID 892970, ActionID 76] : LaunchApp changed task to Completed with ""
[2024-11-15 08:16:01] GameOverlay: started '/home/jay/.local/share/Steam/ubuntu12_32/gameoverlayui' (pid 814680) for game process 814551
[2024-11-15 08:16:02] Game process removed: AppID 892970 "STEAM_LINUX_RUNTIME_VERBOSE=1 "/home/jay/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" /home/jay/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- /home/jay/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=892970 -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point --verb=waitforexitandrun -- '/home/jay/Games/SteamLibrary/steamapps/common/SteamLinuxRuntime'/scout-on-soldier-entry-point-v2 --  '/home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim.x86_64' '--doorstop-enable' 'true' '--doorstop-target' '/home/jay/.config/r2modmanPlus-local/Valheim/profiles/Takers_Underworld/BepInEx/core/BepInEx.Preloader.dll' '--r2profile' 'Takers_Underworld' -console", ProcID 814551
[2024-11-15 08:16:02] ThreadGetProcessExitCode: no such process 814559
[2024-11-15 08:16:02] ThreadGetProcessExitCode: no such process 814554
[2024-11-15 08:16:02] ThreadGetProcessExitCode: no such process 814552

What's new here is that the Player.log file from Valheim was populated, and there is a dump file named 'mono_crash.mem.814551.1.blob'

I think that your workaround is actually having a side-effect of making the game not run in the container environment, because this stack trace does not look like the container:

This migh as well be right.. The game itself in this condition could be.. unreliable.. and as such does crash. Thus the main issue would be 'steam-launch-wrapper' crashing without a trace I can find...

Any other specific log files are created/stuffed with more content when 'STEAM_LINUX_RUNTIME_VERBOSE=1' is set ?
I will investigate as much as you are willing to question me :)

As a former QA analyst and actual functional analyse, I do such for work when bugs arise, but unrelated to games hehe

@smcv
Copy link
Contributor

smcv commented Nov 15, 2024

So the issue here is that you're putting a custom wrapper around the %command%, but the %command% is the entire composite command-line including the reaper, the steam-launch-wrapper, and the SLR compatibility tool (which is actually two compatibility tools stacked one on top of the other). https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/steam-compat-tool-interface.md?ref_type=heads has some more technical information about how compatibility tools and Launch Options fit together.

Your top-level wrapper script receives the rest of the command line in the special shell array "$@", and from then on, what happens is entirely under the control of your wrapper script - which gives you lots of options, but also equally many ways you can make it go wrong.

From your start_game_bepinex.sh:

if [ "$2" = "SteamLaunch" ]; then
    cmd="$1 $2 $3 $4 $0"
   shift 4
    exec $cmd $@

This is making a big assumption about the structure of the command-line that Steam uses to launch the actual game: it's assuming that if the second argument is SteamLaunch, then the first 4 arguments (exactly 4, no more, no fewer) are a wrapper or compatibility tool or similar. And then it's using that assumption to permute the arguments on the command-line to re-exec itself inside that wrapper.

Without having done the analysis, I'm 99% sure that this assumption is no longer true: the Steam Linux Runtime 1.0 compatibility tool takes up more items in the command-line than the older legacy runtime. Sorry, if you're going to modify the command-line in-place like this, then it's up to you to figure out how to do it without making assumptions that will go wrong over time.

Looking for the main executable by identifying an argument that starts with / and ends with /valheim.x86_64, and using that as your reference for where to inject custom mod code, might work more reliably?

I see that this script is writing various things to R2Valheim.log, including some of the commands that it runs. You would probably find that R2Valheim.log contains information that will help you.

This is also going to do the wrong thing if any of the items on the command-line contain spaces. Shell script is an awful programming language, mostly consisting of sharp edges, and you might find that writing logic in Python using the subprocess standard-library module works better. A copy of python3 is available inside the Steam Linux Runtime container - it's version 3.7, with no extra modules beyond the standard library, but that should be enough for simple scripting.

Similarly, linux_wrapper.sh is not robust against items in $args that contain spaces, but it does log some potentially useful things to R2Valheim.log.

I can't provide a complete tutorial on writing robust shell scripts (that would take weeks, if not years) but the tl;dr version is "always use shellcheck".

Any other specific log files are created/stuffed with more content when 'STEAM_LINUX_RUNTIME_VERBOSE=1' is set ?

There's no specific log file as such, the extra information goes to wherever the stderr for the invoked process has been sent. That's under the control of your wrapper scripts.

There is another variable you can set, STEAM_LINUX_RUNTIME_LOG=1, which redirects output from everything in the process hierarchy, from steamapps/common/SteamLinuxRuntime_soldier/_v2-entry-point downwards, to arrive in steamapps/common/SteamLinuxRuntime_soldier/var/slr-latest.log (until/unless some other process redirects output again, in which case it's that other process that gets control over what happens).

But if you're bypassing SLR completely (as I suspect you are when you exec "$exec"), that will never happen: we'll never enter steamapps/common/SteamLinuxRuntime_soldier/_v2-entry-point and the output will never be redirected.

Similarly, if something "outside" SLR is crashing, then we'll never get far enough for that redirection to happen: the crash could well be happening before SLR gets involved, in which case SLR never gets the opportunity to redirect output.

@AngelQC
Copy link
Author

AngelQC commented Nov 15, 2024

if [ "$2" = "SteamLaunch" ]; then
    cmd="$1 $2 $3 $4 $0"
    shift 4
    exec $cmd $@
    exit

Hmmm the behavior is:

If the 2nd argument is SteamLaunch, then insert the script file name as the 4th argument and re-launch the line, including all the former arguments.
Shift 4 deletes the first 4 arguments, which are now in the new cmd variable, so $0 now contains $5 $6 .. $#

But as it is, this is not executed on my machine, SteamLaunch being the 4th argument. This could be a previous version compatibility which could be removed/reworked, but still, we must not forget that the previous version of valheim do work.

I see that this script is writing various things to R2Valheim.log, including some of the commands that it runs. You would probably find that R2Valheim.log contains information that will help you.

Sorry, this was not obvious in my previous posts, but this is in fact my log file, I did add those echo redirection lines. R2Valheim.log with the variables was in the last post, but quoted as 'home [log]'

I insist, those are not my scripts, they are provided by r2modman 😄
I know a bit about scripting and I would have done it differently I'm sure, abs has been my bible for a long time.
So far it seems to work either ways, as I log the full command lines with echo "* $*">> ~/R2Valheim.log and nothing has spaces beside "name" "value" pairs, which are balanced.

There's no specific log file as such, the extra information goes to wherever the stderr for the invoked process has been sent. That's under the control of your wrapper scripts.

Oh.. I didn't think of that, will add a redirect to check if there are lost outputs.
[edit]
The only added line when launching the wrapper is:
Found UnityPlayer, hooking into it instead
[/edit]

There is another variable you can set, STEAM_LINUX_RUNTIME_LOG=1, which redirects output from everything in the process hierarchy, from steamapps/common/SteamLinuxRuntime_soldier/_v2-entry-point downwards, to arrive in steamapps/common/SteamLinuxRuntime_soldier/var/slr-latest.log (until/unless some other process redirects output again, in which case it's that other process that gets control over what happens).

Ah! I will also enable and check these.

[Edit]
With mods, the execution does not make it to the Soldier, a log file is not created, but it is when launching from Steam.
Although, the game stops now from steam.
Removing any commands, leaving -console which is a Valheim argument, the game crashes.. I'll continue later on..
[/edit]

But if you're bypassing SLR completely (as I suspect you are when you exec "$exec"), that will never happen:

True, and as we kinda agreed in a previous post, this is not the way as it's launched out of the container. I did that earlier in hope for this to go farther, but it seems to only mislead the search as it's evaluating the host's libs and bins instead of the container's, resulting in the mono-related crash so I won't do that anymore -- the goal is to fix, and also hopefully understand, steam-launch-wrapper startup and crashing.

@smcv
Copy link
Contributor

smcv commented Nov 15, 2024

I see you're passing -compat-force-slr off as arguments to the Valheim executable. Unless there has been a quite spectacular coincidence, I think that might be based on a misunderstanding.

-compat-force-slr off are arguments for the main Steam client executable. If you want to go back to the legacy runtime that was used before November, you should run Steam (e.g. from a terminal) as

steam -compat-force-slr off

but you should not add -compat-force-slr off to the Launch Options of any individual game.

@AngelQC
Copy link
Author

AngelQC commented Nov 15, 2024

Yes, quite the misunderstanding from
Issue 706 😄
I removed it at some point but it's not there anymore as far as I can remember. I did not understand it was a Steam parameter, I thought the wrapper would enable/disable this. I put the logging at the same place and it does work though 🤔 .

I'll dig some more through the weekend, as whatever I did today didn't give me any new information.
I updated and rebooted my machine when I stopped earlier, when the game started crashing all the time, even starting with no arguments what so ever which removes any scripts from the equation. The mod manager does not put any new files in the game folder itself, so that's now weird..

@AngelQC
Copy link
Author

AngelQC commented Nov 16, 2024

Well.. The most craziest thing is happening..

The [default_old] version of the game actually works out of the container / by bypassing the wrapper. Player.log

I was trying to set the game as the server (long story short, a lot of mods even though they have been updated by their modders to the latest version of the game, have issues so we reverted to the last backup and old version the mods while they are being fixed) and I could not get it to work with the wrapper..
So in my previous explanations, I think I had forgotten that I was playing while the script was launching the game instead of the full steam-launch-wrapper command line...

So it's been a while since the wrapper stopped working, and it just happened that the newer version of the game has issue with my Linux distro's current libraries.. 🤣

I will remove and reinstall Steam completely because.. that's not normal and you would have a lot of opened issues from linux users if it was a real Steam issue.. I think.. I can't believe only Valheim does not work with the wrapper, but it'S my only native game .. so I can't really test more widely.

Will keep this posted.

@stele95
Copy link

stele95 commented Nov 18, 2024

I'm having the same issue, unable to launch the modded game with Soldier. Starting Steam with -compat-force-slr off works for me and the modded game starts and runs without problems.
My launch options are:
"/home/stele/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" mangohud gamemoderun %command% -console --doorstop-enable true --doorstop-target "/home/stele/.config/r2modmanPlus-local/Valheim/profiles/Modovi/BepInEx/core/BepInEx.Preloader.dll" --r2profile "Modovi"

After digging through logs and manually logging things inside Soldier's scripts, I found out that this is what Soldier receives:
--ld-preloads=libgamemodeauto.so.0:libdoorstop_x64.so::/home/stele/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so:/home/stele/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so:/usr/mangohud/libMangoHud_opengl.so --env-if-host=LD_PRELOAD=libgamemodeauto.so.0:libdoorstop_x64.so::/home/stele/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so:/home/stele/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so:/usr/mangohud/libMangoHud_opengl.so -- Found UnityPlayer, hooking into it instead /mnt/games_partition/games/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/steam-runtime-launcher-interface-0 container-runtime /mnt/games_partition/games/SteamLibrary/steamapps/common/SteamLinuxRuntime/scout-on-soldier-entry-point-v2 -- /mnt/games_partition/games/SteamLibrary/steamapps/common/Valheim/valheim.x86_64 -console

Notice the Found UnityPlayer, hooking into it instead string inside. This is what Bepinex (I think) is logging and that somehow ends up in the arguments for Soldier, which completely messes up everything and that results in the following error in the logs:
/mnt/games_partition/games/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier/_v2-entry-point: line 285: /mnt/games_partition/games/SteamLibrary/steamapps/common/Valheim/Found UnityPlayer, hooking into it instead [2024-11-18 12:36:43] /mnt/games_partition/games/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier/run: No such file or directory

This is consistently happening for me with every single attempt to run the game with mods. I start the game directly from Steam instead of through r2modman, that's why I have all those arguments in the launch options after -console

I'm using Steam Beta and Soldier beta, and starting game without mods works as expected.

@smcv
Copy link
Contributor

smcv commented Nov 18, 2024

Notice the Found UnityPlayer, hooking into it instead string inside. This is what Bepinex (I think) is logging and that somehow ends up in the arguments for Soldier, which completely messes up everything

If some component of this mod manager emits arbitrary strings on standard output (stdout), then that does seem likely to break things. Some of the setup that is done internally by the container runtime framework almost certainly relies on having a "clean" standard output stream.

Components that emit arbitrary diagnostic information should use standard error (stderr), for example echo Message >&2 or fprintf(stderr, "Message\n").

LD_PRELOAD=...:libdoorstop_x64.so:...

If some component of this mod manager is injecting this libdoorstop_x64.so into executables not under its control (such as the steam-launch-wrapper and the Steam Linux Runtime framework), then the code in libdoorstop_x64.so needs to be aware that it is running in an extremely precarious environment, and it needs to be very careful what it does.

@stele95
Copy link

stele95 commented Nov 18, 2024

I think I found the problem, I believe it's this line of code that causes the issue. This is where it's called with the string from above.

I'll try and build a version with changes to logging and see if that helps. Will report back with results.

@smcv
Copy link
Contributor

smcv commented Nov 18, 2024

I think I found the problem, I believe it's this line of code that causes the issue. This is where it's called with the string from above.

I'll try and build a version with changes to logging and see if that helps. Will report back with results.

Yes, logging via printf() is not appropriate for a LD_PRELOAD module. With great power comes great responsibility! As a minimum it should use fprintf(stderr, ...) instead of printf(...).

A side benefit of redirecting that logging to stderr would be that stderr is line-buffered rather than fully buffered by default, so the diagnostic message is less likely to be buffered in memory (and therefore lost if the application crashes), and more likely to get written out to the terminal, Journal and/or console-linux.txt in a timely way.

@stele95
Copy link

stele95 commented Nov 18, 2024

I found out that BepInEx used for Valheim uses an old version of Doorstop libs for Unix found here. After I removed this line of code, built the lib myself and replaced the one used by BepInEx, the game was launching and loading mods without a problem.

@AngelQC
Copy link
Author

AngelQC commented Nov 20, 2024

@stele95 there is a pull request dating Nov28 2021 about this file!

Thanks for your input, it seems to show that, as I was starting to think, the problem has nothing to do with Steam itself.. But BepInEx is mandatory for mods geez.

Are you going to open an issue on the UnityDoorstop git ? Leave a link here, I will hop in. You are better placed than me to do so I think. You found the problem in the code.

I'm quite astonished, however, that I could run the game for so long out of the container, with mods, and it would just work! It took the "Bog witch" update to break it.

I did not re-try starting Steam with the command line option until now but it still does not fully work.
Using versions prior to Bog Witch (the latest version), it does, otherwise it crashes.

So until the server I play on updates to the latest version, I might be able to play..

@stele95
Copy link

stele95 commented Nov 21, 2024

@AngelQC That PR is basically what needs to be done so BepInEx works in the new Steam runtime.
I'll create an issue on the new version of Doorstop since newer versions of BepInEx use the new version.

If I understand correctly, it wasn't the Bog witch update that broke the game, it's Steam forcing their runtime that exposed a problem that was always there but wasn't detected since games were starting in a different way before.

As for the fix, in case you want to do it yourself now, in case BepInEx for Valheim doesn't get updated to a new version anytime soon, here are the steps:

  • git clone https://github.com/NeighTools/UnityDoorstop.Unix
  • Open doorstop.c file and remove the printf line here
  • Build the lib by running make build_x64
  • Copy the libdoorstop_x64.so to ~/.config/r2modmanPlus-local/Valheim/profiles/{nameOfYourProfile}/doorstop_libs/

This worked for me and I'm launching Valheim now without any problems and mods that I use work as expected

@smcv
Copy link
Contributor

smcv commented Nov 21, 2024

If I understand correctly, it wasn't the Bog witch update that broke the game, it's Steam forcing their runtime that exposed a problem that was always there but wasn't detected since games were starting in a different way before.

Yes, I think so. The container runtime involves more processes during its startup, and is more reliant on LD_PRELOAD modules not corrupting stdout with diagnostic messages.

@AngelQC
Copy link
Author

AngelQC commented Nov 22, 2024

@stele95 thanks for this, I am reviewing the code and building as I write. Worked with a previous version, not the latest.

State

With stele95's lib build solution, I could start the game, modded, but not the latest version. The game, however, now starts within the Soldier container and the game's Player.log file shows path from Steam:

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7c7b4a11565a - /home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so : 
	0x7c7b4a0be049 - /home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so : 
	0x7c7b4a0434d0 - /home/jay/Games/SteamLibrary/steamapps/common/Valheim/valheim_Data/MonoBleedingEdge/x86_64/libmonobdwgc-2.0.so : 
	0x7c7b8c24c1d0 - /usr/lib/libc.so.6 : 
	0x7c7aa497009e - /usr/lib/libvulkan_radeon.so : 
	0x7c7aa497422f - /usr/lib/libvulkan_radeon.so : 
	0x7c7aa497222f - /usr/lib/libvulkan_radeon.so : 
	0x7c7b8d3369f7 - /home/jay/Games/SteamLibrary/steamapps/common/Valheim/UnityPlayer.so : 

And despite the logging command line arguments, I somehow do not have any new Soldier logs since the 15th... what ever means I use to launch the game.

Setup: Game v0.219.14 - Small patch after Bog Witch update

  • Steam launched with -compat-force-slr off argument
  • Steam Linux Runtime 2.0 (Soldier) set at client_beta
  • Game set as normal/no beta
  • Launch options:
STEAM_LINUX_RUNTIME_LOG=1 STEAM_LINUX_RUNTIME_KEEP_LOGS=1 STEAM_LINUX_RUNTIME_VERBOSE=1 "/home/jay/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" %command% -console

Results:

  • Vanilla, through Steam: Launches
  • Vanilla, through r2modman*: Launches
  • Modded, through r2modman*: Crashes (R2Valheim.log, Player.log)

Setup: Game v0.218.21 - Last stable before Bog Witch update

  • Steam launched with -compat-force-slr off argument
  • Steam Linux Runtime 2.0 (Soldier) set at client_beta
  • Game set at default_prebw
  • Launch options: (Same as previous)
STEAM_LINUX_RUNTIME_LOG=1 STEAM_LINUX_RUNTIME_KEEP_LOGS=1 STEAM_LINUX_RUNTIME_VERBOSE=1 "/home/jay/.config/r2modmanPlus-local/Valheim/linux_wrapper.sh" %command% -console

Results:

  • Vanilla, through Steam: Launches
  • Vanilla, through r2modman*: Launches
  • Modded, through r2modman*: Launches

* Note: r2modman does launch through Steam as well, adding options to the %command% variable as a result.

The last error line, which has a hint of relevancy, is
ERROR: ld.so: object '/home/jay/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.

@smcv
Copy link
Contributor

smcv commented Nov 22, 2024

Steam launched with -compat-force-slr off argument

If you run Steam as steam -compat-force-slr off, and you have not specifically configured this specific game with Properties → Compatibility → Force the use of → Steam Linux Runtime 1.0 (scout), then your game is not running in the SLR container at all. Instead, it will be using the legacy LD_LIBRARY_PATH-based runtime - the same as the default behaviour before November 2024.

You can check this by running the game in a configuration that doesn't crash (for example, without enabling any mods), and looking at the process list with pstree -alp or systemd-cgls or similar (perhaps in another terminal window or in a ssh session).

If the game is running in the SLR container, you'll see a srt-bwrap (or possibly bwrap) process with references to the SteamLinuxRuntime_soldier directory in its arguments. If not, you will not see those.

despite the logging command line arguments, I somehow do not have any new Soldier logs

If the game is not running in a SLR container, then you will not get any new SLR logs. SLR logs are only recorded when you run something via SLR.

If you are not running the game via a SLR container, then any problems you are still having with the game presumably can't be SLR's fault.

after Bog Witch update: Vanilla, through r2modman*: Launches; Modded, through r2modman*: Crashes
before Bog Witch update: Vanilla, through r2modman*: Launches; Modded, through r2modman*: Launches

To me, this sounds like one of the mods you are running is not compatible with the new game update. This is out-of-scope for the Steam Runtime: we do not control mods' code or whether they crash the game.

@AngelQC
Copy link
Author

AngelQC commented Nov 22, 2024

If you run Steam as steam -compat-force-slr off, and you have not specifically configured this specific game with Properties → Compatibility → Force the use of → Steam Linux Runtime 1.0 (scout), then your game is not running in the SLR container at all. Instead, it will be using the legacy LD_LIBRARY_PATH-based runtime - the same as the default behaviour before November 2024.

Wow.. I just realized now what the -force-slr means, and what turning it off involves.. When I first did that, if you remember, I was applying this to the game start, which had no effect thus not having logs... /facepalm

And suddenly, I have logs now.. Geez..

Ok then lets look at that with a proper Soldier container.. lol
R2Valheim-241122-2.log
Player-241122-2.log
slr-latest-241122.log

To me, this sounds like one of the mods you are running is not compatible with the new game update. This is out-of-scope for the Steam Runtime: we do not control mods' code or whether they crash the game.

All the previous execution were out of the container, so anything that went wrong then is out of scope I agree. Now I got Soldier logs, so it should be more obvious to the keen eye. Not my eye as I cannot be certain what errors in the log is causing the issue. Some files not loaded are sometime okay/we'll not use the feature, others are more important..

  • There are a bunch of inexistant Vulkan-related files and folders
  • The same with dri module

Then pressure-vessel tries to regenerate ld.so.conf and can't process libgpg-error.so.0 from a couple of locations and that's pretty much the end of it..

In the game log, no error regarding the two mods I left, the BepInEx which I rebuild a library for, and CursorSwapper, both for the current game's version with no pending update.

@AngelQC
Copy link
Author

AngelQC commented Dec 1, 2024

Someone else opened a ticket with BepInEx, the main mods library. Not sure so far where the trouble lies, so I'm trying to monitor both sides..

@Omnifarious
Copy link

Omnifarious commented Dec 13, 2024

The problematic script snippet pointed to here: #708 (comment)

can be replaced with this:

relaunch=0
found_exe=0
echo "Searching for SteamLaunch: $*" 1>&2
for argument in "$@"; do
    if [ $relaunch -eq 0 ]; then
        if [[ "$argument" == "SteamLaunch" ]]; then
           relaunch=1
           echo "Found SteamLaunch" 1>&2
        fi
    elif [ $found_exe -eq 0 ]; then
        case "$argument" in
          *"valheim.x86_64"*)
            newcmd+=("$0")
            found_exe=1
            ;;
        esac
    fi
    newcmd+=("$argument")
done
if [ $found_exe -ne 0 ]; then
    echo "Re-execing as: ${newcmd[*]}" 1>&2
    exec "${newcmd[@]}"
    exit 1
elif [ $relaunch -eq 1 ]; then
    echo "A relaunch is indiciated, but valheim.x86_64 not found." 1>&2
    exit 1
else
    unset newcmd relaunch found_exe
fi

This still depends on the actual game executable being named valheim.x86_64, but is otherwise pretty flexible about the argument layout. Other parts of the script have the same dependency. Oh, also, to be safe, you should change the interpreter at the top of the script from #!/bin/sh to #!/bin/bash because I think that advanced array handling stuff won't work in POSIX compatibility mode .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants