Not an issue, just a suggestion for the next step #1
Comments
|
Silent doesn't really care much for the games... |
|
@mirh I see your point. Those crashes are almost 100% reproducible on all modern hardware though, especially the one under liara's office which is also mentioned on PCGW. |
|
I'm sure of it, but I don't really think that's a bug to be fixed in the executable. |
|
Can you provide saves that repro this quickly? |
|
Here's a save on Illium under Liara's office: Cleaner_13_Vanguard_271216.zip Here's another save at the end of Kasumi's loyalty mission: To use the saves, extract the 2 folders into My Documents\BioWare\Mass Effect 2\Save |
If that's the case, then it should be pretty easy to spot files not found with procmon. |
|
I'm not using mods at all, just the game, fully patched and with all the DLC installed. You can replicate the problem on the origin version. |
|
From what I can tell this only happens to users in full screen mode. The engine will look in multiple locations to find files, it not finding them at the first location is normal. |
|
@adolfintel Do you have a workaround for the "Unable to authorize the listed DLC"? I used an option in Mass Effect 1 to bypass this, does one exist for Mass Effect 2? |
I have it but I don't know if it's a good idea to post it here since it's technically a crack. Do you need it? |
|
@adolfintel No worries, got the DLC unlocker! |
|
@adolfintel Can't reproduce the crashes sadly. :( |
|
That's good to know. What resolution are you playing at? Did you try fullscreen and borderless modes? |
|
Still can't reproduce this, any other ideas? Tried windowed, fullscreen windowed, fullscreen, various resolutions. I'm using the latest Steam version of the executable, with SHA-256 of MassEffect2.exe: Example of one tested configuration |
|
Some questions:
|
Nope, just mouse and keyboard
No mods, I'm using a cracked exe with the binkw32.dll that unlock all the DLC. I was also able to replicate the issue with the origin version so the crack is not to blame.
Nothing special, Realtek HD Audio. The device is set to its default 44100Hz 16bit stereo configuration. NVIDIA HDMI Audio is also installed but not in use.
https://github.com/Erik-JS/masseffect-binkw32 I know the kasumi crash is kinda rare but the crash on Illium can be replicated on all of my machines. |
|
@riverar By the way, I have some experience with tools like IDA pro. I wasn't able to find out anything useful by myself but you clearly know more about it than I do, so if you give me a few pointers I might be able to investigate it further |
|
I have no insight to share on this issue, ha! At this point, I'm just trying to reproduce the issue so we have somewhere to start. Can you perform the following steps and reproduce the crash? We'll at least have a stack to look at.
|
|
Ok, I'll get back to you in a day or two with all the info I can find. By the way, can you make a pastebin with the hashes of all the files in your ME2 installation? Especially those regarding the Kasumi DLC |
|
@adolfintel Per request report.zip |
|
@riverar Thank you. Did you get my email with the dumps? |
|
@adolfintel oops just dug it out of my junk folder, let's take a peek! |
|
@adolfintel Thanks, sent you a few emails back. Just a heads up, in case those end in the junk mail folder. |
|
I got them, running the tests now. I can tell you right away that RTSS was not the problem and OBS was not running |
|
@riverar Check your email :) |
|
@riverar I sent you another email with a full dump of a kasumi crash, hope that helps |
|
So, I'm 90% sure that the kasumi crash is caused by some Windows update that came out between 2014 and 2016. I have some old hardware and software here so I decided to try various combinations:
Personally, I recall playing the game without issues in 2014, but having the crash in 2016, and this matches what I'm seeing here. Thoughts? |
|
@riverar I'm sending you an email with a modified exe and more info, but here's what I found out: struct stuff=myFunction();
if(stuff.a==NULL) return; else{code...}
if(stuff.b==NULL) return; else{more code...}
...The thing is: it never checks if stuff is NULL in the first place. So, I modified it so that after getting the value of stuff, it would jump to a small piece of code that I put in one of the empty areas in the exe where it checks if it's NULL, and if it is it returns from the function, otherwise it runs the original code The crash seems to have been fixed, and while I haven't played through the entire game yet, it doesn't seem to cause problems so far. Now obviously I can't post a cracked exe on github, even if it's just a fix, so with your help, I'd like to make something like SilentPatch that injects the modified code. Can you do that? |
|
There are really countless of reports before 2014, so I'm afraid it's not Windows (also, I don't think 2011 drivers should have problems with a 2010 game?). Also.. Maybe the issue isn't that the NULL pointer has no checks, but rather that sometimes it ends up being NULL? But I'm sure @Mgamerz will have some ideas (hex editing seems a no-go on origin stupidly obfuscated exe, but maybe the binkw32 loader could come to the rescue) |
For me it crashed in both fullscreen and borderless windowed mode. Windowed mode just reduced the probability of crashing in a particular place, for example on Illium. My impression is most crashes in this game is related to race conditions between threads. That's also why it crashes more on a modern hardware. Some threads run faster than they used to, which exposes bugs that were always there, but were hidden during the game testing because they almost never caused crashes in practice before. A good example of it is the crash after the Kasumi's loyalty mission that @adolfintel fixed. Before I found his fix, I tried to finish the mission several times and it crashed on every single attempt. At first I thought it might be due to a missing file and tried to check it with Process Monitor. And guess what, the game didn't crash with Process Monitor running in the background! Of course Process Monitor didn't magically fix the game, all it did was slowing down the right thread that was running too fast before, so this time another thread had enough time to populate the data that the first thread required, and the pointer mentioned in https://fdossena.com/?p=kasumifix/i.md wasn't null anymore. At least that's the only logical explanation I can think of. |
|
That's sensible.. And FWIW I just finished some kind of theory for my aforementioned discovery after like half a day of tests and loosing my faith in god (at one point I even came to think autologin could control the unexplained behaviour I still can not get at the bottom of, but then it suddenly didn't seem to make a difference anymore) |
|
@mirh I remember trying to set single thread affinity when I first encountered these issues and it didn't do anything other than make the game more stuttery. That wine report is probably just a wined3d bug from back then, I had no issues with it on wine-ge with dxvk (tested a few weeks ago). I think our best bet if we want to fix the Illium crash is some variant of what @JohnA2 suggested with the modified video file. Ideally, we would want some way to either modify the Illium map so that it loads an empty video file or replace the original file with an empty one and modify the Liara apartment map so that it points to a copy of the original video with a different name. |
|
Not really, even though I'm sure everything and anything can be done with them by now. |
The video plays in the Liara's office, not apartment. As I mentioned above, replacing the video with either an empty one or a different one from the game with ME Explorer works but doesn't fix the crash, so I had to disable it with hex editing instead. I think not having that 8 seconds video is a small price to pay to avoid the crash that happens so often. The video only plays once when you start the Shadow Broker mission, so people who want to see it can temporary replace
It may reduce the chances of a crash if you're very lucky, but it can't fix it completely. The only good fix would be implementing the proper thread synchronization by changing the game code. But it can be difficult even when having the source code. Fixing thread synchronization with reverse engineering seems next to impossible to me, but I'm not an expert.
They mention the Illium car chase issue. I remember reading that some people also had a crash during that scene, but it never crashed there in my case. I only noticed a slight corruption of the video for a second there. Maybe that can cause an issue. If that's true, re-encoding that video might fix it. But I can't try it as it never crashes for me.
I think not many people play in windowed mode, so it was likely tested the least. I finished the game in fullscreen mode. After applying the @adolfintel's fix and my fix, I didn't have any major issues anymore. It still crashed a few times at random places, but I was always able to continue without the same crash happening at the same place again. The only other persistent crash I encountered was at the beginning of Prometheus Station (Overlord DLC). The fix in that case was not bringing Legion as a squad member, see https://fextralife.com/forums/t216365/overlord-crash-at-prometheus-station . Overall it seems that most issues are with DLCs that were released after the last patch. Probably Bioware stopped working on fixing bugs at that point. |
|
Yeah, I know about that car chase crash (the whole LTSB mission has several spots where it occasionally crashes: this one, the scene with Vasir in Liara's apartment, the scene with Liara after killing Vasir, on the shadow broker's ship near those stabilizer things that move up and down and spark, when the door opens after the fight while waiting for Liara's hacking thing to finish, and probably more that I'm forgetting. @JohnA2 did you try reencoding the video inside BioD_TwrHub_504LiaraDLC_LOC_int.pcc? |
|
@adolfintel No, I didn't try reencoding that video. I believe that if replacing it with other videos didn't fix the crash, the problem is in how the game processes the video (during texture caching I guess) in that particular spot, not the video itself. I checked other .pcc files for similar cases and found that a video embedded in a texture like that is a pretty rare case in the game, that's probably why that bug doesn't cause crashes in other places. The car chase crash sounds like a different issue, that's why I thought reencoding might help there. |
|
Quick update on the Illium freeze. I spent a few hours investigating the issue. I'm not 100% confident about the stuff that I'm about to write, my reverse engineering skills are very limited, but here goes. I think the freeze is caused by the rendering thread (conveniently named RenderingThread, thanks game): at address 0x51147f it tries to do a mov eax,[ecx], where [ecx] seems to be a function pointer loaded from memory. When the freeze occurs, the pointer contains garbage data and the rendering thread crashes. Edit: placing a breakpoint at 0x511470 (before the pointer is read from memory) seems to fix the freeze by introducing some delay, I just need some way to automatically resume execution after a few ms instead of having to press the resume button in IDA. |
|
Oh my god, I think I fixed it 😱 I modified the exe to add a short sleep at the beginning of that function, that seems to be enough, instead of crashing it does a small stutter. I haven't tested it extensively yet, but I went back and forth over 20 times without crashing. I'll keep you posted 🤞 |
|
@adolfintel That sounds promising. A sleep is still not a proper way to do thread synchronization, but it's probably the best chance to fix it when the source code is not available. If you can tell the offset in the .exe to modify, I can test it as well. Also maybe worth noting that this Illium issue was a bit different for me on Windows 11 and 7. On 11 it's a freeze with no error, but on 7 it's a crash with the usual Windows error message. So maybe if you still have access to win7, it would be easier to debug there. When it crashed, the exception offset was either 00111481 or 0011147f. |
|
@JohnA2 It's definitely not ideal (and I don't like seeing a hiccup every time I go through that tunnel), but it's better than nothing. As I said, it's not extensively tested yet (just got back from work) and it may not work at all, but here's what I changed in my exe: For reference, my exe has a SHA1 of I'll test this a bit and then make an ASI to share it if it works. |
|
@adolfintel My .exe is a bit different. The SHA1 doesn't match and also the "56 8B 71 04" sequence starts at 00111470 instead of 00111471. Not sure if it's just a mistake you made because the shift is only 1 byte. Anyway, I think the main question is how many times the game hits the code where you added sleep. If it's rare, occasional hiccup is fine. If not, personally, I would still rather sacrifice the 8 seconds video instead. It's not an important video for the story, it's just a little weird when Liara looks at an empty datapad screen for a few seconds and says "It looks like a leaked transmission between Shadow Broker operatives" before the picture of Feron finally appears 😄 |
|
@JohnA2 I'm attaching the ASI so you can test it (source included of course), hopefully it works with different exes too, the one I have is a cracked one from reloaded I think. |
|
@adolfintel Thanks! I've tried it and it says the patch was successful. I also noticed the hiccup that you mentioned, but unfortunately on the fifth attempt to pass the tunnel it still crashed for me. The exception offset was 00111481 as before. But the frequency of this crash definitely reduced with your patch.
These are exactly the places where I found that other "TextureMovie", as ME Explorer calls it, are used, so modifying that code shouldn't affect much of the game, which is really great if a more reliable fix can somehow be made. |
|
Hmm, that sucks. I didn't get a single crash in about 50 tries, I even went into Liara's office and it played the video just fine, I guess the different thread scheduling in linux is involved. 1 chance of crashing out of 5 is a good start though, it used to be ~100% for me on windows before this. Offset 111481 corresponds to 0x511481 in memory, which was one of the crash spots without the sleep. If the sleep is not enough to reliably fix this maybe we can figure out a better solution. You seem to know ME Explorer better than me, do you think there's some kind of "loading volume" near liara's office? If so, would it be possible to remove it or move it out of the way? |
That part also never crashed for me. Only going through the tunnel under the Liara's office or (more rarely) near the tunnel entrance did.
Yes, this is definitely a factor when a race condition is involved. In my case the crash frequency with the unpatched game was also different depending on the OS and hardware, even though I only tried it on Windows.
With my fix it never crashed on my 2 systems anymore. Not sure if it's just luck with the race condition or if the code that causes the crash is simply not executed after I "disable" the video.
I'm also not that good with ME Explorer. I just tried it and was able to figure out how to do things because its UI was intuitive enough for me. I tried to adjust all variables that were associated with the texture/video, but it made no difference. Honestly, I'm satisfied with my "disabling video" fix enough to stop looking for other options. Fixing thread synchronization would've been a perfect fix because it might fix other random crashes as well, but doing it in assembly is above my skills. Another alternative I thought of was replacing the video on the datapad with a series of quickly changing images because images don't seem to cause crashes. But that would require some ME modding skills. |
|
The reason I don't like the "disabling video" fix is not just because you lose that 8 second video, it's because the problem is still there and waiting to be triggered again by unknown conditions. I don't like the sleep workaround much better. |
|
Only difference I see is that there are several copies of this stupid video in the localization files, while the other TextureMovies only appear once. Replacing the video file with another TextureMovie seems to always cause a crash when approaching the door |
I completely agree here, but I think it would take a reverse engineering expert to come up with a proper fix in this case.
I think there's no difference in the embedded bik files. One of the replacement videos that I tried was from another TextureMovie from Freedom's Progress and that video still caused a crash on Illium, even though it doesn't seem to crash on Freedom's Progress. It feels like that Illium location is a somehow unique combination of TextureMovie and other things. Like I mentioned, disabling dynamic shadows was another thing that seemingly 100% fixed the crash for me (I just didn't like that solution). I also noticed that while the crash inside the tunnel can happen at any time, a more rare crash just outside the tunnel only happened closer to sunrise or sunset and not during the day. These 2 observations with shadows and the sun position might be just a coincidence and not mean much, but if they do make a difference, it would explain why it doesn't crash on Freedom's Progress and the Shadow Broker base with TextureMovies present there as well. |
|
Indeed, given that Btw FWIW I see the BioMoviePlayer class in SFXGame has a lot of null checks. Maybe you could try playing with them? p.s. wine tries to stick with windows threading model as much as it can IIRC, what happens here could even be (somewhat paradoxically) that they don't have the same compatibility regards that windows would have. |
|
Another quick update on the Illium freeze. I added a little "safety net" before the function pointer read from memory is used at 0x511481, a simple loop that freezes the rendering thread until that pointer in memory points inside the code segment. This brought down the crashes to 0% (on linux) when running in borderless mode, but fullscreen still freezes indefinitely when the usual freeze occurs (pointer gets corrupted maybe?). I'll keep you posted. |
|
Alright, time for another update on the Illium freeze. Here's what I got at the moment:
|
|
@adolfintel Your findings are very interesting. I suspected that this bug is caused by a combination of dynamic shadows and TextureMovie because eliminating one of these two things seems to fix it. I used ME3Explorer 5.1.0.2851 from https://github.com/ME3Tweaks/LegendaryExplorer/releases . I also experienced what you described with older ME3Explorer versions. There are newer versions called Legendary Explorer as well, but I haven't tried those. |
With this you may potentially be able to find the function producing that pointer - if it's done from only one or two places, it may be enough to jerry rig proper event-based synchronization there. The downside is obviously that it's hard to prove you cover all the code so if something remains unpatched it might introduce deadlocks, which isn't optimal either. |
|
@CookiePLMonster I have no idea how to do it. I tried placing a breakpoint on writes but it's in a different place every time. |
|
That may be tricky then, maybe time travel debugging in WinDbg could help but I never used it. Perhaps if you go up the call stack, you are eventually going to hit some object that is on a static address or at least easier to debug? |
|
The thing is, I don't even know which thread is writing that data, I looked at the stacks for the main thread, the rendering thread and the IO thread and I couldn't figure out what was going on, they're long and by the time the game crashes the function that messed up the pointer may have already finished running and its data overwritten by other calls. There are also other threads for audio and who knows what else, but their stack is very small and I doubt they're involved. I'll try that time travel feature tomorrow (gotta work today) and let you know if I discover something. (I have to say, this and the kasumi fix combined would be prime SilentPatch material 😉 ) |
|
Been a while, huh? After giving up in february, today I tested the game again, and it seems that this option in Valve's version of Wine fixes the freezing on Illium: So if you're using Proton, Wine-GE (or my TDF project) you might want to give it a try. |
|
You mean, that it fixes the crashing problem even without your patch? |
|
My patch was just a shitty workaround that didn't work 100% of the time, that's why I never released it. This seems to have solved it... for now at least. |
First of all, congratulations on fixing this problem. I tried fixing this game a few years ago but I was only able to determine that the game was not using 3DNow at all and that the problem was some incorrect calculation. I loved your article, very well written, great fix.
The next logical step now would be... Mass Effect 2: determining what causes the crashes on Illium (when you walk under Liara's office and at the end of Kasumi's loyalty mission). The first one I think is just memory corruption caused by some incorrect handling of pointers, as it can be worked around by switching between fullscreen and borderless; the second one I haven't got a clue, but it might be somethng inside the bink playback code in the game, since it crashes when a bik file is supposed to play.
Again, congratulations on this massive achievement ❤️
The text was updated successfully, but these errors were encountered: