libfaketime 0.9.7 and up clashes with jemalloc #130
Comments
Whissi
changed the title
|
OK, this is related to jemalloc usage: When I build rsyslog without jemalloc ( |
|
Problems with jemalloc confirmed; but no solution yet. |
|
I spent a little time investigating this (in the process I found this issue). The root cause here is that jemalloc does neat stuff with time and threads, it is using timers to do things like arena-decay etc. So the problem here is that during libfaketime's initialization, it is calling dlsym(), which needs malloc, which initializes jemalloc, which creates a bunch of threads, which captures a bunch of timestamps. In the test case I had in front of me, I was able to get things working by avoiding faking during ftpl_init: DONT_FAKE_TIME(ftpl_init()); with that change, the only other issue is jemalloc also needs to initialize its condition-variable, so I removed stubbing on cond_init from libfaketime; and this was sufficient to get everything working. If a PR would be more helpful, I could clean up what I have working. I was doing all of this to help out a |
|
Thanks, that sounds very interesting. A PR would be awesome; we could then check whether it breaks anything and use it as a base. |
|
@timkordas A PR would still be useful w.r.t. rust-lang/rust#60461 I'm getting your point with DONT_FAKE_TIME on ftpl_init, but could you please elaborate on "removed stubbing on cond_init"? |
|
This is supposed to be fixed with the commit referenced above and in v0.9.8 / current HEAD, but lacks confirmation. Also, more information is still needed on the "cond_init" part. Please re-open if you have any news. |
|
Rust still doesn't work with version 0.9.8 with default config: Details |
|
Well, I can confirm such an endless recursion with a more minimum example of jemalloc than Rust. Apparently not fixed. :-( However, even with FAKE_PTHREAD not enabled and wrapping all of ftpl_init() in DONT_FAKE_TIME, it does not work for me, so I cannot reproduce the fix sketched by @timkordas yet. Please provide more details. I'm currently uncertain whether this can be fixed at all in libfaketime. As @timkordas already pointed out, libfaketime needs dlsym() in order to pass intercepted function calls through to their original version; in this case it is clock_gettime(). dlsym() apparently allocates some memory, which is handled by jemalloc, and jemalloc needs to fetch a timestamp, which libfaketime cannot provide unless the dlsym() call succeeded. Hence the endless recursion. Trying to break the recursion by returning an error just leads to another endless hang. Eventually jemalloc expects clock_gettime to always succeed (jemalloc's src/nstime.c does not have any apparent error handling for its calls to clock_gettime, but I didn't really dig into its source yet). I'm open for suggestions... |
|
Sorry, I haven't had a chance to clean up and submit a PR.
the old branch I have lying around has this for every instance of the
!initialized check:
if (!initialized)
{
#ifndef SUPPORT_JEMALLOC
ftpl_init();
#else
DONT_FAKE_TIME(ftpl_init());
#endif
}
that is, until we're initialized, we can't support any faking.
…-Tim
On Tue, Sep 3, 2019 at 6:20 AM Wolfgang Hommel ***@***.***> wrote:
Well, I can confirm such an endless recursion with a more minimum example
of jemalloc than Rust. Apparently not fixed. :-(
However, even with FAKE_PTHREAD not enabled and wrapping all of
ftpl_init() in DONT_FAKE_TIME, it does not work for me, so I cannot
reproduce the fix sketched by @timkordas <https://github.com/timkordas>
yet. Please provide more details.
I'm currently uncertain whether this can be fixed at all in libfaketime.
As @timkordas <https://github.com/timkordas> already pointed out,
libfaketime needs dlsym() in order to pass intercepted function calls
through to their original version; in this case it is clock_gettime().
dlsym() apparently allocates some memory, which is handled by jemalloc, and
jemalloc needs to fetch a timestamp, which libfaketime cannot provide
unless the dlsym() call succeeded. Hence the endless recursion.
Trying to break the recursion by returning an error just leads to another
endless hang. Eventually jemalloc expects clock_gettime to always succeed
(jemalloc's src/nstime.c does not have any apparent error handling for its
calls to clock_gettime, but I didn't really dig into its source yet).
I'm open for suggestions...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#130?email_source=notifications&email_token=AHGXTGNWQNDJJARFMTQ6RXDQHZQANA5CNFSM4EFI55W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5YFG7I#issuecomment-527455101>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHGXTGMWTBH2BIH5MTYMXRDQHZQANANCNFSM4EFI55WQ>
.
|
|
Thanks for sharing! Interestingly enough, this doesn't change anything for me, and I'm a bit puzzled why that works for you. My jemalloc test program simply malloc()s a couple of bytes and uses jemalloc, just like libfaketime, as LD_PRELOAD library. Are you linking it with your application? One aspect here is that each of the intercepted time-related system calls checks whether libfaketime is initialized as its first step and calls ftpl_init() if required. ftpl_init() then pretty quickly dives into dlsym() of all the original functions, where the recursion with jemalloc happens. I don't see how DONT_FAKE_TIME(), which simply sets the global variable dont_fake to false as a wrapper around arbitrary calls, would even influence that. |
|
826f8b7 will now detect and warn about recursive calls to clock_gettime() as they happen when used together with current jemalloc. First, it will try to go the DONT_FAKE_TIME(ftpl_init()) way suggested by @timkordas, even if that never worked for me. On the next recursion, it will return -1 to the call to clock_gettime(), which signals an error that should help the caller (jemalloc's nstime_get() in this particular case) to detect that something is wrong. This does not fix compatibility with current jemalloc for me. I assume that jemalloc currently does not expect and handle such an error on calls to clock_gettime(). I'd appreciate any suggestions on how we can fix this purely on the libfaketime side, but at the moment I assume that it can't work without changes made to jemalloc as well. Therefore, I don't know what to do about rust-lang/rust#60461 right now, sorry. |
|
I think this cannot be easily solved, as the initial stacktrace shows that the malloc called by dlsym causes the recursion. Since I think libfaketime cannot know whether jemalloc is done initialising (or if it is used at all), I think it cannot know when it is allowed to start using dlsym. What do you think? Also the stacktrace does not show an endless recursion, but I think a deadlock. jemalloc locks its semaphore, then calls a clock function ending in faketime, faketime calls dlsym to load original clock function, dlsym calls malloc and ends up in jemalloc again. jemalloc locks its ... Wait, its already locked, so it waits. Forever. jemalloc init is not reentrant. I see not much what faketime could do about jemallocs deadlock. Do you think I misread the initial stracktrace and understood well? In case I didn't missunderstand, I think it would only help to link either jemalloc or libfaketime statically (against libc and libdl, and whatever is needed). At least on Linux with its extremely stable kernel ABI I think this should be even quite portable across systems without recompiling (on other systems, statically linked binaries might be very system specific). For me malloc is really basic, so I think it should avoid any dependencies (I don't know what it needs clock for, maybe it writes a log file or so?). |
|
mmm I just wonder if normal clock_gettime() in glibc could in a future version also do some lazy initialization and need malloc. If there is nothing that forbids that (some standard or so), then I think this is more likely a jemalloc issue than a libfaketime one - because in that case it would fail without libfaketime as well :) |
|
I just tested this again with latest libfaketime (0.9.9) and every application linked against jemalloc is still failing. So we will stick to 0.9.6... |
Older versions of BIND require commenting out qname-minimization configuration. BIND builds linked to jemalloc or Clang ASAN will fail to start on systems with libfaketime versions > 0.9.6: - jemalloc wolfcw/libfaketime#130 - Clang ASAN wolfcw/libfaketime#365
Older versions of BIND require commenting out qname-minimization configuration. BIND builds linked to jemalloc or Clang ASAN will fail to start on systems with libfaketime versions > 0.9.6: - jemalloc wolfcw/libfaketime#130 - Clang ASAN wolfcw/libfaketime#365
|
I'm reopening this issue as it does not seem to be resolved. Even if there's no idea how to get to a resolution, it's useful to be able to find known outstanding issues in the list of open issues. |
dkg
changed the title
So you think my analysis (that it is a dead lock because of non-reentrant init in remalloc) is incorrect? |
Older versions of BIND require commenting out qname-minimization configuration. BIND builds linked to jemalloc or Clang ASAN will fail to start on systems with libfaketime versions > 0.9.6: - jemalloc wolfcw/libfaketime#130 - Clang ASAN wolfcw/libfaketime#365
|
@sdettmer i'm not claiming that your analysis is incorrect -- it looks reasonable to me. i'm trying to understand whether a fix has been applied that would let this work with jemalloc, and i don't think it's resolved. I tried building this simple test binary with And it does hang when run under That makes me think we don't have a solution to this problem. is there a fix i'm missing, beyond "don't do that, then"? |
|
No, we don't have a fix for that. Last time we traced this, we figured out that https://github.com/jemalloc/jemalloc/blob/dev/src/nstime.c has calls to clock_gettime() but does not check the return value for errors. So while libfaketime can detect the problem (it leads to indirect recursive calls, see Lines 2261 to 2291 in 0ca35dd As a consequence, libfaketime cannot initialize properly (because its use of dlsym() apparently needs to malloc() memory somewhere), and things hang. Given that we can neither return the real time from libfaketime nor a properly faked time without initialisation, and the caller ignores the error-indicating return value, there is likely little to nothing that can be done on libfaketime's side. We could try with returning a constant (non-zero) fake timestamp and postpone initialisation to a later point, but similar to unrelated "hang" issues with other libraries (e.g., glibc and FORCE_MONOTONIC_FIX), this eventually means we'd have to return constant timestamps more than once before it is "safe" to use dlsym(), and thus might need a detection of which other libraries (and in which version) are in use. However, the use of constant timestamps may still break the calling application (once we can then switch to properly faked timestamps), so I don't think a clean solution is possible without the calling function honouring clock_gettime()'s return value. |
|
But is the change from 0bde083 correct? I.e. previous versions without this change are working, so you could argue this change is causing the problem. But maybe it is required/fixing another issue or just uncovered this long standing bug? Should we try to connect with people from jemalloc to get their input? Maybe they have an idea or could implement |
|
The recursion seems to be like this:
libfaketime cannot work properly unless initialised, and the change you pointed to just makes sure that libfaketime initialisation is done no matter which of its functions is called first. To make things work for every component involved, including libfaketime, the recursion must be broken. Once libfaketime is initalised, this should no longer be a problem. Alternatively, we'd need a dlsym() implementation that does not rely on malloc() (and thus calls jemalloc recursively). Currently, libfaketime detects the recursion and tries to break it (by returning an error value). But given that, so far, jemalloc apparently ignores this error value, recursion detection alone does not solve the problem yet. My current guess is that this either would have to be addressed in jemalloc (i.e., handle clock_gettime() return values appropriately), or we'd need a dlsym() without dynamic memory allocation for libfaketime initialisation (i.e., don't call jemalloc's malloc() recursively during dlsym()). |
|
Thanks you all for your valuable input. Let me try to summarize what I think is the current state of analysis. To @wolfcw summary
I would like to add that strictly speaking it is not just a recursion but a cyclic dependency. Dependencies need to form a DAG (https://en.wikipedia.org/wiki/Directed_acyclic_graph) to be resolvable, but we we have the cycle. The special form of recursion we find here I think is called "reentrance". I don't think libfaketime can detect a recursive call, because as pointed by @wolfcw summary, there is only one instance of clock_gettime(). In contrast, jemalloc() calls itself indirectly and jemalloc() possibly could detect and work around this (I did not though through, possibly it is not safely feasible when for example including signal handlers calling malloc when a signal arrives while in jemalloc code - signal safe reentrace usually is hard). Because I don't see faketime called twice in the analysis (only jemalloc is), I think the initialization patches will do no good (and don't work) and should not be merged into. I don't see where this error value introduced by the patch could be returned - the problem is, that dlopen blocks and thus never returns. faketime also cannot load libc to get the "real" malloc, because it would also need to call dlopen. I think malloc() is a very essential function and to make it dependent on clock_gettime(), which might be implemented as system call with context switch, for example, is possibly not the best idea. Code simply relies on malloc. jemalloc could initialize all its dependency (by calling all needed functions) before hijacking malloc, but still it's not safe. jemalloc could try to implement a "reentrance" detection and simply call original malloc in that case, however this possibly breaks things (for example, the allocated block then would not be traced, but the free of it could be, so it would look like a memory corruption, because a block is freed without mallocing according to jemalloc log file!). jemalloc could also try to link its dependencies statically. |
|
Let's collect our options. Link staticallyAs pointed before, like jemalloc, I think also libfaketime could link dl* and malloc (libc.a, libdl.a) statically to solve this particular issue, but I don't know whether every user wants to pay its price. Initialization FunctionThe initialization order of technically independent libraries is, as far I know, undefined. However, libdl will use a fixed one (either in order of appearance or in reverse order I think). So we possibly could practically enforce to init libfaketime first (although not guaranteed because officially undefined). If so, I think we could use an initialization function in libfaketime to dlload first (https://stackoverflow.com/a/9759936 shows ways to accomplish this). This approach would be Linux (or GCC) specific. I think other systems may use other ways to archive that; every system supporting C++ should have a way to do so, but I'm not sure if C standard guarantees that such exists everywhere). Implement real clock_gettimeWhen being Linux specific, we could also consider implementing the syscall our selves. Possibly its something like /* Get current value of CLOCK and store it in TP. */
int
__clock_gettime (clockid_t clock_id, struct timespec *tp)
{
return INLINE_VSYSCALL (clock_gettime, 2, clock_id, tp);
}(see https://code.woboq.org/userspace/glibc/sysdeps/unix/sysv/linux/clock_gettime.c.html#__clock_gettime) but this would be very slowwithout vDSO. vDSO could be complicated If we cannot call __vdso_clock_gettime I think code would be too complicated. In summary I think we could investigate the initialization function option. What do you think? |
|
Thanks for your precise summary and listing the options!! I need to dig deeper into them, just wanted to point out that we are already using
and I'm not sure whether using Also, a solution that only works for Linux would be fine as a first step, we just have to make sure not to break things for other OS. |
So, does someone have a makefile building a correct static |
|
Just pinging if there was an update for this issue. I had been using a patched version of libfaketime to get around the jemalloc issue for builds I am doing. The patch was not accepted, so I am not sure if there is any update for this going forward. |
|
No progress yet. Could you please point to the PR/patch you submitted? |
|
I did not submit a patch I was build from 0.9.9 based on a patch .. details here in this issue #355 I believe there was a concern in the patch, so it was not merged. |
|
The patch was this PR.. #333 |
|
Thanks for pointing those out. Yes, PR #333 was withdrawn by its author, @ronrother, as it could not cover all cases reliably. Your issue with stale shared memory files might be a consequence of failed initialisations, but working around those will not fix the jemalloc compatibility issue. As I posted above on Sep 3, 2019, I think one key aspect here is that jemalloc does not consider that calls to clock_gettime() may ever fail. I've just checked their current src/nstime.c again and nothing seems to have changed there in this regard. I still think that we cannot fix this compatibility issue purely on libfaketime's end, altough I'd be happy to be proven wrong. I didn't find an issue related to libfaketime in jemalloc's repository, so opening one might be a logical step to help with progress. There are other tools like libasan, which apparently have similar incompatibilities. It'd really help if those affected could assist with digging into both ends. :-) |
|
I hit this issue with libfaketime 0.9.10, with glibc 2.39, Firefox 123.0, on ArchLinux. I explored ways to avoid the issue by avoiding One of the issues is that
I created a patch (#463) to allow users to avoid calling How to avoid (indirect) malloc calls from libfaketimeWhen I tried to launch Firefox through Stack trace: malloc -> time -> dlsym -> malloc -> exception/malloc recursion#95207 0x000056111b95921a in malloc () #95208 0x00007f9982983af2 in malloc (size=87) at ../include/rtld-malloc.h:56 #95209 __GI__dl_exception_create_format (exception=exception@entry=0x7ffc36a0e8e0, objname=0x7f99829b6f80 "/tmp/libfaketime/src/libfaketime.so.1", fmt=fmt@entry=0x7f99829a9e2b "undefined symbol: %s%s%s") at dl-exception.c:157 #95210 0x00007f998298a72d in _dl_lookup_symbol_x (undef_name=0x7f9982974b90 "__ftime", undef_map=<optimized out>, ref=0x7ffc36a0e958, symbol_scope=<optimized out>, version=<optimized out>, type_class=0, flags=2, skip_map=0x7f998297e000) at dl-lookup.c:809 #95211 0x00007f99824860ce in do_sym (handle=<optimized out>, name=0x7f9982974b90 "__ftime", who=0x7f998296f323 <ftpl_really_init+1049>, vers=vers@entry=0x0, flags=flags@entry=2) at dl-sym.c:146 #95212 0x00007f99824863f1 in _dl_sym (handle=<optimized out>, name=<optimized out>, who=<optimized out>) at dl-sym.c:195 #95213 0x00007f99823b94f8 in dlsym_doit (a=a@entry=0x7ffc36a0ebc0) at dlsym.c:40 #95214 0x00007f99829814e1 in __GI__dl_catch_exception (exception=exception@entry=0x7ffc36a0eb20, operate=0x7f99823b94e0 <dlsym_doit>, args=0x7ffc36a0ebc0) at dl-catch.c:237 #95215 0x00007f9982981603 in _dl_catch_error (objname=0x7ffc36a0eb78, errstring=0x7ffc36a0eb80, mallocedp=0x7ffc36a0eb77, operate=<optimized out>, args=<optimized out>) at dl-catch.c:256 #95216 0x00007f99823b8f07 in _dlerror_run (operate=operate@entry=0x7f99823b94e0 <dlsym_doit>, args=args@entry=0x7ffc36a0ebc0) at dlerror.c:138 #95217 0x00007f99823b9585 in dlsym_implementation (dl_caller=<optimized out>, name=<optimized out>, handle=<optimized out>) at dlsym.c:54 #95218 ___dlsym (handle=<optimized out>, name=<optimized out>) at dlsym.c:68 #95219 0x00007f998296f323 in ftpl_really_init () from /tmp/libfaketime/src/libfaketime.so.1 #95220 0x00007f998296ffb4 in ftpl_init () from /tmp/libfaketime/src/libfaketime.so.1 #95221 0x00007f998296e025 in time () from /tmp/libfaketime/src/libfaketime.so.1 #95222 0x000056111b9a0af3 in ?? () #95223 0x000056111b95935f in malloc () #95224 0x00007f99826ac4fb in (anonymous namespace)::pool::pool (this=0x7f9982876280 <(anonymous namespace)::emergency_pool>) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:235 #95225 __static_initialization_and_destruction_0 () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:373 #95226 _GLOBAL__sub_I_eh_alloc.cc(void) () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:456 #95227 0x00007f99829851be in call_init (env=0x7ffc36a0f010, argv=0x7ffc36a0efe8, argc=4, l=<optimized out>) at dl-init.c:74 #95228 call_init (l=<optimized out>, argc=4, argv=0x7ffc36a0efe8, env=0x7ffc36a0f010) at dl-init.c:26 #95229 0x00007f99829852ac in _dl_init (main_map=0x7f99829b62e0, argc=4, argv=0x7ffc36a0efe8, env=0x7ffc36a0f010) at dl-init.c:121 #95230 0x00007f998299bbc0 in _dl_start_user () from /lib64/ld-linux-x86-64.so.2 I digged in the source code of glibc and found that it tries to allocate memory for an error message when a symbol is not found: https://github.com/bminor/glibc/blob/b6e3898194bbae78910bbe9cd086937014961e45/elf/dl-lookup.c#L800-L812 But after starting with Stack trace: malloc -> time -> ftpl_init -> ftpl_really_init -> ft_shm_init -> sem_open -> malloc -> exception/malloc recursion#91000 0x00005585f3e09d49 in malloc () #91001 0x00007fb718cb0fab in __sem_check_add_mapping (name=name@entry=0x7ffc4cc95410 "/faketime_sem_1112645", fd=fd@entry=3, existing=existing@entry=0x0) at ../sysdeps/pthread/sem_routines.c:104 #91002 0x00007fb718cb0b68 in __sem_open (name=0x7ffc4cc95410 "/faketime_sem_1112645", oflag=0) at ../sysdeps/pthread/sem_open.c:83 #91003 0x00007fb7192b6b07 in ft_shm_init () from /tmp/libfaketime/src/libfaketime.so.1 #91004 0x00007fb7192bd6a2 in ftpl_init () from /tmp/libfaketime/src/libfaketime.so.1 #91005 0x00007fb7192bb997 in time () from /tmp/libfaketime/src/libfaketime.so.1 #91006 0x00005585f3e18846 in GMut* InfallibleAllocPolicy::new_<GMut>() () #91007 0x00005585f3e09dde in malloc () #91008 0x00007fb718eac4fb in (anonymous namespace)::pool::pool (this=0x7fb719076280 <(anonymous namespace)::emergency_pool>) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:235 #91009 __static_initialization_and_destruction_0 () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:373 #91010 _GLOBAL__sub_I_eh_alloc.cc(void) () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:456 #91011 0x00007fb7192d31be in call_init (env=0x7ffc4cc95a40, argv=0x7ffc4cc95a18, argc=4, l=<optimized out>) at dl-init.c:74 #91012 call_init (l=<optimized out>, argc=4, argv=0x7ffc4cc95a18, env=0x7ffc4cc95a40) at dl-init.c:26 #91013 0x00007fb7192d32ac in _dl_init (main_map=0x7fb7193042e0, argc=4, argv=0x7ffc4cc95a18, env=0x7ffc4cc95a40) at dl-init.c:121 #91014 0x00007fb7192e9bc0 in _dl_start_user () from /lib64/ld-linux-x86-64.so.2 I don't see a reasonable way to avoid the $ cd /tmp/libfaketime # this repository
$ make PREFIX= LIBDIRNAME=$PWD FAKETIME_COMPILE_CFLAGS=-DFAKE_STATELESS=1
$ LD_DEBUG=unused LD_PRELOAD=/tmp/libfaketime/src/libfaketime.so.1 FAKETIME='2024-06-07 12:34:56' firefox --no-remote -profile /tmp/profiledirAnd then it crashes again, but this time in Stack trace: malloc -> time -> ftpl_init -> ftpl_really_init -> parse_ft_string -> mktime -> free -> malloc/exception recursion#95212 0x000055606fbe121a in malloc () #95213 0x00007ff08a4aef35 in __cxxabiv1::__cxa_allocate_exception (thrown_size=136) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:398 #95214 0x00007ff08a49c4a0 in __cxxabiv1::throw_recursive_init_exception () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/guard.cc:248 #95215 __cxxabiv1::__cxa_guard_acquire (g=0x55606fca2628) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/guard.cc:299 #95216 0x000055606fc22673 in free () #95217 0x00007ff08a1fc1da in tzset_internal (always=always@entry=1) at tzset.c:401 #95218 0x00007ff08a1fc343 in __tzset () at tzset.c:551 #95219 0x00007ff08a1fb01d in __GI_mktime (tp=0x7ffc0f69dc20) at mktime.c:541 #95220 0x00007ff08a757803 in parse_ft_string () from /tmp/libfaketime/src/libfaketime.so.1 #95221 0x00007ff08a758f61 in ftpl_really_init () from /tmp/libfaketime/src/libfaketime.so.1 #95222 0x00007ff08a758fb4 in ftpl_init () from /tmp/libfaketime/src/libfaketime.so.1 #95223 0x00007ff08a757025 in time () from /tmp/libfaketime/src/libfaketime.so.1 #95224 0x000055606fc28af3 in ?? () #95225 0x000055606fbe135f in malloc () #95226 0x00007ff08a4ac4fb in (anonymous namespace)::pool::pool (this=0x7ff08a676280 <(anonymous namespace)::emergency_pool>) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:235 #95227 __static_initialization_and_destruction_0 () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:373 #95228 _GLOBAL__sub_I_eh_alloc.cc(void) () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:456 #95229 0x00007ff08a76e1be in call_init (env=0x7ffc0f69e110, argv=0x7ffc0f69e0e8, argc=4, l=<optimized out>) at dl-init.c:74 #95230 call_init (l=<optimized out>, argc=4, argv=0x7ffc0f69e0e8, env=0x7ffc0f69e110) at dl-init.c:26 #95231 0x00007ff08a76e2ac in _dl_init (main_map=0x7ff08a79f2e0, argc=4, argv=0x7ffc0f69e0e8, env=0x7ffc0f69e110) at dl-init.c:121 #95232 0x00007ff08a784bc0 in _dl_start_user () from /lib64/ld-linux-x86-64.so.2 Okay, next try: now avoiding
Now from
After patching libfaketime once more (#463), to enable skipping of dlsym/dlvsym calls without the use of Because v0.9.10 works I wondered whether the issue was with the master branch, and indeed, when I checked out https://github.com/wolfcw/libfaketime/commits/v0.9.10/ , applied #463 and #465 on top, then I finally managed to launch Firefox 123.0 with faketime:
EDIT: libfaketime has only started to be an issue in Firefox since version 120, here is the bug I filed: https://bugzilla.mozilla.org/show_bug.cgi?id=1884996 (and fixed in Firefox 125!) |
|
Thank you for going down this rabbit hole!! :-) I like the approach as it does not impact regular use compatibility. As you point out with the limitation to FAKE_STATELESS already, it's hard to tell whether really all nested malloc() calls will be avoidable (e.g., if libfaketime adds further functionality). But it seems more than interesting enough when also working with jemalloc and similar libraries in a setting where FAKE_STATELESS is currently sufficient. :-) |
Hi,
sorry, I don't know a good summary at the moment.
rsyslog is using libfaketime via LD_PRELOAD [1,2] in its test suite.
It was working fine until libfaketime-0.9.7 release, see rsyslog/rsyslog#2099
I bisected libfaketime and found the commit which broke libfaketime for rsyslog:
Bad commit: 0bde083
[1] https://github.com/rsyslog/rsyslog/blob/master/tests/faketime_common.sh
[2] https://github.com/rsyslog/rsyslog/blob/master/tests/now_family_utc.sh
The text was updated successfully, but these errors were encountered: