building release fails when ERL_LIBS points to read-only filesystem

[dlesl]

When relx copies dependencies from ERL_LIBS, it preserves their mode and as a result subsequently fails to rewrite .app files with {error, eacces}.

[tsloughter]

Hm, I thought we had a fix for that recently...

To be clear, you aren't using dev mode, right?

[dlesl]

No, but I was worried for a second and had to double check :D. I just managed to confirm that this change fixes it: https://github.com/dlesl/relx/tree/make-files-writeable-3_33 -- it's this commit that made the difference dlesl@76c57fb

[dlesl]

OTOH if the fix for this was recent and I'm still using 3.33.0 that might explain it :)

[tsloughter]

Can you try the latest rebar3?

[tsloughter]

Oh, this (#825) only touches beams, it needs to also update .app

[dlesl]

I tried to try the latest rebar3 but I ran into another issue, however this time I'm not sure if its a bug in rebar3 or relx. My interpretation of the problem is that for some apps, relx doesn't look in ERL_LIBS to resolve them, and I think this might be because these apps are also dependencies of rebar3. I put together a minimal repro here: https://github.com/dlesl/relxcrashrepro

I have added certifi to the .app.src, but not added it as a dependency (because I would like to provide a compiled version via ERL_LIBS).

$ make DEBUG=1 REBAR3_VERSION=3.13.0
...
===> Failed to solve release:
 Dependency certifi is specified as a dependency but is not reachable by the system. 

As expected, rebar3 3.13.0 can't find the dependency (and would have found it in ERL_LIBS).
However with the latest rebar3:

$ make DEBUG=1 REBAR3_VERSION=3.14.3 
...
===> Uncaught error: {badmatch,{error,enotdir}}
...

If you change certifi to doesntexist in the .app.src, it behaves correctly.
I ran it with strace -f and saw stuff like this:

[pid  8351] readlink("/home/david/projects/relxcrashrepro/rebar3/certifi",  <unfinished ...>
[pid  8337] <... sched_yield resumed>)  = 0
[pid  8351] <... readlink resumed>0x7fb059f16410, 256) = -1 ENOTDIR (Not a directory)

which almost looks like it's trying to resolve the dependency "inside" the rebar3 escript?

[dlesl]

I did a bit more investigation into this and of the released rebar3 versions, the problem first arises in 3.14.0-rc1. I think the issue lies in relx_resolve:find_app_in_code_path -- because at this point the code path also contains the rebar3 escript's dependencies.
I'm not sure how to solve this, if I reset the code path before calling relx:build_release then of course the relx module no longer exists 😆. I also wonder if this could affect other rebar3 commands, for example would eunit tests be run with the versions of apps from rebar3's dependencies instead of those on ERL_LIBS.

[ferd]

yeah that's a common issue and we try to keep rebar3 dependencies as limited as possible to prevent clashes. For other cases we keep separate sets of code paths for plugins and for deps, but for the dependencies we use ourselves we're very limited.

[dlesl]

@ferd makes sense!

I had a few ideas regarding a path forward, do any of these sound reasonable?

• "Renaming" rebar3's dependencies to avoid clashes, eg. rebar3_certifi
• Implementing custom logic for resolving dependencies in relx (I'm guessing this is how it worked before 3.14?)
• Keeping the current behaviour but shelling out to a fresh erlang instance to call code:lib_dir, along the lines of erl -noshell -eval 'io:format("~p~n", [code:lib_dir(certifi)]),erlang:halt(0).'. This could be done only when resolving rebar3 dependencies to avoid a performance hit.