Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dune subst no longer works on empty directories in ocaml-ci without Dune 3.17 #11200

Closed
shonfeder opened this issue Dec 12, 2024 · 1 comment · Fixed by ocaml/opam-repository#27137
Closed

dune subst no longer works on empty directories in ocaml-ci without Dune 3.17 #11200

shonfeder opened this issue Dec 12, 2024 · 1 comment · Fixed by ocaml/opam-repository#27137

Comments

@shonfeder
Copy link
Collaborator

shonfeder commented Dec 12, 2024 •
edited
Loading

Expected Behavior

dune subst (invovked by opam) continues to work on empty directories of vendored projects for pinned dependencies, as it did up until 3.16

Actual Behavior

Failes with

### output ###
# File ".", line 1, characters 0-0:
# Error: There is no dune-project file in the current directory, please add one
# with a (name <name>) field in it.
# Hint: 'dune subst' must be executed from the root of the project.

Reproduction

This is not at all minimal, but is a starter, taken from an ocaml-ci job (where this problem has been encountered):

git clone --recursive "https://github.com/ocurrent/ocaml-ci.git" -b "master" && cd "ocaml-ci" && git reset --hard bc5c5e55
cat > Dockerfile <<'END-OF-DOCKERFILE'
FROM ocaml/opam:ubuntu-22.04-ocaml-4.14@sha256:1c5bf3c2bec3ee454185e5155264394ba79f70e5c3985b66f7bd57810d9cbebb
# ubuntu-22.04-4.14_opam-2.3
USER 1000:1000
ENV CLICOLOR_FORCE="1"
ENV OPAMCOLOR="always"
WORKDIR /src
RUN sudo ln -f /usr/bin/opam-2.3 /usr/bin/opam
RUN opam init --reinit -ni
RUN uname -rs && opam exec -- ocaml -version && opam --version
WORKDIR /src
RUN sudo chown opam /src
RUN cd ~/opam-repository && (git cat-file -e 9d811310def03190e9b9b1db9c9ea19b4f954c17 || git fetch origin master) && git reset -q --hard 9d811310def03190e9b9b1db9c9ea19b4f954c17 && git log --no-decorate -n1 --oneline && opam update -u
RUN mkdir -p 'solver-service/' 'ocurrent/' 'ocluster/' 'ocluster/obuilder/' 'ocluster/' 'ocaml-version/' 'ocaml-dockerfile/'
COPY --chown=1000:1000 solver-service/solver-worker.opam solver-service/solver-service.opam solver-service/solver-service-api.opam solver-service/
COPY --chown=1000:1000 ocurrent/current_web.opam ocurrent/current_ssh.opam ocurrent/current_slack.opam ocurrent/current_rpc.opam ocurrent/current_gitlab.opam ocurrent/current_github.opam ocurrent/current_git.opam ocurrent/current_examples.opam ocurrent/current_docker.opam ocurrent/current.opam ocurrent/
COPY --chown=1000:1000 ocluster/ocluster.opam ocluster/ocluster-worker.opam ocluster/ocluster-api.opam ocluster/
COPY --chown=1000:1000 ocluster/obuilder/obuilder.opam ocluster/obuilder/obuilder-spec.opam ocluster/obuilder/
COPY --chown=1000:1000 ocluster/current_ocluster.opam ocluster/
COPY --chown=1000:1000 ocaml-version/ocaml-version.opam ocaml-version/
COPY --chown=1000:1000 ocaml-dockerfile/dockerfile.opam ocaml-dockerfile/dockerfile-opam.opam ocaml-dockerfile/dockerfile-cmd.opam ocaml-dockerfile/
COPY --chown=1000:1000 ocaml-ci.opam ocaml-ci-web.opam ocaml-ci-service.opam ocaml-ci-gitlab.opam ocaml-ci-client.opam ocaml-ci-api.opam ./
RUN opam pin add -yn solver-worker.dev 'solver-service/' && \
    opam pin add -yn solver-service.dev 'solver-service/' && \
    opam pin add -yn solver-service-api.dev 'solver-service/' && \
    opam pin add -yn current_web.dev 'ocurrent/' && \
    opam pin add -yn current_ssh.dev 'ocurrent/' && \
    opam pin add -yn current_slack.dev 'ocurrent/' && \
    opam pin add -yn current_rpc.dev 'ocurrent/' && \
    opam pin add -yn current_gitlab.dev 'ocurrent/' && \
    opam pin add -yn current_github.dev 'ocurrent/' && \
    opam pin add -yn current_git.dev 'ocurrent/' && \
    opam pin add -yn current_examples.dev 'ocurrent/' && \
    opam pin add -yn current_docker.dev 'ocurrent/' && \
    opam pin add -yn current.dev 'ocurrent/' && \
    opam pin add -yn ocluster.dev 'ocluster/' && \
    opam pin add -yn ocluster-worker.dev 'ocluster/' && \
    opam pin add -yn ocluster-api.dev 'ocluster/' && \
    opam pin add -yn obuilder.dev 'ocluster/obuilder/' && \
    opam pin add -yn obuilder-spec.dev 'ocluster/obuilder/' && \
    opam pin add -yn current_ocluster.dev 'ocluster/' && \
    opam pin add -yn ocaml-version.dev 'ocaml-version/' && \
    opam pin add -yn dockerfile.dev 'ocaml-dockerfile/' && \
    opam pin add -yn dockerfile-opam.dev 'ocaml-dockerfile/' && \
    opam pin add -yn dockerfile-cmd.dev 'ocaml-dockerfile/' && \
    opam pin add -yn ocaml-ci.dev './' && \
    opam pin add -yn ocaml-ci-web.dev './' && \
    opam pin add -yn ocaml-ci-service.dev './' && \
    opam pin add -yn ocaml-ci-gitlab.dev './' && \
    opam pin add -yn ocaml-ci-client.dev './' && \
    opam pin add -yn ocaml-ci-api.dev './'
# RUN echo '(lang dune 3.0)' > 'solver-service/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocurrent/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocluster/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocluster/obuilder/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocluster/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocaml-version/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocaml-dockerfile/dune-project' && \
#   echo '(lang dune 3.0)' > './dune-project'
ENV DEPS="0install-solver.2.18 alcotest.1.8.0 alcotest-lwt.1.8.0 angstrom.0.16.1 ansi.0.7.0 arp.3.1.1 asetmap.0.8.1 asn1-combinators.0.2.6 astring.0.8.5 atd.2.15.0 atdgen.2.15.0 atdgen-runtime.2.15.0 awa.0.3.0 awa-mirage.0.3.0 base.v0.16.3 base-bigarray.base base-bytes.base base-threads.base base-unix.base base64.3.5.1 bigarray-compat.1.1.0 bigarray-overlap.0.2.1 bigstringaf.0.10.0 biniou.1.2.2 bos.0.2.1 ca-certs.0.2.3 ca-certs-nss.3.101 camlp-streams.5.0.1 capnp.3.6.0 capnp-rpc.1.2.3 capnp-rpc-lwt.1.2.3 capnp-rpc-net.1.2.3 capnp-rpc-unix.1.2.3 caqti.2.1.2 caqti-lwt.2.1.1 carton.0.7.2 carton-git.0.7.2 carton-lwt.0.7.2 cf.0.5.0 cf-lwt.0.5.0 checkseum.0.5.2 cmdliner.1.3.0 cohttp.6.0.0 cohttp-lwt.6.0.0 cohttp-lwt-unix.6.0.0 conduit.6.2.3 conduit-lwt.6.2.3 conduit-lwt-unix.6.2.3 conf-bash.1 conf-capnproto.2 conf-git.1.1 conf-gmp.4 conf-gmp-powm-sec.3 conf-graphviz.0.1 conf-libev.4-12 conf-libffi.2.0.0 conf-libssl.4 conf-pkg-config.3 conf-sqlite3.1 conf-which.1 cppo.1.8.0 crunch.3.3.1 csexp.1.5.2 cstruct.6.2.0 cstruct-lwt.6.2.0 cstruct-unix.6.2.0 csv.2.4 ctypes.0.23.0 ctypes-foreign.0.23.0 current.dev current_docker.dev current_git.dev current_github.dev current_gitlab.dev current_incr.0.6.1 current_ocluster.dev current_rpc.dev current_web.dev decompress.1.5.3 digestif.1.2.0 dns.7.0.3 dns-client.7.0.3 dns-client-lwt.7.0.3 dns-client-mirage.7.0.3 dockerfile.dev dockerfile-opam.dev domain-name.0.4.0 dream.1.0.0~alpha6 dream-httpaf.1.0.0~alpha3 dream-pure.1.0.0~alpha2 duff.0.5 dune.3.17.0 dune-build-info.3.17.0 dune-configurator.3.17.0 dune-private-libs.3.17.0 dune-site.3.17.0 duration.0.2.1 dyn.3.17.0 easy-format.1.3.4 emile.1.1 encore.0.8 eqaf.0.9 ethernet.3.2.0 extunix.0.4.3 faraday.0.8.2 faraday-lwt.0.8.2 faraday-lwt-unix.0.8.2 fmt.0.9.0 fpath.0.7.3 fsevents.0.3.0 fsevents-lwt.0.3.0 git.3.15.0 git-mirage.3.15.0 git-paf.3.15.0 git-unix.3.15.0 github.4.4.1 github-data.4.4.1 github-unix.4.4.1 gitlab.0.1.8 gitlab-unix.0.1.8 gmap.0.3.0 graphql.0.14.0 graphql-lwt.0.14.0 graphql_parser.0.14.0 h2.0.13.0 happy-eyeballs.0.6.0 happy-eyeballs-lwt.0.6.0 happy-eyeballs-mirage.0.6.0 hex.1.5.0 hkdf.1.0.4 hmap.0.8.1 host-arch-x86_64.1 host-system-other.1 hpack.0.13.0 http.6.0.0 httpaf.0.7.1 httpun-types.0.2.0 hxd.0.3.3 ISO8601.0.2.6 inotify.2.6 integers.0.7.0 ipaddr.5.6.0 ipaddr-cstruct.5.6.0 ipaddr-sexp.5.6.0 irmin-watcher.0.5.0 jsonm.1.0.2 ke.0.6 lambdasoup.1.1.1 logs.0.7.0 lru.0.3.1 lwt.5.9.0 lwt-dllist.1.0.1 lwt_ppx.5.8.0 lwt_ssl.1.2.0 macaddr.5.6.0 macaddr-cstruct.5.6.0 magic-mime.1.3.1 markup.1.0.3 menhir.20240715 menhirCST.20240715 menhirLib.20240715 menhirSdk.20240715 metrics.0.4.1 mimic.0.0.6 mimic-happy-eyeballs.0.0.6 mirage-clock.4.2.0 mirage-clock-unix.4.2.0 mirage-crypto.0.11.3 mirage-crypto-ec.0.11.3 mirage-crypto-pk.0.11.3 mirage-crypto-rng.0.11.3 mirage-crypto-rng-lwt.0.11.3 mirage-flow.3.0.0 mirage-kv.6.1.1 mirage-net.4.0.0 mirage-random.3.0.0 mirage-runtime.4.8.2 mirage-time.3.0.0 mirage-unix.5.0.1 mtime.2.1.0 multipart_form.0.6.0 multipart_form-lwt.0.6.0 num.1.5-1 obuilder-spec.dev ocaml.4.14.2 ocaml-base-compiler.4.14.2 ocaml-compiler-libs.v0.12.4 ocaml-config.2 ocaml-options-vanilla.1 ocaml-syntax-shims.1.0.0 ocaml-version.dev ocamlbuild.0.15.0 ocamlfind.1.9.6 ocamlgraph.2.1.0 ocluster-api.dev ocplib-endian.1.2 omigrate.0.3.2 opam-0install.0.5 opam-core.2.3.0 opam-file-format.2.1.6 opam-format.2.3.0 opam-repository.2.3.0 opam-state.2.3.0 optint.0.3.0 ordering.3.17.0 paf.0.5.0 parsexp.v0.16.0 pbkdf.1.2.0 pecu.0.7 pgx.2.2 pgx_lwt.2.2 pgx_lwt_unix.2.2 pp.2.0.0 ppx_compare.v0.16.0 ppx_custom_printf.v0.16.0 ppx_derivers.1.2.1 ppx_deriving.6.0.3 ppx_deriving_yojson.3.9.0 ppx_sexp_conv.v0.16.0 ppxlib.0.33.0 prettym.0.0.3 prometheus.1.2 prometheus-app.1.2 psq.0.2.1 ptime.1.2.0 randomconv.0.1.3 re.1.12.0 res.5.0.2 result.1.5 routes.2.0.0 rresult.0.7.0 seq.base session.0.5.0 session-cohttp.0.5.0 session-cohttp-lwt.0.5.0 sexplib.v0.16.0 sexplib0.v0.16.0 sha.1.15.4 solver-service.dev solver-service-api.dev spdx_licenses.1.2.0 sqlite3.5.2.0 ssl.0.7.0 stdint.0.7.2 stdio.v0.16.0 stdlib-shims.0.3.0 stdune.3.17.0 stringext.1.6.0 swhid_core.0.1 tailwindcss.dev tar.2.6.0 tar-unix.2.6.0 tcpip.8.0.0 timedesc.3.1.0 timedesc-tzdb.3.1.0 timedesc-tzlocal.3.1.0 tls.0.17.3 tls-lwt.0.17.3 tls-mirage.0.17.3 topkg.1.0.7 tyxml.4.6.0 uchar.0.0.2 unstrctrd.0.4 uri.4.4.0 uri-sexp.4.4.0 uuidm.0.9.9 uutf.1.0.3 x509.0.16.5 yojson.2.2.2 zarith.1.14"
ENV CI="true"
ENV OCAMLCI="true"
RUN opam update --depexts && opam install --cli=2.3 --depext-only -y ocaml-ci.dev ocaml-ci-web.dev ocaml-ci-service.dev ocaml-ci-gitlab.dev ocaml-ci-client.dev ocaml-ci-api.dev $DEPS
RUN opam install $DEPS
COPY --chown=1000:1000 . /src
RUN opam exec -- dune build @install @check @runtest && rm -rf _build

END-OF-DOCKERFILE
docker build .

Note that the commented out block where we create dummy dune-project files is the current workaround we have hacked into ocaml-ci to prevent spurious CI failures for users.

Running the above produces the following errors during the final opam install:

313.3 #=== ERROR while compiling solver-service-api.dev =============================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/solver-service)
313.3 # path        ~/.opam/4.14/.opam-switch/build/solver-service-api.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/solver-service-api-7-831243.env
313.3 # output-file ~/.opam/log/solver-service-api-7-831243.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling ocluster-api.dev ===================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocluster)
313.3 # path        ~/.opam/4.14/.opam-switch/build/ocluster-api.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/ocluster-api-7-607c66.env
313.3 # output-file ~/.opam/log/ocluster-api-7-607c66.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling ocaml-version.dev ==================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocaml-version)
313.3 # path        ~/.opam/4.14/.opam-switch/build/ocaml-version.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/ocaml-version-7-b1c25b.env
313.3 # output-file ~/.opam/log/ocaml-version-7-b1c25b.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling obuilder-spec.dev ==================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocluster/obuilder)
313.3 # path        ~/.opam/4.14/.opam-switch/build/obuilder-spec.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/obuilder-spec-7-c37df3.env
313.3 # output-file ~/.opam/log/obuilder-spec-7-c37df3.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling dockerfile.dev =====================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocaml-dockerfile)
313.3 # path        ~/.opam/4.14/.opam-switch/build/dockerfile.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/dockerfile-7-554e0b.env
313.3 # output-file ~/.opam/log/dockerfile-7-554e0b.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling current_rpc.dev ====================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocurrent)
313.3 # path        ~/.opam/4.14/.opam-switch/build/current_rpc.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/current_rpc-7-89bbf3.env
313.3 # output-file ~/.opam/log/current_rpc-7-89bbf3.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling current.dev ========================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocurrent)
313.3 # path        ~/.opam/4.14/.opam-switch/build/current.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/current-7-aaf51c.env
313.3 # output-file ~/.opam/log/current-7-aaf51c.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
  • PR with a reproducing test: WIP

Specifications

  • Version of dune (output of dune --version): dune.3.17.0
  • Version of ocaml (output of ocamlc --version): ocaml.4.14.2
  • Operating system (distribution and version): all tested in ocaml-ci

Additional information
@shonfeder shonfeder changed the title dune subst no longer works on vendored deps without Dune 3.17 dune subst no longer works on vendored deps in ocaml-ci without Dune 3.17 Dec 12, 2024
shonfeder added a commit to shonfeder/dune that referenced this issue Dec 12, 2024
@shonfeder
Regrssion test for ocaml#11200
a223344
shonfeder added a commit to shonfeder/dune that referenced this issue Dec 12, 2024
@shonfeder
Regression test for ocaml#11200
8ecdda9
@shonfeder shonfeder mentioned this issue Dec 12, 2024
Merged
shonfeder added a commit to shonfeder/dune that referenced this issue Dec 12, 2024
@shonfeder
Regression test for ocaml#11200
5b54491
@shonfeder shonfeder changed the title dune subst no longer works on vendored deps in ocaml-ci without Dune 3.17 dune subst no longer works on enoty directories in ocaml-ci without Dune 3.17 Dec 13, 2024
@shonfeder shonfeder changed the title dune subst no longer works on enoty directories in ocaml-ci without Dune 3.17 dune subst no longer works on empty directories in ocaml-ci without Dune 3.17 Dec 13, 2024
shonfeder added a commit to shonfeder/dune that referenced this issue Dec 13, 2024
@shonfeder
Regression test for ocaml#11200
72fe9f3 
Signed-off-by: Shon Feder <[email protected]>
rgrinberg pushed a commit to shonfeder/dune that referenced this issue Dec 13, 2024
@shonfeder @rgrinberg
test: Regression test for ocaml#11200
ccef0be 
Signed-off-by: Shon Feder <[email protected]>
rgrinberg pushed a commit that referenced this issue Dec 13, 2024
@shonfeder
test: Regression test for #11200 (#11201)
38c0743 
Signed-off-by: Shon Feder <[email protected]>
@Leonidas-from-XIV
Copy link
Collaborator

Fixed by #11204.

maiste added a commit to maiste/opam-repository that referenced this issue Dec 17, 2024
@maiste
[new release] dune (15 packages) (3.17.1)
de2a709 
CHANGES:

### Fixed

- When a library declares `(no_dynlink)`, then the `.cmxs` file for it
  is no longer built. (ocaml/dune#11176, @nojb)

- Fix bug that could result in corrupted file copies by Dune, for example when
  using the `copy_files#` stanza or the `copy#` action. (@nojb, ocaml/dune#11194, fixes
  ocaml/dune#11193)

- Remove useless error message when running `$ dune subst` in empty projects.
  (@rgrinberg, ocaml/dune#11204, fixes ocaml/dune#11200)
@maiste maiste mentioned this issue Dec 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[new release] dune (15 packages) (3.17.1) maiste/opam-repository
2 participants
@Leonidas-from-XIV @shonfeder