Hitting Tokio panic during encapsulation

[jlebon]

Hit this while building latest RHCOS locally:

$ RUST_BACKTRACE=1 rpm-ostree container-encapsulate --max-layers=50 --format-version=1 \
    --repo=/run/host/var/srv/rhcos/rhcos-unpriv/tmp/repo \
    --label=coreos-assembler.image-config-checksum=be1ef79777533f7d21945ce0faf1edaca80e408fe0824193d01ab6c1effaa852 \
    --label=coreos-assembler.image-input-checksum=5c60d23211d19183058f453c5c357c85e15204572891f1233e0c35d5d4bf2ae3 \
    --label=org.opencontainers.image.source=https://github.com/openshift/os \
    --label=org.opencontainers.image.revision=v0.0.2-782-g05f485bf94830b03beba309357a861fdbb9debe0 \
    412.86.202208251815-0 \
    oci-archive:rhcos-412.86.202208251815-0-ostree.x86_64.ociarchive.tmp:latest
warning: This entrypoint is deprecated; use `rpm-ostree compose container-encapsulate` instead
warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend.
25020 objects in 504 packages (340 source)
rpm size: 2616705560
Earliest changed package: grep-3.1-6.el8.x86_64 at 2018-08-12 07:50:42 UTC
1721 duplicates
Multiple owners:
  /usr/lib/.build-id/58/c5bafd304b4adee2ab63b1a8c41a7ef5338994
  /usr/lib/.build-id/72/b331b204a4dfb0ba490b895302cb51054711ae
  /usr/lib/.build-id/8b/e949ccb51866e0e8e039704e245c04bdb96a78
  /usr/lib/.build-id/90/99a855bc827b3075231463350599e124ef80f5
  /usr/lib/.build-id/c1/b65e0f84638eb5a8a98a60ef78c3fd9704540b
  /usr/lib/.build-id/e2/f667d8e32cdee5ca63471613be8f9c57033128
  /usr/lib/.build-id/f3/b87780af9bc8750df3c34ee6c38a043f9ccd75
  /usr/lib/.build-id/f6/1fbfeb41d733f8325276e4fd4b70f92a8ddac3
  /usr/lib64/libgcc_s-8-20210514.so.1
  /usr/lib64/libgcc_s.so.1
  /usr/share/licenses/geolite2/COPYRIGHT.txt
  /usr/share/licenses/geolite2/LICENSE.txt
  /usr/share/licenses/libgcc/COPYING
  /usr/share/licenses/libgcc/COPYING.LIB
  /usr/share/licenses/libgcc/COPYING.RUNTIME
  /usr/share/licenses/libgcc/COPYING3
  /usr/share/licenses/libgcc/COPYING3.LIB
  /usr/share/licenses/systemd/LICENSE.LGPL2.1
thread 'main' panicked at 'Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.', /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/enter.rs:39:9
stack backtrace:
   0: std::panicking::begin_panic
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/std/src/panicking.rs:616:12
   1: tokio::runtime::enter::enter
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/enter.rs:39:9
   2: tokio::runtime::handle::Handle::block_on
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/handle.rs:352:34
   3: rpmostree_rust::container::container_encapsulate
             at /var/home/jlebon/Code/github.com/coreos/rpm-ostree/rust/src/container.rs:388:18
   4: rpm_ostree::inner_async_main::{{closure}}
             at /var/home/jlebon/Code/github.com/coreos/rpm-ostree/rust/src/main.rs:38:13
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/future/mod.rs:91:19
   6: rpm_ostree::dispatch_multicall::{{closure}}
             at /var/home/jlebon/Code/github.com/coreos/rpm-ostree/rust/src/main.rs:77:36
   7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/future/mod.rs:91:19
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/future/future.rs:124:9
   9: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/basic_scheduler.rs:516:48
  10: tokio::coop::with_budget::{{closure}}
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/coop.rs:102:9
  11: std::thread::local::LocalKey<T>::try_with
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/std/src/thread/local.rs:445:16
  12: std::thread::local::LocalKey<T>::with
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/std/src/thread/local.rs:421:9
  13: tokio::coop::with_budget
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/coop.rs:95:5
  14: tokio::coop::budget
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/coop.rs:72:5
  15: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}::{{closure}}
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/basic_scheduler.rs:516:25
  16: tokio::runtime::basic_scheduler::Context::enter
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/basic_scheduler.rs:374:19
  17: tokio::runtime::basic_scheduler::CoreGuard::block_on::{{closure}}
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/basic_scheduler.rs:515:36
  18: tokio::runtime::basic_scheduler::CoreGuard::enter::{{closure}}
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/basic_scheduler.rs:582:57
  19: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/macros/scoped_tls.rs:61:9
  20: tokio::runtime::basic_scheduler::CoreGuard::enter
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/basic_scheduler.rs:582:27
  21: tokio::runtime::basic_scheduler::CoreGuard::block_on
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/basic_scheduler.rs:506:9
  22: tokio::runtime::basic_scheduler::BasicScheduler::block_on
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/basic_scheduler.rs:182:24
  23: tokio::runtime::Runtime::block_on
             at /var/home/jlebon/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.16.1/src/runtime/mod.rs:475:46
  24: rpm_ostree::inner_main
             at /var/home/jlebon/Code/github.com/coreos/rpm-ostree/rust/src/main.rs:144:5
  25: rpm_ostree::main
             at /var/home/jlebon/Code/github.com/coreos/rpm-ostree/rust/src/main.rs:167:13
  26: core::ops::function::FnOnce::call_once
             at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[cgwalters]

I think this is fallout from #3904 - in the legacy path we're still calling it a native async method, but it isn't anymore and needs to be run from the worker thread.

[jlebon]

Forgot to mention, this was with git main.
v2022.12 works fine.

So something in v2022.12...main.

[cgwalters]

Should work to use the new rpm-ostree compose container-encapsulate; let's cut over cosa to detect and use that.

[cgwalters]

It's worth noting how this snuck through CI:

• For rawhide and openshift/os master which do use the chunked path, we use stable rpm-ostree in coreos-assembler
• We do have CI in this repo for running this, but when I renamed the CLI I also switched our test to use the new entrypoint; I'd only sanity tested --help, thinking there wasn't really any way anything could break other than that. But clearly, I was wrong.

I think this was a one-off; once we pull the trigger on coreos/coreos-assembler#3014 we can pretty quickly delete the old entrypoint and hence the possibility of any bugs like this.