Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to install from bootstrap script (package list fails to download from entware) #324

Closed
IvoryTinplate opened this issue Apr 2, 2021 · 19 comments

Comments

@IvoryTinplate
Copy link

Hello lovely people-

Many apologies if this has already been posted, but I couldn't find it.

I'm trying to install on my RM2 for the first time, using the bootstrap script, RM version 2.6.2.75. Here's the output:

bootstrap: OK
INFO:  Fetching secure wget
INFO:  Creating /home/root/.entware and mounting to /opt
INFO:  Installing Entware to /opt
failed: Network is unreachable.
failed: Network is unreachable.
2021-04-02 01:52:15 URL:https://bin.entware.net/armv7sf-k3.2/installer/opkg [752476/752476] -> "/opt/bin/opkg" [1]
failed: Network is unreachable.
failed: Network is unreachable.
2021-04-02 01:52:17 URL:https://bin.entware.net/armv7sf-k3.2/installer/opkg.conf [171/171] -> "/opt/etc/opkg.conf" [1]
failed: Network is unreachable.
failed: Network is unreachable.
2021-04-02 01:52:25 URL:https://bin.entware.net/armv7sf-k3.2/installer/ld-2.27.so [134428/134428] -> "/opt/lib/ld-2.27.so" [1]
failed: Network is unreachable.
failed: Network is unreachable.
2021-04-02 01:52:28 URL:https://bin.entware.net/armv7sf-k3.2/installer/libc-2.27.so [1247408/1247408] -> "/opt/lib/libc-2.27.so" [1]
failed: Network is unreachable.
failed: Network is unreachable.
2021-04-02 01:52:29 URL:https://bin.entware.net/armv7sf-k3.2/installer/libgcc_s.so.1 [50848/50848] -> "/opt/lib/libgcc_s.so.1" [1]
failed: Network is unreachable.
failed: Network is unreachable.
2021-04-02 01:52:31 URL:https://bin.entware.net/armv7sf-k3.2/installer/libpthread-2.27.so [92656/92656] -> "/opt/lib/libpthread-2.27.so" [1]
Downloading https://bin.entware.net/armv7sf-k3.2/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Installing entware-opt (227000-3) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/entware-opt_227000-3_all.ipk
Installing libgcc (8.4.0-11) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/libgcc_8.4.0-11_armv7-3.2.ipk
Installing libc (2.27-11) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/libc_2.27-11_armv7-3.2.ipk
Installing libssp (8.4.0-11) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/libssp_8.4.0-11_armv7-3.2.ipk
Installing libpthread (2.27-11) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/libpthread_2.27-11_armv7-3.2.ipk
Installing librt (2.27-11) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/librt_2.27-11_armv7-3.2.ipk
Installing libstdcpp (8.4.0-11) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/libstdcpp_8.4.0-11_armv7-3.2.ipk
Installing entware-release (1.0-2) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/entware-release_1.0-2_all.ipk
Installing zoneinfo-asia (2021a-1) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/zoneinfo-asia_2021a-1_armv7-3.2.ipk
Installing zoneinfo-europe (2021a-1) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/zoneinfo-europe_2021a-1_armv7-3.2.ipk
Installing findutils (4.7.0-3) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/findutils_4.7.0-3_armv7-3.2.ipk
Installing terminfo (6.2-1) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/terminfo_6.2-1_armv7-3.2.ipk
Installing libpcre (8.44-4) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/libpcre_8.44-4_armv7-3.2.ipk
Installing grep (3.6-1a) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/grep_3.6-1a_armv7-3.2.ipk
Installing locales (2.27-9) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/locales_2.27-9_armv7-3.2.ipk
Installing opkg (2020-12-24-9bbc7eae-1) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/opkg_2020-12-24-9bbc7eae-1_armv7-3.2.ipk
Installing entware-upgrade (1.0-1) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/entware-upgrade_1.0-1_all.ipk
Configuring libgcc.
Configuring libc.
Configuring libssp.
Configuring libpthread.
Configuring librt.
Configuring terminfo.
Configuring libpcre.
Configuring grep.
Configuring locales.
Entware uses separate locale-archive file independent from main system
Creating locale archive /opt/usr/lib/locale/locale-archive
Adding en_EN.UTF-8
Adding ru_RU.UTF-8
You can download locale sources from http://bin.entware.net/other/i18n_glib227.tar.gz
You can add new locales to Entware using /opt/bin/localedef.new
Configuring entware-upgrade.
Upgrade operations are not required.
Configuring opkg.
Configuring zoneinfo-europe.
Configuring zoneinfo-asia.
Configuring libstdcpp.
Configuring entware-release.
Configuring findutils.
Configuring entware-opt.
INFO:  Adding the Toltec stable repository
Downloading https://bin.entware.net/armv7sf-k3.2/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Downloading https://toltec-dev.org/stable/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/toltec
Installing ca-certificates (20210119-1) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/ca-certificates_20210119-1_all.ipk
Installing wget-nossl (1.21.1-1) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/wget-nossl_1.21.1-1_armv7-3.2.ipk
Installing zlib (1.2.11-3) to root...
Downloading https://bin.entware.net/armv7sf-k3.2/zlib_1.2.11-3_armv7-3.2.ipk
Configuring ca-certificates.
Configuring zlib.
Configuring wget-nossl.
Downloading https://bin.entware.net/armv7sf-k3.2/Packages.gz
*** Failed to download the package list from https://bin.entware.net/armv7sf-k3.2/Packages.gz

Downloading https://toltec-dev.org/stable/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/toltec
Collected errors:
 * opkg_download: Failed to download https://bin.entware.net/armv7sf-k3.2/Packages.gz, wget returned 8.
ERROR: Unexpected error on line bootstrap:251 in function toltec-install
ERROR: This script failed to install. If you can't solve the above
       issue yourself, please report it at:

Internet access from the remarkable seems fine, and I can download the entware package list from other devices on my LAN.

@matteodelabre
Copy link
Member

Hi @IvoryTinplate! Thanks for your report.

This unrelated to your issue, but please note that Toltec support for version 2.6 is not yet complete (see #322).

In the output you posted, wget seems unable to fetch resources from the network. Could you try running the following command manually to see if it reports more detailed information?

$ /usr/bin/wget https://example.com

The command above will test the system wget, which is only used as a bootstrap to fetch another wget that is actually used for the rest of the install. (This is because the default wget lacks TLS support.) Could you also check that Toltec’s wget works? You can try the following commands to verify that:

$ /usr/bin/wget https://toltec-dev.org/thirdparty/bin/wget-v1.21.1
$ echo "8798fcdabbe560722a02f95b30385926e4452e2c98c15c2c217583eaa0db30fc  wget-v1.21.1" | sha256sum -c
$ chmod u+x wget-v1.21.1
$ ./wget-v1.21.1 https://example.com

Thanks!

@matteodelabre matteodelabre added the pending Further information is requested label Apr 4, 2021
@briankaemingk
Copy link

Hi @matteodelabre thanks for all your hard work on this. Appears that I'm getting the same error as @IvoryTinplate. Please see below for the output when I tried your commands...

reMarkable: ~/ opkg update
Downloading https://bin.entware.net/armv7sf-k3.2/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
Downloading https://toltec-dev.org/testing/Packages.gz
*** Failed to download the package list from https://toltec-dev.org/testing/Packages.gz

Collected errors:
 * opkg_download: Failed to download https://toltec-dev.org/testing/Packages.gz, wget returned 8.
reMarkable: ~/ /usr/bin/wget https://example.com
Connecting to example.com (93.184.216.34:443)
wget: note: TLS certificate validation not implemented
saving to 'index.html'
index.html           100% |********************************|  1256  0:00:00 ETA
'index.html' saved
reMarkable: ~/ rm index.html 
reMarkable: ~/ /usr/bin/wget https://toltec-dev.org/thirdparty/bin/wget-v1.21.1
Connecting to toltec-dev.org (172.104.246.107:443)
wget: note: TLS certificate validation not implemented
saving to 'wget-v1.21.1'
wget-v1.21.1         100% |********************************| 3042k  0:00:00 ETA
'wget-v1.21.1' saved
reMarkable: ~/ echo "8798fcdabbe560722a02f95b30385926e4452e2c98c15c2c217583eaa0db30fc  wget-v1.21.1" | sha256sum -c
wget-v1.21.1: OK
reMarkable: ~/  chmod u+x wget-v1.21.1
reMarkable: ~/ ./wget-v1.21.1 https://example.com
--2021-04-05 16:13:36--  https://example.com/
Resolving example.com... 2606:2800:220:1:248:1893:25c8:1946, 93.184.216.34
Connecting to example.com|2606:2800:220:1:248:1893:25c8:1946|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1256 (1.2K) [text/html]
Saving to: 'index.html'

index.html          100%[===================>]   1.23K  --.-KB/s    in 0s      

2021-04-05 16:13:37 (5.56 MB/s) - 'index.html' saved [1256/1256]

@matteodelabre
Copy link
Member

Hi @briankaemingk. Based on the output you provided, I think you’re seeing a separate issue. It appears that you’re using the testing branch, whose directory layout has recently been changed (see #310). To fix your issue, you have two choices:

  • Revert back to the stable branch, which still has the old repository layout. When we move the new layout to the stable branch, it will have an automatic transition mechanism in place so you won’t have to edit your configuration again. In general, the testing branch is only intended for devs to test packages and features more thoroughly before they land in stable. So if you wish to avoid similar issues in the future, I would advise moving back to the stable branch (at the cost of having slightly older packages). To do this change, you need to edit /opt/etc/opkg.conf on your device, replace each occurrence of testing with stable, then run opkg update && opkg upgrade.

  • Update your configuration for the new repository layout in testing. To do this, you need to download the updated bootstrap script (not the one hosted at https://toltec-dev.org/bootstrap) and run it on your device with bash bootstrap. The script will detect your existing install and automatically update its configuration.

All of this does not apply to @IvoryTinplate since their logs show that they’re using the stable branch.

@briankaemingk
Copy link

briankaemingk commented Apr 5, 2021 via email

@matteodelabre
Copy link
Member

We’re still working on 2.6 support—you can track its status in #322. Currently, most apps using the Qt framework (e.g., draft) will not work on 2.6 unless recompiled using the 2.x version of our toolchain. But on the flip side, apps compiled with this version of the toolchain will not work on older system versions such as 2.5, so it’s not simply a matter of updating all of our the packages to use the new toolchain, since that would break support for users who have not received the update yet.

@briankaemingk
Copy link

briankaemingk commented Apr 5, 2021 via email

@matteodelabre
Copy link
Member

Here are some instructions to compile draft with toolchain v2.0.1. Make sure to install on your computer all the required dependencies listed here first.

  • Clone this Git repository.
  • Switch to the testing branch.
  • Edit the package/draft/package file, change image=qt:v1.4 to image=qt:v2.0.1.
  • Run make draft.
  • The resulting package can be found in build/repo/rmall/draft_…_rmall.ipk—copy it to your device.
  • On your device, run opkg install path/to/draft_…_rmall.ipk (uninstall draft first if it’s already installed).

@briankaemingk
Copy link

Thank you! And forgive me for taking over this thread, but this is really helpful, hoping I can help pay it forward and contribute in the future..

Here's what I'm getting after verifying I've installed all the dependencies (docker, bsdtar, python, and the requirements.txt python files)...

(env) (base) briankaemingk@brians-MacBook-Pro toltec % make draft
./scripts/package_build.py  "draft"
Traceback (most recent call last):
  File "./scripts/package_build.py", line 44, in <module>
    repo = Repo(paths.RECIPE_DIR, paths.REPO_DIR)
  File "/Users/briankaemingk/Documents/toltec/scripts/toltec/repo.py", line 56, in __init__
    self.generic_recipes[name] = GenericRecipe.from_file(
  File "/Users/briankaemingk/Documents/toltec/scripts/toltec/recipe.py", line 45, in from_file
    return GenericRecipe(name, path, recipe.read())
  File "/Users/briankaemingk/Documents/toltec/scripts/toltec/recipe.py", line 58, in __init__
    variables, functions = bash.get_declarations(definition)
  File "/Users/briankaemingk/Documents/toltec/scripts/toltec/bash.py", line 144, in get_declarations
    assert next_token == "("
AssertionError
make: *** [draft] Error 1

@matteodelabre
Copy link
Member

No problem. So this looks like an error with our Bash parser. Can you please share your Bash version and Python version?

@briankaemingk
Copy link

GNU bash, version 5.0.18(1)-release (x86_64-apple-darwin19.6.0)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
(env) (base) briankaemingk@brians-MacBook-Pro toltec % python --version  
Python 3.8.5

@matteodelabre
Copy link
Member

Unfortunately I can’t reproduce this using the same versions as you, but I’m not on macOS so that might be why. Could you please insert a print(declarations) statement between lines 122 and 123 in scripts/toltec/bash.py and share the output so that we can see exactly what is causing the parser to fail? (The output should normally be free of personal info, but please double check before posting.) Thanks!

@briankaemingk
Copy link

Happy to, please see below.. I was also able to successfully run your make instructions on ubuntu, so I have a working copy of draft on my rm2, thanks again!

Let me know what else I can do to help debug the parser issue..

toltec % make draft                
./scripts/package_build.py  "draft"
build () 
{ 
    rm .cargo/config;
    cargo build --release
}
package () 
{ 
    install -D -m 755 -t "$pkgdir"/opt/bin "$srcdir"/target/armv7-unknown-linux-gnueabihf/release/retris;
    install -D -m 644 "$srcdir"/oxide "$pkgdir"/opt/etc/draft/retris;
    install -D -m 644 "$srcdir"/icon.png "$pkgdir"/opt/etc/draft/icons/retris.png
}
BASH=/usr/local/bin/bash
BASH_ARGC=()
BASH_ARGV=()
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="3" [1]="2" [2]="57" [3]="1" [4]="release" [5]="x86_64-apple-darwin20")
BASH_VERSION='3.2.57(1)-release'
DIRSTACK=()
EUID=501
GROUPS=()
HOSTNAME=<HOSTNAME>
HOSTTYPE=x86_64
IFS=$' \t\n'
MACHTYPE=x86_64-apple-darwin20
OPTERR=1
OPTIND=1
OSTYPE=darwin20
PATH=/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.
PIPESTATUS=([0]="0")
PPID=59039
PS4='+ '
PWD=/Users/<USER>/Documents/toltec
SHELL=/bin/zsh
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
TERM=dumb
UID=501
_=-f
flags=([0]="patch_rm2fb")
image=rust:v1.4
installdepends=([0]="display")
license=MIT
maintainer='Linus K. <[email protected]>'
pkgdesc='Tetris game'
pkgnames=([0]="retris")
pkgver=0.6.3-2
section=games
sha256sums=([0]="ecc7215098c03e79cd92b1835626e6739a5a932d5aa709899d183347e2a4108e")
source=([0]="https://github.com/LinusCDE/retris/archive/0.6.3-1.zip")
timestamp=2021-01-30T02:41Z
url=https://github.com/LinusCDE/retris
build () 
{ 
    rm .cargo/config;
    cargo build --release
}
package () 
{ 
    install -D -m 755 -t "$pkgdir"/opt/bin "$srcdir"/target/armv7-unknown-linux-gnueabihf/release/retris;
    install -D -m 644 "$srcdir"/oxide "$pkgdir"/opt/etc/draft/retris;
    install -D -m 644 "$srcdir"/icon.png "$pkgdir"/opt/etc/draft/icons/retris.png
}

Traceback (most recent call last):
  File "./scripts/package_build.py", line 44, in <module>
    repo = Repo(paths.RECIPE_DIR, paths.REPO_DIR)
  File "/Users/<USER>/Documents/toltec/scripts/toltec/repo.py", line 56, in __init__
    self.generic_recipes[name] = GenericRecipe.from_file(
  File "/Users/<USER>/Documents/toltec/scripts/toltec/recipe.py", line 45, in from_file
    return GenericRecipe(name, path, recipe.read())
  File "/Users/<USER>/Documents/toltec/scripts/toltec/recipe.py", line 58, in __init__
    variables, functions = bash.get_declarations(definition)
  File "/Users/<USER>/Documents/toltec/scripts/toltec/bash.py", line 145, in get_declarations
    assert next_token == "("
AssertionError
make: *** [draft] Error 1

@matteodelabre
Copy link
Member

Thanks for sharing more details. Your output says BASH_VERSION='3.2.57(1)-release' and BASH=/usr/local/bin/bash, which is different from the version information you provided above. Could it be that there are two different versions of Bash on your system and that Toltec picks up the older one?

@briankaemingk
Copy link

Very strange indeed... a which bash gives me the expected later version...

/usr/local/bin/bash
% bash --version
GNU bash, version 5.0.18(1)-release (x86_64-apple-darwin19.6.0)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

@matteodelabre
Copy link
Member

This may be because our script launches Bash subprocesses with empty environments (in particular with empty PATHs). So if you have customized your PATH to point to a non-default location, the Toltec script may end up launching a different Bash version. To try preserving the PATH value, can you try changing line 97 of scripts/toltec/bash.py from

    env: Dict[str, str] = {}

to

    import os
    env: Dict[str, str] = {"PATH": os.environ["PATH"]}

@briankaemingk
Copy link

briankaemingk commented Apr 8, 2021 via email

matteodelabre added a commit that referenced this issue Apr 8, 2021
Our current Python code spawns a Bash process as part of the recipe
parsing process. To make sure the calling environment does not influence
how the recipe is parsed, the subprocess is created with a clean
environment. Unfortunately, this means the PATH variable is cleared and
therefore that the subprocess may use a Bash binary different from the
one in the user’s PATH. This PR changes that behavior and forwards the
PATH value to the subprocess.

See the following comment and its follow-ups for more context:
<#324 (comment)>

Test plan: Added a dummy `bash` binary (a file containing only
`#!/usr/bin/env false`) to my PATH and checked that the recipe parsing
fails (indicating that it invokes the dummy Bash and not the system
one). Without the current PR, the parsing succeeds even if the parent
PATH points to the dummy Bash.
@matteodelabre
Copy link
Member

@briankaemingk Thanks for helping me troubleshoot this! I opened the PR mentioned above to fix this in our codebase.

@IvoryTinplate Sorry that we used your thread to diagnose a different issue. If your issue still persists, could you please provide more details as per my comment above (#324 (comment))?

matteodelabre added a commit that referenced this issue Apr 9, 2021
Our current Python code spawns a Bash process as part of the recipe parsing process. To make sure the calling environment does not influence how the recipe is parsed, the subprocess is created with a clean environment. Unfortunately, this means the PATH variable is cleared and therefore that the subprocess may use a Bash binary different from the one in the user’s PATH. This PR changes that behavior and forwards the PATH value to the subprocess.

See the following comment and its follow-ups for more context: <#324 (comment)>

Test plan: Added a dummy `bash` binary (a file containing only `#!/usr/bin/env false`) to my PATH and checked that the recipe parsing fails (indicating that it invokes the dummy Bash and not the system one). Without the current PR, the parsing succeeds even if the parent PATH points to the dummy Bash.
danshick pushed a commit to danshick/toltec that referenced this issue May 5, 2021
Our current Python code spawns a Bash process as part of the recipe parsing process. To make sure the calling environment does not influence how the recipe is parsed, the subprocess is created with a clean environment. Unfortunately, this means the PATH variable is cleared and therefore that the subprocess may use a Bash binary different from the one in the user’s PATH. This PR changes that behavior and forwards the PATH value to the subprocess.

See the following comment and its follow-ups for more context: <toltec-dev#324 (comment)>

Test plan: Added a dummy `bash` binary (a file containing only `#!/usr/bin/env false`) to my PATH and checked that the recipe parsing fails (indicating that it invokes the dummy Bash and not the system one). Without the current PR, the parsing succeeds even if the parent PATH points to the dummy Bash.
LinusCDE pushed a commit that referenced this issue Jun 3, 2021
Our current Python code spawns a Bash process as part of the recipe parsing process. To make sure the calling environment does not influence how the recipe is parsed, the subprocess is created with a clean environment. Unfortunately, this means the PATH variable is cleared and therefore that the subprocess may use a Bash binary different from the one in the user’s PATH. This PR changes that behavior and forwards the PATH value to the subprocess.

See the following comment and its follow-ups for more context: <#324 (comment)>

Test plan: Added a dummy `bash` binary (a file containing only `#!/usr/bin/env false`) to my PATH and checked that the recipe parsing fails (indicating that it invokes the dummy Bash and not the system one). Without the current PR, the parsing succeeds even if the parent PATH points to the dummy Bash.
@matteodelabre
Copy link
Member

@IvoryTinplate Any update?

@matteodelabre
Copy link
Member

I’m closing this issue for now. Please feel free to re-open it if the same issue happens again.

@matteodelabre matteodelabre removed the pending Further information is requested label Jun 18, 2021
matteodelabre added a commit to toltec-dev/build that referenced this issue Jul 31, 2021
Our current Python code spawns a Bash process as part of the recipe parsing process. To make sure the calling environment does not influence how the recipe is parsed, the subprocess is created with a clean environment. Unfortunately, this means the PATH variable is cleared and therefore that the subprocess may use a Bash binary different from the one in the user’s PATH. This PR changes that behavior and forwards the PATH value to the subprocess.

See the following comment and its follow-ups for more context: <toltec-dev/toltec#324 (comment)>

Test plan: Added a dummy `bash` binary (a file containing only `#!/usr/bin/env false`) to my PATH and checked that the recipe parsing fails (indicating that it invokes the dummy Bash and not the system one). Without the current PR, the parsing succeeds even if the parent PATH points to the dummy Bash.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants