Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The number of packages downloaded from the remote source is inconsistent with that of the local source. #2016

Closed
weli-l opened this issue Nov 13, 2023 · 19 comments

Comments

@weli-l
Copy link

weli-l commented Nov 13, 2023

When I configure both remote and local sources (where the rpm package for the local source is downloaded from the remote source using dnf download *) . When the same software package list is installed, the specified remote source cannot download weak dependencies, but the specified local source can download weak dependencies. The number of displayed software package lists is different. The number of software package lists using the remote source is 424, but the number of software package lists using the local source is 449. The missing 25 are weak dependencies.

dnf install --installroot /usr1/k_root/ [package list]

dnf install -c local.repo --installroot /usr1/k_root/ [package list]

@weli-l
Copy link
Author

weli-l commented Nov 13, 2023

In addition, I found that even if I use the -c command to specify local resources, dnf will still look for it in the remote source

@jan-kolarik
Copy link
Member

Hello, I am not sure what do you want to achieve with the -c local.repo parameter. For running the dnf using the locally cached metadata only, you have to pass the capitalized -C option. If you have specified the local repository in the local.repo file, you can configure dnf to run using only this repository f.e. with the --repofrompath and --repo parameters, try looking in the dnf man pages.

@weli-l
Copy link
Author

weli-l commented Nov 13, 2023

Hello, I am not sure what do you want to achieve with the -c local.repo parameter. For running the dnf using the locally cached metadata only, you have to pass the capitalized -C option. If you have specified the local repository in the local.repo file, you can configure dnf to run using only this repository f.e. with the --repofrompath and --repo parameters, try looking in the dnf man pages.

Thank you for your reply. local.repo is the configuration file I created locally. I created another folder localRPMS locally to store all the rpm packages, and then I used -c local.repo to download files from the localRPMS folder. But this way I was able to download the number of rpm packages that I was able to download unlike using a remote source, which had fewer weak dependency packages. If I use -C, then the resource is still downloaded from the remote source.

@jan-kolarik
Copy link
Member

Hello, I am not sure what do you want to achieve with the -c local.repo parameter. For running the dnf using the locally cached metadata only, you have to pass the capitalized -C option. If you have specified the local repository in the local.repo file, you can configure dnf to run using only this repository f.e. with the --repofrompath and --repo parameters, try looking in the dnf man pages.

Thank you for your reply. local.repo is the configuration file I created locally. I created another folder localRPMS locally to store all the rpm packages, and then I used -c local.repo to download files from the localRPMS folder. But this way I was able to download the number of rpm packages that I was able to download unlike using a remote source, which had fewer weak dependency packages. If I use -C, then the resource is still downloaded from the remote source.

I see, if -c option is used to pass to specify a configuration from a different path, the repositories configuration is still loaded from the reposdir path (by default "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d"), so packages from other repositories are taken into account. This could be modified f.e. using the --disablerepo parameter.

Additionally, I observed that when running the first command you provided, the default /etc/dnf/dnf.conf is used. However, in the second command with the -c option, you are overriding it by using the local.repo. Depending on the configuration in /etc/dnf/dnf.conf, this might result in different behavior in both cases.

@weli-l
Copy link
Author

weli-l commented Nov 13, 2023

Hello, I am not sure what do you want to achieve with the -c local.repo parameter. For running the dnf using the locally cached metadata only, you have to pass the capitalized -C option. If you have specified the local repository in the local.repo file, you can configure dnf to run using only this repository f.e. with the --repofrompath and --repo parameters, try looking in the dnf man pages.

Thank you for your reply. local.repo is the configuration file I created locally. I created another folder localRPMS locally to store all the rpm packages, and then I used -c local.repo to download files from the localRPMS folder. But this way I was able to download the number of rpm packages that I was able to download unlike using a remote source, which had fewer weak dependency packages. If I use -C, then the resource is still downloaded from the remote source.

I see, if -c option is used to pass to specify a configuration from a different path, the repositories configuration is still loaded from the reposdir path (by default "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d"), so packages from other repositories are taken into account. This could be modified f.e. using the --disablerepo parameter.

Additionally, I observed that when running the first command you provided, the default /etc/dnf/dnf.conf is used. However, in the second command with the -c option, you are overriding it by using the local.repo. Depending on the configuration in /etc/dnf/dnf.conf, this might result in different behavior in both cases.

Yes, when I run these two commands in turn, executing the first command does not download weak dependencies, but the second command does. In fact, these weak dependency packages are already downloaded in my environment.

@jan-kolarik
Copy link
Member

jan-kolarik commented Nov 13, 2023

Hello, I am not sure what do you want to achieve with the -c local.repo parameter. For running the dnf using the locally cached metadata only, you have to pass the capitalized -C option. If you have specified the local repository in the local.repo file, you can configure dnf to run using only this repository f.e. with the --repofrompath and --repo parameters, try looking in the dnf man pages.

Thank you for your reply. local.repo is the configuration file I created locally. I created another folder localRPMS locally to store all the rpm packages, and then I used -c local.repo to download files from the localRPMS folder. But this way I was able to download the number of rpm packages that I was able to download unlike using a remote source, which had fewer weak dependency packages. If I use -C, then the resource is still downloaded from the remote source.

I see, if -c option is used to pass to specify a configuration from a different path, the repositories configuration is still loaded from the reposdir path (by default "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d"), so packages from other repositories are taken into account. This could be modified f.e. using the --disablerepo parameter.
Additionally, I observed that when running the first command you provided, the default /etc/dnf/dnf.conf is used. However, in the second command with the -c option, you are overriding it by using the local.repo. Depending on the configuration in /etc/dnf/dnf.conf, this might result in different behavior in both cases.

Yes, when I run these two commands in turn, executing the first command does not download weak dependencies, but the second command does. In fact, these weak dependency packages are already downloaded in my environment.

So, could you confirm whether the install_weak_deps option in /etc/dnf/dnf.conf is not set to False? Or provide the whole configuration file? The default value, if not specified by the user or configuration, is True, and this could potentially explain your situation.

@weli-l
Copy link
Author

weli-l commented Nov 13, 2023

Hello, I am not sure what do you want to achieve with the -c local.repo parameter. For running the dnf using the locally cached metadata only, you have to pass the capitalized -C option. If you have specified the local repository in the local.repo file, you can configure dnf to run using only this repository f.e. with the --repofrompath and --repo parameters, try looking in the dnf man pages.

Thank you for your reply. local.repo is the configuration file I created locally. I created another folder localRPMS locally to store all the rpm packages, and then I used -c local.repo to download files from the localRPMS folder. But this way I was able to download the number of rpm packages that I was able to download unlike using a remote source, which had fewer weak dependency packages. If I use -C, then the resource is still downloaded from the remote source.

I see, if -c option is used to pass to specify a configuration from a different path, the repositories configuration is still loaded from the reposdir path (by default "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d"), so packages from other repositories are taken into account. This could be modified f.e. using the --disablerepo parameter.
Additionally, I observed that when running the first command you provided, the default /etc/dnf/dnf.conf is used. However, in the second command with the -c option, you are overriding it by using the local.repo. Depending on the configuration in /etc/dnf/dnf.conf, this might result in different behavior in both cases.

Yes, when I run these two commands in turn, executing the first command does not download weak dependencies, but the second command does. In fact, these weak dependency packages are already downloaded in my environment.

So, could you confirm whether the install_weak_deps option in /etc/dnf/dnf.conf is not set to False? Or provide the whole configuration file? The default value, if not specified by the user or configuration, is True, and this could potentially explain your situation
/

these are my configure
local.repo
[xxx]
name=xxx
baseurl=/root/downrpms
enabled=1
gpgcheck=0
priority=1

/etc/dnf/dnf.conf
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=True
skip_if_unavailable=False
sslverify=False

/etc/yum.repos.d/remote.repo
[1xxx]
name=1xxx
baseurl=https://xxx
enabled=1
gpgcheck=0
priority=10

@jan-kolarik
Copy link
Member

Hello, I am not sure what do you want to achieve with the -c local.repo parameter. For running the dnf using the locally cached metadata only, you have to pass the capitalized -C option. If you have specified the local repository in the local.repo file, you can configure dnf to run using only this repository f.e. with the --repofrompath and --repo parameters, try looking in the dnf man pages.

Thank you for your reply. local.repo is the configuration file I created locally. I created another folder localRPMS locally to store all the rpm packages, and then I used -c local.repo to download files from the localRPMS folder. But this way I was able to download the number of rpm packages that I was able to download unlike using a remote source, which had fewer weak dependency packages. If I use -C, then the resource is still downloaded from the remote source.

I see, if -c option is used to pass to specify a configuration from a different path, the repositories configuration is still loaded from the reposdir path (by default "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d"), so packages from other repositories are taken into account. This could be modified f.e. using the --disablerepo parameter.
Additionally, I observed that when running the first command you provided, the default /etc/dnf/dnf.conf is used. However, in the second command with the -c option, you are overriding it by using the local.repo. Depending on the configuration in /etc/dnf/dnf.conf, this might result in different behavior in both cases.

Yes, when I run these two commands in turn, executing the first command does not download weak dependencies, but the second command does. In fact, these weak dependency packages are already downloaded in my environment.

So, could you confirm whether the install_weak_deps option in /etc/dnf/dnf.conf is not set to False? Or provide the whole configuration file? The default value, if not specified by the user or configuration, is True, and this could potentially explain your situation
/

these are my configure local.repo [xxx] name=xxx baseurl=/root/downrpms enabled=1 gpgcheck=0 priority=1

/etc/dnf/dnf.conf [main] gpgcheck=1 installonly_limit=3 clean_requirements_on_remove=True best=True skip_if_unavailable=False sslverify=False

/etc/yum.repos.d/remote.repo [1xxx] name=1xxx baseurl=https://xxx enabled=1 gpgcheck=0 priority=10

OK, I see another difference of your two provided use cases. The first one is running with best=True, the second one with best=False as the /etc/dnf/dnf.conf is not being applied. Could you try running the second use case with --best option and compare the transaction packages then?

@weli-l
Copy link
Author

weli-l commented Nov 13, 2023

Hello, I am not sure what do you want to achieve with the -c local.repo parameter. For running the dnf using the locally cached metadata only, you have to pass the capitalized -C option. If you have specified the local repository in the local.repo file, you can configure dnf to run using only this repository f.e. with the --repofrompath and --repo parameters, try looking in the dnf man pages.

Thank you for your reply. local.repo is the configuration file I created locally. I created another folder localRPMS locally to store all the rpm packages, and then I used -c local.repo to download files from the localRPMS folder. But this way I was able to download the number of rpm packages that I was able to download unlike using a remote source, which had fewer weak dependency packages. If I use -C, then the resource is still downloaded from the remote source.

I see, if -c option is used to pass to specify a configuration from a different path, the repositories configuration is still loaded from the reposdir path (by default "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d"), so packages from other repositories are taken into account. This could be modified f.e. using the --disablerepo parameter.
Additionally, I observed that when running the first command you provided, the default /etc/dnf/dnf.conf is used. However, in the second command with the -c option, you are overriding it by using the local.repo. Depending on the configuration in /etc/dnf/dnf.conf, this might result in different behavior in both cases.

Yes, when I run these two commands in turn, executing the first command does not download weak dependencies, but the second command does. In fact, these weak dependency packages are already downloaded in my environment.

So, could you confirm whether the install_weak_deps option in /etc/dnf/dnf.conf is not set to False? Or provide the whole configuration file? The default value, if not specified by the user or configuration, is True, and this could potentially explain your situation
/

these are my configure local.repo [xxx] name=xxx baseurl=/root/downrpms enabled=1 gpgcheck=0 priority=1
/etc/dnf/dnf.conf [main] gpgcheck=1 installonly_limit=3 clean_requirements_on_remove=True best=True skip_if_unavailable=False sslverify=False
/etc/yum.repos.d/remote.repo [1xxx] name=1xxx baseurl=https://xxx enabled=1 gpgcheck=0 priority=10

OK, I see another difference of your two provided use cases. The first one is running with best=True, the second one with best=False as the /etc/dnf/dnf.conf is not being applied. Could you try running the second use case with --best option and compare the transaction packages then?

when I use dnf install -b -c local.repo --installroot /usr1/k_root/ [package list]
problem is still alive

@jan-kolarik
Copy link
Member

Hello, I am not sure what do you want to achieve with the -c local.repo parameter. For running the dnf using the locally cached metadata only, you have to pass the capitalized -C option. If you have specified the local repository in the local.repo file, you can configure dnf to run using only this repository f.e. with the --repofrompath and --repo parameters, try looking in the dnf man pages.

Thank you for your reply. local.repo is the configuration file I created locally. I created another folder localRPMS locally to store all the rpm packages, and then I used -c local.repo to download files from the localRPMS folder. But this way I was able to download the number of rpm packages that I was able to download unlike using a remote source, which had fewer weak dependency packages. If I use -C, then the resource is still downloaded from the remote source.

I see, if -c option is used to pass to specify a configuration from a different path, the repositories configuration is still loaded from the reposdir path (by default "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d"), so packages from other repositories are taken into account. This could be modified f.e. using the --disablerepo parameter.
Additionally, I observed that when running the first command you provided, the default /etc/dnf/dnf.conf is used. However, in the second command with the -c option, you are overriding it by using the local.repo. Depending on the configuration in /etc/dnf/dnf.conf, this might result in different behavior in both cases.

Yes, when I run these two commands in turn, executing the first command does not download weak dependencies, but the second command does. In fact, these weak dependency packages are already downloaded in my environment.

So, could you confirm whether the install_weak_deps option in /etc/dnf/dnf.conf is not set to False? Or provide the whole configuration file? The default value, if not specified by the user or configuration, is True, and this could potentially explain your situation
/

these are my configure local.repo [xxx] name=xxx baseurl=/root/downrpms enabled=1 gpgcheck=0 priority=1
/etc/dnf/dnf.conf [main] gpgcheck=1 installonly_limit=3 clean_requirements_on_remove=True best=True skip_if_unavailable=False sslverify=False
/etc/yum.repos.d/remote.repo [1xxx] name=1xxx baseurl=https://xxx enabled=1 gpgcheck=0 priority=10

OK, I see another difference of your two provided use cases. The first one is running with best=True, the second one with best=False as the /etc/dnf/dnf.conf is not being applied. Could you try running the second use case with --best option and compare the transaction packages then?

when I use dnf install -b -c local.repo --installroot /usr1/k_root/ [package list] problem is still alive

It might be useful to provide also the output of both commands used. Also with the second command I guess you want to use only the local repository, right? Then as described above, you need to disable the remote repositories. Or use --repo=xxx with the second command.

@weli-l
Copy link
Author

weli-l commented Nov 13, 2023

Because of Information security, I can only provide two pictures as follows :
when I use the first command, the result is this :
image
when I use the second command, the result is this :
image
I think the output of both methods should be the same, because the local.repo I used is a list of rpm packages downloaded from the remote source, but that's not the case.
I don't know if the reason for this is whether the default values for the parameter (install_weak_deps)of the two commands are different.

@weli-l
Copy link
Author

weli-l commented Nov 13, 2023

Hello, I am not sure what do you want to achieve with the -c local.repo parameter. For running the dnf using the locally cached metadata only, you have to pass the capitalized -C option. If you have specified the local repository in the local.repo file, you can configure dnf to run using only this repository f.e. with the --repofrompath and --repo parameters, try looking in the dnf man pages.

Thank you for your reply. local.repo is the configuration file I created locally. I created another folder localRPMS locally to store all the rpm packages, and then I used -c local.repo to download files from the localRPMS folder. But this way I was able to download the number of rpm packages that I was able to download unlike using a remote source, which had fewer weak dependency packages. If I use -C, then the resource is still downloaded from the remote source.

I see, if -c option is used to pass to specify a configuration from a different path, the repositories configuration is still loaded from the reposdir path (by default "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d"), so packages from other repositories are taken into account. This could be modified f.e. using the --disablerepo parameter.
Additionally, I observed that when running the first command you provided, the default /etc/dnf/dnf.conf is used. However, in the second command with the -c option, you are overriding it by using the local.repo. Depending on the configuration in /etc/dnf/dnf.conf, this might result in different behavior in both cases.

Yes, when I run these two commands in turn, executing the first command does not download weak dependencies, but the second command does. In fact, these weak dependency packages are already downloaded in my environment.

So, could you confirm whether the install_weak_deps option in /etc/dnf/dnf.conf is not set to False? Or provide the whole configuration file? The default value, if not specified by the user or configuration, is True, and this could potentially explain your situation
/

these are my configure local.repo [xxx] name=xxx baseurl=/root/downrpms enabled=1 gpgcheck=0 priority=1
/etc/dnf/dnf.conf [main] gpgcheck=1 installonly_limit=3 clean_requirements_on_remove=True best=True skip_if_unavailable=False sslverify=False
/etc/yum.repos.d/remote.repo [1xxx] name=1xxx baseurl=https://xxx enabled=1 gpgcheck=0 priority=10

OK, I see another difference of your two provided use cases. The first one is running with best=True, the second one with best=False as the /etc/dnf/dnf.conf is not being applied. Could you try running the second use case with --best option and compare the transaction packages then?

when I use dnf install -b -c local.repo --installroot /usr1/k_root/ [package list] problem is still alive

It might be useful to provide also the output of both commands used. Also with the second command I guess you want to use only the local repository, right? Then as described above, you need to disable the remote repositories. Or use --repo=xxx with the second command.

Because of Information security, I can only provide two pictures as follows :
when I use the first command, the result is this :
image
when I use the second command, the result is this :
image
I think the output of both methods should be the same, because the local.repo I used is a list of rpm packages downloaded from the remote source, but that's not the case.
I don't know if the reason for this is whether the default values for the parameter (install_weak_deps)of the two commands are different.

@jan-kolarik
Copy link
Member

jan-kolarik commented Nov 13, 2023

I don't know if the reason for this is whether the default values for the parameter (install_weak_deps)of the two commands are different.

You can verify this by providing --setopt=install_weak_deps=True for both commands and comparing the output. However, given the configuration file you provided, it appears that this is not the case.

I'm still uncertain if the environment is the same in both use cases. All the packages from the connected and enabled repositories in the first use case should be identical to the downloaded ones in the second local use case. If there are any differences, it might impact the solver's decision, leading to different resolutions being offered to the user.

@weli-l
Copy link
Author

weli-l commented Nov 13, 2023

I don't know if the reason for this is whether the default values for the parameter (install_weak_deps)of the two commands are different.

You can verify this by providing --setopt=install_weak_deps=True for both commands and comparing the output. However, given the configuration file you provided, it appears that this is not the case.

I'm still uncertain if the environment is the same in both use cases. All the packages from the connected and enabled repositories in the first use case should be identical to the downloaded ones in the second local use case. If there are any differences, it might impact the solver's decision, leading to different resolutions being offered to the user.

Before I configured local.repo, I used dnf download * to download all rpm packages, and both commands were run on the same virtual machine and the environment was the same. In addition, through --setopt=install_weak_deps =False, the output results of the two methods are exactly the same, both are 424. I don’t know what causes the weak dependencies not to be downloaded when downloading the package list from the remote source.

@m-blaha
Copy link
Member

m-blaha commented Nov 13, 2023

Well, there are edge cases when dnf download will not download all packages needed for installation - see issue #2002 . But to be sure about the root cause, I'm afraid we need more data. The best would be a reproducer of the issue.
If you need to download all the packages from the repo, then you might want to use the dnf reposync plugin.

@weli-l
Copy link
Author

weli-l commented Nov 14, 2023

Well, there are edge cases when dnf download will not download all packages needed for installation - see issue #2002 . But to be sure about the root cause, I'm afraid we need more data. The best would be a reproducer of the issue. If you need to download all the packages from the repo, then you might want to use the dnf reposync plugin.

yes, I know. But the problem is still alive after I use the dnf reposync plugin. I think the key of the problem is why weak dependencies packages cannot be downloaded when I use remote sources.

@m-blaha
Copy link
Member

m-blaha commented Nov 14, 2023

Just a long shot - dnf has a feature to autodetect weak dependencies that the user explicitly does not want on the system. This is to prevent re-installation of such weakdeps in case of upgrade. For details see man dnf.conf and search for exclude_from_weak there. This feature is enabled by default - would disabling of it (via setting exclude_from_weak_autodetect=false) help to download all the weakdeps? But without more data, I'm just guessing what might be going on.

@weli-l
Copy link
Author

weli-l commented Nov 14, 2023

Just a long shot - dnf has a feature to autodetect weak dependencies that the user explicitly does not want on the system. This is to prevent re-installation of such weakdeps in case of upgrade. For details see man dnf.conf and search for exclude_from_weak there. This feature is enabled by default - would disabling of it (via setting exclude_from_weak_autodetect=false) help to download all the weakdeps? But without more data, I'm just guessing what might be going on.

image
After I set this option, weak dependencies are still not downloaded, I guess if these weak dependencies will cause installation errors, so automatic filtering. Does dnf has such a mechanism? Also, I have another question is that why I still download a small number of packages from a remote source when I specify local.repo with -c option

@weli-l
Copy link
Author

weli-l commented Nov 16, 2023

The root cause of the problem is that the configuration of the remote source is incorrect. The XML file in the remote source fails to parse the weak dependency.

@weli-l weli-l closed this as completed Nov 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants