Unstable behaviour of resolving dependencies

[andygrunwald]

The dependency resolver of perseus seems to be unstable.
If you have a list of packages that need to resolve the dependencies, there is a high chance that a different number of dependencies will be resolved.
But this should be stable and return the same amount every time.

Expected Behavior

Resolving the same amount of dependencies every time.

Current Behavior

Different amount of resolved dependencies

Possible Solution

Not sure yet, what the detailed issue is or how to solve it.

Steps to Reproduce (for bugs)

perseus ships multiple configuration files with: ./.docker/medusa-small.json, ./.docker/medusa-medium.json and ./.docker/medusa-big.json

Those can be used for testing.
How to reproduce this issue:

1. Clone this repository
2. Fire make build && ./perseus mirror ./.docker/medusa-medium.json --numOfWorkers=98
3. Let it run, wait for the message Start concurrent download process and check the amountPackages=201-Output in the log message.
4. Do it multiple times for the same configuration file and compare the amount of resolved packages to download
5. Expected behavour should be that the amount is every time the same for the same configuration file

Context

How has this issue affected you?

This can lead to missing packages for local repositories.
If packagist is complete deactivated, it is not able to install a PHP application via composer, because not all deps can be resolved.

A few numbers

Numbers of today from my local network:

➜  perseus git:(master) ✗ make build && ./perseus mirror ./.docker/medusa-small.json --numOfWorkers=98
go build -ldflags "-X main.Version=`git rev-parse --abbrev-ref HEAD 2>/dev/null` -X main.CommitHash=`git rev-parse --short HEAD 2>/dev/null` -X main.BuildDate=`date +%FT%T%z`" github.com/andygrunwald/perseus/cmd/perseus
INFO[2017-05-15T13:06:22+02:00] Using configuration file                      path="./.docker/medusa-small.json"
INFO[2017-05-15T13:06:22+02:00] Running "mirror" command
INFO[2017-05-15T13:06:22+02:00] Configuration                                 error="No repositories defined/configured."
INFO[2017-05-15T13:06:23+02:00] Start concurrent download process             amountPackages=33 amountWorker=98
...

➜  perseus git:(master) ✗ make build && ./perseus mirror ./.docker/medusa-medium.json --numOfWorkers=98
go build -ldflags "-X main.Version=`git rev-parse --abbrev-ref HEAD 2>/dev/null` -X main.CommitHash=`git rev-parse --short HEAD 2>/dev/null` -X main.BuildDate=`date +%FT%T%z`" github.com/andygrunwald/perseus/cmd/perseus
INFO[2017-05-15T13:04:25+02:00] Using configuration file                      path="./.docker/medusa-medium.json"
INFO[2017-05-15T13:04:25+02:00] Running "mirror" command
INFO[2017-05-15T13:04:25+02:00] Configuration                                 error="No repositories defined/configured."
INFO[2017-05-15T13:04:26+02:00] Start concurrent download process             amountPackages=201 amountWorker=98
^C

➜  perseus git:(master) ✗ make build && ./perseus mirror ./.docker/medusa-big.json --numOfWorkers=78
go build -ldflags "-X main.Version=`git rev-parse --abbrev-ref HEAD 2>/dev/null` -X main.CommitHash=`git rev-parse --short HEAD 2>/dev/null` -X main.BuildDate=`date +%FT%T%z`" github.com/andygrunwald/perseus/cmd/perseus
INFO[2017-05-15T15:45:42+02:00] Using configuration file                      path="./.docker/medusa-big.json"
INFO[2017-05-15T15:45:42+02:00] Running "mirror" command
INFO[2017-05-15T15:45:42+02:00] Configuration                                 error="No repositories defined/configured."
INFO[2017-05-15T15:45:43+02:00] Error while resolving dependencies of package  error="API returned status code 404: Expected a return code within 2xx for package "thecodingmachine/nodejs-installer". Got 404" package="thecodingmachine/nodejs-installer" responseCode=404
INFO[2017-05-15T15:45:43+02:00] Error while resolving dependencies of package  error="API returned status code 404: Expected a return code within 2xx for package "psr/cache-implementation". Got 404" package="psr/cache-implementation" responseCode=404
INFO[2017-05-15T15:45:44+02:00] Start concurrent download process             amountPackages=373 amountWorker=78
^C

➜  perseus git:(master) ✗ make build && ./perseus mirror ./.docker/medusa-big.json --numOfWorkers=98
go build -ldflags "-X main.Version=`git rev-parse --abbrev-ref HEAD 2>/dev/null` -X main.CommitHash=`git rev-parse --short HEAD 2>/dev/null` -X main.BuildDate=`date +%FT%T%z`" github.com/andygrunwald/perseus/cmd/perseus
INFO[2017-05-15T14:34:41+02:00] Using configuration file                      path="./.docker/medusa-big.json"
INFO[2017-05-15T14:34:41+02:00] Running "mirror" command
INFO[2017-05-15T14:34:41+02:00] Configuration                                 error="No repositories defined/configured."
INFO[2017-05-15T14:34:42+02:00] Error while resolving dependencies of package  error="API returned status code 404: Expected a return code within 2xx for package "thecodingmachine/nodejs-installer". Got 404" package="thecodingmachine/nodejs-installer" responseCode=404
INFO[2017-05-15T14:34:42+02:00] Error while resolving dependencies of package  error="API returned status code 404: Expected a return code within 2xx for package "psr/cache-implementation". Got 404" package="psr/cache-implementation" responseCode=404
INFO[2017-05-15T14:34:42+02:00] Start concurrent download process             amountPackages=362 amountWorker=98
^C

If you have more test results, feel free to post your results here as well

Your Environment

• Version used (perseus version): master (rev 1dcc4e8)
• Operating System and version: macOS Sierra v10.12.4
• Medusa configuration: Located in ./docker/-Folder
• Satis configuration: Located in ./docker/-Folder

[mre]

Ran it on my local machine as discussed. I get 363 packages:

# make build && ./perseus mirror ./.docker/medusa-big.json --numOfWorkers=98                                                                         ⏎
go build -ldflags "-X main.Version=`git rev-parse --abbrev-ref HEAD 2>/dev/null` -X main.CommitHash=`git rev-parse --short HEAD 2>/dev/null` -X main.BuildDate=`date +%FT%T%z`" github.com/andygrunwald/perseus/cmd/perseus
INFO[2017-05-15T16:06:26+02:00] Using configuration file                      path="./.docker/medusa-big.json"
INFO[2017-05-15T16:06:26+02:00] Running "mirror" command
INFO[2017-05-15T16:06:26+02:00] Configuration                                 error="No repositories defined/configured."
INFO[2017-05-15T16:06:27+02:00] Error while resolving dependencies of package  error="API returned status code 404: Expected a return code within 2xx for package \"psr/cache-implementation\". Got 404" package="psr/cache-implementation" responseCode=404
INFO[2017-05-15T16:06:27+02:00] Error while resolving dependencies of package  error="API returned status code 404: Expected a return code within 2xx for package \"thecodingmachine/nodejs-installer\". Got 404" package="thecodingmachine/nodejs-installer" responseCode=404
INFO[2017-05-15T16:06:28+02:00] Start concurrent download process             amountPackages=363 amountWorker=98

[andygrunwald]

Maybe this can help: https://jorin.me/use-go-channels-to-build-a-crawler/