Scanner freezes when using ClearlyDefined as a scan storage provider #4540

martencassel · 2021-10-04T20:22:43Z

When i run the scanner using clearlydefined with a file based storage, the scanner seem to hang.

I don't know if its a problem using the clearlydefined api or if my configuration file is incorrect ?

Is there any flag to see more information from the scanner, like debug messages ?

The process doesn't seem to do much

sudo sudo strace -f -e trace=network -s 10000 -p `pidof java`
strace: Process 110620 attached with 91 threads
[pid 110770] recvfrom(295, 
 <unfinished ...>
[pid 110767] sendto(295, "\27\3\3\0#\315K\17\366~\3\365\5\37\r\327.N.\274\215y>\264\10\355u\244\277\35K^{9\69\233\305i\"", 40, 0, NULL, 0) = 40
[pid 110767] sendto(295, "\27\3\3\0#\326?|j\201\320\224\257I\316\261\207\333(\\\311>\10\2217x1\255\227\223J\271\377,R\224!>\236\301", 40, 0, NULL, 0) = 40
[pid 110767] shutdown(295, SHUT_WR)     = 0
[pid 110767] shutdown(295, SHUT_RD)     = 0
[pid 110770] <... recvfrom resumed>"", 5, 0, NULL, NULL) = 0
[pid 110770] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7f231368e008} ---
[pid 110767] +++ exited with 0 +++
[pid 110770] +++ exited with 0 +++
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---

~/scripts/scan.sh 
20:06:35.331 [main] INFO  org.ossreviewtoolkit.model.config.OrtConfiguration - Using ORT configuration file '/ort-home/.ort/ort.conf'.
20:06:35.638 [main] INFO  org.ossreviewtoolkit.model.config.LicenseFilenamePatterns - Configuring the license file patterns.
________ _____________________
\_____  \\______   \__    ___/ the OSS Review Toolkit, version DOCKER-SNAPSHOT.
 /   |   \|       _/ |    |
/    |    \    |   \ |    |    Running 'scan' under Java 11.0.8 on Linux with
\_______  /____|_  / |____|    8 CPUs and a maximum of 7942 MiB of memory.
        \/       \/
Environment variables:
ORT_CONFIG_DIR = /ort-home/.ort
ORT_DATA_DIR = /ort-home/.ort
JAVA_HOME = /opt/java/openjdk
ANDROID_HOME = /opt/android-sdk
GOPATH = /go

20:06:35.682 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - Using file based storage with local directory '/ort-home/.ort/scan-results'.
20:06:35.689 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - Using ClearlyDefined storage with URL 'https://api.clearlydefined.io'.
20:06:35.694 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - Using composite storage with readers FileBasedStorage with XZCompressedLocalFileStorage backend, ClearlyDefinedStorage and writers FileBasedStorage with XZCompressedLocalFileStorage backend.
20:06:35.695 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - ScanResultStorage has been configured to composite[readers:[FileBasedStorage with XZCompressedLocalFileStorage backend, ClearlyDefinedStorage], writers:[FileBasedStorage with XZCompressedLocalFileStorage backend]].
Using scan storage 'composite[readers:[FileBasedStorage with XZCompressedLocalFileStorage backend, ClearlyDefinedStorage], writers:[FileBasedStorage with XZCompressedLocalFileStorage backend]]'.
Using scanner 'ScanCode'.
20:06:36.733 [main] INFO  org.ossreviewtoolkit.scanner.scanners.scancode.ScanCode - Searching scan results for 1 packages.
20:06:36.739 [main] INFO  org.ossreviewtoolkit.scanner.scanners.scancode.ScanCode - Searching scan results for 380 packages.
20:06:37.199 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/cavaliercoder/badio:v0.0.0-20160213150051-ce5280129e9e'.
20:06:37.199 [DefaultDispatcher-worker-9] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/anmitsu/go-shlex:v0.0.0-20161002113705-648efa622239'.
20:06:37.199 [DefaultDispatcher-worker-63] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.62.0'.
20:06:37.200 [DefaultDispatcher-worker-7] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/asaskevich/govalidator:v0.0.0-20200907205600-7a23bdc65eef'.
20:06:37.200 [DefaultDispatcher-worker-37] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/Masterminds/squirrel:v1.4.0'.
20:06:37.200 [DefaultDispatcher-worker-61] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go/storage:v1.6.0'.
20:06:37.200 [DefaultDispatcher-worker-13] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.65.0'.
20:06:37.200 [DefaultDispatcher-worker-41] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/Masterminds/goutils:v1.1.0'.
20:06:37.200 [DefaultDispatcher-worker-45] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/cavaliercoder/go-rpm:v0.0.0-20200122174316-8cb9fd9c31a8'.
20:06:37.200 [DefaultDispatcher-worker-38] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/alecthomas/jsonschema:v0.0.0-20200530073317-71f438968921'.
20:06:37.200 [DefaultDispatcher-worker-10] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/BurntSushi/toml:v0.3.1'.
20:06:37.200 [DefaultDispatcher-worker-59] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go/storage:v1.5.0'.
20:06:37.200 [DefaultDispatcher-worker-46] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::code.gitea.io/sdk/gitea:v0.13.0'.
20:06:37.200 [DefaultDispatcher-worker-60] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.76.0'.
20:06:37.200 [DefaultDispatcher-worker-43] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/Masterminds/sprig:v2.22.0+incompatible'.
20:06:37.201 [DefaultDispatcher-worker-48] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.34.0'.
20:06:37.200 [DefaultDispatcher-worker-18] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/asaskevich/govalidator:v0.0.0-20200428143746-21a406dcc535'.
20:06:37.201 [DefaultDispatcher-worker-24] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go/storage:v1.0.0'.
20:06:37.201 [DefaultDispatcher-worker-50] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.72.0'.
20:06:37.201 [DefaultDispatcher-worker-30] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go/storage:v1.10.0'.
20:06:37.201 [DefaultDispatcher-worker-33] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.26.0'.
20:06:37.198 [DefaultDispatcher-worker-35] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/cavaliercoder/go-cpio:v0.0.0-20180626203310-925f9528c45e'.
20:06:37.198 [DefaultDispatcher-worker-36] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/Masterminds/semver:v1.5.0'.
20:06:37.198 [DefaultDispatcher-worker-64] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go/storage:v1.8.0'.
20:06:37.198 [DefaultDispatcher-worker-62] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.74.0'.
20:06:37.210 [DefaultDispatcher-worker-8] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.57.0'.
20:06:37.225 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.utils.OrtAuthenticator - Authenticator was successfully installed.
20:06:37.240 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.utils.OrtProxySelector - Proxy selector was successfully installed.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by retrofit2.Platform (file:/opt/ort/lib/retrofit-2.9.0.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of retrofit2.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20:06:37.446 [DefaultDispatcher-worker-30] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.54.0'.
20:06:37.446 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.44.3'.
20:06:37.446 [DefaultDispatcher-worker-35] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.46.3'.
20:06:37.447 [DefaultDispatcher-worker-30] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.44.2'.
20:06:37.447 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.44.1'.
20:06:37.447 [DefaultDispatcher-worker-35] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.38.0'.
20:06:37.446 [DefaultDispatcher-worker-61] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.50.0'.
20:06:37.447 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/cespare/xxhash/v2:v2.1.1'.
20:06:37.446 [DefaultDispatcher-worker-41] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.52.0'.
20:06:37.447 [DefaultDispatcher-worker-30] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.45.1'.
20:06:37.446 [DefaultDispatcher-worker-50] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.56.0'.
20:06:37.446 [DefaultDispatcher-worker-60] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.53.0'.

cat ~/scripts/scan.sh 
#!/bin/bash

export CURRENT_DIRNAME=$(basename "$PWD")

sudo rm -rf ~/.ort/project/packaged/ort/scanner/native-scan-results

docker run -e ORT_CONFIG_DIR=/ort-home/.ort \
           -e ORT_DATA_DIR=/ort-home/.ort \
           -v $HOME:/ort-home \
           -v $(pwd):/project \
	   -v ~/.gitconfig:/root/.gitconfig \
           ort:latest --info scan -i /ort-home/.ort/project/$CURRENT_DIRNAME/ort/analyzer/analyzer-result.json -o /ort-home/.ort/project/$CURRENT_DIRNAME/ort/scanner

 cat ~/.ort/ort.conf
ort {
  scanner {
    storages {
      clearlyDefined {
        serverUrl = "https://api.clearlydefined.io"
      }
      fileBasedStorage {
        backend {
          localFileStorage {
            directory = "/ort-home/.ort/scan-results"
            compression = true
          }
        }
      }
    }

    storageReaders: [
       "fileBasedStorage", "clearlyDefined"
    ]

    storageWriters: [
      "fileBasedStorage"
    ]
  }
}

sschuberth · 2021-10-05T06:58:11Z

We're also facing timeout issues with ClearlyDefined recently. My hope is that working on #3905 will address this.

schvvarzekatze · 2021-12-15T13:39:07Z

If I configure to read and write the scan result only on postgres it seems to stop due to a timeout:

3:13:43.225 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running '/usr/local/bin/scancode --copyright --license --info --strip-root --timeout 300 --processes 7 /tmp/ort-ScanCode<id>/Maven/com.azure/azure-core-http-netty/1.5.0 --json-pp /sample-project/build/oss-review-toolkit/gradle/scan/native-scan-results/Maven/com.azure/azure-core-http-netty/1.5.0/scan-results_ScanCode.json' in '/'...

Would it help to configure the timeout of ScanCode

sschuberth · 2021-12-15T13:49:12Z

Hmm, I don't see ScanCode running into a timeout in the logs you posted (you should see a message like "Timeout ... while scanning file ..." then).

As Netty is a rather big package IIRC, the scan could simply take long. A ScanCode scan can easily take several hours depending on the code size. PS: The timeout is per file, not per total scan duration.

schvvarzekatze · 2021-12-15T15:51:28Z

It does not even continue after more than one hour from this package only on postgres. But also with clearlyDefined it stops after some minutes scanning - not at the same package when I want to reproduce it. It does not continue even after several hours. I have never seen the timeout message in the logs so a longer timeout probably won't be a solution.

ort {
  analyzer {
  }

  scanner {
    storages {

      postgres {
        url = ${POSTGRES_URL}
        schema = "public"
        username = "ort"
        password = ${POSTGRES_PASSWORD}
        sslmode = "disable"
      }

    }

    storageReaders: [
      "postgres"
    ]

    storageWriters: [
      "postgres"
    ]
  }
}

ort {
  analyzer {
  }

  scanner {
    storages {
      clearlyDefined {
       serverUrl = "https://api.clearlydefined.io"
     }

      postgres {
        url = ${POSTGRES_URL}
        schema = "public"
        username = "ort"
        password = ${POSTGRES_PASSWORD}
        sslmode = "disable"
      }

      fileBasedStorage {
              backend {
                localFileStorage {
                  directory = "/ort/analyzer"
                  compression = true
                }
              }
            }
    }

    storageReaders: [
      "fileBasedStorage", "postgres", "clearlyDefined"
    ]

    storageWriters: [
      "fileBasedStorage", "postgres"
    ]
  }
}

schvvarzekatze · 2021-12-15T16:00:32Z

The scan passed azure-core-http-nettythis time after 1.5 hours. So it really takes very long. ;)
Finally the scan with postgres was successful and took about 5 h.

software-testing-professional · 2022-02-09T07:13:12Z

I'm also facing this freeze issue when using clearlyDefined as storageReader.

...
storageReaders = [
  fileBasedStorage, clearlyDefined
]

storageWriters = [
  fileBasedStorage 
]
...

root@2af0304336d8:/# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   7244  4104 pts/0    Ss   06:45   0:00 /bin/bash
root        1089  0.8  1.1 14265044 379328 pts/0 Sl+  06:53   0:11 /opt/java/openjdk/bin/java -classpath /opt/ort/lib/*:/opt/ort/plugin/* org.ossreviewtoolkit.cli.OrtMainKt --info scan -i /project/ort/analyzer/analyzer-result.json -o /pro
root        1242  0.0  0.0   7244  3892 pts/1    Ss   07:15   0:00 /bin/bash
root        1252  0.0  0.0   8900  3252 pts/1    R+   07:16   0:00 ps aux

ORT is in Sl+ (interruptible sleep).

Any news on this?

sschuberth · 2022-02-09T07:37:25Z

Any news on this?

I guess no one is currently working on this because AFAIK ClearlyDefined uses a very old version of ScanCode by now, and thus it's quite unattractive to use ClearlyDefined as a scan storage reader currently...

sschuberth · 2022-02-24T08:02:42Z

I accidentally came across this issue that I fixed, which likely also fixes this issue. Can anyone confirm?

martencassel · 2022-02-25T17:45:10Z

I would like to test this. Any particular projects that i can use as test cases ?

Apart from the scanning not freezing or anything like that, how do i know that this works ?

martencassel · 2022-02-25T18:03:09Z

I just ran the scanner (main/73af92a4b86cb5e01c0b867e6eb64115ed692bf0) of ort with a https://github.com/tornadoweb/tornado/tree/master with no issues running the scanner.

martencassel · 2022-02-25T18:04:41Z

Just curios, do you perform any e2e tests of clearlydefined in a pipline or such ?

sschuberth · 2022-02-25T19:46:57Z

Just curios, do you perform any e2e tests of clearlydefined in a pipline or such ?

No, currently not.

sschuberth · 2022-02-25T19:47:49Z

I just ran the scanner (main/73af92a4b86cb5e01c0b867e6eb64115ed692bf0) of ort with a https://github.com/tornadoweb/tornado/tree/master with no issues running the scanner.

That's good news. We should probably wait for other affected users to confirm the fix before closing this issue.

software-testing-professional · 2022-02-28T06:03:13Z

I'll give it a try within the next days. With this configuration:

storageReaders = [
  fileBasedStorage, clearlyDefined
]

storageWriters = [
  fileBasedStorage 
]

schvvarzekatze · 2022-02-28T08:08:39Z

In our case there no timeout anymore with this configuration using the recent docker image of ORT:

ort {
  analyzer {
  }

  scanner {
    storages {
      clearlyDefined {
        serverUrl = "https://api.clearlydefined.io"
      }

      postgres {
        url = ${POSTGRES_URL}
        schema = "public"
        username = "ort"
        password = ${POSTGRES_PASSWORD}
        sslmode = "disable"
      }
    }

    storageReaders: [
      "postgres", "clearlyDefined"
    ]

    storageWriters: [
      "postgres"
    ]
  }
}

sschuberth · 2022-02-28T09:32:04Z

Thanks for these confirmations, I'm closing this then.

software-testing-professional · 2022-02-28T13:12:31Z

Hmm.
It's still freezing.

I'm on commit main/ddfbf6289b4aef2b5acaf29dee6dd0718b27b52a and running the scanner locally (after ./gradlew installDist).

Config is

storageReaders = [
  clearlyDefined, fileBasedStorage
]

So clearlyDefined is used first.

borisbaldassari · 2022-03-01T17:10:23Z

As mentioned in issue #5116 I also have the issue on latest dev. Commit ID is 98616cc52da42fbe9cfd2b317e90fd4cc2925d04.
Storage readers is defined as [ fileBasedStorage, clearlyDefined ].

Caveat: it actually works on some projects (e.g. Eclipse JGit) but freezes on others (e.g. Eclipse Tycho).

martencassel · 2022-03-01T17:39:49Z

I tested with https://github.com/tornadoweb/tornado (python project) and it worked fine.

janderssonse · 2022-03-16T08:19:12Z

Confirm freeze after a few minutes with latest dev and clearlyDefined
Storage readers is defined as [ fileBasedStorage, clearlyDefined ].

mawl · 2022-04-29T08:20:42Z

Confirm freeze on a dotnetcore project:

ort {
  analyzer {
    ignoreToolVersions = true
    allowDynamicVersions = true
  }

  scanner {
    skipConcluded = false
    
    storages {
      artifactoryStorage {
        backend {
          httpFileStorage {
            url = "https://repo.mycompany.local/artifactory/scan-results/license-scanner"
            headers {
              X-JFrog-Art-Api = "${ARTIFACTORY_API_TOKEN}"
            }
          }
        }
      }

      clearlyDefined {
        serverUrl = "https://api.clearlydefined.io"
      }      
    }

    storageReaders: [
      "artifactoryStorage", "clearlyDefined"
    ]

    storageWriters: [
      "artifactoryStorage"
    ]
  }
}

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by retrofit2.Platform (file:/opt/ort/lib/retrofit-2.9.0.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of retrofit2.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
07:00:07.015 [DefaultDispatcher-worker-48] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'NuGet::System.Diagnostics.Contracts:4.3.0'.
07:00:07.015 [DefaultDispatcher-worker-47] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'NuGet::Microsoft.NETCore.Targets:1.0.1'.
07:00:07.015 [DefaultDispatcher-worker-38] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'NuGet::System.Data.Common:4.1.0'.
07:00:07.027 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'NuGet::System.Diagnostics.DiagnosticSource:4.0.0'.
07:00:07.053 [DefaultDispatcher-worker-69] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'NuGet::System.Diagnostics.DiagnosticSource:4.3.0'.
07:00:07.053 [DefaultDispatcher-worker-47] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'NuGet::System.Diagnostics.Process:4.1.0'.
ERROR: Job failed: execution took longer than 1h0m0s seconds

sschuberth · 2022-08-26T10:24:01Z

For anyone who's still affected by this, can you please share your ORT analyzer result for a public (and as small as possible) project where you can reproduce the scanner freeze?

schvvarzekatze · 2022-08-30T13:22:27Z

I still have the freeze for the following package:

11:29:43.338 [DefaultDispatcher-worker-15] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'Maven:com.github.junrar:junrar:7.5.2'.

sschuberth · 2022-08-31T07:05:44Z

Looking up results for 'Maven:com.github.junrar:junrar:7.5.2'.

Just using that single artifact in ClearlyDefinedStorageFunTest still passes. So it's not a matter of a specific artifact being used, but a sum of artifacts that probably triggers some race condition. I'd really need a full analyzer result file for which the hanging in the scanner reproducibly happens.

sschuberth · 2023-01-18T14:37:47Z

Can anyone still reproduce the hanging with today's version of ORT?

schvvarzekatze · 2023-02-08T12:55:19Z

I still have the freeze for the following package:

11:29:43.338 [DefaultDispatcher-worker-15] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'Maven:com.github.junrar:junrar:7.5.2'.

Now the freeze does not occur anymore for gradle packages. I used 32 GB and 32 CPUs because this also solved the Freeze of the ORT scan without ClearlyDefined as Read Only storage.

sschuberth · 2023-02-08T20:38:38Z

Thanks for confirming. With the recent fixes for ORT's calls to the ClearlyDefined API (esp. for packages without namespaces) I indeed believe this to be fixed. If anyone still sees this, we can reopen the issue.

software-testing-professional · 2023-02-23T09:04:24Z

I can confirm that the scanner does not freeze anymore, if ClearlyDefined is defined as storageReader (together with artifactoryStorage).

storageReaders: [
  "clearlyDefined",
  "artifactoryStorage"
]

...
08:30:57.397 [main] INFO org.ossreviewtoolkit.scanner.ScanResultsStorage - Read 0 scan result(s) for 'Maven:io.swagger.core.v3:swagger-core:2.2.7' from ClearlyDefinedStorage in 694.389us.
08:30:57.979 [main] INFO org.ossreviewtoolkit.scanner.ScanResultsStorage - Read 1 scan result(s) for 'Maven:org.apache.tomcat.embed:tomcat-embed-el:10.1.4' from ClearlyDefinedStorage in 582.341739ms.
08:30:57.981 [main] INFO org.ossreviewtoolkit.scanner.ScanResultsStorage - Read 0 scan result(s) for 'Maven:net.logstash.logback:logstash-logback-encoder:7.2' from ClearlyDefinedStorage in 1.274609ms.
08:30:57.982 [main] INFO org.ossreviewtoolkit.scanner.ScanResultsStorage - Read 0 scan result(s) for 'Maven:org.springframework:spring-test:6.0.3' from ClearlyDefinedStorage in 890.483us.
...
Tested with Docker-based ORT, built from commit 19c89ff.

Thank you for fixing it!!! smiley

martencassel changed the title ~~scanner seems to hang when using ClearlyDefined~~ scanner freezes when using ClearlyDefined Oct 4, 2021

This comment was marked as off-topic.

Sign in to view

sschuberth closed this as completed Feb 28, 2022

sschuberth reopened this Feb 28, 2022

sschuberth mentioned this issue Mar 1, 2022

ORT Scanner freezes on some projects #5116

Closed

sschuberth added bug scanner About the scanner tool labels Mar 1, 2022

sschuberth changed the title ~~scanner freezes when using ClearlyDefined~~ Scanner freezes when using ClearlyDefined as a scan storage provider May 3, 2022

sschuberth mentioned this issue Jul 19, 2022

scanner runs very slow / no status #3296

Closed

sschuberth pinned this issue Aug 18, 2022

sschuberth mentioned this issue Aug 26, 2022

Set up a scheduled job that runs ORT on ORT #5696

Open

sschuberth moved this to Q3 2022 – Jul-Sep in Roadmap Aug 31, 2022

sschuberth added this to Roadmap Aug 31, 2022

sschuberth moved this from Q3 2022 – Jul-Sep to Q4 2022 - Oct-Dec in Roadmap Nov 2, 2022

This comment was marked as off-topic.

Sign in to view

sschuberth moved this from Q4 2022 - Oct-Dec to Q1 2023 - Jan-Mar in Roadmap Jan 11, 2023

sschuberth closed this as completed Feb 8, 2023

sschuberth unpinned this issue Feb 8, 2023

sschuberth mentioned this issue Feb 23, 2023

Analyzer fails to authenticate with Artifactory when downloading artifacts (http-401) #5507

Closed

sschuberth self-assigned this Feb 28, 2023

sschuberth removed this from Roadmap Nov 20, 2023

sschuberth marked this as a duplicate of #5116 Feb 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scanner freezes when using ClearlyDefined as a scan storage provider #4540

Scanner freezes when using ClearlyDefined as a scan storage provider #4540

martencassel commented Oct 4, 2021 •

edited

Loading

sschuberth commented Oct 5, 2021

This comment was marked as off-topic.

This comment was marked as off-topic.

schvvarzekatze commented Dec 15, 2021

sschuberth commented Dec 15, 2021

schvvarzekatze commented Dec 15, 2021

schvvarzekatze commented Dec 15, 2021 •

edited

Loading

software-testing-professional commented Feb 9, 2022 •

edited

Loading

sschuberth commented Feb 9, 2022

sschuberth commented Feb 24, 2022

martencassel commented Feb 25, 2022 •

edited

Loading

martencassel commented Feb 25, 2022 •

edited

Loading

martencassel commented Feb 25, 2022

sschuberth commented Feb 25, 2022

sschuberth commented Feb 25, 2022

software-testing-professional commented Feb 28, 2022 •

edited

Loading

schvvarzekatze commented Feb 28, 2022

sschuberth commented Feb 28, 2022

software-testing-professional commented Feb 28, 2022

borisbaldassari commented Mar 1, 2022 •

edited

Loading

martencassel commented Mar 1, 2022

janderssonse commented Mar 16, 2022 •

edited

Loading

mawl commented Apr 29, 2022

sschuberth commented Aug 26, 2022

schvvarzekatze commented Aug 30, 2022

sschuberth commented Aug 31, 2022

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

sschuberth commented Jan 18, 2023

schvvarzekatze commented Feb 8, 2023

sschuberth commented Feb 8, 2023

software-testing-professional commented Feb 23, 2023

Scanner freezes when using ClearlyDefined as a scan storage provider #4540

Scanner freezes when using ClearlyDefined as a scan storage provider #4540

Comments

martencassel commented Oct 4, 2021 • edited Loading

sschuberth commented Oct 5, 2021

This comment was marked as off-topic.

This comment was marked as off-topic.

schvvarzekatze commented Dec 15, 2021

sschuberth commented Dec 15, 2021

schvvarzekatze commented Dec 15, 2021

schvvarzekatze commented Dec 15, 2021 • edited Loading

software-testing-professional commented Feb 9, 2022 • edited Loading

sschuberth commented Feb 9, 2022

sschuberth commented Feb 24, 2022

martencassel commented Feb 25, 2022 • edited Loading

martencassel commented Feb 25, 2022 • edited Loading

martencassel commented Feb 25, 2022

sschuberth commented Feb 25, 2022

sschuberth commented Feb 25, 2022

software-testing-professional commented Feb 28, 2022 • edited Loading

schvvarzekatze commented Feb 28, 2022

sschuberth commented Feb 28, 2022

software-testing-professional commented Feb 28, 2022

borisbaldassari commented Mar 1, 2022 • edited Loading

martencassel commented Mar 1, 2022

janderssonse commented Mar 16, 2022 • edited Loading

mawl commented Apr 29, 2022

sschuberth commented Aug 26, 2022

schvvarzekatze commented Aug 30, 2022

sschuberth commented Aug 31, 2022

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

sschuberth commented Jan 18, 2023

schvvarzekatze commented Feb 8, 2023

sschuberth commented Feb 8, 2023

software-testing-professional commented Feb 23, 2023

martencassel commented Oct 4, 2021 •

edited

Loading

schvvarzekatze commented Dec 15, 2021 •

edited

Loading

software-testing-professional commented Feb 9, 2022 •

edited

Loading

martencassel commented Feb 25, 2022 •

edited

Loading

martencassel commented Feb 25, 2022 •

edited

Loading

software-testing-professional commented Feb 28, 2022 •

edited

Loading

borisbaldassari commented Mar 1, 2022 •

edited

Loading

janderssonse commented Mar 16, 2022 •

edited

Loading