fix(crawler): find sha1 by filename instead of using versions from metadata

[DmitriyLewen]

Description

Find sha1 by filename instead of using versions from metadata.
This is required to avoid cases when metadata file doesn't contains all version.

Also currently href is used if the file/directory name is too long.

Some examples from maven repository:

• multiple file from version directory - https://repo1.maven.org/maven2/org/jasypt/jasypt/1.9.3/
• metadata.xml files doesn't contains all versions - https://repo1.maven.org/maven2/org/tuckey/urlrewritefilter/maven-metadata.xml (missed 3.2.0 version)
• file/directory name is too long - https://repo.maven.apache.org/maven2/org/jbundle/util/osgi/wrapped/org.jbundle.util.osgi.wrapped.org.apache.catalina.servlets/7.0.23/ (inspect this page to see differences between href and text)
• same sha1 for 2 files - https://repo.maven.apache.org/maven2/ai/rapids/cudf/0.14/ (cudf-0.14-cuda10-1.jar.sha1 and cudf-0.14.jar.sha1)

test run:

• https://github.com/DmitriyLewen/trivy-java-db/actions/runs/8245358817/job/22549204555

test image:

• https://github.com/DmitriyLewen/trivy-java-db/pkgs/container/trivy-java-db/189767366?tag=1

DB changes:

	old	new
number of indexes	9443336	10790262
build time (crawler + build)	(6m - 11m) + (22m - 23m)	18m + 26m
db size before compress	535MB	589MB

Related Issues:

• Close fix(crawler): don't use metadata.xml files to get list of version #28

[DmitriyLewen]

An idea to better track wrong files:
we can add a file to this repo with a list of wrong sha1 files found.
If we find a new wrong file, we will return an error.
If the file already contains an wrong sha1 file, simply skip this sha1 file.

[DmitriyLewen]

This is not a required change. But it will help me (and other contributors with capital letters in their names) work with the changes from the fork.

[knqyf263]

What if using bash? To reduce the risk of supply chain attacks, I want to avoid introducing third-party actions unless it's truly necessary.

      lowercase_repo=$(echo "${{ github.repository }}" | tr '[:upper:]' '[:lower:]')

[DmitriyLewen]

no problem. changed in 2dddbfb

[knqyf263]

Now, we don't need this step and can merge it into the uploading step.

      - name: Upload assets to GHCR
        run: |
          lowercase_repo=$(echo "${{ github.repository }}" | tr '[:upper:]' '[:lower:]')"
          oras version
          oras push --artifact-type application/vnd.aquasec.trivy.config.v1+json \
          ghcr.io/${lowercase_repo}:${DB_VERSION} \
          javadb.tar.gz:application/vnd.aquasec.trivy.javadb.layer.v1.tar+gzip

[DmitriyLewen]

Right! Thanks!
Changed b358571
Test run - https://github.com/DmitriyLewen/trivy-java-db/actions/runs/8794956548