Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature improved java cataloging #2

Draft
wants to merge 57 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
f26d089
WIP: gather all properties in hierarchy
GijsCalis Feb 26, 2024
87df551
Merge remote-tracking branch 'gcalis/main' into recurse-parent-poms-a…
GijsCalis Feb 27, 2024
e96dbb1
WIP: recurse into parents and boms
GijsCalis Feb 28, 2024
acdf111
WIP: Getting there, rework of recursion needed
GijsCalis Feb 29, 2024
a0dd2ef
Added resolving properties by name (reference value)
GijsCalis Mar 4, 2024
eb3f72d
Property resolution working
GijsCalis Mar 4, 2024
2602b81
Get pom from local repo, recursively gather properties
GijsCalis Mar 17, 2024
3668766
Use local Maven repository for resolving artifacts
GijsCalis Apr 4, 2024
e586b31
fix logging of error
GijsCalis Apr 4, 2024
62684c7
fix load default java cataloger config on start
GijsCalis Apr 4, 2024
902520b
fix logging of license parsing errors
GijsCalis Apr 4, 2024
78f0c24
cleanup logging: start with lowercase
GijsCalis Apr 4, 2024
7d4c5a4
Make local Maven repository dir configurable
GijsCalis Apr 7, 2024
a77af37
Add unit tests for using remote Maven repo and local Maven repo
GijsCalis Apr 7, 2024
7f1c79d
Fix unit tests by resetting caches before test
GijsCalis Apr 8, 2024
c727765
Recurse into parent poms with cycle detection
GijsCalis Apr 9, 2024
26587c8
fix bug: missed changed function signature
GijsCalis Apr 9, 2024
bf8ebfd
Remove unneeded logging
GijsCalis Apr 9, 2024
ceedb47
Merge branch 'main' into feature-improved-java-cataloging
GijsCalis Apr 9, 2024
eb80105
remove unused/duplicate modules
GijsCalis Apr 10, 2024
ec208f9
Retry resolving version property after processing all parent poms
GijsCalis Apr 10, 2024
d3efe1f
Update instructions for java configuration
GijsCalis Apr 11, 2024
1df8583
remove accidentally created SBOM files
GijsCalis Apr 11, 2024
f01788c
Code clean: style fixes
GijsCalis Apr 12, 2024
ac2e1ea
Merge remote-tracking branch 'origin/main' into feature-improved-java…
GijsCalis Apr 12, 2024
ff1c843
Merge remote-tracking branch 'origin/main' into feature-improved-java…
GijsCalis Apr 16, 2024
ebda837
Merge remote-tracking branch 'upstream/main' into feature-improved-ja…
kzantow Jun 27, 2024
f941def
chore: initial refactor to use mavenResolver
kzantow Jul 17, 2024
2ad3c5d
Merge remote-tracking branch 'upstream/main' into feature-improved-ja…
kzantow Jul 17, 2024
2922853
chore: refactor maven_repo_utils
kzantow Jul 17, 2024
9b7f3dc
Merge remote-tracking branch 'upstream/main' into feature-improved-ja…
kzantow Jul 17, 2024
8bcd53c
chore: cache maven pom files directly
kzantow Jul 17, 2024
9864efe
chore: refactor parsedPomProject
kzantow Jul 18, 2024
14536a0
Merge remote-tracking branch 'upstream/main' into feature-improved-ja…
kzantow Jul 18, 2024
b51c5eb
chore: fix env var names
kzantow Jul 18, 2024
a3485b3
chore: update some comments
kzantow Jul 18, 2024
f11cb49
chore: maven_resolver refactoring
kzantow Jul 18, 2024
1960f70
chore: trim test files
kzantow Jul 18, 2024
3486497
Merge remote-tracking branch 'upstream/main' into feature-improved-ja…
kzantow Jul 18, 2024
4a7b5b6
Merge remote-tracking branch 'upstream/main' into feature-improved-ja…
kzantow Jul 22, 2024
bbcf965
feat: add maven relativePath parent resolution
kzantow Jul 23, 2024
6472bdf
chore: lint-fix
kzantow Jul 23, 2024
a1fb9d7
chore: refactor pom cataloger to scan and index all poms in the resolver
kzantow Jul 23, 2024
7b2fb7a
chore: improve property resolution for boms
kzantow Jul 23, 2024
0f41319
chore: properly resolve maven ID info
kzantow Jul 24, 2024
9a047e4
fix: improve determinism in java archive identification
kzantow Jul 25, 2024
d06334b
Merge remote-tracking branch 'upstream/main' into feature-improved-ja…
kzantow Jul 25, 2024
103a608
Merge remote-tracking branch 'origin/fix/deterministic-java-pom-prope…
kzantow Jul 25, 2024
e8b14f7
chore: use structured logging
kzantow Jul 25, 2024
80253f7
chore: use structured logging
kzantow Jul 25, 2024
984f21c
Merge remote-tracking branch 'upstream/main' into feature-improved-ja…
kzantow Jul 29, 2024
697b4e1
chore: don't trim existing pom
kzantow Jul 29, 2024
084e1f7
chore: reorganize test utils
kzantow Jul 29, 2024
7ff89e5
Merge remote-tracking branch 'upstream/main' into feature-improved-ja…
kzantow Jul 30, 2024
7837e26
fix: properly respect max parent depth, default to unlimited
kzantow Jul 31, 2024
e11085c
Merge remote-tracking branch 'upstream/main' into feature-improved-ja…
kzantow Aug 4, 2024
a2a695f
chore: pr feedback
kzantow Aug 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions cmd/syft/internal/options/catalog.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ func DefaultCatalog() Catalog {
Package: defaultPackageConfig(),
LinuxKernel: defaultLinuxKernelConfig(),
Golang: defaultGolangConfig(),
Java: defaultJavaConfig(),
File: defaultFileConfig(),
Relationships: defaultRelationshipsConfig(),
Source: defaultSourceConfig(),
Expand Down Expand Up @@ -150,6 +151,8 @@ func (cfg Catalog) ToPackagesConfig() pkgcataloging.Config {
GuessUnpinnedRequirements: cfg.Python.GuessUnpinnedRequirements,
},
JavaArchive: java.DefaultArchiveCatalogerConfig().
WithUseMavenLocalRepository(cfg.Java.UseMavenLocalRepository).
WithMavenLocalRepositoryDir(cfg.Java.MavenLocalRepositoryDir).
WithUseNetwork(cfg.Java.UseNetwork).
WithMavenBaseURL(cfg.Java.MavenURL).
WithArchiveTraversal(archiveSearch, cfg.Java.MaxParentRecursiveDepth),
Expand Down
38 changes: 30 additions & 8 deletions cmd/syft/internal/options/java.go
Original file line number Diff line number Diff line change
@@ -1,24 +1,46 @@
package options

import "github.com/anchore/clio"
import (
"github.com/anchore/clio"
"github.com/anchore/syft/syft/pkg/cataloger/java"
)

type javaConfig struct {
UseNetwork bool `yaml:"use-network" json:"use-network" mapstructure:"use-network"`
UseMavenLocalRepository bool `yaml:"use-maven-local-repository" json:"use-maven-local-repository" mapstructure:"use-maven-local-repository"`
MavenLocalRepositoryDir string `yaml:"maven-local-repository-dir" json:"maven-local-repository-dir" mapstructure:"maven-local-repository-dir"`
MavenURL string `yaml:"maven-url" json:"maven-url" mapstructure:"maven-url"`
MaxParentRecursiveDepth int `yaml:"max-parent-recursive-depth" json:"max-parent-recursive-depth" mapstructure:"max-parent-recursive-depth"`
}

func defaultJavaConfig() javaConfig {
def := java.DefaultArchiveCatalogerConfig()

return javaConfig{
UseNetwork: def.UseNetwork,
MaxParentRecursiveDepth: def.MaxParentRecursiveDepth,
UseMavenLocalRepository: def.UseMavenLocalRepository,
MavenLocalRepositoryDir: def.MavenLocalRepositoryDir,
MavenURL: def.MavenBaseURL,
}
}

var _ interface {
clio.FieldDescriber
} = (*javaConfig)(nil)

func (o *javaConfig) DescribeFields(descriptions clio.FieldDescriptionSet) {
descriptions.Add(&o.UseNetwork, `enables Syft to use the network to fill in more detailed information about artifacts
currently this enables searching maven-url for license data
when running across pom.xml files that could have more information, syft will
explicitly search maven for license information by querying the online pom when this is true
this option is helpful for when the parent pom has more data,
that is not accessible from within the final built artifact`)
descriptions.Add(&o.UseNetwork, `enables Syft to use the network to fetch version and license information for packages when
a parent or imported pom file is not found in the local maven repository.
the pom files are downloaded from the remote Maven repository at 'maven-url'`)
descriptions.Add(&o.MavenURL, `maven repository to use, defaults to Maven central`)
descriptions.Add(&o.MaxParentRecursiveDepth, `depth to recursively resolve parent POMs`)
descriptions.Add(&o.MaxParentRecursiveDepth, `depth to recursively resolve parent POMs, no limit if <= 0`)
descriptions.Add(&o.UseMavenLocalRepository, `use the local Maven repository to retrieve pom files. When Maven is installed and was previously used
for building the software that is being scanned, then most pom files will be available in this
repository on the local file system. this greatly speeds up scans. when all pom files are available
in the local repository, then 'use-network' is not needed.
TIP: If you want to download all required pom files to the local repository without running a full
build, run 'mvn help:effective-pom' before performing the scan with syft.`)
descriptions.Add(&o.MavenLocalRepositoryDir, `override the default location of the local Maven repository.
the default is the subdirectory '.m2/repository' in your home directory`)
}
2 changes: 1 addition & 1 deletion syft/file/license.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ type LicenseEvidence struct {
func NewLicense(value string) License {
spdxExpression, err := license.ParseExpression(value)
if err != nil {
log.Trace("unable to parse license expression: %s, %w", value, err)
log.WithFields("error", err, "value", value).Trace("unable to parse license expression")
}

return License{
Expand Down
5 changes: 5 additions & 0 deletions syft/pkg/cataloger/internal/pkgtest/test_generic_parser.go
Original file line number Diff line number Diff line change
Expand Up @@ -310,6 +310,11 @@ func TestFileParser(t *testing.T, fixturePath string, parser generic.Parser, exp
NewCatalogTester().FromFile(t, fixturePath).Expects(expectedPkgs, expectedRelationships).TestParser(t, parser)
}

func TestCataloger(t *testing.T, fixtureDir string, cataloger pkg.Cataloger, expectedPkgs []pkg.Package, expectedRelationships []artifact.Relationship) {
t.Helper()
NewCatalogTester().FromDirectory(t, fixtureDir).Expects(expectedPkgs, expectedRelationships).TestCataloger(t, cataloger)
}

func TestFileParserWithEnv(t *testing.T, fixturePath string, parser generic.Parser, env *generic.Environment, expectedPkgs []pkg.Package, expectedRelationships []artifact.Relationship) {
t.Helper()

Expand Down
Loading
Loading