Add p2-aware model converter for CycloneDX SBOM generation #3258
Conversation
You can take a look at TychoWorkspaceReader that has a method
As described in the discussions around this I think this is somehow impossible to solve without configuration in all cases, but you can do the following, given you have already found the P2 coordinates:
I think this would require a configuration somewhere to what URL the reactor should be mapped, maybe just as simple as start with a user-property like
What do you mean by this? |
|
By the way please don't use tycho-extras, this is an old historic construct and should be avoided for new designs, simply put it in the root of the project. |
Thanks, I'll have a look.
And for the most part, I agree. But I also believe we should at least attempt to find the location of an artifact. Otherwise the package URL is not much of a locator... I also don't believe it has to be a link to the jar file. I would also consider the base URL of the update site, a link to the Git repository or anything similar to be valid locators. There are of course difficult/impossible edge-cases, but I'd rather have something that works for 95% of all projects and go from there. Problems that come to mind are e.g. with file-based repositories or builds that directly assemble a product and therefore don't upload their artifacts. But as said, I first want to get this working for simple projects.
That's a good point. I assume most projects already have a property in their pom, describing the repository the build artifacts are uploaded to. One might as well simply use the same property for both.
The SBOM contains components of the form: Where the group is simply the artificial group id created by Tycho. Given that there is no real counterpart in p2, I'm not exactly sure what to do with it or if it even matters. But that's simply something I have to ask the CycloneDX developers about.
Noted. |
As described above you can try to guess at least what URLs can supply the artifact with the described methods that works in a good 80% of cases. |
|
I made some good progress and a good chunk seems to be working already. A general problems seems to be that the SBOM is generate really early, at a point where Tycho is not fully initialized yet.
I didn't get that to work yet, which is why I'm still constructing the ArtifactKey "by hand".
That is currently done via a Maven property but in the end should function exactly the same way. I also believe I need to also create a CycloneDX dependency model for p2 artifacts. Because dependencies are usually flat. Example: |
This shouldn't be an issue, Tycho itself is initialized far earlier than most (if not all) plugins in a regular maven build, so maybe you can give an example where it causes issues?
Can you explain what was the problem?
Do you mean to construct a tree of items? There is already a dependency-tree mojo for P2 model
|
The The expanded version is set by the BuildQualifierMojo during the But this goal is executed before it has a chance to run and so far, I haven't managed to bind it to a later phase.
Initially, I was calling This initially wouldn't work, because Tycho didn't recognize them as part of the reactor build, so I had to calculate the corresponding project first. The next problem is that the version of the returned ArtifactKey isn't expanded, so inside the SBOM, it would still be Which is why I do all of this by hand.
I'm pretty sure CycloneDX doesn't use the dependency-tree mojo but rather works on I could fix this by also writing a custom ProjectDependenciesConverter which uses DependencyArtifacts instead, but that all depends on how much effort it is... |
I think you probably want to use What I can think of is to make Tycho compute the qualifier on demand at first call as we now do for almost everything, but as you said its done by the mojo actually and it is configurable so could be different from the default, especially if you choose different timestamp providers so one defiantly has to take the project config into account.
What do you mean by "not part of reactor build" are you calling the mojo directly on the commandline?
That's for sure, I just wanted to give a hint how you can reconstruct a tree from a flat list, but I'm not sure if you like the other way round? |
That's what I do ;D
The artifacts Tycho gets from CycloneDX are not of type
Hm... That looks really usefull, actually. If I can reuse this dependency tree, then it should hopefully be pretty straightforward to synchronize this with the SBOM tree. Thanks! |
Yes usually you need to first map an artifact to a reactor project by building a map from the session, I think mostly because CycloneDX will likely use dependencies instead of real project artifacts, you can for example use TychoReactorReader and make |
|
I finally got some time to work on this stuff again... I ended up reusing the algorithm that is used by the dependencies-tree mojo and moved parts of it to tycho-core. While it's not finished yet, it's (for the most part) functional. I still need to extend the documentation and do some cleanup, though. |
Great, if you like you can split this into a separate PR then we can merge the refactoring already.
Products don't have artifacts (in P2) as they are metadata only. You can take the metadata IU and perform a Planner operation on it or use the dependency artifacts itself. |
|
Got a little stuck with creating the CycloneDX dependency tree, but the result should now match the output of the |
This change consists of two parts. The first part consists of generating a valid package URL for a given p2 artifact. The PURL contains the symbolic name, version and classifier of the artifact key, as well as the repository it is located in. The second part handles the dependencies between two PURLs. For the sake of consistency, this dependency tree matches the tree calculated by the dependency-tree mojo.
|
@ptziegler do you want me to merge (and backport?) your change. |
|
/request-license-review |
License review requests:
After all reviews have concluded, re-run the license-vetting check from the Github Actions web-interface to update its status. Workflow run (with attached summary files): |
|
Sorry, the past few days have been a little busy on my side... I'd like to first update the specification in package-url/purl-spec#272 because I now use the base url of the repository as location, instead of the full path to the file. There are also a few instances where the SBOM is using the group id. I'd also like to ask the CycloneDX developers, whether it's fine to keep using the artifical p2.* entry or whether there's a better alternative. I'll try to work on this evening, but I don't believe they require any significant changes to the current implementation. So I think it would be good if the current state is merged, so that other people can try it on some larger projects. |
|
@ptziegler If you like please cherry-pick / backport this to Tycho 4 then it could be included in the upcomming bugfix release! |
laeubi
added
the
backport-to-tycho-4.0.x
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation and see the Github Action logs for details |
|
Ah, drat. I just tried to generate the SBOM for GEF-classic and an error is thrown because the expanded version isn't set for artifacts of type I don't really see the use-case for having the bill-of-materials for test fragments, so it probably makes more sense to restrict this converter to bundles and features, rather than to add the BuildQualifierMojo to the default-lifecycle-mapping for this artifact type. Until then, the |
|
@ptziegler actually the build-qualifier is part of the default-lifecycle-mapping already for the Beside that it is common for mojos to have a configuration parameter that controls the packaging types a mojo can run, but as far as I understand you don't have a mojo here but just extending the cyclone-dx one? In any case there is likely a next bugfix release after 15.03.2024 when Eclipse become GA, so any improvements can go in there and as you said its good to test it out in the wild what will likely uncover some improvements we can make here. |
After debugging, the build-qualifier is indeed calculated, also for the test bundles. To be precise, the problem shows up, the moment you have more than one test fragment. Example:
Looking at the dependency tree, the Draw2D tests have a dependency on an artifact with IU Now in order to calculate the version string in the PURL, I need to resolve the build qualifiers for all reactor projects, which is done via the ProjectDependencyClosure: I would assume that this method returns |
In hindsight, I'm probably simply using the wrong method... |
|
You can try to use Also I always enable |
It depends, as obviously it gives you the MavenProject but this does not mean it is build, I have already refactored Tycho in almost all places to behave well defined even if project is not yet build but build qualifier is the only place where this currently not happens. |
Initial draft based on the discussions here and here.
The big obstacles right now are:
Convert the Maven artifact back into a p2 artifact.Find the download URL for a given p2 artifact.Handle artifacts from the current reactor build.Handle group name of the SBOM component.The reactor artifacts are especially tricky, because none of the artifacts have been built when the SBOM is generated.
For illustration, this is the SBOM for org.eclipse.gef-feature:
bom.zip