Linking failed assertions to resources

[splatch]

A follow up of discussions in junit-team/junit5#2504.

I have a set of dynamic tests which are based on a XML document. The XML document role is to define a hex-encoded byte sequence, fire a deserializer and compare generated object tree with expected XML structure.
So far I managed to use testSourceUri parameter in instantiated dynamic tests to hint IDE where to look for test definitions (?line=&column=). I seek a solution for a second step - assertion errors.

Currently I found no way to report a test failure which is result of an unmatched assertion or comparison coming from an external resource. Please note that I do not seek a way to compare XML elements, I am able to conduct that operation using existing tools. What I currently can not do is pointing a line in which failed assertion starts.
An example sequence of file looks like this:
https://github.com/apache/plc4x/blob/e93b442e77b3d5594a0bc1ecd6311c86c9395b4e/protocols/ads/src/test/resources/protocols/ads/AdsDiscoverySerializerTest.xml#L77L108
Any lines between 77 and 108 can lead to assertion error hence it would be extremely useful from development point of view to tell which one exactly is wrong.

Above situation is not specific to an Apache PLC4X where above feature would be consumed. I believe that use of various scenario files and "externalized" scenario definitions is widely used across industry. Yet it still miss a common way to point failures which are beyond written java code.

[marcphilipp]

I think we could address this by introducing a new LocationAwareThrowable interface with a Set<URI> getLocations() method and implementing it in all AssertionError/Exception implementations.

[splatch]

I can't comment much over your proposal, cause this is more specific to tooling maintainers who would consume it.

From naming/design perspective I'd say that for me it sounds like LocationAware and LocationsAware could be used do distinct situations where there is one pointer or multiple (ie. dynamic test with single source or set of dynamic tests with multiple sources?). A specific Throwables are just another type which can bring location/locations information.
[Note, I have limited knowledge about ota4j, above is just free comment in topic]

[sbrannen]

I'm wondering if "source" might be a better term than "location" for this use case, as in "the source of the (cause of the) exception" so to speak.

I also wonder if the method should return a collection or single location/source.

For example, MultipleFailuresError could be used to aggregate multiple LocationAwareThrowable exceptions, where each individual exception points to a single external resource/location/source.

[leonard84]

One thing to keep in mind, that absolute URLs can be problematic, when the report displayed a browser.
Relative URLs can also be problematic, when the report does not have the same base directory and the respective resource is not moved along.

Also, do we define a standard to address specific content in a file, e.g., what github does with line numbers #L42?

[marcphilipp]

Also, do we define a standard to address specific content in a file, e.g., what github does with line numbers #L42?

I think we should do what JUnit Platform already supports in FilePosition:

scheme://...?line=42&column=23

[marcphilipp]

I'm wondering if "source" might be a better term than "location" for this use case, as in "the source of the (cause of the) exception" so to speak.

I went with "location" since "source" (cf. TestSource in the JUnit Platform) might be too restrictive, e.g. when you want to point at resource files.