[WorkflowTesting] Use CustomDump to better visualize failure messages

[DJBen]

What is it?

Better visualize the error message by using CustomDump in WorkflowTesting.

• Introduces dependency to CustomDump from WorkflowTesting, which is a lightweight opinion-less library that improves printing of any objects by leveraging Mirrors.
• In WorkflowTesting, replaces all the XCTAssertEquals into XCTAssertNoDifference, which is the go-to method to compare the diff the objects.

Why the change?

It has been a constant pain to reveal the differences of two very large structs when XCTAssertEquals fails, especially when the difference is minor. Some people resort to https://www.diffchecker.com/, or even subscribe to it to alleviate the issue. The manual copy and paste process is still a sink of engineering time.

Introducing the diffing visualization helps developers pinpoint the assertion failure in both local test sor CI pipelines, and speed up development in a long run.

Before:

("State(..... 1000+ characters omitted)") is not equal to ("State(..... 1000+ characters omitted)")

After:

XCTAssertNoDifference failed: …
State(
... (the method automatically omits overlapping states)
  − property: false
  + property: true
)
(First: −, Second: +)
            

Impact of the added dependency

The added dependency is added into WorkflowTesting, which is meant for the tests modules to depend on. It does not contain any semantic-altering change in workflows.

If the consumer module already depends on CustomDump, it may cause a version conflict. Fortunately, I synced the dependency to CustomDump with Square's internal dependent version to avoid this issue.

Checklist

• Unit Tests
• UI Tests (n/a)
• Snapshot Tests (iOS only) (n/a)
• I have made corresponding changes to the documentation (n/a)

[CLAassistant]

All committers have signed the CLA.

[jamieQ]

my guess is this is not going to work as a change to the public repo because this has been hosted internally as a pod, but AFAIK does not exist in the public CocoaPods spec repo

[dnkoutso]

That is correct. Podspecs may not specify "repos" on where the pod comes from and if the podspec is published into the cocoapods trunk (read its open source like Workflow is) then all of its dependencies must be public too.

[square-tomb]

Confirmed here: https://github.com/square/workflow-swift/actions/runs/5006839279/jobs/8973201477?pr=214#step:5:8

[DJBen]

Although I came across this and it seems public https://github.com/Golface/CocoaPods-Composable-Architecture.

[DJBen]

@dnkoutso what do you recommend the next step to unblock it would be?

[dnkoutso]

@DJBen this is a pain...we'd have to ask for someone to publish this in the open source repo or we cannot depend on it. An alternative is to duplicate all of it inside Workflow which is meh. Trying to think of alternatives..........

[DJBen]

If I publish it to the trunk, will it solve the issue? Because SQ internally is using a fork, will that cause a version conflict?

[dnkoutso]

If you do it would solve this issue here I believe yes.

For SQ if it uses an internal fork it would have to ensure the version requirements specified here match and if they do then I think it will take the fork

[DJBen]

I tried to untangle the public dependency and here's what I find:

• CustomDump depends on XCTestDynamicOverlay, but it needs > 0.4. Unfortunately, an old 0.1.1 is occupied by another person github.com/Arafo ([email protected]).

Is it okay that I alias the pods with PFC (pointfreeco), so they become PFCXCTestDynamicOverlay and PFCCustomDump, and so I can upload our versions to the public trunk?

I anticipate that may cause source file conflict if in our test, we have both files from CustomDump and PFCCustomDump. Another alternative is to cherry-pick and prefix all the files necessary from CustomDump to this repo. I wonder what you all think.

[DJBen]

@gilgtm I managed to contact Rafa and persuade him to delete the XCTestDynamicOverlay repo so I can push a new version. I managed to push CustomDump 0.11.0 to cocoapods.

[square-tomb]

Thank you for contributing this! I share your expectation that just about everyone will prefer this output.

I think we could avoid introducing a new third-party dependency to workflow-swift by either:

• adding a parameter to tester or assert, that accepts a function to use in place of XCTAssertEqual
• writing an extension within Register that adds a new set of assert methods

But both of these would require updating each existing tester/assert call site in Register that wants the new behavior, which we don't want to do if this is strictly an improvement. So I think making the change in WorkflowTesting itself is justified.

Can you put up a WIP PR in ios-register that points at this branch of workflow-swift, to verify with CI that none of our tests are somehow depending on specific behaviors of XCTAssertEqual?

[DJBen]

Thank you for contributing this! I share your expectation that just about everyone will prefer this output.

I think we could avoid introducing a new third-party dependency to workflow-swift by either:

• adding a parameter to tester or assert, that accepts a function to use in place of XCTAssertEqual
• writing an extension within Register that adds a new set of assert methods

But both of these would require updating each existing tester/assert call site in Register that wants the new behavior, which we don't want to do if this is strictly an improvement. So I think making the change in WorkflowTesting itself is justified.

Can you put up a WIP PR in ios-register that points at this branch of workflow-swift, to verify with CI that none of our tests are somehow depending on specific behaviors of XCTAssertEqual?

I agree. however I am experencing difficulty of setting up the dependency. I think the follow up steps are blocked until it is resolved