Troubleshooting Timed Out Builds

[finestructure]

Discussed in #2615

^{Originally posted by stephencelis September 13, 2023}
Is there a way to figure out why builds are timing out? Maybe a way to get a partial log?

We have a project with visionOS and watchOS builds timing out:

https://swiftpackageindex.com/pointfreeco/swift-snapshot-testing/builds

Would love to track the issue down, as we are able to build for both platforms locally.

[finestructure]

I can confirm this seems to be hanging during "Resolve Package Graph" (which is pretty much always where xcodebuild builds hang) when I build locally with our setup for 5.9/visionOS:

Command SwiftCompile failed with a nonzero exit code

Resolve Package Graph"
�[36;1mResolving Package Graph�[0m

In the case of the 5.9 builds this is very likely due to us still using Xcode-15.0.0-Beta.2 to build those. It works fine for example when I build with Xcode-15.0.0-Release.Candidate.

That doesn't explain the timeouts for watchOS on 5.7 and 5.8 though, looking into that.

[finestructure]

Ok, it also hangs when I build watchOS with Xcode-14.3.1 (which is what I have locally, our builders actually use 14.3.0 for Swift 5.8 but that shouldn't make a difference). It's actually tripping over building TestHelpers.swift:

** BUILD FAILED **


The following build commands failed:
	SwiftCompile normal armv7k Compiling\ TestHelpers.swift /Users/sas/Projects/SPI/spi-builder/checkout/Tests/SnapshotTestingTests/TestHelpers.swift (in target 'SnapshotTestingTests' from project 'swift-snapshot-testing')
	SwiftCompile normal armv7k /Users/sas/Projects/SPI/spi-builder/checkout/Tests/SnapshotTestingTests/TestHelpers.swift (in target 'SnapshotTestingTests' from project 'swift-snapshot-testing')
(2 failures)"

and then eventually hanging with

SwiftDriverJobDiscovery normal arm64_32 Compiling SyntaxStmtNodes.swift, SyntaxTypeNodes.swift (in target 'SwiftSyntax' from project 'swift-syntax')

Resolve Package Graph"
Resolving Package Graph

It's unclear to me why the build error leads to a hang but we've seen this a few times with xcodebuild builds.

Here are the full logs of the build, as run through our builder tool:

swift-snapshot-testing-build.log.zip

We effectively run

env DEVELOPER_DIR="/Applications/Xcode-14.3.1.app" xcrun xcodebuild -IDEClonedSourcePackagesDirPathOverride="$PWD/.dependencies" -derivedDataPath "$PWD/.derivedData" build -scheme swift-snapshot-testing-Package -destination "generic/platform=watchos"

which fails the similarly (but not identically) but without the hang:

** BUILD FAILED **


The following build commands failed:
	SwiftCompile normal armv7k Compiling\ WaitTests.swift /Users/sas/Projects/SPI/spi-builder/checkout/Tests/SnapshotTestingTests/WaitTests.swift (in target 'SnapshotTestingTests' from project 'swift-snapshot-testing')
	SwiftCompile normal armv7k /Users/sas/Projects/SPI/spi-builder/checkout/Tests/SnapshotTestingTests/WaitTests.swift (in target 'SnapshotTestingTests' from project 'swift-snapshot-testing')
(2 failures)

It's unclear to me why the run through the builder tool would continue on and hang 🤔

(It's not that we hang on the doc build, that isn't attempted if the build has already failed.)

[finestructure]

Sorry, this took a while but I've found a fix for the hangs and I also know why the watchOS builds were failing - it's two separate issues.

The watchOS build failure is because we pick the wrong scheme: swift-snapshot-testing-Package instead of SnapshotTesting.

env DEVELOPER_DIR=/Applications/Xcode-15.0.0.app xcrun xcodebuild build -scheme SnapshotTesting -destination generic/platform=watchos

succeeds whereas

env DEVELOPER_DIR=/Applications/Xcode-15.0.0.app xcrun xcodebuild build -scheme swift-snapshot-testing-Package -destination generic/platform=watchos

fails.

The hang in case of failure seems to be caused by an issue when collecting stdout/stderr output. We've seen this happening across a number of packages - always xcodebuild builds.

A newer version of our process runner (a forked version of ShellOut) avoids this problem and properly collects the output.

I'll open a PR against snapshot-testing to set the proper scheme for the watchOS builds.

[finestructure]

pointfreeco/swift-snapshot-testing#785

[finestructure]

Fixed in https://gitlab.com/finestructure/swiftpackageindex-builder/-/merge_requests/224

Pending test runs and deployment.

[finestructure]

All tests passed & deployed.