Allow callers to run a subprocess and provide low and high water marks when using SequenceOutput to emit standard output and standard error as soon as it arrives. #40

rdingman · 2025-05-08T17:10:38Z

Resolves #39

when using SequenceOutput to emit standard output and standard error as soon as it arrives. Resolves swiftlang#39

iCharlesHu

Thanks so much for the issue and PR! I agree this is a great addition to the API surface.

As a overall comment, could you add a test to make sure the new behavior works as intended?

iCharlesHu · 2025-05-09T19:03:10Z

Sources/Subprocess/AsyncBufferSequence.swift

            self.buffer = []
            self.currentPosition = 0
            self.finished = false
+            self.streamIterator = Self.createDataStream(with: diskIO.dispatchIO, bufferSize: bufferSize).makeAsyncIterator()


AsyncBufferSequence is a shared type across all platforms therefore we can't unconditionally refer to platform specific type dispatchIO here. We may see Windows build failure as a result

Thanks for the feedback. I'll look into implementing this in such a way that platforms without dispatchIO don't break.

iCharlesHu · 2025-05-09T19:14:38Z

Sources/Subprocess/IO/Output.swift

+    internal let lowWater: Int?
+    internal let highWater: Int?
+    internal let bufferSize: Int
+
+    internal init(lowWater: Int? = nil, highWater: Int? = nil, bufferSize: Int = readBufferSize) {
+        self.lowWater = lowWater
+        self.highWater = highWater
+        self.bufferSize = bufferSize
+    }


I don’t think it’s appropriate to include these parameters here for a couple of reasons:

(This isn’t directly related to your change) right now, we’re in the middle of some major architectural updates: Adopt ~Copyable in Subprocess #38. This PR makes SequenceOutput internal, so you can’t use .sequence or .sequence(lowWater: …) anymore.

More importantly, this looks like a platform-specific feature. Setting this parameter won’t have any impact on Windows, and (also unrelated to your change) we’re planning to move away from DispatchIO on Linux soon, so it won’t work there either.

Considering all this, I suggest we move these parameters to Darwin’s specific PlatformOptions, maybe under a nested struct PlatformOptions.StreamOptions.

Noted. I'll look into moving these parameters.

rdingman · 2025-05-09T19:33:36Z

Thanks so much for the issue and PR! I agree this is a great addition to the API surface.

As a overall comment, could you add a test to make sure the new behavior works as intended?

I did add a test named "testSlowDripRedirectedOutputRedirectToSequence". Does that not cover the new behavior like you intend?

rdingman · 2025-05-10T22:20:40Z

@iCharlesHu Any suggestions on how to build and test this on windows? I have things building on windows, but I'm having trouble debugging the tests.

rdingman · 2025-05-11T23:21:03Z

@iCharlesHu I figured out how to get the debugger working in VSCode on Windows.

However, it appears that several of the tests crash with an exception because a file descriptor is being closed more than once (this is the case on main). Is this a known issue?

I tried out your PR #38 to see if it fixed those issues, but it has not.

For what its worth I'm running on:

VSCode 1.100.0
Swift Extension 2.2.0
LLDB DAP 0.2.13
Windows 11 Pro 24h2
Swift version 6.1 (swift-6.1-RELEASE)
Target: aarch64-unknown-windows-msvc

iCharlesHu · 2025-05-12T18:16:27Z

Thanks so much for the issue and PR! I agree this is a great addition to the API surface.
As a overall comment, could you add a test to make sure the new behavior works as intended?

I did add a test named "testSlowDripRedirectedOutputRedirectToSequence". Does that not cover the new behavior like you intend?

Ahh yes! Sorry I totally missed testSlowDripRedirectedOutputRedirectToSequence. That should work thanks!

iCharlesHu · 2025-05-12T18:18:28Z

@iCharlesHu I figured out how to get the debugger working in VSCode on Windows.

However, it appears that several of the tests crash with an exception because a file descriptor is being closed more than once (this is the case on main). Is this a known issue?

I tried out your PR #38 to see if it fixed those issues, but it has not.

For what its worth I'm running on:

VSCode 1.100.0

Swift Extension 2.2.0

LLDB DAP 0.2.13

Windows 11 Pro 24h2

Swift version 6.1 (swift-6.1-RELEASE)
Target: aarch64-unknown-windows-msvc

Thanks so much for looking into the Windows build. Unfortunately we do have some known test failures on Windows currently (#22) and I'll address them separately. Right now we want to make sure all new changes at least build on Windows.

rdingman · 2025-05-12T18:22:44Z

@iCharlesHu I figured out how to get the debugger working in VSCode on Windows.
However, it appears that several of the tests crash with an exception because a file descriptor is being closed more than once (this is the case on main). Is this a known issue?
I tried out your PR #38 to see if it fixed those issues, but it has not.
For what its worth I'm running on:

VSCode 1.100.0

Swift Extension 2.2.0

LLDB DAP 0.2.13

Windows 11 Pro 24h2

Swift version 6.1 (swift-6.1-RELEASE)
Target: aarch64-unknown-windows-msvc

@iCharlesHu I figured out how to get the debugger working in VSCode on Windows.
However, it appears that several of the tests crash with an exception because a file descriptor is being closed more than once (this is the case on main). Is this a known issue?
I tried out your PR #38 to see if it fixed those issues, but it has not.
For what its worth I'm running on:

VSCode 1.100.0

Swift Extension 2.2.0

LLDB DAP 0.2.13

Windows 11 Pro 24h2

Swift version 6.1 (swift-6.1-RELEASE)
Target: aarch64-unknown-windows-msvc

Thanks so much for looking into the Windows build. Unfortunately we do have some known test failures on Windows currently (#22) and I'll address them separately. Right now we want to make sure all new changes at least build on Windows.

@iCharlesHu Great, that's good to know. Thanks!

Sources/Subprocess/Platforms/Subprocess+Unix.swift

Sources/Subprocess/Platforms/Subprocess+Windows.swift

Sources/Subprocess/AsyncBufferSequence.swift

Sources/Subprocess/Platforms/Subprocess+Unix.swift

iCharlesHu · 2025-05-13T22:22:22Z

Sources/Subprocess/AsyncBufferSequence.swift

+                    streamIterator = diskIO.readDataStream(upToLength: readBufferSize).makeAsyncIterator()
+                    return data


This does not seem right. Why are we creating a new iterator when the first one ends? There will be nothing to read from this second iterator because all the data in the pipe would already been read.

Thanks for the pushback on this. I was going to explain my thinking and then realize if I have to explain this then it probably should be written in a more straightforward manner. The first implementation was using one iterator per chunk and when we reached a chunk boundary we'd switch to a new iterator. While this did work, it was admittedly a little clunky. Now, we use one AsyncThrowingStream (and iterator) across all chunks (even if the chunk is broken up into sub-chunks fora single read as in my original motivation for this issue. Please check on the new implementation to see if it makes sense to you.

Sources/Subprocess/Platforms/Subprocess+Unix.swift

iCharlesHu · 2025-05-14T03:49:05Z

Tests/SubprocessTests/SubprocessTests+Unix.swift

@@ -665,6 +665,48 @@ extension SubprocessUnixTests {
        #expect(catResult.terminationStatus.isSuccess)
        #expect(catResult.standardError == expected)
    }
+
+    @Test func testSlowDripRedirectedOutputRedirectToSequence() async throws {
+        let threshold: Double = 0.5


Unfortunately in tests you'll have to write

guard #available(SubprocessSpan , *) else { return }

In the beginning to work around the same availability issue. See other tests for examples.

@iCharlesHu When I first started working on this, I was very confused as to why some of the tests weren't running my new code and it was because of this check. Wouldn't it be better to have them skipped and noted as such in the test output rather than falsely succeeding? I'm thinking something like this:

@Test( .enabled( if: { if #available(SubprocessSpan , *) { true } else { false } }(), "This test requires SubprocessSpan" ) ) func testSlowDripRedirectedOutputRedirectToSequence() async throws { }

Of course, we can have a helper function to make this less verbose.

Thoughts?

@iCharlesHu I went ahead and conditionalized this one test this way as an example. Let me know if you don't like that and would like me to revert to a guard

iCharlesHu · 2025-05-14T05:39:26Z

Sources/Subprocess/Platforms/Subprocess+Linux.swift

+    public struct StreamOptions: Sendable {
+        let lowWater: Int?
+        let highWater: Int?
+
+        init(lowWater: Int? = nil, highWater: Int? = nil) {
+            self.lowWater = lowWater
+            self.highWater = highWater
+        }


Sorry I know I initially suggested using a StreamOptions nested struct, but after revisiting this API, I think we should reconsider the lowWater and highWater properties. Here’s why: 1) Their names can be quite confusing outside of the DispatchIO context, and 2) we’d need to add runtime validation to ensure lowWater < highWater.

How about we try something like this instead?

struct PlatformOptions { … let preferredStreamBufferSizeRange: Range<Int>? = nil }

This approach makes it clear that we’re requesting a range (with a lower and upper bound), and it eliminates the need for validation.

@iCharlesHu The issue I see with this suggestion is that it presumes that you will set both the lower and upper bound, or neither. There is no mechanism for setting just one or the other. As I see it, we have a few options:

Adopt your suggestion and take the stance that you cannot set these independently.

Recognize that these are platform specific options and rename them to indicate that. On Linux, these options would be removed once the Linux implementation moves away from DispatchIO because they are DispatchIO specific. We'd add the appropriate runtime check here. (FWIW, DispatchIO handles when lowWater > highWater and makes them the same).

Attempt to use some sort of sentinel values to represent the "don't set this" or "use the default" case. For lowWater mark, this could be something like -1 (the default is "unspecified"). For highWater this is tougher because the documentation says the default value is SIZE_MAX which isn't representable by Int. IMO, this option doesn't seem very intuitive, but I thought I'd include it anyways.

Change to use an enum which represents the four cases of set neither, set lower, set upper, set both. Something like:

enum BufferSizeOptions { case none case lowerBound(Int) case upperBound(Int) case range(Range<Int>) }

Thoughts?

I think option 3 is too awkward and unintuitive, so I don't think we should consider it. If you feel strongly about option 1, we can go with that, but I wanted to bring up this issue.

@iCharlesHu I modified your proposal a bit to use a RangeExpression rather than just a concrete Range that way we can express things like 0... to only set the lower bounds or ...4096 to only set the upper bounds. I didn't want to make PlatformOptions fully generic because this would be more cumbersome. Instead, I added some API on PlatformOptions to enforce the requirement that RangeExpression.Bound must be an Int.

Check it out and let me know what you think.

iCharlesHu · 2025-05-14T05:41:22Z

Sources/Subprocess/Platforms/Subprocess+Linux.swift

+    public var outputOptions: StreamOptions = .init()
+    public var errorOptions: StreamOptions = .init()


IMO we only need one (see my comments below).

I replaced these with preferredStreamBufferSizeRange above.

…cessSpan, *) so they compile with Swift 6.2

…o read all then data rather than one per chunk

… RangeExpression to express the various ways to configure the low and high watermark

rdingman requested a review from iCharlesHu as a code owner May 8, 2025 17:10

Allow callers to run a subprocess and provide low and high water marks

7324ac4

when using SequenceOutput to emit standard output and standard error as soon as it arrives. Resolves swiftlang#39

rdingman force-pushed the rdingman/issue-39 branch from ae5935d to 7324ac4 Compare May 8, 2025 17:19

iCharlesHu added the API Change label May 9, 2025

iCharlesHu reviewed May 9, 2025

View reviewed changes

rdingman marked this pull request as draft May 9, 2025 23:59

rdingman force-pushed the rdingman/issue-39 branch 5 times, most recently from 73a31d4 to a1abbf5 Compare May 10, 2025 00:35

rdingman force-pushed the rdingman/issue-39 branch from a1abbf5 to 14584a9 Compare May 11, 2025 19:09

Move stream creation and configure to platform specific files

7b6899c

rdingman force-pushed the rdingman/issue-39 branch from d9f85be to 7b6899c Compare May 11, 2025 21:16

rdingman marked this pull request as ready for review May 11, 2025 23:15

iCharlesHu reviewed May 14, 2025

View reviewed changes

rdingman added 8 commits May 14, 2025 08:49

Mark APIs that reference SequenceOutput.Buffer with @available(Subpro…

9b173ab

…cessSpan, *) so they compile with Swift 6.2

Simplify AsyncBufferSequence to only create one AsyncThrowingStream t…

ab43ae4

…o read all then data rather than one per chunk

Add fatalError for unexpected result from DispatchIO.read

10aa523

Fix build on Windows

aa903ad

Update PlatformOptions to have preferredStreamBufferSizeRange and use…

bb57bf2

… RangeExpression to express the various ways to configure the low and high watermark

Fix build on Linux

955e9c9

Fix Swift 6.2 build

5f2df5f

Update how we disable tests that require SubprocessSpan

4bac06a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow callers to run a subprocess and provide low and high water marks when using SequenceOutput to emit standard output and standard error as soon as it arrives. #40

Allow callers to run a subprocess and provide low and high water marks when using SequenceOutput to emit standard output and standard error as soon as it arrives. #40

rdingman commented May 8, 2025 •

edited

Loading

iCharlesHu left a comment

iCharlesHu May 9, 2025 •

edited

Loading

rdingman May 9, 2025

iCharlesHu May 9, 2025

rdingman May 9, 2025

rdingman commented May 9, 2025

rdingman commented May 10, 2025

rdingman commented May 11, 2025

iCharlesHu commented May 12, 2025

iCharlesHu commented May 12, 2025

rdingman commented May 12, 2025

iCharlesHu May 13, 2025

rdingman May 14, 2025

iCharlesHu May 14, 2025

rdingman May 14, 2025 •

edited

Loading

rdingman May 14, 2025

iCharlesHu May 14, 2025

rdingman May 14, 2025

rdingman May 14, 2025 •

edited

Loading

iCharlesHu May 14, 2025

rdingman May 14, 2025

		streamIterator = diskIO.readDataStream(upToLength: readBufferSize).makeAsyncIterator()
		return data

		public var outputOptions: StreamOptions = .init()
		public var errorOptions: StreamOptions = .init()

Allow callers to run a subprocess and provide low and high water marks when using SequenceOutput to emit standard output and standard error as soon as it arrives. #40

Are you sure you want to change the base?

Allow callers to run a subprocess and provide low and high water marks when using SequenceOutput to emit standard output and standard error as soon as it arrives. #40

Conversation

rdingman commented May 8, 2025 • edited Loading

iCharlesHu left a comment

Choose a reason for hiding this comment

iCharlesHu May 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdingman commented May 9, 2025

rdingman commented May 10, 2025

rdingman commented May 11, 2025

iCharlesHu commented May 12, 2025

iCharlesHu commented May 12, 2025

rdingman commented May 12, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdingman May 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdingman May 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdingman commented May 8, 2025 •

edited

Loading

iCharlesHu May 9, 2025 •

edited

Loading

rdingman May 14, 2025 •

edited

Loading

rdingman May 14, 2025 •

edited

Loading