-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Have MergedLinesEnumerable implement IAsyncEnumerable<string> #109
base: release-1.7
Are you sure you want to change the base?
Changes from 1 commit
3fddb89
15815ce
ca60f2a
da45635
e45f1f6
87316df
7976a96
d3720b9
990605e
0ab96b0
fdd9137
750b1e0
caa9428
3b99204
6a2baae
983a28f
7104cea
f6a555a
9030874
b2b7bbe
f9f0813
8d18b9d
5151a25
97198a3
1420db9
4c68836
263d7dd
16b38df
7830e3e
ff9af41
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -97,46 +97,37 @@ void TestOneThrows(bool reverse) | |
TestOneThrows(reverse: true); | ||
} | ||
|
||
[Test, Timeout(10000)] // something's wrong if it's taking more than 10 seconds | ||
public void FuzzTest() | ||
[Test, Timeout(5_000)] // something's wrong if it's taking more than 5 seconds | ||
public async Task FuzzTest() | ||
{ | ||
var pipe1 = new Pipe(); | ||
var pipe2 = new Pipe(); | ||
Pipe pipe1 = new(), pipe2 = new(); | ||
|
||
var asyncEnumerable = this.Create(new StreamReader(pipe1.OutputStream), new StreamReader(pipe2.OutputStream)); | ||
|
||
var strings1 = Enumerable.Range(0, 2000).Select(_ => Guid.NewGuid().ToString()).ToArray(); | ||
var strings2 = Enumerable.Range(0, 2300).Select(_ => Guid.NewGuid().ToString()).ToArray(); | ||
|
||
static void WriteStrings(IReadOnlyList<string> strings, TextWriter writer) | ||
static void WriteStrings(IReadOnlyList<string> strings, Pipe pipe) | ||
{ | ||
var spinWait = default(SpinWait); | ||
var random = new Random(Guid.NewGuid().GetHashCode()); | ||
SpinWait spinWait = default; | ||
Random random = new(Guid.NewGuid().GetHashCode()); | ||
using StreamWriter writer = new(pipe.InputStream); | ||
foreach (var line in strings) | ||
{ | ||
if (random.Next(110) == 1) | ||
if (random.Next(10) == 1) | ||
{ | ||
spinWait.SpinOnce(); | ||
} | ||
|
||
writer.WriteLine(line); | ||
} | ||
} | ||
|
||
var task1 = Task.Run(() => | ||
{ | ||
using StreamWriter writer1 = new(pipe1.InputStream); | ||
WriteStrings(strings1, writer1); | ||
}); | ||
var task2 = Task.Run(() => | ||
{ | ||
using StreamWriter writer2 = new(pipe2.InputStream); | ||
WriteStrings(strings2, writer2); | ||
}); | ||
var consumeTask = asyncEnumerable.ToListAsync(); | ||
Task.WaitAll(task1, task2, consumeTask); | ||
|
||
CollectionAssert.AreEquivalent(strings1.Concat(strings2).ToList(), consumeTask.Result); | ||
var task1 = Task.Run(() => WriteStrings(strings1, pipe1)); | ||
var task2 = Task.Run(() => WriteStrings(strings2, pipe2)); | ||
Task.WaitAll(task1, task2); // need to dispose the writer to end the stream | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think I had to swap the order for tests to pass. Is this a red flag? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah let's revert this change and make sure it still passes. Also, does this pass or fail on main? You didn't make any changes to Pipe I think so it may be an issue with the release branch. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @madelson I looked more into this and gathered some numbers, but I'm lost as to how I should debug this. Note:
However, I noticed that a small change makes a difference.
var consumeTask = Task.Run(enumerable.ToListAsync);
Task.WaitAll(task1, task2);
Task.WaitAll(task1, task2);
var consumeTask = Task.Run(enumerable.ToListAsync);
// originally 2000
var strings1 = Enumerable.Range(0, 20).Select(_ => Guid.NewGuid().ToString()).ToArray();
// originally 2300
var strings2 = Enumerable.Range(0, 23).Select(_ => Guid.NewGuid().ToString()).ToArray();
...
// same as master or release-1.7
var consumeTask = Task.Run(enumerable.ToListAsync);
Task.WaitAll(task1, task2, consumeTask);
CollectionAssert.AreEquivalent(strings1.Concat(strings2).ToList(), consumeTask.Result); Therefore, I believe that my Any idea how I should debug this? For one thing, I think replacing There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I also found that There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems like ChatGPT came to the rescue again. (It wasn't helping a few hours ago.) I lost its message, but it said something along the lines of " Now the test passes, but I can't run There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Makes sense; we have do dispose the writer to end the stream. Can you point me to the relevant code change?
I'm not sure I follow here. As often as what? Does it fail when it runs more often? In what way? How does the overall time for this test case compare before and after the changes (I would expect it to be the same). Who would be overriding the frequency if it were protected virtual? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @madelson Let me know if the above makes sense! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @madelson Bumping this thread! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This makes me feel like there is a bug somewhere. It could be in the MergedLinesEnumerable changes, it could be in the test code, or it could be in the What I would suggest is to (temporarily) add some logging statements to the code like this:
My assumption is that at some point we should stop seeing log statements as the code will enter a hung state. We can then add additional logs to try to get closer and closer to the point where each thread stops. From there, hopefully we can deduce why it is hanging. |
||
CollectionAssert.AreEquivalent(strings1.Concat(strings2), await asyncEnumerable.ToListAsync()); | ||
} | ||
Bartleby2718 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
} | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Bartleby2718 can we try
await Task.WhenAll(task1, task2, consumeTask);
here?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@madelson I've been trying to get that working, but this test fails for the async case if I do that. Not sure if the test logic is flawed (i.e. shouldn't await consumeTask if the input streams may not have been closed?) or there's a bug somewhere else.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you try specifically with the test being async and using
await Task.WhenAll
instead ofTask.WaitAll
? If that doesn't work, could be some kind of threading bug in 1.7There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@madelson Yes,
MergedLinesEnumerableTestAsync
'sFuzzTest
fails butMergedLinesEnumerableTestSync
'sFuzzTest
is fine. It fails even when it's run alone, but I did notice thatTestPipeline(2)
always fails with it if all tests are run together:There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@madelson I somehow got the test to pass with
await Task.WhenAll(task1, task2, consumeTask);
! Not sure if this is the fix or it means something else needs to be fixed, but this does look promising.Let me know what you think! (FWIW the test didn't pass within 10 secodns with
if (random.Next(4) == 1)
.)