Outputting streams to a text file in chunks

[mickvikt]

I have a question about chunking a long stream before outputting it to a text file.
Chunk operator returns IStream<Enumerable<TIn>>, but extension method ToTextFileValue accepts IStream<TIn>.

How do I transform IStream<Enumerable<TIn>> to IStream<TIn> to be able to output to file?

Thank you.

[paillave]

Let me get it straight: from one stream of something you want to issue several files of X lines?

Anyway, this page about file creation might help you. Let me know otherwise.

[mickvikt]

Not exactly, I have a long data stream from the database query and I want to output it to one text file.
The problem is that I get Stream too long or Out of memory errors.
I thought that chunking the stream could mitigate this problem.

[paillave]

Ah ok, I'll come back to with an example. It's an interesting use case 🙂

…

________________________________ From: Viktoras Mickūnas ***@***.***> Sent: Saturday, October 14, 2023 9:50:59 AM To: paillave/Etl.Net ***@***.***> Cc: Stéphane Royer ***@***.***>; Comment ***@***.***> Subject: Re: [paillave/Etl.Net] Outputting streams to a text file in chunks (Discussion #475) Not exactly, I have a long data stream from the database query and I want to output it to one text file. The problem is that I get Stream too long or Out of memory errors. I thought that chunking the stream could mitigate this problem. — Reply to this email directly, view it on GitHub<#475 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABDFY2HLJGWSD6HBGU62C5LX7I76HAVCNFSM6AAAAAA56XZGUSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TENZZGIZTO>. You are receiving this because you commented.Message ID: ***@***.***>