Allow for atomic appends to end of file:: Round 2

[JoeShook]

Link to original proposal: #53432 (comment)

I am not sure why this proposal never resulted in a provision to enable atomic appends, at least as an opt-in. The discussions on this topic are quite technical and dig deeper into the API's and OS layers than I feel comfortable explaining. But let me just explain my simple scenario and why it blocks me from deploying dotnet core to Linux, at least in my environment.

Environment

In our Linux environment we control the life cycle of our applications in our own way. It has been the way for deploying Java apps for some number of years. So, lets rule out containers or cloud. Logs are rotated with a utility called "logrotate". This is all orchestrated for all apps and again has worked great for years.

It is natural then to start migrating dotnet core apps to Linux. Well imagine our surprise when before long apps with any kind of logging volume, during the moment logs are rotated we begin to find log files growing much larger than the limit. When we investigate the logs, we find them filled with NUL bytes. Basically, if the logger is writing 50MB deep into the file during log rotation, you will get a new file with 50MB of NUL bytes and then continue from there. It is a mess. After searching around, I found my way to the original proposal and this issue in serilog-sinks-file/issues/221.

Interim Solution

Of course one could just control the log rotation themselves. But we want to keep things consistent. After all, one reason to consolidate on Linux is for consistency. So, we decided to just write to console (stdout). It is a good long term way to go after all, if we plan on going to containers. It turns out that redirecting output to a log file with ">>" solved our problem.

W3CLogger

I am checking out W3CLogger . After moving off IIS and running behind Kestrel only, it was logical to investigate lighting up a w3c style log like we get from IIS. But the ASP.NET Core W3CLogger requires a file to log to. Ugh! Back to wishing I could turn on atomic appends.

Please reconsider enabling atomic appends as an opt-in.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Link to original proposal: #53432 (comment)

I am not sure why this proposal never resulted in a provision to enable atomic appends, at least as an opt-in. The discussions on this topic are quite technical and dig deeper into the API's and OS layers than I feel comfortable explaining. But let me just explain my simple scenario and why it blocks me from deploying dotnet core to Linux, at least in my environment.

Environment

In our Linux environment we control the life cycle of our applications in our own way. It has been the way for deploying Java apps for some number of years. So, lets rule out containers or cloud. Logs are rotated with a utility called "logrotate". This is all orchestrated for all apps and again has worked great for years.

It is natural then to start migrating dotnet core apps to Linux. Well imagine our surprise when before long apps with any kind of logging volume, during the moment logs are rotated we begin to find log files growing much larger than the limit. When we investigate the logs, we find them filled with NUL bytes. Basically, if the logger is writing 50MB deep into the file during log rotation, you will get a new file with 50MB of NUL bytes and then continue from there. It is a mess. After searching around, I found my way to the original proposal and this issue in serilog-sinks-file/issues/221.

Interim Solution

Of course one could just control the log rotation themselves. But we want to keep things consistent. After all, one reason to consolidate on Linux is for consistency. So, we decided to just write to console (stdout). It is a good long term way to go after all, if we plan on going to containers. It turns out that redirecting output to a log file with ">>" solved our problem.

W3CLogger

I am checking out W3CLogger . After moving off IIS and running behind Kestrel only, it was logical to investigate lighting up a w3c style log like we get from IIS. But the ASP.NET Core W3CLogger requires a file to log to. Ugh! Back to wishing I could turn on atomic appends.

Please reconsider enabling atomic appends as an opt-in.

Author:	JoeShook
Assignees:	-
Labels:	area-System.IO, untriaged
Milestone:	-