Question: How to reliably write to a file in ZFS

[jean-airoldie]

Hi, I'm trying to find some documentation concerning behavior not specified by POSIX. I found lots of conflicting information online and nothing official. Specifically I would like to know what is the minimum set of syscalls required to write to a file atomically using ZFS, and I don't care about my code not being portable to another file system. By atomicity, I mean that in the event of a crash, either the file system detects and removes the write that couldn't be completely written to disk, or that a user space application has the ability to detect and remove the corrupted file.

For some context, since POSIX is dogshit, you need to go through multiple hoops just to be able to persist your file correctly in the event of a crash. This blog post explains this in more details. But even if you do everything correctly, you still run in the issue of most classical file systems potentially corrupting themselves on power failure (even XFS AFAIK).

I would like to know:

• Does ZFS guarantee that it will never get corrupted in the even of a power failure (or OS/FS crash), at least from the perspective of a user space application (e.g. detect and remove the corruption automatically would work)?
• What is the minimum set of syscalls required for a user space application to guarantee that a newly created file has been written to disk atomically? Do I need to do any of the insanity mentioned in the blog post (e.g temporary log file with checksum in the advent of a failure)?
• Is there some other way to use a ZFS file system directly other than by going through the POSIX API that I don't know of that might be more efficient?
• Do I need to disable the write cache for my disk device?

[grahamperrin]

Suggested reading

https://openzfs.github.io/openzfs-docs/man/8/fsck.zfs.8.html

… ZFS datasets are checked by running zpool scrub on the containing pool. An individual ZFS dataset is never checked independently of its pool, which is unlike a regular filesystem. …

#7912 – Feature: Ability to repair defective on-disk data

https://openzfs.github.io/openzfs-docs/man/8/zdb.8.html options -F and -X with regard to transaction rewinds.

In practice (with FreeBSD), I never needed either of those approaches to transactions.

ZFS fundamentals: transaction groups | Delphix (2012-12-11)

[jean-airoldie]

I'm gonna answer my own question in case someone is looking for it. I couldn't find official doc, I just read the code & relied on some user / expert testimony as well as current zfs code.

• Does ZFS guarantee that it will never get corrupted in the even of a power failure (or OS/FS crash), at least from the perspective of a user space application (e.g. detect and remove the corruption automatically would work)?

Basically yes if the drives don't lie about writes. The file system shouldn't corrupt itself normally, except in case of a bug. If the file system does get corrupted, such as in the case of hardware failure, it will get detected, as even the file sytem metadata is checksummed. This is different from classical file systems that are expected to corrupt themselves, thus why fsck is required to be run periodically on such systems.

As for the data, basically the file system will be restored to the last commited transaction group.

• What is the minimum set of syscalls required for a user space application to guarantee that a newly created file has been written to disk atomically? Do I need to do any of the insanity mentioned in the blog post (e.g temporary log file with checksum in the advent of a failure)?

I couldn't find good documentation on this, but the way I'm doing it currently is basically to openat then one big pwrite64 to the file. This isn't purely atomic because there is the possibility that the act of opening the file and then the write are located on two separate transaction groups, but that's good enough. In that case you're just gonna have a empty file, which is pretty easy to deal with. Then you could flush which would basically force & wait for the current transaction group to be committed, assuming the file system is mounted with the right options (sync=always). Since its pretty wasteful, I instead have a single thread in my program that calls zpool sync <pool> in the same way the CLI does it (aka directly via libc::ioctl). This is more efficient because you only block once event if you have tons of pending flushes.

• Is there some other way to use a ZFS file system directly other than by going through the POSIX API that I don't know of that might be more efficient?

Couldn't find anything on this. However, see the previous paragraph about the zfs sync which is basically a fsync but for the entire pool.

• Do I need to disable the write cache for my disk device?

Nope.

[jean-airoldie]

Also see this reddit post that has some decent comments https://www.reddit.com/r/zfs/comments/m3d6ao/looking_for_doc_about_zfs_file_durability/.