Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix sparse image file cause file system corrputed #5869

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

wangqiang1588
Copy link

fix linux virtual machine (ubuntu 23.10.1) report error

"
EXT4-fs error (device vda2): ext4_validate_block_bitmap:421: comm kworker/u20:0 bg 2230: bad block bitmap checksum
EXT4-fs (vda2): Delayed block allocation failed for inode 14327754 at logical offset 0 with max blocks 47 with error 74
EXT4-fs (vda2): THis should not happen!! Data will be lost
"
the reason is virtual machine disk image is created as sparse file

truncate command make a sparse file, the space will not alloc before really used
this should be better at most time.
but maybe not suitable for virtual machines, especially in the case of heavy IO loads
this may give extra time delay and operational interruptions when system do really space alloc
this behavior may cause later write completed before previous write
the incorrect write order may cause file system corrputed

osy and others added 7 commits October 27, 2023 20:57
"
EXT4-fs error (device vda2): ext4_validate_block_bitmap:421: comm kworker/u20:0 bg 2230: bad block bitmap checksum
EXT4-fs  (vda2): Delayed block allocation failed for inode 14327754 at logical offset 0 with max blocks 47 with error 74
EXT4-fs  (vda2):  THis should not happen!! Data will be lost
"
the reason is virtual machine disk image is created as sparse file

truncate command make a sparse file, the space will not alloc before really used
this should be better at most time.
but maybe not suitable for virtual machines, especially in the case of heavy IO loads
this may give extra time delay and operational interruptions when system do really space alloc
this behavior may cause later write completed before previous write
the incorrect write order may cause file system corrputed
@wangqiang1588
Copy link
Author

Screenshot

@wangqiang1588 wangqiang1588 marked this pull request as draft November 21, 2023 08:55
@wangqiang1588 wangqiang1588 marked this pull request as ready for review November 21, 2023 08:56
@gnattu
Copy link
Contributor

gnattu commented Nov 22, 2023

I'm afraid using this does not help with extreme io cases. By running stress-ng --iomix 2 on a btrfs fs would still result in fs error after a couple of minutes.
Screenshot 2023-11-23 at 06 46 34

@wangqiang1588
Copy link
Author

we do test on EXT4 , it works better. test use EXT4 fs ?

@gnattu
Copy link
Contributor

gnattu commented Nov 23, 2023

May I ask which hypervisor you are using? If it's Apple Virtualization, simply using a non-sparse file might not be sufficient to prevent filesystem errors. We've had an extensive discussion in #4840. I notice you are using a raw image, so I assume you are using Apple Virtualization because the default image format for QEMU is qcow2.

@wangqiang1588
Copy link
Author

we are using Apple Virtualization , we try to build Android AOSP on mac M2

…default value),may cause linux file system corrputed, especially in the case of heavy IO loads
@wangqiang1588
Copy link
Author

we have try to set VZDiskImageCachingMode to uncached , found it works more better , please try this modify

func vzDiskImage() throws -> VZDiskImageStorageDeviceAttachment? {
    if let imageURL = imageURL {
        if #available(macOS 12, *) {
            /*
             * virtual disk cache mode have bugs,
             * when it is enabled or set to auto (default value)
             * may cause linux file system corrputed, especially in the case of heavy IO loads
             */
            return try VZDiskImageStorageDeviceAttachment(url: imageURL, readOnly: isReadOnly, cachingMode:VZDiskImageCachingMode.uncached, synchronizationMode: VZDiskImageSynchronizationMode.full)
        } else {
            return try VZDiskImageStorageDeviceAttachment(url: imageURL, readOnly: isReadOnly)
        }
    } else {
        return nil
    }
}

@gnattu
Copy link
Contributor

gnattu commented Nov 23, 2023

We already discussed and tried this approach before: #4840 (comment)

We don't get filesystem error initially, but after a reboot we still have a filesystem error showing up. The most reliable way I've found is to switch to VZNVMExpressControllerDeviceConfiguration instead of the VZVirtioBlockDeviceConfiguration for Linux VMs, as demonstrated in pr #5919. But the nvme device config is only available on macOS 14+ host. We also tried to patch the kernel to throttle cache flushing frequency and it makes the heavy-io workloads much more stable, but it does not fix the issue 100%.

@wangqiang1588
Copy link
Author

maybe we should defalut disable cache below macOS 14 and switch to VZNVMExpressControllerDeviceConfiguration on macOS 14+

for now we test aosp build works better than cache enabled

@wangqiang1588
Copy link
Author

we found when cache disabled , after a reboot we do'nt found filesystem error showing up again on macOS 14.1.1 host. maybe apple have fixed this problem on macOS 14.1.1

@wpiekutowski
Copy link

wpiekutowski commented Nov 23, 2023

For me, the following combinations work reliably:

  • NVMe with any caching mode
  • virtio with cached caching mode

Details: #4840

@wpiekutowski
Copy link

Anyway spare files are interesting and would be cool to have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants