Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

delay write of all metadata #7

Open
grustam opened this issue Mar 18, 2018 · 5 comments
Open

delay write of all metadata #7

grustam opened this issue Mar 18, 2018 · 5 comments

Comments

@grustam
Copy link

grustam commented Mar 18, 2018

will be good to have option to delay writing of all of metadata (Directory and FAT entries).
So it will be possible to sequential write data of many files without interrupting to write its metadata.
Sequential write is huge faster than write non-sequentially relatively small chunks.
Flush metadata either explicitly on convenient time or optionally by configurable timer.
Timer can be different for different operations, like 30 seconds for adding/moving/renaming/changing files and 1 second for deleting/shrinking files.
Either to have 2 streams, one for file data and second for metadata, so stream for metadata can be buffered.

@picrap
Copy link
Owner

picrap commented Mar 18, 2018

I think a full delay is a good idea, however, let's not confuse responsibilities: if you want an efficient cache, this is probably another product you need to look for (something that would cache all writes and flush them to disk in an ordered way in order to reduce seek delay, for example) on which you would back the ExFat library.
So to me a good option in our scope is to allow caching everything, which is already the case 😉.
I need to think more about this.

@grustam
Copy link
Author

grustam commented Mar 20, 2018

My thoughts for thinking :)
Write cache is good solution but not the best in case of updating volume with many files - open volume, read directory entries, delete part of directories and files, replace or create many new files and directories, close volume.
With LRU/LFU block write cache need to consider that:
File system allocates space for directories in random places, so it is almost impossible to write it consecutively and all at once.
With postponed write of metadata:
File system can keep in memory all new/modified directory entries without real allocating space until flush and then do allocation and do contiguous write of metadata. Optionally relocates all fragmented directories to consecutive block, so potentially all directories can be in single consecutive block.
Windows utilize file cache feature which rushes to write all of metadata multiple times and then file data, because it worries much about file system consistency. So file can be exist but without correct data.
With suggested approach USB stick can be created much faster, ideally with a few consecutive block writes and in case of failure file system will not show new files which for me is better than Windows approach.

@picrap
Copy link
Owner

picrap commented Mar 20, 2018

And are you interested in contributing to the project? 😉

@grustam
Copy link
Author

grustam commented Mar 20, 2018

I don't have time currently, but if you don't mind I can develop and contribute changes in which I interested.

@picrap
Copy link
Owner

picrap commented Mar 20, 2018

That's the whole point of open source software: anyone can contribute and is usually welcome to!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants