How would you create a merge of two filesystems, where one filesystem is always used except for a few selected files?

[danielkrajnik]

I'm trying to merge two filesystems. One is "hot" and used every day, second one "cold" and used rarely only as an archive of large files.

You want to see both of them in one "tree" and I thought that you can use mergefs for that?

For example:

Cold:
-dir
|_file1 4GB

Hot:
-dir
|_file2 1KB
|_file3 1KB

Merge:
-dir
|_file1 4GB
|_file2 1KB
|_file3 1KB

Would it be possible to have the "hot" filesystem always active? "Cold" filesystem would be then only used explicitly by the user for a few selected files?

I think that typically mergeFS follows a "policy" to decide where to write each file based on criteria (e.g. free space), but I'm wondering if it's possible to leave it to the user who would explicitly make that decision for each file (and mergefs would always write to one filesystem by default)?

[trapexit]

No, there is no file level choices to anything. You're the first person ever to ask for such a thing. At most people have asked about files matching some pattern.

I still don't really understand what you mean by "active". If you combine things together they are expected that you want to use them. Otherwise why are you including them? Filesystems have a lot more functionality than creating or writing to a file. You have to explain at the function level what you want.

Lots and lots of people combine multiple filesystems together (often local and remote) and have the remote effectively read only. That's the whole point of all the mode values per branch. And they write to the remote out of band whenever they wish to. But I don't know under what situation you would want file "abc" to be interacted with only to one branch but everything else on another branch. I suppose hierarchical policies could do that but that would be a very invasive rewrite of the software.

[trapexit]

You need to describe things as input and behavior. Give input X, Y, Z it chooses branch B.

mfs... given the absolute free space of all branches choose the one with the large value.

[danielkrajnik]

Thanks a lot for explaining this. It sounds like I've had a different use case in mind though:

That's the whole point of all the mode values per branch

When you have a few files in a folder and you want to archive (move to the second remote) only a few of them this sounds like it won't work. You wouldn't want to move the whole folder neither.

write to the remote out of band

I hoped that this can be integrated, because traversing many nested directories just to change remote sounds really tedious when done frequently.

If you combine things together they are expected that you want to use them

I assumed that you can still see what files were archived (moved to a remote) in a directory and retrieve them when needed. If you move something completely you may never discover it again.

Not really sure where to find a solution. You are right that I've never seen anyone merge filesystems and allow "file-level" control. I think that this would be really useful though.

[trapexit]

I hoped that this can be integrated, because traversing many nested directories just to change remote sounds really tedious when done frequently.

Integrated how? A list of files that mergerfs should treat differently? Whether it was part of mergerfs or you did it out of band you still have to list out N file paths. You aren't describing a set of characteristics on which mergerfs could somehow know to treat these things differently.

It would help if instead of describing a solution you described the problem.

[trapexit]

I assumed that you can still see what files were archived (moved to a remote) in a directory and retrieve them when needed. If you move something completely you may never discover it again.

I'm not exactly sure what you mean by that last part. Yes... mergerfs combines the output of readdir for every relative path found in the branches configured. It's no different from you doing ls /mnt/a/ and ls /mnt/b/ and combining them logically.

[trapexit]

Even if you had the ability to treat individual files differently... differently how?

[danielkrajnik]

If you move something completely you may never discover it again.

I'm not exactly sure what you mean by that last part.

Sorry, I meant that even though one remote is an "archive", which you may use infrequently you still want to see its files in their original locations. It was following:

If you combine things together they are expected that you want to use them. Otherwise why are you including them?

Two remotes just behave quite differently (one is space-limited and other one is very slow).

[danielkrajnik]

It would help if instead of describing a solution you described the problem.

The two remotes in question are constrained: Google Drive allows for maximum 5TB and AWS Glacier takes a few hours to retrieve files. Their advantages are: Google Drive retrieves files instantaneously and AWS Glacier can store hundreds of TBs.

Ideally it would be possible to have one single tree that would share files between two remotes. Sharing wouldn't be based on a policy however, but user could pick and choose, which files should be stored in each remote, mostly to go below Google Drive's 5TB limit.

It would be useful if it was possible to move specific files, because often only a few files take the larger amount of space (virtual machine images, cache files, movies) .

I might have started with incorrect assumptions, which must have made this question sound confusing. Thank you for explaining mechanism of mergefs and I'd be grateful to hear your thoughts on how to approach this problem (with mergefs or otherwise).

[trapexit]

It would be useful if it was possible to move specific files,

There is no movement of any files currently.

Your "problem" isn't unique to having two remotes. That's fundamentally the same as every other person using mergerfs with some smaller fixed size local storage and then some large/"unlimited" cloud storage. Probably thousands of people use mergerfs for that setup. That's why rclone added their union feature. But I still don't understand what you want from the software.

A filesystem is made up of different functions. You have to speak about your expectations/requirements in that sense. You open a file... what happens. A file is created. What happens? An unlink is requested for "foo/bar/baz"... what happens?

https://github.com/trapexit/mergerfs#terminology
https://github.com/trapexit/mergerfs#functions-and-their-category-classifications

[danielkrajnik]

Navigation between read only mergerfs and read-write corrseponding directory is what confuses me a little. How people switch between long nested folders - just practically - do you copy the path, remove the mount point prefix and add the read/write corresponding filesystem each time? I suppose you can automate it in a shell script (I've been using zsh's zle functions with some success for that), but what about all other file manager (terminal ones, GUIs). It sounds much more tedious outside of shell.

I was really hoping to see some recorded demos on youtube of people's union filesystems setups, but these are unfortunately very rare. I realize that I may be asking completely unrelated questions and there may be very simple solutions for this already.

[trapexit]

The process is no different than any other copying between locations. Nothing different from common, standard rclone, rsync, etc. usage.

rclone src:foo/bar/baz dst:foo/bar/baz

So yes... people write scripts / apps to do whatever movement of files they want. No different from backups, cleanups, dedupes or any other thing.

There are lots of videos/articles on union filesystems and mergerfs specifically but what you're describing is just standard fair with filesystems. There is nothing specific to mergerfs here.

https://github.com/trapexit/mergerfs/wiki/Tutorials,-Articles,-Videos,-Podcasts

[danielkrajnik]

https://github.com/trapexit/mergerfs/wiki/Tutorials,-Articles,-Videos,-Podcasts

Thank you, I haven't seen this.

rclone src:foo/bar/baz dst:foo/bar/baz

If that's the standard way I think that's fine, I was just hoping that you may do this instead:
rclone switch foo/bar/baz dst

Another confusion was how do you know where any one file is stored when you are using multiple src branches (based on a policy). I assume that depending on free space file may be stored anywhere between available sources.

It would be less error prone I think. But I will go through the tutorials and maybe the other way is better.

[trapexit]

You could put in a feature request to have something like rclone mv --mirror-path src:foo/bar/baz dst: or something but given that's as easy to manage yourself I don't think anyone cared enough to add it. If you already know the source path I'm not sure the complication in reusing the path for the destination.

Another confusion was how do you know where any one file is stored when you are using multiple src branches

Because you as the person who is making up the arbitrary rules as to what goes where knows those arbitrary rules and can code them into the script doing the movement. I still feel like you are massively over-complicating or misunderstanding this or making assumptions about what I understand you to what.

Someone has local disk and gdrive. They want to store files on local disk first when they download them and then move to gdrive when finished torrenting or something. Your source is local... your destination is gdrive. It's simple. rclone move local:/media/ gdrive:/media/ or whatever particular command you want. Exactly the same as copying data between any two other filesystems. A USB drive and your SSD. Cloud and local. Dropbox and Jottacloud. Floppy disk and zip drive. There is absolutely nothing special here. Define your source locations, define your destination locations. Unless you can define logic to find the files by looking at their metadata or data then you have to expressly define your setup. If you want to drive from your home to the mall you have to provide a list of locations and movements. No different here.

[danielkrajnik]

If that's the common practice then I will give it a try. I assumed perhaps incorrectly that you create a central "monorepo" which has special interfaces to move files across its "backends". But I may be overcomplicating this and it's not such a big problem to do the same manually and just add name to the files file1-ARCHIVE file2 file3-ARCHIVE etc.

[trapexit]

Sorry again for stating something obvious, but I am not sure when creating a new file in mergerfs how do you know where it will be stored? Is it dependent on it's location in the "tree" (sorry for wrong terminology) or does it depend on the mergerfs policy? Is there no way to decide where it will be stored on a "file-level"?

A program asks the kernel to create a file, make a directory, whatever. That request is sent to mergerfs. mergerfs runs the policy you set for create and determines the underlying branch to do the create/mkdir/etc. and does it. Returning the details required back. That's how everything works.

Of course it can decide on a file by file basis what to do. That's exactly what the policy does. But you need to articulate the algorithm to use. It doesn't magically know where you want files. It has to be able to determine it from the information it has. You have to tell me how you would decide how something would be placed on branch A vs B vs C. As I asked before... given what information do you make the choice?

The file size? the name? the age of the directory? the time of day? ... f(x) = y... what is x? what is f(x)?

[trapexit]

So you want to create a config file with a bunch of mappings for specific filepaths to branches on the chance you're going to write that specific file... rather than just write the file to the target filesystem? That seems like a very roundabout way of doing something very simple. You clearly aren't talking about a situation where that file is unlinked and created over an over. And you're not talking about a class of files. Like "all files ending in .png should go to branch X".

At some point I may add the ability to define different config for different parts of the tree based on regex or fnmatch patterns which could theoretically allow for this but I certainly would not add specific file to branch mapping. This is a very very bespoke usecase.

[danielkrajnik]

I think that you are right, I may just not know any better way to manage it. In the other thread you mentioned having mergerfs as a separate read-only view of the system and use separate read-write filesystems. I suppose that would be better way to do it although as mentioned I am not sure how one navigates between them.

Maybe there are other tools to "orchestrate" files in mergerfs. I'm clearly missing something obvious here.

[trapexit]

I don't understand what you mean by "how one navigates between them." I feel like there is some fundamental assumption you're making that isn't articulated. It's all just a tree. A standard filesystem layout. Windows, MacOS, Linux... they are all the same in this regard. You just use the paths as you want.

[danielkrajnik]

No worries, I will try the methods you suggested and follow the articles from the link above and maybe I will just find at some point that this all just started with wrong assumptions.

[trapexit]

I'm not sure there is anything about this in any tutorial as it isn't something specific to mergerfs. It's the same pattern with any backup or other transfer of files.

[arakis]

I want to expose mergefs via NFS, and only via NFS, and only that one mount. So the client has no idea that there are multiple branches "behind the scenes".

So I ran into similar problems as @danielkrajnik

So, the only way that could work is to have a script on the server, that will move specific files/folders based on defined rules from one dir(drive) to another. Not sure if this is really the most nice way. I would really wish a way to define a per-file/(sub)-folder policy, defined in a global config file.

[trapexit]

It would help if you defined your problem rather than proposed solutions.

Also, his original question was about specific users writing to specific places through mergerfs. That wouldn't be possible without specific, mergerfs specific interactions and therefore not all that useful. What would you do? You "touch /mnt/mergerfs/foo/bar:0" and it picks branch zero and creates the file there named "bar"? How do you articulate the branch every time? How could that work outside interactive situations? mergerfs could have policies per userid, per relative path, but per file isn't really practical.

[arakis]

Sometimes, it's simpler to ask for a solution rather than explaining the problem. But here we go:

/mnt/cheap-archive --> Cheap to write and store, but very expensive and slow to read (for example, Google Archive Storage tier, or AWS Glacier).
/mnt/expensive-volatile --> Normal disk.

/mnt/merged --> Combines both.

Now, I have these relative paths:

./media/movies --> Should always be placed into the "archive" branch.
./media/pictures --> Always on the "volatile" branch.
./media/disks/*.iso --> "Archive" branch.
./media/disks/readme.txt --> "Volatile" branch.

This is only a "sample". What I mean is: There are dozens of situations where you have a "project folder" (somewhere deep in the tree), but you have mixed tiers: Volatile files like a "project index file", "project work file", and archive files, like untransformed raw data for the project, such as raw meshes, videos, etc.

Without a detailed distinction, you always need to decide: "Do I put the whole project into the archive or the whole project into the volatile?". And of course, you also need to "expose" the underlining folders then...which I do not want!

The client/consumer of mergefs should always be "unaware" and should never care about what is happening behind the scenes. Picking up files at intervals/moving them via a server script could work, but could cause conflicts if a file is being written at that exact moment - I'm not sure how mergefs would resolve this conflict.

So, back to my so-called "solution", which I consider more of an idea or a concept of what mechanics could work: Having a config file with this content:

./media/movies archive
*.iso archive
* volatile

Similar to a gitattribute file. So, what would happen if create is called? It would look up the branch, depending on the relative file name and the config file. That's all. After the file is created, the path never need to resolved again.

[trapexit]

Sometimes, it's simpler to ask for a solution rather than explaining the problem.

It is impossible to suggest or design a solution to an unknown or unclear problem.

The client/consumer of mergefs should always be "unaware" and should never care about what is happening behind the scenes.

It is improbable for mergerfs to manage all usecases nor would I care to try. Custom behavior requires custom solutions. Especially when the API (the filesystem) does not provide any way to present or consume the necessary information to make decisions.

Picking up files at intervals/moving them via a server script could work, but could cause conflicts if a file is being written at that exact moment - I'm not sure how mergefs would resolve this conflict.

This is always a risk. Files are always at risk of race conditions using standard APIs unless carefully accounted for (non-standard exclusive locks, taking advantage of atomic behaviors, etc.) Software must be written to properly deal with atomicity or lack thereof of a POSIX filesystem. Tools like rsync will use some simple logic to try to ensure this doesn't happen... hence why I generally recommend rsync for such situations rather than cp/mv/etc. Point is... mergerfs is not in the business of trying to fix poorly made software.

So, back to my so-called "solution", which I consider more of an idea or a concept of what mechanics could work: Having a config file with this content:

It isn't that simple. create is only one of a dozen or so functions. You have to be able to ensure that these rules apply to them or you need rules for every function. And what if you have 20 branches rather than 2? And how does picking individual branches jive with policies? What happens if you try to rename or link between two paths with different rules? There is a lot more to this than I believe you understand... this is why articulating a problem is better. Providing a "user story". https://xyproblem.info

The user story is, as I understand it, something like: "You wish to be able to target a specific branch for which a create (creat,mkdir,mknod,etc.) will be directed based on user defined regex? / fnmatch? / glob? rules."

And lets not forget that your "archive" storage is not a traditional filesystem, likely using another fuse filesystem like that of rclone which has a bunch of unfun failure conditions and concerns that are kinda outside mergerfs' purview... which is why everyone using the two technologies together code their own side channel behavior to specifically deal with those edge cases. API usage quotas, retries, duplication, writing to multiple targets, etc.