Merge of directory level

[marc0janssen]

Hi, I would like mergerfs to omnit merging a directory if a directory with the same name exists in the second branch.

For example.

brach1
/--dir1--file1
| file2
|-- dir2

branch2
/--dir1--file3
|
|--dir3

wil add up to:
result:
/--dir1--file1
| file2
|-- dir2
|--dir3

So dir1--file3 from branch2 is not in the result.

Is this possible?

[trapexit]

No. It does not support that. If something isn't described in the documentation is doesn't exist.

What is the usecase? The whole point of the software is to create a union of the content. Why do you want it not to?

[marc0janssen]

For me it would be necessary that the files in branch1 are "more wanted" than the files in branch2.... SO I do't want to see these files from branch2... The missing files in branch2 are wanted.

I want this so, media with a higher quality is not mixed with media from in a lower quality in branch 2.

Does that make sense?

[trapexit]

Not exactly. Why not just delete the file or move it outside the location of the service scanning those paths?

How would mergerfs know which directory to include in the readdir?

You want a union of all branches but with only ever 1 directory's content? That doesn't really work as a blanket rule. It would fail immediately. The root path is naturally shared across all branches.

[marc0janssen]

You could make branch dominant of the other....

If the dominant branch has the "dir1" then this is in the result of the merge.... in favor of branch2.

[trapexit]

Yes, but what is that algorithm? mergerfs has policies for many things. You need to clearly define how it would choose.

[trapexit]

As I said above... it would break immediately to just "first found". / is on all branches. So it would pick the first branch and you'd have nothing merged.

[marc0janssen]

Can’t you say: If directory exist in dominant branch then this shown in favor of branch2 If directory does not exist in dominant branch, then the version of branch2 is shown (if it exists there). So the merged directory is still the sum of dirs in both branches.. only for dirs that do not exist in both branches... plus the dirs that are in the dominant branch if they exist in both branches

…

Op 8 dec 2023, om 15:48 heeft trapexit ***@***.***> het volgende geschreven: Yes, but what is that algorithm? mergerfs has policies for many things. You need to clearly define how it would choose. — Reply to this email directly, view it on GitHub <#1286 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABMFKPEARIJV5L2J4L5B3N3YIMSENAVCNFSM6AAAAABAMX6JE6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMBQGU2TM>. You are receiving this because you authored the thread.

[trapexit]

There is no such thing as "dominant branch." There is a list of N branches. I'm asking you to define the way in which you choose your "dominant branch." Given a list of branches how would you choose the content to show? Look at mergerfs policies for examples. You have to explain how to sort and/or filter the branches.

And you're talking only about directory listing. That is entirely separate from all other functions. open, rmdir, create, etc all already have a bunch of different of policies. You would want/need those to functionally match.

[marc0janssen]

Ok Trapexit sorry for my shortcoming and thinking out of the box.

I guess it is not possible if I see your responses...

Have a great weekend!

[trapexit]

I never said it wasn't possible. I'm asking you to define your algorithm for it to work.

[marc0janssen]

I  was thinking. in only a master and a slave. so two branches not a bunch of them SO this feature will only work with 2 branches.  a master and a slave.

…

On 8 dec 2023 16:00 +0100, trapexit ***@***.***>, wrote: I never said it wasn't possible. I'm asking you to define your algorithm for it to work. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[trapexit]

I'm not sure I understand why that would be. You have an ordered list of branches. If you look at existing policies I already provide diverse collection of behaviors. "first found" works just fine in your theoretical "master / slave" setup. But again... that doesn't work in all cases (like the root path) hence my asking for more details.

[marc0janssen]

Ok Trapexit... I will look a bit better at "first found" but I this it will still try to mix the files for those directories involved.

I investigate it again....

[marc0janssen]

First found, will pick the branch that is first in the list, and uses this to add the new files, right?

If so, not what I mean.

[trapexit]

You need to understand that every function in mergerfs has it's own policy and behavior. This is what I was saying above. Just because you list (or don't list) something doesn't mean anything with regard to opening that same file path. What you seem to talk about is only the readdir behavior. Which is fine... but it isn't the whole story. And even just for readdir their would need to be very specific details about your expected behavior. How would you choose what to present given a list of branches? First one found? One with the most files in it? WIth the biggest files? There are a bunch of information to make the decision and existing policies for other functions use some of them.

[trapexit]

First found, will pick the branch that is first in the list, and uses this to add the new files, right?

You are conflating policies with the function. They are not the same. A function has a policy. Software calls "create" and the policy is how mergerfs decides how to behave. Choose what branch to use. But ff policy for "open" has nothing to do with new files. It has to do with which file is opened.

[marc0janssen]

It seems so Trapexit… I will find another way for my needs, Thanks for thinking with me in this matter. Succes with Mergerfs!!!

…

Op 8 dec 2023, om 16:17 heeft trapexit ***@***.***> het volgende geschreven: You need to understand that every function in mergerfs has it's own policy and behavior. This is what I was saying above. Just because you list (or don't list) something doesn't mean anything with regard to opening that same file path. What you seem to talk about is only the readdir behavior. Which is fine... but it isn't the whole story. And even just for readdir their would need to be very specific details about your expected behavior. How would you choose what to present given a list of branches? First one found? One with the most files in it? WIth the biggest files? There are a bunch of information to make the decision and existing policies for other functions use some of them. — Reply to this email directly, view it on GitHub <#1286 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABMFKPFTGKQPQPWKSKRGWSLYIMVSFAVCNFSM6AAAAABAMX6JE6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMBQHA4DK>. You are receiving this because you authored the thread.

[trapexit]

If you don't find a good way to do it otherwise feel free to come back and we can discuss further.

[marc0janssen]

I will Trapexit... have a great weekend!

…

On 8 dec 2023 16:36 +0100, trapexit ***@***.***>, wrote: If you don't find a good way to do it otherwise feel free to come back and we can discuss further. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[marc0janssen]

I have it fixed now with overlayfs.... working like a charm

…

On 8 dec 2023 16:36 +0100, trapexit ***@***.***>, wrote: If you don't find a good way to do it otherwise feel free to come back and we can discuss further. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[trapexit]

How? I'm not aware of overlayfs having such a feature.

Directories¶
Overlaying mainly involves directories. If a given name appears in both upper and lower filesystems and refers to a non-directory in either, then the lower object is hidden - the name refers only to the upper object.

Where both upper and lower objects are directories, a merged directory is formed.

At mount time, the two directories given as mount options "lowerdir" and "upperdir" are combined into a merged directory:

mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,workdir=/work /merged

The "workdir" needs to be an empty directory on the same filesystem as upperdir.

Then whenever a lookup is requested in such a merged directory, the lookup is performed in each actual directory and the combined result is cached in the dentry belonging to the overlay filesystem. If both actual lookups find directories, both are stored and a merged directory is created, otherwise only one is stored: the upper if it exists, else the lower.

Only the lists of names from directories are merged. Other content such as metadata and extended attributes are reported for the upper directory only. These attributes of the lower directory are hidden.

[trapexit]

You are using whiteouts / opaque to individually tag every file?

[marc0janssen]

Hi Trapexit... I got it working like Want with Mergerfs... Thanks for your patience and keep up the good work.

[marc0janssen]

Overlayfs did not to what I wanted after all.. I jumped too high too soon.... I think.... Mergerfs does the trick... have not tested it long enough.. but it looks promising....