Add option to include untracked files in git and mercurial fetchers

[jfroche]

It can be confusing (especially for new users) that the git and mercurial fetchers do not include untracked files. This commit adds an option for the fetchers to include untracked files.

The option is disabled by default and still take into account the repository ignored files.

Priorities

Add 👍 to pull requests you find important.

This subject has been referenced before in
#8389 #7107 #3231 #6858

[sellout]

I agree that the current behavior is confusing and odd. If there is a line to be drawn, it should be between committed/uncommitted or maybe staged/unstaged. But the current split includes some changes that aren't expected to be committed, but not others. Untracked is odd on its own because there's no way to make it unstaged. The closest you can get is intent-to-add, which stages an empty file, leaving the staged changes in a state that you probably don't want to commit.

It's also odd because the current behavior will fetch the whole tree if there is no repo, but as soon as you git init, the fetcher fetches nothing.

This PR at least gives the option to avoid this oddness, without change the behavior for any existing usage.

[roberth]

I'm in favor of this, but unfortunately it seems that the git part of the implementation will have to be redone after #9240, which is a priority because it is a (long awaited) fix for reproducibility issues with git.

Mercurial might suffer from similar issues, but we haven't made a priority out of stabilizing that fetcher yet.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/the-garn-devlog/36297/2

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/use-nix-file-excluded-from-git/37196/7

[NobbZ]

What is the difference between this flag and using path-type explicitely?

[v-kat]

Is this still planned for release? It'd be helpful with some agenix issues I was facing

[jfroche]

@roberth I realize I may not have been clear earlier. Three days after your comment, I've pushed an implementation that should align with the new approach based on libgit2. I'm curious about the likelihood of it being reviewed and eventually merged. Any insights on this?

[jfroche]

What is the difference between this flag and using path-type explicitely?

New users are most of time unaware of the distinctions between running it within or outside a Git repository, often leading to confusion. This option explicitly clarifies its intended functionality.

This implementation also overlooks files that are specified in the .gitignore.