Content addressing and adding to store cleanup

[Ericson2314]

Motivation

Organize content addressing

We now better distinguish ways to:

• Content address a store object (to CA store path):
  • Text
  • Flat
  • NAR
  • (someday soon) Git
• Content-address a file system object (to hash):
  • Text
  • Flat
  • NAR
  • (someday soon) Git

The former has a new underlying method to give the method it uses for the latter.

Use SourceAccessor with Store::addToStore

This helps promote SourceAccessor --- eventually (I think) we want just about all file system access to go directly through it and not use native operations directly. This will help us with in-memory story (good for unit tests!) and dovetails with #9278.

Remove now-redundant text-hashing store methods

addTextToStore and computeStorePathFromDump are now redundant because the other methods are upgraded to take a ContentAddressMethod not FileIngestionMethod --- we have a single suite of methods that works with all types store object content adressing, and we don't need any additional methods for the Text case.

Context

I was reminded how much duplicated code we had working on #8918. This is a long standing issue and it's high time we clean it up. Cleaning it up first will also make it easier to review git hashing support to libstore.

Priorities

Add 👍 to pull requests you find important.

[edolstra]

Just to note but some of this code is here to minimize the diff with lazy-trees. So if we start changing this, it will increases the lazy-trees diff again.

[Ericson2314]

@edolstra happy to fix that merge conflict :)

[Ericson2314]

Right now there is a conflict in lazy trees from #8848 that is very domain-specific on both side. I can't help with that so much, but I am happy to do all the other conflicts!

[Ericson2314]

fetchToStore stays non-virtual on lazy-trees so I think there is no issue here. This PR makes Store able to consume SourceAccessors directly so we just use that. Anything in an InputAcessor beyond the underlying SourceAccessor Store::addToStore won't care about anyways.

[Ericson2314]

For the fingerprinting, Sure that method could go back, but we could also flip around the logic so the SourceAccesor itself has a maybeLstat-like method to try to cheaply content-address file system object, and the InputAccesor implementations do the getCache() to look this up.

[edolstra]

On lazy-trees, InputAccessor::fetchToStore() implements a SQLite cache for SourcePath -> StorePath mappings that I want to backport to master. It optimizes stuff like src = <some big tree>. That's why it's important not to get rid of fetchToStore().

[Ericson2314]

OK I'll put it back

[Ericson2314]

It is now put back (not sure why this thread isn't marked outdated).

[roberth]

Not on a red or green line, is probably why.

[Ericson2314]

Fixed the bug!

[roberth]

Just one suggestion.
Great stuff: less code, more consistent use of fewer interfaces, progress towards in-memory store.

Ok to merge after the suggestion and resolving the work priority inversion. It seems that you can't help @edolstra with lazy-trees until the existing conflict is resolved.

[Ericson2314]

@amjoseph-nixpkgs BTW this addresses some duplication you brought up a while back :)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-11-17-nix-team-meeting-minutes-104/35753/1

[Ericson2314]

Just rebased this fixing conflicts. it would be very good to get it reviewed @edolstra so I can continue with the rest of the git hashing stuff!

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-11-27-nix-team-meeting-minutes-107/36112/1