Split into core and generics packages

[matchwood]

See #70 and #61 .

The goal would be to provide two packages:

sop-core (or a better name) - definition of data structures and classes for working with them: NP, HTraverse etc
generics-sop - depends on sop-core and re-exports everything so the API remains the same. Any generics specific code will live here.

Question: What should the core package be called?

I am keen on providing specialisations for heterogenous lists and extensible records in the core package, as well as curry/uncurry functionality. The core package will then be in the same category as packages like HList and Vinyl. My reasoning for this is that the sop approach in generics-sop is great, but at the moment it is not immediately obvious that it can or should be used for general purpose programming with heterogenous data structures. The name should ideally be something that reflects its general purpose possibilities, rather than something specifically related to generics.

[matchwood]

@kosmikus A chat would definitely be good -I will email you about this.

[adamConnerSax]

I think "sop-core" is pretty good. It leaves the generic part to the generics-sop part.
I would be happy with this split as well since, as was pointed out, this would allow a TH free subset to be available for compilation in environments without TH (cross-compiling for IOS, e.g.,). The one puzzle I would still have is that the code I'm hoping to compile in more environments still depends on generics-sop.Generic. E.g., f :: (Generic a, Generic b, Code a ~ SomeTypeFamily (Code b)) => a -> SomeType b
Maybe I can make it polymorphic over to/from functions? That could substitute for the the Generic constraint, I guess?
Like f :: (a -> SOP Identity) -> (SOP Identity -> b) -> a -> SomeType b ? But do I still need a way to capture the Code a ~ SomeTypeFamily (Code b)? Maybe I don't...
Anyway, I support the split, regardless. I love the library for it's use as a heterogenous container. I just also use the generics piece.

[kosmikus]

I think prefer sop-base over sop-core, but both work.

[kosmikus]

Another name that has been suggested for the base package is simply sop.

[matchwood]

I was thinking about sop as well. I prefer this actually, as it sounds like a standalone package, rather than just some underlying component of something else.

[matchwood]

On a related note, what would you like the module names to be? If we call it sop then just SOP.NP, SOP.Classes would make sense.

[kosmikus]

While I'm somewhat guilty in advocating to steal the unofficial top-level identifier Generics for various generic programming packages I co-authored, I don't think that SOP is a good top-level choice.

I see the following options:

• keep Generics.SOP.NP which preserves compatibility and denotes the primary motivation, but might otherwise be a misnomer; I think if we do not go with this, we at least have to mirror the modules under these names, either from sop(-base) or generics-sop, possibly with a deprecation warning;

• use Data.SOP.NP which correctly places the sop library under the Data hierarchy, which I think is adequate. Products and sums are data structures. Having the SOP component in there still allows us to keep all the package modules in a similar place.

• use Data.NP, but this has a larger possibility of conflicts and also makes it unclear what to do with modules such as Generics.SOP.Classes.

[phadej]

Data.SOP is the best compromise IMO

…

Sent from my iPhone

On 24 Aug 2018, at 17.05, Andres Löh ***@***.***> wrote: While I'm somewhat guilty in advocating to steal the unofficial top-level identifier Generics for various generic programming packages I co-authored, I don't think that SOP is a good top-level choice. I see the following options: keep Generics.SOP.NP which preserves compatibility and denotes the primary motivation, but might otherwise be a misnomer; I think if we do not go with this, we at least have to mirror the modules under these names, either from sop(-base) or generics-sop, possibly with a deprecation warning; use Data.SOP.NP which correctly places the sop library under the Data hierarchy, which I think is adequate. Products and sums are data structures. Having the SOP component in there still allows us to keep all the package modules in a similar place. use Data.NP, but this has a larger possibility of conflicts and also makes it unclear what to do with modules such as Generics.SOP.Classes. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[edsko]

Another vote for Data.SOP :)

[matchwood]

I'm happy with Data.SOP as well. And yes I was planning to re-export everything in Generics.SOP so the interface for generics-sop won't change at all.

[matchwood]

First steps here.

Outstanding questions:

1. Version numbers. Should sop release at 0.1? Or should we keep it in line with generics-sop, and move them both to 0.4? Or something else?

2. Documentation: some of the documentation in sop is generics specific, but @kosmikus suggests that a separate documentation branch would be better for resolving this (I guess this would cover updating readmes etc as well). I simply re-exported Data.SOP modules as Generics.SOP modules where relevant - should each of these have a cursory documentation (like "Export of core module for api consistency") or is it fine to just leave them as they are?

3. sop tests. It feels strange not to have any tests at all. I'm happy to write some - any thoughts on what would be worth covering?

4. transformers dependency. I haven't investigated thoroughly, but the only import I have found so far is Data.Functor.Classes in Data.SOP.BasicFunctors, which is in base from 4.9. Is it worth conditionally excluding transformers in sop.cabal? And I think it can be removed entirely from generics-sop.cabal.

[kosmikus]

Thanks a lot.

1. After thinking about this for a while longer, I think having the same version number for both makes sense here, but I won't have the goal to keep them in sync. They both share common history though, and that's best clarified by keeping the Changelog up to this point for both (as you suggested), and having the same version number for now. I'm inclined to go to 0.4 for both, even if it may not strictly be required. But introducing a new library dependency is a significant change in my opinion. @phadej may have a different opinion here, though.

2. Yes, I'm somewhat tempted to keep the split as straight-forward as possible and then do further improvements to the library separately.

3. Similar to 2. I'm all in favour, but it's an improvement that makes sense independently of the split, so it should perhaps be done independently of the split.

4. I think conditionally depending on transformers may make sense. I'm also planning to move to depending on 8.0 soon anyway, so if it makes everything easier, we can as well put base > 4.9 in the cabal file now, then really bump to 0.4, and thereby make it clear that all 8.0-and-above-only changes become possible from now on.

[matchwood]

Great, I agree with all of this. I'll go ahead with this in mind and create a PR - we can always change things if others have objections.

[phadej]

• "1" both are 👍, i.e. having the same version number after split, and letting them live own lives afterwards.
• "3" sop will have tests, there are doctests. They aren't run as test-suite, but externally. I can take care about making travis run them after the split (though haskell-ci should just work).
• "4", note that we only conditionally depend on transformers already:

  if !impl (ghc >= 8.0)
    build-depends:     transformers-compat  >= 0.3  && < 0.7,
                       transformers         >= 0.3  && < 0.6

is in generics-sop.cabal :)

[matchwood]

3. Ah yes, I'm running into issues with the doctests actually. doctest doesn't seem to like that the modules are now spread across two directories. I've tried using -isrc:../sop/src and variations but I haven't found anything that works thus far. I don't have much experience with doctest so it may be that I'm missing something obvious.

4. Haha excellent point I just realised I've been reading that as an if / else! But I guess @kosmikus's point about depending on 8.0 is still relevant - if the packages don't support <8.0.2 then we can remove the conditional transfomers dependency.

[matchwood]

So specifically, running the doctest.sh script with the addition of -i../sop/src in the generics-sop directory also seems to try to run the tests in the sop src, and fails with Could not find module [..] when it runs into import Data.SOP.

[phadej]

It will be easier to discuss over the concrete code. Make a PR. It's ok, even it's not perfect yet.

…

Sent from my iPhone

On 10 Sep 2018, at 20.40, matchwood ***@***.***> wrote: So specifically, running the doctest.sh script with the addition of -i../sop/src in the generics-sop directory also seems to try to run the tests in the sop src, and fails with Could not find module [..] when it runs into import Data.SOP. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[matchwood]

Sure thing, see above (not sure if you were notified of the PR or not).

[phadej]

Ok. Let's not worry about doctests for now. I can fix them when other stuff is settled.

About versions. If generics-sop re-exports sop's modules (in a way it does now in #78), it has to depend on the minor version of sop (i.e ==0.4.0.*) . Otherwise: when a new symbol is added to sop, it would leak through generics-sop which version number stays the same. Therefore users of generics-sop API won't be described by its version only.

I think that that re-exporting explicit member lists is better (though tedious):

• When new stuff is added to sop the users can depend on sop directly to get it
• generics-sop re-export lists could be updated once in a while (bumping the lower bound dependency)

An alternative is not to re-export sop modules in generics-sop. That has different pros and cons.

So in summary, three alternatives

• version numbers tightly coupled, relatively easy to maintain, easy to use
• explicit re-export lists, burden to maintain, easy (and safe) to use
• no-export, easiest to maintain, have to depend on two libs - so slight inconvenience for users

---

  if !impl (ghc >= 8.0)
    build-depends:     transformers-compat  >= 0.3  && < 0.7,
                       transformers         >= 0.3  && < 0.6

says; if not GHC-8.0 or later depend also on transformers. i.e.

• with GHC-8.0 we don't need transformers
• transformers are needed for GHC-7.8/7.10

!impl(ghc >= 8.0) is almost the same as impl(ghc < 8.0).

[matchwood]

I think in the longer term there is no need for generics-sop to export the sop modules - doing so would just ease the transition. So I would be in favour of explicitly exporting with a deprecation warning, not adding any extra exports if sop changes, and removing the re-exports at a later point.

Yeah I get the logic of the conditional build-depends, I was just mentally inserting an else clause whenever I read it! (i.e. I read it as conditionally importing either transformers-compat or transformers). But in any case, if 0.4 drops support for < 8.0 then the conditional import can just be removed (unless I'm missing something?).

[kosmikus]

The PR looks good.

Some questions I still haven't quite decided on:

1. The name of the core package. Currently sop. Other options that have been mentioned are sop-base or sop-core. I like the simplicity of sop. I think sop-base may be too close to basic-sop, even though the latter is a rarely used and unimportant package. However, sop-core might be easier to Google, whereas sop without extra terms such as "Hackage" or "Haskell" is clearly leading to lots of nonsense.

2. The re-export problem mentioned by Oleg. I think I'm leaning towards option 1 here (version numbers tightly coupled). This seems to make sense to me. If sop is being bumped, then generics-sop should be as well, unless no exports change. I'm worried about the "easy of use" for someone just interested in generics-sop. Having to "find" another package and what it exports and importing it separately isn't going to make anything easier. I'm not at all against the split, because these parts make sense on their own. But they're still also a "part of" generics-sop as far as I'm concerned. If we reach a point where sop is widely popular yet generics-sop isn't, we could reconsider.

3. Should we drop GHC 7 compatibility or not? On one hand, there's no critical reason to do so for the current changes. On the other hand, I've always been holding back some other changes because we still have to support GHC 7, so dropping the compatibility now would be somewhat liberating. I'm happy to go with overall consensus here.

[matchwood]

1. I wondered about the searchability of sop as well. Personally I routinely add "Haskell" to all my searches anyway and I guess most people would if necessary (it would take a while to find transformers from a Google search for just "transformers"!). But I don't feel strongly one way or another.

2. How would you want to handle additions to sop that were not really relevant to generics-sop? Say, for example, a module that provides specialisations for heterogenous lists or extensible records? Particularly that might be an issue if there are breaking changes in the code not used by generics-sop then generics-sop will be forced into bumping versions for no particular reason.

3. I have no opinion on this, aside from that feeling liberated is a concrete positive!

[matchwood]

2. Having said that I realise that if they are new modules then I guess they just wouldn't be exported by generics-sop so that shouldn't matter. Oleg's suggestion for explicit exports though does allow new additions to existing modules without having to bump generics-sop. If curry_NP and uncurry_NP live in Data.SOP.NP then would you want generics-sop to re-export them for example? And if the re-exports aren't deprecated (thinking about it again I don't think they should be deprecated) then the ease of use issue doesn't arise.

[kosmikus]

I would probably want to re-export curry_NP and uncurry_NP, yes. It doesn't make much sense to make subtle differences between Data.SOP.NP and Generics.SOP.NP, because there'll be nobody who's able to remember what these differences are. If there are extra modules added to sop, then that's a different story. But I'd probably argue that if anything is to be added to sop that's somehow "in conflict" with generics-sop, then it should go into yet another package depending on sop.

[matchwood]

Ha true - I've been bitten by that sort of idiosyncrasy before. I don't feel strongly about this, happy to go with whatever you decide.

[kosmikus]

Ok, in the interest of making progress, I'm going to decide here:

1. Let's call it sop-core after all. I might rename the overall repository to sop at some point.

2. Let's go with "version numbers tightly coupled".

3. Let's drop compatibility with GHC 7.

[matchwood]

Great, I've updated the PR. There is now some obsolete CPP code (in Data.SOP.BasicFunctors) for example. Would you like me to go through removing this?

[kosmikus]

@matchwood Thank you! Yes, I think we can and should remove CPP fragments that are no longer needed. Keeping them around serves no purpose and will just be dead code and noise.

[kosmikus]

CPP cleanup done.

[matchwood]

Many thanks - apologies for not getting around to doing the CPP cleanup. Had a busy few days, and it was next on my list for this week!

[kosmikus]

@matchwood No worries. Thanks a lot for your help.