Code sharing between different training courses

[stephenswat]

Just now I opened a merge request, #17, which deals with the way this course is built and served. That is to say, something outside of the content. This left me a little bit worried about the structure of the repositories of the HSF trainings in a more general sense. If I go to any of the other training repositories, the same issue is still there. Updating anything that is not directly related to the contents of one particular course requires you to mirror that change a dozen times! This seems like a potential recipe for disaster and unmaintainability in the future.

Have we put any thought into how we can maybe resolve this issue? To somehow deduplicate all the non-content files between the different trainings?

[stephenswat]

@klieret, I was told to tag you, since you are the most likely candidate to have thought about this. 😄

[stephenswat]

Since opening this issue I have found out about Jekyll themes and how to create one. I have managed to move a lot of the assets and layouts in this repository into a theme, which could significantly reduce the amount of code duplication between projects; they could all just use this theme.

If everyone agrees, I could create a repository for this theme and see if we can convert the existing trainings to it. 😃

[klieret]

Hi @stephenswat

So far we have exactly copied the approach that the carpentries have taken: They use a repository "style repository" with all shared elements and use it as a template whenever they start a new module. The disadvantage is, as you mentioned that it's hard to pass on updates to all the modules.

In our case this styles repository lives here: https://github.com/hsf-training/hsf-styles but I haven't updated it in a while. This should be kept up to date with its counter part from the carpentries (to which we made some adjustments) https://github.com/carpentries/styles

I haven't done that in a while though :/
In particular I see that they seemed to have fixed this bit of the Makefile: https://github.com/carpentries/styles/blob/gh-pages/Makefile though by moving the set rather than deleting it.

I will update our styles repository today and then create PRs for all the modules so that they are up to date.

At one of the hackathons I also talked to @henryiii about better ways to do this. I think that using themes was also what he proposed.

I currently don't have much time to look into this, but for me the only important thing is that we don't get cut away from changes from https://github.com/hsf-training/hsf-styles because they have many more people working on this.

[stephenswat]

Thank you for the extensive answer, Kilian. I'd still like to see how far we can come with a Jekyll theme (keeping the hsf-styles repository in mind), so I will continue looking into that, and report back soon. :)

[stephenswat]

Alright, so I read up a little bit on Jekyll themes and such, and I have come up with something that might work. If you check out https://github.com/stephenswat/hsf-training-cpp-webpage/tree/minimized, you will find my minimized version of this C++ training repository, which has the following two changes made:

1. The assets, layouts, and includes are moved to a new theme repository at https://github.com/stephenswat/hsf-training-theme.
2. All the unnecessary files and directories are removed.

I think the advantage of this approach is that there is simply a lot less code to maintain. By using this theme across trainings we would avoid a ton of work updating files whenever the base theme changes.

The instructions for serving the site are slightly different (not not much more complicated, explained in the README), and serving this site is actually a lot faster.

[klieret]

Thanks a lot for looking into this @stephenswat ! I agree that this is a much much nicer solution than the current one.

It seems carpentries have actually discussed this: First brought up in this issue, then decided to reevaluate the decision in the future (source). This is probably the same as what you're doing, right?

In order to keep the current workflow of starting a new module (which is to import the styles repo and then to run bin/lesson_initialize.py) we would need to change that script.

More importantly for maintenance though, we would have to split up the styles repository in two parts (one with the lesson_initialize.py) and one with the shared style files. Bringing in the changes from the carpentries via simply merging their repository might no longer be possible then, right? (though we might hack something together)

[stephenswat]

Looking at what the Carpentry people proposed in those issues, it looks rather similar, yes.

Regarding the lesson_initialize.py script, I must admit I don't really understand what purpose is serves. From what I can tell, it simply copies a bunch of files in the training, but why would we not just put those files there by default? Github provides convenient template repositories that allow us to do exactly this - without the use of a Python script. It would seem to me like that would be a a much easier, and more robust, workflow for creating new modules.

Regarding your second question, I think the ideal approach to all of this would be to have a minimum amount of content in the lessons themselves that would even need to be updated from the Carpentries. Looking at the branch I created as an example, there is already rather little there that is dependent on upstream updates: the _extras directory, some configuration files, and about five markdown files. I would like to reduce that number further - ideally we would have zero files that are copies of the upstream - but that will take a bit more work. Still, I think the setup is already quite maintainable; most of the upstream Carpentries changes would need to be merged into the theme repository, but they would propagate automatically to the individual modules because of the theme-based design.

[klieret]

I just asked the people from the carpentries why they didn't follow through with this theme based approach. Perhaps there were some reasons.

[klieret]

Indeed this is looking quite promising. I just tried your minimized version and it worked flawlessly!

So here's the path of migration that I currently see:

1. Fork hsf-boilerplate from hsf-styles to replace it in the future. Remove everything that won't be in the hsf-training-theme. Let's keep the structure from the carpentries with the bin/lesson_initialize.py for now, even though it might not make much sense for us. We could archive hsf-styles to make sure that it won't be used anymore.
2. Create hsf-training-theme (we could e.g. move your repository. Note that we need to give a proper license that includes the carpentries, because we're very much standing on their shoulders). It is probably best to base that on the carpentry-theme rather than our styles repo so we can simply pull changes from upstream
3. Migrate existing repositories
4. New lessons start by importing hsf-boilerplate as template repo

For maintenance:

• hsf-boilerplate pulls from carpentry's styles (since we only have fewer files this should work trivially)
• hsf-training-theme pulls from carpentries-theme (should work trivially)

[klieret]

Ah regarding lesson_initialize.py: I think this was done because there were both files where changes should be propagated to the lessons and files that were only boilerplate (those in bin/boilerplate) where you'd like to be able to change default values without having merge conflicts when you merge the styles repo in the lesson repo for updates.

So I think this actually still makes sense.

[klieret]

I had to

-gem 'hsf-training-theme', git: "https://github.com/stephenswat/hsf-training-theme.git"
+gem 'hsf-training-theme', git: "https://github.com/stephenswat/hsf-training-theme.git", branch: 'main'

btw

[klieret]

Hi @stephenswat : I now finally followed your approach and created hsf-training/hsf-training-theme (which is based on a similar repo from the carpentries with our changes on top of them).

There is still the issue of the few files that cannot be shared by this, but for that I will use multi-gitter to make bulk changes (see maintenance repo).

Thanks a lot again for getting this started! Sorry that it took so long to make it happen...

[stephenswat]

Hi Kilian, it's me who should be sorry for proposing this and not helping to implement it. 😉

Very glad to hear it has worked out! Great job!

[klieret]

Looking at your repository (in particular your Gemfile) helped a lot! :)