Investigate the option of using submodules instead of cloning at build time

[strogonoff]

Currently, hub site pulls project data, and project site pulls software package (and possibly eventually spec) data by checking out their respective repositories during site build. Keeping project data in submodules could reduce build time and remove some of the build logic.

However, it would mean that extra commits would be necessary to “pull” the necessary data into Jekyll site each time.

1. For hub sites, whenever a child project or a “grandchild” software package changes

2. For project sites, whenever a child software package changes

In addition to the process becoming less automated, I believe it’s not possible to add a subdirectory of a repo as a submodule of another repo, which would be important if we want to preserve the sanity of data layout.

See the original riboseinc/open.ribose.com#11.

[strogonoff]

(FTR my first idea was to use GitHub GraphQL API, before I switched to cloning. API could be faster and again avoid the need to manually commit each submodule change across dependent site sources.)

[ronaldtse]

@strogonoff we definitely don't want to lock the git flow to GitHub API, but cloning is fine for now 👍

[strogonoff]

@ronaldtse good point, marrying to GitHub could be a show-stopper for some otherwise interested future consumers of Open Project theme

(Edit: If both Gitlab and GitHub APIs are capable, though, it could be an opt-in setting for consumers who use those host systems and wish to speed up the build. However, API integration maintenance would be an extra headache, so unless this project blows up in popularity I don’t feel like it’d be worth it)

[strogonoff]

@ronaldtse wrote in metanorma/metanorma.org#7 (comment)

Regarding submodules -- we can also clone them for the latest version to save ourselves the need of "updating" the main repo.

[strogonoff]

Right now my vision for framework implementation assumes site build fetching parts of repositories and operating on them, for example, to construct zero-effort multi-version software documentation based on repository version tags.

Using Git submodules means letting Git take care of some of this stuff (e.g., doing the fetching). I see how that would decrease complexity, and it’s a native & more reliable way to do things.

Thing is, with upcoming features such as described above I believe we’re likely going to need finer control over fetched repositories than what the submodule feature is built for, and while this can be investigated in detail and there’s a possibility that submodules would work here, I kinda doubt it’s worth focusing on right now in general based on resources available.

Anyone please feel free to chime in and expand on benefits of submodules, and if someone is willing to join to help investigate and attempt a refactor into submodules, that’d be cool.