Convert Dependencies to Submodules

[th3w1zard1]

Nesting a .git in a subfolder of our main repo creates a submodule. This strategy allows us to point to another repository, enabling these repositories to connect and interact more efficiently. PyKotor has become a giant monster of a project, and it's been difficult to organize things with our current system. This is why I recommend the feature: sub-modules

Organizes Commits:

• Example with Holopatcher: When releasing a new version of Holopatcher, the commits for other tools like PyKotorGL or PyKotorFont won't clutter the history. Submodules prevent this.
• Developers can concentrate on specific components by filtering the commit history for the relevant submodule, reducing clutter and enhancing productivity.
• Encourages a modular approach to code organization, leading to a cleaner, more navigable codebase.

Supports Version-Specific Dependencies:

• Submodules allow for pinning specific versions of external repositories, crucial for maintaining stability in the face of external changes.

Reduces Risk of Merge Conflicts:

• Isolating different project areas into submodules minimizes the chance of merge conflicts, making integration smoother.

Allows repos to communicate:

• Allows the master repo PyKotor to always know where a nested sub-module repo should be pathed inside the master.
• Submodules enable the master repo, like PyKotor, to reference specific commits in nested submodule repos. This maintains clear and precise references to the exact state of each component within the overall project.

More control over what the user is pulling:

• Considering the size of toolset kits, submodules allow users to clone only the parts of PyKotor they need, like BWM or PyKotor.Extract.Installation, avoiding unnecessary downloads.

Improved Dependency Management:

• Submodules help in managing dependencies by linking specific versions to the main project, ensuring consistent performance across various branches

[ProjectSynchro]

Just my two cents (as an outsider):

Creating multiple repositories would be a good idea for this project, keeping the git history might be a bit annoying, however it should be possible to retain using git log <path> and scripting all those commits to be applied into another repo.

If you go this route you may want to look into starting a Github Org for this project, which should allow you to easily manage permissions at a organization level, which allows consistent permissions. (make sure you have some kind of governance plan in place in case of emergencies).

You can also make use of dependabot, to manage updates to what would (in theory) become the new meta-repository pykotor.

You'd also be able to split your issue trackers (or not and keep them in the meta-repository) quite easily, as well as transfer them to the right repositories if one is made somewhere in error.

As a contributor it definitely would make things a bit less crazy (no fear of breaking the entire stack) if you kept all the different tools in separate repos.

As an end-user you'd be able to clone the pykotor repository and end up with what you are expecting (a folder tree of all the tools).

[th3w1zard1]

Ironically this project used to be 4 different repositories

• PyKotor
• PyKotorGL
• ToolsetData
• HolocronToolset

@NickHugi why did we end up combining the repositories?

[th3w1zard1]

As a contributor it definitely would make things a bit less crazy (no fear of breaking the entire stack) if you kept all the different tools in separate repos.

Can confirm I feel fear anytime I update code due to how much is relying on it.

The real problem is the project is too big. Usually Python projects are a handful of scripts.

I notice most of the bugs I end up fixing, would have been caught before run in a compiled language (e.g. Java, C/C++, c#.)

In Python: The syntax errors, indentation errors, type checks, other typo's won't be discovered until the interpreter sees it during runtime. So each .py feels like a bomb that could go off at any moment.

Add in Python's lack of hot editing, how often the toolset needs to load, and yes it has been becoming quite the issue.

[th3w1zard1]

MyPy is a strong solution but I've spent a year going through all of the typecheck errors/warnings yet there's still thousands remaining.

[th3w1zard1]

Thought I'd post here since I'm doing another large repo refactor over on my fork.

I still am not sure the best pythonic way to do all of this. I would like PyKotor and its tools to at least be different pip installs or poetry dependencies. everything in 'utility' must be converted either into poetry packages unrelated to the repo, or put in a non-repo-folder-structured namespace like e.g. as part of Libraries/PyKotor. Perhaps they could all just be optional poetry packages for not until this.