Package/publish generated files

[ev3nvy]

Instead of providing generated files via GitHub releases, we should directly publish them to package repository/repositories for the given language.

Unresolved questions

• What versioning scheme to use?
  • Semver doesn't seem like a good choice since flatc doesn't follow it either.
    • Maybe something like major.YYMMDD.minor.patch (1.250124.0.1)?
    • Look for examples of how other projects do it (I think gitlab crate for rust does something similar).
• What do we do with languages like C++, where there isn't really a central package repository?
  • I think the best approach would be to have a separate repo for those packages (c++ projects often utilize git modules from my limited experience).
  • Could this be a good candidate for josh?
  • TODO: categorize languages into these two groups.
• What would the project structure look like?
  • Root folder called generated with language specific sub-folders?

Needs investigating

• Version constraints for official language specific packages.
  • Let's take rust as an example. Generated rust schema file depends on flatbuffers crate. If a user of rman-schema-rs already has flatbuffers somewhere in the dependency tree, we should also depend on the same version. This is desirable because flatbuffers format claims to be forwards and backwards compatible (assuming this also applies to crates.io releases). To achieve this in rust, flatbuffers dependency should be declared as seen here:

    flatbuffers = "*"

    If forwards and backwards compatibility doesn't apply to crates.io releases however, we should instead pin the version to the version of flatc that was used to generate the file. Users of the rust crate, can also utilize dependency patching if they want to force rman-schema-rs to use a different flatbuffers version (see: https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html).
  • To apply this point more generically to other languages and package managers, there are some considerations to be made.
    • I assume not every package manger handles dependencies the same way, so this should be investigated case-by-case for every package-manager.
    • Dependency patching is pretty unique to cargo/rust, it might make more sense to pretend it doesn't exist.
    • Amount of projects that already depend on flatbuffers should be next-to-none, so I think a safe option would be to just have strict version requirement (and even if they do, it should not be a big deal).

GitHub Actions

• Changes to the schema and updating flatc should re-trigger schema generation.
  • There should be a way to check which files a commit/pr touches.
• Changes should automatically be commited and pushed to the repo via CI.
  • I think doing most steps using just would be cool, so users could build locally as well. It would also simplify CI quite a bit.
• Publish changes to respective package repository.
  • Investigate best practices for each package manager, based on popular projects using it.
  • What should we do if a subset of publishes fails?
    • We should design workflows in a way, where manual dispatch for each publish workflow is possible.

[ev3nvy]

Many if not all questions would be resolved with a reference implementation.

Current plan is to package and publish a schema for rust via cargo to see what makes sense and what doesn't. Reason for doing rust first is because it's what I'm most familiar with.

Next would probably be a C++ version, purely to address the following:

What do we do with languages like C++, where there isn't really a central package repository

Finally, I also have confidence to package js/ts version (I could technically also give python a go, but I don't have enough experience on how python packages are published). Packaging for other languages will stay unimplemented until someone with the required know-how comes along and is willing to contribute.