-
Notifications
You must be signed in to change notification settings - Fork 4
Package unification plans
The idea is to have single package for all functions exposed from librustzcash instead of providing set of smaller packages with smaller interfaces. So instead of zcash_wallet_ops and zcash_backend_keys packages there will be only one package with name like zcash that will expose everything from there.
During work on Milestone 3 we have noticed that there are problems with using data structures across different packages in dynamically typed languages (Python and Ruby) which would make integration of such modules needlessly complex in these languages.
In addition to above there is concern about version mismatches in librustzcash, all packages used by the UniFFI Zcash project are in pre-1.0.0 versions which mean that the API is prone to changes in non-compatible ways so there is possibility that user of the multiple UniFFI Zcash packages would accidentally use different versions of the packages, which could result in parsing or cooperation issues that could be confusing to the user. With single project we remove such possibility as with that we guarantee that there will be always single package version used.
Yet another concern, which is a little bit related to the above, is that with multiple packages we can quickly bloat the compilation times of the dependant projects, as each package will need to independently compile all Rust packages used by each of the projects. Even when the versions would match, the derivative language tooling do not do any cross-package caching (at least we aren't aware of such systems in any of the tooling of these languages). That mean that if user need to use 2 or more packages then they will for example compile N times uniffi project (once for each of the packages). This can substantially increase build times. So instead we can sacrifice a little bit of the build time of the single project (by having more code in one place) to mitigate problem of duplicated compilation of the dependencies.
We could have done it through manual serialisation data into some transferable format (like sequence of bytes) and then pass it to another package and parse that bytes there again. That would partially solve the problem, but at the cost of runtime performance of serialisation and deserialisation data constantly. Additionally that greatly expands the required API surface, additional requirements on wrapper libraries that would handle all that shifting of the formats as well as it would require additional extensive testing against other parts of the universe. Not only against current releases, but also any past release to check if we do not have breakage. This brings a lot of additional complexity to the system without any real gains.
This would partially mitigate the requirement of the serialisation, but it expose the internals of the data structures which in most cases we want to keep "secret". Additionally in many cases (for values that need to be opaque, because librustzcash do not expose its internals anyway) it would bring the performance issues related to the serialisation even within single package. Again, no real gains.
With single big package it may become harder to navigate and maintain .udl file as it grows. Fortunately for us we have found that there is easy way to mitigate this problem on Rust side, as we can use public exports in generated code easily, which gives us the capability to simply split the Rust code into modules.
For .udl we have come up with an idea of having some kind of "preprocessor" that will be ran by build.rs script that will concatenate multiple .udl files before build. That will allow us to provide simple way to manage that codebase while keeping everything in single package.