From 439a1ee133aa20904982c7fc1c17fb032ca162c5 Mon Sep 17 00:00:00 2001 From: Viktor Dick Date: Sun, 29 Dec 2024 14:33:35 +0100 Subject: [PATCH 1/3] Describe new layering concept --- README.md | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/README.md b/README.md index 3187c0a..9bbed37 100644 --- a/README.md +++ b/README.md @@ -101,6 +101,9 @@ and should output a json dictionary in the form of ] ``` +### `layers` +Path to folder on the filesystem that contains layer definitions. See below. + ## Usage The executable `zodbsync` provides several subcommands @@ -285,6 +288,85 @@ Detailed steps: Hint: This requires `git` in version 2.22 or above. +## Layer handling + +Since version 23.1.0, `zodbsync` has gained the possibility to handle multiple +file system source trees that each contribute a separate layer to the objects +in the Data.FS. Layer handling was reworked for 23.3.0 and the following +describes the new handling. + +The configuration option `layers` points to a folder on the file system that +contains separate configuration files or symlinks that may be contributed by +different layer packages. These are read alphabetically and provide the +following options: + +### `workdir` +A path where the layer is placed. This needs to be owned by the user that +executes `zodbsync`. It will be initialized as a git repository if it is not +already one. + +### `source` +This is a path that provides the objects of a layer, possibly read-only to the +user executing `zodbsync` and provided by a Debian package or similar. + +An implicit layer is added at the top where `workdir` is set to the `base_dir` +provided in the main config for compatibility with a non-layered setup. + +The representation rules for objects in a multi-layer setup are as follows: + +- Each object is defined by the topmost layer that provides a `__meta__` file + for it. +- If a folder in some layer contains a `__frozen__` file, both the folder and + any subobjects from any layer below this are ignored - the layer is expected + to fully replicate the intended state of the folder and anything below it. +- If a folder in some layer contains a `__frozen__`, but no `__meta__` file, + the object is supposed to be deleted even if it is defined in some layer + below. +- It is cause for a warning if an object is definend in multiple layers without + explicitly marking it as `__frozen__` and therefore declaring the intent for + a higher layer to shadow the lower layer definition. +- Layers can define subobjects without defining their parent objects, with the + assumption that some lower layer dependency will provide the parent object. + However, if some subobject is defined while no active layer provides the + parent, re-recording will remove it. + +During `record`, the following rules hold: + +- If a new object is found, it is recorded into the `workdir` of the layer that + defines its parent. +- If an object is changed, it is changed in the layer that defined the object. +- If an object is deleted, it is deleted in all layers that define the object, + unless shadowed by a `__frozen__` marker. As mentioned above, this should + usually only be one layer. +- If an object is deleted that is defined as `__frozen__` in one layer and + additionally in a lower layer, that marker is kept. + +The following subcommands for `zodbsync` provide layer handling: + +### `layer-init` +Initialize all layers by initializing the `workdir`s from their `source`s and +uploading all changes into the Data.FS. + +### `layer-update` +For each layer, the file `.checksums` in the `source` is compared to that in +the `workdir`. If it deviates, any unstaged changes in the `workdir` are +committed and the layer content is reset to that found in the `source`. This is +used to update a layer to a newer version. Note that any changes done directly +in the layer since the last update are overwritten by this - they can still be +found in the `git` history in the `workdir`, but the working directory and the +resulting Data.FS content are reset. It is therefore possible to pre-apply +changes that will be part of the next release, but if there is a change that is +not yet merged upstream, the layer should not be updated until it is. + +### TBD +Some more commands are needed for the following use cases: +- A new object is recorded into the layer where its parent is defined. However, + it should instead be an additional object defined in a different layer. +- An object is changed, which is recorded into the layer where the object is + defined. However, the intention is not to pre-apply a change that is about to + also be included upstream, but to freeze and record the object into some + other layer. + ## Compatibility This package replaces similar functionality that was previously found in `python-perfact` and `perfact-dbutils-zope2`. For backwards compatibility, From 084949423ac1fd2d22128eeb4c22a869de9ca293 Mon Sep 17 00:00:00 2001 From: Viktor Dick Date: Sun, 29 Dec 2024 14:59:17 +0100 Subject: [PATCH 2/3] Add notes regarding needed migration paths --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index 9bbed37..7e90ef3 100644 --- a/README.md +++ b/README.md @@ -366,6 +366,16 @@ Some more commands are needed for the following use cases: defined. However, the intention is not to pre-apply a change that is about to also be included upstream, but to freeze and record the object into some other layer. +- A migration path for systems that don't use layers yet, but have a lot of the + same objects that are to be provided by a separate layer, which will probably + have some deviations. It needs to be possible to decide which objects to + reset to their upstream state, which to freeze as changed into the custom + layer and which to add as change to the separate layer (intending to include + that change upstream until the next release). +- Similarly, for the few systems that use the "original" layering system from + 23.2.0, a migration path needs to be provided. For any deviating object, it + needs to be decided if that object is to be frozen in the custom layer or + reset to its upstream state. ## Compatibility This package replaces similar functionality that was previously found in From 9471174a99447012b13134e1b52c9459bf4159be Mon Sep 17 00:00:00 2001 From: Viktor Dick Date: Mon, 30 Dec 2024 10:33:32 +0100 Subject: [PATCH 3/3] Fix version reference --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7e90ef3..4006001 100644 --- a/README.md +++ b/README.md @@ -292,7 +292,7 @@ Hint: This requires `git` in version 2.22 or above. Since version 23.1.0, `zodbsync` has gained the possibility to handle multiple file system source trees that each contribute a separate layer to the objects -in the Data.FS. Layer handling was reworked for 23.3.0 and the following +in the Data.FS. Layer handling was reworked for 23.4.0 and the following describes the new handling. The configuration option `layers` points to a folder on the file system that @@ -373,7 +373,7 @@ Some more commands are needed for the following use cases: layer and which to add as change to the separate layer (intending to include that change upstream until the next release). - Similarly, for the few systems that use the "original" layering system from - 23.2.0, a migration path needs to be provided. For any deviating object, it + 23.3.0, a migration path needs to be provided. For any deviating object, it needs to be decided if that object is to be frozen in the custom layer or reset to its upstream state.