Document embedded entities addition and removal tracking (PR #8091)

# Description

Documentation pr for inmanta/lsm#788

# Self Check:

Strike through any lines that are not applicable (`~~line~~`) then check the box

- [ ] Attached issue to pull request
- [ ] Changelog entry
- [ ] Type annotations are present
- [ ] Code is clear and sufficiently documented
- [ ] No (preventable) type errors (check using make mypy or make mypy-diff)
- [ ] Sufficient test cases (reproduces the bug/tests the requested feature)
- [ ] Correct, in line with design
- [ ] End user documentation is included or an issue is created for end-user documentation (add ref to issue here: )
- [ ] If this PR fixes a race condition in the test suite, also push the fix to the relevant stable branche(s) (see [test-fixes](https://internal.inmanta.com/development/core/tasks/build-master.html#test-fixes) for more info)

changelogs/unreleased/788-document-embedded-entities-tracking.yml

@@ -0,0 +1,3 @@
+description: "Document embedded entities addition and removal tracking"
+change-type: patch
+destination-branches: [master, iso7]

docs/lsm/embedded_entities/embedded_entities.rst

@@ -73,7 +73,7 @@ of the relationship to the embedding entity starts with an underscore as require
 .. literalinclude:: embedded_entities_sources/example_bidirectional_relationship.cf
     :linenos:
     :language: inmanta
-    :emphasize-lines: 16,17
+    :emphasize-lines: 15
     :caption: main.cf
 
 
@@ -100,7 +100,7 @@ modifiers on relationships:
 
 .. literalinclude:: embedded_entities_sources/example_attribute_modifiers_on_relations.cf
     :linenos:
-    :emphasize-lines: 16,17
+    :emphasize-lines: 15,16
     :language: inmanta
     :caption: main.cf
 
@@ -129,16 +129,84 @@ to uniquely identify the embedded entity.
 
 .. literalinclude:: embedded_entities_sources/example_key_attributes.cf
     :linenos:
-    :emphasize-lines: 26,29,30
+    :emphasize-lines: 25,28,29
     :language: inmanta
     :caption: main.cf
 
+
 If the upper arity of the relationship towards an embedded entity is one, it's not required to define an
 index on the embedded entity. In that case, the embedded entity will always have the same identity, no matter what the
 values of its attributes are. This means that there will be no difference in behavior whether the attribute modifier is
 set to ``rw`` or ``rw+``. If an index is defined on the embedded entity, the attribute modifiers will be enforced in
 the same way as for relationships with an upper arity larger than one.
 
+.. _tracking_embedded_entities_across_updates:
+
+Tracking embedded entities across updates
+#########################################
+
+
+Depending on what the embedded entities are modeling, you might want to keep track of which embedded entities
+were added or removed during an update. This section describes how to track embedded entities during the update flow of
+a service. When using the simple lifecycle, this is supported out of the box by wrapping the call to ``lsm::all()`` with the :py:meth:`inmanta_plugins.lsm.insert_removed_embedded_entities<inmanta_plugins.lsm.insert_removed_embedded_entities>` plugin.
+
+The following sections describe 3 flavours of update flows through examples.
+
+
+Update flow with implicit deletion
+**********************************
+
+In this update flow, the embedded entities are side-effect free
+and fully under control of the parent entity. The following model
+demonstrate this case:
+
+- The parent entity is a file on a file system
+- The embedded entities represent individual lines in this file
+
+In this example, the deployed resources (i.e. the deployed files) will mirror exactly the embedded entities
+present in the model since the content of the file is derived from the set of embedded entities.
+If an embedded entity is removed during an update, the file content will reflect this accordingly.
+
+
+.. literalinclude:: embedded_entities_sources/example_lines_in_file.cf
+    :linenos:
+    :language: inmanta
+    :caption: main.cf
+
+Update flow with explicit deletion
+**********************************
+
+
+In this update flow, the embedded entities are not side-effect free
+or not fully under control of the parent entity. The following model
+demonstrate this case:
+
+- The parent entity is a directory on a file system
+- The embedded entities represent individual files in this directory
+
+In this example, we have to take extra steps to make sure the deployed resources (i.e. the deployed directories and files
+below them) match the embedded entities present in the model.
+The content of the directories is derived from the set of embedded entities. If an embedded entity is removed during an
+update, we have to make sure to remove it from disk explicitly.
+
+.. literalinclude:: embedded_entities_sources/example_files_in_folder.cf
+    :linenos:
+    :language: inmanta
+    :caption: main.cf
+
+
+Update flow with mutually explicit desired state
+************************************************
+
+The last possible update scenario is one with mutually exclusive desired state throughout the update, e.g. a database
+migration from cluster A to cluster B:
+
+1. Initial desired state: data lives in cluster A
+2. Intermediate desired state:  data is replicated in cluster A and cluster B
+3. Final desired state: data lives in cluster B
+
+
+For these more involved update scenarios we recommend updating the lifecycle specifically for this update.
 
 .. _legacy_no_strict_modifier_enforcement:
 

docs/lsm/embedded_entities/embedded_entities_sources/example_files_in_folder.cf

@@ -0,0 +1,83 @@
+import std::testing
+import lsm
+import lsm::fsm
+import fs
+import mitogen
+
+entity Folder extends lsm::ServiceEntity:
+    """
+        Top-level service representing a folder on a file system.
+
+        :attr path: Unique path to this folder
+    """
+    string path
+end
+
+index Folder(path)
+Folder.files [0:] lsm::__rwplus__ File._folder [1]
+
+entity File extends lsm::EmbeddedEntity:
+    """
+        Embedded entity representing a file in a folder.
+
+        :attr name: name of this line
+        :attr content: Content of this file
+
+    """
+    string name = ""
+    string content
+    lsm::attribute_modifier content__modifier="rw+"
+
+    # These private attributes keep track of added/removed
+    # embedded entities across updates
+    bool _added = false
+    bool _removed = false
+end
+
+index File(_folder, name)
+
+binding = lsm::ServiceEntityBindingV2(
+    service_entity="__config__::Folder",
+    lifecycle=lsm::fsm::simple,
+    service_entity_name="folder",
+)
+
+# We wrap the call to lsm:all() with the plugin call:
+for instance in lsm::insert_removed_embedded_entities(lsm::all(binding)):
+    folder = Folder(
+        instance_id=instance["id"],
+        entity_binding=binding,
+        path=instance["attributes"]["path"],
+    )
+    for file in instance["attributes"]["files"]:
+        folder.files += File(**file)
+    end
+end
+
+implementation folder for Folder:
+    self.resources += std::testing::NullResource(name=self.path)
+end
+
+mitogen_local = mitogen::Local()
+
+implementation file for File:
+    self._folder.resources += fs::File(
+        path=self._folder.path+"/"+self.name,
+        host=std::Host(
+            via=mitogen_local, name="internal", os=std::linux,
+        ),
+        mode=777,
+        owner="inmanta",
+        group="inmanta",
+        content = self.content,
+        # By keeping track of embedded entities removed during an update
+        # we can purge the underlying resources accordingly.
+        # Alternatively, if the parent folder is removed, we want to
+        # purge all embedded entities.
+        purged=self._removed or self._folder.purge_resources,
+    )
+end
+
+implement File using parents
+implement File using file
+implement Folder using parents, folder

docs/lsm/embedded_entities/embedded_entities_sources/example_lines_in_file.cf

@@ -0,0 +1,74 @@
+import std::testing
+import lsm
+import lsm::fsm
+import fs
+import mitogen
+
+entity File extends lsm::ServiceEntity:
+    """
+        Top-level service representing a file on a file system.
+
+        :attr path: Unique path to this file
+    """
+    string path
+end
+
+index File(path)
+File.lines [0:] lsm::__rwplus__ Line
+
+entity Line extends lsm::EmbeddedEntity:
+    """
+        Embedded entity representing a single line in a file.
+
+        :attr line_no: The line number
+        :attr content: Content of this line
+    """
+    int line_no
+    string content = ""
+    lsm::attribute_modifier content__modifier="rw+"
+
+    # These private attributes keep track of added/removed
+    # embedded entities across updates
+    bool _added = false
+    bool _removed = false
+end
+
+index Line(line_no)
+
+binding = lsm::ServiceEntityBindingV2(
+    service_entity="__config__::File",
+    lifecycle=lsm::fsm::simple,
+    service_entity_name="file",
+)
+
+# We wrap the call to lsm:all() with the plugin call:
+for instance in lsm::insert_removed_embedded_entities(lsm::all(binding)):
+    file = File(
+        instance_id=instance["id"],
+        entity_binding=binding,
+        path=instance["attributes"]["path"],
+    )
+    for line in instance["attributes"]["lines"]:
+        file.lines += Line(**line)
+    end
+end
+
+mitogen_local=mitogen::Local()
+
+implementation file for File:
+    self.resources += std::testing::NullResource(name=self.path)
+    self.resources += fs::File(
+        path=self.path,
+        host=std::Host(
+            via=mitogen_local, name="internal", os=std::linux,
+        ),
+        mode=777,
+        owner="inmanta",
+        group="inmanta",
+        content = string([string(l.content) for l in std::key_sort(self.lines, 'line_no')]),
+    )
+end
+
+implement File using parents
+implement File using file
+implement Line using parents

docs/lsm/index.rst

@@ -5,7 +5,7 @@ Inmanta Lifecycle Service Manager
 *********************************
 
 
-The Inmanta LSM is an active component that governs the lifeycle of services in the orchestration model. LSM extends the
+The Inmanta LSM is an active component that governs the lifecycle of services in the orchestration model. LSM extends the
 Orchestration Engine (OrE) and Resource Controller (ResC) with a service catalog, a service inventory and a lifecycle manager.
 
 .. image:: ../_static/architecture.png
@@ -63,7 +63,7 @@ Creating service entities
 =========================
 .. _intro_example:
 
-Service entities are :ref:`entities <lang-entity>` that extend ``lsm::ServiceEntity``
+Service entities are :ref:`entities <lang-entity>` that extend ``lsm::ServiceEntity``.
 We define attributes for the service entity the same way as for entities. We can also define a modifier for the attribute.
 If no modifier is defined for an attribute, it will be ``rw`` by default.
 Here is an example of an entity definition where the modifier of the ``address`` attribute is set to ``rw+``.
@@ -79,7 +79,7 @@ We also need to add a lifecycle and a name to the service. This is done by creat
     :linenos:
     :language: inmanta
     :lines: 30-36
-    :lineno-start: 25
+    :lineno-start: 30
 
 It's also possible to define a service identity for a service. For more information, see :ref:`service_identity`.
 
@@ -158,7 +158,7 @@ source state, a target state and an error state. The following events can trigge
 
 - the creation of the service instance: The state of the new service instance is set to the start state defined in the state machine. Set attributes provided with the API call are stored in the candidate_attributes set of the instance.
 - ``auto``: This transfer is automatically performed when the lifecycle arrives in the source state. Auto transfers can be disabled by adding a configuration option.
-- ``api set state`` call: When a set state API call is performed with matching source and target states
+- ``api_set_state`` call: When a set state API call is performed with matching source and target states
 - ``on_update``: Transfers marked as on_update are executed when a PATCH is performed on a service instance. The update attributes are stored based on the target_operation or error_operation attribute.
 - ``on_delete``: Transfers marked as on_delete are executed when a DELETE is performed on a service instance.
 - resource based: This transfer is triggered when the orchestrator finishes deploying the resources that this service instance consists off.
@@ -219,7 +219,7 @@ Glossary
         entity defines the attributes of a :term:`service instance` and the lifecycle state machine.
 
     service instance
-        The lifecycle manager manages the lifeycle of service instance.
+        The lifecycle manager manages the lifecycle of service instance.
 
 Dict Path Library
 -----------------