Transforming nested containers

doc/persist.rst

@@ -145,6 +145,7 @@ This example also demonstrates a case where the main document type ``doc_2`` con
 As you can see in the resulting JSON, nested types are also serialized with pools: ``"extra": {"comments": 1}``. Only the ID of the ``comments`` ``vector``
 is serialized instead of its content.
 
+.. _transformations-with-pools:
 
 Transformations with pools
 --------------------------
@@ -416,6 +417,71 @@ a ``immer::persist::incompatible_hash_wrapper`` as the result of the ``immer::pe
 We can see that the transformation has been applied, the keys have the ``_key`` suffix.
 
 
+Transforming nested containers
+------------------------------
+
+Let's look at a situation where a transforming function, while operating on one item of some ``immer`` container, has another ``immer`` container to transform.
+We define the types like this:
+
+.. literalinclude:: ../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: start-define-nested-types
+   :end-before:  end-define-nested-types
+
+The important property here is that we have a ``vector<nested_t>`` and the ``nested_t`` contains ``vector<int>``, so we can say a ``vector`` is nested inside another ``vector``.
+We can prepare a value with some structural sharing and serialize it:
+
+.. literalinclude:: ../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: start-prepare-nested-value
+   :end-before:  end-prepare-nested-value
+
+The resulting JSON looks like:
+
+.. literalinclude:: ../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: start-nested-value-json
+   :end-before:  end-nested-value-json
+
+Looking at the JSON we can confirm that the node ``{"key": 2, "value": [1, 2]}`` is reused.
+Let's define a ``conversion_map`` like this:
+
+.. literalinclude:: ../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: start-nested-conversion_map
+   :end-before:  end-nested-conversion_map
+
+While the transforming function for ``vector_one`` is simple, it transforms an ``int`` into a ``std::string``,
+the function for the ``vector<nested_t>`` is more involved. When we try to transform one item of that vector, ``nested_t``,
+we quickly realize that inside that function we have a ``vector<int>`` to deal with. We're back to the problems described in the beginning of :ref:`transformations-with-pools`.
+To solve this problem, ``immer::persist`` provides the optional second argument to the transforming function, a function ``convert_container``.
+It can be called with two arguments: the desired container type and the ``immer`` container to convert. That way, we have access back to the ``conversion_map`` we're defining.
+This transformation will be performed using pools, as expected, and will preserve structural sharing.
+
+Having defined the ``conversion_map``, we apply it in the usual way and get the ``new_value``:
+
+.. literalinclude:: ../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: start-apply-nested-transformations
+   :end-before:  end-apply-nested-transformations
+
+We can verify that the ``new_value`` has the expected content:
+
+.. literalinclude:: ../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: start-verify-nested-value
+   :end-before:  end-verify-nested-value
+
+And we can serialize it again to confirm that the structural sharing of the nested vectors has been preserved:
+
+.. literalinclude:: ../test/extra/persist/test_for_docs.cpp
+   :language: c++
+   :start-after: start-verify-structural-sharing-of-nested
+   :end-before:  end-verify-structural-sharing-of-nested
+
+We can see that the ``{"key": 2, "value": ["_1_", "_2_"]}`` node is still being reused in the two vectors.
+
+
 Policy
 ------
 

test/extra/persist/test_for_docs.cpp

@@ -609,3 +609,194 @@ TEST_CASE("Transform table's ID type", "[docs]")
         // include:end-new_table_t-new-hash-transformation
     }
 }
+
+namespace {
+// include:start-define-nested-types
+struct nested_t
+{
+    BOOST_HANA_DEFINE_STRUCT(nested_t, //
+                             (vector_one, ints));
+
+    friend bool operator==(const nested_t&, const nested_t&) = default;
+
+    template <class Archive>
+    void serialize(Archive& ar)
+    {
+        ar(CEREAL_NVP(ints));
+    }
+};
+
+struct with_nested_t
+{
+    BOOST_HANA_DEFINE_STRUCT(with_nested_t, //
+                             (immer::vector<nested_t>, nested));
+
+    friend bool operator==(const with_nested_t&,
+                           const with_nested_t&) = default;
+
+    template <class Archive>
+    void serialize(Archive& ar)
+    {
+        ar(CEREAL_NVP(nested));
+    }
+};
+// include:end-define-nested-types
+
+struct new_nested_t
+{
+    BOOST_HANA_DEFINE_STRUCT(new_nested_t, //
+                             (vector_str, str));
+
+    friend bool operator==(const new_nested_t&, const new_nested_t&) = default;
+
+    template <class Archive>
+    void serialize(Archive& ar)
+    {
+        ar(CEREAL_NVP(str));
+    }
+};
+
+struct with_new_nested_t
+{
+    BOOST_HANA_DEFINE_STRUCT(with_new_nested_t,
+                             (immer::vector<new_nested_t>, nested) //
+    );
+
+    friend bool operator==(const with_new_nested_t&,
+                           const with_new_nested_t&) = default;
+
+    template <class Archive>
+    void serialize(Archive& ar)
+    {
+        ar(CEREAL_NVP(nested));
+    }
+};
+} // namespace
+
+TEST_CASE("Transform nested containers", "[docs]")
+{
+    // include:start-prepare-nested-value
+    const auto v1    = vector_one{1, 2, 3};
+    const auto v2    = v1.push_back(4).push_back(5).push_back(6);
+    const auto value = with_nested_t{
+        .nested =
+            {
+                nested_t{.ints = v1},
+                nested_t{.ints = v2},
+            },
+    };
+
+    const auto policy =
+        immer::persist::hana_struct_auto_member_name_policy(with_nested_t{});
+    const auto str = immer::persist::cereal_save_with_pools(value, policy);
+    // include:end-prepare-nested-value
+
+    // include:start-nested-value-json
+    const auto expected_json = json_t::parse(R"(
+{
+  "pools": {
+    "ints": {
+      "B": 5,
+      "BL": 1,
+      "inners": [
+        {"key": 0, "value": {"children": [2], "relaxed": false}},
+        {"key": 3, "value": {"children": [2, 5], "relaxed": false}}
+      ],
+      "leaves": [
+        {"key": 1, "value": [3]},
+        {"key": 2, "value": [1, 2]},
+        {"key": 4, "value": [5, 6]},
+        {"key": 5, "value": [3, 4]}
+      ],
+      "vectors": [{"root": 0, "tail": 1}, {"root": 3, "tail": 4}]
+    },
+    "nested": {
+      "B": 5,
+      "BL": 3,
+      "inners": [{"key": 0, "value": {"children": [], "relaxed": false}}],
+      "leaves": [{"key": 1, "value": [{"ints": 0}, {"ints": 1}]}],
+      "vectors": [{"root": 0, "tail": 1}]
+    }
+  },
+  "value0": {"nested": 0}
+}
+    )");
+    // include:end-nested-value-json
+    REQUIRE(json_t::parse(str) == expected_json);
+
+    namespace hana = boost::hana;
+
+    // include:start-nested-conversion_map
+    const auto conversion_map = hana::make_map(
+        hana::make_pair(
+            hana::type_c<vector_one>,
+            [](int val) -> std::string { return fmt::format("_{}_", val); }),
+        hana::make_pair(
+            hana::type_c<immer::vector<nested_t>>,
+            [](const nested_t& item, const auto& convert_container) {
+                return new_nested_t{
+                    .str =
+                        convert_container(hana::type_c<vector_str>, item.ints),
+                };
+            }));
+    // include:end-nested-conversion_map
+
+    // include:start-apply-nested-transformations
+    const auto pools = immer::persist::get_auto_pool(value, policy);
+    auto transformed_pools =
+        immer::persist::transform_output_pool(pools, conversion_map);
+
+    const auto new_value = with_new_nested_t{
+        .nested = immer::persist::convert_container(
+            pools, transformed_pools, value.nested),
+    };
+    // include:end-apply-nested-transformations
+
+    // include:start-verify-nested-value
+    const auto expected_new = with_new_nested_t{
+        .nested =
+            {
+                new_nested_t{.str = {"_1_", "_2_", "_3_"}},
+                new_nested_t{.str = {"_1_", "_2_", "_3_", "_4_", "_5_", "_6_"}},
+            },
+    };
+    REQUIRE(new_value == expected_new);
+    // include:end-verify-nested-value
+
+    // include:start-verify-structural-sharing-of-nested
+    const auto transformed_str = immer::persist::cereal_save_with_pools(
+        new_value,
+        immer::persist::hana_struct_auto_member_name_policy(
+            with_new_nested_t{}));
+    const auto expected_transformed_json = json_t::parse(R"(
+{
+  "pools": {
+    "nested": {
+      "B": 5,
+      "BL": 3,
+      "inners": [{"key": 0, "value": {"children": [], "relaxed": false}}],
+      "leaves": [{"key": 1, "value": [{"str": 0}, {"str": 1}]}],
+      "vectors": [{"root": 0, "tail": 1}]
+    },
+    "str": {
+      "B": 5,
+      "BL": 1,
+      "inners": [
+        {"key": 0, "value": {"children": [2], "relaxed": false}},
+        {"key": 3, "value": {"children": [2, 5], "relaxed": false}}
+      ],
+      "leaves": [
+        {"key": 1, "value": ["_3_"]},
+        {"key": 2, "value": ["_1_", "_2_"]},
+        {"key": 4, "value": ["_5_", "_6_"]},
+        {"key": 5, "value": ["_3_", "_4_"]}
+      ],
+      "vectors": [{"root": 0, "tail": 1}, {"root": 3, "tail": 4}]
+    }
+  },
+  "value0": {"nested": 0}
+}
+    )");
+    // include:end-verify-structural-sharing-of-nested
+    REQUIRE(json_t::parse(transformed_str) == expected_transformed_json);
+}