YAML loading and dumping improvements

changelog/62932.fixed

@@ -0,0 +1,37 @@
+Improvements to YAML processing (`salt.utils.yaml`) that take effect
+immediately:
+  * Passing a non-`dict` class to the `salt.utils.yaml.SaltYamlSafeLoader`
+    constructor no longer causes all future `!!omap` nodes to raise an exception
+    when loaded.
+  * Loading a sequence node tagged with `!!python/tuple` is now supported, and
+    produces a Python `tuple` object.
+
+Improvements to YAML processing that are expected to take effect in Salt 3007
+but can be previewed now by setting the option `yaml_compatibility` to `3007`:
+  * Loading a well-formed `!!omap` node (a sequence of single-entry mapping
+    nodes) will always produce a `collections.OrderedDict` object.  (Currently
+    it sometimes produces a `list` of `tuple` (key, value) pairs and sometimes
+    raises an exception, depending on the value of
+    `salt.utils.yaml.SaltYamlSafeLoader`'s `dictclass` constructor parameter.)
+  * Loading a mapping node tagged with `!!omap` will always raise an exception
+    due to invalid syntax.  (Currently it sometimes raises an exception and
+    sometimes produces a `collections.OrderedDict` object, depending on the
+    value of `salt.utils.yaml.SaltYamlSafeLoader`'s `dictclass` constructor
+    parameter.)
+  * Dumping a `collections.OrderedMap` will consistently produce an `!!omap`
+    node (a sequence of single-entry mapping nodes).
+  * Loading an explicitly tagged `!!timestamp` node will produce a
+    `datetime.datetime` object instead of a string.
+  * Dumping a `datetime.datetime` object will explicitly tag the node with
+    `!!timestamp`.  (Currently the nodes are untagged.)
+  * Dumping a `tuple` object will consistently produce a sequence node
+    explicitly tagged with `!!python/tuple`.  (Currently `safe_dump()` omits the
+    tag and `dump()` includes it.)
+  * `salt.utils.yaml.dump()` will default to `salt.utils.yaml.OrderedDumper`
+    instead of `yaml.Dumper`.
+  * Dumping a YAML sequence with `salt.utils.yaml.IndentedSafeOrderedDumper`
+    will be properly indented.
+  * Dumping an object with an unsupported type via `salt.utils.yaml.safe_dump()`
+    (or via `dump()` with the `SafeOrderedDumper` or `IndentedSafeOrderedDumper`
+    classes) will raise an exception.  (Currently such objects produce a null
+    node.)

changelog/63158.fixed

@@ -0,0 +1,2 @@
+Updated YAML idiosyncrasies documentation, improved YAML tests, and improved
+readability of YAML code

doc/ref/configuration/master.rst

@@ -1177,6 +1177,62 @@ See the :ref:`2019.2.1 release notes <release-2019-2-1>` for more details.
 
     use_yamlloader_old: False
 
+.. conf_master:: yaml_compatibility
+
+``yaml_compatibility``
+----------------------
+
+.. versionadded:: 3006.0
+
+Default: ``None``
+
+Force the behavior of the YAML loader (parser) and dumper (generator) to match
+the default behavior from a specific version of Salt.  Omitting this or setting
+it to ``None`` (in YAML: ``null``, ``~``, or an empty value) is equivalent to
+setting it to the current version of Salt.
+
+Changes that are expected in a future version of Salt can be previewed by
+setting this to that future version number.
+
+This setting does not take effect until after the config file is parsed (so
+setting this value does not change how the config file itself is parsed).
+
+Note that pillar `.sls` files are parsed on the master so setting this on the
+master affects pillar YAML files.  State `.sls` files are parsed by the minions,
+so setting this on the master does not affect how state YAML files are parsed.
+
+.. code-block:: yaml
+
+    # Use the default behavior.
+    yaml_compatibility: null
+
+    # Any of the following will force the YAML loader/dumper to behave like
+    # it did in Salt v3006.0:
+    yaml_compatibility: 3006
+    yaml_compatibility: 3006.0
+    yaml_compatibility: v3006
+    yaml_compatibility: v3006.0
+    yaml_compatibility: sulfur
+    yaml_compatibility: Sulfur
+    yaml_compatibility: SULFUR
+
+.. conf_master:: yaml_compatibility_warnings
+
+``yaml_compatibility_warnings``
+-------------------------------
+
+.. versionadded:: 3006.0
+
+Default: ``True``
+
+Boolean indicating whether to print warnings about upcoming changes to the YAML
+loader/dumper that could affect compatibility.  Set to `False` to silence the
+warnings.
+
+.. code-block:: yaml
+
+    yaml_compatibility_warnings: True
+
 .. conf_master:: req_server_niceness
 
 ``req_server_niceness``

doc/ref/configuration/minion.rst

@@ -1687,6 +1687,63 @@ See the :ref:`2019.2.1 release notes <release-2019-2-1>` for more details.
 
     use_yamlloader_old: False
 
+.. conf_minion:: yaml_compatibility
+
+``yaml_compatibility``
+----------------------
+
+.. versionadded:: 3006.0
+
+Default: ``None``
+
+Force the behavior of the YAML loader (parser) and dumper (generator) to match
+the default behavior from a specific version of Salt.  Omitting this or setting
+it to ``None`` (in YAML: ``null``, ``~``, or an empty value) is equivalent to
+setting it to the current version of Salt.
+
+Changes that are expected in a future version of Salt can be previewed by
+setting this to that future version number.
+
+This setting does not take effect until after the config file is parsed (so
+setting this value does not change how the config file itself is parsed).
+
+Note that pillar `.sls` files are parsed on the master so setting this on the
+minion does not affect pillar YAML files.  State `.sls` files are parsed by the
+minions, so setting this on the minion does affect how state YAML files are
+parsed.
+
+.. code-block:: yaml
+
+    # Use the default behavior.
+    yaml_compatibility: null
+
+    # Any of the following will force the YAML loader/dumper to behave like
+    # it did in Salt v3006.0:
+    yaml_compatibility: 3006
+    yaml_compatibility: 3006.0
+    yaml_compatibility: v3006
+    yaml_compatibility: v3006.0
+    yaml_compatibility: sulfur
+    yaml_compatibility: Sulfur
+    yaml_compatibility: SULFUR
+
+.. conf_minion:: yaml_compatibility_warnings
+
+``yaml_compatibility_warnings``
+-------------------------------
+
+.. versionadded:: 3006.0
+
+Default: ``True``
+
+Boolean indicating whether to print warnings about upcoming changes to the YAML
+loader/dumper that could affect compatibility.  Set to `False` to silence the
+warnings.
+
+.. code-block:: yaml
+
+    yaml_compatibility_warnings: True
+
 Docker Configuration
 ====================
 

doc/topics/troubleshooting/yaml_idiosyncrasies.rst

@@ -13,6 +13,25 @@ unexpected times.
 
 .. _`YAML`: https://yaml.org/spec/1.1/
 
+.. versionchanged:: 3006.0
+
+    All of the behavior changes slated for Salt v3007.0 can be previewed by
+    setting the ``yaml_compatibility`` option to 3007.
+
+.. versionchanged:: 3006.0
+
+    YAML compatibility warnings can be disabled by setting the
+    ``yaml_compatibility_warnings`` option to False.
+
+.. versionchanged:: 3007.0
+
+    Dumping an object of unsupported type to YAML with a one of the safe dumpers
+    (:py:func:`~salt.utils.yaml.safe_dump`, or :py:func:`~salt.utils.yaml.dump`
+    with the :py:class:`~salt.utils.yaml.SafeOrderedDumper` or
+    :py:class:`~salt.utils.yaml.IndentedSafeOrderedDumper` classes) now raises
+    an exception.  Previously it produced a YAML ``NULL`` node.  Set the
+    ``yaml_compatibility`` option to 3006 to revert to the previous behavior.
+
 Spaces vs Tabs
 ==============
 
@@ -382,50 +401,134 @@ Here's an example:
 Automatic ``datetime`` conversion
 =================================
 
-If there is a value in a YAML file formatted ``2014-01-20 14:23:23`` or
-similar, YAML will automatically convert this to a Python ``datetime`` object.
-These objects are not msgpack serializable, and so may break core salt
-functionality.  If values such as these are needed in a salt YAML file
-(specifically a configuration file), they should be formatted with surrounding
-strings to force YAML to serialize them as strings:
+.. versionchanged:: 2018.3.0
+
+    A YAML scalar node containing a timestamp now always produces a string.
+    Previously, Salt would attempt to create a Python ``datetime.datetime``
+    object, even if the node contained an invalid date (for example,
+    ``4017-16-20``).
+
+.. versionchanged:: 3007.0
+
+    Loading a YAML ``!!timestamp`` node now produces a ``datetime.datetime``
+    object.  Previously, nodes tagged with ``!!timestamp`` produced strings.
+    Set the ``yaml_compatibility`` option to 3006 to revert to the previous
+    behavior.
+
+.. versionchanged:: 3007.0
+
+    Dumping a ``datetime.datetime`` object to YAML now explicitly tags the node
+    with ``!!timestamp``.  Previously, the ``!!timestamp`` tag was omitted.
+    Set the ``yaml_compatibility`` option to 3006 to revert to the previous
+    behavior.
+
+Salt overrides PyYAML's default behavior and loads YAML nodes that look like
+timestamps as strings:
 
 .. code-block:: pycon
 
-    >>> import yaml
-    >>> yaml.safe_load("2014-01-20 14:23:23")
-    datetime.datetime(2014, 1, 20, 14, 23, 23)
-    >>> yaml.safe_load('"2014-01-20 14:23:23"')
+    >>> import salt.utils.yaml
+    >>> salt.utils.yaml.safe_load("2014-01-20 14:23:23")
     '2014-01-20 14:23:23'
 
-Additionally, numbers formatted like ``XXXX-XX-XX`` will also be converted (or
-YAML will attempt to convert them, and error out if it doesn't think the date
-is a real one).  Thus, for example, if a minion were to have an ID of
-``4017-16-20`` the minion would not start because YAML would complain that the
-date was out of range.  The workaround is the same, surround the offending
-string with quotes:
+To force Salt to produce a ``datetime.datetime`` object instead of a string,
+explicitly tag the node with ``!!timestamp``:
 
 .. code-block:: pycon
 
-    >>> import yaml
-    >>> yaml.safe_load("4017-16-20")
-    Traceback (most recent call last):
-      File "<stdin>", line 1, in <module>
-      File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 93, in safe_load
-        return load(stream, SafeLoader)
-      File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 71, in load
-        return loader.get_single_data()
-      File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 39, in get_single_data
-        return self.construct_document(node)
-      File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 43, in construct_document
-        data = self.construct_object(node)
-      File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 88, in construct_object
-        data = constructor(self, node)
-      File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 312, in construct_yaml_timestamp
-        return datetime.date(year, month, day)
-    ValueError: month must be in 1..12
-    >>> yaml.safe_load('"4017-16-20"')
-    '4017-16-20'
+    >>> import salt.utils.yaml
+    >>> salt.utils.yaml.safe_load("!!timestamp 2014-01-20 14:23:23")
+    datetime.datetime(2014, 1, 20, 14, 23, 23)
+
+When dumping a ``datetime.datetime`` object to YAML, Salt tags the node with
+``!!timestamp`` so that it will be loaded back as a ``datetime.datetime``
+object.
+
+Beware that Salt is currently unable to serialize ``datetime.datetime`` objects,
+so ``!!timestamp`` nodes cannot be used in pillar SLS files.
+
+Ordered Dictionaries
+====================
+
+.. versionchanged:: 3007.0
+
+    Loading a YAML ``!!omap`` node now reliably produces a
+    ``collections.OrderedDict`` object.  Previously, an ``!!omap`` node would
+    sometimes produce a ``list`` of ``tuple`` (key, value) pairs and sometimes
+    raise an exception.  Set the ``yaml_compatibility`` option to 3006 to revert
+    to the previous behavior.
+
+.. versionchanged:: 3007.0
+
+    Dumping any ``collections.OrderedDict`` object to YAML now reliably produces
+    an ``!!omap`` node.  Previously, only the subtype
+    ``salt.utils.odict.OrderedDict`` was supported, and it always produced a
+    plain mapping node.  Other types produced various constructs that could not
+    be loaded back as any kind of ``dict``.  Set the ``yaml_compatibility``
+    option to 3006 to revert to the previous behavior.
+
+The YAML specification defines an `ordered mapping type
+<https://yaml.org/type/omap>`_ which is equivalent to a plain mapping except
+iteration order is preserved.  (YAML makes no guarantees about iteration order
+for entries loaded from a plain mapping.)
+
+Ordered mappings are represented as an ``!!omap`` tagged sequence of
+single-entry mappings:
+
+.. code-block:: yaml
+
+    !!omap
+    - key1: value1
+    - key2: value2
+
+Starting with Python 3.6, plain ``dict`` objects iterate in insertion order so
+there is no longer a strong need for the ``!!omap`` type.  However, some users
+may prefer the ``!!omap`` type over the plain ``!!map`` type because (1) it
+makes it obvious that the order of entries is significant, and (2) it provides a
+stronger guarantee of iteration order (plain mapping iteration order can be
+thought of as a Salt implementation detail that may change in the future).
+
+Salt produces a ``collections.OrderedDict`` object when it loads an ``!!omap``
+node.  (Salt's behavior differs from PyYAML's default behavior, which is to
+produce a ``list`` of (key, value) ``tuple`` objects.)  These objects are a
+subtype of ``dict``, so ``!!omap`` is a drop-in replacement for a plain mapping.
+
+When dumping a ``collections.OrderedDict`` object to YAML, Salt produces an
+``!!omap`` node.
+
+Beware that Salt currently serializes ``collections.OrderedDict`` objects the
+same way it serializes plain ``dict`` objects, so they become plain ``dict``
+objects when deserialized by the recipient.
+
+Tuples
+======
+
+.. versionchanged:: 3006.0
+
+    Loading a YAML ``!!python/tuple`` node is now supported.
+
+.. versionchanged:: 3007.0
+
+    Dumping a ``tuple`` object to YAML now always produces a sequence node
+    tagged with ``!!python/tuple``.  Previously, ``salt.utils.yaml.safe_dump()``
+    did not tag the node.  Set the ``yaml_compatibility`` option to 3006 to
+    revert to the previous behavior.
+
+The YAML ``!!python/tuple`` type can be used to produce a Python ``tuple``
+object when loaded:
+
+.. code-block:: yaml
+
+    !!python/tuple
+    - first item
+    - second item
+
+When dumped to YAML, a ``tuple`` object produces a sequence node tagged with
+``!!python/tuple``.
 
+Beware that Salt currently serializes ``tuple`` objects the same way it
+serializes ``list`` objects, so they become ``list`` objects when deserialized
+by the recipient.
 
 Keys Limited to 1024 Characters
 ===============================