integrate WTH guidelines into main document

cookbook/sections/data-publishers.adoc

@@ -303,3 +303,147 @@ def run():
 if __name__ == '__main__':
     run()
 ----
+
+=== Defining and proposing topic for WMO Earth system disciplines
+
+The https://wmo-im.github.io/wis2-topic-hierarchy/standard/wis2-topic-hierarchy-STABLE.html[WIS2 Topic Hierarchy] describes the topic structure and levels to be used when publishing WIS2 notification messages.
+
+The purpose of this document is to provide guidelines to domain experts so that the Topic Hierarchy definition is consistent and useful to address the needs of WIS2 users.
+
+==== Core levels
+
+Core topics are defined in the first 7 levels and address all Earth system disciplines in a consistent manner.
+
+.Core topic examples:
+*cache/a/wis2/ma-marocmeteo/core/data/weather*
+*origin/a/wis2/de-dwd/core/data/ocean*
+*cache/a/wis2/jp-jma/core/data/weather
+
+The Earth system disciplines https://codes.wmo.int/wis/topic-hierarchy/_earth-system-discipline[defined] by WTH are as follows:
+
+* `atmospheric-composition`
+* `climate`
+* `cryosphere`
+* `hydrology`
+* `ocean`
+* `space-weather`
+* `weather`
+
+Topics within earch Earth system discipline are then defined by domain experts, reviewed, approved, and published by WMO.
+
+==== Additional Earth domain specific topics levels
+
+===== What is the use of the Topic Hierarchy ?
+
+The definition of the Topic Hierarchy is heavily linked to the use of Publish-Subscribe (Pub/Sub) protocols, here MQTT[S], in WIS2. Users, when subscribing to the Global Broker, can decide which notification messages they wish to receive.
+
+===== MQTT wildcards
+
+For example, connecting to a Global Broker and subscribing to:
+
+``cache/a/wis2/+/core/data/weather/surface-weather-observation/synop``
+
+users will receive a notification when a new synop is made available from *any* centre-id publishing message on WIS2.  The `+` character is a single level wildcard for MQTT subscription.
+
+A usder can also choose to subscribe to:
+
+``cache/a/wis2/de-dwd/core/data/weather/#``
+
+In this case, users will receive notification messages from the `de-dwd` centre-id for *all* weather data.  The `#` character is a multiple level wildcard for MQTT subscription, and can only be used at the end of a topic subscription.
+
+By using `+` and `#` (the two defined wildcards in the MQTT[S] protocol), users can extend their subscription and receive all messages they need.
+
+The purpose of additional topic hierarchy levels is to allow users to get the messages they need and to be more specific in their request.
+
+==== Dos and don'ts when defining Earth system discipline topics
+
+*Define only a _small number_ of additional levels*
+
+According the the MQTT[S] protocol specification, the accepted length of a given topic is more than 65kB.  This means that the number of levels can be extremely large.  However, as explained in https://www.emqx.com/en/blog/advanced-features-of-mqtt-topics:
+
+_Try not to use more topic levels “just because I can”. For example, `my-home/room1/data` is a better choice than `my/home/room1/data`._
+
+By default, some MQTT brokers are configured to accept a maximum of 10 levels. This can be changed in the configuration of the broker, however, this limit shows that a usable, practical topic structure should not be too deep (i.e. with a large number of levels).
+
+For WIS2, and considering the various Earth system disciplines, a limit of *4* sublevels seems appropriate.
+
+*Do not use the topic as a metadata record*
+
+When defining the topic, experts must focus on the needs of users.  The purpose of the WIS2 Topic Hierarchy is *not* to describe as precisely as possible what data users will obtain if they decide to subscribe and then download.  Rather, it is _only_ to provide a filtering mechanism so that users will not be flooded by WIS2 Notification Messages that may not be useful for them.
+
+In WIS2, all datasets *must* be described using the https://wmo-im.github.io/wcmp2/standard/wcmp2-STABLE.html[WMO Core Metadata Profile version 2(WCMP2)].  Users will be able to discover the data they need by searching the WIS2 catalogue using (via search engines, directly, or from portals and applications).  The topic hierarchy information will be part of the metadata record for data which provides real-time notifications of publication.
+
+*Do not allow locally defined sublevels outside the `experimental` topic
+
+Each Earth system discipline provides an `experimental` topic.  For example, for `weather`, the first additional levels in the domain topic hierarchy are:
+
+[cols="1,1"]
+|===
+|Name|Description
+|`advisories-warnings`
+|Advisories and warnings
+
+|`aviation`
+|Aviation
+
+|`prediction`
+|Data sets produced by quantitative algorithms, such as numerical or statistical prediction models, describing the past, present and future meteorological states
+
+|`space-based-observations`
+|Space based observations
+
+|`surface-based-observations`
+|Surface based observations
+|`experimental`
+|Experimental topics
+|===
+
+As the name suggests, `experimental` allows for defining additional topics for tests and experiments.  This is not meant to be used for operational data exchange.  It should only be used for testing purposes.
+
+With the exclusion of the `experimental` topic level, the topic Hierarchy must be _fully_ defined for each Earth system discipline.
+
+In some situations, it might be tempting for a data producer to use additional topic levels to restrict even more the number of messages received by the users. 
+
+This is *forbidden*. 
+
+In each Earth system discipline community, all WIS Centres will be able to use the entire topic hierarchy of the domain if they provide data corresponding to each topic. A WIS Centre will not be allowed to add additional sublevels or undefined level within the Topic Hierarchy for its own needs.
+
+Modification of the Topic Hierarchy will be possible by using the WMO fast-track approval process.
+
+*Consider users needs and prevent complex wildcard subscriptions*
+
+The purpose of the WIS2 Topic Hierarchy is to inform users about the availability of new data. In WIS2, obtaining data will start, in most cases, by configuring one or more subscription to topics, as defined in the associated WCMP2 discovery metadata records, so that users will receive notifications when new data is available.
+
+The Topic Hierarchy should be defined so that users will not need to configure a very large number of different subscriptions to get the data they are interested in. 
+
+Each level in the WIS2 Topic Hierarchy should be seen as a "logical" group (as the Earth system disciplines `weather`, `ocean`... or like `synop` for `surface-based-observations`).
+
+Then, and considering that wildcard subscription (using `+`and `#` as described above), are "expensive" to manage for the brokers.
+
+For example, a topic hierarchy resulting in users subscribing to:
+
+``cache/a/wis2/+/core/data/ocean/+/some/+/thing/+/else/#``
+
+should be avoided. A subscription to the following topic:
+
+``cache/a/wis2/+/core/data/ocean/some/thing/else/#``
+
+is much more effective for both the client and the broker side.
+
+If it is likely that most users will use wildcards for particular topic levels, then, either removing that level altogether, of moving that level a the end of the topic hierarchy is also more efficient for clients and producers.
+
+If most users end up subscribing to:
+
+``cache/a/wis2/+/core/data/ocean/+/thing/+/#``
+
+then, the Topic Hierarchy could be reconsidered, so that the above subscription can be replaced by:
+
+``cache/a/wis2/+/core/data/ocean/thing/#``
+
+Reordering the levels of topics and potentially reducing the number of sublevels makes the topic hierarchy simpler and more efficient.
+
+*Facilitate client side filtering*
+
+Notification messages are small pieces of information.  MQTT[S] broker and clients are able to handle a very large number of messages.  In that sense, receiving, potentially, too many messages is not a problem. However, downloading data, depending on the size of the data might be slower and less efficient.  If, for a particular dataset, the geometry information available in the notification message is not sufficient to allow client-side filtering before download, it is suggested to provide additional information in the `properties` object of the notification message so that users can decide _before_ downloading if the data in this particular messsage is useful for them.
+
+See <<_advertise_client_side_filters_for_data_subscriptions_in_wcmp2_and_wnm>> for more information on client side filtering

cookbook/sections/introduction.adoc

@@ -4,3 +4,11 @@
 In conjunction with the Manual on the WMO Information System volume II (WMO-No. 1060) (Manual on WIS volume II: WIS 2.0), the present Guide to the WMO Information System volume II (Guide to WIS volume II: WIS 2.0) is designed to ensure adequate uniformity and standardization in the data, information and communication practices, procedures and specifications employed by Members of the World Meteorological Organization (WMO) in the operation of the WMO Information System WIS 2.0 as it supports the mission of the Organization. The Manual on WIS contains standard and recommended practices, procedures and specifications. The Guide to WIS contains additional information concerning practices, procedures, and specifications that Members are invited to follow or implement in establishing and conducting their arrangements in compliance with the WMO Technical Regulations and in developing meteorological and hydrological services.
 
 This cookbook provides various code snippets, recipes and workflow examples in support of WIS2 requirements.  This cookbook is a working document; contributions are encouraged and can be added via https://github.com/wmo-im/wis2-cookbook[GitHub].
+
+=== Key documents
+
+WIS2 Manuals and Guides can be found at the following locations:
+
+* https://library.wmo.int/idurl/4/68731[Manual on the WMO Information System]
+* https://library.wmo.int/idurl/4/69130[Guide to the WMO Information System Volume II - WMO Information System 2.0]
+* https://library.wmo.int/idurl/4/69050[Provisions for the Transition from the WMO Information System (WIS) 1.0 and Global Telecommunication System to WIS 2.0]