add documentation to precise mode

docs/overview/deep_dive/data_types.rst

@@ -95,8 +95,7 @@ Data Type Promotion
 In order to ensure that the same data type is always returned when operations are performed on arrays with different data types, regardless of which backend framework is set, Ivy has it's own set of data type promotion rules and corresponding functions.
 These rules build directly on top of the `rules <https://data-apis.org/array-api/latest/API_specification/type_promotion.html>`_ outlined in the `Array API Standard`_.
 
-The rules are simple: all data type promotions in Ivy should adhere to this `promotion table <https://github.com/unifyai/ivy/blob/db96e50860802b2944ed9dabacd8198608699c7c/ivy/__init__.py#L366>`_,
-which is the union of the Array API Standard `promotion table <https://github.com/unifyai/ivy/blob/db96e50860802b2944ed9dabacd8198608699c7c/ivy/__init__.py#L223>`_ and an extra `promotion table <https://github.com/unifyai/ivy/blob/db96e50860802b2944ed9dabacd8198608699c7c/ivy/__init__.py#L292>`_.
+The rules are simple: all data type promotions in Ivy should adhere a promotion table that extends Array API Standard `promotion table <https://github.com/unifyai/ivy/blob/7a048c1ad7193bc3033a68c1c80f0dfd5d4e74df/ivy/__init__.py#L1245-L1285>`_ using this `promotion table <https://github.com/unifyai/ivy/blob/7a048c1ad7193bc3033a68c1c80f0dfd5d4e74df/ivy/__init__.py#L1290-L1354>`_, and one of two extra `promotion tables <https://github.com/unifyai/ivy/blob/7a048c1ad7193bc3033a68c1c80f0dfd5d4e74df/ivy/__init__.py#L1356-L1400>`_ depending on precision mode that will be explained in the following section.
 
 In order to ensure adherence to this promotion table, many backend functions make use of the functions `ivy.promote_types <https://github.com/unifyai/ivy/blob/db96e50860802b2944ed9dabacd8198608699c7c/ivy/functional/ivy/data_type.py#L1804>`_, `ivy.type_promote_arrays <https://github.com/unifyai/ivy/blob/db96e50860802b2944ed9dabacd8198608699c7c/ivy/functional/ivy/data_type.py#L1940>`_, or `ivy.promote_types_of_inputs <https://github.com/unifyai/ivy/blob/db96e50860802b2944ed9dabacd8198608699c7c/ivy/functional/ivy/data_type.py#L2085>`_.
 These functions: promote data types in the inputs and return the new data types, promote the data types of the arrays in the input and return new arrays, and promote the data types of the numeric or array values inputs and return new type promoted values, respectively.
@@ -182,10 +181,39 @@ Whenever the user defines data with a specific data type, they expect a certain
 The user expects specific behaviour and memory constraints whenever they specify and use concrete data types, and those decisions should be respected.
 Therefore, Ivy does not upcast specific values to improve the stability or precision of the computation.
 
+Precise Mode
+~~~~~~~~~~~~~~~
 
-Arguments in other Functions
-----------------------------
+There are cases that arise in mixed promotion (Integer and Float, Complex and Float) that aren't covered by the Array API Standard promotion table, and depending on each use case,
+the mixed promotion rules differ as observed in different frameworks, for example Tensorflow leaves integer/floating mixed promotion undefined to make behavior utterly predictable (at some cost to user convenience), while Numpy avoids precision loss at all costs even if that meant casting the arrays to wider-than-necessary dtypes
+
+Precise Promotion Table
+"""""""""""""""""""""""""
+
+This table focuses on numerical accuracy at the cost of a higher memory footprint. A 16-bit signed or unsigned integer cannot be represented at full precision by a 16-bit float, which has only 10 bits of mantissa. Therefore, it might make sense to promote integers to floats represented by twice the number of bits. There are two disadvantages of this approach:
+
+#. It still leaves int64 and uint64 promotion undefined, because there is no standard floating point type with enough bits of mantissa to represent their full range of values. We could relax the precision constraint and use ``float64`` as the upper bound for this case.
+#. Some operations result in types that are much wider than necessary; for example mixed operations between ``uint16`` and float16 would promote all the way to ``float64``, which is not ideal.
+
+.. code-block:: python
+
+    with ivy.PreciseMode(True):
+        print(ivy.promote_types("float32","int32"))
+    # float64
 
+Non-Precise Promotion Table
+"""""""""""""""""""""""""""""""""
+The advantage of this approach is that, outside unsigned ints, it avoids all wider-than-necessary promotions: you can never get an f64 output without a 64-bit input, and you can never get an ``float32`` output without a 32-bit input: this results in convenient semantics for working on accelerators while avoiding unwanted 64-bit values. This feature of giving primacy to floating point types resembles the type promotion behavior of PyTorch.
+the disadvantage of this approach is that mixed float/integer promotion is very prone to precision loss: for example, ``int64`` (with a maximum value of 9.2*10^18 can be promoted to ``float16`` (with a maximum value of 6.5*10^4, meaning most representable values will become inf, but we are fine accepting potential loss of precision (but not loss of magnitude) in mixed type promotion which satisfies most of the use cases in deep learning scenarios.
+
+.. code-block:: python
+
+    with ivy.PreciseMode(False):
+        print(ivy.promote_types("float32","int32"))
+    # float32
+
+Arguments in other Functions
+-------------------
 All ``dtype`` arguments are keyword-only.
 All creation functions include the ``dtype`` argument, for specifying the data type of the created array.
 Some other non-creation functions also support the ``dtype`` argument, such as :func:`ivy.prod` and :func:`ivy.sum`, but most functions do not include it.
@@ -653,4 +681,4 @@ If you have any questions, please feel free to reach out on `discord`_ in the `d
 
     <iframe width="420" height="315" allow="fullscreen;"
     src="https://www.youtube.com/embed/2qOBzQdLXn4" class="video">
-    </iframe>
+    </iframe>