Upgrade Machine Learning Dependencies (#451)

Adds infrastructure to fetch RedisAI's dependencies. This removes the
need to call RedisAI's `get_deps.sh` script so that we can fetch newer
versions of our machine learning backends than the ones officially
supported by RedisAI.

Additionally, this upgrades the machine learning python packages
required by SmartSim so that they stay up to date with the backends.
This in turn allows us to add Python3.10+ONNX support.

[ committed by @MattToast ]
[ reviewed by @ashao ]

.github/workflows/run_tests.yml

@@ -110,13 +110,8 @@ jobs:
 
 
       - name: Install ML Runtimes with Smart (with pt, tf, and onnx support)
-        if: (matrix.py_v != '3.10')
         run: smart build --device cpu --onnx -v
 
-      - name: Install ML Runtimes with Smart (with pt and tf support)
-        if: (matrix.py_v == '3.10')
-        run: smart build --device cpu -v
-
       - name: Run mypy
         run: |
           python -m pip install .[mypy]

README.md

@@ -100,8 +100,8 @@ before using it on your system. Each tutorial is a Jupyter notebook that can be
 which will run a jupyter lab with the tutorials, SmartSim, and SmartRedis installed.
 
 ```bash
-docker pull ghcr.io/craylabs/smartsim-tutorials:v0.4.1
-docker run -p 8888:8888 ghcr.io/craylabs/smartsim-tutorials:v0.4.1
+docker pull ghcr.io/craylabs/smartsim-tutorials:latest
+docker run -p 8888:8888 ghcr.io/craylabs/smartsim-tutorials:latest
 # click on link to open jupyter lab
 ```
 
@@ -452,8 +452,8 @@ Each tutorial is a Jupyter notebook that can be run through the
 which will run a jupyter lab with the tutorials, SmartSim, and SmartRedis installed.
 
 ```bash
-docker pull ghcr.io/craylabs/smartsim-tutorials:v1
-docker run -p 8888:8888 ghcr.io/craylabs/smartsim-tutorials:v0.4.1
+docker pull ghcr.io/craylabs/smartsim-tutorials:latest
+docker run -p 8888:8888 ghcr.io/craylabs/smartsim-tutorials:latest
 ```
 Each of the following examples can be found in the
 [SmartSim documentation](https://www.craylabs.org/docs/tutorials/getting_started/getting_started.html).
@@ -640,15 +640,15 @@ from C, C++, Fortran and Python with the SmartRedis Clients:
     <tr>
       <td rowspan="3">1.2.7</td>
       <td>PyTorch</td>
-      <td>1.11.x</td>
+      <td>2.0.1</td>
     </tr>
     <tr>
       <td>TensorFlow\Keras</td>
-      <td>2.8.x</td>
+      <td>2.13.1</td>
     </tr>
     <tr>
       <td>ONNX</td>
-      <td>1.11.x</td>
+      <td>1.14.1</td>
     </tr>
   </tbody>
 </table>

doc/changelog.rst

@@ -17,16 +17,25 @@ Development branch
 
 To be released at some future point in time
 
+Description
+
 - Override the sphinx-tabs extension background color
+- Updated SmartSim's machine learning backends
+- Added ONNX support for Python 3.10
 
 Detailed Notes
 
 - The sphinx-tabs documentation extension uses a white background for the tabs component.
   A custom CSS for those components to inherit the overall theme color has
   been added. (SmartSim-PR453_)
+- Updated SmartSim's machine learning backends to PyTorch 2.0.1, Tensorflow
+  2.13.1, ONNX 1.14.1, and ONNX Runtime 1.16.1. As a result of this change,
+  there is now an available ONNX wheel for use with Python 3.10.
+  (SmartSim-PR451_)
 
-.. _SmartSim-PR453: https://github.com/CrayLabs/SmartSim/pull/453
 
+.. _SmartSim-PR451: https://github.com/CrayLabs/SmartSim/pull/451
+.. _SmartSim-PR453: https://github.com/CrayLabs/SmartSim/pull/453
 
 
 0.6.0

doc/installation_instructions/basic.rst

@@ -84,15 +84,14 @@ versions is dictated by our dependency on RedisAI_ 1.2.7.
 +------------------+----------+-------------+---------------+
 | RedisAI          | PyTorch  | Tensorflow  | ONNX Runtime  |
 +==================+==========+=============+===============+
-| 1.2.7 (default)  | 1.11.0   | 2.8.0       | 1.11.1        |
+| 1.2.7 (default)  | 2.0.1    | 2.13.1      | 1.16.3        |
 +------------------+----------+-------------+---------------+
 
 TensorFlow_ 2.0 and Keras_ are supported through `graph freezing`_.
 
 ScikitLearn_ and Spark_ models are supported by SmartSim as well
 through the use of the ONNX_ runtime (which is not built by
-default due to issues with glibc on a variety of Linux
-platforms and lack of support for Mac OS X).
+default due to issues with glibc on a variety of Linux platforms).
 
 .. _Spark: https://spark.apache.org/mllib/
 .. _Keras: https://keras.io
@@ -242,9 +241,9 @@ SmartSim does.
    * - Platform
      - Python Versions
    * - MacOS
-     - 3.7 - 3.10
+     - 3.8 - 3.10
    * - Linux
-     - 3.7 - 3.10
+     - 3.8 - 3.10
 
 The Python client for SmartRedis is installed through ``pip`` as follows:
 

doc/ml_features.rst

@@ -169,7 +169,7 @@ to the DB using the SmartRedis client.
 
     .. group-tab:: PyTorch
 
-        PyTorch requires models to be `jit-traced <https://pytorch.org/docs/1.11/generated/torch.jit.save.html>`__.
+        PyTorch requires models to be `jit-traced <https://pytorch.org/docs/2.0/generated/torch.jit.save.html>`__.
         The method ``torch.jit.save()`` can either store the model in memory or on file. Here,
         we will keep it in memory as a bytestring.
 
@@ -239,7 +239,7 @@ it can be uploaded to the DB using the SmartRedis client.
 
     .. group-tab:: PyTorch
 
-        PyTorch requires models to be `jit-traced <https://pytorch.org/docs/1.11/generated/torch.jit.save.html>`__.
+        PyTorch requires models to be `jit-traced <https://pytorch.org/docs/2.0/generated/torch.jit.save.html>`__.
         The method ``torch.jit.save()`` can either store the model in memory or on file. Here,
         we will save it to a file located at ``./traced_model.pt``.
 

doc/requirements-doc.txt

@@ -6,9 +6,9 @@ sphinx-copybutton==0.5.2
 sphinx-tabs==3.4.4
 nbsphinx==0.9.3
 docutils==0.18.1
-torch==1.11.0
-tensorflow==2.8.1
+torch==2.0.1
+tensorflow==2.13.1
 ipython
 jinja2==3.1.2
 protobuf
-numpy
+numpy

smartsim/_core/_cli/build.py

@@ -60,14 +60,6 @@
 
 def check_py_onnx_version(versions: Versioner) -> None:
     """Check Python environment for ONNX installation"""
-    if not versions.ONNX:
-        py_version = sys.version_info
-        msg = (
-            "An onnx wheel is not available for "
-            f"Python {py_version.major}.{py_version.minor}. "
-            "Instead consider using Python 3.8 or 3.9 for ONNX 1.11 support"
-        )
-        raise SetupError(msg)
     _check_packages_in_python_env(
         {
             "onnx": Version_(versions.ONNX),
@@ -153,7 +145,7 @@ def build_redis_ai(
     backends_table = [
         ["PyTorch", versions.TORCH, color_bool(use_torch)],
         ["TensorFlow", versions.TENSORFLOW, color_bool(use_tf)],
-        ["ONNX", versions.ONNX or "Unavailable", color_bool(use_onnx)],
+        ["ONNX", versions.ONNX, color_bool(use_onnx)],
     ]
     print(tabulate(backends_table, tablefmt="fancy_outline"), end="\n\n")
     print(f"Building for GPU support: {color_bool(device == 'gpu')}\n")

smartsim/_core/_install/buildenv.py

@@ -193,23 +193,17 @@ class RedisAIVersion(Version_):
 
     defaults = {
         "1.2.7": {
-            "tensorflow": "2.8.0",
-            "onnx": "1.11.0",
-            "skl2onnx": "1.11.1",
-            "onnxmltools": "1.11.1",
-            "scikit-learn": "1.1.1",
-            "torch": "1.11.0",
+            "tensorflow": "2.13.1",
+            "onnx": "1.14.1",
+            "skl2onnx": "1.16.0",
+            "onnxmltools": "1.12.0",
+            "scikit-learn": "1.3.2",
+            "torch": "2.0.1",
             "torch_cpu_suffix": "+cpu",
-            "torch_cuda_suffix": "+cu113",
-            "torchvision": "0.12.0",
+            "torch_cuda_suffix": "+cu117",
+            "torchvision": "0.15.2",
         },
     }
-    # Remove options with unsported wheels for python>=3.10
-    if sys.version_info >= (3, 10):
-        defaults["1.2.7"].pop("onnx")
-        defaults["1.2.7"].pop("skl2onnx")
-        defaults["1.2.7"].pop("onnxmltools")
-        defaults["1.2.7"].pop("scikit-learn")
 
     def __init__(self, vers: str) -> None:  # pylint: disable=super-init-not-called
         min_rai_version = min(Version_(ver) for ver in self.defaults)
@@ -304,33 +298,19 @@ class Versioner:
     # TensorFlow and ONNX only use the defaults, but these are not built into
     # the RedisAI package and therefore the user is free to pick other versions.
     TENSORFLOW = Version_(REDISAI.tensorflow)
-    try:
-        ONNX = Version_(REDISAI.onnx)
-    except AttributeError:
-        ONNX = None
-
-    def as_dict(self, db_name: DbEngine = "REDIS") -> t.Dict[str, t.Any]:
-        packages = [
-            "SMARTSIM",
-            "SMARTREDIS",
-            db_name,
-            "REDISAI",
-            "TORCH",
-            "TENSORFLOW",
-        ]
-        versions = [
-            self.SMARTSIM,
-            self.SMARTREDIS,
-            self.REDIS,
-            self.REDISAI,
-            self.TORCH,
-            self.TENSORFLOW,
-        ]
-        if self.ONNX:
-            packages.append("ONNX")
-            versions.append(self.ONNX)
-        vers = {"Packages": packages, "Versions": versions}
-        return vers
+    ONNX = Version_(REDISAI.onnx)
+
+    def as_dict(self, db_name: DbEngine = "REDIS") -> t.Dict[str, t.Tuple[str, ...]]:
+        pkg_map = {
+            "SMARTSIM": self.SMARTSIM,
+            "SMARTREDIS": self.SMARTREDIS,
+            db_name: self.REDIS,
+            "REDISAI": self.REDISAI,
+            "TORCH": self.TORCH,
+            "TENSORFLOW": self.TENSORFLOW,
+            "ONNX": self.ONNX,
+        }
+        return {"Packages": tuple(pkg_map), "Versions": tuple(pkg_map.values())}
 
     def ml_extras_required(self) -> t.Dict[str, t.List[str]]:
         """Optional ML/DL dependencies we suggest for the user.