syncing dev

.github/workflows/_publish.yaml_ → .github/workflows/publish-docs.yml

@@ -1,13 +1,16 @@
 name: Publish documentation site
 
-
 on:
   release:
     types: [published]
   push:
     branches:
-      - main
+      - master
       - docs
+    paths:
+      - docs/**
+      - mkdocs.yml
+      - README.md
 
 jobs:
 

CONTRIBUTING.md

@@ -18,10 +18,14 @@ Collective Knowledge (CK) and Collective Mind (CM)
 were created by [Grigori Fursin](https://arxiv.org/abs/2406.16791),
 sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons 
 to benefit everyone. Since then, this open-source automation technology
-(CM/CMX, CM4MLOps, CM4ABTF, CM4Research, etc) is being extended 
+(CM/CMX, CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, etc) is being extended 
 as a community effort thanks to all our volunteers, collaborators 
 and contributors listed here in alphabetical order:
 
+* @Henryfzh
+* @Leonard226
+* @Oseltamivir
+* @Submandarine
 * Resmi Arjun
 * Omar Benjelloun (Google)
 * Alice Cheng (Nvidia)

README.md

@@ -11,30 +11,31 @@
 
 ### About
 
-[Collective Knowledge (CK)](https://cKnowledge.org) in an educational project
-to help researchers and engineers automate their repetitive, tedious and time-consuming tasks
-to build, run, benchmark and optimize AI, ML and other applications and systems 
-across diverse and continuously changing models, data, software and hardware.
+[Collective Knowledge (CK, CM, CM4MLOps, CM4MLPerf and CMX)](https://cKnowledge.org) 
+is an educational community project to learn how to run AI, ML and other emerging workloads 
+in the most efficient and cost-effective way across diverse models, data sets, software and hardware.
 
 CK consists of several sub-projects:
 
 * [Collective Mind framework (CM)](cm) - a very lightweight Python-based framework with minimal dependencies
-  to help users implement, share and reuse cross-platform automation recipes to 
-  build, benchmark and optimize applications on any platform
-  with any software and hardware. 
-
-  * [CM interface to run MLPerf inference benchmarks](https://docs.mlcommons.org/inference)
+  intended to help researchers and engineers automate their repetitive, tedious and time-consuming tasks
+  to build, run, benchmark and optimize AI, ML and other applications and systems 
+  across diverse and continuously changing models, data, software and hardware.
 
   * [CM4MLOPS](https://github.com/mlcommons/cm4mlops) - 
     a collection of portable, extensible and technology-agnostic automation recipes
-    with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications 
+    with a common CLI and Python API (CM scripts) to unify and automate 
+    all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications 
     on diverse platforms with any software and hardware: see [online catalog at CK playground](https://access.cknowledge.org/playground/?action=scripts),
     [online MLCommons catalog](https://docs.mlcommons.org/cm4mlops/scripts) 
 
+  * [CM interface to run MLPerf inference benchmarks](https://docs.mlcommons.org/inference)
+
   * [CM4ABTF](https://github.com/mlcommons/cm4abtf) - a unified CM interface and automation recipes
     to run automotive benchmark across different models, data sets, software and hardware from different vendors.
 
-* [CMX (the next generation of CM and CM4MLOps)](cm/docs/cmx) - we are developing the next generation of CM 
+* [CMX (the next generation of CM, CM4MLOps and CM4MLPerf)](cm/docs/cmx) - 
+    we are developing the next generation of CM 
     to make it simpler and more flexible based on user feedback. Please follow 
     this project [here]( https://github.com/orgs/mlcommons/projects/46 ).
 
@@ -44,6 +45,12 @@ CK consists of several sub-projects:
   and organize [public optimization challenges and reproducibility initiatives](https://access.cknowledge.org/playground/?action=challenges) 
   to co-design more efficient and cost-effiective software and hardware for emerging workloads.
 
+  * [CM4MLPerf-results](https://github.com/mlcommons/cm4mlperf-results) - 
+    a simplified and unified representation of the past MLPerf results 
+    for further visualization and analysis using [CK graphs](https://access.cknowledge.org/playground/?action=experiments)
+    (*the new version is coming soon*).
+
+
 * [Artifact Evaluation](https://cTuning.org/ae) - automating artifact evaluation and reproducibility initiatives at ML and systems conferences.
 
 
@@ -57,11 +64,15 @@ CK consists of several sub-projects:
 * Copyright (c) 2021-2024 MLCommons
 * Copyright (c) 2014-2021 cTuning foundation
 
+### Author
+
+* [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning)
+
 ### Maintainers
 
-* CM/CM4Research: [Grigori Fursin](https://cKnowledge.org/gfursin)
-* CM4MLOps: [Arjun Suresh](https://github.com/arjunsuresh) and [Anandhu Sooraj](https://github.com/anandhu-eng)
-* CMX (the next generation of CM) [Grigori Fursin](https://cKnowledge.org/gfursin)
+* [Collective Mind (CM)](cm): [Grigori Fursin](https://cKnowledge.org/gfursin)
+* CM4MLOps (CM automation recipes): [Arjun Suresh](https://github.com/arjunsuresh) and [Anandhu Sooraj](https://github.com/anandhu-eng)
+* CMX (the next generation of CM, CM4MLOps and CM4MLPerf): [Grigori Fursin](https://cKnowledge.org/gfursin)
 
 ### Citing our project
 
@@ -73,19 +84,20 @@ To learn more about the motivation behind CK and CM technology, please explore t
 * "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ]
 * ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] 
 * ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ]
+* Journal of Royal Society'20: [ [paper](https://royalsocietypublishing.org/doi/10.1098/rsta.2020.0211) ]
 
 
 ### CM Documentation
 
 * [CM installation GUI](https://access.cknowledge.org/playground/?action=install)
-* [CM Getting Started Guide and FAQ](docs/getting-started.md)
-  * [Common CM interface to run MLPerf inference benchmarks](docs/mlperf/inference)
-  * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](docs/tutorials/common-interface-to-reproduce-research-projects.md)
+* [CM Getting Started Guide and FAQ](https://github.com/mlcommons/ck/tree/master/docs/getting-started.md)
+  * [Common CM interface to run MLPerf inference benchmarks](https://github.com/mlcommons/ck/tree/master/docs/mlperf/inference)
+  * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](https://github.com/mlcommons/ck/tree/master/docs/tutorials/common-interface-to-reproduce-research-projects.md)
   * [CM automation recipes for MLOps and DevOps](https://access.cknowledge.org/playground/?action=scripts)
-  * [Other CM tutorials](docs/tutorials)
-* [Full documentation](docs/README.md)
-* [CM development tasks](docs/taskforce.md#current-tasks)
-* [CM and CK history](docs/history.md)
+  * [Other CM tutorials](https://github.com/mlcommons/ck/tree/master/docs/tutorials)
+* [Full documentation](https://github.com/mlcommons/ck/tree/master/docs/README.md)
+* [CM taskforce](https://github.com/mlcommons/ck/tree/master/docs/taskforce.md)
+* [CMX, CM and CK history](https://github.com/mlcommons/ck/tree/master/docs/history.md)
 
 
 ### Acknowledgments
@@ -95,6 +107,8 @@ CM4Research and CMX) was created by [Grigori Fursin](https://cKnowledge.org/gfur
 and sponsored by cTuning.org, OctoAI and HiPEAC.
 Grigori donated CK to MLCommons to benefit the community
 and to advance its development as a collaborative, community-driven effort.
-We thank MLCommons and FlexAI for supporting this project,
+
+We thank [MLCommons](https://mlcommons.org), [FlexAI](https://flex.ai) 
+and [cTuning](https://cTuning.org) for supporting this project,
 as well as our dedicated [volunteers and collaborators](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)
 for their feedback and contributions!

cm/CHANGES.md

@@ -1,3 +1,13 @@
+## V3.4.4
+   - improved error reporting in utils.load_json and utils.load_yaml
+   - added utils.substitute_template
+   - enhanced 'cm pull repo' based on user feedback
+
+## V3.4.3
+   - fixed pyproject.toml
+   - fixed docs generation
+   - improved  cm pull repo with branch and SHA (#1355)
+
 ## V3.4.2
    - added utils.flatten_dict
    - added utils.safe_int

cm/MANIFEST.in

@@ -0,0 +1,3 @@
+include CHANGES.md
+include CONTRIBUTING.md
+include COPYRIGHT.txt

cm/README.md

@@ -9,18 +9,20 @@
 
 ### About
 
-Collective Mind (CM) is a small, modular, cross-platform and decentralized workflow automation framework 
-with a human-friendly interface to make it easier to build, run, benchmark and optimize applications 
-across diverse models, data sets, software and hardware.
+Collective Mind (CM) is a small [Python package](https://pypi.org/project/cmind)
+with a unified CLI and API designed for creating and managing 
+portable and technology-agnostic automations for MLOps, DevOps and ResearchOps.
 
+It is intended to make it easier to build, run, benchmark and optimize applications 
+across diverse models, data sets, software and hardware.
+
 CM is a part of [Collective Knowledge (CK)](https://github.com/mlcommons/ck) - 
-an educational community project to learn how to run emerging workloads 
+an educational community project to learn how to run AI, ML and other emerging workloads 
 in the most efficient and cost-effective way across diverse 
-and continuously changing systems.
+and continuously changing systems using the MLPerf benchmarking methodology.
 
-CM includes a collection of portable, extensible and technology-agnostic automation recipes
-with a common API and CLI (aka CM scripts) to unify and automate different steps 
-required to compose, run, benchmark and optimize complex ML/AI applications 
+CM includes a [collection of portable, extensible and technology-agnostic automation recipes](https://access.cknowledge.org/playground/?action=scripts)
+(aka CM scripts) to unify and automate different steps required to compose, run, benchmark and optimize complex ML/AI applications 
 on any platform with any software and hardware. 
 
 CM scripts extend the concept of `cmake` with simple Python automations, native scripts
@@ -44,9 +46,9 @@ from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors:
   and simple JSON/YAML descriptions instead of inventing new workflow languages;
 * must have the same interface to run all automations natively, in a cloud or inside containers.
 
-### Maintainers
+### Author and maintainer
 
-* [Grigori Fursin](https://cKnowledge.org/gfursin)
+* [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning)
 
 ### Resources
 
@@ -83,6 +85,8 @@ developed by [Grigori Fursin](https://cKnowledge.org/gfursin),
 as a part of the [Collective Knowledge educational initiative](https://cKnowledge.org),
 sponsored by [cTuning.org](https://cTuning.org) and [cKnowledge.org](https://cKnowledge.org), 
 and contributed to MLCommons for the benefit of all. 
+
 This open-source technology, including CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, and more, 
-is a collaborative community-driven project made possible by our 
-[amazing volunteers, collaborators, and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! 
+is a collaborative project supported by [MLCommons](https://mlcommons.org), 
+[FlexAI](https://flex.ai), [cTuning](https://cTuning.org)
+and our [amazing volunteers, collaborators, and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! 

cm/cmind/__init__.py

@@ -2,7 +2,7 @@
 #
 # Written by Grigori Fursin
 
-__version__ = "3.4.2"
+__version__ = "3.4.4"
 
 from cmind.core import access
 from cmind.core import x

cm/cmind/repo/automation/repo/module.py

@@ -34,6 +34,7 @@ def pull(self, i):
           (new_branch) (str): Create new Git branch
           (checkout) (str): Git checkout
           (checkout_only) (bool): only checkout existing repo
+          (dir) (str): use repository in this directory
           (depth) (int): Git depth
           (desc) (str): brief repository description (1 line)
           (prefix) (str): extra directory to keep CM artifacts
@@ -54,11 +55,11 @@ def pull(self, i):
 
         console = i.get('out') == 'con'
 
-        alias = i.get('artifact','')
-        url = i.get('url','')
-        desc = i.get('desc','')
-        prefix = i.get('prefix','')
-        pat = i.get('pat','')
+        alias = i.get('artifact', '')
+        url = i.get('url', '')
+        desc = i.get('desc', '')
+        prefix = i.get('prefix', '')
+        pat = i.get('pat', '')
 
         extra_cmd_git = i.get('extra_cmd_git', '')
         extra_cmd_pip = i.get('extra_cmd_pip', '')
@@ -132,13 +133,14 @@ def pull(self, i):
             branch = i.get('branch', '')
             new_branch = i.get('new_branch', '')
             checkout = i.get('checkout', '')
+            _dir = i.get('dir', '')
 
             r = net.request({'get': {'action': 'check-migration-repo-notes', 'repo': url, 'branch': branch, 'checkout': checkout}})
             notes = r.get('dict', {}).get('notes','')
             if notes !='':
                 print (notes)
 
-            if alias == 'mlcommons@ck' and branch == '' and checkout == '':
+            if alias == 'mlcommons@ck' and branch == '' and checkout == '' and _dir == '':
                 print ('=========================================================================')
                 print ('Warning: mlcommons@ck was automatically changed to mlcommons@cm4mlops.')
                 print ('If you want to use older mlcommons@ck repository, use branch or checkout.')
@@ -153,6 +155,7 @@ def pull(self, i):
                            'branch': branch,
                            'new_branch': new_branch,
                            'checkout': checkout,
+                           'dir': _dir,
                            'depth': i.get('depth', '')}]
 
 
@@ -173,6 +176,7 @@ def pull(self, i):
              checkout = repo.get('checkout','')
              depth = repo.get('depth','')
              path_to_repo = repo.get('path_to_repo', None)
+             _dir = repo.get('dir', '')
 
              if console:
                  print (self.cmind.cfg['line'])
@@ -185,6 +189,8 @@ def pull(self, i):
                      print ('New branch: {}'.format(new_branch))
                  if checkout!='':
                      print ('Checkout:   {}'.format(checkout))
+                 if _dir!='':
+                     print ('Directory:  {}'.format(_dir))
                  if depth!='' and depth!=None:
                      print ('Depth:      {}'.format(str(depth)))
                  print ('')
@@ -197,6 +203,7 @@ def pull(self, i):
                             branch = branch,
                             new_branch = new_branch,
                             checkout = checkout,
+                            _dir = _dir,
                             console = console,
                             desc=desc,
                             prefix=prefix,

cm/cmind/repos.py

@@ -99,6 +99,17 @@ def load(self, init = False):
             r = utils.save_json(full_path_to_repo_paths, meta = self.paths)
             if r['return']>0: return r
 
+        # Skip internal repos
+        skip_internal_repos = os.environ.get('CM_CORE_SKIP_INTERNAL_REPOS','').strip().lower()
+        if skip_internal_repos not in ['1', 'true', 'yes']:
+            import pkgutil
+            for mi, name, ispkg in pkgutil.iter_modules():
+                if name.startswith('cm') and name != 'cmind':
+                    path = os.path.join(mi.path, name, 'repo')
+                    path_cmr = os.path.join(path, 'cmr.yaml')
+                    if os.path.isfile(path_cmr) and path not in self.paths:
+                        self.paths.insert(0, path)
+
         # Check internal repo (will be after local)
         if self.path_to_internal_repo != '' and os.path.isdir(self.path_to_internal_repo):
             self.paths.insert(0, self.path_to_internal_repo)
@@ -294,7 +305,7 @@ def process(self, repo_path, mode='add'):
         return rr
 
     ############################################################
-    def pull(self, alias, url = '', branch = '', checkout = '', console = False, desc = '', prefix = '', depth = None, 
+    def pull(self, alias, url = '', branch = '', checkout = '', _dir = '', console = False, desc = '', prefix = '', depth = None, 
                     path_to_repo = None, checkout_only = False, skip_zip_parent_dir = False,
                     extra_cmd_git = '', extra_cmd_pip = '', new_branch = ''):
         """
@@ -307,6 +318,7 @@ def pull(self, alias, url = '', branch = '', checkout = '', console = False, des
             (new_branch) (str): Create new  branch
             (checkout) (str): Git repository checkout
             (checkout_only) (bool): only checkout existing repo
+            (_dir) (str): use repository in this directory
             (depth) (int): Git repository depth
             (console) (bool): if True, print some info to console
             (desc) (str): optional repository description
@@ -335,6 +347,9 @@ def pull(self, alias, url = '', branch = '', checkout = '', console = False, des
         if path_to_repo == None:
             path_to_repo = os.path.join(self.full_path_to_repos, alias)
 
+        if _dir != '': 
+            path_to_repo = os.path.join(path_to_repo, _dir)
+
         if console:
             print ('Local path: '+path_to_repo)
             print ('')

cm/cmind/utils.py

@@ -204,7 +204,7 @@ def load_json(file_name, check_if_exists = False, encoding='utf8'):
         try:
             meta = json.load(jf)
         except Exception as e:
-            return {'return':4, 'error': format(e)}
+            return {'return':4, 'error': f'detected problem in {file_name}: {e}'}
 
     return {'return':0, 'meta': meta}
 
@@ -274,7 +274,7 @@ def load_yaml(file_name, check_if_exists = False, encoding = 'utf8'):
                 # To support old versions
                 meta = yaml.safe_load(yf)
         except Exception as e:
-            return {'return':4, 'error': format(e)}
+            return {'return':4, 'error': f'detected problem in {file_name}: {e}'}
 
     return {'return':0,
             'meta': meta}
@@ -2152,3 +2152,21 @@ def digits(s, first = True):
        pass
 
     return v
+
+##############################################################################
+def substitute_template(template, variables):
+    """
+    Substitutes variables in a template string with values from a dictionary.
+
+    Args:
+        template (str): The template string with placeholders (e.g., "something-{var1}-something-{var2}").
+        vars (dict): A dictionary containing variable-value pairs (e.g., {'var1': 'a', 'var2': 'b'}).
+
+    Returns:
+        str: The template string with placeholders replaced by the corresponding values.
+    """
+    try:
+        return template.format(**variables)
+    except KeyError as e:
+        return f"Error: Missing value for {e.args[0]} in the vars dictionary."
+

cm/pyproject.toml

@@ -1,5 +1,3 @@
-# Developer(s): Grigori Fursin
-
 [build-system]
 build-backend = "setuptools.build_meta"
 requires = ["setuptools"]
@@ -62,6 +60,7 @@ version = {attr = "cmind.__version__"}
 
 [tool.setuptools]
 zip-safe = false
+include-package-data = true
 
 [tool.setuptools.package-data]
 cmind = ["repo/*",