diff --git a/.github/workflows/_publish.yaml_ b/.github/workflows/publish-docs.yml similarity index 87% rename from .github/workflows/_publish.yaml_ rename to .github/workflows/publish-docs.yml index 97ef2a8665..368007a7dc 100644 --- a/.github/workflows/_publish.yaml_ +++ b/.github/workflows/publish-docs.yml @@ -1,13 +1,16 @@ name: Publish documentation site - on: release: types: [published] push: branches: - - main + - master - docs + paths: + - docs/** + - mkdocs.yml + - README.md jobs: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6a3e3574a9..e012953e8a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,10 +18,14 @@ Collective Knowledge (CK) and Collective Mind (CM) were created by [Grigori Fursin](https://arxiv.org/abs/2406.16791), sponsored by cKnowledge.org and cTuning.org, and donated to MLCommons to benefit everyone. Since then, this open-source automation technology -(CM/CMX, CM4MLOps, CM4ABTF, CM4Research, etc) is being extended +(CM/CMX, CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, etc) is being extended as a community effort thanks to all our volunteers, collaborators and contributors listed here in alphabetical order: +* @Henryfzh +* @Leonard226 +* @Oseltamivir +* @Submandarine * Resmi Arjun * Omar Benjelloun (Google) * Alice Cheng (Nvidia) diff --git a/README.md b/README.md index 9959803e0b..e383508f96 100755 --- a/README.md +++ b/README.md @@ -11,30 +11,31 @@ ### About -[Collective Knowledge (CK)](https://cKnowledge.org) in an educational project -to help researchers and engineers automate their repetitive, tedious and time-consuming tasks -to build, run, benchmark and optimize AI, ML and other applications and systems -across diverse and continuously changing models, data, software and hardware. +[Collective Knowledge (CK, CM, CM4MLOps, CM4MLPerf and CMX)](https://cKnowledge.org) +is an educational community project to learn how to run AI, ML and other emerging workloads +in the most efficient and cost-effective way across diverse models, data sets, software and hardware. CK consists of several sub-projects: * [Collective Mind framework (CM)](cm) - a very lightweight Python-based framework with minimal dependencies - to help users implement, share and reuse cross-platform automation recipes to - build, benchmark and optimize applications on any platform - with any software and hardware. - - * [CM interface to run MLPerf inference benchmarks](https://docs.mlcommons.org/inference) + intended to help researchers and engineers automate their repetitive, tedious and time-consuming tasks + to build, run, benchmark and optimize AI, ML and other applications and systems + across diverse and continuously changing models, data, software and hardware. * [CM4MLOPS](https://github.com/mlcommons/cm4mlops) - a collection of portable, extensible and technology-agnostic automation recipes - with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications + with a common CLI and Python API (CM scripts) to unify and automate + all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications on diverse platforms with any software and hardware: see [online catalog at CK playground](https://access.cknowledge.org/playground/?action=scripts), [online MLCommons catalog](https://docs.mlcommons.org/cm4mlops/scripts) + * [CM interface to run MLPerf inference benchmarks](https://docs.mlcommons.org/inference) + * [CM4ABTF](https://github.com/mlcommons/cm4abtf) - a unified CM interface and automation recipes to run automotive benchmark across different models, data sets, software and hardware from different vendors. -* [CMX (the next generation of CM and CM4MLOps)](cm/docs/cmx) - we are developing the next generation of CM +* [CMX (the next generation of CM, CM4MLOps and CM4MLPerf)](cm/docs/cmx) - + we are developing the next generation of CM to make it simpler and more flexible based on user feedback. Please follow this project [here]( https://github.com/orgs/mlcommons/projects/46 ). @@ -44,6 +45,12 @@ CK consists of several sub-projects: and organize [public optimization challenges and reproducibility initiatives](https://access.cknowledge.org/playground/?action=challenges) to co-design more efficient and cost-effiective software and hardware for emerging workloads. + * [CM4MLPerf-results](https://github.com/mlcommons/cm4mlperf-results) - + a simplified and unified representation of the past MLPerf results + for further visualization and analysis using [CK graphs](https://access.cknowledge.org/playground/?action=experiments) + (*the new version is coming soon*). + + * [Artifact Evaluation](https://cTuning.org/ae) - automating artifact evaluation and reproducibility initiatives at ML and systems conferences. @@ -57,11 +64,15 @@ CK consists of several sub-projects: * Copyright (c) 2021-2024 MLCommons * Copyright (c) 2014-2021 cTuning foundation +### Author + +* [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning) + ### Maintainers -* CM/CM4Research: [Grigori Fursin](https://cKnowledge.org/gfursin) -* CM4MLOps: [Arjun Suresh](https://github.com/arjunsuresh) and [Anandhu Sooraj](https://github.com/anandhu-eng) -* CMX (the next generation of CM) [Grigori Fursin](https://cKnowledge.org/gfursin) +* [Collective Mind (CM)](cm): [Grigori Fursin](https://cKnowledge.org/gfursin) +* CM4MLOps (CM automation recipes): [Arjun Suresh](https://github.com/arjunsuresh) and [Anandhu Sooraj](https://github.com/anandhu-eng) +* CMX (the next generation of CM, CM4MLOps and CM4MLPerf): [Grigori Fursin](https://cKnowledge.org/gfursin) ### Citing our project @@ -73,19 +84,20 @@ To learn more about the motivation behind CK and CM technology, please explore t * "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] * ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] * ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] +* Journal of Royal Society'20: [ [paper](https://royalsocietypublishing.org/doi/10.1098/rsta.2020.0211) ] ### CM Documentation * [CM installation GUI](https://access.cknowledge.org/playground/?action=install) -* [CM Getting Started Guide and FAQ](docs/getting-started.md) - * [Common CM interface to run MLPerf inference benchmarks](docs/mlperf/inference) - * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](docs/tutorials/common-interface-to-reproduce-research-projects.md) +* [CM Getting Started Guide and FAQ](https://github.com/mlcommons/ck/tree/master/docs/getting-started.md) + * [Common CM interface to run MLPerf inference benchmarks](https://github.com/mlcommons/ck/tree/master/docs/mlperf/inference) + * [Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23](https://github.com/mlcommons/ck/tree/master/docs/tutorials/common-interface-to-reproduce-research-projects.md) * [CM automation recipes for MLOps and DevOps](https://access.cknowledge.org/playground/?action=scripts) - * [Other CM tutorials](docs/tutorials) -* [Full documentation](docs/README.md) -* [CM development tasks](docs/taskforce.md#current-tasks) -* [CM and CK history](docs/history.md) + * [Other CM tutorials](https://github.com/mlcommons/ck/tree/master/docs/tutorials) +* [Full documentation](https://github.com/mlcommons/ck/tree/master/docs/README.md) +* [CM taskforce](https://github.com/mlcommons/ck/tree/master/docs/taskforce.md) +* [CMX, CM and CK history](https://github.com/mlcommons/ck/tree/master/docs/history.md) ### Acknowledgments @@ -95,6 +107,8 @@ CM4Research and CMX) was created by [Grigori Fursin](https://cKnowledge.org/gfur and sponsored by cTuning.org, OctoAI and HiPEAC. Grigori donated CK to MLCommons to benefit the community and to advance its development as a collaborative, community-driven effort. -We thank MLCommons and FlexAI for supporting this project, + +We thank [MLCommons](https://mlcommons.org), [FlexAI](https://flex.ai) +and [cTuning](https://cTuning.org) for supporting this project, as well as our dedicated [volunteers and collaborators](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) for their feedback and contributions! diff --git a/cm/CHANGES.md b/cm/CHANGES.md index 492d9a92fd..d15293319a 100644 --- a/cm/CHANGES.md +++ b/cm/CHANGES.md @@ -1,3 +1,13 @@ +## V3.4.4 + - improved error reporting in utils.load_json and utils.load_yaml + - added utils.substitute_template + - enhanced 'cm pull repo' based on user feedback + +## V3.4.3 + - fixed pyproject.toml + - fixed docs generation + - improved cm pull repo with branch and SHA (#1355) + ## V3.4.2 - added utils.flatten_dict - added utils.safe_int diff --git a/cm/MANIFEST.in b/cm/MANIFEST.in new file mode 100644 index 0000000000..f08d39c547 --- /dev/null +++ b/cm/MANIFEST.in @@ -0,0 +1,3 @@ +include CHANGES.md +include CONTRIBUTING.md +include COPYRIGHT.txt diff --git a/cm/README.md b/cm/README.md index e9b96edd06..ac726975ae 100644 --- a/cm/README.md +++ b/cm/README.md @@ -9,18 +9,20 @@ ### About -Collective Mind (CM) is a small, modular, cross-platform and decentralized workflow automation framework -with a human-friendly interface to make it easier to build, run, benchmark and optimize applications -across diverse models, data sets, software and hardware. +Collective Mind (CM) is a small [Python package](https://pypi.org/project/cmind) +with a unified CLI and API designed for creating and managing +portable and technology-agnostic automations for MLOps, DevOps and ResearchOps. +It is intended to make it easier to build, run, benchmark and optimize applications +across diverse models, data sets, software and hardware. + CM is a part of [Collective Knowledge (CK)](https://github.com/mlcommons/ck) - -an educational community project to learn how to run emerging workloads +an educational community project to learn how to run AI, ML and other emerging workloads in the most efficient and cost-effective way across diverse -and continuously changing systems. +and continuously changing systems using the MLPerf benchmarking methodology. -CM includes a collection of portable, extensible and technology-agnostic automation recipes -with a common API and CLI (aka CM scripts) to unify and automate different steps -required to compose, run, benchmark and optimize complex ML/AI applications +CM includes a [collection of portable, extensible and technology-agnostic automation recipes](https://access.cknowledge.org/playground/?action=scripts) +(aka CM scripts) to unify and automate different steps required to compose, run, benchmark and optimize complex ML/AI applications on any platform with any software and hardware. CM scripts extend the concept of `cmake` with simple Python automations, native scripts @@ -44,9 +46,9 @@ from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors: and simple JSON/YAML descriptions instead of inventing new workflow languages; * must have the same interface to run all automations natively, in a cloud or inside containers. -### Maintainers +### Author and maintainer -* [Grigori Fursin](https://cKnowledge.org/gfursin) +* [Grigori Fursin](https://cKnowledge.org/gfursin) (FlexAI, cTuning) ### Resources @@ -83,6 +85,8 @@ developed by [Grigori Fursin](https://cKnowledge.org/gfursin), as a part of the [Collective Knowledge educational initiative](https://cKnowledge.org), sponsored by [cTuning.org](https://cTuning.org) and [cKnowledge.org](https://cKnowledge.org), and contributed to MLCommons for the benefit of all. + This open-source technology, including CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, and more, -is a collaborative community-driven project made possible by our -[amazing volunteers, collaborators, and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! +is a collaborative project supported by [MLCommons](https://mlcommons.org), +[FlexAI](https://flex.ai), [cTuning](https://cTuning.org) +and our [amazing volunteers, collaborators, and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! diff --git a/cm/cmind/__init__.py b/cm/cmind/__init__.py index d63eb4ef44..e996b79326 100644 --- a/cm/cmind/__init__.py +++ b/cm/cmind/__init__.py @@ -2,7 +2,7 @@ # # Written by Grigori Fursin -__version__ = "3.4.2" +__version__ = "3.4.4" from cmind.core import access from cmind.core import x diff --git a/cm/cmind/repo/automation/repo/module.py b/cm/cmind/repo/automation/repo/module.py index eae2345a98..2b7578eb1b 100644 --- a/cm/cmind/repo/automation/repo/module.py +++ b/cm/cmind/repo/automation/repo/module.py @@ -34,6 +34,7 @@ def pull(self, i): (new_branch) (str): Create new Git branch (checkout) (str): Git checkout (checkout_only) (bool): only checkout existing repo + (dir) (str): use repository in this directory (depth) (int): Git depth (desc) (str): brief repository description (1 line) (prefix) (str): extra directory to keep CM artifacts @@ -54,11 +55,11 @@ def pull(self, i): console = i.get('out') == 'con' - alias = i.get('artifact','') - url = i.get('url','') - desc = i.get('desc','') - prefix = i.get('prefix','') - pat = i.get('pat','') + alias = i.get('artifact', '') + url = i.get('url', '') + desc = i.get('desc', '') + prefix = i.get('prefix', '') + pat = i.get('pat', '') extra_cmd_git = i.get('extra_cmd_git', '') extra_cmd_pip = i.get('extra_cmd_pip', '') @@ -132,13 +133,14 @@ def pull(self, i): branch = i.get('branch', '') new_branch = i.get('new_branch', '') checkout = i.get('checkout', '') + _dir = i.get('dir', '') r = net.request({'get': {'action': 'check-migration-repo-notes', 'repo': url, 'branch': branch, 'checkout': checkout}}) notes = r.get('dict', {}).get('notes','') if notes !='': print (notes) - if alias == 'mlcommons@ck' and branch == '' and checkout == '': + if alias == 'mlcommons@ck' and branch == '' and checkout == '' and _dir == '': print ('=========================================================================') print ('Warning: mlcommons@ck was automatically changed to mlcommons@cm4mlops.') print ('If you want to use older mlcommons@ck repository, use branch or checkout.') @@ -153,6 +155,7 @@ def pull(self, i): 'branch': branch, 'new_branch': new_branch, 'checkout': checkout, + 'dir': _dir, 'depth': i.get('depth', '')}] @@ -173,6 +176,7 @@ def pull(self, i): checkout = repo.get('checkout','') depth = repo.get('depth','') path_to_repo = repo.get('path_to_repo', None) + _dir = repo.get('dir', '') if console: print (self.cmind.cfg['line']) @@ -185,6 +189,8 @@ def pull(self, i): print ('New branch: {}'.format(new_branch)) if checkout!='': print ('Checkout: {}'.format(checkout)) + if _dir!='': + print ('Directory: {}'.format(_dir)) if depth!='' and depth!=None: print ('Depth: {}'.format(str(depth))) print ('') @@ -197,6 +203,7 @@ def pull(self, i): branch = branch, new_branch = new_branch, checkout = checkout, + _dir = _dir, console = console, desc=desc, prefix=prefix, diff --git a/cm/cmind/repos.py b/cm/cmind/repos.py index e2d254a00d..e75fcf8299 100644 --- a/cm/cmind/repos.py +++ b/cm/cmind/repos.py @@ -99,6 +99,17 @@ def load(self, init = False): r = utils.save_json(full_path_to_repo_paths, meta = self.paths) if r['return']>0: return r + # Skip internal repos + skip_internal_repos = os.environ.get('CM_CORE_SKIP_INTERNAL_REPOS','').strip().lower() + if skip_internal_repos not in ['1', 'true', 'yes']: + import pkgutil + for mi, name, ispkg in pkgutil.iter_modules(): + if name.startswith('cm') and name != 'cmind': + path = os.path.join(mi.path, name, 'repo') + path_cmr = os.path.join(path, 'cmr.yaml') + if os.path.isfile(path_cmr) and path not in self.paths: + self.paths.insert(0, path) + # Check internal repo (will be after local) if self.path_to_internal_repo != '' and os.path.isdir(self.path_to_internal_repo): self.paths.insert(0, self.path_to_internal_repo) @@ -294,7 +305,7 @@ def process(self, repo_path, mode='add'): return rr ############################################################ - def pull(self, alias, url = '', branch = '', checkout = '', console = False, desc = '', prefix = '', depth = None, + def pull(self, alias, url = '', branch = '', checkout = '', _dir = '', console = False, desc = '', prefix = '', depth = None, path_to_repo = None, checkout_only = False, skip_zip_parent_dir = False, extra_cmd_git = '', extra_cmd_pip = '', new_branch = ''): """ @@ -307,6 +318,7 @@ def pull(self, alias, url = '', branch = '', checkout = '', console = False, des (new_branch) (str): Create new branch (checkout) (str): Git repository checkout (checkout_only) (bool): only checkout existing repo + (_dir) (str): use repository in this directory (depth) (int): Git repository depth (console) (bool): if True, print some info to console (desc) (str): optional repository description @@ -335,6 +347,9 @@ def pull(self, alias, url = '', branch = '', checkout = '', console = False, des if path_to_repo == None: path_to_repo = os.path.join(self.full_path_to_repos, alias) + if _dir != '': + path_to_repo = os.path.join(path_to_repo, _dir) + if console: print ('Local path: '+path_to_repo) print ('') diff --git a/cm/cmind/utils.py b/cm/cmind/utils.py index 23296c6e42..5ba5bbce1a 100644 --- a/cm/cmind/utils.py +++ b/cm/cmind/utils.py @@ -204,7 +204,7 @@ def load_json(file_name, check_if_exists = False, encoding='utf8'): try: meta = json.load(jf) except Exception as e: - return {'return':4, 'error': format(e)} + return {'return':4, 'error': f'detected problem in {file_name}: {e}'} return {'return':0, 'meta': meta} @@ -274,7 +274,7 @@ def load_yaml(file_name, check_if_exists = False, encoding = 'utf8'): # To support old versions meta = yaml.safe_load(yf) except Exception as e: - return {'return':4, 'error': format(e)} + return {'return':4, 'error': f'detected problem in {file_name}: {e}'} return {'return':0, 'meta': meta} @@ -2152,3 +2152,21 @@ def digits(s, first = True): pass return v + +############################################################################## +def substitute_template(template, variables): + """ + Substitutes variables in a template string with values from a dictionary. + + Args: + template (str): The template string with placeholders (e.g., "something-{var1}-something-{var2}"). + vars (dict): A dictionary containing variable-value pairs (e.g., {'var1': 'a', 'var2': 'b'}). + + Returns: + str: The template string with placeholders replaced by the corresponding values. + """ + try: + return template.format(**variables) + except KeyError as e: + return f"Error: Missing value for {e.args[0]} in the vars dictionary." + diff --git a/cm/pyproject.toml b/cm/pyproject.toml index 7159530d3f..f948324810 100644 --- a/cm/pyproject.toml +++ b/cm/pyproject.toml @@ -1,5 +1,3 @@ -# Developer(s): Grigori Fursin - [build-system] build-backend = "setuptools.build_meta" requires = ["setuptools"] @@ -62,6 +60,7 @@ version = {attr = "cmind.__version__"} [tool.setuptools] zip-safe = false +include-package-data = true [tool.setuptools.package-data] cmind = ["repo/*", diff --git a/cm4mlops/README.md b/cm4mlops/README.md index fd10f61166..afc3746840 100644 --- a/cm4mlops/README.md +++ b/cm4mlops/README.md @@ -1 +1,5 @@ -Development repository: https://github.com/mlcommons/cm4mlops +Development repositories: +* https://github.com/mlcommons/cm4mlops + +Deprecated: +* https://github.com/mlcommons/ck-mlops diff --git a/cm4mlperf/README.md b/cm4mlperf/README.md new file mode 100644 index 0000000000..15c19d819c --- /dev/null +++ b/cm4mlperf/README.md @@ -0,0 +1,4 @@ +Development repositories: +* https://github.com/mlcommons/cm4mlperf-results +* https://github.com/mlcommons/cm4mlperf-inference +* https://github.com/mlcommons/cm4mlops/tree/mlperf-inference diff --git a/cmx4mlops/README.md b/cmx4mlops/README.md new file mode 100644 index 0000000000..a0990367ef --- /dev/null +++ b/cmx4mlops/README.md @@ -0,0 +1 @@ +TBD diff --git a/cmx4mlops/cmr.yaml b/cmx4mlops/cmr.yaml new file mode 100644 index 0000000000..13493b9e90 --- /dev/null +++ b/cmx4mlops/cmr.yaml @@ -0,0 +1,12 @@ +alias: cmx4mlops +uid: 428611a6db02407f + +git: true + +version: "0.5.1" + +author: "Grigori Fursin" + +install_python_requirements: false + +min_cm_version: "3.4.4" diff --git a/docs/cm/README.md b/docs/cm/README.md new file mode 100644 index 0000000000..e9b96edd06 --- /dev/null +++ b/docs/cm/README.md @@ -0,0 +1,88 @@ +[![PyPI version](https://badge.fury.io/py/cmind.svg)](https://pepy.tech/project/cmind) +[![Python Version](https://img.shields.io/badge/python-3+-blue.svg)](https://github.com/mlcommons/ck/tree/master/cm/cmind) +[![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE.md) +[![Downloads](https://static.pepy.tech/badge/cmind)](https://pepy.tech/project/cmind) + +[![arXiv](https://img.shields.io/badge/arXiv-2406.16791-b31b1b.svg)](https://arxiv.org/abs/2406.16791) +[![CM test](https://github.com/mlcommons/ck/actions/workflows/test-cm.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm.yml) +[![CM script automation features test](https://github.com/mlcommons/ck/actions/workflows/test-cm-script-features.yml/badge.svg)](https://github.com/mlcommons/ck/actions/workflows/test-cm-script-features.yml) + +### About + +Collective Mind (CM) is a small, modular, cross-platform and decentralized workflow automation framework +with a human-friendly interface to make it easier to build, run, benchmark and optimize applications +across diverse models, data sets, software and hardware. + +CM is a part of [Collective Knowledge (CK)](https://github.com/mlcommons/ck) - +an educational community project to learn how to run emerging workloads +in the most efficient and cost-effective way across diverse +and continuously changing systems. + +CM includes a collection of portable, extensible and technology-agnostic automation recipes +with a common API and CLI (aka CM scripts) to unify and automate different steps +required to compose, run, benchmark and optimize complex ML/AI applications +on any platform with any software and hardware. + +CM scripts extend the concept of `cmake` with simple Python automations, native scripts +and JSON/YAML meta descriptions. They require Python 3.7+ with minimal dependencies and are +[continuously extended by the community and MLCommons members](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) +to run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux +and any other operating system, in a cloud or inside automatically generated containers +while keeping backward compatibility. + +CM scripts were originally developed based on the following requirements from the +[MLCommons members](https://mlcommons.org) +to help them automatically compose and optimize complex MLPerf benchmarks, applications and systems +across diverse and continuously changing models, data sets, software and hardware +from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors: +* must work out of the box with the default options and without the need to edit some paths, environment variables and configuration files; +* must be non-intrusive, easy to debug and must reuse existing + user scripts and automation tools (such as cmake, make, ML workflows, + python poetry and containers) rather than substituting them; +* must have a very simple and human-friendly command line with a Python API and minimal dependencies; +* must require minimal or zero learning curve by using plain Python, native scripts, environment variables + and simple JSON/YAML descriptions instead of inventing new workflow languages; +* must have the same interface to run all automations natively, in a cloud or inside containers. + +### Maintainers + +* [Grigori Fursin](https://cKnowledge.org/gfursin) + +### Resources + +* CM v2.x (2022-cur) (stable): [installation on Linux, Windows, MacOS](https://access.cknowledge.org/playground/?action=install) ; + [docs](https://docs.mlcommons.org/ck) ; [popular commands](https://github.com/mlcommons/ck/tree/master/cm/docs/demos/some-cm-commands.md) ; + [getting started guide](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md) +* CM v3.x aka CMX (2024-cur) (stable): [docs](https://github.com/orgs/mlcommons/projects/46) +* MLPerf inference benchmark automated via CM + * [Run MLPerf for submissions](https://docs.mlcommons.org/inference) + * [Run MLPerf at the Student Cluster Competition'24](https://docs.mlcommons.org/inference/benchmarks/text_to_image/reproducibility/scc24) +* Examples of modular containers and GitHub actions with CM commands: + * [GitHub action with CM commands to test MLPerf inference benchmark](https://github.com/mlcommons/inference/blob/master/.github/workflows/test-bert.yml) + * [Dockerfile to run MLPerf inference benchmark via CM](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/app-mlperf-inference/dockerfiles/bert-99.9/ubuntu_22.04_python_onnxruntime_cpu.Dockerfile) + +### License + +[Apache 2.0](LICENSE.md) + +### Citing CM project + +If you found CM automations useful, please cite this article: +[ [ArXiv](https://arxiv.org/abs/2406.16791) ], [ [BibTex](https://github.com/mlcommons/ck/blob/master/citation.bib) ]. + +You can learn more about the motivation behind these projects from the following presentations: + +* "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ [ArXiv](https://arxiv.org/abs/2406.16791) ] +* ACM REP'23 keynote about the MLCommons CM automation framework: [ [slides](https://doi.org/10.5281/zenodo.8105339) ] +* ACM TechTalk'21 about Collective Knowledge project: [ [YouTube](https://www.youtube.com/watch?v=7zpeIVwICa4) ] [ [slides](https://learning.acm.org/binaries/content/assets/leaning-center/webinar-slides/2021/grigorifursin_techtalk_slides.pdf) ] + +### Acknowledgments + +The Collective Mind (CM) automation framework was originally +developed by [Grigori Fursin](https://cKnowledge.org/gfursin), +as a part of the [Collective Knowledge educational initiative](https://cKnowledge.org), +sponsored by [cTuning.org](https://cTuning.org) and [cKnowledge.org](https://cKnowledge.org), +and contributed to MLCommons for the benefit of all. +This open-source technology, including CM4MLOps/CM4MLPerf, CM4ABTF, CM4Research, and more, +is a collaborative community-driven project made possible by our +[amazing volunteers, collaborators, and contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md)! diff --git a/docs/history.md b/docs/history.md index b92250b128..39b353d616 100644 --- a/docs/history.md +++ b/docs/history.md @@ -24,8 +24,11 @@ and systems (CM4MLOps scripts and workflows)](https://github.com/mlcommons/cm4ml and helped establish the [MLCommons Task Force on Automation and Reproducibility](taskforce.md) co-led with Arjun Suresh. -We continue extending CM to support different MLCommons projects to modularize and unify benchmarking -of ML/AI systems as a collaborative engineering effort based on [user feedback](../CONTRIBUTING.md). - You can learn more about the CM concept and motivation from the [keynote at ACM REP'23](https://doi.org/10.5281/zenodo.8105339) and this [white paper](https://arxiv.org/abs/2406.16791). + +# CMX (the next generation of CM and CM4MLOps) + +Grigori started prototyping CMX during summer 2024 to simplify CM interfaces based on user feedback. +It is available as a part of the standard [CM package at PYPI](https://pypi.org/project/cmind). +Please reach out to learn more about our plans. diff --git a/docs/index.md b/docs/index.md index a972b6da3f..32d46ee883 120000 --- a/docs/index.md +++ b/docs/index.md @@ -1 +1 @@ -../cm/README.md \ No newline at end of file +../README.md \ No newline at end of file diff --git a/docs/tests.md b/docs/tests.md new file mode 100644 index 0000000000..53de4fb893 --- /dev/null +++ b/docs/tests.md @@ -0,0 +1 @@ +# TESTS diff --git a/mkdocs.yml b/mkdocs.yml index 5fe16a7ee0..0b2dbe75bd 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,4 +1,4 @@ -site_name: Collective Mind Documentation +site_name: Collective Knowledge Documentation repo_url: https://github.com/mlcommons/ck theme: name: material @@ -19,13 +19,15 @@ theme: - navigation.top - toc.follow nav: - - Install: - - install/index.md - - Documentation: - - CM CLI: specs/cm-cli.md + - HOME: index.md + - CM: + - Summary: cm/README.md - Getting Started: getting-started.md - - Release Notes: - - What's New: news.md + - CM installation: install/index.md + - CM CLI: specs/cm-cli.md + - Releases: + - CM/CMX releases: https://github.com/mlcommons/ck/releases + - CK/CM/CMX news: news.md markdown_extensions: - pymdownx.tasklist: