diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..4a39a71 --- /dev/null +++ b/.gitignore @@ -0,0 +1,103 @@ +.idea/ +venv/ +dictionary.dic +tmp/ +*.drawio.bkp + +# Created by https://www.toptal.com/developers/gitignore/api/visualstudiocode,macos,windows,linux +# Edit at https://www.toptal.com/developers/gitignore?templates=visualstudiocode,macos,windows,linux + +### Linux ### +*~ + +# temporary files which can be created if a process still has a handle open of a deleted file +.fuse_hidden* + +# KDE directory preferences +.directory + +# Linux trash folder which might appear on any partition or disk +.Trash-* + +# .nfs files are created when an open file is removed but is still being accessed +.nfs* + +### macOS ### +# General +.DS_Store +.AppleDouble +.LSOverride + +# Icon must end with two \r +Icon + + +# Thumbnails +._* + +# Files that might appear in the root of a volume +.DocumentRevisions-V100 +.fseventsd +.Spotlight-V100 +.TemporaryItems +.Trashes +.VolumeIcon.icns +.com.apple.timemachine.donotpresent + +# Directories potentially created on remote AFP share +.AppleDB +.AppleDesktop +Network Trash Folder +Temporary Items +.apdisk + +### macOS Patch ### +# iCloud generated files +*.icloud + +### VisualStudioCode ### +.vscode/* +!.vscode/settings.json +!.vscode/tasks.json +!.vscode/launch.json +!.vscode/extensions.json +!.vscode/*.code-snippets + +# Local History for Visual Studio Code +.history/ + +# Built Visual Studio Code Extensions +*.vsix + +### VisualStudioCode Patch ### +# Ignore all local history of files +.history +.ionide + +### Windows ### +# Windows thumbnail cache files +Thumbs.db +Thumbs.db:encryptable +ehthumbs.db +ehthumbs_vista.db + +# Dump file +*.stackdump + +# Folder config file +[Dd]esktop.ini + +# Recycle Bin used on file shares +$RECYCLE.BIN/ + +# Windows Installer files +*.cab +*.msi +*.msix +*.msm +*.msp + +# Windows shortcuts +*.lnk + +# End of https://www.toptal.com/developers/gitignore/api/visualstudiocode,macos,windows,linux diff --git a/content/modules/ROOT/nav.adoc b/content/modules/ROOT/nav.adoc index 2abdfbb..17feee8 100644 --- a/content/modules/ROOT/nav.adoc +++ b/content/modules/ROOT/nav.adoc @@ -1,4 +1,4 @@ -* 1. xref:01_welcome.adoc[Welcome!] +* 1. xref:01_welcome.adoc[Welcome and Introduction] * 2. xref:05_environment_provisioning.adoc[Environment Provisioning] @@ -12,7 +12,7 @@ * 7. xref:32_model_training_car.adoc[Model Training] -* 8. xref:34_boto3.adoc[Boto3] +* 8. xref:34_using_s3_storage.adoc[Using S3 Storage] * 9. xref:36_deploy_model.adoc[Deploy Model] diff --git a/content/modules/ROOT/pages/01_welcome.adoc b/content/modules/ROOT/pages/01_welcome.adoc index 7f6180a..fcd70ac 100644 --- a/content/modules/ROOT/pages/01_welcome.adoc +++ b/content/modules/ROOT/pages/01_welcome.adoc @@ -33,4 +33,13 @@ It's possible to install all the various components by hand, making notes such a The source code for the AI Accelerator can be found at: https://github.com/redhat-ai-services/ai-accelerator -The accelerator was created and is currently maintained by the Red Hat AI Services organization, however contributions from anywhere are always greatly appreciated! \ No newline at end of file +The accelerator was created and is currently maintained by the Red Hat AI Services organization, however contributions from anywhere are always greatly appreciated! + +## Questions for Further Consideration + +Additional questions that could be discussed for this topic: + +. When would I want to manually install RHOAI and associated components, instead of using the AI Accelerator framework? +. Who maintains the AI Accelerator framework project? +. What if I find a bug, or have a new feature to contribute? +. Can I add my own additional components into the AI Accelerator? \ No newline at end of file diff --git a/content/modules/ROOT/pages/05_environment_provisioning.adoc b/content/modules/ROOT/pages/05_environment_provisioning.adoc index 309fd3d..4250457 100644 --- a/content/modules/ROOT/pages/05_environment_provisioning.adoc +++ b/content/modules/ROOT/pages/05_environment_provisioning.adoc @@ -12,7 +12,7 @@ In this section we will order all three clusters at demo.redhat.com. Note that t The demo clusters typically take 1-2 hours to provision, although they may take a little longer on Monday mornings as they are using AWS spot instances and demand is usually high at the start of the work week. So it's suggested to provision them, and continue with the subsequent sections that don't require cluster access yet. -## Provision a Demo Cluster +## Provision the Demo Cluster The first cluster where we will run our demonstration projects requires a little more resources that the other two, so lets provision this first. @@ -36,13 +36,25 @@ image::clustersettings_Dev_Prod.png[width=50%] ## While You Wait -The provisioning process will take a while to complete, so why not take some time to check out some of the documentation in the AI Accelerator project that we will be installing once the clusters are ready: +The provisioning process will take a while to complete, so why not take some time to check out some of the documentation in the AI Accelerator project that we will be bootstrapping, once the new clusters are ready: * https://github.com/redhat-ai-services/ai-accelerator[Project Introduction README] -* https://github.com/redhat-ai-services/ai-accelerator/blob/main/documentation/overview.md[AI Accelerator Overview]. -* https://github.com/redhat-ai-services/ai-accelerator/blob/main/documentation/installation.md[AI Accelerator Installation Procedure]. -* https://github.com/redhat-ai-services/ai-accelerator/tree/main/tenants[Tenants documentation]. +* https://github.com/redhat-ai-services/ai-accelerator/blob/main/documentation/overview.md[AI Accelerator Overview] +* https://github.com/redhat-ai-services/ai-accelerator/blob/main/documentation/installation.md[AI Accelerator Installation Procedure] +* https://github.com/redhat-ai-services/ai-accelerator/tree/main/tenants[Tenants documentation] ## When the Cluster is Ready -Once the clusters have been provisioned, you should receive an email containing the cluster URLs as well as an administrative user (such as `kubeadmin`) and password. You can also obtain these from the status dashboard at https://demo.redhat.com[demo.redhat.com] as well as perform administrative functions on your clusters, such as starting/stopping or extending the lifespan if desired. \ No newline at end of file +Once the clusters have been provisioned, you should receive an email containing the cluster URLs as well as an administrative user (such as `kubeadmin`) and password. + +You can also obtain these URLs and credentials from your services dashboard at https://demo.redhat.com/[demo.redhat.com]. The dashboard also allows you to perform administrative functions on your clusters, such as starting/stopping or extending the lifespan if desired. + +## Questions for Further Consideration + +Additional questions that could be discussed for this topic: + +. How long can we use the demo.redhat.com OpenShift cluster? When will it get deleted? +. I want to install a demonstration cluster that might last several months for a RHOAI evaluation period. What options are available? +. Can we use our own AWS based OpenShift cluster, other than one from demo.redhat.com? +. Could I install this on my own hardware, such as my desktop PC that is running a single node OpenShift cluster? +. The topic of being able to easily repeat an installation, as discussed in the following GitOps sections may be interesting to discuss, since this means that work done to configure an environment is not lost if the environment is destroyed. \ No newline at end of file diff --git a/content/modules/ROOT/pages/07_installation.adoc b/content/modules/ROOT/pages/07_installation.adoc index c0fd73a..7bc3b90 100644 --- a/content/modules/ROOT/pages/07_installation.adoc +++ b/content/modules/ROOT/pages/07_installation.adoc @@ -21,7 +21,7 @@ Clone (download) the Git repository containing the AI Accelerator, since we will TIP: If you can't or prefer not to run the installation from your local machine (such as in a locked down corporate environment), you can also use the Bastion host instead. This is a Linux virtual machine running on AWS, the SSH login details are provided in the provisioning email you received from demo.redhat.com. Just be aware that the Basion host is deprovisioned when the cluster is deleted, so be sure to git commit any changes frequently. -[start=2] +[start=4] . Git clone the following repository to your local machine. If you're using a fork, then change the repository URL in the command below to match yours: [.console-input] [source,adoc] @@ -33,7 +33,7 @@ git clone https://github.com/redhat-ai-services/ai-accelerator.git Carefully follow the instructions found in https://github.com/redhat-ai-services/ai-accelerator/blob/main/documentation/installation.md[`documentation/installation.md`], with the following specifics: -[start=3] +[start=5] . Use the _**Demo**_ cluster credentials when logging into OpenShift . Select number 3 when prompted: [.bordershadow] @@ -44,7 +44,7 @@ This will install all the applications in the bootstrap script and also provide [.bordershadow] image::Bootstrap_argo_url.png[] -[start=4] +[start=7] . Log into the Argo CD link with the Openshift credentials and wait till everything syncs successfully. [.bordershadow] image::Argo_home_screen.png[] @@ -70,4 +70,15 @@ oc delete deployment granite-predictor-00001-deployment -n ai-example-single-mod We will cover the ai-accelerator project overview in a later section. --- -Continue using the _**DEMO**_ cluster for the subsequent exercises. \ No newline at end of file +Continue using the _**DEMO**_ cluster for the subsequent exercises. + +## Questions for Further Consideration + +Additional questions that could be discussed for this topic: + +. What's the difference between "bootstrapping" and "installing" the new OpenShift cluster? +. Why is forking an open source project a good idea? +. How can a project fork be used to contribute back to the parent project with bug fixes, updates and new features? +. Could, or should the bootstrap shell script be converted to Ansible? +. How does the bootstrap script provision GPU resources in the new OpenShift cluster? Hint: a quick walk through the logic in the source code should be a useful exercise, time permitting. +. Where can I get help if the bootstrap process breaks? diff --git a/content/modules/ROOT/pages/20_ai-accelerator_review.adoc b/content/modules/ROOT/pages/20_ai-accelerator_review.adoc index e00024e..cf509a2 100644 --- a/content/modules/ROOT/pages/20_ai-accelerator_review.adoc +++ b/content/modules/ROOT/pages/20_ai-accelerator_review.adoc @@ -1,10 +1,10 @@ -# AI-Accelerator Project +# AI Accelerator Project -The AI-accelerator project is an automated way to deploy RHOAI, relevent operators, and examples to a cluster. +The AI Accelerator project is an automated way to deploy RHOAI, relevent operators, and examples to a cluster. Navigate to the ai-accelerator github project: https://github.com/redhat-ai-services/ai-accelerator -5. The ai-accelerator project installs Open Shift GitOps (ArgoCD) first, then uses ArgoCD to install RHOAI and the related operators. It also deploys examples such as inference services, a workbench, and pipelines. +The ai-accelerator project installs Open Shift GitOps (ArgoCD) first, then uses ArgoCD to install RHOAI and the related operators. It also deploys examples such as inference services, a workbench, and pipelines. * The list of operators it can install are in the https://github.com/redhat-ai-services/ai-accelerator/tree/main/components/operators[operators] folder. * Examples of model serving, workbench, and pipelines can be found in the https://github.com/redhat-ai-services/ai-accelerator/tree/main/tenants[tenants] folder. @@ -16,6 +16,7 @@ This project is set up with ArgoCD and Kustomize in mind. Meaning ArgoCD will ha If you are unfamiliar with Kustomize, this is a very good tutorial: https://devopscube.com/kustomize-tutorial/[Learn more about Kustomize]. ### Overview of Kustomize in AI-Accelerator Project + Let's try to understand how Kustomize is being used to deploy the different resources in the ai-accelerator. 1. When running the _**bootstrap.sh**_ script, it will apply the Open Shift GitOps operator by using Kustomize on the https://github.com/redhat-ai-services/ai-accelerator/blob/b90f025691e14d8e8a8d5ff3452107f8a0c8f48d/scripts/bootstrap.sh#L11[GitOps_Overlay] https://github.com/redhat-ai-services/ai-accelerator/tree/b90f025691e14d8e8a8d5ff3452107f8a0c8f48d/components/operators/openshift-gitops/operator/overlays/latest[folder]: @@ -120,3 +121,17 @@ If you are using a disconnected environment, you will need to first setup: - the registry for the images - the git repositories, proxies and credentials ==== + +## References + +* Red Hat Blog: https://www.redhat.com/en/blog/your-guide-to-continuous-delivery-with-openshift-gitops-and-kustomize[Your Guide to Continuous Delivery with OpenShift GitOps and Kustomize] - a good article explaining more GitOps concepts +* GitHub: https://github.com/gnunn-gitops/standards/blob/master/folders.md[GitOps Folder Structure] - the original inspiration for the folder structure in the AI Accelerator project +* Red Hat Blog: https://www.redhat.com/en/blog/enterprise-mlops-reference-design[Enterprise MLOps Reference Design] - a conceptual reference design for performing Machine Learning Operations (MLOps) +* Topic: https://www.redhat.com/en/topics/devops/what-is-gitops[What is GitOps?] - 7-minute read on the topic of GitOps + +## Questions for Further Consideration + +Additional questions that could be discussed for this topic: + +. Where can I find a list of curated components that follow the GitOps pattern? Hint, see the https://github.com/redhat-cop/gitops-catalog[GitOps Catalog] GitHub page. +. Wow this project structure is complicated! Is there a way to simplify the project folder structures? Hint, a good discussion could be had on where we came from and how we got here in terms of project design and layout. diff --git a/content/modules/ROOT/pages/30_gitops_env_setup_dev_prod.adoc b/content/modules/ROOT/pages/30_gitops_env_setup_dev_prod.adoc index cf3c846..d111f0d 100644 --- a/content/modules/ROOT/pages/30_gitops_env_setup_dev_prod.adoc +++ b/content/modules/ROOT/pages/30_gitops_env_setup_dev_prod.adoc @@ -1,6 +1,7 @@ # Environment Install and Setup: DEV and PROD Cluster ## Parasol-insurance-dev cluster + Follow the following steps to complete the install and setup: * After the cluster is running and ready, log in as the admin @@ -313,3 +314,11 @@ When running the bootstrap script, select `bootstrap/overlays/parasol-insurance- ==== To check your work please refer to https://github.com/redhat-ai-services/ai-accelerator-qa/tree/30_gitops_env_setup_prod[This Prod Branch] ==== + +## Questions for Further Consideration + +Additional questions that could be discussed for this topic: + +. How familiar are your development teams with CI/CD concepts? +. How do you currently deploy project to development, QA and Production environments? +. Is ArgoCD new to the team? \ No newline at end of file diff --git a/content/modules/ROOT/pages/31_custom_notebook.adoc b/content/modules/ROOT/pages/31_custom_notebook.adoc index 6edeccd..65eeb45 100644 --- a/content/modules/ROOT/pages/31_custom_notebook.adoc +++ b/content/modules/ROOT/pages/31_custom_notebook.adoc @@ -304,3 +304,14 @@ image::01_custom_workbench.png[Custom workbench] ==== Verify your work against https://github.com/redhat-ai-services/ai-accelerator-qa/pull/new/31_custom_notebook:[This custom-workbench branch] ==== + +## Questions for Further Consideration + +Additional questions that could be discussed for this topic: + +. How many Python packages are included in your typical data scientist development environment? Are there any packages that are unique to your team? +. How do you handle continuous updates in your development environment, remembering that AI/ML is an evolving landscape, and new packages are released all the time, and existing packages are undergoing very frequent updates? +. Can data scientists ask for new packages in a securely controlled development environment? +. Where do you store source code for model experimentation and training? +. Do you think that cluster storage (such as an OpenShift PVC) is a good permanent location for source code, so that in the event of failure the source is not lost? +. How do your teams of data scientists collaborate on notebooks when training models or performing other experiments? diff --git a/content/modules/ROOT/pages/32_model_training_car.adoc b/content/modules/ROOT/pages/32_model_training_car.adoc index c14c219..daa8131 100644 --- a/content/modules/ROOT/pages/32_model_training_car.adoc +++ b/content/modules/ROOT/pages/32_model_training_car.adoc @@ -1,8 +1,8 @@ # Model Training with Custom Notebook -## In this module you will sync a git repo and run through a the parasol-insurnace Jupyter notebooks. +In this module you will sync a git repo, and then execute the logic contained in the parasol-insurance Jupyter notebooks. -We will use the custom image we created and uploaded before and we will create a workbench with the customer image we uploaded in module 04. +To perform this task, we will use the custom data science notebook image we created and uploaded in the previous module. ## Steps to create workbench with a custom notebook diff --git a/content/modules/ROOT/pages/34_boto3.adoc b/content/modules/ROOT/pages/34_using_s3_storage.adoc similarity index 82% rename from content/modules/ROOT/pages/34_boto3.adoc rename to content/modules/ROOT/pages/34_using_s3_storage.adoc index dfdb8e4..1357137 100644 --- a/content/modules/ROOT/pages/34_boto3.adoc +++ b/content/modules/ROOT/pages/34_using_s3_storage.adoc @@ -1,13 +1,26 @@ -# Boto3 exploration +# Using S3 Storage -In this module you will use boto3 to explore the existing minio configuration and set up scripts to automate uploading the models to the locations you need for your pipelines to run. +In this module you will set up some S3 based storage in the OpenShift cluster, and the utilize the Python package called https://pypi.org/project/boto3/[Boto3] to explore the existing S3 based storage configuration. -## Minio S3 Storage -Let's add Minio s3 storage to our dev environment project so Argo can deploy it. +We will also set up scripts to automate uploading the ML models to the locations required by the pipelines in subsequent modules. -If you need a reference, the ai-accelerator project has minio set up in under the `tenants/ai-examples folder`. +## What is S3 Storage? -### Set up minio +Amazon Simple Storage Service (S3) is a service offered by Amazon Web Services (AWS) that provides object storage through a web service interface. + +This lab uses https://github.com/minio/minio[MinIO] to implement S3 storage, which is High Performance Object Storage released under GNU Affero General Public License v3.0. MinIO is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads. + +For more information about S3: +* AWS documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html[What is Amazon S3?] +* Wikipedia: https://en.wikipedia.org/wiki/Amazon_S3[Amazon S3] + +## Adding MinIO to the Cluster + +In this section we will add MinIO S3 storage operator to our dev OpenShift cluster configuration project so Argo can deploy it. + +If you need a reference, the ai-accelerator project has MinIO set up in under the `tenants/ai-examples folder`. + +### Set up MinIO . Create a `object-datastore.yaml` file in the `tenants/parasol-insurance/namespaces/base` directory with the following content: @@ -85,7 +98,8 @@ resources: The same content will work for both overlays (dev and prod) ==== -Commit your changes to your fork of ai-accelerator project. Let ArgoCD sync and deploy minio. +Commit your changes to your fork of ai-accelerator project. Wait for ArgoCD to sync and deploy MinIO. + You should find your minio resource in the _**object-datastore**_ namespace. The *minio-ui* route can be found in _**object-datastore**_ namespace under _**Routes**_. Open this in a new tab and log in with `minio:minio123`. @@ -97,9 +111,9 @@ Compare your change to this in git https://github.com/redhat-ai-services/ai-acce Explore the S3 storage. -## Set up an standard data science workbench to explore S3 with boto3 +## Explore S3 with Boto3 -We have previously used a custom workbench to explore how to train a model. Now we will use a standard workbench to explore the S3 storage. +We have previously used a custom workbench to explore how to train a model. Now we will use a standard workbench to explore the S3 storage using the Boto3 Python package. ### Create a standard workbench @@ -293,7 +307,7 @@ resources: - ../../base ---- -. Push the changes to git, and wait for the synchrnization to complete. +. Push the changes to git, and wait for the synchronization to complete. + [TIP] @@ -307,8 +321,11 @@ Validate against https://github.com/redhat-ai-services/ai-accelerator-qa/pull/n [.bordershadow] image::standard-workbench.png[Standard workbench] -## Explore S3 in RHOAI Workbench: -Some S3 technologies do not come with UI or CLI to interact with the buckets or files. A common tool that can be used accross all S3 technologies is boto3. Boto3 is the AWS SDK for Python. It allows you to directly interact with AWS services such as S3, EC2, and more. +## Explore S3 in RHOAI Workbench + +https://pypi.org/project/boto3/[Boto3] is a commonly used Python package, which is the AWS SDK for communicating with S3 storage providers. It allows you to directly interact with AWS services such as S3, EC2, and more. + +Lets create some Python code in a Jupyter notebook to interact with our S3 storage: . Go to RHOAI Dashboard and go to the _**parasol-insurance**_ Data Science Project. @@ -324,11 +341,11 @@ image::standard-workbench.png[Standard workbench] [.bordershadow] image::Workbench_env_vars.png[] -. Launch the workbench and wait for the Jupyter notebook to spin up. +. Launch the workbench and wait for the Jupyter notebook to start up. . Create a new Notebook. -. In a new cell, add and run the content below to install boto3 and ultralytics. +. In a new cell, add and run the content below to install the `boto3` and `ultralytics` packages using pip. + [source, python] @@ -336,7 +353,7 @@ image::Workbench_env_vars.png[] !pip install boto3 ultralytics ---- -. Configure the connection to minio S3 +. Configure the connection to MinIO S3 + [source, python] @@ -449,3 +466,10 @@ def get_minio_content(bucket): get_minio_content('models') ---- +## Questions for Further Consideration + +Additional questions that could be discussed for this topic: + +* What other tools exist for interacting with S3? Hint, https://s3tools.org/s3cmd[s3cmd] is another quite popular S3 CLI tool. +* Could a shortcut to the MinIO Console be added to OpenShift? Hint, see the OpenShift `ConsoleLink` API, https://github.com/redhat-na-ssa/demo-lab-config/blob/main/demo/run-mlflow/link-minio.yaml[here's an example]. +* What's the maximum size of an object, such as a ML model that can be stored in S3? \ No newline at end of file