diff --git a/workshop/00_home.md b/workshop/00_home.md new file mode 100644 index 0000000..98bf039 --- /dev/null +++ b/workshop/00_home.md @@ -0,0 +1,8 @@ +--- +title: "Welcome" +weight: 10 +--- + +# meta-aws-workshop + +Test123 Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean tincidunt eros non enim rhoncus dictum mollis at nulla. In quis nunc urna. Quisque id justo malesuada, fermentum libero quis, consectetur libero. Vivamus faucibus ex at turpis facilisis, et molestie felis vestibulum. Donec tellus velit, iaculis vitae ex vitae, cursus congue felis. Nullam interdum feugiat nisi eu mollis. Nullam nec arcu faucibus, tempor elit sit amet, tempor nibh. Mauris posuere risus vitae nibh placerat blandit. Quisque a dapibus arcu. \ No newline at end of file diff --git a/workshop/01_introduction.md b/workshop/01_introduction.md new file mode 100644 index 0000000..cd3fd65 --- /dev/null +++ b/workshop/01_introduction.md @@ -0,0 +1,8 @@ +--- +title: "Introduction" +weight: 10 +--- +In this section, you will learn how CI/CD fits into the iot product life cycle and the roles participating at each phase during the iot product life cycle. The introduction roughly mirrors the first module in the original training. The introduction has three sections: +- High level overview of the iot product life cycle visualized by a SIPOC diagram +- High level overview of role-based deliverables across the life cycle visualized by a lightweight flowchart categorized by roles +- A workshop roadmap that draws parallels between the iot product life cycle the each module in the workshop, visualized by an execution architecture. \ No newline at end of file diff --git a/workshop/02_setup.md b/workshop/02_setup.md new file mode 100644 index 0000000..aca0759 --- /dev/null +++ b/workshop/02_setup.md @@ -0,0 +1,23 @@ +--- +title: "Setup" +weight: 20 +--- +In this section, learners will invoke mechanisms to setup artifacts for the modules although some artifacts may be setup by the workshop harness. The section will walk the learner through all the created artifacts and tools so a baseline understanding of the toolchain can be set prior to progressing to Module 1 + +## Launch Workshop Resources with CloudFormation +Before starting the Embedded Linux workshop, you need to create the required AWS resources. To do this, we provide an AWS CloudFormation template to create a stack that contains the resources. When you create the stack, +AWS creates a number of resources in your account. + + + +Choose an AWS region from the below list where you want to launch your CloudFormation stack. It is recommended to choose the closest region. The required AWS resource for the workshop are provisioned with AWS CloudFormation. Simply click the AWS region where you want to launch your stack. + +By choosing one of the links below you will be automatically redirected to the CloudFormation section of the AWS Console where your stack will be launched. +* [Launch CloudFormation stack in eu-central-1](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/create/review?templateURL=https://aws-iot-workshop-artifacts.s3.amazonaws.com/4f74bcdb2e45dbf5/2021-07-29/cfn/cfn-iot-c9-v2-generic.json&stackName=EmbeddedLinux¶m_C9InstanceType=c5.9xlarge¶m_C9UserDataScript=c9-ub1804-embeddedlinux.sh¶m_C9ImageId=ubuntu-18.04-x86_64) (Frankfurt) +* [Launch CloudFormation stack in eu-west-1](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/create/review?templateURL=https://aws-iot-workshop-artifacts.s3.amazonaws.com/4f74bcdb2e45dbf5/2021-07-29/cfn/cfn-iot-c9-v2-generic.json&stackName=EmbeddedLinux¶m_C9InstanceType=c5.9xlarge¶m_C9UserDataScript=c9-ub1804-embeddedlinux.sh¶m_C9ImageId=ubuntu-18.04-x86_64) (Ireland) +* [Launch CloudFormation stack in us-east-1](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?templateURL=https://aws-iot-workshop-artifacts.s3.amazonaws.com/4f74bcdb2e45dbf5/2021-07-29/cfn/cfn-iot-c9-v2-generic.json&stackName=EmbeddedLinux¶m_C9InstanceType=c5.9xlarge¶m_C9UserDataScript=c9-ub1804-embeddedlinux.sh¶m_C9ImageId=ubuntu-18.04-x86_64) (N. Virginia) +* [Launch CloudFormation stack in us-west-2](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://aws-iot-workshop-artifacts.s3.amazonaws.com/4f74bcdb2e45dbf5/2021-07-29/cfn/cfn-iot-c9-v2-generic.json&stackName=EmbeddedLinux¶m_C9InstanceType=c5.9xlarge¶m_C9UserDataScript=c9-ub1804-embeddedlinux.sh¶m_C9ImageId=ubuntu-18.04-x86_64) (Oregon) +* [Launch CloudFormation stack in ap-southeast-1](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-1#/stacks/create/review?templateURL=https://aws-iot-workshop-artifacts.s3.amazonaws.com/4f74bcdb2e45dbf5/2021-07-29/cfn/cfn-iot-c9-v2-generic.json&stackName=EmbeddedLinux¶m_C9InstanceType=c5.9xlarge¶m_C9UserDataScript=c9-ub1804-embeddedlinux.sh¶m_C9ImageId=ubuntu-18.04-x86_64) (Singapore) +* [Launch CloudFormation stack in ap-southeast-2](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/create/review?templateURL=https://aws-iot-workshop-artifacts.s3.amazonaws.com/4f74bcdb2e45dbf5/2021-07-29/cfn/cfn-iot-c9-v2-generic.json&stackName=EmbeddedLinux¶m_C9InstanceType=c5.9xlarge¶m_C9UserDataScript=c9-ub1804-embeddedlinux.sh¶m_C9ImageId=ubuntu-18.04-x86_64) (Sydney) + + diff --git a/workshop/03_01_hello_yocto.md b/workshop/03_01_hello_yocto.md new file mode 100644 index 0000000..2f1b3b8 --- /dev/null +++ b/workshop/03_01_hello_yocto.md @@ -0,0 +1,169 @@ +--- +title: "1 – Hello Yocto! - Build your own Linux image" +weight: 10 +--- +Developing a project that use embedded devices requires engineering effort for selecting the right Operating System, building Board Support Package extensions and actually developing the application. In this module we tackle the first one. Choosing the right OS is a critical step in the path to production as it is, after all, the beating heart of your device: it needs to be secure, resilient, updateable, maintainable and may need to be compatible with different hardware architectures. + +The Yocto Project is an open source collaboration project that helps developers create custom Linux-based systems by providing flexible set of tools and a space where embedded developers can share technologies, software stack configurations, and best practices that can be used to create tailored Linux images based on your project needs. In this module, you’ll learn what the process is of creating an embedded OS image using the Yocto Project in the Cloud. + +### What you will learn in this module + +Learn how to setup a cloud development environment to make development easier and manually bake a ready-to-work image using a provided Yocto Recipe and Layers. + +### What you will need + +A Cloud9 Instance with 100GB available: the higher the number of vCPU available, the faster you'll be able to complete this module. (more info can be found here: https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#var-PARALLEL_MAKE ) + +The bitbake process is CPU-intensive and scales automatically with the number of vCPUs available. + +![Graph that shows the time it takes to bitbake the base image based on the Cloud9 CPU instance](/images/01_hello_yocto_bitbaketimes.png) + +We recommend at least a c5.9xlarge. + +### Step 1 - Prepare your development environment + +```bash +sudo apt update +sudo apt upgrade -y +``` + +Now install all the required packages: +```bash +sudo apt install gawk wget git-core \ + diffstat unzip texinfo gcc-multilib \ + chrpath socat cpio build-essential \ + python3 python3-pip python3-pexpect \ + xz-utils debianutils iputils-ping \ + python3-git python3-jinja2 libegl1-mesa \ + libsdl1.2-dev xterm pylint3 -y +``` + +Let's set up our work folder: +``` +mkdir -p $HOME/environment/src/mydev-proto +DEVHOME=$HOME/environment/src/mydev-proto +``` + +And clone Poky, Yocto's reference distribution that will help us build our own custom Linux Distribution. + +``` +git clone -b hardknott git://git.yoctoproject.org/poky $DEVHOME +cd $DEVHOME +``` + +Let's finish up by __sourcing__ the init script, while specifying __build__ as the build folder. + +``` +source ./oe-init-build-env build +``` + +### Step 2 - Bake the minimum image + +While we go through the rest of the module, let's start baking the minimum core image, let's run this command. +This might take some time (17m 53s on c5.9xlarge Cloud9 instance). + +``` +MACHINE=qemux86-64 \ + bitbake \ + core-image-minimal +``` + +{{% notice note %}} + If you receive an error like this: + ![](/images/01_hello_yocto_diskfull.png) + Increase your disk space following this guide: https://docs.aws.amazon.com/cloud9/latest/user-guide/move-environment.html + After you've resized the Cloud9's EBS from the AWS console or via CLI, if you are using Ubuntu, the main commands are: + `sudo growpart /dev/nvme0n1 1` and `sudo resize2fs /dev/nvme0n1p1` +{{% /notice %}} +While we wait, we can create a new shell and proceed to the next step + +### Step 3 - Integrate layers and your application layer + +Let's initialize the shell and download our layers: +``` +DEVHOME=$HOME/environment/src/mydev-proto +cd $DEVHOME +git clone -b hardknott git://git.openembedded.org/meta-openembedded +git clone -b hardknott https://git.yoctoproject.org/git/meta-virtualization +git clone -b hardknott https://github.com/aws4embeddedlinux/meta-aws +``` + +Then modify the `$DEVHOME/build/conf/bblayers.conf` file by adding the layers we downloaded previously to our new custom layer (substitute $DEVHOME with the $DEVHOME path, e.g. `home/ubuntu/environment/src/mydev-proto`) +``` +$DEVHOME/meta-openembedded/meta-oe +$DEVHOME/meta-openembedded/meta-python +$DEVHOME/meta-openembedded/meta-networking +$DEVHOME/meta-aws +``` + +It should look like this: + +``` +BBLAYERS ?= " \ + /home/ubuntu/environment/src/mydev-proto/meta \ + /home/ubuntu/environment/src/mydev-proto/meta-poky \ + /home/ubuntu/environment/src/mydev-proto/meta-yocto-bsp \ + /home/ubuntu/environment/src/mydev-proto/meta-openembedded/meta-oe \ + /home/ubuntu/environment/src/mydev-proto/meta-openembedded/meta-python \ + /home/ubuntu/environment/src/mydev-proto/meta-openembedded/meta-networking \ + /home/ubuntu/environment/src/mydev-proto/meta-aws \ + " +``` +This basically enables the layers in the build system. + +Before baking the image, let's add the aws-ioto-device-client to the image. +Let's modify `$DEVHOME/build/conf/local.conf` and add the following line at the end of the file. +``` +IMAGE_INSTALL_append = "aws-iot-device-client" +``` + +Wonder what this does? Check https://github.com/aws4embeddedlinux/meta-aws/blob/hardknott/recipes-iot/aws-iot-device-client/aws-iot-device-client_1.2.0.bb + +Now let's bake the image again. + +``` +MACHINE=qemux86-64 \ + bitbake \ + core-image-minimal +``` + +Notice how this time, it take less time because it only needs to bake the incremental layers we just added. + +### Step 4 - Test the image + + +``` +runqemu \ + qemux86-64 \ + core-image-minimal \ + ext4 \ + qemuparams="-m 2048" \ + nographic +``` + +provide user __root__ and test that the aws-device-client-sdk is installed by running the following command: +``` +/sbin/aws-iot-device-client --help +``` + +{{% notice note %}} + You can fix the name lookup by modifying the /etc/resolv.conf and adding your preferred nameservers (e.g. 1.1.1.1 and 1.0.0.1). + Wonder how to do it the "Yocto" way? Head over to: https://www.yoctoproject.org/docs/1.6/dev-manual/dev-manual.html#using-bbappend-files +{{% /notice %}} + + +If you want to exit the simulation, just run Ctrl+A and then press X + +### Checkpoint + +1. You have successfully logged onto the Cloud9 instance and set up the prerequisites +1. You have baked the image without any additional layer +1. You have modified the configuration to include the cloned layers +1. You have run the non graphical simulation of the firmware you just baked and ensured that the aws-iot-device-client sdk is present + +### Considerations +Whew, this is fine if you are a single developer and are not maintaining a plethora of architectures, branches and distributions. + +What if we had an automation that the bitbake process would kick-off everytime our team did a pull request/committed to the code repository and generate and archive the different layers to further speed up the bitbake times for every set of PCBs, Firmware versions, Architectures? + +Follow along in the next module to discover more! diff --git a/workshop/03_02_build_images.md b/workshop/03_02_build_images.md new file mode 100644 index 0000000..404219f --- /dev/null +++ b/workshop/03_02_build_images.md @@ -0,0 +1,120 @@ +--- +title: "2 – Build images in a fast and repeatable way" +weight: 10 +--- + +### Introduction + +In this module we use AWS CloudFormation, AWS CodePipelines and the Yocto project to define and deploy a solution that constructs a Linux image compatible with a Raspberry Pi 4. + +Here is a high-level diagram of how the solution works and the services used: + +![Solution Architecture](/images/02_build_images_solution_architecture.png) + +### Prerequisites +- An AWS Account +- A Dockerhub Account +- Experience building images using Yocto +- Familiarity AWS CloudFormation, the AWS CLI and shell scripts +- (Optional) A Raspberry Pi 4 and an SD card to test the produced image + +### Expected Environment +If you are continuing from Module 1, you can continue to use Cloud9. Otherwise you can use AWS CloudShell which is a browser-based shell that makes it easy to securely manage, explore, and interact with your AWS resources. CloudShell is pre-authenticated with your console credentials. Common development and operations tools are pre-installed, so no local installation or configuration is required. + + +### Step 1 - Setup your environment + +Open the AWS CloudShell service and run the following command to clone this repository and set `$PREFIX` to something unique like "mod2-YOUR_AWS_ACCOUNT_NUMBER". + +```bash +cd ~/ +git clone https://github.com/aws4embeddedlinux/meta-aws-ci +cd ~/meta-aws-ci/core/scripts/ + +export PREFIX=mod2-<> +``` + +### Step 2 – Securely store your Dockerhub credentials + +When building containers, you will need to setup a secret that contains your Dockerhub username and password in AWS Secrets Manager. This is used to authenticate the CodePipeline with Dockerhub and used when composing images. + +In AWS CloudShell, run this script without arguments and enter your Dockerhub username and password. It will create a Secrets Manager entry and return an ARN that you will use when doing setup for the container projects. + +```bash +./setup_dockerhub_secret.sh $PREFIX +``` +Once this process is complete, store the secret ARN in an environment variable for later use. + +```bash +export SECRET_ARN=arn:aws:secretsmanager:eu-west-1:123456789123:secret:dockerhub_EXAMPLE +``` + +### Step 3 – Create the baseline components +Baseline components are required for all other automation areas. + +In AWS CloudShell, run the script to create the network layer. The network layer is a Virtual Private Cloud (VPC) for AWS CodeBuild. + +```bash +./setup_ci_network.sh $PREFIX +``` + +### Step 4 – Install the container build layer and invoke the build process + +In AWS CloudShell, run the script to create the container build layer. This script installs an AWS CodeBuild project to construct a custom container that is used to build Linux compatible images for the reference distribution named ‘Poky’. + +```bash +./setup_ci_container_poky.sh $PREFIX $SECRET_ARN +``` + +Once this process is complete, invoke the build process. The process takes about 15 minutes to complete. You can monitor it using the CLI or by logging into the [AWS CodeBuild console](https://console.aws.amazon.com/codesuite/codebuild/projects). Make sure you select the right region. + + +```bash +aws codebuild start-build --project-name $PREFIX-el-ci-container-poky +``` + +Finally, find out the image URI and store it in an environment variable for later use. + +```bash +aws ecr describe-repositories --query repositories[].repositoryUri --output text +export CONTAINER_URI=123456789123.dkr.ecr.eu-west-1.amazonaws.com/yoctoproject/EXAMPLE/buildmachine-poky +``` + +### Step 5 – Install the Linux build layer and invoke the build process + +In AWS CloudShell, run the script to create the Linux build layer. This script installs an AWS CodeBuild project to construct the core-image-minimal image for the QEMU x86-64 MACHINE target that includes the AWS IoT Device Client. The AWS CodeBuild project file for this project is in the [meta-aws-demos](https://github.com/aws-samples/meta-aws-demos) repository. It also creates a new S3 bucket to store images it creates. + +```bash +export VENDOR=rpi_foundation +export BOARD=rpi4-64 +export DEMO=aws-iot-greengrass-v2 +export YOCTO_RELEASE=dunfell +export COMPUTE_TYPE=BUILD_GENERAL1_LARGE +./setup_build_demos_prod.sh $PREFIX $CONTAINER_URI $VENDOR $BOARD $DEMO $YOCTO_RELEASE $COMPUTE_TYPE +``` +Once the process is complete, find out the name of the newly created S3 bucket and store in an environment variable for later use + +```bash +aws s3 ls | grep $PREFIX-el-build- | awk '{print $3}' +export S3_BUCKET=EXAMPLE-el-build-rpi4-64-aws-iot-gre-buildbucket-EXAMPLE +``` + +Invoke the build process. The process takes about 90 minutes to complete using `BUILD_GENERAL1_LARGE`. You can monitor it using the CLI or by logging into the [AWS CodeBuild console](https://console.aws.amazon.com/codesuite/codebuild/projects). Make sure you select the right region. + +```bash +aws codebuild start-build --project-name $PREFIX-el-build-$BOARD-$DEMO-$YOCTO_RELEASE +``` +Once the build process is complete you can review the contents of the S3 bucket + +```bash +aws s3 ls $S3_BUCKET --recursive +``` + +### Step 6 (Optional) - Download the image from S3 and test it + +Download the image using the CLI or the AWS console and then use your favorite software to write the downloaded image to the SD card. Make sure you choose the right device. This process will overwrite the card. + +```bash +dd if=image.bin of=/dev/ bs=4M status=progress +``` + diff --git a/workshop/03_03_dev_experience.md b/workshop/03_03_dev_experience.md new file mode 100644 index 0000000..3cfabf8 --- /dev/null +++ b/workshop/03_03_dev_experience.md @@ -0,0 +1,5 @@ +--- +title: "3 - How to integrate the dev experience" +weight: 10 +--- +Delivering an integrated developer experience is key to adopt a sustainable CI/CD practice. \ No newline at end of file diff --git a/workshop/03_04_auto_testing.md b/workshop/03_04_auto_testing.md new file mode 100644 index 0000000..9ba2611 --- /dev/null +++ b/workshop/03_04_auto_testing.md @@ -0,0 +1,6 @@ +--- +title: "4 – Automate build and test processes" +weight: 10 +--- +In this module participants will learn about various methods for device on-boarding at scale and device fleet provisioning/fleet management. Needless to say that an IoT device that don’t have a strong root-of-trust and robust update capabilities will be eventually compromised or become dysfunctional. +While AWS IoT Greengrass allows for reliable management and orchestration of components, {TBC} \ No newline at end of file diff --git a/workshop/03_05_remote_update.md b/workshop/03_05_remote_update.md new file mode 100644 index 0000000..59284f7 --- /dev/null +++ b/workshop/03_05_remote_update.md @@ -0,0 +1,13 @@ +--- +title: "5 – Remotely update devices" +weight: 10 +--- + +In this module participants will learn about different update options for updating a device +- File level vs. Block level +- As well as different options for tooling to achieve device updates +-- block level options: Mender, swupdate +-- file level options: OSTree, swupd +- We will introduce the supporting AWS IoT functions that can be used to trigger and process to update jobs, e.g. CodeBuild, IoT Jobs, CloudFront +- During the hands-on section the particpants will use iot-jobs and meta-swupdate to update a device/image + diff --git a/workshop/03_modules.md b/workshop/03_modules.md new file mode 100644 index 0000000..ca9f697 --- /dev/null +++ b/workshop/03_modules.md @@ -0,0 +1,5 @@ +--- +title: "Modules" +weight: 30 +--- +TBD \ No newline at end of file diff --git a/workshop/04_conclusion.md b/workshop/04_conclusion.md new file mode 100644 index 0000000..c8e099c --- /dev/null +++ b/workshop/04_conclusion.md @@ -0,0 +1,5 @@ +--- +title: "Conclusion" +weight: 40 +--- +TBD \ No newline at end of file diff --git a/workshop/static/images/01_hello_yocto_bitbaketimes.png b/workshop/static/images/01_hello_yocto_bitbaketimes.png new file mode 100644 index 0000000..3ec3e56 Binary files /dev/null and b/workshop/static/images/01_hello_yocto_bitbaketimes.png differ diff --git a/workshop/static/images/01_hello_yocto_diskfull.png b/workshop/static/images/01_hello_yocto_diskfull.png new file mode 100644 index 0000000..86d5293 Binary files /dev/null and b/workshop/static/images/01_hello_yocto_diskfull.png differ diff --git a/workshop/static/images/02_build_images_solution_architecture.png b/workshop/static/images/02_build_images_solution_architecture.png new file mode 100644 index 0000000..379a88b Binary files /dev/null and b/workshop/static/images/02_build_images_solution_architecture.png differ