Supplementary Material for Lectures

discord.gg/cudamode The PMPP Book: Programming Massively Parallel Processors: A Hands-on Approach (Amazon link) YouTube Channel

Lecture 1: Profiling and Integrating CUDA kernels in PyTorch

• Video
• Date: 2024-01-13, Speaker: Mark Saroufim
• Notebook and slides in lecture_001 folder

Lecture 2: Recap Ch. 1-3 from the PMPP book

• Video
• Date: 2024-01-20, Speaker: Andreas Koepf
• Slides: The powerpoint file lecture_002/cuda_mode_lecture2.pptx can be found in the root directory of this repository. Alternatively here as Google docs presentation.

Lecture 3: Getting Started With CUDA

• Video
• Date: 2024-01-27, Speaker: Jeremy Howard
• Notebook: See the lecture_003 folder, or run the Colab version

Lecture 4: Intro to Compute and Memory Architecture

• Video
• Date: 2024-02-03, Speaker: Thomas Viehmann
• Notebook and slides in the lecture_004 folder.

Lecture 5: Going Further with CUDA for Python Programmers

• Video
• Date: 2024-02-10, Speaker: Jeremy Howard
• Notebook in the lecture_005 folder.

Lecture 6: Optimizing PyTorch Optimizers

• Video
• Date: 2024-02-17, Speaker: Jane Xu
• Slides

Lecture 7: Advanced Quantization

• Video
• Date: 2024-02-25, Speaker: Charles Hernandez
• Slides

Lecture 8: CUDA Performance Checklist

• Video
• Date: 2024-03-09, Speaker: Mark Saroufim
• Code in the lecture_008 folder
• Slides

Lecture 9: Reductions

• Video
• Date: 2024-03-09, Speaker: Mark Saroufim
• Code in the lecture_009 folder
• Slides

Lecture 10: Build a Prod Ready CUDA Library

• Video
• Date: 2024-03-16, Speaker: Oscar Amoros Huguet
• slides

Lecture 11: Sparsity

• Video
• Date: 2024-03-23, Speaker: Jesse Cai
• Slides

Lecture 12: Flash Attention

• Video
• Date: 2024-03-30, Speaker: Thomas Viehmann

Lecture 13: Ring Attention

• Video
• Date: 2024-04-06, Speaker: Andreas Koepf
• Slides

Lecture 14: Practitioner's Guide to Triton

• Video
• Date: 2024-04-13, Speaker: Umer Adil
• Notebook

Lecture 15: CUTLASS

• Date: 2024-04-20, Speaker: Eric Auld

Lecture 16: On Hands profiling

• Date: 2024-04-27, Speaker: Taylor Robbie

Bonus Lecture: CUDA C++ llm.cpp

• Date: 2024-04-27, Speaker: Jake Hemstad & Georgii Evtushenko
• Slides

Lecture 17: GPU Collective Communication (NCCL)

• Date: 2024-05-04, Speaker: Dan Johnson
• Code in the lecture_017 folder

Lecture 18: Fused Kernels

• Date: 2024-05-11, Speaker: Kapil Sharma
• Code in the lecture_018 folder

Lecture 19: Data Processing on GPUs

• Date: 2024-05-18, Speaker: Devavret Makkar

Lecture 20: Scan Algorithm

• Date: 2024-05-25, Speaker: Izzat El Haj
• Slides

Lecture 21: Scan Algorithm Part 2

• Date: 2024-05-31, Speaker: Izzat El Haj
• Slides

Lecture 22: Hacker's Guide to Speculative Decoding in VLLM

• Date: 2024-06-01, Speaker: Cade Daniel
• Slides

Lecture 23: Tensor Cores

• Date: 2024-06-07, Speaker: Vijay Thakkar & Pradeep Ramani
• Slides

Lecture 24: Scan at the Speed of Light

• Date: 2024-06-08, Speaker: Jake Hemstad & Georgii Evtushenko
• Slides