+
+ Machine learning is not a separate industry, instead, it's a powerful way of thinking about data. We aim to educate and enable the community to responsibly design, develop, deploy and iterate on production ML applications.
+
+
+
+
+
+
+
Meet your instructor
+
+
+
+
+
+
Hi, I'm Goku Mohandas
+
+
+
+
+
+
+
+
+
+ I've spent my career developing ML applications across all scales and industries. Specifically over the last four years (through Made With ML), I’ve had the opportunity to help dozens of F500 companies + startups build out their ML platforms and launch high-impact ML applications on top of them. I started Made With ML to address the gaps in education and share the best practices on how to deliver value with ML in production.
+
+
+
+ While this was an amazing experience, it was also a humbling one because there were obstacles around scale, integrations and productionization that I didn’t have great solutions for. So, I decided to join a team that has been addressing these precise obstacles with some of the best ML teams in the world and has an even bigger vision I could stand behind. So I'm excited to announce that Made With ML is now part of Anyscale to accelerate the path towards production ML.
+
+
+
+🎉 Made With ML is now part of Anyscale, read more about it here!
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
In the RNN lesson, we were constrained to using the representation at the very end but what if we could give contextual weight to each encoded input (\(h_i\)) when making our prediction? This is also preferred because it can help mitigate the vanishing gradient issue which stems from processing very long sequences. Below is attention applied to the outputs from an RNN. In theory, the outputs can come from anywhere where we want to learn how to weight amongst them but since we're working with the context of an RNN from the previous lesson , we'll continue with that.
+
+
+
+
+
\[ \alpha = softmax(W_{attn}h) \]
+
\[ c_t = \sum_{i=1}^{n} \alpha_{t,i}h_i \]
+
+
+
+
+
Variable
+
Description
+
+
+
+
+
\(N\)
+
batch size
+
+
+
\(M\)
+
max sequence length in the batch
+
+
+
\(H\)
+
hidden dim, model dim, etc.
+
+
+
\(h\)
+
RNN outputs (or any group of outputs you want to attend to) \(\in \mathbb{R}^{NXMXH}\)
+
+
+
\(\alpha_{t,i}\)
+
alignment function context vector \(c_t\) (attention in our case) $
+
+
+
\(W_{attn}\)
+
attention weights to learn \(\in \mathbb{R}^{HX1}\)
+
+
+
\(c_t\)
+
context vector that accounts for the different inputs with attention
+
+
+
+
+
+
Objective:
+
At it's core, attention is about learning how to weigh a group of encoded representations to produce a context-aware representation to use for downstream tasks. This is done by learning a set of attention weights and then using softmax to create attention values that sum to 1.
+
+
+
Advantages:
+
Learn how to account for the appropriate encoded representations regardless of position.
+
+
+
Disadvantages:
+
Another compute step that involves learning weights.
+
+
+
Miscellaneous:
+
Several state-of-the-art approaches extend on basic attention to deliver highly context-aware representations (ex. self-attention).
+
+
+
+
Set up
+
Let's set our seed and device for our main task.
+
We will download the AG News dataset, which consists of 120K text samples from 4 unique classes (Business, Sci/Tech, Sports, World)
+
1
+2
+3
+4
+5
# Load data
+url="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/news.csv"
+df=pd.read_csv(url,header=0)# load
+df=df.sample(frac=1).reset_index(drop=True)# shuffle
+df.head()
+
+
+
+
+
+
+
title
+
category
+
+
+
+
+
0
+
Sharon Accepts Plan to Reduce Gaza Army Operation...
+
World
+
+
+
1
+
Internet Key Battleground in Wildlife Crime Fight
+
Sci/Tech
+
+
+
2
+
July Durable Good Orders Rise 1.7 Percent
+
Business
+
+
+
3
+
Growing Signs of a Slowing on Wall Street
+
Business
+
+
+
4
+
The New Faces of Reality TV
+
World
+
+
+
+
+
+
Preprocessing
+
We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc.
+
defpreprocess(text,stopwords=STOPWORDS):
+"""Conditional preprocessing on our text unique to our task."""
+ # Lower
+ text=text.lower()
+
+ # Remove stopwords
+ pattern=re.compile(r"\b("+r"|".join(stopwords)+r")\b\s*")
+ text=pattern.sub("",text)
+
+ # Remove words in parenthesis
+ text=re.sub(r"\([^)]*\)","",text)
+
+ # Spacing and filters
+ text=re.sub(r"([-;;.,!?<=>])",r" \1 ",text)
+ text=re.sub("[^A-Za-z0-9]+"," ",text)# remove non alphanumeric chars
+ text=re.sub(" +"," ",text)# remove multiple spaces
+ text=text.strip()
+
+ returntext
+
+
1
+2
+3
# Sample
+text="Great week for the NYSE!"
+preprocess(text=text)
+
+
+great week nyse
+
+
1
+2
+3
+4
# Apply to dataframe
+preprocessed_df=df.copy()
+preprocessed_df.title=preprocessed_df.title.apply(preprocess)
+print(f"{df.title.values[0]}\n\n{preprocessed_df.title.values[0]}")
+
+
+Sharon Accepts Plan to Reduce Gaza Army Operation, Haaretz Says
+
+sharon accepts plan reduce gaza army operation haaretz says
+
+
+
+
Warning
+
If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before applying those operations. This is because we cannot apply any knowledge gained from the test set accidentally (data leak) during preprocessing/training. However for global preprocessing steps like the function above where we aren't learning anything from the data itself, we can perform before splitting the data.
It's important that we only fit using our train data split because during inference, our model will not always know every token so it's important to replicate that scenario with our validation and test splits as well.
# Sample of tokens
+print(take(5,tokenizer.token_to_index.items()))
+print(f"least freq token's freq: {tokenizer.min_token_freq}")# use this to adjust num_tokens
+
Attention applied to the outputs from an RNN. In theory, the outputs can come from anywhere where we want to learn how to weight amongst them but since we're working with the context of an RNN from the previous lesson , we'll continue with that.
+
+
+
+
+
\[ \alpha = softmax(W_{attn}h) \]
+
\[ c_t = \sum_{i=1}^{n} \alpha_{t,i}h_i \]
+
+
+
+
+
Variable
+
Description
+
+
+
+
+
\(N\)
+
batch size
+
+
+
\(M\)
+
max sequence length in the batch
+
+
+
\(H\)
+
hidden dim, model dim, etc.
+
+
+
\(h\)
+
RNN outputs (or any group of outputs you want to attend to) \(\in \mathbb{R}^{NXMXH}\)
+
+
+
\(\alpha_{t,i}\)
+
alignment function context vector \(c_t\) (attention in our case) $
+
+
+
\(W_{attn}\)
+
attention weights to learn \(\in \mathbb{R}^{HX1}\)
+
+
+
\(c_t\)
+
context vector that accounts for the different inputs with attention
+
+
+
+
+
1
importtorch.nn.functionalasF
+
+
The RNN will create an encoded representation for each word in our input resulting in a stacked vector that has dimensions \(NXMXH\), where N is the # of samples in the batch, M is the max sequence length in the batch, and H is the number of hidden units in the RNN.
# Encode
+rnn=nn.RNN(EMBEDDING_DIM,RNN_HIDDEN_DIM,batch_first=True)
+out,h_n=rnn(x)# h_n is the last hidden state
+print("out: ",out.shape)
+print("h_n: ",h_n.shape)
+
In a many-to-many task such as machine translation, our attentional interface will also account for the encoded representation of token in the output as well (via concatenation) so we can know which encoded inputs to attend to based on the encoded output we're focusing on. For more on this, be sure to explore Bahdanau's attention paper.
+
+
Model
+
Now let's create our RNN based model but with the addition of the attention layer on top of the RNN's outputs.
defget_probability_distribution(y_prob,classes):
+"""Create a dict of class probabilities from an array."""
+ results={}
+ fori,class_inenumerate(classes):
+ results[class_]=np.float64(y_prob[i])
+ sorted_results={k:vfork,vinsorted(
+ results.items(),key=lambdaitem:item[1],reverse=True)}
+ returnsorted_results
+
The word tennis was attended to the most to result in the Sports label.
+
Types of attention
+
We'll briefly look at the different types of attention and when to use each them.
+
Soft (global) attention
+
Soft attention the type of attention we've implemented so far, where we attend to all encoded inputs when creating our context vector.
+
+
advantages: we always have the ability to attend to all inputs in case something we saw much earlier/ see later are crucial for determining the output.
+
disadvantages: if our input sequence is very long, this can lead to expensive compute.
+
+
Hard attention
+
Hard attention is focusing on a specific set of the encoded inputs at each time step.
+
+
advantages: we can save a lot of compute on long sequences by only focusing on a local patch each time.
+
disadvantages: non-differentiable and so we need to use more complex techniques (variance reduction, reinforcement learning, etc.) to train.
Local attention blends the advantages of soft and hard attention. It involves learning an aligned position vector and empirically determining a local window of encoded inputs to attend to.
+
+
advantages: apply attention to a local patch of inputs yet remain differentiable.
+
disadvantages: need to determine the alignment vector for each output but it's a worthwhile trade off to determine the right window of inputs to attend to in order to avoid attending to all of them.
We can also use attention within the encoded input sequence to create a weighted representation that based on the similarity between input pairs. This will allow us to create rich representations of the input sequence that are aware of the relationships between each other. For example, in the image below you can see that when composing the representation of the token "its", this specific attention head will be incorporating signal from the token "Law" (it's learned that "its" is referring to the "Law").
In the next lesson, we'll implement Transformers that leverage self-attention to create contextual representations of our inputs for downstream applications.
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Attention - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
At the core of CNNs are filters (aka weights, kernels, etc.) which convolve (slide) across our input to extract relevant features. The filters are initialized randomly but learn to act as feature extractors via parameter sharing.
+
+
+
+
+
+
Objective:
+
Extract meaningful spatial substructure from encoded data.
We will download the AG News dataset, which consists of 120K text samples from 4 unique classes (Business, Sci/Tech, Sports, World)
+
1
+2
+3
+4
+5
# Load data
+url="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/news.csv"
+df=pd.read_csv(url,header=0)# load
+df=df.sample(frac=1).reset_index(drop=True)# shuffle
+df.head()
+
+
+
+
+
+
+
title
+
category
+
+
+
+
+
0
+
Sharon Accepts Plan to Reduce Gaza Army Operation...
+
World
+
+
+
1
+
Internet Key Battleground in Wildlife Crime Fight
+
Sci/Tech
+
+
+
2
+
July Durable Good Orders Rise 1.7 Percent
+
Business
+
+
+
3
+
Growing Signs of a Slowing on Wall Street
+
Business
+
+
+
4
+
The New Faces of Reality TV
+
World
+
+
+
+
+
+
Preprocessing
+
We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc.
+
Our input data is text and we can't feed it directly to our models. So, we'll define a Tokenizer to convert our text input data into token indices. This means that every token (we can decide what a token is char, word, sub-word, etc.) is mapped to a unique index which allows us to represent our text as an array of indices.
+We're going to restrict the number of tokens in our Tokenizer to the top 500 most frequent tokens (stop words already removed) because the full vocabulary size (~35K) is too large to run on Google Colab notebooks.
+
# Sample of tokens
+print(take(5,tokenizer.token_to_index.items()))
+print(f"least freq token's freq: {tokenizer.min_token_freq}")# use this to adjust num_tokens
+
# Convert texts to sequences of indices
+X_train=tokenizer.texts_to_sequences(X_train)
+X_val=tokenizer.texts_to_sequences(X_val)
+X_test=tokenizer.texts_to_sequences(X_test)
+preprocessed_text=tokenizer.sequences_to_texts([X_train[0]])[0]
+print("Text to indices:\n"
+ f" (preprocessed) → {preprocessed_text}\n"
+ f" (tokenized) → {X_train[0]}")
+
+
+Text to indices:
+ (preprocessed) → china <UNK> north korea nuclear talks
+ (tokenized) → [ 16 1 285 142 114 24]
+
+
+
+
Did we need to split the data first?
+
How come we applied the preprocessing functions to the entire dataset but tokenization after splitting the dataset? Does it matter?
+
+Show answer
+
If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before applying those operations. This is because we cannot apply any knowledge gained from the test set accidentally (data leak) during preprocessing/training. So for the tokenization process, it's important that we only fit using our train data split because during inference, our model will not always know every token so it's important to replicate that scenario with our validation and test splits as well. However for global preprocessing steps, like the preprocessing function where we aren't learning anything from the data itself, we can perform before splitting the data.
+
+
+
One-hot encoding
+
One-hot encoding creates a binary column for each unique value for the feature we're trying to map. All of the values in each token's array will be 0 except at the index that this specific token is represented by.
+
There are 5 words in the vocabulary:
+
1
+2
+3
+4
+5
+6
+7
{
+"a":0,
+"e":1,
+"i":2,
+"o":3,
+"u":4
+}
+
+
Then the text aou would be represented by:
+
1
+2
+3
[[1.0.0.0.0.]
+ [0.0.0.1.0.]
+ [0.0.0.0.1.]]
+
+
One-hot encoding allows us to represent our data in a way that our models can process the data and isn't biased by the actual value of the token (ex. if your labels were actual numbers).
+
+
We have already applied one-hot encoding in the previous lessons when we encoded our labels. Each label was represented by a unique index but when determining loss, we effectively use it's one hot representation and compared it to the predicted probability distribution. We never explicitly wrote this out since all of our previous tasks were multi-class which means every input had just one output class, so the 0s didn't affect the loss (though it did matter during back propagation).
+
+
1
+2
+3
+4
+5
+6
defto_categorical(seq,num_classes):
+"""One-hot encode a sequence of tokens."""
+ one_hot=np.zeros((len(seq),num_classes))
+ fori,iteminenumerate(seq):
+ one_hot[i,item]=1.
+ returnone_hot
+
# Convert tokens to one-hot
+vocab_size=len(tokenizer)
+X_train=[to_categorical(seq,num_classes=vocab_size)forseqinX_train]
+X_val=[to_categorical(seq,num_classes=vocab_size)forseqinX_val]
+X_test=[to_categorical(seq,num_classes=vocab_size)forseqinX_test]
+
+
Padding
+
Our inputs are all of varying length but we need each batch to be uniformly shaped. Therefore, we will use padding to make all the inputs in the batch the same length. Our padding index will be 0 (note that this is consistent with the <PAD> token defined in our Tokenizer).
+
+
One-hot encoding creates a batch of shape (N, max_seq_len, vocab_size) so we'll need to be able to pad 3D sequences.
+
+
1
+2
+3
+4
+5
+6
+7
+8
defpad_sequences(sequences,max_seq_len=0):
+"""Pad sequences to max length in sequence."""
+ max_seq_len=max(max_seq_len,max(len(sequence)forsequenceinsequences))
+ num_classes=sequences[0].shape[-1]
+ padded_sequences=np.zeros((len(sequences),max_seq_len,num_classes))
+ fori,sequenceinenumerate(sequences):
+ padded_sequences[i][:len(sequence)]=sequence
+ returnpadded_sequences
+
+
1
+2
+3
+4
# 3D sequences
+print(X_train[0].shape,X_train[1].shape,X_train[2].shape)
+padded=pad_sequences(X_train[0:3])
+print(padded.shape)
+
+
+(6, 500) (5, 500) (6, 500)
+(3, 6, 500)
+
+
+
+
Is our pad_sequences function properly created?
+
Notice any assumptions that could lead to hidden bugs?
+
+Show answer
+
By using np.zeros() to create our padded sequences, we're assuming that our pad token's index is 0. While this is the case for our project, someone could choose to use a different index and this can cause an error. Worst of all, this would be a silent error because all downstream operations would still run normally but our performance will suffer and it may not always be intuitive that this was the cause of issue!
+
+
+
Dataset
+
We're going to create Datasets and DataLoaders to be able to efficiently create batches with our data splits.
We're going to learn about CNNs by applying them on 1D text data.
+
Inputs
+
In the dummy example below, our inputs are composed of character tokens that are one-hot encoded. We have a batch of N samples, where each sample has 8 characters and each character is represented by an array of 10 values (vocab size=10). This gives our inputs the size (N, 8, 10).
+
+
With PyTorch, when dealing with convolution, our inputs (X) need to have the channels as the second dimension, so our inputs will be (N, 10, 8).
# Assume all our inputs are padded to have the same # of words
+batch_size=64
+max_seq_len=8# words per input
+vocab_size=10# one hot size
+x=torch.randn(batch_size,max_seq_len,vocab_size)
+print(f"X: {x.shape}")
+x=x.transpose(1,2)
+print(f"X: {x.shape}")
+
+ This diagram above is for char-level tokens but extends to any level of tokenization.
+
+
+
Filters
+
At the core of CNNs are filters (aka weights, kernels, etc.) which convolve (slide) across our input to extract relevant features. The filters are initialized randomly but learn to act as feature extractors via parameter sharing.
+
We can see convolution in the diagram below where we simplified the filters and inputs to be 2D for ease of visualization. Also note that the values are 0/1s but in reality they can be any floating point value.
+
+
+
+
+
Now let's return to our actual inputs x, which is of shape (8, 10) [max_seq_len, vocab_size] and we want to convolve on this input using filters. We will use 50 filters that are of size (1, 3) and has the same depth as the number of channels (num_channels = vocab_size = one_hot_size = 10). This gives our filter a shape of (3, 10, 50) [kernel_size, vocab_size, num_filters]
+
+
+
+
+
+
stride: amount the filters move from one convolution operation to the next.
+
padding: values (typically zero) padded to the input, typically to create a volume with whole number dimensions.
+
+
So far we've used a stride of 1 and VALID padding (no padding) but let's look at an example with a higher stride and difference between different padding approaches.
+
Padding types:
+
+
VALID: no padding, the filters only use the "valid" values in the input. If the filter cannot reach all the input values (filters go left to right), the extra values on the right are dropped.
+
SAME: adds padding evenly to the right (preferred) and left sides of the input so that all values in the input are processed.
+
+
+
+
+
+
We're going to use the Conv1d layer to process our inputs.
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
# Convolutional filters (VALID padding)
+vocab_size=10# one hot size
+num_filters=50# num filters
+filter_size=3# filters are 3X3
+stride=1
+padding=0# valid padding (no padding)
+conv1=nn.Conv1d(in_channels=vocab_size,out_channels=num_filters,
+ kernel_size=filter_size,stride=stride,
+ padding=padding,padding_mode="zeros")
+print("conv: {}".format(conv1.weight.shape))
+
When we apply these filter on our inputs, we receive an output of shape (N, 6, 50). We get 50 for the output channel dim because we used 50 filters and 6 for the conv outputs because:
Now we'll add padding so that the convolutional outputs are the same shape as our inputs. The amount of padding for the SAME padding can be determined using the same equation. We want out output to have the same width as our input, so we solve for P:
+
\[ \frac{W-F+2P}{S} + 1 = W \]
+
\[ P = \frac{S(W-1) - W + F}{2} \]
+
If \(P\) is not a whole number, we round up (using math.ceil) and place the extra padding on the right side.
+
1
+2
+3
+4
+5
+6
+7
+8
# Convolutional filters (SAME padding)
+vocab_size=10# one hot size
+num_filters=50# num filters
+filter_size=3# filters are 3X3
+stride=1
+conv=nn.Conv1d(in_channels=vocab_size,out_channels=num_filters,
+ kernel_size=filter_size,stride=stride)
+print("conv: {}".format(conv.weight.shape))
+
We will explore larger dimensional convolution layers in subsequent lessons. For example, Conv2D is used with 3D inputs (images, char-level text, etc.) and Conv3D is used for 4D inputs (videos, time-series, etc.).
+
+
Pooling
+
The result of convolving filters on an input is a feature map. Due to the nature of convolution and overlaps, our feature map will have lots of redundant information. Pooling is a way to summarize a high-dimensional feature map into a lower dimensional one for simplified downstream computation. The pooling operation can be the max value, average, etc. in a certain receptive field. Below is an example of pooling where the outputs from a conv layer are 4X4 and we're going to apply max pool filters of size 2X2.
In our use case, we want to just take the one max value so we will use the MaxPool1D layer, so our max-pool filter size will be max_seq_len.
+
1
+2
+3
# Max pooling
+pool_output=F.max_pool1d(z,z.size(2))
+print("Size: {}".format(pool_output.shape))
+
+
+Size: torch.Size([64, 50, 1])
+
+
+
Batch normalization
+
The last topic we'll cover before constructing our model is batch normalization. It's an operation that will standardize (mean=0, std=1) the activations from the previous layer. Recall that we used to standardize our inputs in previous notebooks so our model can optimize quickly with larger learning rates. It's the same concept here but we continue to maintain standardized values throughout the repeated forward passes to further aid optimization.
+
1
+2
+3
+4
# Batch normalization
+batch_norm=nn.BatchNorm1d(num_features=num_filters)
+z=batch_norm(conv(x))# applied to activations (after conv layer & before pooling)
+print(f"z: {z.shape}")
+
+
+z: torch.Size([64, 50, 6])
+
+
1
+2
# Mean and std before batchnorm
+print(f"mean: {torch.mean(conv(x)):.2f}, std: {torch.std(conv(x)):.2f}")
+
+
+mean: -0.00, std: 0.57
+
+
1
+2
# Mean and std after batchnorm
+print(f"mean: {torch.mean(z):.2f}, std: {torch.std(z):.2f}")
+
+
+mean: 0.00, std: 1.00
+
+
+
Modeling
+
Model
+
Let's visualize the model's forward pass.
+
+
We'll first tokenize our inputs (batch_size, max_seq_len).
+
Then we'll one-hot encode our tokenized inputs (batch_size, max_seq_len, vocab_size).
+
We'll apply convolution via filters (filter_size, vocab_size, num_filters) followed by batch normalization. Our filters act as character level n-gram detectors.
+
We'll apply 1D global max pooling which will extract the most relevant information from the feature maps for making the decision.
+
We feed the pool outputs to a fully-connected (FC) layer (with dropout).
+
We use one more FC layer with softmax to derive class probabilities.
We used SAME padding (w/ stride=1) which means that the conv outputs will have the same width (max_seq_len) as our inputs. The amount of padding differs for each batch based on the max_seq_len but you can calculate it by solving for P in the equation below.
\[ P = \frac{\text{stride}(\text{max_seq_len}-1) - \text{max_seq_len} + \text{filter_size}}{2} \]
+
If \(P\) is not a whole number, we round up (using math.ceil) and place the extra padding on the right side.
+
Training
+
Let's create the Trainer class that we'll use to facilitate training for our experiments. Notice that we're now moving the train function inside this class.
defget_probability_distribution(y_prob,classes):
+"""Create a dict of class probabilities from an array."""
+ results={}
+ fori,class_inenumerate(classes):
+ results[class_]=np.float64(y_prob[i])
+ sorted_results={k:vfork,vinsorted(
+ results.items(),key=lambdaitem:item[1],reverse=True)}
+ returnsorted_results
+
# Dataloader
+text="What a day for the new york stock market to go bust!"
+sequences=tokenizer.texts_to_sequences([preprocess(text)])
+print(tokenizer.sequences_to_texts(sequences))
+X=[to_categorical(seq,num_classes=len(tokenizer))forseqinsequences]
+y_filler=label_encoder.encode([label_encoder.classes[0]]*len(X))
+dataset=Dataset(X=X,y=y_filler,max_filter_size=FILTER_SIZE)
+dataloader=dataset.create_dataloader(batch_size=batch_size)
+
We went through all the trouble of padding our inputs before convolution to result in outputs of the same shape as our inputs so we can try to get some interpretability. Since every token is mapped to a convolutional output on which we apply max pooling, we can see which token's output was most influential towards the prediction. We first need to get the conv outputs from our model:
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
In a nutshell, a machine learning model consumes input data and produces predictions. The quality of the predictions directly corresponds to the quality of data you train the model with; garbage in, garbage out. Check out this article on where it makes sense to use AI and how to properly apply it.
+
We're going to go through all the concepts with concrete code examples and some synthesized data to train our models on. The task is to determine whether a tumor will be benign (harmless) or malignant (harmful) based on leukocyte (white blood cells) count and blood pressure. This is a synthetic dataset that we created and has no clinical relevance.
+
Set up
+
We'll set our seeds for reproducibility.
+
1
+2
importnumpyasnp
+importrandom
+
+
1
SEED=1234
+
+
1
+2
+3
# Set seed for reproducibility
+np.random.seed(SEED)
+random.seed(SEED)
+
+
Full dataset
+
We'll first train a model with the entire dataset. Later we'll remove a subset of the dataset and see the effect it has on our model.
We want to choose features that have strong predictive signal for our task. If you want to improve performance, you need to continuously do feature engineering by collecting and adding new signals. So you may run into a new feature that has high correlation (orthogonal signal) with your existing features but it may still possess some unique signal to boost your predictive performance.
+
# Standardize the data (mean=0, std=1) using training data
+X_scaler=StandardScaler().fit(X_train)
+
+
1
+2
+3
+4
# Apply scaler on training and test data (don't standardize outputs for classification)
+X_train=X_scaler.transform(X_train)
+X_val=X_scaler.transform(X_val)
+X_test=X_scaler.transform(X_test)
+
+
1
+2
+3
# Check (means should be ~0 and std should be ~1)
+print(f"X_test[0]: mean: {np.mean(X_test[:,0],axis=0):.1f}, std: {np.std(X_test[:,0],axis=0):.1f}")
+print(f"X_test[1]: mean: {np.mean(X_test[:,1],axis=0):.1f}, std: {np.std(X_test[:,1],axis=0):.1f}")
+
We're going to plot a point, which we know belongs to the malignant tumor class. Our well trained model here would accurately predict that it is indeed a malignant tumor!
+
Great! We received great performances on both our train and test data splits. We're going to use this dataset to show the importance of data quality.
+
Reduced dataset
+
Let's remove some training data near the decision boundary and see how robust the model is now.
+
Load data
+
1
+2
+3
+4
+5
# Raw reduced data
+url="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/tumors_reduced.csv"
+df_reduced=pd.read_csv(url,header=0)# load
+df_reduced=df_reduced.sample(frac=1).reset_index(drop=True)# shuffle
+df_reduced.head()
+
+
+
+
+
+
+
leukocyte_count
+
blood_pressure
+
tumor_class
+
+
+
+
+
0
+
16.795186
+
14.434741
+
benign
+
+
+
1
+
13.472969
+
15.250393
+
malignant
+
+
+
2
+
9.840450
+
16.434717
+
malignant
+
+
+
3
+
16.390730
+
14.419258
+
benign
+
+
+
4
+
13.367974
+
15.741790
+
malignant
+
+
+
+
+
+
1
+2
+3
+4
+5
# Define X and y
+X=df_reduced[["leukocyte_count","blood_pressure"]].values
+y=df_reduced["tumor_class"].values
+print("X: ",np.shape(X))
+print("y: ",np.shape(y))
+
# Standardize inputs using training data
+X_scaler=StandardScaler().fit(X_train)
+X_train=X_scaler.transform(X_train)
+X_val=X_scaler.transform(X_val)
+X_test=X_scaler.transform(X_test)
+
+
Model
+
1
+2
# Initialize model
+model=MLP(input_dim=INPUT_DIM,hidden_dim=HIDDEN_DIM,num_classes=NUM_CLASSES)
+
+
Training
+
1
+2
+3
# Define Loss
+class_weights_tensor=torch.Tensor(list(class_weights.values()))
+loss_fn=nn.CrossEntropyLoss(weight=class_weights_tensor)
+
# Visualize the decision boundary
+plt.figure(figsize=(8,5))
+plt.title("Test")
+plot_multiclass_decision_boundary(model=model,X=X_test,y=y_test)
+
+# Sample point near the decision boundary (same point as before)
+plt.scatter(mean_leukocyte_count+0.05,mean_blood_pressure-0.05,s=200,
+ c="b",edgecolor="w",linewidth=2)
+
+# Annotate
+plt.annotate("true: malignant,\npred: benign",
+ color="white",
+ xy=(mean_leukocyte_count,mean_blood_pressure),
+ xytext=(0.45,0.60),
+ textcoords="figure fraction",
+ fontsize=16,
+ arrowprops=dict(facecolor="white",shrink=0.1))
+plt.show()
+
+
+
+
+
+
This is a very fragile but highly realistic scenario. Based on our reduced synthetic dataset, we have achieved a model that generalized really well on the test data. But when we ask for the prediction for the same point tested earlier (which we known is malignant), the prediction is now a benign tumor. We would have completely missed the tumor. To mitigate this, we can:
+
+
Get more data around the space we are concerned about
+
Consume predictions with caution when they are close to the decision boundary
+
+
Takeaway
+
Models are not crystal balls. So it's important that before any machine learning, we really look at our data and ask ourselves if it is truly representative for the task we want to solve. The model itself may fit really well and generalize well on your data but if the data is of poor quality to begin with, the model cannot be trusted.
+
Once you are confident that your data is of good quality, you can finally start thinking about modeling. The type of model you choose depends on many factors, including the task, type of data, complexity required, etc.
+
So once you figure out what type of model your task needs, start with simple models and then slowly add complexity. You don’t want to start with neural networks right away because that may not be right model for your data and task. Striking this balance in model complexity is one of the key tasks of your data scientists. simple models → complex models
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Data quality - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
While one-hot encoding allows us to preserve the structural information, it does poses two major disadvantages.
+
+
linearly dependent on the number of unique tokens in our vocabulary, which is a problem if we're dealing with a large corpus.
+
representation for each token does not preserve any relationship with respect to other tokens.
+
+
In this notebook, we're going to motivate the need for embeddings and how they address all the shortcomings of one-hot encoding. The main idea of embeddings is to have fixed length representations for the tokens in a text regardless of the number of tokens in the vocabulary. With one-hot encoding, each token is represented by an array of size vocab_size, but with embeddings, each token now has the shape embed_dim. The values in the representation will are not fixed binary values but rather, changing floating points allowing for fine-grained learned representations.
+
+
Objectives:
+
Represent tokens in text that capture the intrinsic semantic relationships.
+
+
+
Advantages:
+
Low-dimensionality while capturing relationships.
+
Interpretable token representations
+
+
+
Disadvantages:
+
Can be computationally intensive to precompute.
+
+
+
Miscellaneous:
+
There are lot's of pretrained embeddings to choose from but you can also train your own from scratch.
+
+
+
+
Learning embeddings
+
We can learn embeddings by creating our models in PyTorch but first, we're going to use a library that specializes in embeddings and topic modeling called Gensim.
+Snape nodded, but did not elaborate.
+['snape', 'nodded', 'but', 'did', 'not', 'elaborate']
+
+
+
But how do we learn the embeddings the first place? The intuition behind embeddings is that the definition of a token doesn't depend on the token itself but on its context. There are several different ways of doing this:
+
+
Given the word in the context, predict the target word (CBOW - continuous bag of words).
+
Given the target word, predict the context word (skip-gram).
+
Given a sequence of words, predict the next word (LM - language modeling).
+
+
All of these approaches involve create data to train our model on. Every word in a sentence becomes the target word and the context words are determines by a window. In the image below (skip-gram), the window size is 2 (2 words to the left and right of the target word). We repeat this for every sentence in our corpus and this results in our training data for the unsupervised task. This in an unsupervised learning technique since we don't have official labels for contexts. The idea is that similar target words will appear with similar contexts and we can learn this relationship by repeatedly training our mode with (context, target) pairs.
+
+
+
+
+
We can learn embeddings using any of these approaches above and some work better than others. You can inspect the learned embeddings but the best way to choose an approach is to empirically validate the performance on a supervised task.
+
Word2Vec
+
When we have large vocabularies to learn embeddings for, things can get complex very quickly. Recall that the backpropagation with softmax updates both the correct and incorrect class weights. This becomes a massive computation for every backwards pass we do so a workaround is to use negative sampling which only updates the correct class and a few arbitrary incorrect classes (NEGATIVE_SAMPLING=20). We're able to do this because of the large amount of training data where we'll see the same word as the target class multiple times.
EMBEDDING_DIM=100
+WINDOW=5
+MIN_COUNT=3# Ignores all words with total frequency lower than this
+SKIP_GRAM=1# 0 = CBOW
+NEGATIVE_SAMPLING=20
+
+
1
+2
+3
+4
+5
+6
# Super fast because of optimized C code under the hood
+w2v=Word2Vec(
+ sentences=sentences,size=EMBEDDING_DIM,
+ window=WINDOW,min_count=MIN_COUNT,
+ sg=SKIP_GRAM,negative=NEGATIVE_SAMPLING)
+print(w2v)
+
+
+Word2Vec(vocab=4937, size=100, alpha=0.025)
+
+
1
+2
# Vector for each word
+w2v.wv.get_vector("potter")
+
# Saving and loading
+w2v.wv.save_word2vec_format("model.bin",binary=True)
+w2v=KeyedVectors.load_word2vec_format("model.bin",binary=True)
+
+
FastText
+
What happens when a word doesn't exist in our vocabulary? We could assign an UNK token which is used for all OOV (out of vocabulary) words or we could use FastText, which uses character-level n-grams to embed a word. This helps embed rare words, misspelled words, and also words that don't exist in our corpus but are similar to words in our corpus.
+
1
fromgensim.modelsimportFastText
+
+
1
+2
+3
+4
+5
# Super fast because of optimized C code under the hood
+ft=FastText(sentences=sentences,size=EMBEDDING_DIM,
+ window=WINDOW,min_count=MIN_COUNT,
+ sg=SKIP_GRAM,negative=NEGATIVE_SAMPLING)
+print(ft)
+
+
+FastText(vocab=4937, size=100, alpha=0.025)
+
+
1
+2
# This word doesn't exist so the word2vec model will error out
+w2v.wv.most_similar(positive="scarring",topn=5)
+
+
1
+2
# FastText will use n-grams to embed an OOV word
+ft.wv.most_similar(positive="scarring",topn=5)
+
# Save and loading
+ft.wv.save("model.bin")
+ft=KeyedVectors.load("model.bin")
+
+
Pretrained embeddings
+
We can learn embeddings from scratch using one of the approaches above but we can also leverage pretrained embeddings that have been trained on millions of documents. Popular ones include Word2Vec (skip-gram) or GloVe (global word-word co-occurrence). We can validate that these embeddings captured meaningful semantic relationships by confirming them.
# Save GloVe embeddings to local directory in word2vec format
+word2vec_output_file="{0}.word2vec".format(embeddings_file)
+glove2word2vec(embeddings_file,word2vec_output_file)
+
+
+(400000, 100)
+
+
1
+2
# Load embeddings (may take a minute)
+glove=KeyedVectors.load_word2vec_format(word2vec_output_file,binary=False)
+
+
1
+2
+3
# (king - man) + woman = ?
+# king - man = ? - woman
+glove.most_similar(positive=["woman","king"],negative=["man"],topn=5)
+
We will download the AG News dataset, which consists of 120K text samples from 4 unique classes (Business, Sci/Tech, Sports, World)
+
1
+2
+3
+4
+5
# Load data
+url="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/news.csv"
+df=pd.read_csv(url,header=0)# load
+df=df.sample(frac=1).reset_index(drop=True)# shuffle
+df.head()
+
+
+
+
+
+
+
title
+
category
+
+
+
+
+
0
+
Sharon Accepts Plan to Reduce Gaza Army Operation...
+
World
+
+
+
1
+
Internet Key Battleground in Wildlife Crime Fight
+
Sci/Tech
+
+
+
2
+
July Durable Good Orders Rise 1.7 Percent
+
Business
+
+
+
3
+
Growing Signs of a Slowing on Wall Street
+
Business
+
+
+
4
+
The New Faces of Reality TV
+
World
+
+
+
+
+
+
Preprocessing
+
We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc.
+
defpreprocess(text,stopwords=STOPWORDS):
+"""Conditional preprocessing on our text unique to our task."""
+ # Lower
+ text=text.lower()
+
+ # Remove stopwords
+ pattern=re.compile(r"\b("+r"|".join(stopwords)+r")\b\s*")
+ text=pattern.sub("",text)
+
+ # Remove words in parenthesis
+ text=re.sub(r"\([^)]*\)","",text)
+
+ # Spacing and filters
+ text=re.sub(r"([-;;.,!?<=>])",r" \1 ",text)
+ text=re.sub("[^A-Za-z0-9]+"," ",text)# remove non alphanumeric chars
+ text=re.sub(" +"," ",text)# remove multiple spaces
+ text=text.strip()
+
+ returntext
+
+
1
+2
+3
# Sample
+text="Great week for the NYSE!"
+preprocess(text=text)
+
+
+great week nyse
+
+
1
+2
+3
+4
# Apply to dataframe
+preprocessed_df=df.copy()
+preprocessed_df.title=preprocessed_df.title.apply(preprocess)
+print(f"{df.title.values[0]}\n\n{preprocessed_df.title.values[0]}")
+
+
+Sharon Accepts Plan to Reduce Gaza Army Operation, Haaretz Says
+
+sharon accepts plan reduce gaza army operation haaretz says
+
+
+
+
Warning
+
If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before applying those operations. This is because we cannot apply any knowledge gained from the test set accidentally (data leak) during preprocessing/training. However for global preprocessing steps like the function above where we aren't learning anything from the data itself, we can perform before splitting the data.
It's important that we only fit using our train data split because during inference, our model will not always know every token so it's important to replicate that scenario with our validation and test splits as well.
# Sample of tokens
+print(take(5,tokenizer.token_to_index.items()))
+print(f"least freq token's freq: {tokenizer.min_token_freq}")# use this to adjust num_tokens
+
Each token in the input is represented via embeddings (all out-of-vocabulary (OOV) tokens are given the embedding for UNK token.) In the model below, we'll see how to set these embeddings to be pretrained GloVe embeddings and how to choose whether to freeze (fixed embedding weights) those embeddings or not during training.
+
Padding
+
Our inputs are all of varying length but we need each batch to be uniformly shaped. Therefore, we will use padding to make all the inputs in the batch the same length. Our padding index will be 0 (note that this is consistent with the <PAD> token defined in our Tokenizer).
+
+
While embedding our input tokens will create a batch of shape (N, max_seq_len, embed_dim) we only need to provide a 2D matrix (N, max_seq_len) for using embeddings with PyTorch.
+
+
1
+2
+3
+4
+5
+6
+7
defpad_sequences(sequences,max_seq_len=0):
+"""Pad sequences to max length in sequence."""
+ max_seq_len=max(max_seq_len,max(len(sequence)forsequenceinsequences))
+ padded_sequences=np.zeros((len(sequences),max_seq_len))
+ fori,sequenceinenumerate(sequences):
+ padded_sequences[i][:len(sequence)]=sequence
+ returnpadded_sequences
+
We'll be using a convolutional neural network on top of our embedded tokens to extract meaningful spatial signal. This time, we'll be using many filter widths to act as n-gram feature extractors.
+
Let's visualize the model's forward pass.
+
+
We'll first tokenize our inputs (batch_size, max_seq_len).
+
Then we'll embed our tokenized inputs (batch_size, max_seq_len, embedding_dim).
+
We'll apply convolution via filters (filter_size, embedding_dim, num_filters) followed by batch normalization. Our filters act as character level n-gram detectors. We have three different filter sizes (2, 3 and 4) and they will act as bi-gram, tri-gram and 4-gram feature extractors, respectively.
+
We'll apply 1D global max pooling which will extract the most relevant information from the feature maps for making the decision.
+
We feed the pool outputs to a fully-connected (FC) layer (with dropout).
+
We use one more FC layer with softmax to derive class probabilities.
We have first have to decide whether to use pretrained embeddings randomly initialized ones. Then, we can choose to freeze our embeddings or continue to train them using the supervised data (this could lead to overfitting). Here are the three experiments we're going to conduct:
defget_probability_distribution(y_prob,classes):
+"""Create a dict of class probabilities from an array."""
+ results={}
+ fori,class_inenumerate(classes):
+ results[class_]=np.float64(y_prob[i])
+ sorted_results={k:vfork,vinsorted(
+ results.items(),key=lambdaitem:item[1],reverse=True)}
+ returnsorted_results
+
We went through all the trouble of padding our inputs before convolution to result is outputs of the same shape as our inputs so we can try to get some interpretability. Since every token is mapped to a convolutional output on which we apply max pooling, we can see which token's output was most influential towards the prediction. We first need to get the conv outputs from our model:
1D global max-pooling would extract the highest value from each of our num_filters for each filter_size. We could also follow this same approach to figure out which n-gram is most relevant but notice in the heatmap above that many filters don't have much variance. To mitigate this, this paper uses threshold values to determine which filters to use for interpretability. But to keep things simple, let's extract which tokens' filter outputs were extracted via max-pooling the most frequently.
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
Our goal is to learn a linear model \(\hat{y}\) that models \(y\) given \(X\) using weights \(W\) and bias \(b\):
+
\[ \hat{y} = XW + b \]
+
+
+
+
+
Variable
+
Description
+
+
+
+
+
\(N\)
+
total numbers of samples
+
+
+
\(\hat{y}\)
+
predictions \(\in \mathbb{R}^{NX1}\)
+
+
+
\(X\)
+
inputs \(\in \mathbb{R}^{NXD}\)
+
+
+
\(W\)
+
weights \(\in \mathbb{R}^{DX1}\)
+
+
+
\(b\)
+
bias \(\in \mathbb{R}^{1}\)
+
+
+
+
+
+
Objective:
+
Use inputs \(X\) to predict the output \(\hat{y}\) using a linear model. The model will be a line of best fit that minimizes the distance between the predicted (model's output) and target (ground truth) values. Training data \((X, y)\) is used to train the model and learn the weights \(W\) using gradient descent.
+
+
+
Advantages:
+
Computationally simple.
+
Highly interpretable.
+
Can account for continuous and categorical features.
+
+
+
Disadvantages:
+
The model will perform well only when the data is linearly separable (for classification).
+
+
+
Miscellaneous:
+
You can also use linear regression for binary classification tasks where if the predicted continuous value is above a threshold, it belongs to a certain class. But we will cover better techniques for classification in future lessons and will focus on linear regression for continuous regression tasks only.
+
+
+
+
Generate data
+
We're going to generate some simple dummy data to apply linear regression on. It's going to create roughly linear data (y = 3.5X + noise); the random noise is added to create realistic data that doesn't perfectly align in a line. Our goal is to have the model converge to a similar linear equation (there will be slight variance since we added some noise).
+
# Set seed for reproducibility
+np.random.seed(SEED)
+
+
1
+2
+3
+4
+5
+6
+7
# Generate synthetic data
+defgenerate_data(num_samples):
+"""Generate dummy data for linear regression."""
+ X=np.array(range(num_samples))
+ random_noise=np.random.uniform(-10,20,size=num_samples)
+ y=3.5*X+random_noise# add some noise
+ returnX,y
+
+
1
+2
+3
+4
# Generate random (linear) data
+X,y=generate_data(num_samples=NUM_SAMPLES)
+data=np.vstack([X,y]).T
+print(data[:5])
+
Now that we have our data prepared, we'll first implement linear regression using just NumPy. This will let us really understand the underlying operations.
+
Split data
+
Since our task is a regression task, we will randomly split our dataset into three sets: train, validation and test data splits.
+
+
train: used to train our model.
+
val : used to validate our model's performance during training.
+
test: used to do an evaluation of our fully trained model.
+
+
+
Be sure to check out our entire lesson focused on properlysplitting data in our MLOps course.
+
+
1
+2
+3
TRAIN_SIZE=0.7
+VAL_SIZE=0.15
+TEST_SIZE=0.15
+
+
1
+2
+3
+4
+5
# Shuffle data
+indices=list(range(NUM_SAMPLES))
+np.random.shuffle(indices)
+X=X[indices]
+y=y[indices]
+
+
+
+
Warning
+
Be careful not to shuffle \(X\) and \(y\) separately because then the inputs won't correspond to the outputs!
+
+
1
+2
+3
+4
+5
+6
# Split indices
+train_start=0
+train_end=int(0.7*NUM_SAMPLES)
+val_start=train_end
+val_end=int((TRAIN_SIZE+VAL_SIZE)*NUM_SAMPLES)
+test_start=val_end
+
# Determine means and stds
+X_mean=np.mean(X_train)
+X_std=np.std(X_train)
+y_mean=np.mean(y_train)
+y_std=np.std(y_train)
+
+
+
+
We need to treat the validation and test sets as if they were hidden datasets. So we only use the train set to determine the mean and std to avoid biasing our training process.
# Check (means should be ~0 and std should be ~1)
+# Check (means should be ~0 and std should be ~1)
+print(f"mean: {np.mean(X_test,axis=0)[0]:.1f}, std: {np.std(X_test,axis=0)[0]:.1f}")
+print(f"mean: {np.mean(y_test,axis=0)[0]:.1f}, std: {np.std(y_test,axis=0)[0]:.1f}")
+
+
+mean: -0.4, std: 0.9
+mean: -0.3, std: 1.0
+
+
+
Weights
+
Our goal is to learn a linear model \(\hat{y}\) that models \(y\) given \(X\) using weights \(W\) and bias \(b\) → \(\hat{y} = XW + b\)
+
Step 1: Randomly initialize the model's weights \(W\).
+
1
+2
INPUT_DIM=X_train.shape[1]# X is 1-dimensional
+OUTPUT_DIM=y_train.shape[1]# y is 1-dimensional
+
Step 3: Compare the predictions \(\hat{y}\) with the actual target values \(y\) using the objective (cost) function to determine the loss \(J\). A common objective function for linear regression is mean squared error (MSE). This function calculates the difference between the predicted and target values and squares it.
The gradient is the derivative, or the rate of change of a function. It's a vector that points in the direction of greatest increase of a function. For example the gradient of our loss function (\(J\)) with respect to our weights (\(W\)) will tell us how to change \(W\) so we can maximize \(J\). However, we want to minimize our loss so we subtract the gradient from \(W\).
+
Update weights
+
Step 5: Update the weights \(W\) using a small learning rate \(\alpha\).
+
\[ W = W - \alpha\frac{\partial{J}}{\partial{W}} \]
+
\[ b = b - \alpha\frac{\partial{J}}{\partial{b}} \]
The learning rate \(\alpha\) is a way to control how much we update the weights by. If we choose a small learning rate, it may take a long time for our model to train. However, if we choose a large learning rate, we may overshoot and our training will never converge. The specific learning rate depends on our data and the type of models we use but it's typically good to explore in the range of \([1e^{-8}, 1e^{-1}]\). We'll explore learning rate update strategies in later lessons.
+
+
Training
+
Step 6: Repeat steps 2 - 5 to minimize the loss and train the model.
+
To keep the code simple, we're not calculating and displaying the validation loss after each epoch here. But in later lessons, the performance on the validation set will be crucial in influencing the learning process (learning rate, when to stop training, etc.).
+
+
Evaluation
+
Now we're ready to see how well our trained model will perform on our test (hold-out) data split. This will be our best measure on how well the model would perform on the real world, given that our dataset's distribution is close to unseen data.
+
# Figure size
+plt.figure(figsize=(15,5))
+
+# Plot train data
+plt.subplot(1,2,1)
+plt.title("Train")
+plt.scatter(X_train,y_train,label="y_train")
+plt.plot(X_train,pred_train,color="red",linewidth=1,linestyle="-",label="model")
+plt.legend(loc="lower right")
+
+# Plot test data
+plt.subplot(1,2,2)
+plt.title("Test")
+plt.scatter(X_test,y_test,label='y_test')
+plt.plot(X_test,pred_test,color="red",linewidth=1,linestyle="-",label="model")
+plt.legend(loc="lower right")
+
+# Show plots
+plt.show()
+
+
+
+
+
+
Interpretability
+
Since we standardized our inputs and outputs, our weights were fit to those standardized values. So we need to unstandardize our weights so we can compare it to our true weight (3.5).
This time we'll use scikit-learn's StandardScaler to standardize our data.
+
1
fromsklearn.preprocessingimportStandardScaler
+
+
1
+2
+3
# Standardize the data (mean=0, std=1) using training data
+X_scaler=StandardScaler().fit(X_train)
+y_scaler=StandardScaler().fit(y_train)
+
+
1
+2
+3
+4
+5
+6
+7
# Apply scaler on training and test data
+X_train=X_scaler.transform(X_train)
+y_train=y_scaler.transform(y_train).ravel().reshape(-1,1)
+X_val=X_scaler.transform(X_val)
+y_val=y_scaler.transform(y_val).ravel().reshape(-1,1)
+X_test=X_scaler.transform(X_test)
+y_test=y_scaler.transform(y_test).ravel().reshape(-1,1)
+
+
1
+2
+3
# Check (means should be ~0 and std should be ~1)
+print(f"mean: {np.mean(X_test,axis=0)[0]:.1f}, std: {np.std(X_test,axis=0)[0]:.1f}")
+print(f"mean: {np.mean(y_test,axis=0)[0]:.1f}, std: {np.std(y_test,axis=0)[0]:.1f}")
+
+
+mean: -0.3, std: 0.7
+mean: -0.3, std: 0.6
+
+
+
Weights
+
We will be using PyTorch's Linear layers in our MLP implementation. These layers will act as out weights (and biases).
+
\[ z = XW \]
+
1
fromtorchimportnn
+
+
1
+2
+3
+4
+5
# Inputs
+N=3# num samples
+x=torch.randn(N,INPUT_DIM)
+print(x.shape)
+print(x.numpy())
+
When we implemented linear regression with just NumPy, we used batch gradient descent to update our weights (used entire training set). But there are actually many different gradient descent optimization algorithms to choose from and it depends on the situation. However, the ADAM optimizer has become a standard algorithm for most cases.
# Figure size
+plt.figure(figsize=(15,5))
+
+# Plot train data
+plt.subplot(1,2,1)
+plt.title("Train")
+plt.scatter(X_train,y_train,label="y_train")
+plt.plot(X_train,pred_train.detach().numpy(),color="red",linewidth=1,linestyle="-",label="model")
+plt.legend(loc="lower right")
+
+# Plot test data
+plt.subplot(1,2,2)
+plt.title("Test")
+plt.scatter(X_test,y_test,label='y_test')
+plt.plot(X_test,pred_test.detach().numpy(),color="red",linewidth=1,linestyle="-",label="model")
+plt.legend(loc="lower right")
+
+# Show plots
+plt.show()
+
+
+
+
+
+
Inference
+
After training a model, we can use it to predict on new data.
+
1
+2
+3
+4
# Feed in your own inputs
+sample_indices=[10,15,25]
+X_infer=np.array(sample_indices,dtype=np.float32)
+X_infer=torch.Tensor(X_scaler.transform(X_infer.reshape(-1,1)))
+
+
Recall that we need to unstandardize our predictions.
Linear regression offers the great advantage of being highly interpretable. Each feature has a coefficient which signifies its importance/impact on the output variable y. We can interpret our coefficient as follows: by increasing X by 1 unit, we increase y by \(W\) (~3.65) units.
+
Regularization helps decrease overfitting. Below is L2 regularization (ridge regression). There are many forms of regularization but they all work to reduce overfitting in our models. With L2 regularization, we are penalizing large weight values by decaying them because having large weights will lead to preferential bias with the respective inputs and we want the model to work with all the inputs and not just a select few. There are also other types of regularization like L1 (lasso regression) which is useful for creating sparse models where some feature coefficients are zeroed out, or elastic which combines L1 and L2 penalties.
+
+
Regularization is not just for linear regression. You can use it to regularize any model's weights including the ones we will look at in future lessons.
Regularization didn't make a difference in performance with this specific example because our data is generated from a perfect linear equation but for large realistic data, regularization can help our model generalize well.
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Linear regression - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
Logistic regression is an extension on linear regression (both are generalized linear methods). We will still learn to model a line (plane) that models \(y\) given \(X\). Except now we are dealing with classification problems as opposed to regression problems so we'll be predicting probability distributions as opposed to discrete values. We'll be using the softmax operation to normalize our logits (\(XW\)) to derive probabilities.
+
Our goal is to learn a logistic model \(\hat{y}\) that models \(y\) given \(X\).
+
\[ \hat{y} = \frac{e^{XW_y}}{\sum_j e^{XW}} \]
+
+
+
+
+
Variable
+
Description
+
+
+
+
+
\(N\)
+
total numbers of samples
+
+
+
\(C\)
+
number of classes
+
+
+
\(\hat{y}\)
+
predictions \(\in \mathbb{R}^{NXC}\)
+
+
+
\(X\)
+
inputs \(\in \mathbb{R}^{NXD}\)
+
+
+
\(W\)
+
weights \(\in \mathbb{R}^{DXC}\)
+
+
+
+
(*) bias term (\(b\)) excluded to avoid crowding the notations
+
+
This function is known as the multinomial logistic regression or the softmax classifier. The softmax classifier will use the linear equation (\(z=XW\)) and normalize it (using the softmax function) to produce the probability for class y given the inputs.
+
+
Objectives:
+
Predict the probability of class \(y\) given the inputs \(X\). The softmax classifier normalizes the linear outputs to determine class probabilities.
+
+
+
Advantages:
+
Can predict class probabilities given a set on inputs.
+
+
+
Disadvantages:
+
Sensitive to outliers since objective is to minimize cross entropy loss. Support vector machines (SVMs) are a good alternative to counter outliers.
+
+
+
Miscellaneous:
+
Softmax classifier is widely in neural network architectures as the last layer since it produces class probabilities.
+
+
+
+
Set up
+
We'll set our seeds for reproducibility.
+
1
+2
importnumpyasnp
+importrandom
+
+
1
SEED=1234
+
+
1
+2
+3
# Set seed for reproducibility
+np.random.seed(SEED)
+random.seed(SEED)
+
+
Load data
+
We'll used some synthesized data to train our models on. The task is to determine whether a tumor will be benign (harmless) or malignant (harmful) based on leukocyte (white blood cells) count and blood pressure. Note that this is a synthetic dataset that has no clinical relevance.
We want to split our dataset so that each of the three splits has the same distribution of classes so that we can train and evaluate properly. We can easily achieve this by telling scikit-learn's train_test_split function what to stratify on.
+
You'll notice that our class labels are text. We need to encode them into integers so we can use them in our models. We could scikit-learn's LabelEncoder to do this but we're going to write our own simple label encoder class so we can see what's happening under the hood.
We also want to calculate our class weights, which are useful for weighting the loss function during training. It tells the model to focus on samples from an under-represented class. The loss section below will show how to incorporate these weights.
+
1
+2
+3
+4
# Class weights
+counts=np.bincount(y_train)
+class_weights={i:1.0/countfori,countinenumerate(counts)}
+print(f"counts: {counts}\nweights: {class_weights}")
+
We need to standardize our data (zero mean and unit variance) so a specific feature's magnitude doesn't affect how the model learns its weights. We're only going to standardize the inputs X because our outputs y are class values.
+
1
fromsklearn.preprocessingimportStandardScaler
+
+
1
+2
# Standardize the data (mean=0, std=1) using training data
+X_scaler=StandardScaler().fit(X_train)
+
+
1
+2
+3
+4
# Apply scaler on training and test data (don't standardize outputs for classification)
+X_train=X_scaler.transform(X_train)
+X_val=X_scaler.transform(X_val)
+X_test=X_scaler.transform(X_test)
+
+
1
+2
+3
# Check (means should be ~0 and std should be ~1)
+print(f"X_test[0]: mean: {np.mean(X_test[:,0],axis=0):.1f}, std: {np.std(X_test[:,0],axis=0):.1f}")
+print(f"X_test[1]: mean: {np.mean(X_test[:,1],axis=0):.1f}, std: {np.std(X_test[:,1],axis=0):.1f}")
+
Now that we have our data prepared, we'll first implement logistic regression using just NumPy. This will let us really understand the underlying operations. It's normal to find the math and code in this section slightly complex. You can still read each of the steps to build intuition for when we implement this using PyTorch.
+
Our goal is to learn a logistic model \(\hat{y}\) that models \(y\) given \(X\).
+
\[ \hat{y} = \frac{e^{XW_y}}{\sum_j e^{XW}} \]
+
+
We are going to use multinomial logistic regression even though our task only involves two classes because you can generalize the softmax classifier to any number of classes.
+
+
Initialize weights
+
Step 1: Randomly initialize the model's weights \(W\).
+
1
+2
INPUT_DIM=X_train.shape[1]# X is 2-dimensional
+NUM_CLASSES=len(label_encoder.classes)# y has two possibilities (benign or malignant)
+
Step 2: Feed inputs \(X\) into the model to receive the logits (\(z=XW\)). Apply the softmax operation on the logits to get the class probabilities \(\hat{y}\) in one-hot encoded form. For example, if there are three classes, the predicted class probabilities could look like [0.3, 0.3, 0.4].
# Normalization via softmax to obtain class probabilities
+exp_logits=np.exp(logits)
+y_hat=exp_logits/np.sum(exp_logits,axis=1,keepdims=True)
+print(f"y_hat: {y_hat.shape}")
+print(f"sample: {y_hat[0]}")
+
Step 3: Compare the predictions \(\hat{y}\) (ex. [0.3, 0.3, 0.4]) with the actual target values \(y\) (ex. class 2 would look like [0, 0, 1]) with the objective (cost) function to determine loss \(J\). A common objective function for logistics regression is cross-entropy loss.
bias term (\(b\)) excluded to avoid crowding the notations
+
1
+2
+3
+4
# Loss
+correct_class_logprobs=-np.log(y_hat[range(len(y_hat)),y_train])
+loss=np.sum(correct_class_logprobs)/len(y_train)
+print(f"loss: {loss:.2f}")
+
+
+loss: 0.69
+
+
+
Gradients
+
Step 4: Calculate the gradient of loss \(J(\theta)\) w.r.t to the model weights. Let's assume that our classes are mutually exclusive (a set of inputs could only belong to one class).
Step 5: Update the weights \(W\) using a small learning rate \(\alpha\). The updates will penalize the probability for the incorrect classes (j) and encourage a higher probability for the correct class (y).
# Training and test accuracy
+train_acc=np.mean(np.equal(y_train,pred_train))
+test_acc=np.mean(np.equal(y_test,pred_test))
+print(f"train acc: {train_acc:.2f}, test acc: {test_acc:.2f}")
+
In our case, we will also incorporate the class weights into our loss function to counter any class imbalances.
+
1
+2
+3
# Define Loss
+class_weights_tensor=torch.Tensor(list(class_weights.values()))
+loss_fn=nn.CrossEntropyLoss(weight=class_weights_tensor)
+
+
Metrics
+
We'll compute accuracy as we train our model because just looking the loss value isn't super intuitive to look at. We'll look at other metrics (precision, recall, f1) in the evaluation section below.
+
# Accuracy (could've also used our own accuracy function)
+train_acc=accuracy_score(y_train,pred_train)
+test_acc=accuracy_score(y_test,pred_test)
+print(f"train acc: {train_acc:.2f}, test acc: {test_acc:.2f}")
+
+
+train acc: 0.98, test acc: 0.98
+
+
+
We can also evaluate our model on other meaningful metrics such as precision and recall. These are especially useful when there is data imbalance present.
# Predict
+y_infer=F.softmax(model(torch.Tensor(X_infer)),dim=1)
+prob,_class=y_infer.max(dim=1)
+label=label_encoder.decode(_class.detach().numpy())[0]
+print(f"The probability that you have a {label} tumor is {prob.detach().numpy()[0]*100.0:.0f}%")
+
+
+The probability that you have a benign tumor is 93%
+
@article{madewithml,
+author={Goku Mohandas},
+title={ Logistic regression - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
Our goal is to learn a model \(\hat{y}\) that models \(y\) given \(X\) . You'll notice that neural networks are just extensions of the generalized linear methods we've seen so far but with non-linear activation functions since our data will be highly non-linear.
+
+
+
+
+
\[ z_1 = XW_1 \]
+
\[ a_1 = f(z_1) \]
+
\[ z_2 = a_1W_2 \]
+
\[ \hat{y} = softmax(z_2) \]
+
+
+
+
+
Variable
+
Description
+
+
+
+
+
\(N\)
+
total numbers of samples
+
+
+
\(D\)
+
number of features
+
+
+
\(H\)
+
number of hidden units
+
+
+
\(C\)
+
number of classes
+
+
+
\(W_1\)
+
1st layer weights \(\in \mathbb{R}^{DXH}\)
+
+
+
\(z_1\)
+
outputs from first layer \(\in \mathbb{R}^{NXH}\)
+
+
+
\(f\)
+
non-linear activation function
+
+
+
\(a_1\)
+
activations from first layer \(\in \mathbb{R}^{NXH}\)
+
+
+
\(W_2\)
+
2nd layer weights \(\in \mathbb{R}^{HXC}\)
+
+
+
\(z_2\)
+
outputs from second layer \(\in \mathbb{R}^{NXC}\)
+
+
+
\(\hat{y}\)
+
prediction \(\in \mathbb{R}^{NXC}\)
+
+
+
+
(*) bias term (\(b\)) excluded to avoid crowding the notations
+
+
+
Objective:
+
Predict the probability of class \(y\) given the inputs \(X\). Non-linearity is introduced to model the complex, non-linear data.
+
+
+
Advantages:
+
Can model non-linear patterns in the data really well.
+
+
+
Disadvantages:
+
Overfits easily.
+
Computationally intensive as network increases in size.
+
Not easily interpretable.
+
+
+
Miscellaneous:
+
Future neural network architectures that we'll see use the MLP as a modular unit for feed forward operations (affine transformation (XW) followed by a non-linear operation).
+
+
+
+
Set up
+
We'll set our seeds for reproducibility.
+
1
+2
importnumpyasnp
+importrandom
+
+
1
SEED=1234
+
+
1
+2
+3
# Set seed for reproducibility
+np.random.seed(SEED)
+random.seed(SEED)
+
+
Load data
+
I created some non-linearly separable spiral data so let's go ahead and download it for our classification task.
+
1
+2
importmatplotlib.pyplotasplt
+importpandasaspd
+
+
1
+2
+3
+4
+5
# Load data
+url="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/spiral.csv"
+df=pd.read_csv(url,header=0)# load
+df=df.sample(frac=1).reset_index(drop=True)# shuffle
+df.head()
+
+
+
+
+
+
+
X1
+
X2
+
color
+
+
+
+
+
0
+
0.106737
+
0.114197
+
c1
+
+
+
1
+
0.311513
+
-0.664028
+
c1
+
+
+
2
+
0.019870
+
-0.703126
+
c1
+
+
+
3
+
-0.054017
+
0.508159
+
c3
+
+
+
4
+
-0.127751
+
-0.011382
+
c3
+
+
+
+
+
+
1
+2
+3
+4
+5
# Data shapes
+X=df[["X1","X2"]].values
+y=df["color"].values
+print("X: ",np.shape(X))
+print("y: ",np.shape(y))
+
+
+X: (1500, 2)
+y: (1500,)
+
+
1
+2
+3
+4
+5
# Visualize data
+plt.title("Generated non-linear data")
+colors={"c1":"red","c2":"yellow","c3":"blue"}
+plt.scatter(X[:,0],X[:,1],c=[colors[_y]for_yiny],edgecolors="k",s=25)
+plt.show()
+
+
+
+
+
+
Split data
+
We'll shuffle our dataset (since it's ordered by class) and then create our data splits (stratified on class).
+
In the previous lesson we wrote our own label encoder class to see the inner functions but this time we'll use scikit-learn LabelEncoder class which does the same operations as ours.
+
We need to standardize our data (zero mean and unit variance) so a specific feature's magnitude doesn't affect how the model learns its weights. We're only going to standardize the inputs X because our outputs y are class values.
+
1
fromsklearn.preprocessingimportStandardScaler
+
+
1
+2
# Standardize the data (mean=0, std=1) using training data
+X_scaler=StandardScaler().fit(X_train)
+
+
1
+2
+3
+4
# Apply scaler on training and test data (don't standardize outputs for classification)
+X_train=X_scaler.transform(X_train)
+X_val=X_scaler.transform(X_val)
+X_test=X_scaler.transform(X_test)
+
+
1
+2
+3
# Check (means should be ~0 and std should be ~1)
+print(f"X_test[0]: mean: {np.mean(X_test[:,0],axis=0):.1f}, std: {np.std(X_test[:,0],axis=0):.1f}")
+print(f"X_test[1]: mean: {np.mean(X_test[:,1],axis=0):.1f}, std: {np.std(X_test[:,1],axis=0):.1f}")
+
Before we get to our neural network, we're going to motivate non-linear activation functions by implementing a generalized linear model (logistic regression). We'll see why linear models (with linear activations) won't suffice for our dataset.
+
1
importtorch
+
+
1
+2
# Set seed for reproducibility
+torch.manual_seed(SEED)
+
+
Model
+
We'll create our linear model using one layer of weights.
+
1
+2
fromtorchimportnn
+importtorch.nn.functionalasF
+
+
1
+2
+3
INPUT_DIM=X_train.shape[1]# X is 2-dimensional
+HIDDEN_DIM=100
+NUM_CLASSES=len(classes)# 3 classes
+
Using the generalized linear method (logistic regression) yielded poor results because of the non-linearity present in our data yet our activation functions were linear. We need to use an activation function that can allow our model to learn and map the non-linearity in our data. There are many different options so let's explore a few.
# Fig size
+plt.figure(figsize=(12,3))
+
+# Data
+x=torch.arange(-5.,5.,0.1)
+
+# Sigmoid activation (constrain a value between 0 and 1.)
+plt.subplot(1,3,1)
+plt.title("Sigmoid activation")
+y=torch.sigmoid(x)
+plt.plot(x.numpy(),y.numpy())
+
+# Tanh activation (constrain a value between -1 and 1.)
+plt.subplot(1,3,2)
+y=torch.tanh(x)
+plt.title("Tanh activation")
+plt.plot(x.numpy(),y.numpy())
+
+# Relu (clip the negative values to 0)
+plt.subplot(1,3,3)
+y=F.relu(x)
+plt.title("ReLU activation")
+plt.plot(x.numpy(),y.numpy())
+
+# Show plots
+plt.show()
+
+
+
+
+
+
The ReLU activation function (\(max(0,z)\)) is by far the most widely used activation function for neural networks. But as you can see, each activation function has its own constraints so there are circumstances where you'll want to use different ones. For example, if we need to constrain our outputs between 0 and 1, then the sigmoid activation is the best choice.
+
+
In some cases, using a ReLU activation function may not be sufficient. For instance, when the outputs from our neurons are mostly negative, the activation function will produce zeros. This effectively creates a "dying ReLU" and a recovery is unlikely. To mitigate this effect, we could lower the learning rate or use alternative ReLU activations, ex. leaky ReLU or parametric ReLU (PReLU), which have a small slope for negative neuron outputs.
+
+
NumPy
+
Now let's create our multilayer perceptron (MLP) which is going to be exactly like the logistic regression model but with the activation function to map the non-linearity in our data.
+
+
It's normal to find the math and code in this section slightly complex. You can still read each of the steps to build intuition for when we implement this using PyTorch.
+
+
Our goal is to learn a model \(\hat{y}\) that models \(y\) given \(X\). You'll notice that neural networks are just extensions of the generalized linear methods we've seen so far but with non-linear activation functions since our data will be highly non-linear.
+
\[ z_1 = XW_1 \]
+
\[ a_1 = f(z_1) \]
+
\[ z_2 = a_1W_2 \]
+
\[ \hat{y} = softmax(z_2) \]
+
Initialize weights
+
Step 1: Randomly initialize the model's weights \(W\) (we'll cover more effective initialization strategies later in this lesson).
+
We'll apply the softmax function to normalize the logits and obtain class probabilities.
+
\[ \hat{y} = softmax(z_2) \]
+
1
+2
+3
+4
+5
# Normalization via softmax to obtain class probabilities
+exp_logits=np.exp(logits)
+y_hat=exp_logits/np.sum(exp_logits,axis=1,keepdims=True)
+print(f"y_hat: {y_hat.shape}")
+print(f"sample: {y_hat[0]}")
+
Step 3: Compare the predictions \(\hat{y}\) (ex. [0.3, 0.3, 0.4]) with the actual target values \(y\) (ex. class 2 would look like [0, 0, 1]) with the objective (cost) function to determine loss \(J\). A common objective function for classification tasks is cross-entropy loss.
Step 5: Update the weights \(W\) using a small learning rate \(\alpha\). The updates will penalize the probability for the incorrect classes (\(j\)) and encourage a higher probability for the correct class (\(y\)).
# Predict
+y_infer=F.softmax(model(torch.Tensor(X_infer)),dim=1)
+prob,_class=y_infer.max(dim=1)
+label=label_encoder.inverse_transform(_class.detach().numpy())[0]
+print(f"The probability that you have {label} is {prob.detach().numpy()[0]*100.0:.0f}%")
+
+
+The probability that you have c1 is 92%
+
+
+
Initializing weights
+
So far we have been initializing weights with small random values but this isn't optimal for convergence during training. The objective is to initialize the appropriate weights such that our activations (outputs of layers) don't vanish (too small) or explode (too large), as either of these situations will hinder convergence. We can do this by sampling the weights uniformly from a bound distribution (many that take into account the precise activation function used) such that all activations have unit variance.
+
+
You may be wondering why we don't do this for every forward pass and that's a great question. We'll look at more advanced strategies that help with optimization like batch normalization, etc. in future lessons. Meanwhile you can check out other initializers here.
A great technique to have our models generalize (perform well on test data) is to increase the size of your data but this isn't always an option. Fortunately, there are methods like regularization and dropout that can help create a more robust model.
+
Dropout is a technique (used only during training) that allows us to zero the outputs of neurons. We do this for dropout_p% of the total neurons in each layer and it changes every batch. Dropout prevents units from co-adapting too much to the data and acts as a sampling strategy since we drop a different set of neurons each time.
Though neural networks are great at capturing non-linear relationships they are highly susceptible to overfitting to the training data and failing to generalize on test data. Just take a look at the example below where we generate completely random data and are able to fit a model with \(2*N*C + D\) (where N = # of samples, C = # of classes and D = input dimension) hidden units. The training performance is good (~70%) but the overfitting leads to very poor test performance. We'll be covering strategies to tackle overfitting in future lessons.
+
1
+2
+3
+4
NUM_EPOCHS=500
+NUM_SAMPLES_PER_CLASS=50
+LEARNING_RATE=1e-1
+HIDDEN_DIM=2*NUM_SAMPLES_PER_CLASS*NUM_CLASSES+INPUT_DIM# 2*N*C + D
+
+
1
+2
+3
+4
+5
# Generate random data
+X=np.random.rand(NUM_SAMPLES_PER_CLASS*NUM_CLASSES,INPUT_DIM)
+y=np.array([[i]*NUM_SAMPLES_PER_CLASSforiinrange(NUM_CLASSES)]).reshape(-1)
+print("X: ",format(np.shape(X)))
+print("y: ",format(np.shape(y)))
+
# Standardize the inputs (mean=0, std=1) using training data
+X_scaler=StandardScaler().fit(X_train)
+X_train=X_scaler.transform(X_train)
+X_val=X_scaler.transform(X_val)
+X_test=X_scaler.transform(X_test)
+
+
1
+2
+3
+4
+5
+6
+7
# Convert data to tensors
+X_train=torch.Tensor(X_train)
+y_train=torch.LongTensor(y_train)
+X_val=torch.Tensor(X_val)
+y_val=torch.LongTensor(y_val)
+X_test=torch.Tensor(X_test)
+y_test=torch.LongTensor(y_test)
+
+
1
+2
+3
+4
# Initialize model
+model=MLP(input_dim=INPUT_DIM,hidden_dim=HIDDEN_DIM,
+ dropout_p=DROPOUT_P,num_classes=NUM_CLASSES)
+print(model.named_parameters)
+
It's important that we experiment, starting with simple models that underfit (high bias) and improve it towards a good fit. Starting with simple models (linear/logistic regression) let's us catch errors without the added complexity of more sophisticated models (neural networks).
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Neural networks - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Set up
+
+
Click on this link to open the accompanying notebook for this lesson or create a blank one on Google Colab.
+
Sign into your Google account to start using the notebook. If you don't want to save your work, you can skip the steps below. If you do not have access to Google, you can follow along using Jupyter Lab.
+
If you do want to save your work, click the COPY TO DRIVE button on the toolbar. This will open a new notebook in a new tab. Rename this new notebook by removing the words Copy of from the title (change Copy of 01_Notebooks to 1_Notebooks).
+
+
+
+
+
+
+
+
Alternatives to Google Colab
+
Alternatively, you can run these notebooks locally by using JupyterLab. You should first set up a directory for our project, create a virtual environment and install jupyterlab.
Notebooks are made up of cells. There are two types of cells:
+
+
code cell: used for writing and executing code.
+
text cell: used for writing text, HTML, Markdown, etc.
+
+
Text cells
+
Click on a desired location in the notebook and create the cell by clicking on the ➕ TEXT (located in the top left corner).
+
+
+
+
+
Once you create the cell, click on it and type the following text inside it:
+
### This is a header
+Helloworld!
+
+
Run a cell
+
Once you type inside the cell, press the SHIFT and RETURN (enter key) together to run the cell.
+
Edit a cell
+
To edit a cell, double click on it and make any changes.
+
Move a cell
+
Move a cell up and down by clicking on the cell and then pressing the ⬆ and ⬇ button on the top right of the cell.
+
+
+
+
+
Delete a cell
+
Delete the cell by clicking on it and pressing the trash can button 🗑️ on the top right corner of the cell. Alternatively, you can also press ⌘/Ctrl + M + D.
+
+
+
+
+
Code cells
+
Repeat the steps above to create and edit a code cell. You can create a code cell by clicking on the ➕ CODE (located in the top left corner).
+
+
+
+
Once you've created the code cell, double click on it, type the following inside it and then press Shift + Enter to execute the code.
+
1
print("Hello world!")
+
+
+Hello world!
+
+
+
These are the basic concepts we'll need to use these notebooks but we'll learn few more tricks in subsequent lessons.
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Notebooks - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Set up
+
First we'll import the NumPy package and set seeds for reproducibility so that we can receive the exact same results every time.
+
1
importnumpyasnp
+
+
1
+2
# Set seed for reproducibility
+np.random.seed(seed=1234)
+
+
+
+
Basics
+
+
+
+
+
1
+2
+3
+4
+5
+6
+7
# Scalar
+x=np.array(6)
+print("x: ",x)
+print("x ndim: ",x.ndim)# number of dimensions
+print("x shape:",x.shape)# dimensions
+print("x size: ",x.size)# size of elements
+print("x dtype: ",x.dtype)# data type
+
We can extract specific values from our tensors using indexing.
+
+
Keep in mind that when indexing the row and column, indices start at 0. And like indexing with lists, we can use negative indices as well (where -1 is the last item).
# Basic math
+x=np.array([[1,2],[3,4]],dtype=np.float64)
+y=np.array([[1,2],[3,4]],dtype=np.float64)
+print("x + y:\n",np.add(x,y))# or x + y
+print("x - y:\n",np.subtract(x,y))# or x - y
+print("x * y:\n",np.multiply(x,y))# or x * y
+
One of the most common NumPy operations we’ll use in machine learning is matrix multiplication using the dot product. Suppose we wanted to take the dot product of two matrices with shapes [2 X 3] and [3 X 2]. We take the rows of our first matrix (2) and the columns of our second matrix (2) to determine the dot product, giving us an output of [2 X 2]. The only requirement is that the inside dimensions match, in this case the first matrix has 3 columns and the second matrix has 3 rows.
+
+
+
+
+
1
+2
+3
+4
+5
+6
# Dot product
+a=np.array([[1,2,3],[4,5,6]],dtype=np.float64)# we can specify dtype
+b=np.array([[7,8],[9,10],[11,12]],dtype=np.float64)
+c=a.dot(b)
+print(f"{a.shape} · {b.shape} = {c.shape}")
+print(c)
+
# Sum across a dimension
+x=np.array([[1,2],[3,4]])
+print(x)
+print("sum all: ",np.sum(x))# adds all elements
+print("sum axis=0: ",np.sum(x,axis=0))# sum across rows
+print("sum axis=1: ",np.sum(x,axis=1))# sum across columns
+
What happens when we try to do operations with tensors with seemingly incompatible shapes? Their dimensions aren’t compatible as is but how does NumPy still gives us the right result? This is where broadcasting comes in. The scalar is broadcast across the vector so that they have compatible shapes.
This kind of unintended broadcasting happens more often then you'd think because this is exactly what happens when we create an array from a list. So we need to ensure that we apply the proper reshaping before using it for any operations.
We often need to change the dimensions of our tensors for operations like the dot product. If we need to switch two dimensions, we can transpose
+the tensor.
+
+
+
+
+
1
+2
+3
+4
+5
+6
+7
# Transposing
+x=np.array([[1,2,3],[4,5,6]])
+print("x:\n",x)
+print("x.shape: ",x.shape)
+y=np.transpose(x,(1,0))# flip dimensions at index 0 and 1
+print("y:\n",y)
+print("y.shape: ",y.shape)
+
Sometimes, we'll need to alter the dimensions of the matrix. Reshaping allows us to transform a tensor into different permissible shapes. Below, our reshaped tensor has the same number of values as the original tensor. (1X6 = 2X3). We can also use -1 on a dimension and NumPy will infer the dimension based on our input tensor.
The way reshape works is by looking at each dimension of the new tensor and separating our original tensor into that many units. So here the dimension at index 0 of the new tensor is 2 so we divide our original tensor into 2 units, and each of those has 3 values.
+
+
+
+
+
+
Unintended reshaping
+
Though reshaping is very convenient to manipulate tensors, we must be careful of its pitfalls as well. Let's look at the example below. Suppose we have x, which has the shape [2 X 3 X 4].
Instead, if we transpose the tensor and then do a reshape, we get our desired tensor. Transpose allows us to put our two vectors that we want to combine together and then we use reshape to join them together. And as a general rule, we should always get our dimensions together before reshaping to combine them.
This becomes difficult when we're dealing with weight tensors with random values in many machine learning tasks. So a good idea is to always create a dummy example like this when you’re unsure about reshaping. Blindly going by the tensor shape can lead to lots of issues downstream.
We can also easily add and remove dimensions to our tensors and we'll want to do this to make tensors compatible for certain operations.
+
1
+2
+3
+4
+5
+6
+7
# Adding dimensions
+x=np.array([[1,2,3],[4,5,6]])
+print("x:\n",x)
+print("x.shape: ",x.shape)
+y=np.expand_dims(x,1)# expand dim 1
+print("y: \n",y)
+print("y.shape: ",y.shape)# notice extra set of brackets are added
+
# Removing dimensions
+x=np.array([[[1,2,3]],[[4,5,6]]])
+print("x:\n",x)
+print("x.shape: ",x.shape)
+y=np.squeeze(x,1)# squeeze dim 1
+print("y: \n",y)
+print("y.shape: ",y.shape)# notice extra set of brackets are gone
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Set up
+
First we'll import the NumPy and Pandas libraries and set seeds for reproducibility. We'll also download the dataset we'll be working with to disk.
+
1
+2
importnumpyasnp
+importpandasaspd
+
+
1
+2
# Set seed for reproducibility
+np.random.seed(seed=1234)
+
+
+
+
Load data
+
We're going to work with the Titanic dataset which has data on the people who embarked the RMS Titanic in 1912 and whether they survived the expedition or not. It's a very common and rich dataset which makes it very apt for exploratory data analysis with Pandas.
+
Let's load the data from the CSV file into a Pandas dataframe. The header=0 signifies that the first row (0th index) is a header row which contains the names of each column in our dataset.
+
1
+2
+3
# Read from CSV to Pandas DataFrame
+url="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/titanic.csv"
+df=pd.read_csv(url,header=0)
+
We can also get statistics across our features for certain groups. Here we wan to see the average of our continuous features based on whether the passenger survived or not.
+
After exploring, we can clean and preprocess our dataset.
+
+
Be sure to check out our entire lesson focused on preprocessing in our MLOps course.
+
+
1
+2
# Rows with at least one NaN value
+df[pd.isnull(df).any(axis=1)].head()
+
+
+
+
+
+
+
pclass
+
name
+
sex
+
age
+
sibsp
+
parch
+
ticket
+
fare
+
cabin
+
embarked
+
survived
+
+
+
+
+
9
+
1
+
Artagaveytia, Mr. Ramon
+
male
+
71.0
+
0
+
0
+
PC 17609
+
49.5042
+
NaN
+
C
+
0
+
+
+
13
+
1
+
Barber, Miss. Ellen "Nellie"
+
female
+
26.0
+
0
+
0
+
19877
+
78.8500
+
NaN
+
S
+
1
+
+
+
15
+
1
+
Baumann, Mr. John D
+
male
+
NaN
+
0
+
0
+
PC 17318
+
25.9250
+
NaN
+
S
+
0
+
+
+
23
+
1
+
Bidois, Miss. Rosalie
+
female
+
42.0
+
0
+
0
+
PC 17757
+
227.5250
+
NaN
+
C
+
1
+
+
+
25
+
1
+
Birnbaum, Mr. Jakob
+
male
+
25.0
+
0
+
0
+
13905
+
26.0000
+
NaN
+
C
+
0
+
+
+
+
+
+
1
+2
+3
+4
# Drop rows with Nan values
+df=df.dropna()# removes rows with any NaN values
+df=df.reset_index()# reset's row indexes in case any rows were dropped
+df.head()
+
+
+
+
+
+
+
index
+
pclass
+
name
+
sex
+
age
+
sibsp
+
parch
+
ticket
+
fare
+
cabin
+
embarked
+
survived
+
+
+
+
+
0
+
0
+
1
+
Allen, Miss. Elisabeth Walton
+
female
+
29.0000
+
0
+
0
+
24160
+
211.3375
+
B5
+
S
+
1
+
+
+
1
+
1
+
1
+
Allison, Master. Hudson Trevor
+
male
+
0.9167
+
1
+
2
+
113781
+
151.5500
+
C22 C26
+
S
+
1
+
+
+
2
+
2
+
1
+
Allison, Miss. Helen Loraine
+
female
+
2.0000
+
1
+
2
+
113781
+
151.5500
+
C22 C26
+
S
+
0
+
+
+
3
+
3
+
1
+
Allison, Mr. Hudson Joshua Creighton
+
male
+
30.0000
+
1
+
2
+
113781
+
151.5500
+
C22 C26
+
S
+
0
+
+
+
4
+
4
+
1
+
Allison, Mrs. Hudson J C (Bessie Waldo Daniels)
+
female
+
25.0000
+
1
+
2
+
113781
+
151.5500
+
C22 C26
+
S
+
0
+
+
+
+
+
+
1
+2
+3
# Dropping multiple columns
+df=df.drop(["name","cabin","ticket"],axis=1)# we won't use text features for our initial basic models
+df.head()
+
We're now going to use feature engineering to create a column called family_size. We'll first define a function called get_family_size that will determine the family size using the number of parents and siblings.
+
1
+2
+3
+4
# Lambda expressions to create new features
+defget_family_size(sibsp,parch):
+ family_size=sibsp+parch
+ returnfamily_size
+
+Once we define the function, we can use lambda to apply that function on each row (using the numbers of siblings and parents in each row to determine the family size for each row).
+
Feature engineering can be done in collaboration with domain experts that can guide us on what features to engineer and use.
+
+
Save data
+
Finally, let's save our preprocessed data into a new CSV file to use later.
+
1
+2
# Saving dataframe to CSV
+df.to_csv("processed_titanic.csv",index=False)
+
+
1
+2
# See the saved file
+!ls-l
+
+
+total 96
+-rw-r--r-- 1 root root 6975 Dec 3 17:36 processed_titanic.csv
+drwxr-xr-x 1 root root 4096 Nov 21 16:30 sample_data
+-rw-r--r-- 1 root root 85153 Dec 3 17:36 titanic.csv
+
+
+
Scaling
+
When working with very large datasets, our Pandas DataFrames can become very large and it can be very slow or impossible to operate on them. This is where packages that can distribute workloads or run on more efficient hardware can come in handy.
+
+
Dask: parallel computing to scale packages like Numpy, Pandas and scikit-learn on one/multiple machines.
+
cuDF: efficient dataframe loading and computation on a GPU.
+
+
And, of course, we can combine these together (Dask-cuDF) to operate on partitions of a dataframe on the GPU.
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Pandas - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
Here we use the variable name x in our examples but when you're working on a specific task, be sure to be explicit (ex. first_name) when creating variables (applies to functions, classes, etc. as well).
+
+
We can change the value of a variable by simply assigning a new value to it.
There are many different types of variables: integers, floats, strings, boolean etc.
+
1
+2
+3
# int variable
+x=5
+print(x,type(x))
+
+
+5 <class 'int'>
+
+
1
+2
+3
# float variable
+x=5.0
+print(x,type(x))
+
+
+5.0 <class 'float'>
+
+
1
+2
+3
# text variable
+x="5"
+print(x,type(x))
+
+
+5 <class 'str'>
+
+
1
+2
+3
# boolean variable
+x=True
+print(x,type(x))
+
+
+True <class 'bool'>
+
+
We can also do operations with variables:
+
1
+2
+3
+4
+5
# Variables can be used with each other
+a=1
+b=2
+c=a+b
+print(c)
+
+
+3
+
+
+
+
Know your types!
+
We should always know what types of variables we're dealing with so we can do the right operations with them. Here's a common mistake that can happen if we're using the wrong variable type.
+
1
+2
+3
+4
# int variables
+a=5
+b=3
+print(a+b)
+
+
+Show answer
+
+8
+
+
+
1
+2
+3
+4
# string variables
+a="5"
+b="3"
+print(a+b)
+
+
+Show answer
+
+53
+
+
+
+
Lists
+
Lists are an ordered, mutable (changeable) collection of values that are comma separated and enclosed by square brackets. A list can be comprised of many different types of variables. Below is a list with an integer, string and a float:
+
1
+2
+3
# Creating a list
+x=[3,"hello",1.2]
+print(x)
+
+
+[3, 'hello', 1.2]
+
+
1
+2
# Length of a list
+len(x)
+
+
+3
+
+
We can add to a list by using the append function:
+
1
+2
+3
+4
# Adding to a list
+x.append(7)
+print(x)
+print(len(x))
+
+
+[3, 'hello', 1.2, 7]
+4
+
+
and just as easily replace existing items:
+
1
+2
+3
# Replacing items in a list
+x[1]="bye"
+print(x)
+
+
+[3, 'bye', 1.2, 7]
+
+
and perform operations with lists:
+
1
+2
+3
+4
# Operations
+y=[2.4,"world"]
+z=x+y
+print(z)
+
+
+[3, 'bye', 1.2, 7, 2.4, 'world']
+
+
+
Tuples
+
Tuples are collections that are ordered and immutable (unchangeable). We will use tuples to store values that will never be changed.
+
1
+2
+3
# Creating a tuple
+x=(3.0,"hello")# tuples start and end with ()
+print(x)
+
+
+(3.0, 'hello')
+
+
+
1
+2
+3
# Adding values to a tuple
+x=x+(5.6,4)
+print(x)
+
+
+(3.0, 'hello', 5.6, 4)
+
+
+
1
+2
# Try to change (it won't work and we get an error)
+x[0]=1.2
+
+
+---------------------------------------------------------------------------
+TypeError Traceback (most recent call last)
+----> 1 x[0] = 1.2
+TypeError: 'tuple' object does not support item assignment
+
+
+
Sets
+
Sets are collections that are unordered and mutable. However, every item in a set much be unique.
+
1
+2
+3
+4
# Sets
+text="Learn ML with Made With ML"
+print(set(text))
+print(set(text.split(" ")))
+
Indexing and slicing from lists allow us to retrieve specific values within lists. Note that indices can be positive (starting from 0) or negative (-1 and lower, where -1 is the last item in the list).
+
+
+
+
+
1
+2
+3
+4
+5
+6
# Indexing
+x=[3,"hello",1.2]
+print("x[0]: ",x[0])
+print("x[1]: ",x[1])
+print("x[-1]: ",x[-1])# the last item
+print("x[-2]: ",x[-2])# the second to last item
+
+
+x[0]: 3
+x[1]: hello
+x[-1]: 1.2
+x[-2]: hello
+
+
+
1
+2
+3
+4
+5
# Slicing
+print("x[:]: ",x[:])# all indices
+print("x[1:]: ",x[1:])# index 1 to the end of the list
+print("x[1:2]: ",x[1:2])# index 1 to index 2 (not including index 2)
+print("x[:-1]: ",x[:-1])# index 0 to last index (not including last index)
+
+Though this does produce results, we should always explicitly use the length of the list to index items from it to avoid incorrect assumptions for downstream processes.
+
+
+
Dictionaries
+
Dictionaries are an unordered, mutable collection of key-value pairs. You can retrieve values based on the key and a dictionary cannot have two of the same keys.
See if you can recall and sort out the similarities and differences of the foundational data structures we've seen so far.
+
+
+
+
+
Mutable
+
Ordered
+
Indexable
+
Unique
+
+
+
+
+
List
+
❓
+
❓
+
❓
+
❓
+
+
+
Tuple
+
❓
+
❓
+
❓
+
❓
+
+
+
Set
+
❓
+
❓
+
❓
+
❓
+
+
+
Dictionary
+
❓
+
❓
+
❓
+
❓
+
+
+
+
+Show answer
+
+
+
+
+
Mutable
+
Ordered
+
Indexable
+
Unique
+
+
+
+
+
List
+
✅
+
✅
+
✅
+
❌
+
+
+
Tuple
+
❌
+
✅
+
✅
+
❌
+
+
+
Set
+
✅
+
❌
+
❌
+
✅
+
+
+
Dictionary
+
✅
+
❌
+
❌
+
✅ keys ❌ values
+
+
+
+
+
+
But of course, there is pretty much a way to do accomplish anything with Python. For example, even though native dictionaries are unordered, we can leverage the OrderedDict data structure to change that (useful if we want to iterate through keys in a certain order, etc.).
After Python 3.7+, native dictionaries are insertion ordered.
+
+
1
+2
# Dictionary items
+print(d.items())
+
+
+dict_items([('a', 2), ('c', 3), ('b', 1)])
+
+
1
+2
# Order by keys
+print(OrderedDict(sorted(d.items())))
+
+
+OrderedDict([('a', 2), ('b', 1), ('c', 3)])
+
+
1
+2
# Order by values
+print(OrderedDict(sorted(d.items(),key=lambdax:x[1])))
+
+
+OrderedDict([('b', 1), ('a', 2), ('c', 3)])
+
+
+
If statements
+
We can use if statements to conditionally do something. The conditions are defined by the words if, elif (which stands for else if) and else. We can have as many elif statements as we want. The indented code below each condition is the code that will execute if the condition is True.
+
1
+2
+3
+4
+5
+6
+7
+8
+9
# If statement
+x=4
+ifx<1:
+ score="low"
+elifx<=4:# elif = else if
+ score="medium"
+else:
+ score="high"
+print(score)
+
+
+medium
+
+
+
1
+2
+3
+4
# If statement with a boolean
+x=True
+ifx:
+ print("it worked")
+
+
+it worked
+
+
+
Loops
+
For loops
+
A for loop can iterate over a collection of values (lists, tuples, dictionaries, etc.) The indented code is executed for each item in the collection of values.
+
1
+2
+3
+4
# For loop
+veggies=["carrots","broccoli","beans"]
+forveggieinveggies:
+ print(veggie)
+
+
+carrots
+broccoli
+beans
+
+
+
When the loop encounters the break command, the loop will terminate immediately. If there were more items in the list, they will not be processed.
+
1
+2
+3
+4
+5
+6
# `break` from a for loop
+veggies=["carrots","broccoli","beans"]
+forveggieinveggies:
+ ifveggie=="broccoli":
+ break
+ print(veggie)
+
+
+carrots
+
+
+
When the loop encounters the continue command, the loop will skip all other operations for that item in the list only. If there were more items in the list, the loop will continue normally.
+
1
+2
+3
+4
+5
+6
# `continue` to the next iteration
+veggies=["carrots","broccoli","beans"]
+forveggieinveggies:
+ ifveggie=="broccoli":
+ continue
+ print(veggie)
+
+
+carrots
+beans
+
+
+
While loops
+
A while loop can perform repeatedly as long as a condition is True. We can use continue and break commands in while loops as well.
+
1
+2
+3
+4
+5
# While loop
+x=3
+whilex>0:
+ x-=1# same as x = x - 1
+ print(x)
+
+
+2
+1
+0
+
+
+
List comprehension
+
We can combine our knowledge of lists and for loops to leverage list comprehensions to create succinct code.
Python syntax is usually very straight forward, so the correct answer involves just directly copying the statements from the nested for loop from top to bottom!
Functions are a way to modularize reusable pieces of code. They're defined by the keyword def which stands for definition and they can have the following components.
+
+
+
+
+
1
+2
+3
+4
+5
# Define the function
+defadd_two(x):
+"""Increase x by 2."""
+ x+=2
+ returnx
+
+
Here are the components that may be required when we want to use the function. we need to ensure that the function name and the input parameters match with how we defined the function above.
+
+
+
+
+
1
+2
+3
+4
# Use the function
+score=0
+new_score=add_two(x=score)
+print(new_score)
+
+
+2
+
+
+
A function can have as many input parameters and outputs as we want.
+
1
+2
+3
+4
+5
# Function with multiple inputs
+defjoin_name(first_name,last_name):
+"""Combine first name and last name."""
+ joined_name=first_name+" "+last_name
+ returnjoined_name
+
+
1
+2
+3
+4
+5
+6
# Use the function
+first_name="Goku"
+last_name="Mohandas"
+joined_name=join_name(
+ first_name=first_name,last_name=last_name)
+print(joined_name)
+
+
+Goku Mohandas
+
+
+
+
We can be even more explicit with our function definitions by specifying the types of our input and output arguments. We cover this in our documentation lesson because the typing information is automatically leveraged to create very intuitive documentation.
+
+
It's good practice to always use keyword argument when using a function so that it's very clear what input variable belongs to what function input parameter. On a related note, you will often see the terms *args and **kwargs which stand for arguments and keyword arguments. You can extract them when they are passed into a function. The significance of the * is that any number of arguments and keyword arguments can be passed into the function.
Classes are object constructors and are a fundamental component of object oriented programming in Python. They are composed of a set of functions that define the class and it's operations.
+
Magic methods
+
Classes can be customized with magic methods like __init__ and __str__, to enable powerful operations. These are also known as dunder methods (ex. dunder init), which stands for double underscores due to the leading and trailing underscores.
+
The __init__ function is used when an instance of the class is initialized.
+
1
+2
+3
+4
+5
+6
+7
+8
# Creating the class
+classPet(object):
+"""Class object for a pet."""
+
+ def__init__(self,species,name):
+"""Initialize a Pet."""
+ self.species=species
+ self.name=name
+
+
1
+2
+3
+4
+5
# Creating an instance of a class
+my_dog=Pet(species="dog",
+ name="Scooby")
+print(my_dog)
+print(my_dog.name)
+
+
+<__main__.Pet object at 0x7fe487e9c358>
+Scooby
+
+
+
The print (my_dog) command printed something not so relevant to us. Let's fix that with the __str__ function.
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
# Creating the class
+# Creating the class
+classPet(object):
+"""Class object for a pet."""
+
+ def__init__(self,species,name):
+"""Initialize a Pet."""
+ self.species=species
+ self.name=name
+
+ def__str__(self):
+"""Output when printing an instance of a Pet."""
+ returnf"{self.species} named {self.name}"
+
+
1
+2
+3
+4
+5
# Creating an instance of a class
+my_dog=Pet(species="dog",
+ name="Scooby")
+print(my_dog)
+print(my_dog.name)
+
+
+dog named Scooby
+Scooby
+
+
+
+
We'll be exploring additional built-in functions in subsequent notebooks (like __len__, __iter__ and __getitem__, etc.) but if you're curious, here is a tutorial on more magic methods.
+
+
Object functions
+
Besides these magic functions, classes can also have object functions.
+
# Creating the class
+classPet(object):
+"""Class object for a pet."""
+
+ def__init__(self,species,name):
+"""Initialize a Pet."""
+ self.species=species
+ self.name=name
+
+ def__str__(self):
+"""Output when printing an instance of a Pet."""
+ returnf"{self.species} named {self.name}"
+
+ defchange_name(self,new_name):
+"""Change the name of your Pet."""
+ self.name=new_name
+
+
1
+2
+3
+4
# Creating an instance of a class
+my_dog=Pet(species="dog",name="Scooby")
+print(my_dog)
+print(my_dog.name)
+
+
+dog named Scooby
+Scooby
+
+
+
1
+2
+3
+4
# Using a class's function
+my_dog.change_name(new_name="Scrappy")
+print(my_dog)
+print(my_dog.name)
+
+
+dog named Scrappy
+Scrappy
+
+
+
Inheritance
+
We can also build classes on top of one another using inheritance, which allows us to inherit all the properties and methods from another class (the parent).
+
Notice how we inherited the initialized variables from the parent Pet class like species and name. We also inherited functions such as change_name().
+
+
Which function is executed?
+
Which function is executed if the parent and child functions have functions with the same name?
+
+Show answer
+
As you can see, both our parent class (Pet) and the child class (Dog) have different __str__ functions defined but share the same function name. The child class inherits everything from the parent classes but when there is conflict between function names, the child class' functions take precedence and overwrite the parent class' functions.
+
+
+
Methods
+
There are two important decorator methods to know about when it comes to classes: @classmethod and @staticmethod. We'll learn about decorators in the next section below but these specific methods pertain to classes so we'll cover them here.
A @classmethod allows us to create class instances by passing in the uninstantiated class itself (cls). This is a great way to create (or load) classes from objects (ie. dictionaries).
Recall that functions allow us to modularize code and reuse them. However, we'll often want to add some functionality before or after the main function executes and we may want to do this for many different functions. Instead of adding more code to the original function, we can use decorators!
+
+
decorators: augment a function with pre/post-processing. Decorators wrap around the main function and allow us to operate on the inputs and or outputs.
+
+
Suppose we have a function called operations which increments the input value x by 1.
+
Now let's say we want to increment our input x by 1 before and after the operations function executes and, to illustrate this example, let's say the increments have to be separate steps. Here's how we would do it by changing the original code:
+
We were able to achieve what we want but we now increased the size of our operations function and if we want to do the same incrementing for any other function, we have to add the same code to all of those as well ... not very efficient. To solve this, let's create a decorator called add which increments x by 1 before and after the main function f executes.
+
Creating a decorator
+
The decorator function accepts a function f which is the function we wish to wrap around, in our case, it's operations(). The output of the decorator is its wrapper function which receives the arguments and keyword arguments passed to function f.
+
Inside the wrapper function, we can:
+
+
extract the input parameters passed to function f.
+
make any changes we want to the function inputs.
+
function f is executed
+
make any changes to the function outputs
+
wrapper function returns some value(s), which is what the decorator returns as well since it returns wrapper.
+
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
# Decorator
+defadd(f):
+ defwrapper(*args,**kwargs):
+"""Wrapper function for @add."""
+ x=kwargs.pop("x")# .get() if not altering x
+ x+=1# executes before function f
+ x=f(*args,**kwargs,x=x)
+ x+=1# executes after function f
+ returnx
+ returnwrapper
+
+
+
We can use this decorator by simply adding it to the top of our main function preceded by the @ symbol.
+
Suppose we wanted to debug and see what function actually executed with operations().
+
1
operations.__name__,operations.__doc__
+
+
+('wrapper', 'Wrapper function for @add.')
+
+
The function name and docstring are not what we're looking for but it appears this way because the wrapper function is what was executed. In order to fix this, Python offers functools.wraps which carries the main function's metadata.
+
Awesome! We were able to decorate our main function operation() to achieve the customization we wanted without actually altering the function. We can reuse our decorator for other functions that may need the same customization!
+
+
This was a dummy example to show how decorators work but we'll be using them heavily during our MLOps lessons. A simple scenario would be using decorators to create uniform JSON responses from each API endpoint without including the bulky code in each endpoint.
+
+
Callbacks
+
Decorators allow for customized operations before and after the main function's execution but what about in between? Suppose we want to conditionally/situationally do some operations. Instead of writing a whole bunch of if-statements and make our functions bulky, we can use callbacks!
+
+
callbacks: conditional/situational processing within the function.
+
+
Our callbacks will be classes that have functions with key names that will execute at various periods during the main function's execution. The function names are up to us but we need to invoke the same callback functions within our main function.
+
It seems like we've just done some operations before and after the function's main process? Isn't that what a decorator is for?
+
+Show answer
+
With callbacks, it's easier to keep track of objects since it's all defined in a separate callback class. It's also now possible to interact with our function, not just before or after but throughout the entire process! Imagine a function with:
+
+
multiple processes where we want to execute operations in between them
+
execute operations repeatedly when loops are involved in functions
+
+
+
+
Putting it all together
+
decorators + callbacks = powerful customization before, during and after the main function’s execution without increasing its complexity. We will be using this duo to create powerful ML training scripts that are highly customizable in future lessons.
+
1
fromfunctoolsimportwraps
+
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
# Decorator
+defadd(f):
+ @wraps(f)
+ defwrap(*args,**kwargs):
+"""Wrapper function for @add."""
+ x=kwargs.pop("x")# .get() if not altering x
+ x+=1# executes before function f
+ x=f(*args,**kwargs,x=x)
+ # can do things post function f as well
+ returnx
+ returnwrap
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Set up
+
We'll import PyTorch and set seeds for reproducibility. Note that PyTorch also required a seed since we will be generating random tensors.
+
1
+2
importnumpyasnp
+importtorch
+
+
1
SEED=1234
+
+
1
+2
+3
# Set seed for reproducibility
+np.random.seed(seed=SEED)
+torch.manual_seed(SEED)
+
+
+
+
+
+
Basics
+
We'll first cover some basics with PyTorch such as creating tensors and converting from common data structures (lists, arrays, etc.) to tensors.
+
1
+2
+3
+4
+5
# Creating a random tensor
+x=torch.randn(2,3)# normal distribution (rand(2,3) -> uniform distribution)
+print(f"Type: {x.type()}")
+print(f"Size: {x.shape}")
+print(f"Values: \n{x}")
+
# Dimensional operations
+x=torch.randn(2,3)
+print(f"Values: \n{x}")
+y=torch.sum(x,dim=0)# add each row's value for every column
+print(f"Values: \n{y}")
+z=torch.sum(x,dim=1)# add each columns's value for every row
+print(f"Values: \n{z}")
+
We can determine gradients (rate of change) of our tensors with respect to their constituents using gradient bookkeeping. The gradient is a vector that points in the direction of greatest increase of a function. We'll be using gradients in the next lesson to determine how to change our weights to affect a particular objective function (ex. loss).
# Tensors with gradient bookkeeping
+x=torch.rand(3,4,requires_grad=True)
+y=3*x+2
+z=y.mean()
+z.backward()# z has to be scalar
+print(f"x: \n{x}")
+print(f"x.grad: \n{x.grad}")
+
We also load our tensors onto the GPU for parallelized computation using CUDA (a parallel computing platform and API from Nvidia).
+
1
+2
# Is CUDA available?
+print(torch.cuda.is_available())
+
+
+False
+
+
+
If False (CUDA is not available), let's change that by following these steps: Go to Runtime > Change runtime type > Change Hardware accelerator to GPU > Click Save
+
1
importtorch
+
+
1
+2
# Is CUDA available now?
+print(torch.cuda.is_available())
+
+
+True
+
+
1
+2
+3
# Set device
+device=torch.device("cuda"iftorch.cuda.is_available()else"cpu")
+print(device)
+
+
+cuda
+
+
1
+2
+3
+4
x=torch.rand(2,3)
+print(x.is_cuda)
+x=torch.rand(2,3).to(device)# Tensor is stored on the GPU
+print(x.is_cuda)
+
+
+False
+True
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ PyTorch - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
So far we've processed inputs as whole (ex. applying filters across the entire input to extract features) but we can also process our inputs sequentially. For example we can think of each token in our text as an event in time (timestep). We can process each timestep, one at a time, and predict the class after the last timestep (token) has been processed. This is very powerful because the model now has a meaningful way to account for the sequential order of tokens in our sequence and predict accordingly.
+
+
+
+
+
$$ \text{RNN forward pass for a single time step } X_t $$:
+
\[ h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h) \]
+
+
+
+
+
Variable
+
Description
+
+
+
+
+
\(N\)
+
batch size
+
+
+
\(E\)
+
embeddings dimension
+
+
+
\(H\)
+
# of hidden units
+
+
+
\(W_{hh}\)
+
RNN weights \(\in \mathbb{R}^{HXH}\)
+
+
+
\(h_{t-1}\)
+
previous timestep's hidden state \(\in in \mathbb{R}^{NXH}\)
+
+
+
\(W_{xh}\)
+
input weights \(\in \mathbb{R}^{EXH}\)
+
+
+
\(X_t\)
+
input at time step \(t \in \mathbb{R}^{NXE}\)
+
+
+
\(b_h\)
+
hidden units bias \(\in \mathbb{R}^{HX1}\)
+
+
+
\(h_t\)
+
output from RNN for timestep \(t\)
+
+
+
+
+
+
Objective:
+
Process sequential data by accounting for the current input and also what has been learned from previous inputs.
+
+
+
Advantages:
+
Account for order and previous inputs in a meaningful way.
+
Conditioned generation for generating sequences.
+
+
+
Disadvantages:
+
Each time step's prediction depends on the previous prediction so it's difficult to parallelize RNN operations.
+
Processing long sequences can yield memory and computation issues.
+
Interpretability is difficult but there are few techniques that use the activations from RNNs to see what parts of the inputs are processed.
+
+
+
Miscellaneous:
+
Architectural tweaks to make RNNs faster and interpretable is an ongoing area of research.
+
+
+
+
Set up
+
Let's set our seed and device for our main task.
+
We will download the AG News dataset, which consists of 120K text samples from 4 unique classes (Business, Sci/Tech, Sports, World)
+
1
+2
+3
+4
+5
# Load data
+url="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/news.csv"
+df=pd.read_csv(url,header=0)# load
+df=df.sample(frac=1).reset_index(drop=True)# shuffle
+df.head()
+
+
+
+
+
+
+
title
+
category
+
+
+
+
+
0
+
Sharon Accepts Plan to Reduce Gaza Army Operation...
+
World
+
+
+
1
+
Internet Key Battleground in Wildlife Crime Fight
+
Sci/Tech
+
+
+
2
+
July Durable Good Orders Rise 1.7 Percent
+
Business
+
+
+
3
+
Growing Signs of a Slowing on Wall Street
+
Business
+
+
+
4
+
The New Faces of Reality TV
+
World
+
+
+
+
+
+
Preprocessing
+
We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc.
+
defpreprocess(text,stopwords=STOPWORDS):
+"""Conditional preprocessing on our text unique to our task."""
+ # Lower
+ text=text.lower()
+
+ # Remove stopwords
+ pattern=re.compile(r"\b("+r"|".join(stopwords)+r")\b\s*")
+ text=pattern.sub("",text)
+
+ # Remove words in parenthesis
+ text=re.sub(r"\([^)]*\)","",text)
+
+ # Spacing and filters
+ text=re.sub(r"([-;;.,!?<=>])",r" \1 ",text)
+ text=re.sub("[^A-Za-z0-9]+"," ",text)# remove non alphanumeric chars
+ text=re.sub(" +"," ",text)# remove multiple spaces
+ text=text.strip()
+
+ returntext
+
+
1
+2
+3
# Sample
+text="Great week for the NYSE!"
+preprocess(text=text)
+
+
+great week nyse
+
+
1
+2
+3
+4
# Apply to dataframe
+preprocessed_df=df.copy()
+preprocessed_df.title=preprocessed_df.title.apply(preprocess)
+print(f"{df.title.values[0]}\n\n{preprocessed_df.title.values[0]}")
+
+
+Sharon Accepts Plan to Reduce Gaza Army Operation, Haaretz Says
+
+sharon accepts plan reduce gaza army operation haaretz says
+
+
+
+
Warning
+
If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before applying those operations. This is because we cannot apply any knowledge gained from the test set accidentally (data leak) during preprocessing/training. However for global preprocessing steps like the function above where we aren't learning anything from the data itself, we can perform before splitting the data.
It's important that we only fit using our train data split because during inference, our model will not always know every token so it's important to replicate that scenario with our validation and test splits as well.
# Sample of tokens
+print(take(5,tokenizer.token_to_index.items()))
+print(f"least freq token's freq: {tokenizer.min_token_freq}")# use this to adjust num_tokens
+
Inputs to RNNs are sequential like text or time-series.
+
1
+2
BATCH_SIZE=64
+EMBEDDING_DIM=100
+
+
1
+2
+3
+4
+5
+6
# Input
+sequence_size=8# words per input
+x=torch.rand((BATCH_SIZE,sequence_size,EMBEDDING_DIM))
+seq_lens=torch.randint(high=sequence_size,size=(BATCH_SIZE,))
+print(x.shape)
+print(seq_lens.shape)
+
+
+torch.Size([64, 8, 100])
+torch.Size([1, 64])
+
+
+
+
+
+
$$ \text{RNN forward pass for a single time step } X_t $$:
+
\[ h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h) \]
+
+
+
+
+
Variable
+
Description
+
+
+
+
+
\(N\)
+
batch size
+
+
+
\(E\)
+
embeddings dimension
+
+
+
\(H\)
+
# of hidden units
+
+
+
\(W_{hh}\)
+
RNN weights \(\in \mathbb{R}^{HXH}\)
+
+
+
\(h_{t-1}\)
+
previous timestep's hidden state \(\in in \mathbb{R}^{NXH}\)
+
+
+
\(W_{xh}\)
+
input weights \(\in \mathbb{R}^{EXH}\)
+
+
+
\(X_t\)
+
input at time step \(t \in \mathbb{R}^{NXE}\)
+
+
+
\(b_h\)
+
hidden units bias \(\in \mathbb{R}^{HX1}\)
+
+
+
\(h_t\)
+
output from RNN for timestep \(t\)
+
+
+
+
+
+
At the first time step, the previous hidden state \(h_{t-1}\) can either be a zero vector (unconditioned) or initialized (conditioned). If we are conditioning the RNN, the first hidden state \(h_0\) can belong to a specific condition or we can concat the specific condition to the randomly initialized hidden vectors at each time step. More on this in the subsequent notebooks on RNNs.
+
+
1
+2
RNN_HIDDEN_DIM=128
+DROPOUT_P=0.1
+
+
1
+2
+3
# Initialize hidden state
+hidden_t=torch.zeros((BATCH_SIZE,RNN_HIDDEN_DIM))
+print(hidden_t.size())
+
+
+torch.Size([64, 128])
+
+
+
We'll show how to create an RNN cell using PyTorch's RNNCell and the more abstracted RNN.
# Forward pass through RNN
+x=x.permute(1,0,2)# RNN needs batch_size to be at dim 1
+
+# Loop through the inputs time steps
+hiddens=[]
+fortinrange(sequence_size):
+ hidden_t=rnn_cell(x[t],hidden_t)
+ hiddens.append(hidden_t)
+hiddens=torch.stack(hiddens)
+hiddens=hiddens.permute(1,0,2)# bring batch_size back to dim 0
+print(hiddens.size())
+
+
+torch.Size([64, 8, 128])
+
+
1
+2
+3
+4
+5
+6
# We also could've used a more abstracted layer
+x=torch.rand((BATCH_SIZE,sequence_size,EMBEDDING_DIM))
+rnn=nn.RNN(EMBEDDING_DIM,RNN_HIDDEN_DIM,batch_first=True)
+out,h_n=rnn(x)# h_n is the last hidden state
+print("out: ",out.shape)
+print("h_n: ",h_n.shape)
+
In our model, we want to use the RNN's output after the last relevant token in the sentence is processed. The last relevant token doesn't refer the <PAD> tokens but to the last actual word in the sentence and its index is different for each input in the batch. This is why we included a seq_lens tensor in our batches.
+
1
+2
+3
+4
+5
+6
+7
+8
defgather_last_relevant_hidden(hiddens,seq_lens):
+"""Extract and collect the last relevant
+ hidden state based on the sequence length."""
+ seq_lens=seq_lens.long().detach().cpu().numpy()-1
+ out=[]
+ forbatch_index,column_indexinenumerate(seq_lens):
+ out.append(hiddens[batch_index,column_index])
+ returntorch.stack(out)
+
+
1
+2
# Get the last relevant hidden state
+gather_last_relevant_hidden(hiddens=out,seq_lens=seq_lens).squeeze(0).shape
+
+
+torch.Size([64, 128])
+
+
+
There are many different ways to use RNNs. So far we've processed our inputs one timestep at a time and we could either use the RNN's output at each time step or just use the final input timestep's RNN output. Let's look at a few other possibilities.
While our simple RNNs so far are great for sequentially processing our inputs, they have quite a few disadvantages. They commonly suffer from exploding or vanishing gradients as a result using the same set of weights (\(W_{xh}\) and \(W_{hh}\)) with each timestep's input. During backpropagation, this can cause gradients to explode (>1) or vanish (<1). If you multiply any number greater than 1 with itself over and over, it moves towards infinity (exploding gradients) and similarly, If you multiply any number less than 1 with itself over and over, it moves towards zero (vanishing gradients). To mitigate this issue, gated RNNs were devised to selectively retain information. If you're interested in learning more of the specifics, this post is a must-read.
+
There are two popular types of gated RNNs: Long Short-term Memory (LSTMs) units and Gated Recurrent Units (GRUs).
+
+
When deciding between LSTMs and GRUs, empirical performance is the best factor but in general GRUs offer similar performance with less complexity (less weights).
We can also have RNNs that process inputs from both directions (first token to last token and vice versa) and combine their outputs. This architecture is known as a bidirectional RNN.
+
Notice that the output for each sample at each timestamp has size 256 (double the RNN_HIDDEN_DIM). This is because this includes both the forward and backward directions from the BiRNN.
defget_probability_distribution(y_prob,classes):
+"""Create a dict of class probabilities from an array."""
+ results={}
+ fori,class_inenumerate(classes):
+ results[class_]=np.float64(y_prob[i])
+ sorted_results={k:vfork,vinsorted(
+ results.items(),key=lambdaitem:item[1],reverse=True)}
+ returnsorted_results
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
Transformers are a very popular architecture that leverage and extend the concept of self-attention to create very useful representations of our input data for a downstream task.
+
+
+
advantages:
+
+
better representation for our input tokens via contextual embeddings where the token representation is based on the specific neighboring tokens using self-attention.
+
sub-word tokens, as opposed to character tokens, since they can hold more meaningful representation for many of our keywords, prefixes, suffixes, etc.
+
attend (in parallel) to all the tokens in our input, as opposed to being limited by filter spans (CNNs) or memory issues from sequential processing (RNNs).
+
+
+
+
disadvantages:
+
+
computationally intensive
+
required large amounts of data (mitigated using pretrained models)
We will download the AG News dataset, which consists of 120K text samples from 4 unique classes (Business, Sci/Tech, Sports, World)
+
1
+2
+3
+4
+5
# Load data
+url="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/news.csv"
+df=pd.read_csv(url,header=0)# load
+df=df.sample(frac=1).reset_index(drop=True)# shuffle
+df.head()
+
+
+
+
+
+
+
title
+
category
+
+
+
+
+
0
+
Sharon Accepts Plan to Reduce Gaza Army Operation...
+
World
+
+
+
1
+
Internet Key Battleground in Wildlife Crime Fight
+
Sci/Tech
+
+
+
2
+
July Durable Good Orders Rise 1.7 Percent
+
Business
+
+
+
3
+
Growing Signs of a Slowing on Wall Street
+
Business
+
+
+
4
+
The New Faces of Reality TV
+
World
+
+
+
+
+
1
+2
+3
# Reduce data size (too large to fit in Colab's limited memory)
+df=df[:10000]
+print(len(df))
+
+
+10000
+
+
+
Preprocessing
+
We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc.
+
defpreprocess(text,stopwords=STOPWORDS):
+"""Conditional preprocessing on our text unique to our task."""
+ # Lower
+ text=text.lower()
+
+ # Remove stopwords
+ pattern=re.compile(r"\b("+r"|".join(stopwords)+r")\b\s*")
+ text=pattern.sub("",text)
+
+ # Remove words in parenthesis
+ text=re.sub(r"\([^)]*\)","",text)
+
+ # Spacing and filters
+ text=re.sub(r"([-;;.,!?<=>])",r" \1 ",text)
+ text=re.sub("[^A-Za-z0-9]+"," ",text)# remove non alphanumeric chars
+ text=re.sub(" +"," ",text)# remove multiple spaces
+ text=text.strip()
+
+ returntext
+
+
1
+2
+3
# Sample
+text="Great week for the NYSE!"
+preprocess(text=text)
+
+
+great week nyse
+
+
1
+2
+3
+4
# Apply to dataframe
+preprocessed_df=df.copy()
+preprocessed_df.title=preprocessed_df.title.apply(preprocess)
+print(f"{df.title.values[0]}\n\n{preprocessed_df.title.values[0]}")
+
+
+Sharon Accepts Plan to Reduce Gaza Army Operation, Haaretz Says
+
+sharon accepts plan reduce gaza army operation haaretz says
+
+
+
+
Warning
+
If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before applying those operations. This is because we cannot apply any knowledge gained from the test set accidentally (data leak) during preprocessing/training. However for global preprocessing steps like the function above where we aren't learning anything from the data itself, we can perform before splitting the data.
We'll first learn about the unique components within the Transformer architecture and then implement one for our text classification task.
+
Scaled dot-product attention
+
The most popular type of self-attention is scaled dot-product attention from the widely-cited Attention is all you need paper. This type of attention involves projecting our encoded input sequences onto three matrices, queries (Q), keys (K) and values (V), whose weights we learn.
Instead of applying self-attention only once across the entire encoded input, we can also separate the input and apply self-attention in parallel (heads) to each input section and concatenate them. This allows the different head to learn unique representations while maintaining the complexity since we split the input into smaller subspaces.
hidden dim (or dimension of the model \(d_{model}\))
+
+
+
+
+
Positional encoding
+
With self-attention, we aren't able to account for the sequential position of our input tokens. To address this, we can use positional encoding to create a representation of the location of each token with respect to the entire sequence. This can either be learned (with weights) or we can use a fixed function that can better extend to create positional encoding for lengths during inference that were not observed during training.
+
\[ PE_{(pos,2i)} = sin({pos}/{10000^{2i/H}}) \]
+
\[ PE_{(pos,2i+1)} = cos({pos}/{10000^{2i/H}}) \]
+
+
+
+
+
Variable
+
Description
+
+
+
+
+
\(pos\)
+
position of the token \((1...M)\)
+
+
+
\(i\)
+
hidden dim \((1..H)\)
+
+
+
+
+
This effectively allows us to represent each token's relative position using a fixed function for very large sequences. And because we've constrained the positional encodings to have the same dimensions as our encoded inputs, we can simply concatenate them before feeding them into the multi-head attention heads.
+
Architecture
+
And here's how it all fits together! It's an end-to-end architecture that creates these contextual representations and uses an encoder-decoder architecture to predict the outcomes (one-to-one, many-to-one, many-to-many, etc.) Due to the complexity of the architecture, they require massive amounts of data for training without overfitting, however, they can be leveraged as pretrained models to finetune with smaller datasets that are similar to the larger set it was initially trained on.
We're going to use a pretrained BertModel to act as a feature extractor. We'll only use the encoder to receive sequential and pooled outputs (is_decoder=False is default).
We decided to work with the pooled output, but we could have just as easily worked with the sequential output (encoder representation for each sub-token) and applied a CNN (or other decoder options) on top of it.
defget_probability_distribution(y_prob,classes):
+"""Create a dict of class probabilities from an array."""
+ results={}
+ fori,class_inenumerate(classes):
+ results[class_]=np.float64(y_prob[i])
+ sorted_results={k:vfork,vinsorted(
+ results.items(),key=lambdaitem:item[1],reverse=True)}
+ returnsorted_results
+
Now we're ready to start the MLOps course to learn how to apply all this foundational modeling knowledge to responsibly develop, deploy and maintain production machine learning applications.
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Transformers - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
Next we'll define a LabelEncoder to encode our text labels into unique indices. We're not going to use scikit-learn's LabelEncoder anymore because we want to be able to save and load our instances the way we want to.
+
We need to standardize our data (zero mean and unit variance) so a specific feature's magnitude doesn't affect how the model learns its weights. We're only going to standardize the inputs X because our outputs y are class values. We're going to compose our own StandardScaler class so we can easily save and load it later during inference.
+
# Standardize the data (mean=0, std=1) using training data
+X_scaler=StandardScaler()
+X_scaler.fit(X_train)
+
+
1
+2
+3
+4
# Apply scaler on training and test data (don't standardize outputs for classification)
+X_train=X_scaler.scale(X_train)
+X_val=X_scaler.scale(X_val)
+X_test=X_scaler.scale(X_test)
+
+
1
+2
+3
# Check (means should be ~0 and std should be ~1)
+print(f"X_test[0]: mean: {np.mean(X_test[:,0],axis=0):.1f}, std: {np.std(X_test[:,0],axis=0):.1f}")
+print(f"X_test[1]: mean: {np.mean(X_test[:,1],axis=0):.1f}, std: {np.std(X_test[:,1],axis=0):.1f}")
+
+We don't really need the collate_fn here but we wanted to make it transparent because we will need it when we want to do specific processing on our batch (ex. padding).
+
So far, we used batch gradient descent to update our weights. This means that we calculated the gradients using the entire training dataset. We also could've updated our weights using stochastic gradient descent (SGD) where we pass in one training example one at a time. The current standard is mini-batch gradient descent, which strikes a balance between batch and SGD, where we update the weights using a mini-batch of n (BATCH_SIZE) samples. This is where the DataLoader object comes in handy.
+
So far we've been running our operations on the CPU but when we have large datasets and larger models to train, we can benefit by parallelizing tensor operations on a GPU. In this notebook, you can use a GPU by going to Runtime > Change runtime type > Select GPU in the Hardware accelerator dropdown. We can what device we're using with the following line of code:
+
1
+2
+3
# Set CUDA seeds
+torch.cuda.manual_seed(SEED)
+torch.cuda.manual_seed_all(SEED)# multi-GPU
+
So far we've been writing training loops that train only using the train data split and then we perform evaluation on our test set. But in reality, we would follow this process:
+
+
Train using mini-batches on one epoch of the train data split.
+
Evaluate loss on the validation split and use it to adjust hyperparameters (ex. learning rate).
+
After training ends (via stagnation in improvements, desired performance, etc.), evaluate your trained model on the test (hold-out) data split.
+
+
We'll create a Trainer class to keep all of these processes organized.
+
The first function in the class is train_step which will train the model using batches from one epoch of the train data split.
Next we'll define the eval_step which will be used for processing both the validation and test data splits. This is because neither of them require gradient updates and display the same metrics.
defeval_step(self,dataloader):
+"""Validation or test step."""
+ # Set model to eval mode
+ self.model.eval()
+ loss=0.0
+ y_trues,y_probs=[],[]
+
+ # Iterate over val batches
+ withtorch.inference_mode():
+ fori,batchinenumerate(dataloader):
+
+ # Step
+ batch=[item.to(self.device)foriteminbatch]# Set device
+ inputs,y_true=batch[:-1],batch[-1]
+ z=self.model(inputs)# Forward pass
+ J=self.loss_fn(z,y_true).item()
+
+ # Cumulative Metrics
+ loss+=(J-loss)/(i+1)
+
+ # Store outputs
+ y_prob=F.softmax(z).cpu().numpy()
+ y_probs.extend(y_prob)
+ y_trues.extend(y_true.cpu().numpy())
+
+ returnloss,np.vstack(y_trues),np.vstack(y_probs)
+
+
The final function is the predict_step which will be used for inference. It's fairly similar to the eval_step except we don't calculate any metrics. We pass on the predictions which we can use to generate our performance scores.
defpredict_step(self,dataloader):
+"""Prediction step."""
+ # Set model to eval mode
+ self.model.eval()
+ y_probs=[]
+
+ # Iterate over val batches
+ withtorch.inference_mode():
+ fori,batchinenumerate(dataloader):
+
+ # Forward pass w/ inputs
+ inputs,targets=batch[:-1],batch[-1]
+ z=self.model(inputs)
+
+ # Store outputs
+ y_prob=F.softmax(z).cpu().numpy()
+ y_probs.extend(y_prob)
+
+ returnnp.vstack(y_probs)
+
+
LR scheduler
+
As our model starts to optimize and perform better, the loss will reduce and we'll need to make smaller adjustments. If we keep using a fixed learning rate, we'll be overshooting back and forth. Therefore, we're going to add a learning rate scheduler to our optimizer to adjust our learning rate during training. There are many schedulers schedulers to choose from but a popular one is ReduceLROnPlateau which reduces the learning rate when a metric (ex. validation loss) stops improving. In the example below we'll reduce the learning rate by a factor of 0.1 (factor=0.1) when our metric of interest (self.scheduler.step(val_loss)) stops decreasing (mode="min") for three (patience=3) straight epochs.
We should never train our models for an arbitrary number of epochs but instead we should have explicit stopping criteria (even if you are bootstrapped by compute resources). Common stopping criteria include when validation performance stagnates for certain # of epochs (patience), desired performance is reached, etc.
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Our CLI application made it much easier to interact with our models, especially for fellow team members who may not want to delve into the codebase. But there are several limitations to serving our models with a CLI:
+
+
users need access to the terminal, codebase, virtual environment, etc.
+
CLI outputs on the terminal are not exportable
+
+
To address these issues, we're going to develop an application programming interface (API) that will anyone to interact with our application with a simple request.
+
+
The end user may not directly interact with our API but may use UI/UX components that send requests to our it.
+
+
Serving
+
APIs allow different applications to communicate with each other in real-time. But when it comes to serving predictions, we need to first decide if we'll do that in batches or real-time, which is entirely based on the feature space (finite vs. unbound).
+
Batch serving
+
We can make batch predictions on a finite set of inputs which are then written to a database for low latency inference. When a user or downstream process sends an inference request in real-time, cached results from the database are returned.
+
+
+
+
+
+
✅ generate and cache predictions for very fast inference for users.
+
✅ the model doesn't need to be spun up as it's own service since it's never used in real-time.
+
❌ predictions can become stale if user develops new interests that aren’t captured by the old data that the current predictions are based on.
+
❌ input feature space must be finite because we need to generate all the predictions before they're needed for real-time.
+
+
+
Batch serving tasks
+
What are some tasks where batch serving is ideal?
+
+Show answer
+
Recommend content that existing users will like based on their viewing history. However, new users may just receive some generic recommendations based on their explicit interests until we process their history the next day. And even if we're not doing batch serving, it might still be useful to cache very popular sets of input features (ex. combination of explicit interests leads to certain recommended content) so that we can serve those predictions faster.
+
+
+
Real-time serving
+
We can also serve live predictions, typically through a request to an API with the appropriate input data.
+
+
+
+
+
+
✅ can yield more up-to-date predictions which may yield a more meaningful user experience, etc.
+
❌ requires managed microservices to handle request traffic.
+
❌ requires real-time monitoring since input space in unbounded, which could yield erroneous predictions.
+
+
In this lesson, we'll create the API required to enable real-time serving. The interactions in our situation involve the client (users, other applications, etc.) sending a request (ex. prediction request) with the appropriate inputs to the server (our application with a trained model) and receiving a response (ex. prediction) in return.
+
+
+
+
+
Request
+
Users will interact with our API in the form of a request. Let's take a look at the different components of a request:
+
URI
+
A uniform resource identifier (URI) is an identifier for a specific resource.
The method is the operation to execute on the specific resource defined by the URI. There are many possible methods to choose from, but the four below are the most popular, which are often referred to as CRUD because they allow you to Create, Read, Update and Delete.
+
+
GET: get a resource.
+
POST: create or update a resource.
+
PUT/PATCH: create or update a resource.
+
DELETE: delete a resource.
+
+
+
Note
+
You could use either the POST or PUT request method to create and modify resources but the main difference is that PUT is idempotent which means you can call the method repeatedly and it'll produce the same state every time. Whereas, calling POST multiple times can result in creating multiple instance and so changes the overall state each time.
+
POST/models/<new_model>-d{}# error since we haven't created the `new_model` resource yet
+POST/models-d{}# creates a new model based on information provided in data
+POST/models/<existing_model>-d{}# updates an existing model based on information provided in data
+
+PUT/models/<new_model>-d{}# creates a new model based on information provided in data
+PUT/models/<existing_model>-d{}# updates an existing model based on information provided in data
+
+
+
We can use cURL to execute our API calls with the following options:
+
curl--help
+
+
+Usage: curl [options...]
+-X, --request HTTP method (ie. GET)
+-H, --header headers to be sent to the request (ex. authentication)
+-d, --data data to POST, PUT/PATCH, DELETE (usually JSON)
+...
+
+
+
For example, if we want to GET all models, our cURL command would look like this:
+
curl-XGET"http://localhost:8000/models"
+
+
+
Headers
+
Headers contain information about a certain event and are usually found in both the client's request as well as the server's response. It can range from what type of format they'll send and receive, authentication and caching info, etc.
+
curl-XGET"http://localhost:8000/"\ # method and URI
+-H"accept: application/json"\ # client accepts JSON
+-H"Content-Type: application/json"\ # client sends JSON
+
+
+
Body
+
The body contains information that may be necessary for the request to be processed. It's usually a JSON object sent during POST, PUT/PATCH, DELETE request methods.
+
curl-XPOST"http://localhost:8000/models"\ # method and URI
+-H"accept: application/json"\ # client accepts JSON
+-H"Content-Type: application/json"\ # client sends JSON
+-d"{'name': 'RoBERTa', ...}"# request body
+
+
+
Response
+
The response we receive from our server is the result of the request we sent. The response also includes headers and a body which should include the proper HTTP status code as well as explicit messages, data, etc.
We may also want to include other metadata in the response such as model version, datasets used, etc. Anything that the downstream consumer may be interested in or metadata that might be useful for inspection.
+
+
There are many HTTP status codes to choose from depending on the situation but here are the most common options:
+
+
+
+
+
Code
+
Description
+
+
+
+
+
200 OK
+
method operation was successful.
+
+
+
201 CREATED
+
POST or PUT method successfully created a resource.
+
+
+
202 ACCEPTED
+
the request was accepted for processing (but processing may not be done).
+
+
+
400 BAD REQUEST
+
server cannot process the request because of a client side error.
+
+
+
401 UNAUTHORIZED
+
you're missing required authentication.
+
+
+
403 FORBIDDEN
+
you're not allowed to do this operation.
+
+
+
404 NOT FOUND
+
the resource you're looking for was not found.
+
+
+
500 INTERNAL SERVER ERROR
+
there was a failure somewhere in the system process.
+
+
+
501 NOT IMPLEMENTED
+
this operation on the resource doesn't exist yet.
+
+
+
+
+
Best practices
+
When designing our API, there are some best practices to follow:
+
+
URI paths, messages, etc. should be as explicit as possible. Avoid using cryptic resource names, etc.
+
Use nouns, instead of verbs, for naming resources. The request method already accounts for the verb (✅ GET /users not ❌ GET /get_users).
+
Plural nouns (✅ GET /users/{userId} not ❌ GET /user/{userID}).
+
Use dashes in URIs for resources and path parameters but use underscores for query parameters (GET /nlp-models/?find_desc=bert).
+
Return appropriate HTTP and informative messages to the user.
+
+
Implementation
+
We're going to define our API in a separate app directory because, in the future, we may have additional packages like tagifai and we don't want our app to be attached to any one package. Inside our app directory, we'll create the follow scripts:
api.py: the main script that will include our API initialization and endpoints.
+
gunicorn.py: script for defining API worker configurations.
+
schemas.py: definitions for the different objects we'll use in our resource endpoints.
+
+
FastAPI
+
We're going to use FastAPI as our framework to build our API service. There are plenty of other framework options out there such as Flask, Django and even non-Python based options like Node, Angular, etc. FastAPI combines many of the advantages across these frameworks and is maturing quickly and becoming more widely adopted. It's notable advantages include:
Your choice of framework also depends on your team's existing systems and processes. However, with the wide adoption of microservices, we can wrap our specific application in any framework we choose and expose the appropriate resources so all other systems can easily communicate with it.
+
+
Initialization
+
The first step is to initialize our API in our api.py script` by defining metadata like the title, description and version:
Our first endpoint is going to be a simple one where we want to show that everything is working as intended. The path for the endpoint will just be / (when a user visit our base URI) and it'll be a GET request. This simple endpoint is often used as a health check to ensure that our application is indeed up and running properly.
We let our application know that the endpoint is at / through the path operation decorator in line 4 and we return a JSON response with the 200 OK HTTP status code.
+
+
In our actual api.py script, you'll notice that even our index function looks different. Don't worry, we're slowly adding components to our endpoints and justifying them along the way.
+
+
Launching
+
We're using Uvicorn, a fast ASGI server that can run asynchronous code in a single process to launch our application.
+
pipinstalluvicorn==0.17.6
+
+
# Add to requirements.txt
+uvicorn==0.17.6
+
+
We can launch our application with the following command:
+
uvicornapp.api:app\ # location of app (`app` directory > `api.py` script > `app` object)
+--host0.0.0.0\ # localhost
+--port8000\ # port 8000
+--reload\ # reload every time we update
+--reload-dirtagifai\ # only reload on updates to `tagifai` directory
+--reload-dirapp# and the `app` directory
+
+
+INFO: Will watch for changes in these directories: ['/Users/goku/Documents/madewithml/mlops/app', '/Users/goku/Documents/madewithml/mlops/tagifai']
+INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
+INFO: Started reloader process [57609] using statreload
+INFO: Started server process [57611]
+INFO: Waiting for application startup.
+INFO: Application startup complete.
+
+
+
+
Notice that we only reload on changes to specific directories, as this is to avoid reloading on files that won't impact our application such as log files, etc.
+
+
If we want to manage multiple uvicorn workers to enable parallelism in our application, we can use Gunicorn in conjunction with Uvicorn. This will usually be done in a production environment where we'll be dealing with meaningful traffic. We've included a app/gunicorn.py script with the customizable configuration and we can launch all the workers with the follow command:
+
Access endpoints via code. Here we show how to do it with the requests library in Python but it can be done with most popular languages. You can even use an online tool to convert your cURL commands into code!
+
In our GET \ request's response above, there was not a whole lot of information about the actual request, but it's useful to have details such as URL, timestamp, etc. But we don't want to do this individually for each endpoint, so let's use decorators to automatically add relevant metadata to our responses
We're passing in a Request instance in line 10 so we can access information like the request method and URL. Therefore, our endpoint functions also need to have this Request object as an input argument. Once we receive the results from our endpoint function f, we can append the extra details and return a more informative response. To use this decorator, we just have to wrap our functions accordingly.
There are also some built-in decorators we should be aware of. We've already seen the path operation decorator (ex. @app.get("/")) which defines the path for the endpoint as well as other attributes. There is also the events decorator (@app.on_event()) which we can use to startup and shutdown our application. For example, we use the (@app.on_event("startup")) event to load the artifacts for the model to use for inference. The advantage of doing this as an event is that our service won't start until this is complete and so no requests will be prematurely processed and cause errors. Similarly, we can perform shutdown events with (@app.on_event("shutdown")), such as saving logs, cleaning, etc.
When we define an endpoint, FastAPI automatically generates some documentation (adhering to OpenAPI standards) based on the it's inputs, typing, outputs, etc. We can access the Swagger UI for our documentation by going to /docs endpoints on any browser while the api is running.
+
+
+
+
+
Click on an endpoint > Try it out > Execute to see what the server's response will look like. Since this was a GET request without any inputs, our request body was empty but for other method's we'll need to provide some information (we'll illustrate this when we do a POST request).
+
+
+
+
+
Notice that our endpoint is organized under sections in the UI. We can use tags when defining our endpoints in the script:
+
Notice that we're passing an optional query parameter filter here to indicate the subset of performance we care about. We can include this parameter in our GET request like so:
Our next endpoint will be to GET the arguments used to train the model. This time, we're using a path parameter args, which is a required field in the URI.
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
@app.get("/args/{arg}",tags=["Arguments"])
+@construct_response
+def_arg(request:Request,arg:str)->Dict:
+"""Get a specific parameter's value used for the run."""
+ response={
+ "message":HTTPStatus.OK.phrase,
+ "status-code":HTTPStatus.OK,
+ "data":{
+ arg:vars(artifacts["args"]).get(arg,""),
+ },
+ }
+ returnresponse
+
+
We can perform our GET request like so, where the param is part of the request URI's path as opposed to being part of it's query string.
+
Now it's time to define our endpoint for prediction. We need to consume the inputs that we want to classify and so we need to define the schema that needs to be followed when defining those inputs.
Here we're defining a PredictPayload object as a List of Text objects called texts. Each Text object is a string that defaults to None and must have a minimum length of 1 character.
+
+
Note
+
We could've just defined our PredictPayload like so:
+
We're using pydantic's BaseModel object here because it offers built-in validation for all of our schemas. In our case, if a Text instance is less than 1 character, then our service will return the appropriate error message and code:
Lastly, we can add a schema_extra object under a Config class to depict what an example PredictPayload should look like. When we do this, it automatically appears in our endpoint's documentation (click Try it out).
classPredictPayload(BaseModel):
+ texts:List[Text]
+
+ @validator("texts")
+ deflist_must_not_be_empty(cls,value):
+ ifnotlen(value):
+ raiseValueError("List of texts to classify cannot be empty.")
+ returnvalue
+
+classConfig:
+schema_extra={
+"example":{
+"texts":[
+{"text":"Transfer learning with transformers for text classification."},
+{"text":"Generative adversarial networks in both PyTorch and TensorFlow."},
+]
+}
+}
+
+
+
+
+
+
Product
+
To make our API a standalone product, we'll need to create and manage a database for our users and resources. These users will have credentials which they will use for authentication and use their privileges to be able to communicate with our service. And of course, we can display a rendered frontend to make all of this seamless with HTML forms, buttons, etc. This is exactly how the old MWML platform was built and we leveraged FastAPI to deliver high performance for 500K+ daily service requests.
+
If you are building a product, then I highly recommending forking this generation template to get started. It includes the backbone architecture you need for your product:
+
+
Databases (models, migrations, etc.)
+
Authentication via JWT
+
Asynchronous task queue with Celery
+
Customizable frontend via Vue JS
+
Docker integration
+
so much more!
+
+
However, for the majority of ML developers, thanks to the wide adoption of microservices, we don't need to do all of this. A well designed API service that can seamlessly communicate with all other services (framework agnostic) will fit into any process and add value to the overall product. Our main focus should be to ensure that our service is working as it should and constantly improve, which is exactly what the next cluster of lessons will focus on (testing and monitoring)
+
Model server
+
Besides wrapping our models as separate, scalable microservices, we can also have a purpose-built model server to host our models. Model servers provide a registry with an API layer to seamlessly inspect, update, serve, rollback, etc. multiple versions of models. They also offer automatic scaling to meet throughput and latency needs. Popular options include BentoML, MLFlow, TorchServe, RedisAI, Nvidia Triton Inference Server, etc.
+
+
Model servers are experiencing a lot of adoption for their ability to standardize the model deployment and serving processes across the team -- enabling seamless upgrades, validation and integration.
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ APIs for Model Serving - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
We'll often want to increase the size and diversity of our training data split through data augmentation. It involves using the existing samples to generate synthetic, yet realistic, examples.
+
+
+
Split the dataset. We want to split our dataset first because many augmentation techniques will cause a form of data leak if we allow the generated samples to be placed across different data splits.
+
+
For example, some augmentation involves generating synonyms for certain key tokens in a sentence. If we allow the generated sentences from the same origin sentence to go into different splits, we could be potentially leaking samples with nearly identical embedding representations across our different splits.
+
+
+
+
Augment the training split. We want to apply data augmentation on only the training set because our validation and testing splits should be used to provide an accurate estimate on actual data points.
+
+
+
Inspect and validate. It's useless to augment just for the same of increasing our training sample size if the augmented data samples are not probable inputs that our model could encounter in production.
+
+
+
The exact method of data augmentation depends largely on the type of data and the application. Here are a few ways different modalities of data can be augmented:
While the transformations on some data modalities, such as images, are easy to inspect and validate, others may introduce silent errors. For example, shifting the order of tokens in text can significantly alter the meaning (“this is really cool” → “is this really cool”). Therefore, it’s important to measure the noise that our augmentation policies will introduce and do have granular control over the transformations that take place.
+
+
Libraries
+
Depending on the feature types and tasks, there are many data augmentation libraries which allow us to extend our training data.
# Load tokenizers and transformers
+substitution=naw.ContextualWordEmbsAug(model_path="distilbert-base-uncased",action="substitute")
+insertion=naw.ContextualWordEmbsAug(model_path="distilbert-base-uncased",action="insert")
+text="Conditional image generation using Variational Autoencoders and GANs."
+
+
1
+2
# Substitutions
+substitution.augment(text)
+
+
+hierarchical risk mapping using variational signals and gans.
+
+
+
Substitution doesn't seem like a great idea for us because there are certain keywords that provide strong signal for our tags so we don't want to alter those. Also, note that these augmentations are NOT deterministic and will vary every time we run them. Let's try insertion...
+
1
+2
# Insertions
+insertion.augment(text)
+
+
+automated conditional inverse image generation algorithms using multiple variational autoencoders and gans.
+
+
+
A little better but still quite fragile and now it can potentially insert key words that can influence false positive tags to appear. Maybe instead of substituting or inserting new tokens, let's try simply swapping machine learning related keywords with their aliases. We'll use Snorkel's transformation functions to easily achieve this.
print(flattened_aliases["natural language processing"])
+print(flattened_aliases["nlp"])
+
+
+['nlp', 'nlproc']
+['nlproc', 'natural language processing']
+
+
+
+
For now we'll use tags and aliases as they are in aliases_by_tag but we could account for plurality of tags using the inflect package or apply stemming before replacing aliases, etc.
+
+
1
+2
+3
# We want to match with the whole word only
+print("gan"in"This is a gan.")
+print("gan"in"This is gandalf.")
+
@transformation_function()
+defswap_aliases(x):
+"""Swap ML keywords with their aliases."""
+ # Find all matches
+ matches=[]
+ fori,taginenumerate(flattened_aliases):
+ match=find_word(tag,x.text)
+ ifmatch:
+ matches.append(match)
+ # Swap a random match with a random alias
+ iflen(matches):
+ match=random.choice(matches)
+ tag=x.text[match.start():match.end()]
+ x.text=f"{x.text[:match.start()]}{random.choice(flattened_aliases[tag])}{x.text[match.end():]}"
+ returnx
+
+
1
+2
+3
+4
+5
# Swap
+foriinrange(3):
+ sample_df=pd.DataFrame([{"text":"a survey of reinforcement learning for nlp tasks."}])
+ sample_df.text=sample_df.text.apply(clean_text,lower=True,stem=False)
+ print(swap_aliases(sample_df.iloc[0]).text)
+
+
1
+2
+3
+4
+5
# Undesired behavior (needs contextual insight)
+foriinrange(3):
+ sample_df=pd.DataFrame([{"text":"Autogenerate your CV to apply for jobs using NLP."}])
+ sample_df.text=sample_df.text.apply(clean_text,lower=True,stem=False)
+ print(swap_aliases(sample_df.iloc[0]).text)
+
+
+autogenerate vision apply jobs using nlp
+autogenerate cv apply jobs using natural language processing
+autogenerate cv apply jobs using nlproc
+
+
+
Now we'll define a augmentation policy to apply our transformation functions with certain rules (how many samples to generate, whether to keep the original data point, etc.)
big bad nlp database collection 400 nlp datasets...
+
natural-language-processing
+
+
+
2
+
big bad natural language processing database c...
+
natural-language-processing
+
+
+
2
+
big bad nlproc database collection 400 nlp dat...
+
natural-language-processing
+
+
+
+
+
+
1
len(train_df),len(train_df_augmented)
+
+
+(668, 913)
+
+
+
For now, we'll skip the data augmentation because it's quite fickle and empirically it doesn't improvement performance much. But we can see how this can be very effective once we can control what type of vocabulary to augment on and what exactly to augment with.
+
+
Warning
+
Regardless of what method we use, it's important to validate that we're not just augmenting for the sake of augmentation. We can do this by executing any existing data validation tests and even creating specific tests to apply on augmented data.
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Data Augmentation - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
In the previous lesson, we learned how to manually execute our ML workloads with Jobs and Services. However, we want to be able to automatically execute these workloads when certain events occur (new data, performance regressions, elapsed time, etc.) to ensure that our models are always up to date and increasing in quality. In this lesson, we'll learn how to create continuous integration and delivery (CI/CD) pipelines to achieve an application that is capable of continual learning.
+
GitHub Actions
+
We're going to use GitHub Actions to create our CI/CD pipelines. GitHub Actions allow us to define workflows that are triggered by events (pull request, push, etc.) and execute a series of actions.
+
+
+
+
+
Our GitHub Actions are defined under our repository's .github/workflows directory where we have workflows for documentation (documentation.yaml), workloads (workloads.yaml) to train/validate a model and a final workflow for serving our model (serve.yaml). Let's start by understanding the structure of a workflow.
+
Events
+
Workflows are triggered by an event, which can be something that occurs on an event (like a push or pull request), schedule (cron), manually and many more. In our application, our workloads workflow is triggered on a pull request to the main branch and then our serve workflow and documentation workflows are triggered on a push to the main branch.
Each of our workflows only have one job but if we had multiple, the jobs would all run in parallel. If we wanted to create dependent jobs, where if a particular job fails all it's dependent jobs will be skipped, then we'd use the needs keyword.
+
+
Steps
+
Each job contains a series of steps which are executed in order. Each step has a name, as well as actions to use from the GitHub Action marketplace and/or commands we want to run. For example, here's a look at one of the steps in our workloads job inside our workloads.yaml workflow:
Now that we understand the basic components of a GitHub Actions workflow, let's take a closer look at each of our workflows. Most of our workflows will require access to our Anyscale credentials so we'll start by setting those up. We can set these secrets for our repository under the Settings tab.
+
+
+
+
+
And our first workflow will be our workloads workflow which will be triggered on a pull request to the main branch. This means that we'll need to push our local code changes to Git and then submit a pull request to the main branch. But in order to push our code to GitHub, we'll need to first authenticate with our credentials before pushing to our repository:
+
gitconfig--globaluser.name$GITHUB_USERNAME
+gitconfig--globaluser.emailyou@example.com# <-- CHANGE THIS to your email
+gitadd.
+gitcommit-m""# <-- CHANGE THIS to your message
+gitpushorigindev
+
+
Now you will be prompted to enter your username and password (personal access token). Follow these steps to get personal access token: New GitHub personal access token → Add a name → Toggle repo and workflow → Click Generate token (scroll down) → Copy the token and paste it when prompted for your password.
+
+
Note that we should be on a dev branch, which we set up in our setup lesson. If you're not, go ahead and run gitcheckout-bdev first.
+
+
And when any of our GitHub Actions workflows execute, we will be able to view them under the Actions tab of our repository. Here we'll find all the workflows that have been executed and we can inspect each one to see the details of the execution.
+
Workloads
+
Our workloads workflow is triggered on a pull request to the main branch. It contains a single job that runs our model development workloads with an Anyscale Job. The steps in this job are as follows:
+
+
We start by configuring our AWS credentials so that we can push/pull from our S3 buckets. Recall that we store our model registry and results in S3 buckets so we need to be able to access them. We created an IAM role for this course so that only certain repositories can access our S3 buckets.
+
Next, we checkout our repository code and install our Python dependencies so that we can execute our Anyscale Job.
+
1
+2
+3
+4
+5
+6
+7
# Set up dependencies
+-uses:actions/checkout@v3
+-uses:actions/setup-python@v4
+with:
+python-version:'3.10.11'
+cache:'pip'
+-run:python3 -m pip install anyscale==0.5.128 typer==0.9.0
+
+
Next, we can run our Anyscale Job but note that since this will be running on a GitHub hosted runner, we need to export our Anyscale credentials first (which we already set up earlier on our repository).
+
1
+2
+3
+4
+5
+6
# Run workloads
+-name:Workloads
+run:|
+export ANYSCALE_HOST=$\{\{ secrets.ANYSCALE_HOST \}\}
+export ANYSCALE_CLI_TOKEN=$\{\{ secrets.ANYSCALE_CLI_TOKEN \}\}
+anyscale jobs submit deploy/jobs/workloads.yaml --wait
+
+
Recall that our Anyscale Job in the previous step saves our model registry and results to S3 buckets. So in this step, we'll read the artifacts from S3 (from our unique path using our GitHub username) and save them locally on our GitHub runner. We have a small utility script called .github/workflows/json_to_md.py to convert our JSON results to markdown tables that we can comment on our PR.
+
We use a GitHub Action from the marketplace to comment our results markdown tables on our PR.
+
1
+2
+3
+4
+5
+6
+7
+8
+9
# Comment results to PR
+-name:Comment training results on PR
+uses:thollander/actions-comment-pull-request@v2
+with:
+filePath:results/training_results.md
+-name:Comment evaluation results on PR
+uses:thollander/actions-comment-pull-request@v2
+with:
+filePath:results/evaluation_results.md
+
+
+
So when this workloads workflow completes, we'll have a comment on our PR (example) with our training and evaluation results. We can now collaboratively analyze the details and decide if we want to merge the PR.
+
+
+
+
+
+
Tip
+
We could easily extend this by retrieving evaluation results from our currently deployed model in production as well. Recall that we defined a /evaluate/ endpoint for our service that expects a dataset location and returns the evaluation results. And we can submit this request as a step in our workflow and save the results to a markdown table that we can comment on our PR.
It contains a single job that serves our model with Anyscale Services. The steps in this job are as follows:
+
+
We start by configuring our AWS credentials so that we can push/pull from our S3 buckets. Recall that we store our model registry and results in S3 buckets so we need to be able to access them.
+
Next, we checkout our repository code and install our Python dependencies so that we can execute our Anyscale Job.
+
1
+2
+3
+4
+5
+6
+7
# Set up dependencies
+-uses:actions/checkout@v3
+-uses:actions/setup-python@v4
+with:
+python-version:'3.10.11'
+cache:'pip'
+-run:python3 -m pip install anyscale==0.5.128 typer==0.9.0
+
+
Next, we can run our Anyscale Service but note that since this will be running on a GitHub hosted runner, we need to export our Anyscale credentials first (which we already set up earlier on our repository).
+
1
+2
+3
+4
+5
+6
# Run workloads
+-name:Workloads
+run:|
+export ANYSCALE_HOST=$\{\{ secrets.ANYSCALE_HOST \}\}
+export ANYSCALE_CLI_TOKEN=$\{\{ secrets.ANYSCALE_CLI_TOKEN \}\}
+anyscale service rollout --service-config-file deploy/services/serve_model.yaml
+
+
+
So when this serve workflow completes, our model will be deployed to production and we can start making inference requests with it.
+
+
Note
+
The anyscale service rollout command will update our existing service (if there was already one running) without changing the SECRET_TOKEN or SERVICE_ENDPOINT. So this means that our downstream applications that were making inference requests to our service can continue to do so without any changes.
+
+
Documentation
+
Our documentation workflow is also triggered on a push to the main branch. It contains a single job that builds our docs. The steps in this job are as follows:
+
+
We checkout our repository code and install our Python dependencies so that we can build our documentation.
+
1
+2
+3
+4
+5
+6
+7
# Set up dependencies
+-uses:actions/checkout@v3
+-uses:actions/setup-python@v4
+with:
+python-version:'3.10.11'
+cache:'pip'
+-run:python3 -m pip install mkdocs==1.4.2 mkdocstrings==0.21.2 "mkdocstrings[python]>=0.18"
+
And with that, we're able to automatically update our ML application when ever we make changes to the code and want to trigger a new deployment. We have fully control because we can decide not to trigger an event (ex. push to main branch) if we're not satisfied with the results of our model development workloads. We can easily extend this to include other events (ex. new data, performance regressions, etc.) to trigger our workflows, as well as, integrate with more functionality around orchestration (ex. Prefect, Kubeflow, etc.), monitoring, etc.
+
+
+
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ CI/CD workflows - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
In the previous lesson, we organized our code from our notebook into individual Python scripts. We moved our functions and classes into their respective scripts and also created new workload functions to execute the main ML workloads (ex. train_model function from madewithml/train.py script). We now want to enable users to execute these workloads from the terminal without having to know anything about our code itself.
+
Methods
+
One way to execute these workloads is to import the functions in the Python script and execute them one at a time:
Caution: Don't forget to run export PYTHONPATH=$PYTHONPATH:$PWD in your terminal to ensure that Python can find the modules in our project.
+
+
While this may seem simple, it still requires us to import packages, identify the input arguments, etc. Therefore, another alternative is to place the main function call under a if__name__=="__main__" conditional so that it's only executed when we run the script directly. Here we can pass in the input arguments directly into the function in the code.
However, the limitation here is that we can't choose which function from a particular script to execute. We have to set the one we want to execute under the if__name__=="__main__" conditional. It's also very rigid since we have to set the input argument values in the code, unless we use a library like argparse.
+
1
+2
+3
+4
+5
+6
+7
+8
# madewithml/serve.py
+if__name__=="__main__":
+ parser=argparse.ArgumentParser()
+ parser.add_argument("--run_id",help="run ID to use for serving.")
+ parser.add_argument("--threshold",type=float,default=0.9,help="threshold for `other` class.")
+ args=parser.parse_args()
+ ray.init()
+ serve.run(ModelDeployment.bind(run_id=args.run_id,threshold=args.threshold))
+
+Which we can call from the terminal like so (note that --threshold is optional since it has a default value):
+
pythonmadewithml/serve.py--run_id$RUN_ID
+
+
+
We use argparse in our madewithml/serve.py script because it's the only workload in the script and it's a one-line function call (serve.run()).
+
+
Compared to using functions or the __main__ conditional, a much better user experience would be to execute these workloads from the terminal. In this lesson, we'll learn how to build a command-line interface (CLI) so that execute our main ML workloads.
+
Typer
+
We're going to create our CLI using Typer. It's as simple as initializing the app and then adding the appropriate decorator to each function operation we wish to use as a CLI command in our script:
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
importtyper
+fromtyping_extensionsimportAnnotated
+app=typer.Typer()
+
+@app.command()
+deftrain_model(
+ experiment_name:Annotated[str,typer.Option(help="name of the experiment.")]=None,
+ ...):
+ pass
+
+if__name__=="__main__":
+ app()
+
+
Inputs
+
You may notice that our function inputs have a lot of information besides just the input name. We'll cover typing (str, List, etc.) in our documentation lesson but for now, just know that Annotated allows us to specify metadata about the input argument's type and details about the (required) option (typer.Option).
+
+
We make all of our input arguments optional so that we can explicitly define them in our CLI commands (ex. --experiment-name).
+
+
We can also add some helpful information about the input parameter (with typer.Option(help="...")) and a default value (ex. None).
+
Usage
+
With our CLI commands defined and our input arguments enriched, we can execute our workloads. Let's start by executing our train_model function by assuming that we don't know what the required input parameters are. Instead of having to look in the code, we can just do the following:
+
pythonmadewithml/train.py--help
+
+
+Usage: train.py [OPTIONS]
+Main train function to train our model as a distributed workload.
+
+
+
+
+
+
+
We can follow this helpful message to execute our workload with the appropriate inputs.
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
When developing an application, there are a lot of technical decisions and results (preprocessing, performance, etc.) that are integral to our system. How can we effectively communicate this to other developers and business stakeholders? One option is a Jupyter notebook but it's often cluttered with code and isn't very easy for non-technical team members to access and run. We need to create a dashboard that can be accessed without any technical prerequisites and effectively communicates key findings. It would be even more useful if our dashboard was interactive such that it provides utility even for the technical developers.
+
Streamlit
+
There are some great tooling options, such as Dash, Gradio, Streamlit, Tableau, Looker, etc. for creating dashboards to deliver data oriented insights. Traditionally, interactive dashboards were exclusively created using front-end programming languages such as HTML Javascript, CSS, etc. However, given that many developers working in machine learning are using Python, the tooling landscape has evolved to bridge this gap. These tools now allow ML developers to create interactive dashboards and visualizations in Python while offering full customization via HTML, JS, and CSS. We'll be using Streamlit to create our dashboards because of it's intuitive API, sharing capabilities and increasing community adoption.
+
Set up
+
With Streamlit, we can quickly create an empty application and as we develop, the UI will update as well.
+
+You can now view your Streamlit app in your browser.
+
+ Local URL: http://localhost:8501
+ Network URL: http://10.0.1.93:8501
+
+
+
This will automatically open up the streamlit dashboard for us on http://localhost:8501.
+
+
Be sure to add this package and version to our requirements.txt file.
+
+
API Reference
+
Before we create a dashboard for our specific application, we need to learn about the different Streamlit components. Instead of going through them all in this lesson, take a few minutes and go through the API reference. It's quite short and we promise you'll be amazed at how many UI components (styled text, latex, tables, plots, etc.) you can create using just Python. We'll explore the different components in detail as they apply to creating different interactions for our specific dashboard below.
+
Sections
+
We'll start by outlining the sections we want to have in our dashboard by editing our streamlit/app.py script:
In this section, we'll display the performance of from our latest trained model. Again, we're going to keep it simple but we could also overlay more information such as improvements or regressions from previous deployments by accessing the model store.
+
1
+2
+3
+4
+5
+6
+7
+8
+9
st.header("📊 Performance")
+performance_fp=Path(config.CONFIG_DIR,"performance.json")
+performance=utils.load_dict(filepath=performance_fp)
+st.text("Overall:")
+st.write(performance["overall"])
+tag=st.selectbox("Choose a tag: ",list(performance["class"].keys()))
+st.write(performance["class"][tag])
+tag=st.selectbox("Choose a slice: ",list(performance["slices"].keys()))
+st.write(performance["slices"][tag])
+
+
+
+
+
+
Inference
+
With the inference section, we want to be able to quickly predict with the latest trained model.
+
1
+2
+3
+4
+5
st.header("🚀 Inference")
+text=st.text_input("Enter text:","Transfer learning with transformers for text classification.")
+run_id=st.text_input("Enter run ID:",open(Path(config.CONFIG_DIR,"run_id.txt")).read())
+prediction=main.predict_tag(text=text,run_id=run_id)
+st.write(prediction)
+
+
+
+
+
+
+
Tip
+
Our dashboard is quite simple but we can also more comprehensive dashboards that reflect some of the core topics we covered in our machine learning canvas.
View false +/- interactively and connect with annotation pipelines so that changes to the data can be reviewed and incorporated.
+
Compare performances across multiple releases to visualize improvements/regressions over time (using model store, git tags, etc.)
+
+
+
Caching
+
Sometimes we may have views that involve computationally heavy operations, such as loading data or model artifacts. It's best practice to cache these operations by wrapping them as a separate function with the @st.cache decorator. This calls for Streamlit to cache the function by the specific combination of it's inputs to deliver the respective outputs when the function is invoked with the same inputs.
We have several different options for deploying and managing our Streamlit dashboard. We could use Streamlit's sharing feature (beta) which allows us to seamlessly deploy dashboards straight from GitHub. Our dashboard will continue to stay updated as we commit changes to our repository. Another option is to deploy the Streamlit dashboard along with our API service. We can use docker-compose to spin up a separate container or simply add it to the API service's Dockerfile's ENTRYPOINT with the appropriate ports exposed. The later might be ideal, especially if your dashboard isn't meant to be public and it you want added security, performance, etc.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Dashboard - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
So far we've had the convenience of using local CSV files as data source but in reality, our data can come from many disparate sources. Additionally, our processes around transforming and testing our data should ideally be moved upstream so that many different downstream processes can benefit from them. Our ML use case being just one among the many potential downstream applications. To address these shortcomings, we're going to learn about the fundamentals of data engineering and construct a modern data stack that can scale and provide high quality data for our applications.
This process is more commonly known as ELT, but there are variants such as ETL and reverse ETL, etc. They are all essentially the same underlying workflows but have slight differences in the order of data flow and where data is processed and stored.
+
+
+
+
+
+
Utility and simplicity
+
It can be enticing to set up a modern data stack in your organization, especially with all the hype. But it's very important to motivate utility and adding additional complexity:
+
+
Start with a use case that we already have data sources for and has direct impact on the business' bottom line (ex. user churn).
+
Start with the simplest infrastructure (source → database → report) and add complexity (in infrastructure, performance and team) as needed.
+
+
+
Data systems
+
+
+
Before we start working with our data, it's important to understand the different types of systems that our data can live in. So far in this course we've worked with files, but there are several types of data systems that are widely adopted in industry for different purposes.
+
+
+
+
+
Data lake
+
A data lake is a flat data management system that stores raw objects. It's a great option for inexpensive storage and has the capability to hold all types of data (unstructured, semi-structured and structured). Object stores are becoming the standard for data lakes with default options across the popular cloud providers. Unfortunately, because data is stored as objects in a data lake, it's not designed for operating on structured data.
Another popular storage option is a database (DB), which is an organized collection of structured data that adheres to either:
+
+
relational schema (tables with rows and columns) often referred to as a Relational Database Management System (RDBMS) or SQL database.
+
non-relational (key/value, graph, etc.), often referred to as a non-relational database or NoSQL database.
+
+
A database is an online transaction processing (OLTP) system because it's typically used for day-to-day CRUD (create, read, update, delete) operations where typically information is accessed by rows. However, they're generally used to store data from one application and is not designed to hold data from across many sources for the purpose of analytics.
A data warehouse (DWH) is a type of database that's designed for storing structured data from many different sources for downstream analytics and data science. It's an online analytical processing (OLAP) system that's optimized for performing operations across aggregating column values rather than accessing specific rows.
The first step in our data pipeline is to extract data from a source and load it into the appropriate destination. While we could construct custom scripts to do this manually or on a schedule, an ecosystem of data ingestion tools have already standardized the entire process. They all come equipped with connectors that allow for extraction, normalization, cleaning and loading between sources and destinations. And these pipelines can be scaled, monitored, etc. all with very little to no code.
We're going to use the open-source tool Airbyte to create connections between our data sources and destinations. Let's set up Airbyte and define our data sources. As we progress in this lesson, we'll set up our destinations and create connections to extract and load data.
+
+
Ensure that we have Docker installed, but if not, download it here. For Windows users, be sure to have these configurations enabled.
+
In a parent directory, outside our project directory for the MLOps course, execute the following commands to load the Airbyte repository locally and launch the service.
+
Our data sources we want to extract from can be from anywhere. They could come from 3rd party apps, files, user click streams, physical devices, data lakes, databases, data warehouses, etc. But regardless of the source of our data, they type of data should fit into one of these categories:
+
+
structured: organized data stored in an explicit structure (ex. tables)
+
semi-structured: data with some structure but no formal schema or data types (web pages, CSV, JSON, etc.)
+
unstructured: qualitative data with no formal structure (text, images, audio, etc.)
+
+
For our application, we'll define two data sources:
+
+
projects.csv: data containing projects with their ID, create date, title and description.
+
tags.csv: labels for each of project IDs in projects.csv
+
+
+
Ideally, these data assets would be retrieved from a database that contains projects that we extracted and perhaps another database that stores labels from our labeling team's workflows. However, for simplicity we'll use CSV files to demonstrate how to define a data source.
+
+
Define file source in Airbyte
+
We'll start our ELT process by defining the data source in Airbyte:
+
+
On our Airbyte UI, click on Sources on the left menu. Then click the + New source button on the top right corner.
+
Click on the Source type dropdown and choose File. This will open a view to define our file data source.
+
Once we know the source we want to extract data from, we need to decide the destination to load it. The choice depends on what our downstream applications want to be able to do with the data. And it's also common to store data in one location (ex. data lake) and move it somewhere else (ex. data warehouse) for specific processing.
+
Set up Google BigQuery
+
Our destination will be a data warehouse since we'll want to use the data for downstream analytical and machine learning applications. We're going to use Google BigQuery which is free under Google Cloud's free tier for up to 10 GB storage and 1TB of queries (which is significantly more than we'll ever need for our purpose).
+
+
Log into your Google account and then head over to Google CLoud. If you haven't already used Google Cloud's free trial, you'll have to sign up. It's free and you won't be autocharged unless you manually upgrade your account. Once the trial ends, we'll still have the free tier which is more than plenty for us.
Project name:made-with-ml# Google will append a unique ID to the end of it
+Location:No organization
+
+
Once the project has been created, refresh the page and we should see it (along with few other default projects from Google).
+
+
# Google BigQuery projects
+├──made-with-ml-XXXXXX👈ourproject
+├──bigquery-publicdata
+├──imjasonh-storage
+└──nyc-tlc
+
+
+
Console or code
+
Most cloud providers will allow us to do everything via console but also programmatically via API, Python, etc. For example, we manually create a project but we could've also done so with code as shown here.
+
+
Define BigQuery destination in Airbyte
+
Next, we need to establish the connection between Airbyte and BigQuery so that we can load the extracted data to the destination. In order to authenticate our access to BigQuery with Airbyte, we'll need to create a service account and generate a secret key. This is basically creating an identity with certain access that we can use for verification. Follow these instructions to create a service and generate the key file (JSON). Note down the location of this file because we'll be using it throughout this lesson. For example ours is /Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json.
+
+
On our Airbyte UI, click on Destinations on the left menu. Then click the + New destination button on the top right corner.
+
Click on the Destination type dropdown and choose BigQuery. This will open a view to define our file data source.
+
Name:BigQuery
+Default Dataset ID:mlops_course# where our data will go inside our BigQuery project
+Project ID:made-with-ml-XXXXXX# REPLACE this with your Google BiqQuery Project ID
+Credentials JSON:SERVICE-ACCOUNT-KEY.json# REPLACE this with your service account JSON location
+Dataset location:US# select US or EU, all other options will not be compatible with dbt later
+
+
Click the Set up destination button and our data destination will be tested and saved.
+
+
+
+
+
+
Connections
+
So we've set up our data sources (public CSV files) and destination (Google BigQuery data warehouse) but they haven't been connected yet. To create the connection, we need to think about a few aspects.
+
Frequency
+
How often do we want to extract data from the sources and load it into the destination?
+
+
batch: extracting data in batches, usually following a schedule (ex. daily) or when an event of interest occurs (ex. new data count)
+
streaming: extracting data in a continuous stream (using tools like Kafka, Kinesis, etc.)
+
+
+
Micro-batch
+
As we keep decreasing the time between batch ingestion (ex. towards 0), do we have stream ingestion? Not exactly. Batch processing is deliberately deciding to extract data from a source at a given interval. As that interval becomes <15 minutes, it's referred to as a micro-batch (many data warehouses allow for batch ingestion every 5 minutes). However, with stream ingestion, the extraction process is continuously on and events will keep being ingested.
+
+
+
Start simple
+
In general, it's a good idea to start with batch ingestion for most applications and slowly add the complexity of streaming ingestion (and additional infrastructure). This was we can prove that downstream applications are finding value from the data source and evolving to streaming later should only improve things.
+
+
+
We'll learn more about the different system design implications of batch vs. stream in our systems design lesson.
+
+
Connecting File source to BigQuery destination
+
Now we're ready to create the connection between our sources and destination:
+
+
On our Airbyte UI, click on Connections on the left menu. Then click the + New connection button on the top right corner.
+
Under Select a existing source, click on the Source dropdown and choose Projects and click Use existing source.
+
Under Select a existing destination, click on the Destination dropdown and choose BigQuery and click Use existing destination.
+
Click the Set up connection button and our connection will be tested and saved.
+
Repeat the same for our Tags source with the same BigQuery destination.
+
+
+
Notice that our sync mode is Full refresh | Overwrite which means that every time we sync data from our source, it'll overwrite the existing data in our destination. As opposed to Full refresh | Append which will add entries from the source to bottom of the previous syncs.
+
+
+
+
+
+
Data sync
+
Our replication frequency is Manual because we'll trigger the data syncs ourselves:
+
+
On our Airbyte UI, click on Connections on the left menu. Then click the Projects <> BigQuery connection we set up earlier.
+
Press the 🔄 Sync now button and once it's completed we'll see that the projects are now in our BigQuery data warehouse.
+
Repeat the same with our Tags <> BigQuery connection.
A curated list of Monte Carlo tree search papers...
+
+
+
4
+
19
+
2020-03-03 13:54:31
+
Diffusion to Vector
+
Reference implementation of Diffusion2Vec (Com...
+
+
+
+
+
+
+
Best practices
+
With the advent of cheap storage and cloud SaaS options to manage them, it's become a best practice to store raw data into data lakes. This allows for storage of raw, potentially unstructured, data without having to justify storage with downstream applications. When we do need to transform and process the data, we can move it to a data warehouse so can perform those operations efficiently.
+
+
+
+
+
Transform
+
Once we've extracted and loaded our data, we need to transform the data so that it's ready for downstream applications. These transformations are different from the preprocessing we've seen before but are instead reflective of business logic that's agnostic to downstream applications. Common transformations include defining schemas, filtering, cleaning and joining data across tables, etc. While we could do all of these things with SQL in our data warehouse (save queries as tables or views), dbt delivers production functionality around version control, testing, documentation, packaging, etc. out of the box. This becomes crucial for maintaining observability and high quality data workflows.
+
+
+
+
+
+
Popular transformation tools include dbt, Matillion, custom jinja templated SQL, etc.
+
+
+
Note
+
In addition to data transformations, we can also process the data using large-scale analytics engines like Spark, Flink, etc.
+
+
dbt Cloud
+
Now we're ready to transform our data in our data warehouse using dbt. We'll be using a developer account on dbt Cloud (free), which provides us with an IDE, unlimited runs, etc.
+
+
We'll learn how to use the dbt-core in our orchestration lesson. Unlike dbt Cloud, dbt core is completely open-source and we can programmatically connect to our data warehouse and perform transformations.
Inside our models/labeled_projects/labeled_projects.sql file we'll create a view that joins our project data with the appropriate tags. This will create the labeled data necessary for downstream applications such as machine learning models. Here we're joining based on the matching id between the projects and tags:
We can view the queried results by clicking the Preview button and view the data lineage as well.
+
Schemas
+
Inside our models/labeled_projects/schema.yml file we'll define the schemas for each of the features in our transformed data. We also define several tests that each feature should pass. View the full list of dbt tests but note that we'll use Great Expectations for more comprehensive tests when we orchestrate all these data workflows in our orchestration lesson.
At the bottom of the IDE, we can execute runs based on the transformations we've defined. We'll run each of the following commands and once they finish, we can see the transformed data inside our data warehouse.
+
dbtrun
+dbttest
+
+
Once these commands run successfully, we're ready to move our transformations to a production environment where we can insert this view in our data warehouse.
+
Jobs
+
In order to apply these transformations to the data in our data warehouse, it's best practice to create an Environment and then define Jobs:
+
+
Click Environments on the left menu > New Environment button (top right corner) and fill out the details:
+
There is so much more to dbt so be sure to check out their official documentation to really customize any workflows. And be sure to check out our orchestration lesson where we'll programmatically create and execute our dbt transformations.
+
+
Implementations
+
Hopefully we created our data stack for the purpose of gaining some actionable insight about our business, users, etc. Because it's these use cases that dictate which sources of data we extract from, how often and how that data is stored and transformed. Downstream applications of our data typically fall into one of these categories:
+
+
dataanalytics: use cases focused on reporting trends, aggregate views, etc. via charts, dashboards, etc.for the purpose of providing operational insight for business stakeholders.
machinelearning: use cases centered around using the transformed data to construct predictive models (forecasting, personalization, etc.).
+
+
While it's very easy to extract data from our data warehouse:
+
pipinstallgoogle-cloud-bigquery==1.21.0
+
+
fromgoogle.cloudimportbigquery
+fromgoogle.oauth2importservice_account
+
+# Replace these with your own values
+project_id="made-with-ml-XXXXXX"# REPLACE
+SERVICE_ACCOUNT_KEY_JSON="/Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json"# REPLACE
+
+# Establish connection
+credentials=service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_KEY_JSON)
+client=bigquery.Client(credentials=credentials,project=project_id)
+
+# Query data
+query_job=client.query("""
+ SELECT *
+ FROM mlops_course.labeled_projects""")
+results=query_job.result()
+results.to_dataframe().head()
+
+
+
+
+
+
+
id
+
created_on
+
title
+
description
+
tag
+
+
+
+
+
0
+
1994.0
+
2020-07-29 04:51:30
+
Understanding the Effectivity of Ensembles in ...
+
The report explores the ideas presented in Dee...
+
computer-vision
+
+
+
1
+
1506.0
+
2020-06-19 06:26:17
+
Using GitHub Actions for MLOps & Data Science
+
A collection of resources on how to facilitate...
+
mlops
+
+
+
2
+
807.0
+
2020-05-11 02:25:51
+
Introduction to Machine Learning Problem Framing
+
This course helps you frame machine learning (...
+
mlops
+
+
+
3
+
1204.0
+
2020-06-05 22:56:38
+
Snaked: Classifying Snake Species using Images
+
Proof of concept that it is possible to identi...
+
computer-vision
+
+
+
4
+
1706.0
+
2020-07-04 11:05:28
+
PokeZoo
+
A deep learning based web-app developed using ...
+
computer-vision
+
+
+
+
+
+
+
Warning
+
Check out our notebook where we extract the transformed data from our data warehouse. We do this in a separate notebook because it requires the google-cloud-bigquery package and until dbt loosens it's Jinja versioning constraints... it'll have to be done in a separate environment. However, downstream applications are typically analytics or ML applications which have their own environments anyway so these conflicts are not inhibiting.
+
+
many of the analytics (ex. dashboards) and machine learning solutions (ex. feature stores) allow for easy connection to our data warehouses so that workflows can be triggered when an event occurs or on a schedule. We're going to take this a step further in the next lesson where we'll use a central orchestration platform to control all these workflows.
+
+
Analytics first, then ML
+
It's a good idea for the first several applications to be analytics and reporting based in order to establish a robust data stack. These use cases typically just involve displaying data aggregations and trends, as opposed to machine learning systems that involve additional complex infrastructure and workflows.
+
+
Observability
+
When we create complex data workflows like this, observability becomes a top priority. Data observability is the general concept of understanding the condition of data in our system and it involves:
+
+
dataquality: testing and monitoring our data quality after every step (schemas, completeness, recency, etc.).
+
datalineage: mapping the where data comes from and how it's being transformed as it moves through our pipelines.
+
discoverability: enabling discovery of the different data sources and features for downstream applications.
+
privacy+security: are the different data assets treated and restricted appropriately amongst the applications?
The data stack ecosystem to create the robust data workflows is growing and maturing. However, it can be overwhelming when it comes to choosing the best tooling options, especially as needs change over time. Here are a few important factors to consider when making a tooling decision in this space:
+
+
What is the cost per time per employee? Some of the tooling options can rack up quite the bill!
+
Does the tool have the proper connectors to integrate with our data sources and the rest of the stack?
+
Does the tool fit with our team's technical aptitude (SQL, Spark, Python, etc.)?
+
What kind of support does the tool offer (enterprise, community, etc.)?
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Data engineering - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
So far we've had the convenience of using local CSV files as data source but in reality, our data can come from many disparate sources. Additionally, our processes around transforming and testing our data should ideally be moved upstream so that many different downstream processes can benefit from them. Our ML use case being just one among the many potential downstream applications. To address these shortcomings, we're going to learn about the fundamentals of data engineering and construct a modern data stack that can scale and provide high quality data for our applications.
This process is more commonly known as ELT, but there are variants such as ETL and reverse ETL, etc. They are all essentially the same underlying workflows but have slight differences in the order of data flow and where data is processed and stored.
+
+
+
+
+
+
Utility and simplicity
+
It can be enticing to set up a modern data stack in your organization, especially with all the hype. But it's very important to motivate utility and adding additional complexity:
+
+
Start with a use case that we already have data sources for and has direct impact on the business' bottom line (ex. user churn).
+
Start with the simplest infrastructure (source → database → report) and add complexity (in infrastructure, performance and team) as needed.
+
+
+
Data systems
+
+
+
Before we start working with our data, it's important to understand the different types of systems that our data can live in. So far in this course we've worked with files, but there are several types of data systems that are widely adopted in industry for different purposes.
+
+
+
+
+
Data lake
+
A data lake is a flat data management system that stores raw objects. It's a great option for inexpensive storage and has the capability to hold all types of data (unstructured, semi-structured and structured). Object stores are becoming the standard for data lakes with default options across the popular cloud providers. Unfortunately, because data is stored as objects in a data lake, it's not designed for operating on structured data.
Another popular storage option is a database (DB), which is an organized collection of structured data that adheres to either:
+
+
relational schema (tables with rows and columns) often referred to as a Relational Database Management System (RDBMS) or SQL database.
+
non-relational (key/value, graph, etc.), often referred to as a non-relational database or NoSQL database.
+
+
A database is an online transaction processing (OLTP) system because it's typically used for day-to-day CRUD (create, read, update, delete) operations where typically information is accessed by rows. However, they're generally used to store data from one application and is not designed to hold data from across many sources for the purpose of analytics.
A data warehouse (DWH) is a type of database that's designed for storing structured data from many different sources for downstream analytics and data science. It's an online analytical processing (OLAP) system that's optimized for performing operations across aggregating column values rather than accessing specific rows.
The first step in our data pipeline is to extract data from a source and load it into the appropriate destination. While we could construct custom scripts to do this manually or on a schedule, an ecosystem of data ingestion tools have already standardized the entire process. They all come equipped with connectors that allow for extraction, normalization, cleaning and loading between sources and destinations. And these pipelines can be scaled, monitored, etc. all with very little to no code.
We're going to use the open-source tool Airbyte to create connections between our data sources and destinations. Let's set up Airbyte and define our data sources. As we progress in this lesson, we'll set up our destinations and create connections to extract and load data.
+
+
Ensure that we still have Docker installed from our Docker lesson but if not, download it here. For Windows users, be sure to have these configurations enabled.
+
In a parent directory, outside our project directory for the MLOps course, execute the following commands to load the Airbyte repository locally and launch the service.
+
Our data sources we want to extract from can be from anywhere. They could come from 3rd party apps, files, user click streams, physical devices, data lakes, databases, data warehouses, etc. But regardless of the source of our data, they type of data should fit into one of these categories:
+
+
structured: organized data stored in an explicit structure (ex. tables)
+
semi-structured: data with some structure but no formal schema or data types (web pages, CSV, JSON, etc.)
+
unstructured: qualitative data with no formal structure (text, images, audio, etc.)
+
+
For our application, we'll define two data sources:
+
+
projects.csv: data containing projects with their ID, create date, title and description.
+
tags.csv: labels for each of project IDs in projects.csv
+
+
+
Ideally, these data assets would be retrieved from a database that contains projects that we extracted and perhaps another database that stores labels from our labeling team's workflows. However, for simplicity we'll use CSV files to demonstrate how to define a data source.
+
+
Define file source in Airbyte
+
We'll start our ELT process by defining the data source in Airbyte:
+
+
On our Airbyte UI, click on Sources on the left menu. Then click the + New source button on the top right corner.
+
Click on the Source type dropdown and choose File. This will open a view to define our file data source.
+
Once we know the source we want to extract data from, we need to decide the destination to load it. The choice depends on what our downstream applications want to be able to do with the data. And it's also common to store data in one location (ex. data lake) and move it somewhere else (ex. data warehouse) for specific processing.
+
Set up Google BigQuery
+
Our destination will be a data warehouse since we'll want to use the data for downstream analytical and machine learning applications. We're going to use Google BigQuery which is free under Google Cloud's free tier for up to 10 GB storage and 1TB of queries (which is significantly more than we'll ever need for our purpose).
+
+
Log into your Google account and then head over to Google CLoud. If you haven't already used Google Cloud's free trial, you'll have to sign up. It's free and you won't be autocharged unless you manually upgrade your account. Once the trial ends, we'll still have the free tier which is more than plenty for us.
Project name:made-with-ml# Google will append a unique ID to the end of it
+Location:No organization
+
+
Once the project has been created, refresh the page and we should see it (along with few other default projects from Google).
+
+
# Google BigQuery projects
+├──made-with-ml-XXXXXX👈ourproject
+├──bigquery-publicdata
+├──imjasonh-storage
+└──nyc-tlc
+
+
+
Console or code
+
Most cloud providers will allow us to do everything via console but also programmatically via API, Python, etc. For example, we manually create a project but we could've also done so with code as shown here.
+
+
Define BigQuery destination in Airbyte
+
Next, we need to establish the connection between Airbyte and BigQuery so that we can load the extracted data to the destination. In order to authenticate our access to BigQuery with Airbyte, we'll need to create a service account and generate a secret key. This is basically creating an identity with certain access that we can use for verification. Follow these instructions to create a service and generate the key file (JSON). Note down the location of this file because we'll be using it throughout this lesson. For example ours is /Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json.
+
+
On our Airbyte UI, click on Destinations on the left menu. Then click the + New destination button on the top right corner.
+
Click on the Destination type dropdown and choose BigQuery. This will open a view to define our file data source.
+
Name:BigQuery
+Default Dataset ID:mlops_course# where our data will go inside our BigQuery project
+Project ID:made-with-ml-XXXXXX# REPLACE this with your Google BiqQuery Project ID
+Credentials JSON:SERVICE-ACCOUNT-KEY.json# REPLACE this with your service account JSON location
+Dataset location:US# select US or EU, all other options will not be compatible with dbt later
+
+
Click the Set up destination button and our data destination will be tested and saved.
+
+
+
+
+
+
Connections
+
So we've set up our data sources (public CSV files) and destination (Google BigQuery data warehouse) but they haven't been connected yet. To create the connection, we need to think about a few aspects.
+
Frequency
+
How often do we want to extract data from the sources and load it into the destination?
+
+
batch: extracting data in batches, usually following a schedule (ex. daily) or when an event of interest occurs (ex. new data count)
+
streaming: extracting data in a continuous stream (using tools like Kafka, Kinesis, etc.)
+
+
+
Micro-batch
+
As we keep decreasing the time between batch ingestion (ex. towards 0), do we have stream ingestion? Not exactly. Batch processing is deliberately deciding to extract data from a source at a given interval. As that interval becomes <15 minutes, it's referred to as a micro-batch (many data warehouses allow for batch ingestion every 5 minutes). However, with stream ingestion, the extraction process is continuously on and events will keep being ingested.
+
+
+
Start simple
+
In general, it's a good idea to start with batch ingestion for most applications and slowly add the complexity of streaming ingestion (and additional infrastructure). This was we can prove that downstream applications are finding value from the data source and evolving to streaming later should only improve things.
+
+
+
We'll learn more about the different system design implications of batch vs. stream in our systems design lesson.
+
+
Connecting File source to BigQuery destination
+
Now we're ready to create the connection between our sources and destination:
+
+
On our Airbyte UI, click on Connections on the left menu. Then click the + New connection button on the top right corner.
+
Under Select a existing source, click on the Source dropdown and choose Projects and click Use existing source.
+
Under Select a existing destination, click on the Destination dropdown and choose BigQuery and click Use existing destination.
+
Click the Set up connection button and our connection will be tested and saved.
+
Repeat the same for our Tags source with the same BigQuery destination.
+
+
+
Notice that our sync mode is Full refresh | Overwrite which means that every time we sync data from our source, it'll overwrite the existing data in our destination. As opposed to Full refresh | Append which will add entries from the source to bottom of the previous syncs.
+
+
+
+
+
+
Data sync
+
Our replication frequency is Manual because we'll trigger the data syncs ourselves:
+
+
On our Airbyte UI, click on Connections on the left menu. Then click the Projects <> BigQuery connection we set up earlier.
+
Press the 🔄 Sync now button and once it's completed we'll see that the projects are now in our BigQuery data warehouse.
+
Repeat the same with our Tags <> BigQuery connection.
A curated list of Monte Carlo tree search papers...
+
+
+
4
+
19
+
2020-03-03 13:54:31
+
Diffusion to Vector
+
Reference implementation of Diffusion2Vec (Com...
+
+
+
+
+
+
+
Best practices
+
With the advent of cheap storage and cloud SaaS options to manage them, it's become a best practice to store raw data into data lakes. This allows for storage of raw, potentially unstructured, data without having to justify storage with downstream applications. When we do need to transform and process the data, we can move it to a data warehouse so can perform those operations efficiently.
+
+
+
+
+
Transform
+
Once we've extracted and loaded our data, we need to transform the data so that it's ready for downstream applications. These transformations are different from the preprocessing we've seen before but are instead reflective of business logic that's agnostic to downstream applications. Common transformations include defining schemas, filtering, cleaning and joining data across tables, etc. While we could do all of these things with SQL in our data warehouse (save queries as tables or views), dbt delivers production functionality around version control, testing, documentation, packaging, etc. out of the box. This becomes crucial for maintaining observability and high quality data workflows.
+
+
+
+
+
+
Popular transformation tools include dbt, Matillion, custom jinja templated SQL, etc.
+
+
+
Note
+
In addition to data transformations, we can also process the data using large-scale analytics engines like Spark, Flink, etc.
+
+
dbt Cloud
+
Now we're ready to transform our data in our data warehouse using dbt. We'll be using a developer account on dbt Cloud (free), which provides us with an IDE, unlimited runs, etc.
+
+
We'll learn how to use the dbt-core in our orchestration lesson. Unlike dbt Cloud, dbt core is completely open-source and we can programmatically connect to our data warehouse and perform transformations.
Inside our models/labeled_projects/labeled_projects.sql file we'll create a view that joins our project data with the appropriate tags. This will create the labeled data necessary for downstream applications such as machine learning models. Here we're joining based on the matching id between the projects and tags:
We can view the queried results by clicking the Preview button and view the data lineage as well.
+
Schemas
+
Inside our models/labeled_projects/schema.yml file we'll define the schemas for each of the features in our transformed data. We also define several tests that each feature should pass. View the full list of dbt tests but note that we'll use Great Expectations for more comprehensive tests when we orchestrate all these data workflows in our orchestration lesson.
At the bottom of the IDE, we can execute runs based on the transformations we've defined. We'll run each of the following commands and once they finish, we can see the transformed data inside our data warehouse.
+
dbtrun
+dbttest
+
+
Once these commands run successfully, we're ready to move our transformations to a production environment where we can insert this view in our data warehouse.
+
Jobs
+
In order to apply these transformations to the data in our data warehouse, it's best practice to create an Environment and then define Jobs:
+
+
Click Environments on the left menu > New Environment button (top right corner) and fill out the details:
+
There is so much more to dbt so be sure to check out their official documentation to really customize any workflows. And be sure to check out our orchestration lesson where we'll programmatically create and execute our dbt transformations.
+
+
Implementations
+
Hopefully we created our data stack for the purpose of gaining some actionable insight about our business, users, etc. Because it's these use cases that dictate which sources of data we extract from, how often and how that data is stored and transformed. Downstream applications of our data typically fall into one of these categories:
+
+
dataanalytics: use cases focused on reporting trends, aggregate views, etc. via charts, dashboards, etc.for the purpose of providing operational insight for business stakeholders.
machinelearning: use cases centered around using the transformed data to construct predictive models (forecasting, personalization, etc.).
+
+
While it's very easy to extract data from our data warehouse:
+
pipinstallgoogle-cloud-bigquery==1.21.0
+
+
fromgoogle.cloudimportbigquery
+fromgoogle.oauth2importservice_account
+
+# Replace these with your own values
+project_id="made-with-ml-XXXXXX"# REPLACE
+SERVICE_ACCOUNT_KEY_JSON="/Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json"# REPLACE
+
+# Establish connection
+credentials=service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_KEY_JSON)
+client=bigquery.Client(credentials=credentials,project=project_id)
+
+# Query data
+query_job=client.query("""
+ SELECT *
+ FROM mlops_course.labeled_projects""")
+results=query_job.result()
+results.to_dataframe().head()
+
+
+
+
+
+
+
id
+
created_on
+
title
+
description
+
tag
+
+
+
+
+
0
+
1994.0
+
2020-07-29 04:51:30
+
Understanding the Effectivity of Ensembles in ...
+
The report explores the ideas presented in Dee...
+
computer-vision
+
+
+
1
+
1506.0
+
2020-06-19 06:26:17
+
Using GitHub Actions for MLOps & Data Science
+
A collection of resources on how to facilitate...
+
mlops
+
+
+
2
+
807.0
+
2020-05-11 02:25:51
+
Introduction to Machine Learning Problem Framing
+
This course helps you frame machine learning (...
+
mlops
+
+
+
3
+
1204.0
+
2020-06-05 22:56:38
+
Snaked: Classifying Snake Species using Images
+
Proof of concept that it is possible to identi...
+
computer-vision
+
+
+
4
+
1706.0
+
2020-07-04 11:05:28
+
PokeZoo
+
A deep learning based web-app developed using ...
+
computer-vision
+
+
+
+
+
+
+
Warning
+
Check out our notebook where we extract the transformed data from our data warehouse. We do this in a separate notebook because it requires the google-cloud-bigquery package and until dbt loosens it's Jinja versioning constraints... it'll have to be done in a separate environment. However, downstream applications are typically analytics or ML applications which have their own environments anyway so these conflicts are not inhibiting.
+
+
many of the analytics (ex. dashboards) and machine learning solutions (ex. feature stores) allow for easy connection to our data warehouses so that workflows can be triggered when an event occurs or on a schedule. We're going to take this a step further in the next lesson where we'll use a central orchestration platform to control all these workflows.
+
+
Analytics first, then ML
+
It's a good idea for the first several applications to be analytics and reporting based in order to establish a robust data stack. These use cases typically just involve displaying data aggregations and trends, as opposed to machine learning systems that involve additional complex infrastructure and workflows.
+
+
Observability
+
When we create complex data workflows like this, observability becomes a top priority. Data observability is the general concept of understanding the condition of data in our system and it involves:
+
+
dataquality: testing and monitoring our data quality after every step (schemas, completeness, recency, etc.).
+
datalineage: mapping the where data comes from and how it's being transformed as it moves through our pipelines.
+
discoverability: enabling discovery of the different data sources and features for downstream applications.
+
privacy+security: are the different data assets treated and restricted appropriately amongst the applications?
The data stack ecosystem to create the robust data workflows is growing and maturing. However, it can be overwhelming when it comes to choosing the best tooling options, especially as needs change over time. Here are a few important factors to consider when making a tooling decision in this space:
+
+
What is the cost per time per employee? Some of the tooling options can rack up quite the bill!
+
Does the tool have the proper connectors to integrate with our data sources and the rest of the stack?
+
Does the tool fit with our team's technical aptitude (SQL, Spark, Python, etc.)?
+
What kind of support does the tool offer (enterprise, community, etc.)?
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Data Stack for Machine Learning - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
So far we've performed our data processing operations on a single machine. Our dataset was able to fit into a single Pandas DataFrame and we were able to perform our operations in a single Python process. But what if our dataset was too large to fit into a single machine? We would need to distribute our data processing operations across multiple machines. And with the increasing trend in ML for larger unstructured datasets and larger models (LLMs), we can quickly outgrow our single machine constraints and will need to go distributed.
+
+
Note
+
Our dataset is intentionally small for this course so that we can quickly execute the code. But with our distributed set up in this lesson, we can easily switch to a mcuh larger dataset and the code will continue to execute perfectly. And if we add more compute resources, we can scale our data processing operations to be even faster with no changes to our code.
+
+
Implementation
+
There are many frameworks for distributed computing, such as Ray, Dask, Modin, Spark, etc. All of these are great options but for our application we want to choose a framework that is will allow us to scale our data processing operations with minimal changes to our existing code and all in Python. We also want to choose a framework that will integrate well when we want to distributed our downstream workloads (training, tuning, serving, etc.).
+
To address these needs, we'll be using Ray, a distributed computing framework that makes it easy to scale your Python applications. It's a general purpose framework that can be used for a variety of applications but we'll be using it for our data processing operations first (and more later). And it also has great integrations with the previously mentioned distributed data processing frameworks (Dask, Modin, Spark).
+
+
+
+
+
Setup
+
The only setup we have to do is set Ray to preserve order when acting on our data. This is important for ensuring reproducible and deterministic results.
We'll start by ingesting our dataset. Ray has a range of input/output functions that supports all major data formats and sources.
+
1
+2
+3
+4
# Data ingestion
+ds=ray.data.read_csv(DATASET_LOC)
+ds=ds.random_shuffle(seed=1234)
+ds.take(1)
+
+
+[{'id': 2166,
+ 'created_on': datetime.datetime(2020, 8, 17, 5, 19, 41),
+ 'title': 'Pix2Pix',
+ 'description': 'Tensorflow 2.0 Implementation of the paper Image-to-Image Translation using Conditional GANs by Philip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei A. Efros.',
+ 'tag': 'computer-vision'}]
+
+
+
Splitting
+
Next, we'll split our dataset into our training and validation splits. Ray has a built-in train_test_split function but we're using a modified version so that we can stratify our split based on the tag column.
And finally, we're ready to preprocess our data splits. One of the advantages of using Ray is that we won't have to change anything to our original Pandas-based preprocessing function we implemented in the previous lesson. Instead, we can use it directly with Ray's map_batches utility to map our preprocessing function across batches in our data in a distributed manner.
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
The last step in achieving reproducibility is to deploy our versioned code and artifacts in a reproducible environment. This goes well beyond the virtual environment we configured for our Python applications because there are system-level specifications (operating system, required implicit packages, etc.) we aren't capturing. We want to be able to encapsulate all the requirements we need so that there are no external dependencies that would prevent someone else from reproducing our exact application.
+
Docker
+
There are actually quite a few solutions for system-level reproducibility (VMs, container engines, etc.) but the Docker container engine is by far the most popular for several key advantages:
+
+
reproducibility via Dockerfile with explicit instructions to deploy our application in a specific system.
+
isolation via containers as to not affect other applications that may also run on the same underlying operating system.
+
and many more advantages including size (no separate OS needed for each application), speed, Docker Hub, etc.
+
+
We're going to use Docker to deploy our application locally in an isolated, reproducible and scalable fashion. Once we do this, any machine with the Docker engine installed can reproduce our work. However, there is so much more to Docker, which you can explore in the docs, that goes beyond what we'll need.
+
Architecture
+
Before we install Docker, let's take a look at how the container engine works on top our operating system, which can be our local hardware or something managed on the cloud.
+
+
+
+
+
The Docker container engine is responsible for spinning up configured containers, which contains our application and it's dependencies (binaries, libraries, etc.). The container engine is very efficient in that it doesn't need to create a separate operating system for each containerized application. This also means that our containers can share the system's resources via the Docker engine.
+
Set up
+
Now we're ready to install Docker based on our operating system. Once installed, we can start the Docker Desktop which will allow us to create and deploy our containerized applications.
+
docker--version
+
+
+Docker version 20.10.8, build 3967b7d
+
+
+
Images
+
The first step is to build a docker image which has the application and all it's specified dependencies. We can create this image using a Dockerfile which outlines a set of instructions. These instructions essentially build read-only image layers on top of each other to construct our entire image. Let's take a look at our application's Dockerfile and the image layers it creates.
+
Dockerfile
+
We'll start by creating a Dockerfile:
+
touchDockerfile
+
+
The first line we'll write in our Dockerfile specifies the base image we want to pull FROM. Here we want to use the base image for running Python based applications and specifically for Python 3.7 with the slim variant. Since we're only deploying a Python application, this slim variant with minimal packages satisfies our requirements while keeping the size of the image layer low.
+
# Base image
+FROMpython:3.7-slim
+
+
Next we're going to install our application dependencies. First, we'll COPY the required files from our local file system so we can use them for installation. Alternatively, if we were running on some remote infrastructure, we could've pulled from a remote git host. Once we have our files, we can install the packages required to install our application's dependencies using the RUN command. Once we're done using the packages, we can remove them to keep our image layer's size to a minimum.
Since our application (API) requires PORT 8000 to be open, we need to specify in our Dockerfile to expose it.
+
# Export ports
+EXPOSE8000
+
+
The final step in building our image is to specify the executable to be run when a container is built from our image. For our application, we want to launch our API with gunicorn since this Dockerfile may be used to deploy our service to production at scale.
There are many more commands available for us to use in the Dockerfile, such as using environment variables (ENV) and arguments (ARG), command arguments (CMD), specifying volumes (VOLUME), setting the working directory (WORKDIR) and many more, all of which you can explore through the official docs.
+
+
Build images
+
Once we're done composing the Dockerfile, we're ready to build our image using the build command which allows us to add a tag and specify the location of the Dockerfile to use.
+
dockerbuild-ttagifai:latest-fDockerfile.
+
+
We can inspect all built images and their attributes like so:
+
dockerimages
+
+
+REPOSITORY TAG IMAGE ID CREATED SIZE
+tagifai latest 02c88c95dd4c 23 minutes ago 2.57GB
+
+
+
We can also remove any or all images based on their unique IDs.
+
dockerrmi<IMAGE_ID># remove an image
+dockerrmi$(dockerimages-a-q)# remove all images
+
+
Run containers
+
Once we've built our image, we're ready to run a container using that image with the run command which allows us to specify the image, port forwarding, etc.
+
dockerrun-p8000:8000--nametagifaitagifai:latest
+
+
Once we have our container running, we can use the API thanks for the port we're sharing (8000):
+
curl-X'POST'\
+'http://localhost:8000/predict'\
+-H'accept: application/json'\
+-H'Content-Type: application/json'\
+-d'{
+ "texts": [
+ {
+ "text": "Transfer learning with transformers for text classification."
+ }
+ ]
+}'
+
+
We can inspect all containers (running or stopped) like so:
+
+CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
+ee5f1b08abd5 tagifai:latest "gunicorn -c config…" 19 minutes ago Created 0.0.0.0:8000->8000/tcp tagifai
+
+
+
We can also stop and remove any or all containers based on their unique IDs:
+
dockerstop<CONTAINER_ID># stop a running container
+dockerrm<CONTAINER_ID># remove a container
+dockerstop$(dockerps-a-q)# stop all containers
+dockerrm$(dockerps-a-q)# remove all containers
+
+
+
If our application required multiple containers for different services (API, database, etc.) then we can bring them all up at once using the docker compose functionality and scale and manage them using a container orchestration system like Kubernetes (K8s). If we're specifically deploying ML workflows, we can use a toolkit like KubeFlow to help us manage and scale.
+
+
Debug
+
In the event that we run into errors while building our image layers, a very easy way to debug the issue is to run the container with the image layers that have been build so far. We can do this by only including the commands that have ran successfully so far (and all COPY statements) in the Dockerfile. And then we need to rebuild the image (since we altered the Dockerfile) and run the container:
Once we have our container running, we can use our application as we would on our local machine but now it's reproducible on any operating system that can run the Docker container engine. We've covered just what we need from Docker to deploy our application but there is so much more to Docker, which you can explore in the docs.
+
Production
+
This Dockerfile is commonly the end artifact a data scientist or ML engineer delivers to their DevOps teams to deploy and scale their services, with a few changes:
+
+
data assets would be pulled from a remote storage location (ex. S3).
+
model artifacts would be loaded from a remote model registry.
+
code would be loaded from a remote repository (ex. GitHub) via git clone.
+
+
All of these changes would involve using the proper credentials (via encrypted secrets and can even be automatically deployed via CI/CD workflows. But, of course, there are subsequent responsibilities such as monitoring.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Docker - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
+
Code tells you how, comments tell you why. -- Jeff Atwood
+
+
We can really improve the quality of our codebase by documenting it to make it easier for others (and our future selves) to easily navigate and extend it. We know our code base best the moment we finish writing it but fortunately documenting it will allow us to quickly get back to that familiar state of mind. Documentation can mean many different things to developers, so let's define the most common components:
+
+
comments: short descriptions as to why a piece of code exists.
+
typing: specification of a function's inputs and outputs' data types, providing information pertaining to what a function consumes and produces.
+
docstrings: meaningful descriptions for functions and classes that describe overall utility, arguments, returns, etc.
+
docs: rendered webpage that summarizes all the functions, classes, workflows, examples, etc.
+
+
Typing
+
It's important to be as explicit as possible with our code. We've already discussed choosing explicit names for variables, functions but another way we can be explicit is by defining the types for our function's inputs and outputs by using the typing library.
+
So far, our functions have looked like this:
+
1
+2
defsome_function(a,b):
+ returnc
+
+
But we can incorporate so much more information using typing:
+
input parameter b is an integer with default value 0
+
output parameter c is a NumPy array
+
+
There are many other data types that we can work with, including List, Set, Dict, Tuple, Sequence and more, as well as included types such as int, float, etc. You can also use types from packages we install (ex. np.ndarray) and even from our own defined classes (ex. LabelEncoder).
+
+
Starting from Python 3.9+, common types are built in so we don't need to import them with from typing import List, Set, Dict, Tuple, Sequence anymore.
+
+
Docstrings
+
We can make our code even more explicit by adding docstrings to describe overall utility, arguments, returns, exceptions and more. Let's take a look at an example:
# madewithml/data.py
+fromtypingimportList
+
+defclean_text(text:str,stopwords:List=STOPWORDS)->str:
+"""Clean raw text string.
+
+ Args:
+ text (str): Raw text to clean.
+ stopwords (List, optional): list of words to filter out. Defaults to STOPWORDS.
+
+ Returns:
+ str: cleaned text.
+ """
+ pass
+
+
+
Tip
+
If using Visual Studio Code, be sure to use the Python Docstrings Generator extension so you can type """ under a function and then hit the Shift key to generate a template docstring. It will autofill parts of the docstring using the typing information and even exception in your code!
+
+
+
Docs
+
So we're going through all this effort of including typing and docstrings to our functions but it's all tucked away inside our scripts. What if we can collect all this effort and automatically surface it as documentation? Well that's exactly what we'll do with the following open-source packages → final result here.
+
+
+
Initialize mkdocs
+
python3-mmkdocsnew.
+
+This will create the following files:
+
.
+├─docs/
+│└─index.md
+└─mkdocs.yml
+
+
+
+
We'll start by overwriting the default index.md file in our docs directory with information specific to our project:
+
index.md
1
+2
+3
+4
+5
+6
+7
+8
## Documentation
+-[madewithml](madewithml/config.md): documentation for functions and classes.
+
+## Course
+Learn how to combine machine learning with software engineering to design, develop, deploy and iterate on production ML applications.
+
+-Lessons: [https://madewithml.com/](https://madewithml.com/#course)
+-Code: [GokuMohandas/Made-With-ML](https://github.com/GokuMohandas/Made-With-ML)
+
+
+
+
Next we'll create documentation files for each script in our madewithml directory:
+
It's helpful to have the docs directory structure mimic our project's structure as much as possible.
+
+
+
+
Next we'll add madewithml.<SCRIPT_NAME> to each file under docs/madewithml. This will populate the file with information about the functions and classes (using their docstrings) from madewithml/<SCRIPT_NAME>.py thanks to the mkdocstrings plugin.
+
+
Be sure to check out the complete list of mkdocs plugins.
+
# docs/madewithml/data.md
+:::madewithml.data
+
+
+
+
+
Finally, we'll add some configurations to our mkdocs.yml file that mkdocs automatically created:
+
site_name:Made With ML
+site_url:https://madewithml.com/
+repo_url:https://github.com/GokuMohandas/Made-With-ML/
+nav:
+-Home:index.md
+-madewithml:
+-data:madewithml/data.md
+-models:madewithml/models.md
+-train:madewithml/train.md
+-tune:madewithml/tune.md
+-evaluate:madewithml/evaluate.md
+-predict:madewithml/predict.md
+-serve:madewithml/serve.md
+-utils:madewithml/utils.md
+theme:readthedocs
+plugins:
+-mkdocstrings
+watch:
+-.# reload docs for any file changes
+
We can easily serve our documentation for free using GitHub pages for public repositories as wells as private documentation for private repositories. And we can even host it on a custom domain (ex. company's subdomain).
+
+
Be sure to check out the auto-generated documentation page for our repository. We'll learn how to automatically generate and update this docs page every time we make changes to our codebase later in our CI/CD lesson.
+
+
In the next lesson, we'll learn how to style and format our codebase in a consistent manner.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Documentation - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Evaluation is an integral part of modeling and it's one that's often glossed over. We'll often find evaluation to involve simply computing the accuracy or other global metrics but for many real work applications, a much more nuanced evaluation process is required. However, before evaluating our model, we always want to:
+
+
be clear about what metrics we are prioritizing
+
be careful not to over optimize on any one metric because it may mean you're compromising something else
+
+
Setup
+
Let's start by setting up our metrics dictionary that we'll fill in as we go along and all the data we'll need for evaluation: grounds truth labels (y_test, predicted labels (y_pred) and predicted probabilities (y_prob).
While we were developing our models, our evaluation process involved computing the coarse-grained metrics such as overall precision, recall and f1 metrics.
+
+
True positives (TP): we correctly predicted class X.
+
False positives (FP): we incorrectly predicted class X but it was another class.
+
True negatives (TN): we correctly predicted that it's wasn't the class X.
+
False negatives (FN): we incorrectly predicted that it wasn't the class X but it was.
The precision_recall_fscore_support() function from scikit-learn has an input parameter called average which has the following options below. We'll be using the different averaging methods for different metric granularities.
+
+
None: metrics are calculated for each unique class.
+
binary: used for binary classification tasks where the pos_label is specified.
+
micro: metrics are calculated using global TP, FP, and FN.
+
macro: per-class metrics which are averaged without accounting for class imbalance.
+
weighted: per-class metrics which are averaged by accounting for class imbalance.
+
samples: metrics are calculated at the per-sample level.
+
+
+
Fine-grained
+
Inspecting these coarse-grained, overall metrics is a start but we can go deeper by evaluating the same fine-grained metrics at the categorical feature levels.
Besides just inspecting the metrics for each class, we can also identify the true positives, false positives and false negatives. Each of these will give us insight about our model beyond what the metrics can provide.
+
+
True positives (TP): learn about where our model performs well.
+
False positives (FP): potentially identify samples which may need to be relabeled.
+
False negatives (FN): identify the model's less performant areas to oversample later.
+
+
+
It's a good to have our FP/FN samples feed back into our annotation pipelines in the event we want to fix their labels and have those changes be reflected everywhere.
+=== True positives ===
+Mention Classifier Category prediction model
+This repo contains AllenNLP model for prediction of Named Entity categories by its mentions.
+ true: natural-language-processing
+ pred: natural-language-processing
+
+Finetune: Scikit-learn Style Model Finetuning for NLP Finetune is a library that allows users to leverage state-of-the-art pretrained NLP models for a wide variety of downstream tasks.
+ true: natural-language-processing
+ pred: natural-language-processing
+
+Finetuning Transformers with JAX + Haiku Walking through a port of the RoBERTa pre-trained model to JAX + Haiku, then fine-tuning the model to solve a downstream task.
+ true: natural-language-processing
+ pred: natural-language-processing
+
+
+=== False positives ===
+How Docker Can Help You Become A More Effective Data Scientist A look at Docker from the perspective of a data scientist.
+ true: mlops
+ pred: natural-language-processing
+
+Transfer Learning & Fine-Tuning With Keras Your 100% up-to-date guide to transfer learning & fine-tuning with Keras.
+ true: computer-vision
+ pred: natural-language-processing
+
+Exploratory Data Analysis on MS COCO Style Datasets A Simple Toolkit to do exploratory data analysis on MS COCO style formatted datasets.
+ true: computer-vision
+ pred: natural-language-processing
+
+
+=== False negatives ===
+The Unreasonable Effectiveness of Recurrent Neural Networks A close look at how RNNs are able to perform so well.
+ true: natural-language-processing
+ pred: other
+
+Machine Learning Projects This Repo contains projects done by me while learning the basics. All the familiar types of regression, classification, and clustering methods have been used.
+ true: natural-language-processing
+ pred: other
+
+BERT Distillation with Catalyst How to distill BERT with Catalyst.
+ true: natural-language-processing
+ pred: mlops
+
+
+
+
+
Tip
+
It's a really good idea to do this kind of analysis using our rule-based approach to catch really obvious labeling errors.
+
+
Confidence learning
+
While the confusion-matrix sample analysis was a coarse-grained process, we can also use fine-grained confidence based approaches to identify potentially mislabeled samples. Here we’re going to focus on the specific labeling quality as opposed to the final model predictions.
+
Simple confidence based techniques include identifying samples whose:
+
+
+
Categorical
+
+
prediction is incorrect (also indicate TN, FP, FN)
+
confidence score for the correct class is below a threshold
+
confidence score for an incorrect class is above a threshold
+
standard deviation of confidence scores over top N samples is low
+
different predictions from same model using different parameters
+
+
+
+
Continuous
+
+
difference between predicted and ground-truth values is above some %
+
+
+
+
1
+2
+3
+4
# Tag to inspect
+tag="natural-language-processing"
+index=class_to_index[tag]
+indices=np.where(y_test==index)[0]
+
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
# Confidence score for the correct class is below a threshold
+low_confidence=[]
+min_threshold=0.5
+foriinindices:
+ prob=y_prob[i][index]
+ ifprob<=0.5:
+ low_confidence.append({
+ "text":f"{test_df.iloc[i].text}",
+ "true":test_df.tag[i],
+ "pred":test_df.prediction[i],
+ "prob":prob})
+
+
1
low_confidence[0:3]
+
+
+[{'text': 'The Unreasonable Effectiveness of Recurrent Neural Networks A close look at how RNNs are able to perform so well.',
+ 'true': 'natural-language-processing',
+ 'pred': 'other',
+ 'prob': 0.0023471832},
+ {'text': 'Machine Learning Projects This Repo contains projects done by me while learning the basics. All the familiar types of regression, classification, and clustering methods have been used.',
+ 'true': 'natural-language-processing',
+ 'pred': 'other',
+ 'prob': 0.0027675298},
+ {'text': 'BERT Distillation with Catalyst How to distill BERT with Catalyst.',
+ 'true': 'natural-language-processing',
+ 'pred': 'mlops',
+ 'prob': 0.37908182}]
+
+
+
But these are fairly crude techniques because neural networks are easily overconfident and so their confidences cannot be used without calibrating them.
Assumption: “the probability associated with the predicted class label should reflect its ground truth correctness likelihood.”
+
Reality: “modern (large) neural networks are no longer well-calibrated”
+
Solution: apply temperature scaling (extension of Platt scaling) on model outputs
+
+
Recent work on confident learning (cleanlab) focuses on identifying noisy labels (with calibration), which can then be properly relabeled and used for training.
Not all of these are necessarily labeling errors but situations where the predicted probabilities were not so confident. Therefore, it will be useful to attach the predicted outcomes along side results. This way, we can know if we need to relabel, upsample, etc. as mitigation strategies to improve our performance.
+
+
The operations in this section can be applied to entire labeled dataset to discover labeling errors via confidence learning.
+
+
Slicing
+
Just inspecting the overall and class metrics isn't enough to deploy our new version to production. There may be key slices of our dataset that we need to do really well on:
+
+
Target / predicted classes (+ combinations)
+
Features (explicit and implicit)
+
Metadata (timestamps, sources, etc.)
+
Priority slices / experience (minority groups, large users, etc.)
+
+
An easy way to create and evaluate slices is to define slicing functions.
@slicing_function()
+defnlp_llm(x):
+"""NLP projects that use LLMs."""
+ nlp_project="natural-language-processing"inx.tag
+ llm_terms=["transformer","llm","bert"]
+ llm_project=any(s.lower()inx.text.lower()forsinllm_terms)
+ return(nlp_projectandllm_project)
+
+
1
+2
+3
+4
@slicing_function()
+defshort_text(x):
+"""Projects with short titles and descriptions."""
+ returnlen(x.text.split())<8# less than 8 words
+
+
Here we're using Snorkel's slicing_function to create our different slices. We can visualize our slices by applying this slicing function to a relevant DataFrame using slice_dataframe.
We can define even more slicing functions and create a slices record array using the PandasSFApplier. The slices array has N (# of data points) items and each item has S (# of slicing functions) items, indicating whether that data point is part of that slice. Think of this record array as a masking layer for each slicing function on our data.
To calculate metrics for our slices, we could use snorkel.analysis.Scorer but we've implemented a version that will work for multiclass or multilabel scenarios.
Slicing can help identify sources of bias in our data. For example, our model has most likely learned to associated algorithms with certain applications such as CNNs used for computer vision or transformers used for NLP projects. However, these algorithms are not being applied beyond their initial use cases. We’d need ensure that our model learns to focus on the application over algorithm. This could be learned with:
+
+
enough data (new or oversampling incorrect predictions)
+
masking the algorithm (using text matching heuristics)
+
+
Interpretability
+
Besides just comparing predicted outputs with ground truth values, we can also inspect the inputs to our models. What aspects of the input are more influential towards the prediction? If the focus is not on the relevant features of our input, then we need to explore if there is a hidden pattern we're missing or if our model has learned to overfit on the incorrect features. We can use techniques such as SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations) to inspect feature importance. On a high level, these techniques learn which features have the most signal by assessing the performance in their absence. These inspections can be performed on a global level (ex. per-class) or on a local level (ex. single prediction).
We can also use model-specific approaches to interpretability we we did in our embeddings lesson to identify the most influential n-grams in our text.
+
+
Behavioral testing
+
Besides just looking at metrics, we also want to conduct some behavioral sanity tests. Behavioral testing is the process of testing input data and expected outputs while treating the model as a black box. They don't necessarily have to be adversarial in nature but more along the types of perturbations we'll see in the real world once our model is deployed. A landmark paper on this topic is Beyond Accuracy: Behavioral Testing of NLP Models with CheckList which breaks down behavioral testing into three types of tests:
+
+
invariance: Changes should not affect outputs.
+
1
+2
+3
+4
# INVariance via verb injection (changes should not affect outputs)
+tokens=["revolutionized","disrupted"]
+texts=[f"Transformers applied to NLP have {token} the ML field."fortokenintokens]
+[preprocessor.index_to_class[y_prob.argmax()]fory_probinclassifier_fn(texts=texts)]
+
minimumfunctionality: Simple combination of inputs and expected outputs.
+
1
+2
+3
+4
# Minimum Functionality Tests (simple input/output pairs)
+tokens=["natural language processing","mlops"]
+texts=[f"{token} is the next big wave in machine learning."fortokenintokens]
+[preprocessor.index_to_class[y_prob.argmax()]fory_probinclassifier_fn(texts=texts)]
+
+
+
+['natural-language-processing', 'mlops']
+
+
+
+
We'll learn how to systematically create tests in our testing lesson.
+
+
Online evaluation
+
Once we've evaluated our model's ability to perform on a static dataset we can run several types of online evaluation techniques to determine performance on actual production data. It can be performed using labels or, in the event we don't readily have labels, proxy signals.
+
+
manually label a subset of incoming data to evaluate periodically.
+
asking the initial set of users viewing a newly categorized content if it's correctly classified.
+
allow users to report misclassified content by our model.
+
+
And there are many different experimentation strategies we can use to measure real-time performance before committing to replace our existing version of the system.
+
AB tests
+
AB testing involves sending production traffic to our current system (control group) and the new version (treatment group) and measuring if there is a statistical difference between the values for two metrics. There are several common issues with AB testing such as accounting for different sources of bias, such as the novelty effect of showing some users the new system. We also need to ensure that the same users continue to interact with the same systems so we can compare the results without contamination.
+
+
+
+
+
+
In many cases, if we're simply trying to compare the different versions for a certain metric, AB testing can take while before we reach statical significance since traffic is evenly split between the different groups. In this scenario, multi-armed bandits will be a better approach since they continuously assign traffic to the better performing version.
+
+
Canary tests
+
Canary tests involve sending most of the production traffic to the currently deployed system but sending traffic from a small cohort of users to the new system we're trying to evaluate. Again we need to make sure that the same users continue to interact with the same system as we gradually roll out the new system.
+
+
+
+
+
Shadow tests
+
Shadow testing involves sending the same production traffic to the different systems. We don't have to worry about system contamination and it's very safe compared to the previous approaches since the new system's results are not served. However, we do need to ensure that we're replicating as much of the production system as possible so we can catch issues that are unique to production early on. But overall, shadow testing is easy to monitor, validate operational consistency, etc.
+
+
+
+
+
+
What can go wrong?
+
If shadow tests allow us to test our updated system without having to actually serve the new results, why doesn't everyone adopt it?
+
+Show answer
+
With shadow deployment, we'll miss out on any live feedback signals (explicit/implicit) from our users since users are not directly interacting with the product using our new version.
+
We also need to ensure that we're replicating as much of the production system as possible so we can catch issues that are unique to production early on. This is rarely possible because, while your ML system may be a standalone microservice, it ultimately interacts with an intricate production environment that has many dependencies.
+
+
+
Capability vs. alignment
+
We've seen the many different metrics that we'll want to calculate when it comes to evaluating our model but not all metrics mean the same thing. And this becomes very important when it comes to choosing the "best" model(s).
+
+
capability: the ability of our model to perform a task, measured by the objective function we optimize for (ex. log loss)
+
alignment: desired behavior of our model, measure by metrics that are not differentiable or don't account for misclassifications and probability differences (ex. accuracy, precision, recall, etc.)
+
+
While capability (ex. loss) and alignment (ex. accuracy) metrics may seem to be aligned, their differences can indicate issues in our data:
+
+
↓ accuracy, ↑ loss = large errors on lots of data (worst case)
+
↓ accuracy, ↓ loss = small errors on lots of data, distributions are close but tipped towards misclassifications (misaligned)
+
↑ accuracy, ↑ loss = large errors on some data (incorrect predictions have very skewed distributions)
+
↑ accuracy, ↓ loss = no/few errors on some data (best case)
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
So far, we've been training and evaluating our different baselines but haven't really been tracking these experiments. We'll fix this but defining a proper process for experiment tracking which we'll use for all future experiments (including hyperparameter optimization). Experiment tracking is the process of managing all the different experiments and their components, such as parameters, metrics, models and other artifacts and it enables us to:
+
+
Organize all the necessary components of a specific experiment. It's important to have everything in one place and know where it is so you can use them later.
+
Reproduce past results (easily) using saved experiments.
+
Log iterative improvements across time, data, ideas, teams, etc.
+
+
Tools
+
There are many options for experiment tracking but we're going to use MLFlow (100% free and open-source) because it has all the functionality we'll need. We can run MLFlow on our own servers and databases so there are no storage cost / limitations, making it one of the most popular options and is used by Microsoft, Facebook, Databricks and others. There are also several popular options such as a Comet ML (used by Google AI, HuggingFace, etc.), Neptune (used by Roche, NewYorker, etc.), Weights and Biases (used by Open AI, Toyota Research, etc.). These are fully managed solutions that provide features like dashboards, reports, etc.
+
Setup
+
We'll start by setting up our model registry where all of our experiments and their artifacts will be stores.
In this course, our MLflow artifact and backend store will both be on our local machine. In a production setting, these would be remote such as S3 for the artifact store and a database service (ex. PostgreSQL RDS) as our backend store.
+
+
Integration
+
While we could use MLflow directly to log metrics, artifacts and parameters:
+
1
+2
+3
+4
# Example mlflow calls
+mlflow.log_metrics({"train_loss":train_loss,"val_loss":val_loss},step=epoch)
+mlflow.log_artifacts(dir)
+mlflow.log_params(config)
+
+
We'll instead use Ray to integrate with MLflow. Specifically we'll use the MLflowLoggerCallback which will automatically log all the necessary components of our experiments to the location specified in our MLFLOW_TRACKING_URI. We of course can still use MLflow directly if we want to log something that's not automatically logged by the callback. And if we're using other experiment trackers, Ray has integrations for those as well.
Once we have the callback defined, all we have to do is update our RunConfig to include it.
+
1
+2
+3
+4
+5
# Run configuration with MLflow callback
+run_config=RunConfig(
+ callbacks=[mlflow_callback],
+ checkpoint_config=checkpoint_config,
+)
+
+
Training
+
With our updated RunConfig, with the MLflow callback, we can now train our model and all the necessary components will be logged to MLflow. This is the exact same training workflow we've been using so far from the training lesson.
We're going to use the search_runs function from the MLflow python API to identify the best run in our experiment so far (we' only done one run so far so it will be the run from above).
Once we're done training, we can use the MLflow dashboard to visualize our results. To do so, we'll use the mlflow server command to launch the MLflow dashboard and navigate to the experiment we just created.
If you're on Anyscale Workspaces, then we need to first expose the port of the MLflow server. Run the following command on your Anyscale Workspace terminal to generate the public URL to your MLflow server.
We're going to create a small utility function that uses an MLflow run's artifact path to load a Ray Result object. We'll then use the Result object to load the best checkpoint.
+
1
+2
+3
+4
defget_best_checkpoint(run_id):
+ artifact_dir=urlparse(mlflow.get_run(run_id).info.artifact_uri).path# get path from mlflow
+ results=Result.from_path(artifact_dir)
+ returnresults.best_checkpoints[0][0]
+
+
With a particular run's best checkpoint, we can load the model from it and use it.
+
1
+2
+3
+4
+5
# Evaluate on test split
+best_checkpoint=get_best_checkpoint(run_id=best_run.run_id)
+predictor=TorchPredictor.from_checkpoint(best_checkpoint)
+performance=evaluate(ds=test_ds,predictor=predictor)
+print(json.dumps(performance,indent=2))
+
# Predict on sample
+title="Transfer learning with transformers"
+description="Using transformers for transfer learning on text classification tasks."
+sample_df=pd.DataFrame([{"title":title,"description":description,"tag":"other"}])
+predict_with_proba(df=sample_df,predictor=predictor)
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Exploratory data analysis (EDA) to understand the signals and nuances of our dataset. It's a cyclical process that can be done at various points of our development process (before/after labeling, preprocessing, etc. depending on how well the problem is defined. For example, if we're unsure how to label or preprocess our data, we can use EDA to figure it out.
+
We're going to start our project with EDA, a vital (and fun) process that's often misconstrued. Here's how to think about EDA:
+
+
not just to visualize a prescribed set of plots (correlation matrix, etc.).
+
goal is to convince yourself that the data you have is sufficient for the task.
+
use EDA to answer important questions and to make it easier to extract insight
+
not a one time process; as your data grows, you want to revisit EDA to catch distribution shifts, anomalies, etc.
Is there enough signal in the title and description that's unique to each tag? This is important to know because we want to verify our initial hypothesis that the project's title and description are high quality features for predicting the tag. And to visualize this, we're going to use a wordcloud. We also use a jupyter widget, which you can view in the notebook, to interactively select a tag and see the wordcloud for that tag.
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
# Most frequent tokens for each tag
+tag="natural-language-processing"
+plt.figure(figsize=(10,3))
+subset=df[df.tag==tag]
+text=subset.title.values
+cloud=WordCloud(
+ stopwords=STOPWORDS,background_color="black",collocations=False,
+ width=500,height=300).generate(" ".join(text))
+plt.axis("off")
+plt.imshow(cloud)
+
+
+
+
+
+
Looks like the title text feature has some good signal for the respective classes and matches our intuition. We can repeat this for the description text feature as well and see similar quality signals. This information will become useful when we decide how to use our features for modeling.
+
There's a lot more exploratory data analysis that we can do but for now we've answered our questions around our class distributions and the quality of our text features. In the next lesson we'll preprocess our dataset in preparation for model training.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Exploration - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
What is a feature store
+
Let's motivate the need for a feature store by chronologically looking at what challenges developers face in their current workflows. Suppose we had a task where we needed to predict something for an entity (ex. user) using their features.
+
+
Duplication: feature development in isolation (for each unique ML application) can lead to duplication of efforts (setting up ingestion pipelines, feature engineering, etc.).
+
Solution: create a central feature repository where the entire team contributes maintained features that anyone can use for any application.
+
+
+
Skew: we may have different pipelines for generating features for training and serving which can introduce skew through the subtle differences.
+
Solution: create features using a unified pipeline and store them in a central location that the training and serving pipelines pull from.
+
+
+
Values: once we set up our data pipelines, we need to ensure that our input feature values are up-to-date so we aren't working with stale data, while maintaining point-in-time correctness so we don't introduce data leaks.
+
Solution: retrieve input features for the respective outcomes by pulling what's available when a prediction would be made.
+
+
+
+
Point-in-time correctness refers to mapping the appropriately up-to-date input feature values to an observed outcome at \(t_{n+1}\). This involves knowing the time (\(t_n\)) that a prediction is needed so we can collect feature values (\(X\)) at that time.
+
+
+
+
+
When actually constructing our feature store, there are several core components we need to have to address these challenges:
+
+
data ingestion: ability to ingest data from various sources (databases, data warehouse, etc.) and keep them updated.
+
feature definitions: ability to define entities and corresponding features
+
historical features: ability to retrieve historical features to use for training.
+
online features: ability to retrieve features from a low latency origin for inference.
+
+
Each of these components is fairly easy to set up but connecting them all together requires a managed service, SDK layer for interactions, etc. Instead of building from scratch, it's best to leverage one of the production-ready, feature store options such as Feast, Hopsworks, Tecton, Rasgo, etc. And of course, the large cloud providers have their own feature store options as well (Amazon's SageMaker Feature Store, Google's Vertex AI, etc.)
+
+
Tip
+
We highly recommend that you explore this lesson after completing the previous lessons since the topics (and code) are iteratively developed. We did, however, create the feature-store repository for a quick overview with an interactive notebook.
+
+
Over-engineering
+
Not all machine learning platforms require a feature store. In fact, our use case is a perfect example of a task that does not benefit from a feature store. All of our data points are independent, stateless, from client-side and there is no entity that has changing features over time. The real utility of a feature store shines when we need to have up-to-date features for an entity that we continually generate predictions for. For example, a user's behavior (clicks, purchases, etc.) on an e-commerce platform or the deliveries a food runner recently made in the last hour, etc.
+
When do I need a feature store?
+
To answer this question, let's revisit the main challenges that a feature store addresses:
+
+
Duplication: if we don't have too many ML applications/models, we don't really need to add the additional complexity of a feature store to manage transformations. All the feature transformations can be done directly inside the model processing or as a separate function. We could even organize these transformations in a separate central repository for other team members to use. But this quickly becomes difficult to use because developers still need to know which transformations to invoke and which are compatible with their specific models, etc.
+
+
+
Note
+
Additionally, if the transformations are compute intensive, then they'll incur a lot of costs by running on duplicate datasets across different applications (as opposed to having a central location with upt-o-date transformed features).
+
+
+
+
Skew: similar to duplication of efforts, if our transformations can be tied to the model or as a standalone function, then we can just reuse the same pipelines to produce the feature values for training and serving. But this becomes complex and compute intensive as the number of applications, features and transformations grow.
+
+
+
Value: if we aren't working with features that need to be computed server-side (batch or streaming), then we don't have to worry about concepts like point-in-time, etc. However, if we are, a feature store can allow us to retrieve the appropriate feature values across all data sources without the developer having to worry about using disparate tools for different sources (batch, streaming, etc.)
+
+
+
Feast
+
We're going to leverage Feast as the feature store for our application for it's ease of local setup, SDK for training/serving, etc.
+
# Install Feast and dependencies
+pipinstallfeast==0.10.5PyYAML==5.3.1-q
+
+
+
👉 Follow along interactive notebook in the feature-store repository as we implement the concepts below.
+
+
Set up
+
We're going to create a feature repository at the root of our project. Feast will create a configuration file for us and we're going to add an additional features.py file to define our features.
+
+
Traditionally, the feature repository would be it's own isolated repository that other services will use to read/write features from.
We're going to configure the locations for our registry and online store (SQLite) in our feature_store.yaml file.
+
+
+
+
+
+
registry: contains information about our feature repository, such as data sources, feature views, etc. Since it's in a DB, instead of a Python file, it can very quickly be accessed in production.
+
online store: DB (SQLite for local) that stores the (latest) features for defined entities to be used for online inference.
+
+
If all our feature definitions look valid, Feast will sync the metadata about Feast objects to the registry. The registry is a tiny database storing most of the same information you have in the feature repository. This step is necessary because the production feature serving infrastructure won't be able to access Python files in the feature repository at run time, but it will be able to efficiently and securely read the feature definitions from the registry.
+
+
When we run Feast locally, the offline store is effectively represented via Pandas point-in-time joins. Whereas, in production, the offline store can be something more robust like Google BigQuery, Amazon RedShift, etc.
+
+
We'll go ahead and paste this into our features/feature_store.yaml file (the notebook cell is automatically do this):
The first step is to establish connections with our data sources (databases, data warehouse, etc.). Feast requires it's data sources to either come from a file (Parquet), data warehouse (BigQuery) or data stream (Kafka / Kinesis). We'll convert our generated features file from the DataOps pipeline (features.json) into a Parquet file, which is a column-major data format that allows fast feature retrieval and caching benefits (contrary to row-major data formats such as CSV where we have to traverse every single row to collect feature values).
Next, we need to define the main entity that each data point pertains to. In our case, each project has a unique ID with features such as text and tags.
Finally, we're ready to create a FeatureView that loads specific features (features), of various value types, from a data source (input) for a specific period of time (ttl).
# Define a Feature View for each project
+project_details_view=FeatureView(
+ name="project_details",
+ entities=["id"],
+ttl=Duration(
+seconds=(datetime.today()-datetime.strptime(START_TIME,"%Y-%m-%d")).days*24*60*60
+ ),
+features=[
+Feature(name="text",dtype=ValueType.STRING),
+ Feature(name="tag",dtype=ValueType.STRING),
+ ],
+ online=True,
+input=project_details,
+tags={},
+)
+
+
So let's go ahead and define our feature views by moving this code into our features/features.py script (the notebook cell is automatically do this):
+
+Show code
+
fromdatetimeimportdatetime
+frompathlibimportPath
+
+fromfeastimportEntity,Feature,FeatureView,ValueType
+fromfeast.data_sourceimportFileSource
+fromgoogle.protobuf.duration_pb2importDuration
+
+
+# Read data
+START_TIME="2020-02-17"
+project_details=FileSource(
+ path="/content/data/features.parquet",
+ event_timestamp_column="created_on",
+)
+
+# Define an entity for the project
+project=Entity(
+ name="id",
+ value_type=ValueType.INT64,
+ description="project id",
+)
+
+# Define a Feature View for each project
+# Can be used for fetching historical data and online serving
+project_details_view=FeatureView(
+ name="project_details",
+ entities=["id"],
+ ttl=Duration(
+ seconds=(datetime.today()-datetime.strptime(START_TIME,"%Y-%m-%d")).days*24*60*60
+ ),
+ features=[
+ Feature(name="text",dtype=ValueType.STRING),
+ Feature(name="tag",dtype=ValueType.STRING),
+ ],
+ online=True,
+ input=project_details,
+ tags={},
+)
+
+
+
Once we've defined our feature views, we can apply it to push a version controlled definition of our features to the registry for fast access. It will also configure our registry and online stores that we've defined in our feature_store.yaml.
+
cdfeatures
+feastapply
+
+
+Registered entity id
+Registered feature view project_details
+Deploying infrastructure for project_details
+
+
+
Historical features
+
Once we've registered our feature definition, along with the data source, entity definition, etc., we can use it to fetch historical features. This is done via joins using the provided timestamps using pandas for our local setup or BigQuery, Hive, etc. as an offline DB for production.
# Get historical features
+store=FeatureStore(repo_path="features")
+training_df=store.get_historical_features(
+ entity_df=entity_df,
+ feature_refs=["project_details:text","project_details:tag"],
+).to_df()
+training_df.head()
+
+
+
+
+
+
+
event_timestamp
+
id
+
project_details__text
+
project_details__tag
+
+
+
+
+
0
+
2022-06-23 00:00:00+00:00
+
6
+
Comparison between YOLO and RCNN on real world...
+
computer-vision
+
+
+
1
+
2022-06-23 00:00:00+00:00
+
7
+
Show, Infer & Tell: Contextual Inference for C...
+
computer-vision
+
+
+
2
+
2022-06-23 00:00:00+00:00
+
9
+
Awesome Graph Classification A collection of i...
+
graph-learning
+
+
+
+
+
+
Materialize
+
For online inference, we want to retrieve features very quickly via our online store, as opposed to fetching them from slow joins. However, the features are not in our online store just yet, so we'll need to materialize them first.
+Materializing 1 feature views to 2022-06-23 19:16:05+00:00 into the sqlite online store.
+project_details from 2020-02-17 19:16:06+00:00 to 2022-06-23 19:16:05+00:00:
+100%|██████████████████████████████████████████████████████████| 955/955 [00:00<00:00, 10596.97it/s]
+
+
+
This has moved the features for all of our projects into the online store since this was first time materializing to the online store. When we subsequently run the materialize-incremental command, Feast keeps track of previous materializations and so we'll only materialize the new data since the last attempt.
+
Online features
+
Once we've materialized the features (or directly sent to the online store in the stream scenario), we can use the online store to retrieve features.
+
1
+2
+3
+4
+5
+6
+7
# Get online features
+store=FeatureStore(repo_path="features")
+feature_vector=store.get_online_features(
+ feature_refs=["project_details:text","project_details:tag"],
+ entity_rows=[{"id":6}],
+).to_dict()
+feature_vector
+
+
1
+2
+3
{'id':[6],
+ 'project_details__tag':['computer-vision'],
+ 'project_details__text':['Comparison between YOLO and RCNN on real world videos Bringing theory to experiment is cool. We can easily train models in colab and find the results in minutes.']}
+
+
Architecture
+
Batch processing
+
The feature store we implemented above assumes that our task requires batch processing. This means that inference requests on specific entity instances can use features that have been materialized from the offline store. Note that they may not be the most recent feature values for that entity.
+
+
+
+
+
+
Application data is stored in a database and/or a data warehouse, etc. And it goes through the necessary pipelines to be prepared for downstream application (analytics, machine learning, etc.).
+
These features are written to the offline store which can then be used to retrieve historical training data to train a model with. In our local set up, this was join via Pandas DataFrame joins for given timestamps and entity IDs but in a production setting, something like Google BigQuery or Hive would receive the feature requests.
+
Once we have our training data, we can start the workflows to optimize, train and validate a model.
+
We can incrementally materialize features to the online store so that we can retrieve an entity's feature values with low latency. In our local set up, this was join via SQLite for a given set of entities but in a production setting, something like Redis or DynamoDB would be used.
+
These online features are passed on to the deployed model to generate predictions that would be used downstream.
+
+
+
Warning
+
Had our entity (projects) had features that change over time, we would materialize them to the online store incrementally. But since they don't, this would be considered over engineering but it's important to know how to leverage a feature store for entities with changing features over time.
+
+
Stream processing
+
Some applications may require stream processing where we require near real-time feature values to deliver up-to-date predictions at low latency. While we'll still utilize an offline store for retrieving historical data, our application's real-time event data will go directly through our data streams to an online store for serving. An example where stream processing would be needed is when we want to retrieve real-time user session behavior (clicks, purchases) in an e-commerce platform so that we can recommend the appropriate items from our catalog.
+
+
+
+
+
+
Real-time event data enters our running data streams (Kafka / Kinesis, etc.) where they can be processed to generate features.
+
These features are written to the online store which can then be used to retrieve online features for serving at low latency. In our local set up, this was join via SQLite for a given set of entities but in a production setting, something like Redis or DynamoDB would be used.
+
Streaming features are also written from the data stream to the batch data source (data warehouse, db, etc.) to be processed for generating training data later on.
+
Historical data will be validated and used to generate features for training a model. This cadence for how often this happens depends on whether there are data annotation lags, compute constraints, etc.
+
+
+
There are a few more components we're not visualizing here such as the unified ingestion layer (Spark), that connects data from the varied data sources (warehouse, DB, etc.) to the offline/online stores, or low latency serving (<10 ms). We can read more about all of these in the official Feast Documentation, which also has guides to set up a feature store with Feast with AWS, GCP, etc.
+
+
Additional functionality
+
Additional functionality that many feature store providers are currently (or recently) trying to integrate within the feature store platform include:
+
+
transform: ability to directly apply global preprocessing or feature engineering on top of raw data extracted from data sources.
+
Currentsolution: apply transformations as a separate Spark, Python, etc. workflow task before writing to the feature store.
Currentsolution: apply data testing and monitoring as upstream workflow tasks before they are written to the feature store.
+
+
+
discover: ability for anyone in our team to easily discover features that they can leverage for their application.
+
Currentsolution: add a data discovery engine, such as Amundsen, on top of our feature store to enable others to search for features.
+
+
+
+
Reproducibility
+
Though we could continue to version our training data with DVC whenever we release a version of the model, it might not be necessary. When we pull data from source or compute features, should they save the data itself or just the operations?
+
+
Version the data
+
This is okay if (1) the data is manageable, (2) if our team is small/early stage ML or (3) if changes to the data are infrequent.
+
But what happens as data becomes larger and larger and we keep making copies of it.
+
+
+
Version the operations
+
We could keep snapshots of the data (separate from our projects) and provided the operations and timestamp, we can execute operations on those snapshots of the data to recreate the precise data artifact used for training. Many data systems use time-travel to achieve this efficiently.
+
But eventually this also results in data storage bulk. What we need is an append-only data source where all changes are kept in a log instead of directly changing the data itself. So we can use the data system with the logs to produce versions of the data as they were without having to store separate snapshots of the the data itself.
+
+
+
+
Regardless of the choice above, feature stores are very useful here. Instead of coupling data pulls and feature compute with the time of modeling, we can separate these two processes so that features are up-to-date when we need them. And we can still achieve reproducibility via efficient point-in-time correctness, low latency snapshots, etc. This essentially creates the ability to work with any version of the dataset at any point in time.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Feature Store - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Whether we're working individually or with a team, it's important that we have a system to track changes to our projects so that we can revert to previous versions and so that others can reproduce our work and contribute to it. Git is a distributed versional control system that allows us do exactly this. Git runs locally on our computer and it keeps track of our files and their histories. To enable collaboration with others, we can use a remote host (GitHub, GitLab, BitBucket, etc.) to host our files and their histories. We'll use git to push our local changes and pull other's changes to and from the remote host.
+
+
Git is traditionally used to store and version small files <100MB (scripts, READMEs, etc.), however, we can still version large artifacts (datasets, model weights, etc.) using text pointers pointing to blob stores. These pointers will contain information such as where the asset is located, it's specific contents/version (ex. via hashing), etc.
+
+
Set up
+
Initialize git
+
+
+
+
+
Initialize a local repository (.git directory) to track our files:
+
gitinit
+
+
+Initialized empty Git repository in /Users/goku/Documents/madewithml/MLOps/.git/
+
+
+
We can see what files are untracked or yet to be committed:
+
gitstatus
+
+
+On branch main
+
+No commits yet
+
+Untracked files:
+ (use "git add ..." to include in what will be committed)
+ .flake8
+ .vscode/
+ Makefile
+ ...
+
+
+
.gitignore
+
We can see that we have some files that we don't want to push to a remote host, such as our virtual environment, logs, large data files, etc. We can create a .gitignore file to make sure we aren't checking in these files.
For now, we're going to add data to our .gitignore file as well but this means that others will not be able to produce the same data assets when they pull from our remote host. To address this, we'll push pointers to our data files in our versioning lesson so that the data too can be reproduced exactly as we have it locally.
+
+
Tip
+
Check out our project's .gitignore for a more complete example that also includes lots of other system artifacts that we would normally not want to push to a remote repository. Our complete .gitignore file is based on GitHub's Python template and we're using a Mac, so we added the relevant global file names as well.
+
+
If we run git status now, we should no longer see the files we've defined in our .gitignore file.
+
Add to stage
+
Next, we'll add our work from the working directory to the staging area.
+
+
We can add one file at a time:
+
gitadd<filename>
+
+
We can add all files at once:
+
gitadd.
+
+
+
Now running git status will show us all the staged files:
+
gitstatus
+
+
+On branch main
+
+No commits yet
+
+Changes to be committed:
+ (use "git rm --cached ..." to unstage)
+ new file: .flake8
+ new file: .gitignore
+ new file: Makefile
+ ...
+
+
+
Commit to repo
+
Now we're ready to commit the files in the staging area to the local repository. The default branch (a version of our project) will be called main.
The commit requires a message indicating what changes took place. We can use git commit --amend to edit the commit message if needed. If we do a git status check we'll see that there is nothing else to commit from our staging area.
+
gitstatus
+
+
+On branch main
+nothing to commit, working tree clean
+
+
+
Push to remote
+
Now we're ready to push the updates from our local repository to a remote repository. Start by creating an account on GitHub (or any other remote repository) and follow the instructions to create a remote repository (it can be private or public). Inside our local repository, we're going to set our username and email credentials so that we can push changes from our local to the remote repository.
+
# Set credentials via terminal
+gitconfig--globaluser.name<USERNAME>
+gitconfig--globaluser.email<EMAIL>
+
+We can quickly validate that we set the proper credentials like so:
+
Next, we need to establish the connection between our local and remote repositories:
+
# Push to remote
+gitremoteaddoriginhttps://github.com/<USERNAME>/<REPOSITORY_NAME>.git
+gitpush-uoriginmain# pushing the contents of our local repo to the remote repo
+# origin signifies the remote repository
+
+
Developing
+
Now we're ready to start adding to our project and committing the changes.
+
Cloning
+
If we (or someone else) doesn't already have the local repository set up and connected with the remote host, we can use the clone command:
+
gitclone<REMOTE_REPO_URL><PATH_TO_PROJECT_DIR>
+
+
And we can clone a specific branch of a repository as well:
<REMOTE_REPO_URL> is the location of the remote repo (ex. https://github.com/GokuMohandas/Made-With-ML).
+
<PATH_TO_PROJECT_DIR> is the name of the local directory you want to clone the project into.
+
+
Create a branch
+
When we want to add or change something, such as adding a feature, fixing a bug, etc., it's always a best practice to create a separate branch before developing. This is especially crucial when working with a team so we can cleanly merge our work with the main branch after discussions and reviews.
+
We'll start by creating a new branch:
+
gitcheckout-b<NEW_BRANCH_NAME>
+
+
We can see all the branches we've created with the following command where the * indicates our current branch:
+
gitbranch
+
+
+* convnet
+main
+
+
+
We can easily switch between existing branches using:
+
gitcheckout<BRANCH_NAME>
+
+
Once we're in a branch, we can make changes to our project and commit those changes.
+
gitadd.
+gitcommit-m"update model to a convnet"
+gitpushoriginconvnet
+
+
Note that we are pushing this branch to our remote repository, which doesn't yet exist there, so GitHub will create it accordingly.
+
Pull request (PR)
+
When we push our new branch to the remote repository, we'll need to create a pull request (PR) to merge with another branch (ex. our main branch in this case). When merging our work with another branch (ex. main), it's called a pull request because we're requesting the branch to pull our committed work. We can create the pull request using steps outlined here: Creating a pull request.
+
+
+
+
+
+
Note
+
We can merge branches and resolve conflicts using git CLI commands but it's preferred to use the online interface because we can easily visualize the changes, have discussion with teammates, etc.
+
# Merge via CLI
+gitpushoriginconvnet
+gitcheckoutmain
+gitmergeconvnet
+gitpushoriginmain
+
+
+
Pull
+
Once we accepted the pull request, our main branch is now updated with our changes. However, the update only happened on the remote repository so we should pull those changes to our local main branch as well.
+
gitcheckoutmain
+gitpulloriginmain
+
+
Delete branches
+
Once we're done working with a branch, we can delete it to prevent our repository from cluttering up. We can easily delete both the local and remote versions of the branch with the following commands:
+
# Delete branches
+gitbranch-d<BRANCH_NAME># local
+gitpushorigin--delete<BRANCH_NAME># remote
+
+
Collaboration
+
So far, the workflows for integrating our iterative development has been very smooth but in a collaborative setting, we may need to resolve conflicts. Let's say there are two branches (a and b) that were created from the main branch. Here's what we're going to try and simulate:
+
+
Developer A and B fork the main branch to make some changes
+
Developer A makes a change and submits a PR to the main branch.
+
Developer B makes a change to the same line as Developer A and submits a PR to main.
+
We have a merge conflict now since both developers altered the same line.
+
Choose which version of the code to keep and resolve the conflict.
+
+
When we try to merge the second PR, we have to resolve the conflicts between this new PR and what already exists in the main branch.
+
+
+
+
+
We can resolve the conflict by choosing which content (current main which merged with the a branch or this b branch) to keep and delete the other one. Then we can merge the PR successfully and update our local main branch.
Once the conflicts have been resolved and we merge the PR, we can update our local repository to reflect the decisions.
+
gitcheckoutmain
+gitpulloriginmain
+
+
+
Note
+
We only have a conflict because both branches were forked from a previous version of the main branch and they both happened to alter the same content. Had we created one branch first and then updated main before creating the second branch, we wouldn't have any conflicts. But in a collaborative setting, different developers may fork off the same version of the branch anytime.
+
+
+
A few more important commands to know include rebase and stash.
+
+
Inspection
+
Git allows us to inspect the current and previous states of our work at many different levels. Let's explore the most commonly used commands.
+
Status
+
We've used the status command quite a bit already as it's very useful to quickly see the status of our working tree.
+
# Status
+gitstatus
+gitstatus-s# short format
+
+
Log
+
If we want to see the log of all our commits, we can do so using the log command. We can also do the same by inspecting specific branch histories on the Git online interface.
+
# Log
+gitlog
+gitlog--oneline# short version
+
+
+704d99c (HEAD -> main) added project files
+
+
+
+
Commit IDs are 40 characters long but we can represent them with the first few (seven digits is the default for a Git SHA). If there is ambiguity, Git will notify us and we can simply add more of the commit ID.
+
+
Diff
+
If we want to know the difference between two commits, we can use the diff command.
+
# Diff
+gitdiff# all changes between current working tree and previous commit
+gitdiff<COMMIT_A><COMMIT_B># diff b/w two commits
+gitdiff<COMMIT_A>:<PATH_TO_FILE><COMMIT_B>:<PATH_TO_FILE># file diff b/w two commits
+
One of the most useful inspection commands is blame, which allows us to see what commit was responsible for every single line in a file.
+
# Blame
+gitblame<PATH_TO_FILE>
+gitblame-L1,3<PATH_TO_FILE># blame for lines 1 and 3
+
+
Time travel
+
Sometimes we may have done something we wish we could change. It's not always possible to do this in life, but in the world of Git, it is!
+
Restore
+
Sometimes we may just want to undo adding or staging a file, which we can easily do with the restore command.
+
# Restore
+gitrestore--<PATH_TO_FILE><PATH_TO_FILE># will undo any changes
+gitrestore--stage<PATH_TO_FILE># will remove from stage (won't undo changes)
+
+
Reset
+
Now if we already made the commit but haven't pushed to remote yet, we can reset to the previous commit by moving the branch pointer to that commit. Note that this will undo all changes made since the previous commit.
+
# Reset
+gitreset<PREVIOUS_COMMIT_ID># or HEAD^
+
+
+
HEAD is a quick way to refer to the previous commit. Both HEAD and any previous commit ID can be accompanied with a ^ or ~ symbol which acts as a relative reference. ^n refers to the nth parent of the commit while ~n refers to the nth grandparent. Of course we can always just explicitly use commit IDs but these short hands can come in handy for quick checks without doing git log to retrieve commit IDs.
+
+
Revert
+
But instead of moving the branch pointer to a previous commit, we can continue to move forward by adding a new commit to revert certain previous commits.
+
# Revert
+gitrevert<COMMIT_ID>...# rollback specific commits
+gitrevert<COMMIT_TO_ROLLBACK_TO>..<COMMIT_TO_ROLLBACK_FROM># range
+
+
Checkout
+
Sometimes we may want to temporarily switch back to a previous commit just to explore or commit some changes. It's best practice to do this in a separate branch and if we want to save our changes, we need to create a separate PR. Note that if you do checkout a previous commit and submit a PR, you may override the commits in between.
+
There so many different works to work with git and sometimes it can became quickly unruly when fellow developers follow different practices. Here are a few, widely accepted, best practices when it comes to working with commits and branches.
+
Commits
+
+
Commit often such that each commit has a clear associated change which you can approve / rollback.
+
Try and squash commits if you have multiple before pushing to the remote host.
+
Avoid monolithic commits (even if you regularly stash and rebase) because it can cause many components to break and creates a code review nightmare.
+
Attach meaningful messages to commits so developers know exactly what the PR entails.
+
Use tags to represent meaningful and stable releases of your application.
+
# Tags
+gittag-av0.1-m"initial release"
+
+
Don't delete commit history (reset), instead use revert to rollback and provide reasoning.
+
+
Branches
+
+
Create branches when working on a feature, bug, etc. because it makes adding and reverting to the main branch very easy.
+
Avoid using cryptic branch names.
+
Maintain your main branch as the "demo ready" branch that always works.
+
Protect branches with rules (especially the main branch).
+
+
Tags
+
Leverage git tags to mark significant release commits. We can create tags either through the terminal or the online remote interface and this can be done to previous commits as well (in case we forgot).
+
# Tags
+gittag# view all existing tags
+gittag-a<TAG_NAME>-m"SGD"# create a tag
+gitcheckout-b<BRANCH_NAME><TAG_NAME># checkout a specific tag
+gittag-d<TAG_NAME># delete local tag
+gitpushorigin--delete<TAG_NAME># delete remote tag
+gitfetch--all--tags# fetch all tags from remote
+
+
+
Tag names usually adhere to version naming conventions, such as v1.4.2 where the numbers indicate major, minor and bug changes from left to right.
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Git - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
+ While the specific task in this course involves fine-tuning an LLM for a supervised task, everything we learn easily extends to all applications (NLP, CV, time-series, etc.), models (regression → LLMs), data modalities (tabular, text, etc.), cloud platforms (AWS, GCP) and scale (local laptop → distributed cluster).
+
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ MLOps Course - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Our ML workloads have been responsible for everything from data ingestion to model validation:
+
+
+
+
+
We can execute these workloads as standalone CLI commands:
+
1
+2
+3
+4
+5
+6
+7
# ML workloads (simplified)
+pytest--dataset-loc=$DATASET_LOCtests/data...# test data
+python-mpytesttests/code--verbose--disable-warnings# test code
+pythonmadewithml/train.py--experiment-name"llm"...# train model
+pythonmadewithml/evaluate.py--run-id$RUN_ID...# evaluate model
+pytest--run-id=$RUN_IDtests/model...# test model
+pythonmadewithml/serve.py--run_id$RUN_ID# serve model
+
+
With all of our ML workloads implemented (and tested), we're ready to go to production. In this lesson, we'll learn how to convert our ML workloads from CLI commands into a scalable, fault-tolerant and reproducible workflow.
+
+
We'll combine our ML workloads up to (and including) model validation into a workflow.
+
This workflow will then produce model artifacts, which will be saved to our model registry.
+
And finally, we can serve that model behind an API endpoint to use in production.
+
+
+
+
+
+
Jobs
+
Script
+
Since we have our CLI commands for our ML workloads, we could just execute them one-by-one on our local machine or Workspace. But for efficiency, we're going to combine them all into one script. We'll organize this under a workloads.sh bash script inside our deploy/jobs directory. Here the workloads are very similar to our CLI commands but we have some additional steps to print and save the logs from each of our workloads. For example, our data validation workload looks like this:
At the end of our workloads.sh script, we save our model registry (with our saved model artifacts) and the results from the different workloads to S3. We'll use these artifacts and results later on when we deploy our model as a Service.
+
# Save to S3
+exportMODEL_REGISTRY=$(python-c"from madewithml import config; print(config.MODEL_REGISTRY)")
+awss3cp$MODEL_REGISTRYs3://madewithml/$GITHUB_USERNAME/mlflow/--recursive
+awss3cpresults/s3://madewithml/$GITHUB_USERNAME/results/--recursive
+
+
+
Note
+
If you're doing this lesson on your local laptop, you'll have to add the proper AWS credentials and up the S3 buckets for our workloads script to run successfully.
+
+If you don't want to set up all of this yourself, we highly recommend joining our upcoming live cohort where we'll provide an environment with all of this infrastructure already set up for you so that you just focused on the machine learning.
+
+
Configuration
+
Now that we have our single script to execute all workloads, we can execute it with one command (./deploy/jobs/workloads.sh). But even better way is to use Anyscale Jobs to get features like automatic failure handling, email alerts and persisted logs all out of the box for our workloads. And with our cluster_env.yaml, cluster_compute.yaml and workloads.sh files, we can create the configuration for our Anyscale Job with an workloads.yaml file:
Line3: name of our Anyscale Project (we're organizing it all under the same madewithml project we used for our Workspace setup)
+
Line4: name of our cluster environment
+
Line5: name of our compute configuration
+
Line6-10: runtime environment for our Anyscale Job. The runtime_env here specifies that we should upload our current working_dir to an S3 bucket so that all of our workers when we execute an Anyscale Job have access to the code to use. We also set some environment variables that our workloads will have access to.
+
Line11: entrypoint for our Anyscale Job. This is the command that will be executed when we submit our Anyscale Job.
+
Line12: maximum number of retries for our Anyscale Job. If our Anyscale Job fails, it will automatically retry up to this number of times.
+
+
+
Warning
+
Be sure to update the $GITHUB_USERNAME slots inside our deploy/jobs/workloads.yaml configuration to your own GitHub username. This is used to save your model registry and results to a unique path on our shared S3 bucket (s3://madewithml).
+
+
Because we're using the exact same cluster environment and compute configuration, what worked during development will work in production. This is a huge benefit of using Anyscale Jobs because we don't have to worry about any environment discrepanices when we deploy our workloads to production. This makes going to production much easier and faster!
+
Execution
+
And now we can execute our Anyscale Job in one line:
+
anyscalejobsubmitdeploy/jobs/workloads.yaml
+
+
+Authenticating
+
+Output
+(anyscale +8.8s) Maximum uptime is disabled for clusters launched by this job.
+(anyscale +8.8s) Job prodjob_zqj3k99va8a5jtd895u3ygraup has been successfully submitted. Current state of job: PENDING.
+(anyscale +8.8s) Query the status of the job with `anyscale job list --job-id prodjob_zqj3k99va8a5jtd895u3ygraup`.
+(anyscale +8.8s) Get the logs for the job with `anyscale job logs --job-id prodjob_zqj3k99va8a5jtd895u3ygraup --follow`.
+(anyscale +8.8s) View the job in the UI at https://console.anyscale.com/jobs/prodjob_zqj3k99va8a5jtd895u3ygraup
+(anyscale +8.8s) Use --follow to stream the output of the job when submitting a job.
+
+
+
+
Tip
+
When we run anyscale cli commands inside our Workspaces, we automatically have our credentials set up for us. But if we're running anyscale cli commands on our local laptop, we'll have to set up the appropriate credentials.
+
We can now go to thie UI link that was provided to us to view the status, logs, etc. of our Anyscale Job.
+
+
+
+
+
And if we inspect our S3 buckets, we'll can see all the artifacts that have been saved from this Anyscale Job.
+
+
+
+
+
Debugging
+
Since we use the exact same cluster (environment and compute) for production as we did for development, we're significantly less likely to run into the environment discrepancy issues that typically arise when going from development to production. However, there can always be small issues that arize from missing credentials, etc. We can easily debug our Anyscale Jobs by inspecting the jobs: Jobs page > choose job > View console logs at the bottom > View Ray workers logs > paste command > Open job-logs directory > View job-driver-raysubmit_XYZ.log. Alternatively, we can also run our Anyscale Job as a Workspace by clicking on the Duplicate as Workspace button the top of a particular Job's page.
+
Services
+
After we execute our Anyscale Job, we will have saved our model artifacts to a particular location. We'll now use Anyscale Services to pull from this location to serve our models in production behind a scalable rest endpoint
# deploy/services/serve_model.py
+
+importos
+importsubprocess
+frommadewithml.configimportMODEL_REGISTRY# NOQA: E402
+frommadewithml.serveimportModelDeployment# NOQA: E402
+
+# Copy from S3
+github_username=os.environ.get("GITHUB_USERNAME")
+subprocess.check_output(["aws","s3","cp",f"s3://madewithml/{github_username}/mlflow/",str(MODEL_REGISTRY),"--recursive"])
+subprocess.check_output(["aws","s3","cp",f"s3://madewithml/{github_username}/results/","./","--recursive"])
+
+# Entrypoint
+run_id=[line.strip()forlineinopen("run_id.txt")][0]
+entrypoint=ModelDeployment.bind(run_id=run_id,threshold=0.9)
+
+# Inference
+data={"query":"What is the default batch size for map_batches?"}
+response=requests.post("http://127.0.0.1:8000/query",json=data)
+print(response.json())
+
+
+# Inference
+data={"query":"What is the default batch size for map_batches?"}
+response=requests.post("http://127.0.0.1:8000/query",json=data)
+print(response.json())
+
+
In this script, we first pull our previously saved artifacts from our S3 bucket to our local storage and then define the entrypoint for our model.
+
+
Tip
+
Recall that we have the option to scale when we define our service inside out madewithml/serve.py script. And we can scale our compute configuration to meet those demands.
Line3: name of our Anyscale Project (we're organizing it all under the same madewithml project we used for our Workspace setup)
+
Line4: name of our cluster environment
+
Line5: name of our compute configuration
+
Line6-12: serving configuration that specifies our entry point and details about the working directory, environment variables, etc.
+
Line13: rollout strategy for our Anyscale Service. We can either rollout a new version of our service or replace the existing version with the new one.
+
+
+
Warning
+
Be sure to update the $GITHUB_USERNAME slots inside our deploy/services/serve_model.yaml configuration to your own GitHub username. This is used to pull model artifacts and results from our shared S3 bucket (s3://madewithml).
+
+
Execution
+
And now we can execute our Anyscale Service in one line:
+
# Rollout service
+anyscaleservicerollout-fdeploy/services/serve_model.yaml
+
+
+Authenticating
+
+Output
+(anyscale +7.3s) Service service2_xwmyv1wcm3i7qan2sahsmybymw has been deployed. Service is transitioning towards: RUNNING.
+(anyscale +7.3s) View the service in the UI at https://console.anyscale.com/services/service2_xwmyv1wcm3i7qan2sahsmybymw
+
+
+
+
Note
+
If we chose the ROLLOUT strategy, we get a canary rollout (increasingly serving traffic to the new version of our service) by default.
+
+
Once our service is up and running, we can query it:
+
# Query
+curl-XPOST-H"Content-Type: application/json"-H"Authorization: Bearer $SECRET_TOKEN"-d'{
+ "title": "Transfer learning with transformers",
+ "description": "Using transformers for transfer learning on text classification tasks."
+}'$SERVICE_ENDPOINT/predict/
+
And we can just as easily rollback to a previous version of our service or terminate it altogether:
+
# Rollback (to previous version of the Service)
+anyscaleservicerollback-f$SERVICE_CONFIG--name$SERVICE_NAME
+
+# Terminate
+anyscaleserviceterminate--name$SERVICE_NAME
+
+
Observability
+
Once we rollout our service, we have several different dashboards that we can use to monitor our service. We can access these dashboards by going to the Services page > choose service > Click the Dashboard button (top right corner) > Ray Dashboard. Here we'll able to see the logs from our Service, metrics, etc.
+
+
+
+
+
On the same Dashboard button, we also have a Metrics option that will take us to a Grafana Dashboard. This view has a lot more metrics on incoming requests, latency, errors, etc.
+
+
+
+
+
Debugging
+
Serving our models may not always work as intended. Even if our model serving logic is correct, there are external dependencies that could causes errors --- such as our model not being stored where it should be, trouble accessing our model registry, etc. For all these cases and more, it's very important to know how to be able to debug our Services.
+
Services page > choose service > Go to Resource usage section > Click on the cluster link (cluster_for_service_XYZ) > Ray logs (tab at bottom) > paste command > Open worker-XYZ directory > View combined_worker.log
+
Scaling
+
The combination of using Workspaces for development and Job & Services for production make it extremely easy and fast to make the transition. The cluster environment and compute configurations are the exact same so the code that's executing runs on the same conditions. However, we may sometimes want to scale up our production compute configurations to execute Jobs faster or meet the availability/latency demands for our Services. We could address this by creating a new compute configuration:
+
# Compute config
+exportCLUSTER_COMPUTE_NAME="madewithml-cluster-compute-prod"
+anyscalecluster-computecreatedeploy/cluster_compute_prod.yaml--name$CLUSTER_COMPUTE_NAME# uses new config with prod compute requirements
+
+
or by using a one-off configuration to specify the compute changes, where instead of pointing to a previously existing compute configuration, we can define it directly in our Jobs/Services yaml configuration:
And with that, we're able to completely productionize our ML workloads! We have a working service that we can use to make predictions using our trained model. However, what happens when we receive new data or our model's performance regresses over time? With our current approach here, we have to manually execute our Jobs and Services again to udpate our application. In the next lesson, we'll learn how to automate this process with CI/CD workflows that execute our Jobs and Services based on an event (e.g. new data).
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Jobs & Services - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
What is data labeling
+
Labeling (or annotation) is the process of identifying the inputs and outputs that are worth modeling (not just what could be modeled).
+
+
use objective as a guide to determine the necessary signals.
+
explore creating new signals (via combining features, collecting new data, etc.).
+
iteratively add more features to justify complexity and effort.
+
+
It's really important to get our labeling workflows in place before we start performing downstream tasks such as data augmentation, model training, etc.
+
+
Warning
+
Be careful not to include features in the dataset that will not be available during prediction, causing data leaks.
+
+
+
What else can we learn?
+
It's not just about identifying and labeling our initial dataset. What else can we learn from it?
+
+Show answer
+
It's also the phase where we can use our deep understanding of the problem to:
Regardless of whether we have a custom labeling platform or we choose a generalized platform, the process of labeling and all it's related workflows (QA, data import/export, etc.) follow a similar approach.
+
Preliminary steps
+
+
[WHAT] Decide what needs to be labeled:
+
identify natural labels you may already have (ex. time-series)
+
consult with domain experts to ensure you're labeling the appropriate signals
+
decide on the appropriate labels (and hierarchy) for your task
+
+
+
[WHERE] Design the labeling interface:
+
intuitive, data modality dependent and quick (keybindings are a must!)
+
avoid option paralysis by allowing the labeler to dig deeper or suggesting likely labels
+
measure and resolve inter-labeler discrepancy
+
+
+
[HOW] Compose labeling instructions:
+
examples of each labeling scenario
+
course of action for discrepancies
+
+
+
+
+
+
+
+ Multi-label text classification for our task using Prodigy (labeling + QA)
+
+
+
Workflow setup
+
+
Establish data pipelines:
+
[IMPORT]new data for annotation
+
[EXPORT]annotated data for QA, testing, modeling, etc.
+
+
+
Create a quality assurance (QA) workflow:
+
separate from labeling workflow (no bias)
+
communicates with labeling workflow to escalate errors
+
+
+
+
+
+
+
+
Iterative setup
+
+
Implement strategies to reduce labeling efforts
+
identify subsets of the data to label next using active learning
focus labeling efforts on long tail of edge cases over time
+
+
+
+
Labeled data
+
For the purpose of this course, our data is already labeled, so we'll perform a basic version of ELT (extract, load, transform) to construct the labeled dataset.
+
+
In our data-stack and orchestration lessons, we'll construct a modern data stack and programmatically deliver high quality data via DataOps workflows.
+
+
+
projects.csv: projects with id, created time, title and description.
+
tags.csv: labels (tag category) for the projects by id.
+
+
Recall that our objective was to classify incoming content so that the community can discover them easily. These data assets will act as the training data for our first model.
+
Extract
+
We'll start by extracting data from our sources (external CSV files). Traditionally, our data assets will be stored, versioned and updated in a database, warehouse, etc. We'll learn more about these different data systems later, but for now, we'll load our data as a stand-alone CSV file.
Apply basic transformations to create our labeled dataset.
+
1
+2
+3
# Join projects and tags
+df=pd.merge(projects,tags,on="id")
+df.head()
+
+
+
+
+
+
+
+
id
+
created_on
+
title
+
description
+
tag
+
+
+
+
+
0
+
6
+
2020-02-20 06:43:18
+
Comparison between YOLO and RCNN on real world...
+
Bringing theory to experiment is cool. We can ...
+
computer-vision
+
+
+
1
+
7
+
2020-02-20 06:47:21
+
Show, Infer & Tell: Contextual Inference for C...
+
The beauty of the work lies in the way it arch...
+
computer-vision
+
+
+
2
+
9
+
2020-02-24 16:24:45
+
Awesome Graph Classification
+
A collection of important graph embedding, cla...
+
graph-learning
+
+
+
3
+
15
+
2020-02-28 23:55:26
+
Awesome Monte Carlo Tree Search
+
A curated list of Monte Carlo tree search papers...
+
reinforcement-learning
+
+
+
4
+
19
+
2020-03-03 13:54:31
+
Diffusion to Vector
+
Reference implementation of Diffusion2Vec (Com...
+
graph-learning
+
+
+
+
+
+
+
1
df=df[df.tag.notnull()]# remove projects with no tag
+
+
Load
+
Finally, we'll load our transformed data locally so that we can use it for our machine learning application.
+
1
+2
# Save locally
+df.to_csv("labeled_projects.csv",index=False)
+
+
Libraries
+
We could have used the user provided tags as our labels but what if the user added a wrong tag or forgot to add a relevant one. To remove this dependency on the user to provide the gold standard labels, we can leverage labeling tools and platforms. These tools allow for quick and organized labeling of the dataset to ensure its quality. And instead of starting from scratch and asking our labeler to provide all the relevant tags for a given project, we can provide the author's original tags and ask the labeler to add / remove as necessary. The specific labeling tool may be something that needs to be custom built or leverages something from the ecosystem.
+
+
As our platform grows, so too will our dataset and labeling needs so it's imperative to use the proper tooling that supports the workflows we'll depend on.
+
+
General
+
+
Labelbox: the data platform for high quality training and validation data for AI applications.
+
Scale AI: data platform for AI that provides high quality training data.
+
Label Studio: a multi-type data labeling and annotation tool with standardized output format.
+
Universal Data Tool: collaborate and label any type of data, images, text, or documents in an easy web interface or desktop app.
+
Prodigy: recipes for the Prodigy, our fully scriptable annotation tool.
+
Superintendent: an ipywidget-based interactive labelling tool for your data to enable active learning.
+
+
Natural language processing
+
+
Doccano: an open source text annotation tool for text classification, sequence labeling and sequence to sequence tasks.
+
BRAT: a rapid annotation tool for all your textual annotation needs.
+
+
Computer vision
+
+
LabelImg: a graphical image annotation tool and label object bounding boxes in images.
+
CVAT: a free, online, interactive video and image annotation tool for computer vision.
+
VoTT: an electron app for building end-to-end object detection models from images and videos.
+
makesense.ai: a free to use online tool for labelling photos.
+
remo: an app for annotations and images management in computer vision.
+
Labelai: an online tool designed to label images, useful for training AI models.
+
+
Audio
+
+
Audino: an open source audio annotation tool for voice activity detection (VAD), diarization, speaker identification, automated speech recognition, emotion recognition tasks, etc.
+
audio-annotator: a JavaScript interface for annotating and labeling audio files.
+
EchoML: a web app to play, visualize, and annotate your audio files for machine learning.
+
+
Miscellaneous
+
+
MedCAT: a medical concept annotation tool that can extract information from Electronic Health Records (EHRs) and link it to biomedical ontologies like SNOMED-CT and UMLS.
+
+
+
Generalized labeling solutions
+
What criteria should we use to evaluate what labeling platform to use?
+
+Show answer
+
It's important to pick a generalized platform that has all the major labeling features for your data modality with the capability to easily customize the experience.
+
+
how easy is it to connect to our data sources (DB, QA, etc.)?
+
how easy was it to make changes (new features, labeling paradigms)?
+
how securely is our data treated (on-prem, trust, etc.)
+
+
However, as an industry trend, this balance between generalization and specificity is difficult to strike. So many teams put in the upfront effort to create bespoke labeling platforms or used industry specific, niche, labeling tools.
+
+
+
Active learning
+
Even with a powerful labeling tool and established workflows, it's easy to see how involved and expensive labeling can be. Therefore, many teams employ active learning to iteratively label the dataset and evaluate the model.
+
+
Label a small, initial dataset to train a model.
+
Ask the trained model to predict on some unlabeled data.
+
Determine which new data points to label from the unlabeled data based on:
+
entropy over the predicted class probabilities
+
samples with lowest predicted, calibrated, confidence (uncertainty sampling)
+
discrepancy in predictions from an ensemble of trained models
+
+
+
Repeat until the desired performance is achieved.
+
+
+
This can be significantly more cost-effective and faster than labeling the entire dataset.
ALiPy: active learning python toolbox, which allows users to conveniently evaluate, compare and analyze the performance of active learning methods.
+
+
Weak supervision
+
If we had samples that needed labeling or if we simply wanted to validate existing labels, we can use weak supervision to generate labels as opposed to hand labeling all of them. We could utilize weak supervision via labeling functions to label our existing and new data, where we can create constructs based on keywords, pattern expressions, knowledge bases, etc. And we can add to the labeling functions over time and even mitigate conflicts amongst the different labeling functions. We'll use these labeling functions to create and evaluate slices of our data in the evaluation lesson.
An easy way to validate our labels (before modeling) is to use the aliases in our auxillary datasets to create labeling functions for the different classes. Then we can look for false positives and negatives to identify potentially mislabeled samples. We'll actually implement a similar kind of inspection approach, but using a trained model as a heuristic, in our dashboards lesson.
+
+
Iteration
+
Labeling isn't just a one time event or something we repeat identically. As new data is available, we'll want to strategically label the appropriate samples and improve slices of our data that are lacking in quality. Once new data is labeled, we can have workflows that are triggered to start the (re)training process to deploy a new version of our system.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Data Labeling - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Logging is the process of tracking and recording key events that occur in our applications for the purpose of inspection, debugging, etc. They're a whole lot more powerful than print statements because they allow us to send specific pieces of information to specific locations with custom formatting, shared interfaces, etc. This makes logging a key proponent in being able to surface insightful information from the internal processes of our application.
+
Components
+
There are a few overarching concepts to be aware of:
+
+
Logger: emits the log messages from our application.
+
Handler: sends log records to a specific location.
+
Formatter: formats and styles the log records.
+
+
There is so much more to logging such as filters, exception logging, etc. but these basics will allows us to do everything we need for our application.
+
Levels
+
Before we create our specialized, configured logger, let's look at what logged messages look like by using the basic configuration.
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
importlogging
+importsys
+
+# Create super basic logger
+logging.basicConfig(stream=sys.stdout,level=logging.DEBUG)
+
+# Logging levels (from lowest to highest priority)
+logging.debug("Used for debugging your code.")
+logging.info("Informative messages from your code.")
+logging.warning("Everything works but there is something to be aware of.")
+logging.error("There's been a mistake with the process.")
+logging.critical("There is something terribly wrong and process may terminate.")
+
+
+DEBUG:root:Used for debugging your code.
+INFO:root:Informative messages from your code.
+WARNING:root:Everything works but there is something to be aware of.
+ERROR:root:There's been a mistake with the process.
+CRITICAL:root:There is something terribly wrong and process may terminate.
+
+
+
These are the basic levels of logging, where DEBUG is the lowest priority and CRITICAL is the highest. We defined our logger using basicConfig to emit log messages to stdout (ie. our terminal console), but we also could've written to any other stream or even a file. We also defined our logging to be sensitive to log messages starting from level DEBUG. This means that all of our logged messages will be displayed since DEBUG is the lowest level. Had we made the level ERROR, then only ERROR and CRITICAL log message would be displayed.
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
importlogging
+importsys
+
+# Create super basic logger
+logging.basicConfig(stream=sys.stdout,level=logging.ERROR)
+
+# Logging levels (from lowest to highest priority)
+logging.debug("Used for debugging your code.")
+logging.info("Informative messages from your code.")
+logging.warning("Everything works but there is something to be aware of.")
+logging.error("There's been a mistake with the process.")
+logging.critical("There is something terribly wrong and process may terminate.")
+
+
+ERROR:root:There's been a mistake with the process.
+CRITICAL:root:There is something terribly wrong and process may terminate.
+
+
+
Configuration
+
First we'll set the location of our logs in our config.py script:
[Lines 6-11]: define two different Formatters (determine format and style of log messages), minimal and detailed, which use various LogRecord attributes to create a formatting template for log messages.
+
[Lines 12-35]: define the different Handlers (details about location of where to send log messages):
+
console: sends log messages (using the minimal formatter) to the stdout stream for messages above level DEBUG (ie. all logged messages).
+
info: send log messages (using the detailed formatter) to logs/info.log (a file that can be up to 1 MB and we'll backup the last 10 versions of it) for messages above level INFO.
+
error: send log messages (using the detailed formatter) to logs/error.log (a file that can be up to 1 MB and we'll backup the last 10 versions of it) for messages above level ERROR.
+
+
+
[Lines 36-40]: attach our different handlers to our root Logger.
+
+
We chose to use a dictionary to configure our logger but there are other ways such as Python script, configuration file, etc. Click on the different options below to expand and view the respective implementation.
# madewithml/config.py
+importlogging
+
+# Logger
+logging_config={...}
+logging.config.dictConfig(logging_config)
+logger=logging.getLogger()
+
+# Sample messages (note that we use configured `logger` now)
+logger.debug("Used for debugging your code.")
+logger.info("Informative messages from your code.")
+logger.warning("Everything works but there is something to be aware of.")
+logger.error("There's been a mistake with the process.")
+logger.critical("There is something terribly wrong and process may terminate.")
+
+
+DEBUG Used for debugging your code. config.py:71
+INFO Informative messages from your code. config.py:72
+WARNING Everything works but there is something to be aware of. config.py:73
+ERROR There's been a mistake with the process. config.py:74
+CRITICAL There is something terribly wrong and process may terminate. config.py:75
+
+
+
Our logged messages become stored inside the respective files in our logs directory:
+
logs/
+├──info.log
+└──error.log
+
+
And since we defined a detailed formatter, we would see informative log messages like these:
+
+INFO2020-10-21 11:18:42,102 [config.py:module:72]
+Informative messages from your code.
+
+
+
Implementation
+
In our project, we can replace all of our print statements into logging statements:
+
1
print("✅ Training complete!")
+
+
+ ──── becomes: ────
+
+
+
1
+2
fromconfigimportlogger
+logger.info("✅ Training complete!")
+
+
All of our log messages are at the INFO level but while developing we may have had to use DEBUG levels and we also add some ERROR or CRITICAL log messages if our system behaves in an unintended manner.
+
+
+
what: log all the necessary details you want to surface from our application that will be useful during development and afterwards for retrospective inspection.
+
+
+
where: a best practice is to not clutter our modular functions with log statements. Instead we should log messages outside of small functions and inside larger workflows. For example, there are no log messages inside any of our scripts except the main.py and train.py files. This is because these scripts use the smaller functions defined in the other scripts (data.py, evaluate.py, etc.). If we ever feel that we the need to log within our other functions, then it usually indicates that the function needs to be broken down further.
+
+
+
+
When it comes to saving our logs, we could simply upload our logs to a cloud blog storage (ex. S3 or Google Cloud Storage). Or for a more production-grade logging option, we could consider the Elastic stack.
+
+
In the next lesson, we'll learn how to document our code and automatically generate high quality docs for our application.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Logging - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Throughout our development so far, there are so many different commands to keep track of. To help organize everything, we're going to use a Makefile which is a automation tool that organizes our commands. We'll start by create this file in our project's root directory.
+
touchMakefile
+
+
At the top of our Makefile we need to specify the shell environment we want all of our commands to execute in:
+
# Makefile
+SHELL=/bin/bash
+
+
Components
+
Inside our Makefile, we'll be creating a list of rules. These rules have a target which can sometimes have prerequisites that need to be met (can be other targets) and on the next line a Tab followed by a recipe which specifies how to create the target.
+
# Makefile
+target:prerequisites
+<TAB>recipe
+
+
For example, if we wanted to create a rule for styling our files, we would add the following to our Makefile:
&& signifies that we want these commands to execute in one shell (more on this below).
+
+
PHONY
+
A Makefile is called as such because traditionally the targets are supposed to be files we can make. However, Makefiles are also commonly used as command shortcuts, which can lead to confusion when a Makefile target and a file share the same name! For example if we have a file called venv (which we do) and a target in your Makefile called venv, when you run make venv we'll get this message:
+
$makevenv
+
+
+make: `venv' is up to date.
+
+
+
In this situation, this is the intended behavior because if a virtual environment already exists, then we don't want ot make that target again. However, sometimes, we'll name our targets and want them to execute whether it exists as an actual file or not. In these scenarios, we want to define a PHONY target in our makefile by adding this line above the target:
+
.PHONY:<target_name>
+
+
Most of the rules in our Makefile will require the PHONY target because we want them to execute even if there is a file sharing the target's name.
Before we make a target, we can attach prerequisites to them. These can either be file targets that must exist or PHONY target commands that need to be executed prior to making this target. For example, we'll set the style target as a prerequisite for the clean target so that all files are formatted appropriately prior to cleaning them.
We can also set and use variables inside our Makefile to organize all of our rules.
+
+
+
We can set the variables directly inside the Makefile. If the variable isn't defined in the Makefile, then it would default to any environment variable with the same name.
+
# Set variable
+MESSAGE:="hello world"
+
+# Use variable
+greeting:
+@echo${MESSAGE}
+
+
+
+
We can also use variables passed in when executing the rule like so (ensure that the variable is not overridden inside the Makefile):
+
makegreetingMESSAGE="hi"
+
+
+
+
Shells
+
Each line in a recipe for a rule will execute in a separate sub-shell. However for certain recipes such as activating a virtual environment and loading packages, we want to execute all steps in one shell. To do this, we can add the .ONESHELL special target above any target.
However this is only available in Make version 3.82 and above and most Macs currently use version 3.81. You can either update to the current version or chain your commands with && as we did previously:
The last thing we'll add to our Makefile (for now at least) is a help target to the very top. This rule will provide an informative message for this Makefile's capabilities:
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Even though we've trained and thoroughly evaluated our model, the real work begins once we deploy to production. This is one of the fundamental differences between traditional software engineering and ML development. Traditionally, with rule based, deterministic, software, the majority of the work occurs at the initial stage and once deployed, our system works as we've defined it. But with machine learning, we haven't explicitly defined how something works but used data to architect a probabilistic solution. This approach is subject to natural performance degradation over time, as well as unintended behavior, since the data exposed to the model will be different from what it has been trained on. This isn't something we should be trying to avoid but rather understand and mitigate as much as possible. In this lesson, we'll understand the short comings from attempting to capture performance degradation in order to motivate the need for drift detection.
+
+
Tip
+
We highly recommend that you explore this lesson after completing the previous lessons since the topics (and code) are iteratively developed. We did, however, create the monitoring-ml repository for a quick overview with an interactive notebook.
+
+
System health
+
The first step to insure that our model is performing well is to ensure that the actual system is up and running as it should. This can include metrics specific to service requests such as latency, throughput, error rates, etc. as well as infrastructure utilization such as CPU/GPU utilization, memory, etc.
+
+
+
+
+
Fortunately, most cloud providers and even orchestration layers will provide this insight into our system's health for free through a dashboard. In the event we don't, we can easily use Grafana, Datadog, etc. to ingest system performance metrics from logs to create a customized dashboard and set alerts.
+
Performance
+
Unfortunately, just monitoring the system's health won't be enough to capture the underlying issues with our model. So, naturally, the next layer of metrics to monitor involves the model's performance. These could be quantitative evaluation metrics that we used during model evaluation (accuracy, precision, f1, etc.) but also key business metrics that the model influences (ROI, click rate, etc.).
+
It's usually never enough to just analyze the cumulative performance metrics across the entire span of time since the model has been deployed. Instead, we should also inspect performance across a period of time that's significant for our application (ex. daily). These sliding metrics might be more indicative of our system's health and we might be able to identify issues faster by not obscuring them with historical data.
+
+
👉 Follow along interactive notebook in the monitoring-ml repository as we implement the concepts below.
# Cumulative f1
+cumulative_f1=[np.mean(hourly_f1[:n])forninrange(1,len(hourly_f1)+1)]
+print(f"Average cumulative f1 on the last day: {np.mean(cumulative_f1[-24:]):.1f}")
+
+
+Average cumulative f1 on the last day: 93.7
+
+
1
+2
+3
+4
# Sliding f1
+window_size=24
+sliding_f1=np.convolve(hourly_f1,np.ones(window_size)/window_size,mode="valid")
+print(f"Average sliding f1 on the last day: {np.mean(sliding_f1[-24:]):.1f}")
+
We may need to monitor metrics at various window sizes to catch performance degradation as soon as possible. Here we're monitoring the overall f1 but we can do the same for slices of data, individual classes, etc. For example, if we monitor the performance on a specific tag, we may be able to quickly catch new algorithms that were released for that tag (ex. new transformer architecture).
+
+
Delayed outcomes
+
We may not always have the ground-truth outcomes available to determine the model's performance on production inputs. This is especially true if there is significant lag or annotation is required on the real-world data. To mitigate this, we could:
+
+
devise an approximate signal that can help us estimate the model's performance. For example, in our tag prediction task, we could use the actual tags that an author attributes to a project as the intermediary labels until we have verified labels from an annotation pipeline.
+
label a small subset of our live dataset to estimate performance. This subset should try to be representative of the various distributions in the live data.
+
+
Importance weighting
+
However, approximate signals are not always available for every situation because there is no feedback on the ML system’s outputs or it’s too delayed. For these situations, a recent line of research relies on the only component that’s available in all situations: the input data.
The core idea is to develop slicing functions that may potentially capture the ways our data may experience distribution shift. These slicing functions should capture obvious slices such as class labels or different categorical feature values but also slices based on implicit metadata (hidden aspects of the data that are not explicit feature columns). These slicing functions are then applied to our labeled dataset to create matrices with the corresponding labels. The same slicing functions are applied to our unlabeled production data to approximate what the weighted labels would be. With this, we can determine the approximate performance! The intuition here is that we can better approximate performance on our unlabeled dataset based on the similarity between the labeled slice matrix and unlabeled slice matrix. A core dependency of this assumption is that our slicing functions are comprehensive enough that they capture the causes of distributional shift.
+
+
Warning
+
If we wait to catch the model decay based on the performance, it may have already caused significant damage to downstream business pipelines that are dependent on it. We need to employ more fine-grained monitoring to identify the sources of model drift prior to actual performance degradation.
+
+
Drift
+
We need to first understand the different types of issues that can cause our model's performance to decay (model drift). The best way to do this is to look at all the moving pieces of what we're trying to model and how each one can experience drift.
Data drift, also known as feature drift or covariate shift, occurs when the distribution of the production data is different from the training data. The model is not equipped to deal with this drift in the feature space and so, it's predictions may not be reliable. The actual cause of drift can be attributed to natural changes in the real-world but also to systemic issues such as missing data, pipeline errors, schema changes, etc. It's important to inspect the drifted data and trace it back along it's pipeline to identify when and where the drift was introduced.
+
+
Warning
+
Besides just looking at the distribution of our input data, we also want to ensure that the workflows to retrieve and process our input data is the same during training and serving to avoid training-serving skew. However, we can skip this step if we retrieve our features from the same source location for both training and serving, ie. from a feature store.
+
+
+
+
+
+ Data drift can occur in either continuous or categorical features.
+
+
+
+
As data starts to drift, we may not yet notice significant decay in our model's performance, especially if the model is able to interpolate well. However, this is a great opportunity to potentially retrain before the drift starts to impact performance.
+
+
Target drift
+
Besides just the input data changing, as with data drift, we can also experience drift in our outcomes. This can be a shift in the distributions but also the removal or addition of new classes with categorical tasks. Though retraining can mitigate the performance decay caused target drift, it can often be avoided with proper inter-pipeline communication about new classes, schema changes, etc.
+
Concept drift
+
Besides the input and output data drifting, we can have the actual relationship between them drift as well. This concept drift renders our model ineffective because the patterns it learned to map between the original inputs and outputs are no longer relevant. Concept drift can be something that occurs in various patterns:
+
+
+
+
+
+
gradually over a period of time
+
abruptly as a result of an external event
+
periodically as a result of recurring events
+
+
+
All the different types of drift we discussed can can occur simultaneously which can complicated identifying the sources of drift.
+
+
Locating drift
+
Now that we've identified the different types of drift, we need to learn how to locate and how often to measure it. Here are the constraints we need to consider:
+
+
reference window: the set of points to compare production data distributions with to identify drift.
+
test window: the set of points to compare with the reference window to determine if drift has occurred.
+
+
Since we're dealing with online drift detection (ie. detecting drift in live production data as opposed to past batch data), we can employ either a fixed or sliding window approach to identify our set of points for comparison. Typically, the reference window is a fixed, recent subset of the training data while the test window slides over time.
+
Scikit-multiflow provides a toolkit for concept drift detection techniques directly on streaming data. The package offers windowed, moving average functionality (including dynamic preprocessing) and even methods around concepts like gradual concept drift.
+
+
We can also compare across various window sizes simultaneously to ensure smaller cases of drift aren't averaged out by large window sizes.
+
+
Measuring drift
+
Once we have the window of points we wish to compare, we need to know how to compare them.
The first form of measurement can be rule-based such as validating expectations around missing values, data types, value ranges, etc. as we did in our data testing lesson. The difference now is that we'll be validating these expectations on live production data.
+
1
+2
# Simulated production data
+prod_df=ge.dataset.PandasDataset([{"text":"hello"},{"text":0},{"text":"world"}])
+
+
1
+2
+3
+4
# Expectation suite
+df.expect_column_values_to_not_be_null(column="text")
+df.expect_column_values_to_be_of_type(column="text",type_="str")
+expectation_suite=df.get_expectation_suite()
+
+
1
+2
# Validate reference data
+df.validate(expectation_suite=expectation_suite,only_return_failures=True)["statistics"]
+
Once we've validated our rule-based expectations, we need to quantitatively measure drift across the different features in our data.
+
Univariate
+
Our task may involve univariate (1D) features that we will want to monitor. While there are many types of hypothesis tests we can use, a popular option is the Kolmogorov-Smirnov (KS) test.
+
Kolmogorov-Smirnov (KS) test
+
The KS test determines the maximum distance between two distribution's cumulative density functions. Here, we'll measure if there is any drift on the size of our input text feature between two different data subsets.
+
+
Tip
+
While text is a direct feature in our task, we can also monitor other implicit features such as % of unknown tokens in text (need to maintain a training vocabulary), etc. While they may not be used for our machine learning model, they can be great indicators for detecting drift.
Similarly, for categorical data (input features, targets, etc.), we can apply the Pearson's chi-squared test to determine if a frequency of events in production is consistent with a reference distribution.
+
+
We're creating a categorical variable for the # of tokens in our text feature but we could very very apply it to the tag distribution itself, individual tags (binary), slices of tags, etc.
We vectorized our text using tf-idf (to keep modeling simple), which has high dimensionality and is not semantically rich in context. However, typically with text, word/char embeddings are used. So to illustrate what drift detection on multivariate data would look like, let's represent our text using pretrained embeddings.
+
+
Be sure to refer to our embeddings and transformers lessons to learn more about these topics. But note that detecting drift on multivariate text embeddings is still quite difficult so it's typically more common to use these methods applied to tabular features or images.
+
+
We'll start by loading the tokenizer from a pretrained model.
Next, we'll load the pretrained model's weights and use the TransformerEmbedding object to extract the embeddings from the hidden state (averaged across tokens).
Now we need to use a dimensionality reduction method to reduce our representations dimensions into something more manageable (ex. 32 dim) so we can run our two-sample tests on to detect drift. Popular options include:
Autoencoders (AE): networks that consume the inputs and attempt to reconstruct it from an lower dimensional space while minimizing the error. These can either be trained or untrained (the Failing loudly paper recommends untrained).
+
Black box shift detectors (BBSD): the actual model trained on the training data can be used as a dimensionality reducer. We can either use the softmax outputs (multivariate) or the actual predictions (univariate).
# Preprocessing with the reducer
+max_len=100
+batch_size=32
+preprocess_fn=partial(preprocess_drift,model=reducer,tokenizer=tokenizer,
+ max_len=max_len,batch_size=batch_size,device=device)
+
+
Maximum Mean Discrepancy (MMD)
+
After applying dimensionality reduction techniques on our multivariate data, we can use different statistical tests to calculate drift. A popular option is Maximum Mean Discrepancy (MMD), a kernel-based approach that determines the distance between two distributions by computing the distance between the mean embeddings of the features from both distributions.
So far we've applied our drift detection methods on offline data to try and understand what reference window sizes should be, what p-values are appropriate, etc. However, we'll need to apply these methods in the online production setting so that we can catch drift as easy as possible.
+
+
Many monitoring libraries and platforms come with online equivalents for their detection methods.
+
+
Typically, reference windows are large so that we have a proper benchmark to compare our production data points to. As for the test window, the smaller it is, the more quickly we can catch sudden drift. Whereas, a larger test window will allow us to identify more subtle/gradual drift. So it's best to compose windows of different sizes to regularly monitor.
As data starts to flow in, we can use the detector to predict drift at every point. Our detector should detect drift sooner in our drifter dataset than in our normal data.
There are also several considerations around how often to refresh both the reference and test windows. We could base in on the number of new observations or time without drift, etc. We can also adjust the various thresholds (ERT, window size, etc.) based on what we learn about our system through monitoring.
+
Outliers
+
With drift, we're comparing a window of production data with reference data as opposed to looking at any one specific data point. While each individual point may not be an anomaly or outlier, the group of points may cause a drift. The easiest way to illustrate this is to imagine feeding our live model the same input data point repeatedly. The actual data point may not have anomalous features but feeding it repeatedly will cause the feature distribution to change and lead to drift.
+
+
+
+
+
Unfortunately, it's not very easy to detect outliers because it's hard to constitute the criteria for an outlier. Therefore the outlier detection task is typically unsupervised and requires a stochastic streaming algorithm to identify potential outliers. Luckily, there are several powerful libraries such as PyOD, Alibi Detect, WhyLogs (uses Apache DataSketches), etc. that offer a suite of outlier detection functionality (largely for tabular and image data for now).
+
Typically, outlier detection algorithms fit (ex. via reconstruction) to the training set to understand what normal data looks like and then we can use a threshold to predict outliers. If we have a small labeled dataset with outliers, we can empirically choose our threshold but if not, we can choose some reasonable tolerance.
When we identify outliers, we may want to let the end user know that the model's response may not be reliable. Additionally, we may want to remove the outliers from the next training set or further inspect them and upsample them in case they're early signs of what future distributions of incoming features will look like.
+
+
Solutions
+
It's not enough to just be able to measure drift or identify outliers but to also be able to act on it. We want to be able to alert on drift, inspect it and then act on it.
+
Alert
+
Once we've identified outliers and/or measured statistically significant drift, we need to a devise a workflow to notify stakeholders of the issues. A negative connotation with monitoring is fatigue stemming from false positive alerts. This can be mitigated by choosing the appropriate constraints (ex. alerting thresholds) based on what's important to our specific application. For example, thresholds could be:
+
+
fixed values/range for situations where we're concretely aware of expected upper/lower bounds.
+
1
+2
ifpercentage_unk_tokens>5%:
+ trigger_alert()
+
+
forecasted thresholds dependent on previous inputs, time, etc.
+
Once we have our carefully crafted alerting workflows in place, we can notify stakeholders as issues arise via email, Slack, PageDuty, etc. The stakeholders can be of various levels (core engineers, managers, etc.) and they can subscribe to the alerts that are relevant for them.
+
Inspect
+
Once we receive an alert, we need to inspect it before acting on it. An alert needs several components in order for us to completely inspect it:
With these pieces of information, we can work backwards from the alert towards identifying the root cause of the issue. Root cause analysis (RCA) is an important first step when it comes to monitoring because we want to prevent the same issue from impacting our system again. Often times, many alerts are triggered but they maybe all actually be caused by the same underlying issue. In this case, we'd want to intelligently trigger just one alert that pinpoints the core issue. For example, let's say we receive an alert that our overall user satisfaction ratings are reducing but we also receive another alert that our North American users also have low satisfaction ratings. Here's the system would automatically assess for drift in user satisfaction ratings across many different slices and aggregations to discover that only users in a specific area are experiencing the issue but because it's a popular user base, it ends up triggering all aggregate downstream alerts as well!
+
Act
+
There are many different ways we can act to drift based on the situation. An initial impulse may be to retrain our model on the new data but it may not always solve the underlying issue.
+
+
ensure all data expectations have passed.
+
confirm no data schema changes.
+
retrain the model on the new shifted dataset.
+
move the reference window to more recent data or give it more weight.
+
determine if outliers are potentially valid data points.
+
+
Production
+
Since detecting drift and outliers can involve compute intensive operations, we need a solution that can execute serverless workloads on top of our event data streams (ex. Kafka). Typically these solutions will ingest payloads (ex. model's inputs and outputs) and can trigger monitoring workloads. This allows us to segregate the resources for monitoring from our actual ML application and scale them as needed.
+
+
+
+
+
When it actually comes to implementing a monitoring system, we have several options, ranging from fully managed to from-scratch. Several popular managed solutions are Arize, Arthur, Fiddler, Gantry, Mona, WhyLabs, etc., all of which allow us to create custom monitoring views, trigger alerts, etc. There are even several great open-source solutions such as EvidentlyAI, TorchDrift, WhyLogs, etc.
+
We'll often notice that monitoring solutions are offered as part of the larger deployment option such as Sagemaker, TensorFlow Extended (TFX), TorchServe, etc. And if we're already working with Kubernetes, we could use KNative or Kubeless for serverless workload management. But we could also use a higher level framework such as KFServing or Seldon core that natively use a serverless framework like KNative.
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
So far we've implemented our DataOps (ELT, validation, etc.) and MLOps (optimization, training, evaluation, etc.) workflows as Python function calls. This has worked well since our dataset is static and small. But happens when we need to:
+
+
schedule these workflows as new data arrives?
+
scale these workflows as our data grows?
+
share these workflows to downstream applications?
+
monitor these workflows?
+
+
We'll need to break down our end-to-end ML pipeline into individual workflows that can be orchestrated as needed. There are several tools that can help us so this such as Airflow, Prefect, Dagster, Luigi, Orchest and even some ML focused options such as Metaflow, Flyte, KubeFlow Pipelines, Vertex pipelines, etc. We'll be creating our workflows using AirFlow for its:
+
+
wide adoption of the open source platform in industry
+
Python based software development kit (SDK)
+
ability to run locally and scale easily
+
maturity over the years and part of the apache ecosystem
+
+
+
We'll be running Airflow locally but we can easily scale it by running on a managed cluster platform where we can run Python, Hadoop, Spark, etc. on large batch processing jobs (AWS EMR, Google Cloud's Dataproc, on-prem hardware, etc.).
+
+
Airflow
+
Before we create our specific pipelines, let's understand and implement Airflow's overarching concepts that will allow us to "author, schedule, and monitor workflows".
+
+
Separate repository
+
Our work in this lesson will live in a separate repository so create a new directory (outside our mlops-course repository) called data-engineering. All the work in this lesson can be found in our data-engineering repository.
+
+
Set up
+
To install and run Airflow, we can either do so locally or with Docker. If using docker-compose to run Airflow inside Docker containers, we'll want to allocate at least 4 GB in memory.
+
# Configurations
+exportAIRFLOW_HOME=${PWD}/airflow
+AIRFLOW_VERSION=2.3.3
+PYTHON_VERSION="$(python--version|cut-d" "-f2|cut-d"."-f1-2)"
+CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
+
+# Install Airflow (may need to upgrade pip)
+pipinstall"apache-airflow==${AIRFLOW_VERSION}"--constraint"${CONSTRAINT_URL}"
+
+# Initialize DB (SQLite by default)
+airflowdbinit
+
+
This will create an airflow directory with the following components:
We're going to edit the airflow.cfg file to best fit our needs:
+
# Inside airflow.cfg
+enable_xcom_pickling=True# needed for Great Expectations airflow provider
+load_examples=False# don't clutter webserver with examples
+
+
And we'll perform a reset to implement these configuration changes.
+
airflowdbreset-y
+
+
Now we're ready to initialize our database with an admin user, which we'll use to login to access our workflows in the webserver.
+
# We'll be prompted to enter a password
+airflowuserscreate\
+--usernameadmin\
+--firstnameFIRSTNAME\
+--lastnameLASTNAME\
+--roleAdmin\
+--emailEMAIL
+
+
Webserver
+
Once we've created a user, we're ready to launch the webserver and log in using our credentials.
The webserver allows us to run and inspect workflows, establish connections to external data storage, manager users, etc. through a UI. Similarly, we could also use Airflow's REST API or Command-line interface (CLI) to perform the same operations. However, we'll be using the webserver because it's convenient to visually inspect our workflows.
+
+
+
+
+
We'll explore the different components of the webserver as we learn about Airflow and implement our workflows.
+
Scheduler
+
Next, we need to launch our scheduler, which will execute and monitor the tasks in our workflows. The schedule executes tasks by reading from the metadata database and ensures the task has what it needs to finish running. We'll go ahead and execute the following commands on the separate terminal window:
+
# Launch scheduler (in separate terminal)
+sourcevenv/bin/activate
+exportAIRFLOW_HOME=${PWD}/airflow
+exportOBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
+airflowscheduler
+
+
Executor
+
As our scheduler reads from the metadata database, the executor determines what worker processes are necessary for the task to run to completion. Since our default database SQLlite, which can't support multiple connections, our default executor is the Sequential Executor. However, if we choose a more production-grade database option such as PostgresSQL or MySQL, we can choose scalable Executor backends Celery, Kubernetes, etc. For example, running Airflow with Docker uses PostgresSQL as the database and so uses the Celery Executor backend to run tasks in parallel.
+
DAGs
+
Workflows are defined by directed acyclic graphs (DAGs), whose nodes represent tasks and edges represent the data flow relationship between the tasks. Direct and acyclic implies that workflows can only execute in one direction and a previous, upstream task cannot run again once a downstream task has started.
+
+
+
+
+
DAGs can be defined inside Python workflow scripts inside the airflow/dags directory and they'll automatically appear (and continuously be updated) on the webserver. Before we start creating our DataOps and MLOps workflows, we'll learn about Airflow's concepts via an example DAG outlined in airflow/dags/example.py. Execute the following commands in a new (3rd) terminal window:
+
mkdirairflow/dags
+touchairflow/dags/example.py
+
+
Inside each workflow script, we can define some default arguments that will apply to all DAGs within that workflow.
+
1
+2
+3
+4
# Default DAG args
+default_args={
+ "owner":"airflow",
+}
+
+
+
Typically, our DAGs are not the only ones running in an Airflow cluster. However, it can be messy and sometimes impossible to execute different workflows when they require different resources, package versions, etc. For teams with multiple projects, it’s a good idea to use something like the KubernetesPodOperator to execute each job using an isolated docker image.
+
+
We can initialize DAGs with many parameters (which will override the same parameters in default_args) and in several different ways:
There are many parameters that we can initialize our DAGs with, including a start_date and a schedule_interval. While we could have our workflows execute on a temporal cadence, many ML workflows are initiated by events, which we can map using sensors and hooks to external databases, file systems, etc.
+
+
Tasks
+
Tasks are the operations that are executed in a workflow and are represented by nodes in a DAG. Each task should be a clearly defined single operation and it should be idempotent, which means we can execute it multiple times and expect the same result and system state. This is important in the event we need to retry a failed task and don't have to worry about resetting the state of our system. Like DAGs, there are several different ways to implement tasks:
Though the graphs are directed, we can establish certain trigger rules for each task to execute on conditional successes or failures of the parent tasks.
+
+
Operators
+
The first method of creating tasks involved using Operators, which defines what exactly the task will be doing. Airflow has many built-in Operators such as the BashOperator or PythonOperator, which allow us to execute bash and Python commands respectively.
There are also many other Airflow native Operators (email, S3, MySQL, Hive, etc.), as well as community maintained provider packages (Kubernetes, Snowflake, Azure, AWS, Salesforce, Tableau, etc.), to execute tasks specific to certain platforms or tools.
Once we've defined our tasks using Operators or as decorated functions, we need to define the relationships between them (edges). The way we define the relationships depends on how our tasks were defined:
+
+
+
using decorated functions
+
1
+2
+3
# Task relationships
+x=task_1()
+y=task_2(x=x)
+
+
+
+
using Operators
+
1
+2
+3
# Task relationships
+task_1>>task_2# same as task_1.set_downstream(task_2) or
+ # task_2.set_upstream(task_1)
+
+
+
+
In both scenarios, we'll setting task_2 as the downstream task to task_1.
+
+
Note
+
We can even create intricate DAGs by using these notations to define the relationships.
When we use task decorators, we can see how values can be passed between tasks. But, how can we pass values when using Operators? Airflow uses XComs (cross communications) objects, defined with a key, value, timestamp and task_id, to push and pull values between tasks. When we use decorated functions, XComs are being used under the hood but it's abstracted away, allowing us to pass values amongst Python functions seamlessly. But when using Operators, we'll need to explicitly push and pull the values as we need it.
We can also view our XComs on the webserver by going to Admin >> XComs:
+
+
+
+
+
+
Warning
+
The data we pass between tasks should be small (metadata, metrics, etc.) because Airflow's metadata database is not equipped to hold large artifacts. However, if we do need to store and use the large results of our tasks, it's best to use an external data storage (blog storage, model registry, etc.) and perform heavy processing using Spark or inside data systems like a data warehouse.
+
+
DAG runs
+
Once we've defined the tasks and their relationships, we're ready to run our DAGs. We'll start defining our DAG like so:
+
1
+2
+3
# Run DAGs
+example1_dag=example_1()
+example2_dag=example_2()
+
+
The new DAG will have appeared when we refresh our Airflow webserver.
+
Manual
+
Our DAG is initially paused since we specified dags_are_paused_at_creation = True inside our airflow.cfg configuration, so we'll have to manually execute this DAG by clicking on it > unpausing it (toggle) > triggering it (button). To view the logs for any of the tasks in our DAG run, we can click on the task > Log.
Had we specified a start_date and schedule_interval when defining the DAG, it would have have automatically executed at the appropriate times. For example, the DAG below will have started two days ago and will be triggered at the start of every day.
Depending on the start_date and schedule_interval, our workflow should have been triggered several times and Airflow will try to catchup to the current time. We can avoid this by setting catchup=False when defining the DAG. We can also set this configuration as part of the default arguments:
However, if we did want to run particular runs in the past, we can manually backfill what we need.
+
+
We could also specify a cron expression for our schedule_interval parameter or even use cron presets.
+
+
Airflow's Scheduler will run our workflows one schedule_interval from the start_date. For example, if we want our workflow to start on 01-01-1983 and run @daily, then the first run will be immediately after 01-01-1983T11:59.
+
+
Sensors
+
While it may make sense to execute many data processing workflows on a scheduled interval, machine learning workflows may require more nuanced triggers. We shouldn't be wasting compute by running executing our workflows just in case we have new data. Instead, we can use sensors to trigger workflows when some external condition is met. For example, we can initiate data processing when a new batch of annotated data appears in a database or when a specific file appears in a file system, etc.
+
+
There's so much more to Airflow (monitoring, Task groups, smart senors, etc.) so be sure to explore them as you need them by using the official documentation.
+
+
DataOps
+
Now that we've reviewed Airflow's major concepts, we're ready to create the DataOps workflows. It's the exact same workflow we defined in our data stack lesson -- extract, load and transform -- but this time we'll be doing everything programmatically and orchestrating it with Airflow.
+
+
+
+
+
We'll start by creating the script where we'll define our workflows:
We're going to use the Airbyte connections we set up in our data-stack lesson but this time we're going to programmatically trigger the data syncs with Airflow. First, let's ensure that Airbyte is running on a separate terminal in it's repository:
+
gitclonehttps://github.com/airbytehq/airbyte.git# skip if already create in data-stack lesson
+cdairbyte
+docker-composeup
+
+
Next, let's install the required packages and establish the connection between Airbyte and Airflow:
The string in the CONNECTION_ID position is the connection's id.
+
+
We can trigger our DAG right now and view the extracted data be loaded into our BigQuery data warehouse but we'll continue developing and execute our DAG once the entire DataOps workflow has been defined.
+
Validate
+
The specific process of where and how we extract our data can be bespoke but what's important is that we have validation at every step of the way. We'll once again use Great Expectations, as we did in our testing lesson, to validate our extracted and loaded data before transforming it.
+
With the Airflow concepts we've learned so far, there are many ways to use our data validation library to validate our data. Regardless of what data validation tool we use (ex. Great Expectations, TFX, AWS Deequ, etc.) we could use the BashOperator, PythonOperator, etc. to run our tests. However, Great Expectations has a Airflow Provider package to make it even easier to validate our data. This package contains a GreatExpectationsOperator which we can use to execute specific checkpoints as tasks.
But first, before we can create our tests, we need to define a new datasource within Great Expectations for our Google BigQuery data warehouse. This will require several packages and exports:
This will open up an interactive notebook where we can define our expectations. Repeat the same for creating a suite for our tags data asset as well.
+
+Expectations for mlops_course.projects
+
Table expectations
+
1
+2
# data leak
+validator.expect_compound_columns_to_be_unique(column_list=["title","description"])
+
+
Column expectations:
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
# id
+validator.expect_column_values_to_be_unique(column="id")
+
+# create_on
+validator.expect_column_values_to_not_be_null(column="created_on")
+
+# title
+validator.expect_column_values_to_not_be_null(column="title")
+validator.expect_column_values_to_be_of_type(column="title",type_="STRING")
+
+# description
+validator.expect_column_values_to_not_be_null(column="description")
+validator.expect_column_values_to_be_of_type(column="description",type_="STRING")
+
+
+
+Expectations for mlops_course.tags
+
Column expectations:
+
1
+2
+3
+4
+5
+6
# id
+validator.expect_column_values_to_be_unique(column="id")
+
+# tag
+validator.expect_column_values_to_not_be_null(column="tag")
+validator.expect_column_values_to_be_of_type(column="tag",type_="STRING")
+
+
+
Checkpoints
+
Once we have our suite of expectations, we're ready to check checkpoints to execute these expectations:
+
great_expectationscheckpointnewprojects
+
+
This will, of course, open up an interactive notebook. Just ensure that the following information is correct (the default values may not be):
+
Once we've validated our extracted and loaded data, we're ready to transform it. Our DataOps workflows are not specific to any particular downstream application so the transformation must be globally relevant (ex. cleaning missing data, aggregation, etc.). Just like in our data stack lesson, we're going to use dbt to transform our data. However, this time, we're going to do everything programmatically using the open-source dbt-core package.
+
In the root of our data-engineering repository, initialize our dbt directory with the following command:
+
While we developed locally, we could just as easily use Airflow’s dbt cloud provider to connect to our dbt cloud and use the different operators to schedule jobs. This is recommended for production because we can design jobs with proper environment, authentication, schemas, etc.
+
+
Connect Airflow with dbt Cloud:
+
+
Go to Admin > Connections > +
+
Connection ID:dbt_cloud_default
+Connection Type:dbt Cloud
+Account ID:View in URL of https://cloud.getdbt.com/
+API Token:View in https://cloud.getdbt.com/#/profile/api/
+
fromairflow.providers.dbt.cloud.operators.dbtimportDbtCloudRunJobOperator
+transform=DbtCloudRunJobOperator(
+ task_id="transform",
+ job_id=118680,# Go to dbt UI > click left menu > Jobs > Transform > job_id in URL
+ wait_for_termination=True,# wait for job to finish running
+ check_interval=10,# check job status
+ timeout=300,# max time for job to execute
+)
+
+
+
Validate
+
And of course, we'll want to validate our transformations beyond dbt's built-in methods, using great expectations. We'll create a suite and checkpoint as we did above for our projects and tags data assets.
+
great_expectationssuitenew# for mlops_course.labeled_projects
+
+
+Expectations for mlops_course.labeled_projects
+
Table expectations
+
1
+2
# data leak
+validator.expect_compound_columns_to_be_unique(column_list=["title","description"])
+
Now we have our entire DataOps DAG define and executing it will prepare our data from extraction to loading to transformation (and with validation at every step of the way) for downstream applications.
+
+
+
+
+
+
Typically we'll use sensors to trigger workflows when a condition is met or trigger them directly from the external source via API calls, etc. For our ML use cases, this could be at regular intervals or when labeling or monitoring workflows trigger retraining, etc.
+
+
MLOps
+
Once we have our data prepared, we're ready to create one of the many potential downstream applications that will depend on it. Let's head back to our mlops-course project and follow the same set up instructions for Airflow (you can stop the Airflow webserver and scheduler from our data-engineering project since we'll reuse PORT 8000).
+
+
+
# Airflow webserver
+source venv/bin/activate
+exportAIRFLOW_HOME=${PWD}/airflow
+exportGOOGLE_APPLICATION_CREDENTIALS=/Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json # REPLACE
+airflow webserver --port 8080
+# Go to http://localhost:8080
+
We already had an tagifai.elt_data function defined to prepare our data but if we want to leverage the data inside our data warehouse, we'll want to connect to it.
# airflow/dags/workflows.py
+fromgoogle.cloudimportbigquery
+fromgoogle.oauth2importservice_account
+
+PROJECT_ID="made-with-ml-XXXXX"# REPLACE
+SERVICE_ACCOUNT_KEY_JSON="/Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json"# REPLACE
+
+def_extract_from_dwh():
+"""Extract labeled data from
+ our BigQuery data warehouse and
+ save it locally."""
+ # Establish connection to DWH
+ credentials=service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_KEY_JSON)
+ client=bigquery.Client(credentials=credentials,project=PROJECT_ID)
+
+ # Query data
+ query_job=client.query("""
+ SELECT *
+ FROM mlops_course.labeled_projects""")
+ results=query_job.result()
+ results.to_dataframe().to_csv(Path(config.DATA_DIR,"labeled_projects.csv"),index=False)
+
+@dag(
+ dag_id="mlops",
+ description="MLOps tasks.",
+ default_args=default_args,
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=["mlops"],
+)
+defmlops():
+"""MLOps workflows."""
+ extract_from_dwh=PythonOperator(
+ task_id="extract_data",
+ python_callable=_extract_from_dwh,
+ )
+
+ # Define DAG
+ extract_from_dwh
+
+
Validate
+
Next, we'll use Great Expectations to validate our data. Even though we've already validated our data, it's a best practice to test for data quality whenever there is a hand-off of data from one place to another. We've already created a checkpoint for our labeled_projects in our testing lesson so we'll just leverage that inside our MLOps DAG.
And with that we have our MLOps workflow defined that uses the prepared data from our DataOps workflow. At this point, we can add additional tasks for offline/online evaluation, deployment, etc. with the same process as above.
+
+
+
+
+
Continual learning
+
The DataOps and MLOps workflows connect to create an ML system that's capable of continually learning. Such a system will guide us with when to update, what exactly to update and how to update it (easily).
+
+
We use the word continual (repeat with breaks) instead of continuous (repeat without interruption / intervention) because we're not trying to create a system that will automatically update with new incoming data without human intervention.
+
+
Monitoring
+
Our production system is live and monitored. When an event of interest occurs (ex. drift), one of several events needs to be triggered:
+
+
continue: with the currently deployed model without any updates. However, an alert was raised so it should analyzed later to reduce false positive alerts.
+
improve: by retraining the model to avoid performance degradation causes by meaningful drift (data, target, concept, etc.).
+
inspect: to make a decision. Typically expectations are reassessed, schemas are reevaluated for changes, slices are reevaluated, etc.
+
rollback: to a previous version of the model because of an issue with the current deployment. Typically these can be avoided using robust deployment strategies (ex. dark canary).
+
+
Retraining
+
If we need to improve on the existing version of the model, it's not just the matter of fact of rerunning the model creation workflow on the new dataset. We need to carefully compose the training data in order to avoid issues such as catastrophic forgetting (forget previously learned patterns when presented with new data).
+
+
labeling: new incoming data may need to be properly labeled before being used (we cannot just depend on proxy labels).
+
activelearning: we may not be able to explicitly label every single new data point so we have to leverage active learning workflows to complete the labeling process.
+
QA: quality assurance workflows to ensure that labeling is accurate, especially for known false positives/negatives and historically poorly performing slices of data.
+
augmentation: increasing our training set with augmented data that's representative of the original dataset.
+
sampling: upsampling and downsampling to address imbalanced data slices.
+
evaluation: creation of an evaluation dataset that's representative of what the model will encounter once deployed.
+
+
Once we have the proper dataset for retraining, we can kickoff the workflows to update our system!
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Orchestration for Machine Learning - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Before performing a commit to our local repository, there are a lot of items on our mental todo list, ranging from styling, formatting, testing, etc. And it's very easy to forget some of these steps, especially when we want to "push to quick fix". To help us manage all these important steps, we can use pre-commit hooks, which will automatically be triggered when we try to perform a commit. These hooks can ensure that certain rules are followed or specific actions are executed successfully and if any of them fail, the commit will be aborted.
+
Installation
+
We'll be using the Pre-commit framework to help us automatically perform important checks via hooks when we make a commit.
+
+
We'll start by installing and autoupdating pre-commit (we only have to do this once).
+
pre-commitinstall
+pre-commitautoupdate
+
+
+
Config
+
We define our pre-commit hooks via a .pre-commit-config.yaml configuration file. We can either create our yaml configuration from scratch or use the pre-commit CLI to create a sample configuration which we can add to.
When it comes to creating and using hooks, we have several options to choose from.
+
Built-in
+
Inside the sample configuration, we can see that pre-commit has added some default hooks from it's repository. It specifies the location of the repository, version as well as the specific hook ids to use. We can read about the function of these hooks and add even more by exploring pre-commit's built-in hooks. Many of them also have additional arguments that we can configure to customize the hook.
Be sure to explore the many other built-in hooks because there are some really useful ones that we use in our project. For example, check-merge-conflict to see if there are any lingering merge conflict strings or detect-aws-credentials if we accidentally left our credentials exposed in a file, and so much more.
+
+
And we can also exclude certain files from being processed by the hooks by using the optional exclude key. There are many other optional keys we can configure for each hook ID.
Besides pre-commit's built-in hooks, there are also many custom, 3rd party popular hooks that we can choose from. For example, if we want to apply formatting checks with Black as a hook, we can leverage Black's pre-commit hook.
This specific hook is defined under a .pre-commit-hooks.yaml inside Black's repository, as are other custom hooks under their respective package repositories.
+
Local
+
We can also create our own local hooks without configuring a separate .pre-commit-hooks.yaml. Here we're defining two pre-commit hooks, test-non-training and clean, to run some commands that we've defined in our Makefile. Similarly, we can run any entry command with arguments to create hooks very quickly.
Our pre-commit hooks will automatically execute when we try to make a commit. We'll be able to see if each hook passed or failed and make any changes. If any of the hooks fail, we have to fix the errors ourselves or, in many instances, reformatting will occur automatically.
In the event that any of the hooks failed, we need to add and commit again to ensure that all hooks are passed.
+
gitadd.
+gitcommit-m<MESSAGE>
+
+
+
+
+
+
Run
+
Though pre-commit hooks are meant to run before (pre) a commit, we can manually trigger all or individual hooks on all or a set of files.
+
# Run
+pre-commitrun--all-files# run all hooks on all files
+pre-commitrun<HOOK_ID>--all-files# run one hook on all files
+pre-commitrun--files<PATH_TO_FILE># run all hooks on a file
+pre-commitrun<HOOK_ID>--files<PATH_TO_FILE># run one hook on a file
+
+
Skip
+
It is highly not recommended to skip running any of the pre-commit hooks because they are there for a reason. But for some highly urgent, world saving commits, we can use the no-verify flag.
+
# Commit without hooks
+gitcommit-m<MESSAGE>--no-verify
+
+
+
Highly recommend not doing this because no commit deserves to be force pushed no matter how "small" your change was. If you accidentally did this and want to clear the cache, run pre-commitrun--all-files and execute the commit message operation again.
+
+
Update
+
In our .pre-commit-config.yaml configuration files, we've had to specify the versions for each of the repositories so we can use their latest hooks. Pre-commit has an autoupdate CLI command which will update these versions as they become available.
+
# Autoupdate
+pre-commitautoupdate
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Pre-commit - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
We'll start by first preparing our data by ingesting it from source and splitting it into training, validation and test data splits.
+
Ingestion
+
Our data could reside in many different places (databases, files, etc.) and exist in different formats (CSV, JSON, Parquet, etc.). For our application, we'll load the data from a CSV file to a Pandas DataFrame using the read_csv function.
# Data ingestion
+DATASET_LOC="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/dataset.csv"
+df=pd.read_csv(DATASET_LOC)
+df.head()
+
+
+
+
+
+
+
id
+
created_on
+
title
+
description
+
tag
+
+
+
+
+
0
+
6
+
2020-02-20 06:43:18
+
Comparison between YOLO and RCNN on real world...
+
Bringing theory to experiment is cool. We can ...
+
computer-vision
+
+
+
1
+
7
+
2020-02-20 06:47:21
+
Show, Infer & Tell: Contextual Inference for C...
+
The beauty of the work lies in the way it arch...
+
computer-vision
+
+
+
2
+
9
+
2020-02-24 16:24:45
+
Awesome Graph Classification
+
A collection of important graph embedding, cla...
+
other
+
+
+
3
+
15
+
2020-02-28 23:55:26
+
Awesome Monte Carlo Tree Search
+
A curated list of Monte Carlo tree search pape...
+
other
+
+
+
4
+
25
+
2020-03-07 23:04:31
+
AttentionWalk
+
A PyTorch Implementation of "Watch Your Step: ...
+
other
+
+
+
+
+
+
+
In our data engineering lesson we'll look at how to continually ingest data from more complex sources (ex. data warehouses)
+
+
Splitting
+
Next, we need to split our training dataset into train and val data splits.
+
+
Use the train split to train the model.
+
Here the model will have access to both inputs (features) and outputs (labels) to optimize its internal weights.
+
+
+
After each iteration (epoch) through the training split, we will use the val split to determine the model's performance.
+
Here the model will not use the labels to optimize its weights but instead, we will use the validation performance to optimize training hyperparameters such as the learning rate, etc.
+
+
+
Finally, we will use a separate holdout test dataset to determine the model's performance after training.
+
This is our best measure of how the model may behave on new, unseen data that is from a similar distribution to our training dataset.
+
+
+
+
+
Tip
+
For our application, we will have a training dataset to split into train and val splits and a separatetesting dataset for the test set. While we could have one large dataset and split that into the three splits, it's a good idea to have a separate test dataset. Over time, our training data may grow and our test splits will look different every time. This will make it difficult to compare models against other models and against each other.
For our multi-class task (where each project has exactly one tag), we want to ensure that the data splits have similar class distributions. We can achieve this by specifying how to stratify the split by using the stratify keyword argument with sklearn's train_test_split() function.
+
+
Creating proper data splits
+
What are the criteria we should focus on to ensure proper data splits?
+
+Show answer
+
+
the dataset (and each data split) should be representative of data we will encounter
+
equal distributions of output values across all splits
+
shuffle your data if it's organized in a way that prevents input variance
+
avoid random shuffles if your task can suffer from data leaks (ex. time-series)
Before we view our validation split's class counts, recall that our validation split is only test_size of the entire dataset. So we need to adjust the value counts so that we can compare it to the training split's class counts.
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Data preprocessing can be categorized into two types of processes: preparation and transformation. We'll explore common preprocessing techniques and then we'll preprocess our dataset.
+
+
Warning
+
Certain preprocessing steps are global (don't depend on our dataset, ex. lower casing text, removing stop words, etc.) and others are local (constructs are learned only from the training split, ex. vocabulary, standardization, etc.). For the local, dataset-dependent preprocessing steps, we want to ensure that we split the data first before preprocessing to avoid data leaks.
+
+
Preparing
+
Preparing the data involves organizing and cleaning the data.
+
Joins
+
Performing SQL joins with existing data tables to organize all the relevant data you need into one view. This makes working with our dataset a whole lot easier.
+
1
+2
SELECT*FROMA
+INNERJOINBonA.id==B.id
+
+
+
Warning
+
We need to be careful to perform point-in-time valid joins to avoid data leaks. For example, if Table B may have features for objects in Table A that were not available at the time inference would have been needed.
+
+
Missing values
+
First, we'll have to identify the rows with missing values and once we do, there are several approaches to dealing with them.
+
+
+
omit samples with missing values (if only a small subset are missing it)
+
1
+2
+3
+4
+5
+6
# Drop a row (sample) by index
+df.drop([4,10,...])
+# Conditionally drop rows (samples)
+df=df[df.value>0]
+# Drop samples with any missing feature
+df=df[df.isnull().any(axis=1)]
+
+
+
+
omit the entire feature (if too many samples are missing the value)
+
1
+2
# Drop a column (feature)
+df.drop(["A"],axis=1)
+
+
+
+
fill in missing values for features (using domain knowledge, heuristics, etc.)
+
1
+2
# Fill in missing values with mean
+df.A=df.A.fillna(df.A.mean())
+
+
+
+
may not always seem "missing" (ex. 0, null, NA, etc.)
+
1
+2
+3
# Replace zeros to NaNs
+importnumpyasnp
+df.A=df.A.replace({"0":np.nan,0:np.nan})
+
+
+
+
Outliers (anomalies)
+
+
craft assumptions about what is a "normal" expected value
+
1
+2
# Ex. Feature value must be within 2 standard deviations
+df[np.abs(df.A-df.A.mean())<=(2*df.A.std())]
+
+
be careful not to remove important outliers (ex. fraud)
+
values may not be outliers when we apply a transformation (ex. power law)
+
anomalies can be global (point), contextual (conditional) or collective (individual points are not anomalous and the collective group is an outlier)
+
+
Feature engineering
+
Feature engineering involves combining features in unique ways to draw out signal.
+
1
+2
# Input
+df.C=df.A+df.B
+
+
+
Tip
+
Feature engineering can be done in collaboration with domain experts that can guide us on what features to engineer and use.
+
+
Cleaning
+
Cleaning our data involves apply constraints to make it easier for our models to extract signal from the data.
+
+
use domain expertise and EDA
+
apply constraints via filters
+
ensure data type consistency
+
removing data points with certain or null column values
binning: convert a continuous feature into categorical using bins
+
1
+2
+3
+4
+5
+6
+7
+8
# Binning
+importnumpyasnp
+x=np.random.random(4)# values between 0 and 1
+print("x:",x)
+bins=np.linspace(0,1,5)# bins between 0 and 1
+print("bins:",bins)
+binned=np.digitize(x,bins)
+print("binned:",binned)
+
We'll often was to retrieve feature values for an entity (user, item, etc.) over time and reuse the same features across different projects. To ensure that we're retrieving the proper feature values and to avoid duplication of efforts, we can use a feature store.
+
+
+
Curse of dimensionality
+
What can we do if a feature has lots of unique values but enough data points for each unique value (ex. URL as a feature)?
+
+Show answer
+
We can encode our data with hashing or using it's attributes instead of the exact entity itself. For example, representing a user by their location and favorites as opposed to using their user ID or representing a webpage with it's domain as opposed to the exact url. This methods effectively decrease the total number of unique feature values and increase the number of data points for each.
+
+
+
Implementation
+
For our application, we'll be implementing a few of these preprocessing steps that are relevant for our dataset.
We can combine existing input features to create new meaningful signal for helping the model learn. However, there's usually no simple way to know if certain feature combinations will help or not without empirically experimenting with the different combinations. Here, we could use a project's title and description separately as features but we'll combine them to create one input feature.
+
1
+2
# Input
+df["text"]=df.title+" "+df.description
+
+
Cleaning
+
Since we're dealing with text data, we can apply some common text preprocessing operations. Here, we'll be using Python's built-in regular expressions library re and the Natural Language Toolkit nltk.
defclean_text(text,stopwords=STOPWORDS):
+"""Clean raw text string."""
+ # Lower
+ text=text.lower()
+
+ # Remove stopwords
+ pattern=re.compile(r'\b('+r"|".join(stopwords)+r")\b\s*")
+ text=pattern.sub('',text)
+
+ # Spacing and filters
+ text=re.sub(r"([!\"'#$%&()*\+,-./:;<=>?@\\\[\]^_`{|}~])",r" \1 ",text)# add spacing
+ text=re.sub("[^A-Za-z0-9]+"," ",text)# remove non alphanumeric chars
+ text=re.sub(" +"," ",text)# remove multiple spaces
+ text=text.strip()# strip white space at the ends
+ text=re.sub(r"http\S+","",text)# remove links
+
+ returntext
+
+
+
Note
+
We could definitely try and include emojis, punctuations, etc. because they do have a lot of signal for the task but it's best to simplify the initial feature set we use to just what we think are the most influential and then we can slowly introduce other features and assess utility.
+
+
Once we're defined our function, we can apply it to each row in our dataframe via pandas.DataFrame.apply.
+
1
+2
+3
+4
# Apply to dataframe
+original_df=df.copy()
+df.text=df.text.apply(clean_text)
+print(f"{original_df.text.values[0]}\n{df.text.values[0]}")
+
+
+Comparison between YOLO and RCNN on real world videos Bringing theory to experiment is cool. We can easily train models in colab and find the results in minutes.
+comparison yolo rcnn real world videos bringing theory experiment cool easily train models colab find results minutes
+
+
+
+
Warning
+
We'll want to introduce less frequent features as they become more frequent or encode them in a clever way (ex. binning, extract general attributes, common n-grams, mean encoding using other feature values, etc.) so that we can mitigate the feature value dimensionality issue until we're able to collect more data.
We'll also want to be able to decode our predictions back into text labels. We can do this by creating an index_to_class dictionary and using that to convert encoded labels back into text labels.
Next we'll encode our text as well. Instead of using a random dictionary, we'll use a tokenizer that was used for a pretrained LLM (scibert) to tokenize our text. We'll be fine-tuning this exact model later when we train our model.
The tokenizer will convert our input text into a list of token ids and a list of attention masks. The token ids are the indices of the tokens in the vocabulary. The attention mask is a binary mask indicating the position of the token indices so that the model can attend to them (and ignore the pad tokens).
+
1
+2
+3
+4
+5
+6
+7
# Bert tokenizer
+tokenizer=BertTokenizer.from_pretrained("allenai/scibert_scivocab_uncased",return_dict=False)
+text="Transfer learning with transformers for text classification."
+encoded_inputs=tokenizer([text],return_tensors="np",padding="longest")# pad to longest item in batch
+print("input_ids:",encoded_inputs["input_ids"])
+print("attention_mask:",encoded_inputs["attention_mask"])
+print(tokenizer.decode(encoded_inputs["input_ids"][0]))
+
+
+input_ids: [[ 102 2268 1904 190 29155 168 3267 2998 205 103]]
+attention_mask: [[1 1 1 1 1 1 1 1 1 1]]
+[CLS] transfer learning with transformers for text classification. [SEP]
+
+
+
+
Note that we use padding="longest" in our tokenizer function to pad our inputs to the longest item in the batch. This becomes important when we use batches of inputs later and want to create a uniform input size, where shorted text sequences will be padded with zeros to meet the length of the longest input in the batch.
+
+
We'll wrap our tokenization into a tokenize function that we can use to tokenize batches of our data.
We'll wrap up by combining all of our preprocessing operations into function. This way we can easily apply it to different datasets (training, inference, etc.)
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Preprocessing - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
Before we start developing any machine learning models, we need to first motivate and design our application. While this is a technical course, this initial product design process is extremely crucial for creating great products. We'll focus on the product design aspects of our application in this lesson and the systems design aspects in the next lesson.
+
Template
+
The template below is designed to guide machine learning product development. It involves both the product and systems design (next lesson) aspects of our application:
👉 Download a PDF of the ML canvas to use for your own products → ml-canvas.pdf (right click the link and hit "Save Link As...")
+
+
Product design
+
Motivate the need for the product and outline the objectives and impact.
+
+
Note
+
Each section below has a part called "Our task", which will discuss how the specific topic relates to the application that we will be building.
+
+
Background
+
Set the scene for what we're trying to do through a user-centric approach:
+
+
users: profile/persona of our users
+
goals: our users' main goals
+
pains: obstacles preventing our users from achieving their goals
+
+
+
Our task
+
+
users: machine learning developers and researchers.
+
goals: stay up-to-date on ML content for work, knowledge, etc.
+
pains: too much unlabeled content scattered around the internet.
+
+
+
Value proposition
+
Propose the value we can create through a product-centric approach:
+
+
product: what needs to be built to help our users reach their goals?
+
alleviates: how will the product reduce pains?
+
advantages: how will the product create gains?
+
+
+
Our task
+
We will build a platform that helps machine learning developers and researchers stay up-to-date on ML content. We'll do this by discovering and categorizing content from popular sources (Reddit, Twitter, etc.) and displaying it on our platform. For simplicity, assume that we already have a pipeline that delivers ML content from popular sources to our platform. We will just focus on developing the ML service that can correctly categorize the content.
+
+
product: a service that discovers and categorizes ML content from popular sources.
+
alleviates: display categorized content for users to discover.
+
advantages: when users visit our platform to stay up-to-date on ML content, they don't waste time searching for that content themselves in the noisy internet.
+
+
+
+
+
+
Objectives
+
Breakdown the product into key objectives that we want to focus on.
+
+
Our task
+
+
Discover ML content from trusted sources to bring into our platform.
+
Classify incoming content for our users to easily discover. [OUR FOCUS]
+
Display categorized content on our platform (recent, popular, recommended, etc.)
+
+
+
Solution
+
Describe the solution required to meet our objectives, including its:
+
+
corefeatures: key features that will be developed.
+
integration: how the product will integrate with other services.
+
alternatives: alternative solutions that we should considered.
+
constraints: limitations that we need to be aware of.
+
out-of-scope.: features that we will not be developing for now.
+
+
+
Our task
+
Develop a model that can classify the content so that it can be organized by category (tag) on our platform.
+
Corefeatures:
+
+
predict the correct tag for a given content. [OUR FOCUS]
+
user feedback process for incorrectly classified content.
+
workflows to categorize ML content that our model is incorrect / unsure about.
+
+
Integrations:
+
+
ML content from reliable sources will be sent to our service for classification.
+
+
Alternatives:
+
+
allow users to add content manually and classify them (noisy, cold start, etc.)
+
+
Constraints:
+
+
maintain low latency (>100ms) when classifying incoming content. [Latency]
+
only recommend tags from our list of approved tags. [Security]
+
avoid duplicate content from being added to the platform. [UI/UX]
+
+
Out-of-scope:
+
+
identify relevant tags beyond our approved list of tags (natural-language-processing, computer-vision, mlops and other).
+
using full-text HTML from content links to aid in classification.
+
+
+
Feasibility
+
How feasible is our solution and do we have the required resources to deliver it (data, $, team, etc.)?
+
+
Our task
+
We have a dataset with ML content that has been labeled. We'll need to assess if it has the necessary signals to meet our objectives.
+
Sample data point
1
+2
+3
+4
+5
+6
+7
{
+"id":443,
+"created_on":"2020-04-10 17:51:39",
+"title":"AllenNLP Interpret",
+"description":"A Framework for Explaining Predictions of NLP Models",
+"tag":"natural-language-processing"
+}
+
+
+
Now that we've set up the product design requirements for our ML service, let's move on to the systems design requirements in the next lesson.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Product - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
In this lesson, we'll discuss how to migrate and organize code from our notebook to Python scripts. We'll be using VSCode in this course, but feel free to use any editor you feel comfortable with.
+
Notebooks have been great so far for development. They're interactive, stateful (don't have to rerun code), and allow us to visualize outputs. However, when we want to a develop quality codebase, we need to move to scripts. Here are some reasons why:
+
+
+
stateless: when we run code in a notebook, it's automatically saved to the global state (memory). This is great for experimentation because code and variables will be readily available across different cells. However, this can be very problematic as well because there can be hidden state that we're not aware of. Scripts, on the other hand, are stateless and we have to explicitly pass variables to functions and classes.
+
+
+
linear: in notebooks, the order in which we execute cells matter. This can be problematic because we can easily execute cells out of order. Scripts, on the other hand, are linear and we have to explicitly execute code for each workload.
+
+
+
testing: As we'll see in our testing lesson, it's significantly easier to compose and run tests on scripts, as opposed to Jupyter notebooks. This is crucial for ensuring that we have quality code that works as expected.
+
+
+
Setup
+
We already have all the scripts provided in our repository so let's discuss how this was all organized.
+
README
+
It's always a good idea to start organizing our scripts with a README.md file. This is where we can organize all of the instructions necessary to walkthrough our codebase. Our README has information on how to set up our environment, how to run our scripts, etc.
+
+
The contents of the README.md file is what everyone will see when they visit your repository on GitHub. So, it's a good idea to keep it updated with the latest information.
+
+
Scripts
+
Let's start by moving our code from notebooks to scripts. We're going to start by creating the different files and directories that we'll need for our project. The exact number and name of these scripts is entirely up to us, however, it's best to organize and choose names that relate to a specific workload. For example, data.py will have all of our data related functions and classes. And we can also have scripts for configurations (config.py), shared utilities (utils.py), etc.
Don't worry about the contents in these files that aren't from our notebooks just yet or if our code looks significantly more documented. We'll be taking a closer look at those in the respective lessons.
+
+
Functions and classes
+
Once we have these ready, we can start moving code from our notebooks to the appropriate scripts. It should intuitive in which script a particular function or class belongs to. If not, we need to rethink how the names of our scripts. For example, train.py has functions from our notebook such as train_step, val_step, train_loop_per_worker, etc.
These code cells are not part of a function or class, so we need to wrap them around a function so that we can easily execute that workload. For example, all of this training logic is wrapped inside a train_model function in train.py that has all the required inputs to execute the workload:
In the next lesson on command-line interfaces (CLI), we'll learn how to execute these main workloads in our scripts from the command line.
+
+
Config
+
In addition to our core workload scripts, recall that we also have a config.py script. This file will include all of the setup and configuration that all/most of our workloads depend on. For example, setting up our model registry:
We wouldn't have configurations like our ScalingConfig here because that's specific to our training workload. The config.py script is for configurations that are shared across different workloads.
+
+
Utilities
+
Similarly, we also have a utils.py script to include components that will be reused across different scripts. It's a good idea to organize these shared components here as opposed to the core scripts to avoid circular dependency conflicts (two scripts call on functions from each other). Here is an example of one of our utility functions, set_seeds, that's used in both our train.py and tune.py scripts.
Recall in our setup lesson that we initialized Ray inside our notebooks. We still need to initialize Ray before executing our ML workloads via scripts but we can decide to do this only for the scripts with Ray dependent workloads. For example, at the bottom of our train.py script, we have:
Now that we've set up our scripts, we can start executing them from the command line. In the next lesson, we'll learn how to do this with command-line interfaces (CLI).
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Scripting - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
In this lesson, we're going to serve the machine learning models that we have developed so that we can use them to make predictions on unseen data. And we want to be able to serve our models in a scalable and robust manner so it can deliver high throughput (handle many requests) and low latency (quickly respond to each request). In an effort to be comprehensive, we will implement both batch inference (offline) and online inference (real-time), though we will focus on the latter in the remaining lessons as it's more appropriate for our application.
+
Frameworks
+
There are many frameworks to choose from when it comes to model serving, such as Ray Serve, Nvidia Triton, HuggingFace, Bento ML, etc. When choosing between these frameworks, we want to choose the option that will allow us to:
+
+
Pythonic: we don't want to learn a new framework to be able to serve our models.
+
framework agnostic: we want to be able to serve models from all frameworks (PyTorch, TensorFlow, etc.)
+
scale: (auto)scaling our service should be as easy as changing a configuration.
+
composition: combine multiple models and business logic into our service.
+
integrations: integrate with popular API frameworks like FastAPI.
+
+
To address all of these requirements (and more), we will be using Ray Serve to create our service. While we'll be specifically using it's integration with FastAPI, there are many other integrations you might want to explore based on your stack (LangChain, Kubernetes, etc.).
+
Batch inference
+
We will first implement batch inference (or offline inference), which is when we make predictions on a large batch of data. This is useful when we don't need to serve a model's prediction on input data as soon as the input data is received. For example, our service can be used to make predictions once at the end of every day on the batches of content collected throughout the day. This can be more efficient than making predictions on each content individually if we don't need that kind of low latency.
+
Let's take a look at our how we can easily implement batch inference with Ray Serve. We'll start with some setup and load the best checkpoint from our training run.
Next, we'll define a Predictor class that will load the model from our checkpoint and then define the __call__ method that will be used to make predictions on our input data.
To do batch inference, we'll be using the map_batches functionality. We previously used map_batches to map (or apply) a preprocessing function across batches (chunks) of our data. We're now using the same concept to apply our predictor across batches of our inference data.
Note that best_checkpoint as a keyword argument to our Predictor class so that we can load the model from that checkpoint. We can pass this in via the fn_constructor_kwargs argument in our map_batches function.
While we can achieve batch inference at scale, many models will need to be served in an real-time manner where we may need to deliver predictions for many incoming requests (high throughput) with low latency. We want to use online inference for our application over batch inference because we want to quickly categorize content as they are received/submitted to our platform so that the community can discover them quickly.
We'll start by defining our FastAPI application which involves initializing a predictor (and preprocessor) from the best checkpoint for a particular run (specified by run_id). We'll also define a predict function that will be used to make predictions on our input data.
classModelDeployment:
+
+ def__init__(self,run_id):
+"""Initialize the model."""
+ self.run_id=run_id
+ mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)# so workers have access to model registry
+ best_checkpoint=get_best_checkpoint(run_id=run_id)
+ self.predictor=TorchPredictor.from_checkpoint(best_checkpoint)
+ self.preprocessor=self.predictor.get_preprocessor()
+
+ @app.post("/predict/")
+ asyncdef_predict(self,request:Request):
+ data=awaitrequest.json()
+ df=pd.DataFrame([{"title":data.get("title",""),"description":data.get("description",""),"tag":""}])
+ results=predict_with_proba(df=df,predictor=self.predictor)
+ return{"results":results}
+
+
+
async def refers to an asynchronous function (when we call the function we don't have to wait for the function to complete executing). The await keyword is used inside an asynchronous function to wait for the completion of the request.json() operation.
+
+
We can now combine our FastAPI application with Ray Serve by simply wrapping our application with the serve.ingress decorator. We can further wrap all of this with the serve.deployment decorator to define our deployment configuration (ex. number of replicas, compute resources, etc.). These configurations allow us to easily scale our service as needed.
Now let's run our service and perform some real-time inference.
+
1
+2
+3
+4
# Run service
+sorted_runs=mlflow.search_runs(experiment_names=[experiment_name],order_by=["metrics.val_loss ASC"])
+run_id=sorted_runs.iloc[0].run_id
+serve.run(ModelDeployment.bind(run_id=run_id))
+
+
+Started detached Serve instance in namespace "serve".
+Deployment 'default_ModelDeployment:IcuFap' is ready at `http://127.0.0.1:8000/`. component=serve deployment=default_ModelDeployment
+RayServeSyncHandle(deployment='default_ModelDeployment')
+
+
+
1
+2
+3
+4
+5
# Query
+title="Transfer learning with transformers"
+description="Using transformers for transfer learning on text classification tasks."
+json_data=json.dumps({"title":title,"description":description})
+requests.post("http://127.0.0.1:8000/predict/",data=json_data).json()
+
The issue with neural networks (and especially LLMs) is that they are notoriously overconfident. For every input, they will always make some prediction. And to account for this, we have an other class but that class only has projects that are not in our accepted tags but are still machine learning related nonetheless. Here's what happens when we input complete noise as our input:
+
1
+2
+3
+4
# Query (noise)
+title=" 65n7r5675"# random noise
+json_data=json.dumps({"title":title,"description":""})
+requests.post("http://127.0.0.1:8000/predict/",data=json_data).json()
+
Let's shutdown our service before we fixed this issue.
+
1
+2
# Shutdown
+serve.shutdown()
+
+
Custom logic
+
To make our service a bit more robust, let's add some custom logic to predict the other class if the probability of the predicted class is below a certain threshold probability.
It's easier to incorporate custom logic instead of altering the model itself. This way, we won't have to collect new data. change the model's architecture or retrain it. This also makes it really easy to change the custom logic as our product specifications may change (clean separation of product and machine learning).
+
+
1
+2
# Run service
+serve.run(ModelDeploymentRobust.bind(run_id=run_id,threshold=0.9))
+
+
+Started detached Serve instance in namespace "serve".
+Deployment 'default_ModelDeploymentRobust:RTbrNg' is ready at `http://127.0.0.1:8000/`. component=serve deployment=default_ModelDeploymentRobust
+RayServeSyncHandle(deployment='default_ModelDeploymentRobust')
+
+
+
Now let's see how we perform on the same random noise with our custom logic incorporate into the service.
+
1
+2
+3
+4
# Query (noise)
+title=" 65n7r5675"# random noise
+json_data=json.dumps({"title":title,"description":""})
+requests.post("http://127.0.0.1:8000/predict/",data=json_data).json()
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
In this lesson, we'll setup the development environment that we'll be using in all of our lessons. We'll have instructions for both local laptop and remote scalable clusters (Anyscale). While everything will work locally on your laptop, you can sign up to join one of our upcoming live cohorts where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day → sign up here.
+
Cluster
+
We'll start with defining our cluster, which refers to a group of servers that come together to form one system. Our clusters will have a head node that manages the cluster and it will be connected to a set of worker nodes that will execute workloads for us. These clusters can be fixed in size or autoscale based on our application's compute needs, which makes them highly scalable and performant. We'll create our cluster by defining a compute configuration and an environment.
+
Environment
+
We'll start by defining our cluster environment which will specify the software dependencies that we'll need for our workloads.
+
+
💻 Local
+
Your personal laptop will need to have Python installed and we highly recommend using Python 3.10. You can use a tool like pyenv (mac) or pyenv-win (windows) to easily download and switch between Python versions.
+
pyenvinstall3.10.11# install
+pyenvglobal3.10.11# set default
+
+
Once we have our Python version, we can create a virtual environment to install our dependencies. We'll download our Python dependencies after we clone our repository from git shortly.
Our cluster environment will be defined inside a cluster_env.yaml file. Here we specify some details around our base image (anyscale/ray:2.6.0-py310-cu118) that has our Python version, GPU dependencies, etc.
We could specify any python packages inside pip_packages or conda_packages but we're going to use a requirements.txt file to load our dependencies under post_build_cmds.
+
+
Compute
+
Next, we'll define our compute configuration, which will specify our hardware dependencies (head and worker nodes) that we'll need for our workloads.
+
+
💻 Local
+
Your personal laptop (single machine) will act as the cluster, where one CPU will be the head node and some of the remaining CPU will be the worker nodes (no GPUs required). All of the code in this course will work in any personal laptop though it will be slower than executing the same workloads on a larger cluster.
+
+
+
🚀 Anyscale
+
Our cluster compute will be defined inside a cluster_compute.yaml file. Here we specify some details around where our compute resources will come from (cloud computing platform like AWS), types of nodes and their counts, etc.
Our worker nodes will be GPU-enabled so we can train our models faster and we set min_workers to 0 so that we can autoscale these workers only when they're needed (up to a maximum of max_workers). This will help us significantly reduce our compute costs without having to manage the infrastructure ourselves.
+
+
Workspaces
+
With our compute and environment defined, we're ready to create our cluster workspace. This is where we'll be developing our ML application on top of our compute, environment and storage.
+
+
💻 Local
+
Your personal laptop will need to have an interactive development environment (IDE) installed, such as VS Code. For bash commands in this course, you're welcome to use the terminal on VSCode or a separate one.
+
+
+
🚀 Anyscale
+
We're going to launch an Anyscale Workspace to do all of our development in. Workspaces allow us to use development tools such as VSCode, Jupyter notebooks, web terminal, etc. on top of our cluster compute, environment and storage. This create an "infinite laptop" experience that feels like a local laptop experience but on a powerful, scalable cluster.
+
+
+
+
We have the option to create our Workspace using a CLI but we're going to create it using the web UI (you will receive the required credentials during the cohort). On the UI, we can fill in the following information:
+
-Workspace name: `madewithml`
+-Project: `madewithml`
+-Cluster environment name: `madewithml-cluster-env`
+# Toggle `Select from saved configurations`
+-Compute config: `madewithml-cluster-compute`
+-Click on the **Start** button to launch the Workspace
+
+
+
+
+
We have already saved created our Project, cluster environment and compute config so we can select them from the dropdowns but we could just as easily create new ones / update these using the CLI.
Toggle Add a README file (very important as this creates a main branch)
+
Scroll down and click Create repository
+
+
Now we're ready to clone the Made With ML repository's contents from GitHub inside our madewithml directory.
+
exportGITHUB_USERNAME="YOUR_GITHUB_UESRNAME"# <-- CHANGE THIS to your username
+gitclonehttps://github.com/GokuMohandas/Made-With-ML.git.
+gitremoteset-urloriginhttps://github.com/$GITHUB_USERNAME/Made-With-ML.git
+gitcheckout-bdev
+exportPYTHONPATH=$PYTHONPATH:$PWD# so we can import modules from our scripts
+
+
+
💻 Local
+
Recall that we created our virtual environment earlier but didn't actually load any Python dependencies yet. We'll clone our repository and then install the packages using the requirements.txt file.
+
python3-mpipinstall-rrequirements.txt
+
+
+
Caution: make sure that we're installing our Python packages inside our virtual environment.
+
+
+
+
🚀 Anyscale
+
Our environment with the appropriate Python version and libraries is already all set for us through the cluster environment we used when setting up our Anyscale Workspace. But if we want to install additional Python packages as we develop, we need to do pip install with the --user flag inside our Workspaces (via terminal) to ensure that our head and all worker nodes receive the package. And then we should also add it to our requirements file so it becomes part of the cluster environment build process next time.
+
pipinstall--user<package_name>:<version>
+
+
+
Notebook
+
Now we're ready to launch our Jupyter notebook to interactively develop our ML application.
+
+
💻 Local
+
We already installed jupyter through our requirements.txt file in the previous step, so we can just launch it.
+
jupyterlabnotebooks/madewithml.ipynb
+
+
+
+
🚀 Anyscale
+
Click on the Jupyter icon at the top right corner of our Anyscale Workspace page and this will open up our JupyterLab instance in a new tab. Then navigate to the notebooks directory and open up the madewithml.ipynb notebook.
+
+
+
+
+
Ray
+
We'll be using Ray to scale and productionize our ML application. Ray consists of a core distributed runtime along with libraries for scaling ML workloads and has companies like OpenAI, Spotify, Netflix, Instacart, Doordash + many more using it to develop their ML applications. We're going to start by initializing Ray inside our notebooks:
+
1
importray
+
+
1
+2
+3
+4
# Initialize Ray
+ifray.is_initialized():
+ ray.shutdown()
+ray.init()
+
+
We can also view our cluster resources to view the available compute resources:
+
1
ray.cluster_resources()
+
+
+
💻 Local
+
If you are running this on a local laptop (no GPU), use the CPU count from ray.cluster_resources() to set your resources. For example if your machine has 10 CPUs:
These cluster resources only reflect our head node (1 m5.2xlarge). But recall earlier in our compute configuration that we also added g4dn.xlarge worker nodes (each has 1 GPU and 4 CPU) to our cluster. But because we set min_workers=0, our worker nodes will autoscale ( up to max_workers) as they're needed for specific workloads (ex. training). So we can set the # of workers and resources by worker based on this insight:
Head on over to the next lesson, where we'll motivate the specific application that we're trying to build from a product and systems design perspective. And after that, we're ready to start developing!
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Setup - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
To determine the efficacy of our models, we need to have an unbiased measuring approach. To do this, we split our dataset into training, validation, and testing data splits.
+
+
Use the training split to train the model.
+
Here the model will have access to both inputs and outputs to optimize its internal weights.
+
+
+
After each loop (epoch) of the training split, we will use the validation split to determine model performance.
+
Here the model will not use the outputs to optimize its weights but instead, we will use the performance to optimize training hyperparameters such as the learning rate, etc.
+
+
+
After training stops (epoch(s)), we will use the testing split to perform a one-time assessment of the model.
+
This is our best measure of how the model may behave on new, unseen data. Note that training stops when the performance improvement is not significant or any other stopping criteria that we may have specified.
+
+
+
+
+
Creating proper data splits
+
What are the criteria we should focus on to ensure proper data splits?
+
+Show answer
+
+
the dataset (and each data split) should be representative of data we will encounter
+
equal distributions of output values across all splits
+
shuffle your data if it's organized in a way that prevents input variance
+
avoid random shuffles if your task can suffer from data leaks (ex. time-series)
+
+
+
+
+
We need to clean our data first before splitting, at least for the features that splitting depends on. So the process is more like: preprocessing (global, cleaning) → splitting → preprocessing (local, transformations).
+
+
Naive split
+
We'll start by splitting our dataset into three data splits for training, validation and testing.
For our multi-class task (each input has one label), we want to ensure that each data split has similar class distributions. We can achieve this by specifying how to stratify the split by adding the stratify keyword argument.
# Get counts for each class
+counts={}
+counts["train_counts"]={tag:label_encoder.decode(y_train).count(tag)fortaginlabel_encoder.classes}
+counts["val_counts"]={tag:label_encoder.decode(y_val).count(tag)fortaginlabel_encoder.classes}
+counts["test_counts"]={tag:label_encoder.decode(y_test).count(tag)fortaginlabel_encoder.classes}
+
It's hard to compare these because our train and test proportions are different. Let's see what the distribution looks like once we balance it out. What do we need to multiply our test ratio by so that we have the same amount as our train ratio?
We can see how much deviance there is in our naive data splits by computing the standard deviation of each split's class counts from the mean (ideal split).
+
\[ \sigma = \sqrt{\frac{(x - \bar{x})^2}{N}} \]
+
1
+2
# Standard deviation
+np.mean(np.std(dist_df.to_numpy(),axis=0))
+
big bad nlp database collection 400 nlp datasets...
+
natural-language-processing
+
+
+
3
+
job classification job classification done usi...
+
natural-language-processing
+
+
+
4
+
optimizing mobiledet mobile deployments learn ...
+
computer-vision
+
+
+
+
+
+
+
Multi-label classification
+
If we had a multi-label classification task, then we would've applied iterative stratification via the skmultilearn library, which essentially splits each input into subsets (where each label is considered individually) and then it distributes the samples starting with fewest "positive" samples and working up to the inputs that have the most labels.
+
fromskmultilearn.model_selectionimportIterativeStratification
+defiterative_train_test_split(X,y,train_size):
+"""Custom iterative train test split which
+ 'maintains balanced representation with respect
+ to order-th label combinations.'
+ """
+ stratifier=IterativeStratification(
+ n_splits=2,order=1,sample_distribution_per_fold=[1.0-train_size,train_size,])
+ train_indices,test_indices=next(stratifier.split(X,y))
+ X_train,y_train=X[train_indices],y[train_indices]
+ X_test,y_test=X[test_indices],y[test_indices]
+ returnX_train,X_test,y_train,y_test
+
+
Iterative stratification essentially creates splits while "trying to maintain balanced representation with respect to order-th label combinations". We used to an order=1 for our iterative split which means we cared about providing representative distribution of each tag across the splits. But we can account for higher-order label relationships as well where we may care about the distribution of label combinations.
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Splitting a Dataset for Machine Learning - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
+
Code is read more often than it is written. -- Guido Van Rossum (author of Python)
+
+
When we write a piece of code, it's almost never the last time we see it or the last time it's edited. So we need to explain what's going on (via documentation) and make it easy to read. One of the easiest ways to make code more readable is to follow consistent style and formatting conventions. There are many options when it comes to Python style conventions to adhere to, but most are based on PEP8 conventions. Different teams follow different conventions and that's perfectly alright. The most important aspects are:
+
+
consistency: everyone follows the same standards.
+
automation: formatting should be largely effortless after initial configuration.
+
+
Tools
+
We will be using a very popular blend of style and formatting conventions that makes some very opinionated decisions on our behalf (with configurable options).
+
+
Black: an in-place reformatter that (mostly) adheres to PEP8.
+
isort: sorts and formats import statements inside Python scripts.
+
flake8: a code linter with stylistic conventions that adhere to PEP8.
+
+
Configuration
+
Before we can properly use these tools, we'll have to configure them because they may have some discrepancies amongst them since they follow slightly different conventions that extend from PEP8.
+
Black
+
To configure Black, we could just pass in options using the CLI method, but it's much cleaner to do this through our pyproject.toml file.
# Black formatting
+[tool.black]
+line-length=150
+include='\.pyi?$'
+exclude='''
+/(
+ .eggs # exclude a few common directories in the
+ | .git # root of the project
+ | .hg
+ | .mypy_cache
+ | .tox
+ | venv
+ | _build
+ | buck-out
+ | build
+ | dist
+ )/
+'''
+
+
Here we're telling Black what our maximum line length should and to include and exclude certain file extensions.
+
+
The pyproject.toml was created to establish a more human-readable configuration file that is meant to replace a setup.py or setup.cfg file and is increasingly adopted by many open-source libraries.
+
+
isort
+
Next, we're going to configure isort in our pyproject.toml file (just below Black's configurations):
Though there is a complete list of configuration options for isort, we've decided to set these explicitly so there are no conflicts with Black.
+
flake8
+
Lastly, we'll set up flake8 by also adding it's configuration details to out pyproject.toml file.
+
1
+2
+3
+4
+5
+6
[tool.flake8]
+exclude="venv"
+ignore=["E501","W503","E226"]
+# E501: Line too long
+# W503: Line break occurred before binary operator
+# E226: Missing white space around arithmetic operator
+
+
Here we're including an ignore option to ignore certain flake8 rules so everything works with our Black and isort configurations. And besides defining configuration options here, which are applied globally, we can also choose to specifically ignore certain conventions on a line-by-line basis. Here is an example of how we utilize this:
+
1
+2
# madewithml/config.py
+importpretty_errors# NOQA: F401 (imported but unused)
+
+
By placing the # NOQA: <error-code> on a line, we're telling flake8 to do NOQuality Assurance for that particular error on this line.
+
Usage
+
To use these tools that we've configured, we have to execute them from the project directory:
+
Take a look at your files to see all the changes that have been made!
+
+
the . signifies that the configuration file for that package is in the current directory
+
+
Makefile
+
Remembering these three lines to style our code is a bit cumbersome so it's a good idea to create a Makefile. This file can be used to define a set of commands that can be executed with a single command. Here's what our Makefile looks like:
Notice that the clean command depends on the style command (clean:style), which means that style will be executed first before clean is executed.
+
+
+
.PHONY
+
As the name suggests, a Makefile is typically used to make a file, where if a file with the name already exists, then the commands below won't be executed. But we're using it in a way where we want to execute some commands with a single alias. Therefore, the .PHONY:$FILENAME lines indicate that even if there is a file called $FILENAME, go ahead and execute the commands below anyway.
+
+
In the next lesson on pre-commit we'll learn how to automatically execute this formatting whenever we make changes to our code.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Styling - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
In the previous lesson, we covered the product design process for our ML application. In this lesson, we'll cover the systems design process where we'll learn how to design the ML system that will address our product objectives.
+
Template
+
The template below is designed to guide machine learning product development. It involves both the product and systems design aspects of our application:
was there sampling of any kind applied to create this dataset?
+
are we introducing any data leaks?
+
+
+
production:
+
access to batches or real-time streams of ML content from various sources
+
how can we trust that this stream only has data that is consistent with what we have historically seen?
+
+
+
+
+
+
+
Assumption
+
Reality
+
Reason
+
+
+
+
+
All of our incoming data is only machine learning related (no spam).
+
We would need a filter to remove spam content that's not ML related.
+
To simplify our ML task, we will assume all the data is ML content.
+
+
+
+
+
Labeling
+
Describe the labeling process (ingestions, QA, etc.) and how we decided on the features and labels.
+
+
+
+
+
+
Our task
+
Labels: categories of machine learning (for simplification, we've restricted the label space to the following tags: natural-language-processing, computer-vision, mlops and other).
+
Features: text features (title and description) that describe the content.
+
+
+
+
Assumption
+
Reality
+
Reason
+
+
+
+
+
Content can only belong to one category (multiclass).
+
Content can belong to more than one category (multilabel).
+
For simplicity and many libraries don't support or complicate multilabel scenarios.
+
+
+
+
+
Metrics
+
One of the hardest challenges with ML systems is tying our core objectives, many of which may be qualitative, with quantitative metrics that our model can optimize towards.
+
+
Our task
+
For our task, we want to have both high precision and recall, so we'll optimize for f1 score (weighted combination of precision and recall). We'll determine these metrics for the overall dataset, as well as specific classes or slices of data.
+
+
True positives (TP): we correctly predicted class X.
+
False positives (FP): we incorrectly predicted class X but it was another class.
+
True negatives (TN): we correctly predicted that it's wasn't the class X.
+
False negatives (FN): we incorrectly predicted that it wasn't the class X but it was.
It entirely depends on the specific task. For example, in an email spam detector, precision is very important because it's better than we some spam then completely miss an important email. Overtime, we need to iterate on our solution so all evaluation metrics improve but it's important to know which one's we can't comprise on from the get-go.
+
+
+
Evaluation
+
Once we have our metrics defined, we need to think about when and how we'll evaluate our model.
+
Offline evaluation
+
Offline evaluation requires a gold standard holdout dataset that we can use to benchmark all of our models.
+
+
Our task
+
We'll be using this holdout dataset for offline evaluation. We'll also be creating slices of data that we want to evaluate in isolation.
+
+
Online evaluation
+
Online evaluation ensures that our model continues to perform well in production and can be performed using labels or, in the event we don't readily have labels, proxy signals.
+
+
Our task
+
+
manually label a subset of incoming data to evaluate periodically.
+
asking the initial set of users viewing a newly categorized content if it's correctly classified.
+
allow users to report misclassified content by our model.
+
+
+
It's important that we measure real-time performance before committing to replace our existing version of the system.
+
+
Internal canary rollout, monitoring for proxy/actual performance, etc.
+
Rollout to the larger internal team for more feedback.
+
A/B rollout to a subset of the population to better understand UX, utility, etc.
+
+
+
Not all releases have to be high stakes and external facing. We can always include internal releases, gather feedback and iterate until we’re ready to increase the scope.
+
+
Modeling
+
While the specific methodology we employ can differ based on the problem, there are core principles we always want to follow:
+
+
End-to-end utility: the end result from every iteration should deliver minimum end-to-end utility so that we can benchmark iterations against each other and plug-and-play with the system.
+
Manual before ML: try to see how well a simple rule-based system performs before moving onto more complex ones.
+
Augment vs. automate: allow the system to supplement the decision making process as opposed to making the actual decision.
+
Internal vs. external: not all early releases have to be end-user facing. We can use early versions for internal validation, feedback, data collection, etc.
+
Thorough: every approach needs to be well tested (code, data + models) and evaluated, so we can objectively benchmark different approaches.
+
+
+
Our task
+
+
creating a gold-standard labeled dataset that is representative of the problem space.
+
rule-based text matching approaches to categorize content.
+
predict labels (probabilistic) from content title and description.
+
+
+
+
+
Assumption
+
Reality
+
Reason
+
+
+
+
+
Solution needs to involve ML due to unstructured data and ineffectiveness of rule-based systems for this task.
+
An iterative approach where we start with simple rule-based solutions and slowly add complexity.
+
This course is about responsibly delivering value with ML, so we'll jump to it right away.
+
+
+
+
+
+
Utility in starting simple
+
Some of the earlier, simpler, approaches may not deliver on a certain performance objective. What are some advantages of still starting simple?
+
+Show answer
+
+
get internal feedback on end-to-end utility.
+
perform A/B testing to understand UI/UX design.
+
deployed locally to start generating more data required for more complex approaches.
+
+
+
+
Inference
+
Once we have a model we're satisfied with, we need to think about whether we want to perform batch (offline) or real-time (online) inference.
+
Batch inference
+
We can use our models to make batch predictions on a finite set of inputs which are then written to a database for low latency inference. When a user or downstream service makes an inference request, cached results from the database are returned. In this scenario, our trained model can directly be loaded and used for inference in the code. It doesn't have to be served as a separate service.
+
+
+
+
+
+
✅ generate and cache predictions for very fast inference for users.
+
✅ the model doesn't need to be spun up as it's own service since it's never used in real-time.
+
❌ predictions can become stale if user develops new interests that aren’t captured by the old data that the current predictions are based on.
+
+
+
Batch serving tasks
+
What are some tasks where batch serving is ideal?
+
+Show answer
+
Recommend content that existing users will like based on their viewing history. However, new users may just receive some generic recommendations based on their explicit interests until we process their history the next day. And even if we're not doing batch serving, it might still be useful to cache very popular sets of input features (ex. combination of explicit interests leads to certain recommended content) so that we can serve those predictions faster.
+
+
+
Online inference
+
We can also serve real-time predictions where input features are fed to the model to retrieve predictions. In this scenario, our model will need to be served as a separate service (ex. api endpoint) that can handle incoming requests.
+
+
+
+
+
+
✅ can yield more up-to-date predictions which may yield a more meaningful user experience, etc.
+
❌ requires managed microservices to handle request traffic.
+
❌ requires real-time monitoring since input space in unbounded, which could yield erroneous predictions.
+
+
+
Online inference tasks
+
In our example task for batch inference above, how can online inference significantly improve content recommendations?
+
+Show answer
+
With batch processing, we generate content recommendations for users offline using their history. These recommendations won't change until we process the batch the next day using the updated user features. But what is the user's taste significantly changes during the day (ex. user is searching for horror movies to watch). With real-time serving, we can use these recent features to recommend highly relevant content based on the immediate searches.
+
+
+
+
Our task
+
For our task, we'll be serving our model as a separate service to handle real-time requests. We want to be able to perform online inference so that we can quickly categorize ML content as they become available. However, we will also demonstrate how to do batch inference for the sake of completeness.
+
+
Feedback
+
How do we receive feedback on our system and incorporate it into the next iteration? This can involve both human-in-the-loop feedback as well as automatic feedback via monitoring, etc.
+
+
Our task
+
+
enforce human-in-loop checks when there is low confidence in classifications.
+
allow users to report issues related to misclassification.
+
+
+
+
Always return to the value proposition
+
While it's important to iterate and optimize on our models, it's even more important to ensure that our ML systems are actually making an impact. We need to constantly engage with our users to iterate on why our ML system exists and how it can be made better.
+
+
+
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Systems - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
In this lesson, we'll learn how to test code, data and machine learning models to construct a machine learning system that we can reliably iterate on. Tests are a way for us to ensure that something works as intended. We're incentivized to implement tests and discover sources of error as early in the development cycle as possible so that we can decrease downstream costs and wasted time. Once we've designed our tests, we can automatically execute them every time we change or add to our codebase.
+
+
Tip
+
We highly recommend that you explore this lesson after completing the previous lessons since the topics (and code) are iteratively developed. We did, however, create the testing-ml repository for a quick overview with an interactive notebook.
+
+
Types of tests
+
There are four majors types of tests which are utilized at different points in the development cycle:
+
+
Unittests: tests on individual components that each have a single responsibility (ex. function that filters a list).
+
Integrationtests: tests on the combined functionality of individual components (ex. data processing).
+
Systemtests: tests on the design of a system for expected outputs given inputs (ex. training, inference, etc.).
+
Acceptancetests: tests to verify that requirements have been met, usually referred to as User Acceptance Testing (UAT).
+
Regressiontests: tests based on errors we've seen before to ensure new changes don't reintroduce them.
+
+
While ML systems are probabilistic in nature, they are composed of many deterministic components that can be tested in a similar manner as traditional software systems. The distinction between testing ML systems begins when we move from testing code to testing the data and models.
+
+
+
+
+
+
There are many other types of functional and non-functional tests as well, such as smoke tests (quick health checks), performance tests (load, stress), security tests, etc. but we can generalize all of these under the system tests above.
+
+
How should we test?
+
The framework to use when composing tests is the Arrange Act Assert methodology.
+
+
Arrange: set up the different inputs to test on.
+
Act: apply the inputs on the component we want to test.
+
Assert: confirm that we received the expected output.
+
+
+
Cleaning is an unofficial fourth step to this methodology because it's important to not leave remnants of a previous test which may affect subsequent tests. We can use packages such as pytest-randomly to test against state dependency by executing tests randomly.
+
+
In Python, there are many tools, such as unittest, pytest, etc. that allow us to easily implement our tests while adhering to the Arrange Act Assert framework. These tools come with powerful built-in functionality such as parametrization, filters, and more, to test many conditions at scale.
+
What should we test?
+
When arranging our inputs and asserting our expected outputs, what are some aspects of our inputs and outputs that we should be testing for?
+
+
inputs: data types, format, length, edge cases (min/max, small/large, etc.)
+
outputs: data types, formats, exceptions, intermediary and final outputs
+
+
+
👉 We'll cover specific details pertaining to what to test for regarding our data and models below.
+
+
Best practices
+
Regardless of the framework we use, it's important to strongly tie testing into the development process.
+
+
atomic: when creating functions and classes, we need to ensure that they have a single responsibility so that we can easily test them. If not, we'll need to split them into more granular components.
+
compose: when we create new components, we want to compose tests to validate their functionality. It's a great way to ensure reliability and catch errors early on.
+
reuse: we should maintain central repositories where core functionality is tested at the source and reused across many projects. This significantly reduces testing efforts for each new project's code base.
+
regression: we want to account for new errors we come across with a regression test so we can ensure we don't reintroduce the same errors in the future.
+
coverage: we want to ensure 100% coverage for our codebase. This doesn't mean writing a test for every single line of code but rather accounting for every single line.
+
automate: in the event we forget to run our tests before committing to a repository, we want to auto run tests when we make changes to our codebase. We'll learn how to do this locally using pre-commit hooks and remotely via GitHub actions in subsequent lessons.
Note that we aren't testing evaluate.py and serve.py because it involves complicated testing that's based on the data and models. We'll be testing these components as part of our integration tests when we test our system end-to-end.
+
+
💻 Code
+
We'll start by testing our code and we'll use pytest as our testing framework for it's powerful builtin features such as parametrization, fixtures, markers and more.
+
Configuration
+
Pytest expects tests to be organized under a tests directory by default. However, we can also add to our existing pyproject.toml file to configure any other test directories as well. Once in the directory, pytest looks for python scripts starting with tests_*.py but we can configure it to read any other file patterns as well.
Let's see what a sample test and it's results look like. Assume we have a simple function that decodes a list of indices into their respective classes using a dictionary mapping.
We can also have assertions about exceptions like we do in lines 6-8 where all the operations under the with statement are expected to raise the specified exception.
+
+
Execution
+
We can execute our tests above using several different levels of granularity:
+
python3-mpytest# all tests
+python3-mpytesttests/code# tests under a directory
+python3-mpytesttests/code/test_predict.py# tests for a single file
+python3-mpytesttests/code/test_predict.py::test_decode# tests for a single function
+
+
Running our specific test above would produce the following output:
+
It's important to test for the variety of inputs and expected outputs that we outlined above and to never assume that a test is trivial. In our example above, it's important that we test for both "apple" and "Apple" in the event that our function didn't account for casing!
+
+
Classes
+
We can also test classes and their respective functions.
There are also more xunit-style testing options available as well for more involved testing with classes.
+
+
Parametrize
+
So far, in our tests, we've had to create individual assert statements to validate different combinations of inputs and expected outputs. However, there's a bit of redundancy here because the inputs always feed into our functions as arguments and the outputs are compared with our expected outputs. To remove this redundancy, pytest has the @pytest.mark.parametrize decorator which allows us to represent our inputs and outputs as parameters.
Parametrization allows us to reduce redundancy inside test functions but what about reducing redundancy across different test functions? For example, suppose that different test functions all have a common component (ex. preprocessor). Here, we can use pytest's builtin fixture, which is a function that is executed before the test function. Let's rewrite our test_fit_transform function from above using a fixture:
All of our test scripts know to look inside a conftest.py script in the same directory for any fixtures. Note that the name of the fixture and the input argument to our function have to be the same.
+
+
Fixture scopes
+
Fixtures can have different scopes depending on how we want to use them. For example our df fixture has the module scope because we don't want to keep recreating it after every test but, instead, we want to create it just once for all the tests in our module (tests/test_data.py).
+
+
function: fixture is destroyed after every test. [default]
+
class: fixture is destroyed after the last test in the class.
+
module: fixture is destroyed after the last test in the module (script).
+
package: fixture is destroyed after the last test in the package.
+
session: fixture is destroyed after the last test of the session.
+
+
+
Markers
+
We've been able to execute our tests at various levels of granularity (all tests, script, function, etc.) but we can create custom granularity by using markers. We've already used one type of marker (parametrize) but there are several other builtin markers as well. For example, the skipif marker allows us to skip execution of a test if a condition is met. For example, supposed we only wanted to test training our model if a GPU is available:
+
1
+2
+3
+4
+5
+6
@pytest.mark.skipif(
+ nottorch.cuda.is_available(),
+ reason="Full training tests require a GPU."
+)
+deftest_training():
+ pass
+
+
We can also create our own custom markers with the exception of a few reserved marker names.
We can execute them by using the -m flag which requires a (case-sensitive) marker expression like below:
+
pytest-m"training"# runs all tests marked with `training`
+pytest-m"not training"# runs all tests besides those marked with `training`
+
+
+
Tip
+
The proper way to use markers is to explicitly list the ones we've created in our pyproject.toml file. Here we can specify that all markers must be defined in this file with the --strict-markers flag and then declare our markers (with some info about them) in our markers list:
+Once we do this, we can view all of our existing list of markers by executing pytest--markers and we'll receive an error when we're trying to use a new marker that's not defined here.
+
+
Coverage
+
As we're developing tests for our application's components, it's important to know how well we're covering our code base and to know if we've missed anything. We can use the Coverage library to track and visualize how much of our codebase our tests account for. With pytest, it's even easier to use this package thanks to the pytest-cov plugin.
Here we're asking to run all tests under tests/code and to check for coverage for all the code in our madewithml directory. When we run this, we'll see the tests from our tests directory executing while the coverage plugin is keeping tracking of which lines in our application are being executed. Once our tests are done, we can view the generated report either through the terminal:
but a more interactive way is to view it through the htmlcov/index.html on a browser. Here we can click on individual files to see which parts were not covered by any tests.
+
+
+
+
+
+
Warning
+
Though we have 100% coverage, this does not mean that our application is perfect. Coverage only indicates that a piece of code executed in a test, not necessarily that every part of it was tested, let alone thoroughly tested. Therefore, coverage should never be used as a representation of correctness. However, it is very useful to maintain coverage at 100% so we can know when new functionality has yet to be tested. In our CI/CD lesson, we'll see how to use GitHub actions to make 100% coverage a requirement when pushing to specific branches.
+
+
Exclusions
+
Sometimes it doesn't make sense to write tests to cover every single line in our application yet we still want to account for these lines so we can maintain 100% coverage. We have two levels of purview when applying exclusions:
+
+
+
Excusing lines by adding this comment # pragma: no cover, <MESSAGE>
+
1
+2
ifresults_fp:# pragma: no cover, saving results
+ utils.save_dict(d,results_fp)
+
+
+
+
Excluding files by specifying them in our pyproject.toml configuration:
The main point is that we were able to add justification to these exclusions through comments so our team can follow our reasoning.
+
+
Now that we have a foundation for testing traditional software, let's dive into testing our data and models in the context of machine learning systems.
+
🔢 Data
+
So far, we've used unit and integration tests to test the functions that interact with our data but we haven't tested the validity of the data itself. We're going to use the great expectations library to test what our data is expected to look like. It's a library that allows us to create expectations as to what our data should look like in a standardized way. It also provides modules to seamlessly connect with backend data sources such as local file systems, S3, databases, etc. Let's explore the library by implementing the expectations we'll need for our application.
+
+
👉 Follow along interactive notebook in the testing-ml repository as we implement the concepts below.
+
+
First we'll load the data we'd like to apply our expectations on. We can load our data from a variety of sources (filesystem, database, cloud etc.) which we can then wrap around a Dataset module (Pandas / Spark DataFrame, SQLAlchemy). Since multiple data tests may want access to this data, we'll create a fixture for it.
When it comes to creating expectations as to what our data should look like, we want to think about our entire dataset and all the features (columns) within it.
Each of these expectations will create an output with details about success or failure, expected and observed values, expectations raised, etc. For example, the expectation df.expect_column_values_to_be_of_type(column="title",type_="str") would produce the following if successful:
and if we have a failed expectation (ex. df.expect_column_values_to_be_of_type(column="title",type_="int")), we'd receive this output(notice the counts and examples for what caused the failure):
+
{
+"success":false,
+"exception_info":{
+"raised_exception":false,
+"exception_traceback":null,
+"exception_message":null
+},
+"expectation_config":{
+"meta":{},
+"kwargs":{
+"column":"title",
+"type_":"int",
+"result_format":"BASIC"
+},
+"expectation_type":"_expect_column_values_to_be_of_type__map"
+},
+"result":{
+"element_count":955,
+"missing_count":0,
+"missing_percent":0.0,
+"unexpected_count":955,
+"unexpected_percent":100.0,
+"unexpected_percent_nonmissing":100.0,
+"partial_unexpected_list":[
+"How to Deal with Files in Google Colab: What You Need to Know",
+"Machine Learning Methods Explained (+ Examples)",
+"OpenMMLab Computer Vision",
+"...",
+]
+},
+"meta":{}
+}
+
+
There are just a few of the different expectations that we can create. Be sure to explore all the expectations, including custom expectations. Here are some other popular expectations that don't pertain to our specific dataset but are widely applicable:
+
+
feature value relationships with other feature values → expect_column_pair_values_a_to_be_greater_than_b
We've added a --dataset-loc flag to pytest by specifying in our tests/data/conftest.py script. This allows us to pass in the dataset location as an argument to our tests.
We're keeping things simple by using our expectations with pytest but Great expectations also has a lot more functionality around connecting to data sources, Checkpoints to execute suites across various parts of the pipeline, data docs to generate reports, etc.
+
+
Production
+
While we're validating our datasets inside our machine learning applications, in most production scenarios, the data validation happens much further upstream. Our dataset may not be used just for our specific application and may actually be feeding into many other downstream application (ML and otherwise). Therefore, it's a great idea to execute these data validation tests as up stream as possible so that downstream applications can reliably use the data.
+
+
+
+
+
+
Learn more about different data systems in our data engineering lesson if you're not familiar with them.
+
+
🤖 Models
+
The final aspect of testing ML systems involves how to test machine learning models during training, evaluation, inference and deployment.
+
Training
+
We want to write tests iteratively while we're developing our training pipelines so we can catch errors quickly. This is especially important because, unlike traditional software, ML systems can run to completion without throwing any exceptions / errors but can produce incorrect systems. We also want to catch errors quickly to save on time and compute.
Behavioral testing is the process of testing input data and expected outputs while treating the model as a black box (model agnostic evaluation). They don't necessarily have to be adversarial in nature but more along the types of perturbations we may expect to see in the real world once our model is deployed. A landmark paper on this topic is Beyond Accuracy: Behavioral Testing of NLP Models with CheckList which breaks down behavioral testing into three types of tests:
+
+
invariance: Changes should not affect outputs.
+
1
+2
+3
# INVariance via verb injection (changes should not affect outputs)
+get_label(text="Transformers applied to NLP have revolutionized machine learning.",predictor=predictor)
+get_label(text="Transformers applied to NLP have disrupted machine learning.",predictor=predictor)
+
# DIRectional expectations (changes with known outputs)
+get_label(text="ML applied to text classification.",predictor=predictor)
+get_label(text="ML applied to image classification.",predictor=predictor)
+get_label(text="CNNs for text classification.",predictor=predictor)
+
minimumfunctionality: Simple combination of inputs and expected outputs.
+
1
+2
+3
+4
# Minimum Functionality Tests (simple input/output pairs)
+get_label(text="Natural language processing is the next big wave in machine learning.",predictor=predictor)
+get_label(text="MLOps is the next big wave in machine learning.",predictor=predictor)
+get_label(text="This is about graph neural networks.",predictor=predictor)
+
And we can convert these tests into proper parameterized tests by first defining from fixtures in our tests/model/conftest.py and our tests/model/utils.py scripts:
# tests/model/test_behavioral.py
+@pytest.mark.parametrize(
+ "input, label",
+ [
+ (
+ "Natural language processing is the next big wave in machine learning.",
+ "natural-language-processing",
+ ),
+ (
+ "MLOps is the next big wave in machine learning.",
+ "mlops",
+ ),
+ (
+ "This is about graph neural networks.",
+ "other",
+ ),
+ ],
+)
+deftest_mft(input,label,predictor):
+"""Minimum Functionality Tests (simple input/output pairs)."""
+ prediction=utils.get_label(text=input,predictor=predictor)
+ assertlabel==prediction
+
+
And we can execute them just like any other test:
+
# Model tests
+exportEXPERIMENT_NAME="llm"
+exportRUN_ID=$(pythonmadewithml/predict.pyget-best-run-id--experiment-name$EXPERIMENT_NAME--metricval_loss--modeASC)
+pytest--run-id=$RUN_IDtests/model--verbose--disable-warnings
+
+
Testing vs. monitoring
+
We'll conclude by talking about the similarities and distinctions between testing and monitoring. They're both integral parts of the ML development pipeline and depend on each other for iteration. Testing is assuring that our system (code, data and models) passes the expectations that we've established offline. Whereas, monitoring involves that these expectations continue to pass online on live production data while also ensuring that their data distributions are comparable to the reference window (typically subset of training data) through \(t_n\). When these conditions no longer hold true, we need to inspect more closely (retraining may not always fix our root problem).
+
With monitoring, there are quite a few distinct concerns that we didn't have to consider during testing since it involves (live) data we have yet to see.
+
+
features and prediction distributions (drift), typing, schema mismatches, etc.
+
determining model performance (rolling and window metrics on overall and slices of data) using indirect signals (since labels may not be readily available).
+
in situations with large data, we need to know which data points to label and upsample for training.
+
identifying anomalies and outliers.
+
+
+
We'll cover all of these concepts in much more depth (and code) in our monitoring lesson.
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Now that we have our data prepared, we can start training our models to optimize on our objective. Ideally, we would start with the simplest possible baseline and slowly add complexity to our models:
+
+
Start with a random (chance) model.
+
Since we have four classes, we may expect a random model to be correct around 25% of the time but recall that not all of our classes have equal counts.
+
+
+
Develop a rule-based approach using if-else statements, regular expressions, etc.
+
We could build a list of common words for each class and if a word in the input matches a word in the list, we can predict that class.
+
+
+
Slowly add complexity by addressing limitations and motivating representations and model architectures.
+
We could start with a simple term frequency (TF-IDF) mode and then move onto embeddings with CNNs, RNNs, Transformers, etc.
+
+
+
Weigh tradeoffs (performance, latency, size, etc.) between performant baselines.
+
Revisit and iterate on baselines as your dataset grows and new model architectures are developed.
+
+
We're going to skip straight to step 3 of developing a complex model because this task involves unstructured data and rule-based systems are not well suited for this. And with the increase adoption of large language models (LLMs) as a proven model architecture for NLP tasks, we'll fine-tune a pretrained LLM on our dataset.
+
+
Iterate on the data
+
Instead of using a fixed dataset and iterating on the models, we could keep the model constant and iterate on the dataset. This is useful to improve the quality of our datasets.
+
+
remove or fix data samples (false positives & negatives)
+
prepare and transform features
+
expand or consolidate classes
+
incorporate auxiliary datasets
+
identify unique slices to boost
+
+
+
Distributed training
+
With the rapid increase in data (unstructured) and model sizes (ex. LLMs), it's becoming increasingly difficult to train models on a single machine. We need to be able to distribute our training across multiple machines in order to train our models in a reasonable amount of time. And we want to be able to do this without having to:
+
+
set up a cluster by individually (and painstakingly) provisioning compute resources (CPU, GPU, etc.)
+
writing complex code to distribute our training across multiple machines
+
worry about communication and resource utilization between our different distributed compute resources
+
worry about fault tolerance and recovery from our large training workloads
+
+
To address all of these concerns, we'll be using Ray Train here in order to create a training workflow that can scale across multiple machines. While there are many options to choose from for distributed training, such as Pytorch Distributed Data Parallel (DDP), Horovod, etc., none of them allow us to scale across different machines with ease and do so with minimal changes to our single-machine training code as Ray does.
+
+
Primer on distributed training
+
With distributed training, there will be a head node that's responsible for orchestrating the training process. While the worker nodes that will be responsible for training the model and communicating results back to the head node. From a user's perspective, Ray abstracts away all of this complexity and we can simply define our training functionality with minimal changes to our code (as if we were training on a single machine).
+
+
Generative AI
+
In this lesson, we're going to be fine-tuning a pretrained large language model (LLM) using our labeled dataset. The specific class of LLMs we'll be using is called BERT. Bert models are encoder-only models and are the gold-standard for supervised NLP tasks. However, you may be wondering how do all the (much larger) LLM, created for generative applications, fare (GPT 4, Falcon 40B, Llama 2, etc.)?
+
+
We chose the smaller BERT model for our course because it's easier to train and fine-tune. However, the workflow for fine-tuning the larger LLMs are quite similar as well. They do require much more compute but Ray abstracts away the scaling complexities involved with that.
+
+
+
Note
+
All the code for this section can be found in our separate benchmarks.ipynb notebook.
+
+
Set up
+
You'll need to first sign up for an OpenAI account and then grab your API key from here.
+
1
+2
importopenai
+openai.api_key="YOUR_API_KEY"
+
+
Load data
+
We'll first load the our training and inference data into dataframes.
+
1
importpandasaspd
+
+
1
+2
+3
+4
# Load training data
+DATASET_LOC="https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/dataset.csv"
+train_df=pd.read_csv(DATASET_LOC)
+train_df.head()
+
We'll define a few utility functions to make the OpenAI request and to store our predictions. While we could perform batch prediction by loading samples until the context length is reached, we'll just perform one at a time since it's not too many data points and we can have fully deterministic behavior (if you insert new data, etc.). We'll also added some reliability in case we overload the endpoints with too many request at once.
We'll first define what a sample call to the OpenAI endpoint looks like. We'll pass in:
+- system_content that has information about how the LLM should behave.
+- assistant_content for any additional context it should have for answering our questions.
+- user_content that has our message or query to the LLM.
+- model should specify which specific model we want to send our request to.
+
We can pass all of this information in through the openai.ChatCompletion.create function to receive our response.
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
+12
+13
# Query OpenAI endpoint
+system_content="you only answer in rhymes"# system content (behavior)
+assistant_content=""# assistant content (context)
+user_content="how are you"# user content (message)
+response=openai.ChatCompletion.create(
+ model="gpt-3.5-turbo-0613",
+ messages=[
+ {"role":"system","content":system_content},
+ {"role":"assistant","content":assistant_content},
+ {"role":"user","content":user_content},
+ ],
+)
+print(response.to_dict()["choices"][0].to_dict()["message"]["content"])
+
+
+I'm doing just fine, so glad you ask,
+Rhyming away, up to the task.
+How about you, my dear friend?
+Tell me how your day did ascend.
+
+
+
Now, let's create a function that can predict tags for a given sample.
# Get tag
+model="gpt-3.5-turbo-0613"
+system_context=f"""
+ You are a NLP prediction service that predicts the label given an input's title and description.
+ You must choose between one of the following labels for each input: {tags}.
+ Only respond with the label name and nothing else.
+ """
+assistant_content=""
+user_context="Transfer learning with transformers: Using transformers for transfer learning on text classification tasks."
+tag=get_tag(model=model,system_content=system_context,assistant_content=assistant_content,user_content=user_context)
+print(tag)
+
+
+natural-language-processing
+
+
+
Next, let's create a function that can predict tags for a list of inputs.
+
1
+2
+3
# List of dicts w/ {title, description} (just the first 3 samples for now)
+samples=test_df[["title","description"]].to_dict(orient="records")[:3]
+samples
+
+
+[{'title': 'Diffusion to Vector',
+ 'description': 'Reference implementation of Diffusion2Vec (Complenet 2018) built on Gensim and NetworkX. '},
+ {'title': 'Graph Wavelet Neural Network',
+ 'description': 'A PyTorch implementation of "Graph Wavelet Neural Network" (ICLR 2019) '},
+ {'title': 'Capsule Graph Neural Network',
+ 'description': 'A PyTorch implementation of "Capsule Graph Neural Network" (ICLR 2019).'}]
+
Next we'll define a function that can clean our predictions in the event that it's not the proper format or has hallucinated a tag outside of our expected tags.
We'll start with zero-shot learning which involves providing the model with the system_content that tells it how to behave but no examples of the behavior (no assistant_content).
+
1
+2
+3
+4
+5
system_content=f"""
+ You are a NLP prediction service that predicts the label given an input's title and description.
+ You must choose between one of the following labels for each input: {tags}.
+ Only respond with the label name and nothing else.
+ """
+
Now, we'll be adding a assistant_context with a few samples from our training data for each class. The intuition here is that we're giving the model a few examples (few-shot learning) of what each class looks like so that it can learn to generalize better.
+
1
+2
+3
+4
+5
+6
+7
+8
# Create additional context with few samples from each class
+num_samples=2
+additional_context=[]
+cols_to_keep=["title","description","tag"]
+fortagintags:
+ samples=train_df[cols_to_keep][train_df.tag==tag][:num_samples].to_dict(orient="records")
+ additional_context.extend(samples)
+additional_context
+
+
+[{'title': 'Comparison between YOLO and RCNN on real world videos',
+ 'description': 'Bringing theory to experiment is cool. We can easily train models in colab and find the results in minutes.',
+ 'tag': 'computer-vision'},
+ {'title': 'Show, Infer & Tell: Contextual Inference for Creative Captioning',
+ 'description': 'The beauty of the work lies in the way it architects the fundamental idea that humans look at the overall image and then individual pieces of it.\r\n',
+ 'tag': 'computer-vision'},
+ {'title': 'Awesome Graph Classification',
+ 'description': 'A collection of important graph embedding, classification and representation learning papers with implementations.',
+ 'tag': 'other'},
+ {'title': 'Awesome Monte Carlo Tree Search',
+ 'description': 'A curated list of Monte Carlo tree search papers with implementations. ',
+ 'tag': 'other'},
+ {'title': 'Rethinking Batch Normalization in Transformers',
+ 'description': 'We found that NLP batch statistics exhibit large variance throughout training, which leads to poor BN performance.',
+ 'tag': 'natural-language-processing'},
+ {'title': 'ELECTRA: Pre-training Text Encoders as Discriminators',
+ 'description': 'PyTorch implementation of the electra model from the paper: ELECTRA - Pre-training Text Encoders as Discriminators Rather Than Generators',
+ 'tag': 'natural-language-processing'},
+ {'title': 'Pytest Board',
+ 'description': 'Continuous pytest runner with awesome visualization.',
+ 'tag': 'mlops'},
+ {'title': 'Debugging Neural Networks with PyTorch and W&B',
+ 'description': 'A closer look at debugging common issues when training neural networks.',
+ 'tag': 'mlops'}]
+
+
+
1
+2
+3
# Add assistant context
+assistant_content=f"""Here are some examples with the correct labels: {additional_context}"""
+print(assistant_content)
+
+
+Here are some examples with the correct labels: [{'title': 'Comparison between YOLO and RCNN on real world videos', ... 'description': 'A closer look at debugging common issues when training neural networks.', 'tag': 'mlops'}]
+
+
+
+
Tip
+
We could increase the number of samples by increasing the context length. We could also retrieve better few-shot samples by extracting examples from the training data that are similar to the current sample (ex. similar unique vocabulary).
As we can see, few shot learning performs better than it's respective zero shot counter part. GPT 4 has had considerable improvements in reducing hallucinations but for our supervised task this comes at an expense of high precision but lower recall and f1 scores. When GPT 4 is not confident, it would rather predict other.
+
OSS LLMs
+
So far, we've only been using closed-source models from OpenAI. While these are currently the gold-standard, there are many open-source models that are rapidly catching up (Falcon 40B, Llama 2, etc.). Before we see how these models perform on our task, let's first consider a few reasons why we should care about open-source models.
+
+
data ownership: you can serve your models and pass data to your models, without having to share it with a third-party API endpoint.
+
fine-tune: with access to our model's weights, we can actually fine-tune them, as opposed to experimenting with fickle prompting strategies.
+
optimization: we have full freedom to optimize our deployed models for inference (ex. quantization, pruning, etc.) to reduce costs.
+
+
1
# Coming soon in August!
+
+
Results
+
Now let's compare all the results from our generative AI LLM benchmarks:
And we can plot these on a bar plot to compare them visually.
+
1
+2
+3
+4
+5
+6
# Transform data into a new dictionary with four keys
+by_model_and_context={}
+forcontext_type,models_datainperformance.items():
+ formodel,metricsinmodels_data.items():
+ key=f"{model}_{context_type}"
+ by_model_and_context[key]=metrics
+
# Extracting the model names and the metric values
+models=list(by_model_and_context.keys())
+metrics=list(by_model_and_context[models[0]].keys())
+
+# Plotting the bar chart with metric scores on top of each bar
+fig,ax=plt.subplots(figsize=(10,4))
+width=0.2
+x=range(len(models))
+
+fori,metricinenumerate(metrics):
+ metric_values=[by_model_and_context[model][metric]formodelinmodels]
+ ax.bar([pos+width*iforposinx],metric_values,width,label=metric)
+ # Displaying the metric scores on top of each bar
+ forpos,valinzip(x,metric_values):
+ ax.text(pos+width*i,val,f'{val:.3f}',ha='center',va='bottom',fontsize=9)
+
+ax.set_xticks([pos+widthforposinx])
+ax.set_xticklabels(models,rotation=0,ha='center',fontsize=8)
+ax.set_ylabel('Performance')
+ax.set_title('GPT Benchmarks')
+ax.legend(loc='upper left',bbox_to_anchor=(1,1))
+
+plt.tight_layout()
+plt.show()
+
+
+
+
+
+
Our best model is GPT 4 with few shot learning at an f1 score of ~93%. We will see, in the rest of the course, how fine-tuning an LLM with a proper training dataset to change the actual weights of the last N layers (as opposed to the hard prompt tuning here) will yield similar/slightly better results to GPT 4 (at a fraction of the model size and inference costs).
+
However, the best system might actually be a combination of using these few-shot hard prompt LLMs alongside fine-tuned LLMs. For example, our fine-tuned LLMs in the course will perform well when the test data is similar to the training data (similar distributions of vocabulary, etc.) but may not perform well on out of distribution. Whereas, these hard prompted LLMs, by themselves or augmented with additional context (ex. arXiv plugins in our case), could be used when our primary fine-tuned model is not so confident.
+
Setup
+
We'll start by defining some setup utilities and configuring our model.
When working with very large datasets, it's a good idea to limit the number of samples in our dataset so that we can execute our code quickly and iterate on bugs, etc. This is why we have a num_samples input argument in our load_data function (None = no limit, all samples).
+
+
We'll also define a custom preprocessor class that we'll to conveniently preprocess our dataset but also to save/load for later. When defining a preprocessor, we'll need to define a _fit method to learn how to fit to our dataset and a _transform_{pandas|numpy} method to preprocess the dataset using any components from the _fit method. We can either define a _transform_pandas method to apply our preprocessing to a Pandas DataFrame or a _transform_numpy method to apply our preprocessing to a NumPy array. We'll define the _transform_pandas method since our preprocessing function expects a batch of data as a Pandas DataFrame.
+
1
+2
+3
+4
+5
+6
+7
+8
classCustomPreprocessor(Preprocessor):
+"""Custom preprocessor class."""
+ def_fit(self,ds):
+ tags=ds.unique(column="tag")
+ self.class_to_index={tag:ifori,taginenumerate(tags)}
+ self.index_to_class={v:kfork,vinself.class_to_index.items()}
+ def_transform_pandas(self,batch):# could also do _transform_numpy
+ returnpreprocess(batch,class_to_index=self.class_to_index)
+
+
Model
+
Now we're ready to start defining our model architecture. We'll start by loading a pretrained LLM and then defining the components needed for fine-tuning it on our dataset. Our pretrained LLM here is a transformer-based model that has been pretrained on a large corpus of scientific text called scibert.
+
+
+
+
+
+
If you're not familiar with transformer-based models like LLMs, be sure to check out the attention and Transformers lessons.
Once our model is loaded, we can tokenize an input text, convert it to torch tensors and pass it through our model to get a sequence and pooled representation of the text.
+
1
+2
+3
+4
+5
+6
# Sample
+text="Transfer learning with transformers for text classification."
+batch=tokenizer([text],return_tensors="np",padding="longest")
+batch={k:torch.tensor(v)fork,vinbatch.items()}# convert to torch tensors
+seq,pool=llm(input_ids=batch["input_ids"],attention_mask=batch["attention_mask"])
+np.shape(seq),np.shape(pool)
+
We're going to use this pretrained model to represent our input text features and add additional layers (linear classifier) on top of it for our specific classification task. In short, the pretrained LLM will process the tokenized text and return a sequence (one representation after each token) and pooled (combined) representation of the text. We'll use the pooled representation as input to our final fully-connection layer (fc1) to result in a vector of size num_classes (number of classes) that we can use to make predictions.
We can iterate through our dataset in batches however we may have batches of different sizes. Recall that our tokenizer padded the inputs to the longest item in the batch (padding="longest"). However, our batches for training will be smaller than our large data processing batches and so our batches here may have inputs with different lengths. To address this, we're going to define a custom collate_fn to repad the items in our training batches.
+
1
fromray.train.torchimportget_device
+
+
Our pad_array function will take an array of arrays and pad the inner arrays to the longest length.
We'll start by defining what one step (or iteration) of training looks like. This will be a function that takes in a batch of data, a model, a loss function, and an optimizer. It will then perform a forward pass, compute the loss, and perform a backward pass to update the model's weights. And finally, it will return the loss.
Note: We're using the ray.data.iter_torch_batches method instead of torch.utils.data.DataLoader to create a generator that will yield batches of data. In fact, this is the only line that's different from a typical PyTorch training loop and the actual training workflow remains untouched. Ray supports many other ways to load/consume data for different frameworks as well.
+
+
The validation step is quite similar to the training step but we don't need to perform a backward pass or update the model's weights.
Next, we'll define the train_loop_per_worker which defines the overall training loop for each worker. It's important that we include operations like loading the datasets, models, etc. so that each worker will have its own copy of these objects. Ray takes care of combining all the workers' results at the end of each iteration, so from the user's perspective, it's the exact same as training on a single machine!
+
The only additional lines of code we need to add compared to a typical PyTorch training loop are the following:
+
+
session.get_dataset_shard("train") and session.get_dataset_shard("val") to load the data splits (session.get_dataset_shard).
+
model=train.torch.prepare_model(model) to prepare the torch model for distributed execution (train.torch.prepare_model).
+
batch_size_per_worker=batch_size//session.get_world_size() to adjust the batch size for each worker (session.get_world_size).
+
session.report(metrics,checkpoint=checkpoint) to report metrics and save our model checkpoint (session.report).
+
+
All the other lines of code are the same as a typical PyTorch training loop!
Our dataset doesn't suffer from horrible class imbalance, but if it did, we could easily account for it through our loss function. There are also other strategies such as over-sampling less frequent classes and under-sampling popular classes.
+
1
+ 2
+ 3
+ 4
+ 5
+ 6
+ 7
+ 8
+ 9
+10
+11
# Class weights
+batch_counts=[]
+forbatchintrain_ds.iter_torch_batches(batch_size=256,collate_fn=collate_fn):
+ batch_counts.append(np.bincount(batch["targets"].cpu().numpy()))
+counts=[sum(count)forcountinzip(*batch_counts)]
+class_weights=np.array([1.0/countfori,countinenumerate(counts)])
+class_weights_tensor=torch.Tensor(class_weights).to(get_device())
+
+# Training components
+loss_fn=nn.BCEWithLogitsLoss(weight=class_weights_tensor)
+...
+
+
+
Configurations
+
Next, we'll define some configurations that will be used to train our model.
Next we'll define our scaling configuration (ScalingConfig) that will specify how we want to scale our training workload. We specify the number of workers (num_workers), whether to use GPU or not (use_gpu), the resources per worker (resources_per_worker) and how much CPU each worker is allowed to use (_max_cpu_fraction_per_node).
_max_cpu_fraction_per_node=0.8 indicates that 20% of CPU is reserved for non-training workloads that our workers will do such as data preprocessing (which we do prior to training anyway).
+
+
Next, we'll define our CheckpointConfig which will specify how we want to checkpoint our model. Here we will just save one checkpoint (num_to_keep) based on the checkpoint with the minval_loss. We'll also configure a RunConfig which will specify the name of our run and where we want to save our checkpoints.
+
1
+2
+3
# Run config
+checkpoint_config=CheckpointConfig(num_to_keep=1,checkpoint_score_attribute="val_loss",checkpoint_score_order="min")
+run_config=RunConfig(name="llm",checkpoint_config=checkpoint_config,local_dir="~/ray_results")
+
+
We'll be naming our experiment llm and saving our results to ~/ray_results, so a sample directory structure for our trained models would look like this:
The TorchTrainer_ objects are the individuals runs in this experiment and each one will have the following contents:
+
/home/ray/ray_results/TorchTrainer_fd40a_00000_0_2023-07-20_18-14-50/
+├──checkpoint_000009/# we only save one checkpoint (the best)
+├──events.out.tfevents.1689902160.ip-10-0-49-200
+├──params.json
+├──params.pkl
+├──progress.csv
+└──result.json
+
+
+
There are several other configs that we could set with Ray (ex. failure handling) so be sure to check them out here.
+
+
+
Stopping criteria
+
While we'll just let our experiments run for a certain number of epochs and stop automatically, our RunConfig accepts an optional stopping criteria (stop) which determines the conditions our training should stop for. It's entirely customizable and common examples include a certain metric value, elapsed time or even a custom class.
+
+
Training
+
Now we're finally ready to train our model using all the components we've setup above.
+
1
+2
+3
# Load and split data
+ds=load_data()
+train_ds,val_ds=stratify_split(ds,stratify="tag",test_size=test_size)
+
Calling materialize here is important because it will cache the preprocessed data in memory. This will allow us to train our model without having to reprocess the data each time.
+
+
Because we've preprocessed the data prior to training, we can use the fit=False and transform=False flags in our dataset config. This will allow us to skip the preprocessing step during training.
We'll pass all of our functions and configs to the TorchTrainer class to start training. Ray supports a wide variety of framework Trainers so if you're using other frameworks, you can use the corresponding Trainer class instead.
While our model is training, we can inspect our Ray dashboard to observe how our compute resources are being utilized.
+
+
💻 Local
+
We can inspect our Ray dashboard by opening http://127.0.0.1:8265 on a browser window. Click on Cluster on the top menu bar and then we will be able to see a list of our nodes (head and worker) and their utilizations.
+
+
+
🚀 Anyscale
+
On Anyscale Workspaces, we can head over to the top right menu and click on 🛠️ Tools → Ray Dashboard and this will open our dashboard on a new tab. Click on Cluster on the top menu bar and then we will be able to see a list of our nodes (head and worker) and their utilizations.
+
+
+
+
+
+
+
Learn about all the other observability features on the Ray Dashboard through this video.
+
+
Evaluation
+
Now that we've trained our model, we can evaluate it on a separate holdout test set. We'll cover the topic of evaluation much more extensively in our evaluation lesson but for now we'll calculate some quick overall metrics.
Now let's load our trained model for inference on new data. We'll create a few utility functions to format the probabilities into a dictionary for each class and to return predictions for each item in a dataframe.
And now we're ready to apply our model to new data. We'll create a sample dataframe with a title and description and then use our predict_with_proba function to get the predictions. Note that we use a placeholder value for tag since our input dataframe will automatically be preprocessed (and it expects a value in the tag column).
+
1
+2
+3
+4
+5
# Predict on sample
+title="Transfer learning with transformers"
+description="Using transformers for transfer learning on text classification tasks."
+sample_df=pd.DataFrame([{"title":title,"description":description,"tag":"other"}])
+predict_with_proba(df=sample_df,predictor=predictor)
+
Distributed training strategies are great for when our data or models are too large for training but there are additional strategies to make the models itself smaller for serving. The following model compression techniques are commonly used to reduce the size of the model:
+
+
Pruning: remove weights (unstructured) or entire channels (structured) to reduce the size of the network. The objective is to preserve the model’s performance while increasing its sparsity.
+
Quantization: reduce the memory footprint of the weights by reducing their precision (ex. 32 bit to 8 bit). We may loose some precision but it shouldn’t affect performance too much.
+
Distillation: training smaller networks to “mimic” larger networks by having it reproduce the larger network’s layers’ outputs.
+
+
+
+
+
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Training - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
Hyperparameter tuning is the process of discovering a set of performant parameter values for our model. It can be a computationally involved process depending on the number of parameters, search space and model architectures. Hyperparameters don't just include the model's parameters but could also include parameters related to preprocessing, splitting, etc. When we look at all the different parameters that can be tuned, it quickly becomes a very large search space. However, just because something is a hyperparameter doesn't mean we need to tune it.
+
+
It's absolutely acceptable to fix some hyperparameters (ex. using lower cased text [lower=True] during preprocessing).
+
You can initially just tune a small, yet influential, subset of hyperparameters that you believe will yield great results.
+
+
We want to optimize our hyperparameters so that we can understand how each of them affects our objective. By running many trials across a reasonable search space, we can determine near ideal values for our different parameters.
+
Frameworks
+
There are many options for hyperparameter tuning (Ray tune, Optuna, Hyperopt, etc.). We'll be using Ray Tune with it's HyperOpt integration for it's simplicity and general popularity. Ray Tune also has a wide variety of support for many other tune search algorithms (Optuna, Bayesian, etc.).
+
Set up
+
There are many factors to consider when performing hyperparameter tuning. We'll be conducting a small study where we'll tune just a few key hyperparameters across a few trials. Feel free to include additional parameters and to increase the number trials in the tuning experiment.
+
1
+2
# Number of trials (small sample)
+num_runs=2
+
+
We'll start with some the set up, data and model prep as we've done in previous lessons.
We can think of tuning as training across different combinations of parameters. For this, we'll need to define several configurations around when to stop tuning (stopping criteria), how to define the next set of parameters to train with (search algorithm) and even the different values that the parameters can take (search space).
Notice that we use the same mlflow_callback from our experiment tracking lesson so all of our runs will be tracked to MLflow automatically.
+
+
Search algorithm
+
Next, we're going to set the initial parameter values and the search algorithm (HyperOptSearch) for our tuning experiment. We're also going to set the maximum number of trials that can be run concurrently (ConcurrencyLimiter) based on the compute resources we have.
+
1
+2
+3
+4
# Hyperparameters to start with
+initial_params=[{"train_loop_config":{"dropout_p":0.5,"lr":1e-4,"lr_factor":0.8,"lr_patience":3}}]
+search_alg=HyperOptSearch(points_to_evaluate=initial_params)
+search_alg=ConcurrencyLimiter(search_alg,max_concurrent=2)
+
+
+
Tip
+
It's a good idea to start with some initial parameter values that you think might be reasonable. This can help speed up the tuning process and also guarantee at least one experiment that will perform decently well.
+
+
Search space
+
Next, we're going to define the parameter search space by choosing the parameters, their distribution and range of values. Depending on the parameter type, we have many different distributions to choose from.
Next, we're going to define a scheduler to prune unpromising trials. We'll be using AsyncHyperBandScheduler (ASHA), which is a very popular and aggressive early-stopping algorithm. Due to our aggressive scheduler, we'll set a grace_period to allow the trials to run for at least a few epochs before pruning and a maximum of max_t epochs.
+
1
+2
+3
+4
+5
# Scheduler
+scheduler=AsyncHyperBandScheduler(
+ max_t=train_loop_config["num_epochs"],# max epoch (<time_attr>) per trial
+ grace_period=5,# min epoch (<time_attr>) per trial
+)
+
+
Tuner
+
Finally, we're going to define a TuneConfig that will combine the search_alg and scheduler we've defined above.
# All trials in experiment
+results.get_dataframe()
+
+
+
+
+
+
+
epoch
+
lr
+
train_loss
+
val_loss
+
timestamp
+
time_this_iter_s
+
should_checkpoint
+
done
+
training_iteration
+
trial_id
+
...
+
pid
+
hostname
+
node_ip
+
time_since_restore
+
iterations_since_restore
+
config/train_loop_config/dropout_p
+
config/train_loop_config/lr
+
config/train_loop_config/lr_factor
+
config/train_loop_config/lr_patience
+
logdir
+
+
+
+
+
0
+
9
+
0.000100
+
0.04096
+
0.217990
+
1689460552
+
6.890944
+
True
+
True
+
10
+
094e2a7e
+
...
+
94006
+
ip-10-0-48-210
+
10.0.48.210
+
76.588228
+
10
+
0.500000
+
0.000100
+
0.800000
+
3.000000
+
/home/ray/ray_results/TorchTrainer_2023-07-15_...
+
+
+
1
+
0
+
0.000027
+
0.63066
+
0.516547
+
1689460571
+
14.614296
+
True
+
True
+
1
+
4f419368
+
...
+
94862
+
ip-10-0-48-210
+
10.0.48.210
+
14.614296
+
1
+
0.724894
+
0.000027
+
0.780224
+
5.243006
+
/home/ray/ray_results/TorchTrainer_2023-07-15_...
+
+
+
+
+
+
And on our MLflow dashboard, we can create useful plots like a parallel coordinates plot to visualize the different hyperparameters and their values across the different trials.
+
+
+
+
+
Best trial
+
And from these results, we can extract the best trial and its hyperparameters:
+
1
+2
+3
# Best trial's epochs
+best_trial=results.get_best_result(metric="val_loss",mode="min")
+best_trial.metrics_dataframe
+
+
+
+
+
+
+
epoch
+
lr
+
train_loss
+
val_loss
+
timestamp
+
time_this_iter_s
+
should_checkpoint
+
done
+
training_iteration
+
trial_id
+
date
+
time_total_s
+
pid
+
hostname
+
node_ip
+
time_since_restore
+
iterations_since_restore
+
+
+
+
+
0
+
0
+
0.0001
+
0.582092
+
0.495889
+
1689460489
+
14.537316
+
True
+
False
+
1
+
094e2a7e
+
2023-07-15_15-34-53
+
14.537316
+
94006
+
ip-10-0-48-210
+
10.0.48.210
+
14.537316
+
1
+
+
+
1
+
1
+
0.0001
+
0.492427
+
0.430734
+
1689460497
+
7.144841
+
True
+
False
+
2
+
094e2a7e
+
2023-07-15_15-35-00
+
21.682157
+
94006
+
ip-10-0-48-210
+
10.0.48.210
+
21.682157
+
2
+
+
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
...
+
+
+
9
+
9
+
0.0001
+
0.040960
+
0.217990
+
1689460552
+
6.890944
+
True
+
True
+
10
+
094e2a7e
+
2023-07-15_15-35-55
+
76.588228
+
94006
+
ip-10-0-48-210
+
10.0.48.210
+
76.588228
+
10
+
+
+
+
+
+
1
+2
# Best trial's hyperparameters
+best_trial.config["train_loop_config"]
+
# Predict on sample
+title="Transfer learning with transformers"
+description="Using transformers for transfer learning on text classification tasks."
+sample_df=pd.DataFrame([{"title":title,"description":description,"tag":"other"}])
+predict_with_proba(df=sample_df,predictor=predictor)
+
Now that we're tuned our model, in the next lesson, we're going to perform a much more intensive evaluation on our model compared to just viewing it's overall metrics on a test set.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Tuning - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Intuition
+
In this lesson, we're going to learn how to version our code, data and models to ensure reproducible behavior in our ML systems. It's imperative that we can reproduce our results and track changes to our system so we can debug and improve our application. Without it, it would be difficult to share our work, recreate our models in the event of system failures and fallback to previous versions in the event of regressions.
+
Code
+
To version our code, we'll be using git, which is a widely adopted version control system. In fact, when we cloned our repository in the setup lesson, we pulled code from a git repository that we had prepared for you.
We can then make changes to the code and Git, which is running locally on our computer, will keep track of our files and it's versions as we add and commit our changes. But it's not enough to just version our code locally, we need to push our work to a central location that can be pulled by us and others we want to grant access to. This is where remote repositories like GitHub, GitLab, BitBucket, etc. provide a remote location to hold our versioned code in.
+
+
+
+
+
Here's a simplified workflow for how we version our code using GitHub:
If you're not familiar with Git, we highly recommend going through our Git lesson to learn the basics.
+
+
Artifacts
+
While Git is ideal for saving our code, it's not ideal for saving artifacts like our datasets (especially unstructured data like text or images) and models. Also, recall that Git stores every version of our files and so large files that change frequently can very quickly take up space. So instead, it would be ideal if we can save locations (pointers) to these large artifacts in our code as opposed to the artifacts themselves. This way, we can version the locations of our artifacts and pull them as they're needed.
+
+
+
+
+
Data
+
While we're saving our dataset on GitHub for easy course access (and because our dataset is small), in a production setting, we would use a remote blob storage like S3 or a data warehouse like Snowflake. There are also many tools available for versioning our data, such as GitLFS, Dolt, Pachyderm, DVC, etc. With any of these solutions, we would be pointing to our remote storage location and versioning the pointer locations (ex. S3 bucket path) to our data instead of the data itself.
+
Models
+
And similarly, we currently store our models locally where the MLflow artifact and backend store are local directories.
In a production setting, these would be remote such as S3 for the artifact store and a database service (ex. PostgreSQL RDS) as our backend store. This way, our models can be versioned and others, with the appropriate access credentials, can pull the model artifacts and deploy them.
+
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Versioning - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
Learn how to combine machine learning with software engineering to design, develop, deploy and iterate on production ML applications. → GokuMohandas/Made-With-ML
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
+ While the specific task in this course involves fine-tuning an LLM for a supervised task, everything we learn easily extends to all applications (NLP, CV, time-series, etc.), models (regression → LLMs), data modalities (tabular, text, etc.), cloud platforms (AWS, GCP) and scale (local laptop → distributed cluster).
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ First principles
+
+
+ Before we jump straight into the code, we develop a first principles understanding for every machine learning concept.
+
+
+
+
+ Best practices
+
+
+ Implement software engineering best practices as we develop and deploy our machine learning models.
+
+
+
+
+ Scale
+
+
+ Easily scale ML workloads (data, train, tune, serve) in Python without having to learn completely new languages.
+
+
+
+
+ MLOps
+
+
+ Connect MLOps components (tracking, testing, serving, orchestration, etc.) as we build an end-to-end machine learning system.
+
+
+
+
+ Dev to Prod
+
+
+ Learn how to quickly and reliably go from development to production without any changes to our code or infra management.
+
+
+
+
+ CI/CD
+
+
+ Learn how to create mature CI/CD workflows to continuously train and deploy better models in a modular way that integrates with any stack.
+
+
+
+
+
+
+
+
+
+
Who is this content for?
+
Machine learning is not a separate industry, instead, it's a powerful way of thinking about data that's not reserved for any one type of person.
+
+
+
+
+ 👩💻 All developers
+
+
+ Whether software/infra engineer or data scientist, ML is increasingly becoming a key part of the products that you'll be developing.
+
+
+
+
+ 👩🎓 College graduates
+
+
+ Learn the practical skills required for industry and bridge gap between the university curriculum and what industry expects.
+
+
+
+
+ 👩💼 Product/Leadership
+
+
+ who want to develop a technical foundation so that they can build amazing (and reliable) products powered by machine learning.
+
+
+
+
+
+
+
Meet your instructor
+
+
+
+
+
+
Hi, I'm Goku Mohandas
+
+
+
+
+
+
+
+
+
+ I've spent my career developing ML applications across all scales and industries. Specifically over the last four years (through Made With ML), I’ve had the opportunity to help dozens of F500 companies + startups build out their ML platforms and launch high-impact ML applications on top of them. I started Made With ML to address the gaps in education and share the best practices on how to deliver value with ML in production.
+
+
+
+ While this was an amazing experience, it was also a humbling one because there were obstacles around scale, integrations and productionization that I didn’t have great solutions for. So, I decided to join a team that has been addressing these precise obstacles with some of the best ML teams in the world and has an even bigger vision I could stand behind. So I'm excited to announce that Made With ML is now part of Anyscale to accelerate the path towards production ML.
+
+
+
+🎉 Made With ML is now part of Anyscale, read more about it here!
+
+
+
+
+
+
❤️ Wall of Love
+
See what the community has to say about Made With ML.
"This course has given me the know-how to make optimal choices around design & implementation of ML engineering for a variety of real-world use-cases."
"This course really mimics the production ML thought process by providing alternative options with different levels of complexity & weighing on the pros/cons."
"Coming from academia with purely model-specific knowledge, Made With ML set the expectations right when it comes to how ML is being applied in the industry."
"The best MLOps resource that I've come across on the web. Goes over whys, hows, tradeoffs, tools & their alternatives via high-quality explanations and code."
"Completely sold out on the clean code and detailed writeup. This is one of the few ML courses which doesn't stop on just training a model but goes beyond."
+ Machine learning is increasingly becoming a key part of many products and so companies are looking for people with deeper knowledge on not only modeling, but how to operationalize it (MLOps). It's a major advantage to understand the fundamentals of this field at this nascent stage so you can responsibly design, develop, deploy and iterate on production ML applications as a foundational developer in your respective industry.
+
+
+
+
+
+
+
+
What is the time commitment?
+
+
+
+
+ You can go through the lessons at your pace or sign up for our upcoming live cohort where we'll provide live lessons, QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
What happens after the course?
+
+
+
+
+ After the course, you'll have access to our private community where you can connect with alumni and meet future cohort members as well. You can continue to ask questions about the topics (especially as new tools enter the market), get feedback on your work, etc.
+
+
+
+
+
+
Is this course fully remote?
+
+
+
+
+ When you sign up for the course, you'll have the choice of attending remotely or at one of our in-person weekend sessions near you.
+
+
+
+
+
+
+
+
+
+ If you have additional questions, send us an email and we'll get back to you very soon.
+
+
+
+
+
To cite this content, please use:
+
1
+2
+3
+4
+5
+6
@article{madewithml,
+author={Goku Mohandas},
+title={ Home - Made With ML },
+howpublished={\url{https://madewithml.com/}},
+year={2023}
+}
+
Thank you for subscribing to our newsletter! A confirmation email was sent to the email address you provided. Please click on the confirmation button in the email to complete your subscription. If you don’t see it within a few minutes, be sure to check your promotions/spam/junk folder (and mark it Not junk so you receive our future emails).
We created Made With ML to educate and enable the community to responsibly develop, deploy and maintain production machine learning applications. While there are many facets to this mission, at its core are the relationships with teams who share this mission. We want to work together to help the community discover and use the best tools for their context and start the flywheel to make the tools even better.
+
Brand
+
A few numbers that reflect the 100% organic traction we've had over the past few years and the need it's filled in the community for bringing ML to production:
All of our lessons focus on first principles when approaching a concept. This helps develop the foundational understanding to be able to adapt to any stack. But to really solidify the understanding, we implement everything in code within the context of an end-to-end project. This helps understand the implicit value a tool provides and develop the decision making framework for constructing the appropriate stack. And because the community adopts what they've learned for their own use cases in industry, it's imperative that we use tools that can offer that enterprise maturity.
+
+
+
Your product will be deeply integrated into the MLOps course, where thousands of developers everyday will use and assess your product for their industry contexts. All of this visibility and traction is invaluable for industry adoption, standing out in the competitive landscape and using the feedback to improve the product.
+
+
We also have many downstream projects in progress to add more value on top of the content (async video course, private community, university labs, corporate onboarding, talent platform).
+
+
Join us
+
If your team is interested in joining our mission, reach out to us via email to learn more!
After you've applied and been accepted to the next cohort, copy and paste this email template below to send to your manager for reimbursement for the course. Feel free to add any additional details as you see fit.
+
+
Subject: Reimbursement for Made With ML's MLOps Course
+
Hi,
+
Hope you're doing well. I was recently accepted into Made With ML's MLOps course, which is an interactive project-based course on MLOps fundamentals. The course costs $1,250 but the value I'll gain for myself and our team/company will pay for the course almost immediately. I added some key information about the course below and would love to get this career development opportunity reimbursed.
An interactive project-based course to learn and apply the fundamentals of MLOps. I'll be learning to combine machine learning with software engineering best practices which I want to extend to build and improve our own systems. This course brings all of the MLOps best practices into one place, allowing me to quickly (and properly) learn it. And best of all, the course can be done before and after work, so it won't be interfering during work hours.
+
Here's a quick look at the curriculum:
+
+
+
+
+
Who's teaching the course?
+
The course is from Made With ML, one of the top ML repositories on GitHub (30K+ stars) with a growing community (30K+) and is a highly recommended resource used by industry. Their content not only covers MLOps concepts but they go deep into actually implementing everything with production quality code.
+
How will this help me?
+
I'll be learning the foundation I need to responsibly develop ML systems. This includes producing clean, production-grade code, testing my work, understanding MLOps (experiment management, monitoring, systems design, etc.) and data engineering (data stack, orchestration, feature stores) concepts.
+
How will this help our company?
+
What I learn will directly translate to better quality ML systems in our products. I'll also be able to engage in conversations with peers and management as we traverse this space to build what's right for us. And, most important of all, I'll be able to pass on what I learn as I collaborate with others in our team so we're all working towards building reliable ML systems.
Send me an email at goku@madewithml.com to say hi, a bit about yourself and what you're currently learning or working on. I personally respond to all emails and always love to meet people from the community.
+
+
+
+
Upcoming live cohorts
+
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+{% endblock %}
+
+
+{% block scripts %}
+{{ super() }}
+
+
+
+
+
+{% endblock %}
+
+{% block content %}
+ {% if page.edit_url %}
+
+ {% include ".icons/material/pencil.svg" %}
+
+ {% endif %}
+ {% if not "\x3ch1" in page.content %}
+
{{ page.meta.title | d(config.site_name, true)}}
+ {% if page and page.meta.description %}
+ {{ page.meta.description }}
+ {% endif %}
+ {% endif %}
+ {{ page.content }}
+ {% if page and page.meta %}
+ {% if page.meta.git_revision_date_localized or page.meta.revision_date%}
+ {% include "partials/source-date.html" %}
+ {% endif %}
+ {% endif %}
+
+{% endblock %}
\ No newline at end of file
diff --git a/overrides/newsletter.html b/overrides/newsletter.html
new file mode 100644
index 00000000..ab3c2300
--- /dev/null
+++ b/overrides/newsletter.html
@@ -0,0 +1,34 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Subscribe to our newsletter
+
📬 Receive new lessons straight to your inbox (once a month) and join 40K+
+ developers in learning how to responsibly deliver value with ML.
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/robots.txt b/robots.txt
new file mode 100644
index 00000000..2daa7737
--- /dev/null
+++ b/robots.txt
@@ -0,0 +1,2 @@
+User-agent: *
+Sitemap: https://www.madewithml.com/sitemap.xml
\ No newline at end of file
diff --git a/search/search_index.json b/search/search_index.json
new file mode 100644
index 00000000..33e82635
--- /dev/null
+++ b/search/search_index.json
@@ -0,0 +1 @@
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"\u00d7 Made With ML
Join 40K+ developers in learning how to responsibly deliver value with ML!
Subscribe View lessons"},{"location":"#course","title":"ML for DevelopersWho is this content for?","text":"
Learn how to combine machine learning with software engineering to design, develop, deploy and iterate on production ML applications. \u2192 GokuMohandas/Made-With-ML
1. \ud83c\udfa8 Design
Setup
Product
Systems
2. \ud83d\udd22 Data
Preparation
Exploration
Preprocessing
Distributed
3. \ud83e\udd16 Model
Training
Tracking
Tuning
Evaluation
Serving
4. \ud83d\udcbb Develop
Scripting
Command-line
5. \ud83d\udce6 Utilities
Logging
Documentation
Styling
Pre-commit
6. \ud83e\uddea Test
Code
Data
Models
7. \u267b\ufe0f Reproducibility
Versioning
8. \ud83d\ude80 Production
Jobs & Services
CI/CD workflows
Monitoring
Data engineering
Live cohort
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more \u00a0 While the specific task in this course involves fine-tuning an LLM for a supervised task, everything we learn easily extends to all applications (NLP, CV, time-series, etc.), models (regression \u2192 LLMs), data modalities (tabular, text, etc.), cloud platforms (AWS, GCP) and scale (local laptop \u2192 distributed cluster). First principles Before we jump straight into the code, we develop a first principles understanding for every machine learning concept. Best practices Implement software engineering best practices as we develop and deploy our machine learning models. Scale Easily scale ML workloads (data, train, tune, serve) in Python without having to learn completely new languages. MLOps Connect MLOps components (tracking, testing, serving, orchestration, etc.) as we build an end-to-end machine learning system. Dev to Prod Learn how to quickly and reliably go from development to production without any changes to our code or infra management. CI/CD Learn how to create mature CI/CD workflows to continuously train and deploy better models in a modular way that integrates with any stack.
Machine learning is not a separate industry, instead, it's a powerful way of thinking about data that's not reserved for any one type of person.
\ud83d\udc69\u200d\ud83d\udcbb\u00a0 All developers Whether software/infra engineer or data scientist, ML is increasingly becoming a key part of the products that you'll be developing. \ud83d\udc69\u200d\ud83c\udf93\u00a0 College graduates Learn the practical skills required for industry and bridge gap between the university curriculum and what industry expects. \ud83d\udc69\u200d\ud83d\udcbc\u00a0 Product/Leadership who want to develop a technical foundation so that they can build amazing (and reliable) products powered by machine learning."},{"location":"#instructor","title":"Meet your instructor","text":"Hi, I'm Goku Mohandas
I've spent my career developing ML applications across all scales and industries. Specifically over the last four years (through Made With ML), I\u2019ve had the opportunity to help dozens of F500 companies + startups build out their ML platforms and launch high-impact ML applications on top of them. I started Made With ML to address the gaps in education and share the best practices on how to deliver value with ML in production.
While this was an amazing experience, it was also a humbling one because there were obstacles around scale, integrations and productionization that I didn\u2019t have great solutions for. So, I decided to join a team that has been addressing these precise obstacles with some of the best ML teams in the world and has an even bigger vision I could stand behind. So I'm excited to announce that Made With ML is now part of Anyscale to accelerate the path towards production ML.
\ud83c\udf89\u00a0 Made With ML is now part of Anyscale, read more about it here!"},{"location":"#wall-of-love","title":"\u2764\ufe0f Wall of LoveFrequently Asked Questions (FAQ)","text":"
See what the community has to say about Made With ML.
Sherry Wang Senior ML Engineer - Cars.com
\"Made with ML is one of the best courses I\u2019ve ever taken. The material covered is very practical; I get to apply some of them to my job right away.\"
Deepak Jayakumaran Lead Data Scientist - Grab
\"This course has given me the know-how to make optimal choices around design & implementation of ML engineering for a variety of real-world use-cases.\"
Jeremy Jordan Senior ML Engineer - Duo Security
\"This will be a great journey for those interested in deploying machine learning models which lead to a positive impact on the product.\"
Clara Matos Head of AI Eng - Sword Health
\"This course really mimics the production ML thought process by providing alternative options with different levels of complexity & weighing on the pros/cons.\"
Ritchie Ng PyTorch Keynote Speaker
\"For production ML, I cannot possibly think of a better resource out there ... this resource is the gold standard.\"
Greg Coquillo AI Product - Amazon
\"One of the best places where you can learn the fundamentals of ML, then practice MLOps by building production grade products and delivering value!\"
Kavin Veerapandian Senior Analyst - Citi
\"Coming from academia with purely model-specific knowledge, Made With ML set the expectations right when it comes to how ML is being applied in the industry.\"
Daniel Bourke Founder - Mrdbourke
\"Built some machine learning models? Want to take them to the next level? I do. And I\u2019m using @madewithml to learn how. Outstanding MLOps lessons!\"
Dmitry Petrov Co-Founder, CEO - DVC
\"This is not a usual ML class, it covers productionalization part of ML projects - the most important part from a business point of view.\"
Lawrence Okegbemi ML Engineer - Enterscale
\"Following all through, it's really amazing to see how you demonstrated best practices in building an ML driven application.\"
Laxman Tomar ML Engineer - Robofied
\"The best MLOps resource that I've come across on the web. Goes over whys, hows, tradeoffs, tools & their alternatives via high-quality explanations and code.\"
Satyabrata Pal Julia Community
\"Completely sold out on the clean code and detailed writeup. This is one of the few ML courses which doesn't stop on just training a model but goes beyond.\"
\u2039 \u203a
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more Who is this course for? Machine learning is not a separate industry, instead, it's a powerful way of thinking about data that's not reserved for any one type of person.
All developers Whether software engineer or data scientist, ML is increaingly becoming a key part of the products that you'll be developing.
College graduates Learn the practical skills required for industry and bridge gap between the university curriculum and what industry expects.
Product / Leadesrhip who want to develop a technical foundation so that they can build amazing (and reliable) products powered by machine learning.
What are the prerequisites?
You should know how to code in Python and the basics of machine learning.
currently working with ML in industry or academia
Python (basics, NumPy, Pandas)
ML / deep learning basics (logistic regression, PyTorch, etc.)
Why should I take this course now? Machine learning is increasingly becoming a key part of many products and so companies are looking for people with deeper knowledge on not only modeling, but how to operationalize it (MLOps). It's a major advantage to understand the fundamentals of this field at this nascent stage so you can responsibly design, develop, deploy and iterate on production ML applications as a foundational developer in your respective industry. What is the time commitment? You can go through the lessons at your pace or sign up for our upcoming live cohort where we'll provide live lessons, QA, compute (GPUs) and community to learn everything in one day. What happens after the course? After the course, you'll have access to our private community where you can connect with alumni and meet future cohort members as well. You can continue to ask questions about the topics (especially as new tools enter the market), get feedback on your work, etc. Is this course fully remote? When you sign up for the course, you'll have the choice of attending remotely or at one of our in-person weekend sessions near you. \u00a0 If you have additional questions, send us an email and we'll get back to you very soon.
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Home - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"about/","title":"Our Mission","text":""},{"location":"about/#instructor","title":"Meet your instructor","text":"Hi, I'm Goku Mohandas
I've spent my career developing ML applications across all scales and industries. Specifically over the last four years (through Made With ML), I\u2019ve had the opportunity to help dozens of F500 companies + startups build out their ML platforms and launch high-impact ML applications on top of them. I started Made With ML to address the gaps in education and share the best practices on how to deliver value with ML in production.
While this was an amazing experience, it was also a humbling one because there were obstacles around scale, integrations and productionization that I didn\u2019t have great solutions for. So, I decided to join a team that has been addressing these precise obstacles with some of the best ML teams in the world and has an even bigger vision I could stand behind. So I'm excited to announce that Made With ML is now part of Anyscale to accelerate the path towards production ML.
\ud83c\udf89\u00a0 Made With ML is now part of Anyscale, read more about it here!
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more "},{"location":"courses/foundations/","title":"Foundations","text":"1. \ud83d\udee0 Toolkit
Notebooks
Python
NumPy
Pandas
PyTorch
2. \ud83d\udd25 Machine Learning
Linear Regression
Logistic Regression
Neural Networks
Data Quality
Utilities
3. \ud83e\udd16 Deep Learning
CNNs
Embeddings
RNNs
Attention
Transformers
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Foundations - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more \u00a0 While the specific task in this course involves fine-tuning an LLM for a supervised task, everything we learn easily extends to all applications (NLP, CV, time-series, etc.), models (regression \u2192 LLMs), data modalities (tabular, text, etc.), cloud platforms (AWS, GCP) and scale (local laptop \u2192 distributed cluster).
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { MLOps Course - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
In the RNN lesson, we were constrained to using the representation at the very end but what if we could give contextual weight to each encoded input (\\(h_i\\)) when making our prediction? This is also preferred because it can help mitigate the vanishing gradient issue which stems from processing very long sequences. Below is attention applied to the outputs from an RNN. In theory, the outputs can come from anywhere where we want to learn how to weight amongst them but since we're working with the context of an RNN from the previous lesson , we'll continue with that.
Variable Description \\(N\\) batch size \\(M\\) max sequence length in the batch \\(H\\) hidden dim, model dim, etc. \\(h\\) RNN outputs (or any group of outputs you want to attend to) \\(\\in \\mathbb{R}^{NXMXH}\\) \\(\\alpha_{t,i}\\) alignment function context vector \\(c_t\\) (attention in our case) $ \\(W_{attn}\\) attention weights to learn \\(\\in \\mathbb{R}^{HX1}\\) \\(c_t\\) context vector that accounts for the different inputs with attention
Objective:
At it's core, attention is about learning how to weigh a group of encoded representations to produce a context-aware representation to use for downstream tasks. This is done by learning a set of attention weights and then using softmax to create attention values that sum to 1.
Advantages:
Learn how to account for the appropriate encoded representations regardless of position.
Disadvantages:
Another compute step that involves learning weights.
Miscellaneous:
Several state-of-the-art approaches extend on basic attention to deliver highly context-aware representations (ex. self-attention).
title category 0 Sharon Accepts Plan to Reduce Gaza Army Operation... World 1 Internet Key Battleground in Wildlife Crime Fight Sci/Tech 2 July Durable Good Orders Rise 1.7 Percent Business 3 Growing Signs of a Slowing on Wall Street Business 4 The New Faces of Reality TV World"},{"location":"courses/foundations/attention/#preprocessing","title":"Preprocessing","text":"
We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc.
\n[nltk_data] Downloading package stopwords to /root/nltk_data...\n[nltk_data] Package stopwords is already up-to-date!\n['i', 'me', 'my', 'myself', 'we']\n
def preprocess(text, stopwords=STOPWORDS):\n\"\"\"Conditional preprocessing on our text unique to our task.\"\"\"\n # Lower\n text = text.lower()\n\n # Remove stopwords\n pattern = re.compile(r\"\\b(\" + r\"|\".join(stopwords) + r\")\\b\\s*\")\n text = pattern.sub(\"\", text)\n\n # Remove words in parenthesis\n text = re.sub(r\"\\([^)]*\\)\", \"\", text)\n\n # Spacing and filters\n text = re.sub(r\"([-;;.,!?<=>])\", r\" \\1 \", text)\n text = re.sub(\"[^A-Za-z0-9]+\", \" \", text) # remove non alphanumeric chars\n text = re.sub(\" +\", \" \", text) # remove multiple spaces\n text = text.strip()\n\n return text\n
# Sample\ntext = \"Great week for the NYSE!\"\npreprocess(text=text)\n
\ngreat week nyse\n
# Apply to dataframe\npreprocessed_df = df.copy()\npreprocessed_df.title = preprocessed_df.title.apply(preprocess)\nprint (f\"{df.title.values[0]}\\n\\n{preprocessed_df.title.values[0]}\")\n
\nSharon Accepts Plan to Reduce Gaza Army Operation, Haaretz Says\n\nsharon accepts plan reduce gaza army operation haaretz says\n
Warning
If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before applying those operations. This is because we cannot apply any knowledge gained from the test set accidentally (data leak) during preprocessing/training. However for global preprocessing steps like the function above where we aren't learning anything from the data itself, we can perform before splitting the data.
# Class weights\ncounts = np.bincount(y_train)\nclass_weights = {i: 1.0/count for i, count in enumerate(counts)}\nprint (f\"counts: {counts}\\nweights: {class_weights}\")\n
class Tokenizer(object):\n def __init__(self, char_level, num_tokens=None,\n pad_token=\"<PAD>\", oov_token=\"<UNK>\",\n token_to_index=None):\n self.char_level = char_level\n self.separator = \"\" if self.char_level else \" \"\n if num_tokens: num_tokens -= 2 # pad + unk tokens\n self.num_tokens = num_tokens\n self.pad_token = pad_token\n self.oov_token = oov_token\n if not token_to_index:\n token_to_index = {pad_token: 0, oov_token: 1}\n self.token_to_index = token_to_index\n self.index_to_token = {v: k for k, v in self.token_to_index.items()}\n\n def __len__(self):\n return len(self.token_to_index)\n\n def __str__(self):\n return f\"<Tokenizer(num_tokens={len(self)})>\"\n\n def fit_on_texts(self, texts):\n if not self.char_level:\n texts = [text.split(\" \") for text in texts]\n all_tokens = [token for text in texts for token in text]\n counts = Counter(all_tokens).most_common(self.num_tokens)\n self.min_token_freq = counts[-1][1]\n for token, count in counts:\n index = len(self)\n self.token_to_index[token] = index\n self.index_to_token[index] = token\n return self\n\n def texts_to_sequences(self, texts):\n sequences = []\n for text in texts:\n if not self.char_level:\n text = text.split(\" \")\n sequence = []\n for token in text:\n sequence.append(self.token_to_index.get(\n token, self.token_to_index[self.oov_token]))\n sequences.append(np.asarray(sequence))\n return sequences\n\n def sequences_to_texts(self, sequences):\n texts = []\n for sequence in sequences:\n text = []\n for index in sequence:\n text.append(self.index_to_token.get(index, self.oov_token))\n texts.append(self.separator.join([token for token in text]))\n return texts\n\n def save(self, fp):\n with open(fp, \"w\") as fp:\n contents = {\n \"char_level\": self.char_level,\n \"oov_token\": self.oov_token,\n \"token_to_index\": self.token_to_index\n }\n json.dump(contents, fp, indent=4, sort_keys=False)\n\n @classmethod\n def load(cls, fp):\n with open(fp, \"r\") as fp:\n kwargs = json.load(fp=fp)\n return cls(**kwargs)\n
Warning
It's important that we only fit using our train data split because during inference, our model will not always know every token so it's important to replicate that scenario with our validation and test splits as well.
# Sample of tokens\nprint (take(5, tokenizer.token_to_index.items()))\nprint (f\"least freq token's freq: {tokenizer.min_token_freq}\") # use this to adjust num_tokens\n
We'll need to do 2D padding to our tokenized text.
def pad_sequences(sequences, max_seq_len=0):\n\"\"\"Pad sequences to max length in sequence.\"\"\"\n max_seq_len = max(max_seq_len, max(len(sequence) for sequence in sequences))\n padded_sequences = np.zeros((len(sequences), max_seq_len))\n for i, sequence in enumerate(sequences):\n padded_sequences[i][:len(sequence)] = sequence\n return padded_sequences\n
Attention applied to the outputs from an RNN. In theory, the outputs can come from anywhere where we want to learn how to weight amongst them but since we're working with the context of an RNN from the previous lesson , we'll continue with that.
Variable Description \\(N\\) batch size \\(M\\) max sequence length in the batch \\(H\\) hidden dim, model dim, etc. \\(h\\) RNN outputs (or any group of outputs you want to attend to) \\(\\in \\mathbb{R}^{NXMXH}\\) \\(\\alpha_{t,i}\\) alignment function context vector \\(c_t\\) (attention in our case) $ \\(W_{attn}\\) attention weights to learn \\(\\in \\mathbb{R}^{HX1}\\) \\(c_t\\) context vector that accounts for the different inputs with attention
import torch.nn.functional as F\n
The RNN will create an encoded representation for each word in our input resulting in a stacked vector that has dimensions \\(NXMXH\\), where N is the # of samples in the batch, M is the max sequence length in the batch, and H is the number of hidden units in the RNN.
In a many-to-many task such as machine translation, our attentional interface will also account for the encoded representation of token in the output as well (via concatenation) so we can know which encoded inputs to attend to based on the encoded output we're focusing on. For more on this, be sure to explore Bahdanau's attention paper.
def get_probability_distribution(y_prob, classes):\n\"\"\"Create a dict of class probabilities from an array.\"\"\"\n results = {}\n for i, class_ in enumerate(classes):\n results[class_] = np.float64(y_prob[i])\n sorted_results = {k: v for k, v in sorted(\n results.items(), key=lambda item: item[1], reverse=True)}\n return sorted_results\n
Soft attention the type of attention we've implemented so far, where we attend to all encoded inputs when creating our context vector.
\n
\n
advantages: we always have the ability to attend to all inputs in case something we saw much earlier/ see later are crucial for determining the output.
\n
disadvantages: if our input sequence is very long, this can lead to expensive compute.
Hard attention is focusing on a specific set of the encoded inputs at each time step.
\n
\n
advantages: we can save a lot of compute on long sequences by only focusing on a local patch each time.
\n
disadvantages: non-differentiable and so we need to use more complex techniques (variance reduction, reinforcement learning, etc.) to train.
\n
\nShow, Attend and Tell: Neural Image Caption Generation with Visual Attention"},{"location":"courses/foundations/attention/#local-attention","title":"Local attention","text":"
Local attention blends the advantages of soft and hard attention. It involves learning an aligned position vector and empirically determining a local window of encoded inputs to attend to.
\n
\n
advantages: apply attention to a local patch of inputs yet remain differentiable.
\n
disadvantages: need to determine the alignment vector for each output but it's a worthwhile trade off to determine the right window of inputs to attend to in order to avoid attending to all of them.
\n
\nEffective Approaches to Attention-based Neural Machine Translation"},{"location":"courses/foundations/attention/#self-attention","title":"Self-attention","text":"
We can also use attention within the encoded input sequence to create a weighted representation that based on the similarity between input pairs. This will allow us to create rich representations of the input sequence that are aware of the relationships between each other. For example, in the image below you can see that when composing the representation of the token \"its\", this specific attention head will be incorporating signal from the token \"Law\" (it's learned that \"its\" is referring to the \"Law\").
\nAttention Is All You Need\n
In the next lesson, we'll implement Transformers that leverage self-attention to create contextual representations of our inputs for downstream applications.
\n
To cite this content, please use:
\n
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Attention - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
At the core of CNNs are filters (aka weights, kernels, etc.) which convolve (slide) across our input to extract relevant features. The filters are initialized randomly but learn to act as feature extractors via parameter sharing.
Objective:
Extract meaningful spatial substructure from encoded data.
title category 0 Sharon Accepts Plan to Reduce Gaza Army Operation... World 1 Internet Key Battleground in Wildlife Crime Fight Sci/Tech 2 July Durable Good Orders Rise 1.7 Percent Business 3 Growing Signs of a Slowing on Wall Street Business 4 The New Faces of Reality TV World"},{"location":"courses/foundations/convolutional-neural-networks/#preprocessing","title":"Preprocessing","text":"
We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc.
# Class weights\ncounts = np.bincount(y_train)\nclass_weights = {i: 1.0/count for i, count in enumerate(counts)}\nprint (f\"counts: {counts}\\nweights: {class_weights}\")\n
Our input data is text and we can't feed it directly to our models. So, we'll define a Tokenizer to convert our text input data into token indices. This means that every token (we can decide what a token is char, word, sub-word, etc.) is mapped to a unique index which allows us to represent our text as an array of indices.
class Tokenizer(object):\n def __init__(self, char_level, num_tokens=None,\n pad_token=\"<PAD>\", oov_token=\"<UNK>\",\n token_to_index=None):\n self.char_level = char_level\n self.separator = \"\" if self.char_level else \" \"\n if num_tokens: num_tokens -= 2 # pad + unk tokens\n self.num_tokens = num_tokens\n self.pad_token = pad_token\n self.oov_token = oov_token\n if not token_to_index:\n token_to_index = {pad_token: 0, oov_token: 1}\n self.token_to_index = token_to_index\n self.index_to_token = {v: k for k, v in self.token_to_index.items()}\n\n def __len__(self):\n return len(self.token_to_index)\n\n def __str__(self):\n return f\"<Tokenizer(num_tokens={len(self)})>\"\n\n def fit_on_texts(self, texts):\n if not self.char_level:\n texts = [text.split(\" \") for text in texts]\n all_tokens = [token for text in texts for token in text]\n counts = Counter(all_tokens).most_common(self.num_tokens)\n self.min_token_freq = counts[-1][1]\n for token, count in counts:\n index = len(self)\n self.token_to_index[token] = index\n self.index_to_token[index] = token\n return self\n\n def texts_to_sequences(self, texts):\n sequences = []\n for text in texts:\n if not self.char_level:\n text = text.split(\" \")\n sequence = []\n for token in text:\n sequence.append(self.token_to_index.get(\n token, self.token_to_index[self.oov_token]))\n sequences.append(np.asarray(sequence))\n return sequences\n\n def sequences_to_texts(self, sequences):\n texts = []\n for sequence in sequences:\n text = []\n for index in sequence:\n text.append(self.index_to_token.get(index, self.oov_token))\n texts.append(self.separator.join([token for token in text]))\n return texts\n\n def save(self, fp):\n with open(fp, \"w\") as fp:\n contents = {\n \"char_level\": self.char_level,\n \"oov_token\": self.oov_token,\n \"token_to_index\": self.token_to_index\n }\n json.dump(contents, fp, indent=4, sort_keys=False)\n\n @classmethod\n def load(cls, fp):\n with open(fp, \"r\") as fp:\n kwargs = json.load(fp=fp)\n return cls(**kwargs)\n
We're going to restrict the number of tokens in our Tokenizer to the top 500 most frequent tokens (stop words already removed) because the full vocabulary size (~35K) is too large to run on Google Colab notebooks.
# Sample of tokens\nprint (take(5, tokenizer.token_to_index.items()))\nprint (f\"least freq token's freq: {tokenizer.min_token_freq}\") # use this to adjust num_tokens\n
# Convert texts to sequences of indices\nX_train = tokenizer.texts_to_sequences(X_train)\nX_val = tokenizer.texts_to_sequences(X_val)\nX_test = tokenizer.texts_to_sequences(X_test)\npreprocessed_text = tokenizer.sequences_to_texts([X_train[0]])[0]\nprint (\"Text to indices:\\n\"\n f\" (preprocessed) \u2192 {preprocessed_text}\\n\"\n f\" (tokenized) \u2192 {X_train[0]}\")\n
\nText to indices:\n (preprocessed) \u2192 china <UNK> north korea nuclear talks\n (tokenized) \u2192 [ 16 1 285 142 114 24]\n
Did we need to split the data first?
How come we applied the preprocessing functions to the entire dataset but tokenization after splitting the dataset? Does it matter?
Show answer
If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before applying those operations. This is because we cannot apply any knowledge gained from the test set accidentally (data leak) during preprocessing/training. So for the tokenization process, it's important that we only fit using our train data split because during inference, our model will not always know every token so it's important to replicate that scenario with our validation and test splits as well. However for global preprocessing steps, like the preprocessing function where we aren't learning anything from the data itself, we can perform before splitting the data.
One-hot encoding creates a binary column for each unique value for the feature we're trying to map. All of the values in each token's array will be 0 except at the index that this specific token is represented by.
One-hot encoding allows us to represent our data in a way that our models can process the data and isn't biased by the actual value of the token (ex. if your labels were actual numbers).
We have already applied one-hot encoding in the previous lessons when we encoded our labels. Each label was represented by a unique index but when determining loss, we effectively use it's one hot representation and compared it to the predicted probability distribution. We never explicitly wrote this out since all of our previous tasks were multi-class which means every input had just one output class, so the 0s didn't affect the loss (though it did matter during back propagation).
def to_categorical(seq, num_classes):\n\"\"\"One-hot encode a sequence of tokens.\"\"\"\n one_hot = np.zeros((len(seq), num_classes))\n for i, item in enumerate(seq):\n one_hot[i, item] = 1.\n return one_hot\n
# Convert tokens to one-hot\nvocab_size = len(tokenizer)\nX_train = [to_categorical(seq, num_classes=vocab_size) for seq in X_train]\nX_val = [to_categorical(seq, num_classes=vocab_size) for seq in X_val]\nX_test = [to_categorical(seq, num_classes=vocab_size) for seq in X_test]\n
Our inputs are all of varying length but we need each batch to be uniformly shaped. Therefore, we will use padding to make all the inputs in the batch the same length. Our padding index will be 0 (note that this is consistent with the <PAD> token defined in our Tokenizer).
One-hot encoding creates a batch of shape (N, max_seq_len, vocab_size) so we'll need to be able to pad 3D sequences.
def pad_sequences(sequences, max_seq_len=0):\n\"\"\"Pad sequences to max length in sequence.\"\"\"\n max_seq_len = max(max_seq_len, max(len(sequence) for sequence in sequences))\n num_classes = sequences[0].shape[-1]\n padded_sequences = np.zeros((len(sequences), max_seq_len, num_classes))\n for i, sequence in enumerate(sequences):\n padded_sequences[i][:len(sequence)] = sequence\n return padded_sequences\n
# 3D sequences\nprint (X_train[0].shape, X_train[1].shape, X_train[2].shape)\npadded = pad_sequences(X_train[0:3])\nprint (padded.shape)\n
\n(6, 500) (5, 500) (6, 500)\n(3, 6, 500)\n
Is our pad_sequences function properly created?
Notice any assumptions that could lead to hidden bugs?
Show answer
By using np.zeros() to create our padded sequences, we're assuming that our pad token's index is 0. While this is the case for our project, someone could choose to use a different index and this can cause an error. Worst of all, this would be a silent error because all downstream operations would still run normally but our performance will suffer and it may not always be intuitive that this was the cause of issue!
In the dummy example below, our inputs are composed of character tokens that are one-hot encoded. We have a batch of N samples, where each sample has 8 characters and each character is represented by an array of 10 values (vocab size=10). This gives our inputs the size (N, 8, 10).
With PyTorch, when dealing with convolution, our inputs (X) need to have the channels as the second dimension, so our inputs will be (N, 10, 8).
import math\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n
# Assume all our inputs are padded to have the same # of words\nbatch_size = 64\nmax_seq_len = 8 # words per input\nvocab_size = 10 # one hot size\nx = torch.randn(batch_size, max_seq_len, vocab_size)\nprint(f\"X: {x.shape}\")\nx = x.transpose(1, 2)\nprint(f\"X: {x.shape}\")\n
This diagram above is for char-level tokens but extends to any level of tokenization."},{"location":"courses/foundations/convolutional-neural-networks/#filters","title":"Filters","text":"
At the core of CNNs are filters (aka weights, kernels, etc.) which convolve (slide) across our input to extract relevant features. The filters are initialized randomly but learn to act as feature extractors via parameter sharing.
We can see convolution in the diagram below where we simplified the filters and inputs to be 2D for ease of visualization. Also note that the values are 0/1s but in reality they can be any floating point value.
Now let's return to our actual inputs x, which is of shape (8, 10) [max_seq_len, vocab_size] and we want to convolve on this input using filters. We will use 50 filters that are of size (1, 3) and has the same depth as the number of channels (num_channels = vocab_size = one_hot_size = 10). This gives our filter a shape of (3, 10, 50) [kernel_size, vocab_size, num_filters]
stride: amount the filters move from one convolution operation to the next.
padding: values (typically zero) padded to the input, typically to create a volume with whole number dimensions.
So far we've used a stride of 1 and VALID padding (no padding) but let's look at an example with a higher stride and difference between different padding approaches.
Padding types:
VALID: no padding, the filters only use the \"valid\" values in the input. If the filter cannot reach all the input values (filters go left to right), the extra values on the right are dropped.
SAME: adds padding evenly to the right (preferred) and left sides of the input so that all values in the input are processed.
We're going to use the Conv1d layer to process our inputs.
# Convolutional filters (VALID padding)\nvocab_size = 10 # one hot size\nnum_filters = 50 # num filters\nfilter_size = 3 # filters are 3X3\nstride = 1\npadding = 0 # valid padding (no padding)\nconv1 = nn.Conv1d(in_channels=vocab_size, out_channels=num_filters,\n kernel_size=filter_size, stride=stride,\n padding=padding, padding_mode=\"zeros\")\nprint(\"conv: {}\".format(conv1.weight.shape))\n
When we apply these filter on our inputs, we receive an output of shape (N, 6, 50). We get 50 for the output channel dim because we used 50 filters and 6 for the conv outputs because:
Variable Description \\(W\\) width of each input = 8 \\(H\\) height of each input = 1 \\(D\\) depth (# of channels) \\(F\\) filter size = 3 \\(P\\) padding = 0 \\(S\\) stride = 1
Now we'll add padding so that the convolutional outputs are the same shape as our inputs. The amount of padding for the SAME padding can be determined using the same equation. We want out output to have the same width as our input, so we solve for P:
\\[ \\frac{W-F+2P}{S} + 1 = W \\] \\[ P = \\frac{S(W-1) - W + F}{2} \\]
If \\(P\\) is not a whole number, we round up (using math.ceil) and place the extra padding on the right side.
# Convolutional filters (SAME padding)\nvocab_size = 10 # one hot size\nnum_filters = 50 # num filters\nfilter_size = 3 # filters are 3X3\nstride = 1\nconv = nn.Conv1d(in_channels=vocab_size, out_channels=num_filters,\n kernel_size=filter_size, stride=stride)\nprint(\"conv: {}\".format(conv.weight.shape))\n
We will explore larger dimensional convolution layers in subsequent lessons. For example, Conv2D is used with 3D inputs (images, char-level text, etc.) and Conv3D is used for 4D inputs (videos, time-series, etc.).
The result of convolving filters on an input is a feature map. Due to the nature of convolution and overlaps, our feature map will have lots of redundant information. Pooling is a way to summarize a high-dimensional feature map into a lower dimensional one for simplified downstream computation. The pooling operation can be the max value, average, etc. in a certain receptive field. Below is an example of pooling where the outputs from a conv layer are 4X4 and we're going to apply max pool filters of size 2X2.
The last topic we'll cover before constructing our model is batch normalization. It's an operation that will standardize (mean=0, std=1) the activations from the previous layer. Recall that we used to standardize our inputs in previous notebooks so our model can optimize quickly with larger learning rates. It's the same concept here but we continue to maintain standardized values throughout the repeated forward passes to further aid optimization.
# Batch normalization\nbatch_norm = nn.BatchNorm1d(num_features=num_filters)\nz = batch_norm(conv(x)) # applied to activations (after conv layer & before pooling)\nprint (f\"z: {z.shape}\")\n
\nz: torch.Size([64, 50, 6])\n
# Mean and std before batchnorm\nprint (f\"mean: {torch.mean(conv(x)):.2f}, std: {torch.std(conv(x)):.2f}\")\n
\nmean: -0.00, std: 0.57\n
# Mean and std after batchnorm\nprint (f\"mean: {torch.mean(z):.2f}, std: {torch.std(z):.2f}\")\n
We'll first tokenize our inputs (batch_size, max_seq_len).
Then we'll one-hot encode our tokenized inputs (batch_size, max_seq_len, vocab_size).
We'll apply convolution via filters (filter_size, vocab_size, num_filters) followed by batch normalization. Our filters act as character level n-gram detectors.
We'll apply 1D global max pooling which will extract the most relevant information from the feature maps for making the decision.
We feed the pool outputs to a fully-connected (FC) layer (with dropout).
We use one more FC layer with softmax to derive class probabilities.
We used SAME padding (w/ stride=1) which means that the conv outputs will have the same width (max_seq_len) as our inputs. The amount of padding differs for each batch based on the max_seq_len but you can calculate it by solving for P in the equation below.
Let's create the Trainer class that we'll use to facilitate training for our experiments. Notice that we're now moving the train function inside this class.
def get_probability_distribution(y_prob, classes):\n\"\"\"Create a dict of class probabilities from an array.\"\"\"\n results = {}\n for i, class_ in enumerate(classes):\n results[class_] = np.float64(y_prob[i])\n sorted_results = {k: v for k, v in sorted(\n results.items(), key=lambda item: item[1], reverse=True)}\n return sorted_results\n
# Dataloader\ntext = \"What a day for the new york stock market to go bust!\"\nsequences = tokenizer.texts_to_sequences([preprocess(text)])\nprint (tokenizer.sequences_to_texts(sequences))\nX = [to_categorical(seq, num_classes=len(tokenizer)) for seq in sequences]\ny_filler = label_encoder.encode([label_encoder.classes[0]]*len(X))\ndataset = Dataset(X=X, y=y_filler, max_filter_size=FILTER_SIZE)\ndataloader = dataset.create_dataloader(batch_size=batch_size)\n
We went through all the trouble of padding our inputs before convolution to result in outputs of the same shape as our inputs so we can try to get some interpretability. Since every token is mapped to a convolutional output on which we apply max pooling, we can see which token's output was most influential towards the prediction. We first need to get the conv outputs from our model:
# Get conv outputs\nconv_outputs = interpretable_trainer.predict_step(dataloader)\nprint (conv_outputs.shape) # (num_filters, max_seq_len)\n
\n(50, 7)\n
# Visualize a bi-gram filter's outputs\ntokens = tokenizer.sequences_to_texts(sequences)[0].split(\" \")\nsns.heatmap(conv_outputs, xticklabels=tokens)\n
The filters have high values for the words stock and market which influenced the Business category classification.
Warning
This is a crude technique loosely based off of more elaborate interpretability methods.
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { CNNs - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/foundations/data-quality/","title":"Data Quality for Machine Learning","text":""},{"location":"courses/foundations/data-quality/#overview","title":"Overview","text":"
In a nutshell, a machine learning model consumes input data and produces predictions. The quality of the predictions directly corresponds to the quality of data you train the model with; garbage in, garbage out. Check out this article on where it makes sense to use AI and how to properly apply it.
We're going to go through all the concepts with concrete code examples and some synthesized data to train our models on. The task is to determine whether a tumor will be benign (harmless) or malignant (harmful) based on leukocyte (white blood cells) count and blood pressure. This is a synthetic dataset that we created and has no clinical relevance.
We want to choose features that have strong predictive signal for our task. If you want to improve performance, you need to continuously do feature engineering by collecting and adding new signals. So you may run into a new feature that has high correlation (orthogonal signal) with your existing features but it may still possess some unique signal to boost your predictive performance.
# Class weights\ncounts = np.bincount(y_train)\nclass_weights = {i: 1.0/count for i, count in enumerate(counts)}\nprint (f\"counts: {counts}\\nweights: {class_weights}\")\n
from sklearn.preprocessing import StandardScaler\n
# Standardize the data (mean=0, std=1) using training data\nX_scaler = StandardScaler().fit(X_train)\n
# Apply scaler on training and test data (don't standardize outputs for classification)\nX_train = X_scaler.transform(X_train)\nX_val = X_scaler.transform(X_val)\nX_test = X_scaler.transform(X_test)\n
# Check (means should be ~0 and std should be ~1)\nprint (f\"X_test[0]: mean: {np.mean(X_test[:, 0], axis=0):.1f}, std: {np.std(X_test[:, 0], axis=0):.1f}\")\nprint (f\"X_test[1]: mean: {np.mean(X_test[:, 1], axis=0):.1f}, std: {np.std(X_test[:, 1], axis=0):.1f}\")\n
We're going to plot a point, which we know belongs to the malignant tumor class. Our well trained model here would accurately predict that it is indeed a malignant tumor!
# Class weights\ncounts = np.bincount(y_train)\nclass_weights = {i: 1.0/count for i, count in enumerate(counts)}\nprint (f\"counts: {counts}\\nweights: {class_weights}\")\n
Now let's see how the same inference point from earlier performs now on the model trained on the reduced dataset.
# Visualize the decision boundary\nplt.figure(figsize=(8,5))\nplt.title(\"Test\")\nplot_multiclass_decision_boundary(model=model, X=X_test, y=y_test)\n\n# Sample point near the decision boundary (same point as before)\nplt.scatter(mean_leukocyte_count+0.05, mean_blood_pressure-0.05, s=200,\n c=\"b\", edgecolor=\"w\", linewidth=2)\n\n# Annotate\nplt.annotate(\"true: malignant,\\npred: benign\",\n color=\"white\",\n xy=(mean_leukocyte_count, mean_blood_pressure),\n xytext=(0.45, 0.60),\n textcoords=\"figure fraction\",\n fontsize=16,\n arrowprops=dict(facecolor=\"white\", shrink=0.1))\nplt.show()\n
This is a very fragile but highly realistic scenario. Based on our reduced synthetic dataset, we have achieved a model that generalized really well on the test data. But when we ask for the prediction for the same point tested earlier (which we known is malignant), the prediction is now a benign tumor. We would have completely missed the tumor. To mitigate this, we can:
Get more data around the space we are concerned about
Consume predictions with caution when they are close to the decision boundary
Models are not crystal balls. So it's important that before any machine learning, we really look at our data and ask ourselves if it is truly representative for the task we want to solve. The model itself may fit really well and generalize well on your data but if the data is of poor quality to begin with, the model cannot be trusted.
Once you are confident that your data is of good quality, you can finally start thinking about modeling. The type of model you choose depends on many factors, including the task, type of data, complexity required, etc.
So once you figure out what type of model your task needs, start with simple models and then slowly add complexity. You don\u2019t want to start with neural networks right away because that may not be right model for your data and task. Striking this balance in model complexity is one of the key tasks of your data scientists. simple models \u2192 complex models
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Data quality - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
While one-hot encoding allows us to preserve the structural information, it does poses two major disadvantages.
linearly dependent on the number of unique tokens in our vocabulary, which is a problem if we're dealing with a large corpus.
representation for each token does not preserve any relationship with respect to other tokens.
In this notebook, we're going to motivate the need for embeddings and how they address all the shortcomings of one-hot encoding. The main idea of embeddings is to have fixed length representations for the tokens in a text regardless of the number of tokens in the vocabulary. With one-hot encoding, each token is represented by an array of size vocab_size, but with embeddings, each token now has the shape embed_dim. The values in the representation will are not fixed binary values but rather, changing floating points allowing for fine-grained learned representations.
Objectives:
Represent tokens in text that capture the intrinsic semantic relationships.
Advantages:
Low-dimensionality while capturing relationships.
Interpretable token representations
Disadvantages:
Can be computationally intensive to precompute.
Miscellaneous:
There are lot's of pretrained embeddings to choose from but you can also train your own from scratch.
We can learn embeddings by creating our models in PyTorch but first, we're going to use a library that specializes in embeddings and topic modeling called Gensim.
import nltk\nnltk.download(\"punkt\");\nimport numpy as np\nimport re\nimport urllib\n
\n[nltk_data] Downloading package punkt to /root/nltk_data...\n[nltk_data] Unzipping tokenizers/punkt.zip.\n
SEED = 1234\n
# Set seed for reproducibility\nnp.random.seed(SEED)\n
# Split text into sentences\ntokenizer = nltk.data.load(\"tokenizers/punkt/english.pickle\")\nbook = urllib.request.urlopen(url=\"https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/harrypotter.txt\")\nsentences = tokenizer.tokenize(str(book.read()))\nprint (f\"{len(sentences)} sentences\")\n
\n12443 sentences\n
def preprocess(text):\n\"\"\"Conditional preprocessing on our text.\"\"\"\n # Lower\n text = text.lower()\n\n # Spacing and filters\n text = re.sub(r\"([-;;.,!?<=>])\", r\" \\1 \", text)\n text = re.sub(\"[^A-Za-z0-9]+\", \" \", text) # remove non alphanumeric chars\n text = re.sub(\" +\", \" \", text) # remove multiple spaces\n text = text.strip()\n\n # Separate into word tokens\n text = text.split(\" \")\n\n return text\n
# Preprocess sentences\nprint (sentences[11])\nsentences = [preprocess(sentence) for sentence in sentences]\nprint (sentences[11])\n
\nSnape nodded, but did not elaborate.\n['snape', 'nodded', 'but', 'did', 'not', 'elaborate']\n
But how do we learn the embeddings the first place? The intuition behind embeddings is that the definition of a token doesn't depend on the token itself but on its context. There are several different ways of doing this:
Given the word in the context, predict the target word (CBOW - continuous bag of words).
Given the target word, predict the context word (skip-gram).
Given a sequence of words, predict the next word (LM - language modeling).
All of these approaches involve create data to train our model on. Every word in a sentence becomes the target word and the context words are determines by a window. In the image below (skip-gram), the window size is 2 (2 words to the left and right of the target word). We repeat this for every sentence in our corpus and this results in our training data for the unsupervised task. This in an unsupervised learning technique since we don't have official labels for contexts. The idea is that similar target words will appear with similar contexts and we can learn this relationship by repeatedly training our mode with (context, target) pairs.
We can learn embeddings using any of these approaches above and some work better than others. You can inspect the learned embeddings but the best way to choose an approach is to empirically validate the performance on a supervised task.
When we have large vocabularies to learn embeddings for, things can get complex very quickly. Recall that the backpropagation with softmax updates both the correct and incorrect class weights. This becomes a massive computation for every backwards pass we do so a workaround is to use negative sampling which only updates the correct class and a few arbitrary incorrect classes (NEGATIVE_SAMPLING=20). We're able to do this because of the large amount of training data where we'll see the same word as the target class multiple times.
EMBEDDING_DIM = 100\nWINDOW = 5\nMIN_COUNT = 3 # Ignores all words with total frequency lower than this\nSKIP_GRAM = 1 # 0 = CBOW\nNEGATIVE_SAMPLING = 20\n
# Super fast because of optimized C code under the hood\nw2v = Word2Vec(\n sentences=sentences, size=EMBEDDING_DIM,\n window=WINDOW, min_count=MIN_COUNT,\n sg=SKIP_GRAM, negative=NEGATIVE_SAMPLING)\nprint (w2v)\n
\nWord2Vec(vocab=4937, size=100, alpha=0.025)\n
# Vector for each word\nw2v.wv.get_vector(\"potter\")\n
What happens when a word doesn't exist in our vocabulary? We could assign an UNK token which is used for all OOV (out of vocabulary) words or we could use FastText, which uses character-level n-grams to embed a word. This helps embed rare words, misspelled words, and also words that don't exist in our corpus but are similar to words in our corpus.
from gensim.models import FastText\n
# Super fast because of optimized C code under the hood\nft = FastText(sentences=sentences, size=EMBEDDING_DIM,\n window=WINDOW, min_count=MIN_COUNT,\n sg=SKIP_GRAM, negative=NEGATIVE_SAMPLING)\nprint (ft)\n
\nFastText(vocab=4937, size=100, alpha=0.025)\n
# This word doesn't exist so the word2vec model will error out\nw2v.wv.most_similar(positive=\"scarring\", topn=5)\n
# FastText will use n-grams to embed an OOV word\nft.wv.most_similar(positive=\"scarring\", topn=5)\n
We can learn embeddings from scratch using one of the approaches above but we can also leverage pretrained embeddings that have been trained on millions of documents. Popular ones include Word2Vec (skip-gram) or GloVe (global word-word co-occurrence). We can validate that these embeddings captured meaningful semantic relationships by confirming them.
from gensim.scripts.glove2word2vec import glove2word2vec\nfrom io import BytesIO\nimport matplotlib.pyplot as plt\nfrom sklearn.decomposition import PCA\nfrom urllib.request import urlopen\nfrom zipfile import ZipFile\n
# Arguments\nEMBEDDING_DIM = 100\n
def plot_embeddings(words, embeddings, pca_results):\n for word in words:\n index = embeddings.index2word.index(word)\n plt.scatter(pca_results[index, 0], pca_results[index, 1])\n plt.annotate(word, xy=(pca_results[index, 0], pca_results[index, 1]))\n plt.show()\n
# Unzip the file (may take ~3-5 minutes)\nresp = urlopen(\"http://nlp.stanford.edu/data/glove.6B.zip\")\nzipfile = ZipFile(BytesIO(resp.read()))\nzipfile.namelist()\n
# Save GloVe embeddings to local directory in word2vec format\nword2vec_output_file = \"{0}.word2vec\".format(embeddings_file)\nglove2word2vec(embeddings_file, word2vec_output_file)\n
\n(400000, 100)\n
# Load embeddings (may take a minute)\nglove = KeyedVectors.load_word2vec_format(word2vec_output_file, binary=False)\n
# (king - man) + woman = ?\n# king - man = ? - woman\nglove.most_similar(positive=[\"woman\", \"king\"], negative=[\"man\"], topn=5)\n
title category 0 Sharon Accepts Plan to Reduce Gaza Army Operation... World 1 Internet Key Battleground in Wildlife Crime Fight Sci/Tech 2 July Durable Good Orders Rise 1.7 Percent Business 3 Growing Signs of a Slowing on Wall Street Business 4 The New Faces of Reality TV World"},{"location":"courses/foundations/embeddings/#preprocessing","title":"Preprocessing","text":"
We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc.
\n[nltk_data] Downloading package stopwords to /root/nltk_data...\n[nltk_data] Package stopwords is already up-to-date!\n['i', 'me', 'my', 'myself', 'we']\n
def preprocess(text, stopwords=STOPWORDS):\n\"\"\"Conditional preprocessing on our text unique to our task.\"\"\"\n # Lower\n text = text.lower()\n\n # Remove stopwords\n pattern = re.compile(r\"\\b(\" + r\"|\".join(stopwords) + r\")\\b\\s*\")\n text = pattern.sub(\"\", text)\n\n # Remove words in parenthesis\n text = re.sub(r\"\\([^)]*\\)\", \"\", text)\n\n # Spacing and filters\n text = re.sub(r\"([-;;.,!?<=>])\", r\" \\1 \", text)\n text = re.sub(\"[^A-Za-z0-9]+\", \" \", text) # remove non alphanumeric chars\n text = re.sub(\" +\", \" \", text) # remove multiple spaces\n text = text.strip()\n\n return text\n
# Sample\ntext = \"Great week for the NYSE!\"\npreprocess(text=text)\n
\ngreat week nyse\n
# Apply to dataframe\npreprocessed_df = df.copy()\npreprocessed_df.title = preprocessed_df.title.apply(preprocess)\nprint (f\"{df.title.values[0]}\\n\\n{preprocessed_df.title.values[0]}\")\n
\nSharon Accepts Plan to Reduce Gaza Army Operation, Haaretz Says\n\nsharon accepts plan reduce gaza army operation haaretz says\n
Warning
If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before applying those operations. This is because we cannot apply any knowledge gained from the test set accidentally (data leak) during preprocessing/training. However for global preprocessing steps like the function above where we aren't learning anything from the data itself, we can perform before splitting the data.
# Class weights\ncounts = np.bincount(y_train)\nclass_weights = {i: 1.0/count for i, count in enumerate(counts)}\nprint (f\"counts: {counts}\\nweights: {class_weights}\")\n
class Tokenizer(object):\n def __init__(self, char_level, num_tokens=None,\n pad_token=\"<PAD>\", oov_token=\"<UNK>\",\n token_to_index=None):\n self.char_level = char_level\n self.separator = \"\" if self.char_level else \" \"\n if num_tokens: num_tokens -= 2 # pad + unk tokens\n self.num_tokens = num_tokens\n self.pad_token = pad_token\n self.oov_token = oov_token\n if not token_to_index:\n token_to_index = {pad_token: 0, oov_token: 1}\n self.token_to_index = token_to_index\n self.index_to_token = {v: k for k, v in self.token_to_index.items()}\n\n def __len__(self):\n return len(self.token_to_index)\n\n def __str__(self):\n return f\"<Tokenizer(num_tokens={len(self)})>\"\n\n def fit_on_texts(self, texts):\n if not self.char_level:\n texts = [text.split(\" \") for text in texts]\n all_tokens = [token for text in texts for token in text]\n counts = Counter(all_tokens).most_common(self.num_tokens)\n self.min_token_freq = counts[-1][1]\n for token, count in counts:\n index = len(self)\n self.token_to_index[token] = index\n self.index_to_token[index] = token\n return self\n\n def texts_to_sequences(self, texts):\n sequences = []\n for text in texts:\n if not self.char_level:\n text = text.split(\" \")\n sequence = []\n for token in text:\n sequence.append(self.token_to_index.get(\n token, self.token_to_index[self.oov_token]))\n sequences.append(np.asarray(sequence))\n return sequences\n\n def sequences_to_texts(self, sequences):\n texts = []\n for sequence in sequences:\n text = []\n for index in sequence:\n text.append(self.index_to_token.get(index, self.oov_token))\n texts.append(self.separator.join([token for token in text]))\n return texts\n\n def save(self, fp):\n with open(fp, \"w\") as fp:\n contents = {\n \"char_level\": self.char_level,\n \"oov_token\": self.oov_token,\n \"token_to_index\": self.token_to_index\n }\n json.dump(contents, fp, indent=4, sort_keys=False)\n\n @classmethod\n def load(cls, fp):\n with open(fp, \"r\") as fp:\n kwargs = json.load(fp=fp)\n return cls(**kwargs)\n
Warning
It's important that we only fit using our train data split because during inference, our model will not always know every token so it's important to replicate that scenario with our validation and test splits as well.
# Sample of tokens\nprint (take(5, tokenizer.token_to_index.items()))\nprint (f\"least freq token's freq: {tokenizer.min_token_freq}\") # use this to adjust num_tokens\n
Each token in the input is represented via embeddings (all out-of-vocabulary (OOV) tokens are given the embedding for UNK token.) In the model below, we'll see how to set these embeddings to be pretrained GloVe embeddings and how to choose whether to freeze (fixed embedding weights) those embeddings or not during training.
Our inputs are all of varying length but we need each batch to be uniformly shaped. Therefore, we will use padding to make all the inputs in the batch the same length. Our padding index will be 0 (note that this is consistent with the <PAD> token defined in our Tokenizer).
While embedding our input tokens will create a batch of shape (N, max_seq_len, embed_dim) we only need to provide a 2D matrix (N, max_seq_len) for using embeddings with PyTorch.
def pad_sequences(sequences, max_seq_len=0):\n\"\"\"Pad sequences to max length in sequence.\"\"\"\n max_seq_len = max(max_seq_len, max(len(sequence) for sequence in sequences))\n padded_sequences = np.zeros((len(sequences), max_seq_len))\n for i, sequence in enumerate(sequences):\n padded_sequences[i][:len(sequence)] = sequence\n return padded_sequences\n
We'll be using a convolutional neural network on top of our embedded tokens to extract meaningful spatial signal. This time, we'll be using many filter widths to act as n-gram feature extractors.
Let's visualize the model's forward pass.
We'll first tokenize our inputs (batch_size, max_seq_len).
Then we'll embed our tokenized inputs (batch_size, max_seq_len, embedding_dim).
We'll apply convolution via filters (filter_size, embedding_dim, num_filters) followed by batch normalization. Our filters act as character level n-gram detectors. We have three different filter sizes (2, 3 and 4) and they will act as bi-gram, tri-gram and 4-gram feature extractors, respectively.
We'll apply 1D global max pooling which will extract the most relevant information from the feature maps for making the decision.
We feed the pool outputs to a fully-connected (FC) layer (with dropout).
We use one more FC layer with softmax to derive class probabilities.
We're going create some utility functions to be able to load the pretrained GloVe embeddings into our Embeddings layer.
def load_glove_embeddings(embeddings_file):\n\"\"\"Load embeddings from a file.\"\"\"\n embeddings = {}\n with open(embeddings_file, \"r\") as fp:\n for index, line in enumerate(fp):\n values = line.split()\n word = values[0]\n embedding = np.asarray(values[1:], dtype='float32')\n embeddings[word] = embedding\n return embeddings\n
def make_embeddings_matrix(embeddings, word_index, embedding_dim):\n\"\"\"Create embeddings matrix to use in Embedding layer.\"\"\"\n embedding_matrix = np.zeros((len(word_index), embedding_dim))\n for word, i in word_index.items():\n embedding_vector = embeddings.get(word)\n if embedding_vector is not None:\n embedding_matrix[i] = embedding_vector\n return embedding_matrix\n
We have first have to decide whether to use pretrained embeddings randomly initialized ones. Then, we can choose to freeze our embeddings or continue to train them using the supervised data (this could lead to overfitting). Here are the three experiments we're going to conduct:
def get_probability_distribution(y_prob, classes):\n\"\"\"Create a dict of class probabilities from an array.\"\"\"\n results = {}\n for i, class_ in enumerate(classes):\n results[class_] = np.float64(y_prob[i])\n sorted_results = {k: v for k, v in sorted(\n results.items(), key=lambda item: item[1], reverse=True)}\n return sorted_results\n
We went through all the trouble of padding our inputs before convolution to result is outputs of the same shape as our inputs so we can try to get some interpretability. Since every token is mapped to a convolutional output on which we apply max pooling, we can see which token's output was most influential towards the prediction. We first need to get the conv outputs from our model:
import collections\nimport seaborn as sns\n
class InterpretableCNN(nn.Module):\n def __init__(self, embedding_dim, vocab_size, num_filters,\n filter_sizes, hidden_dim, dropout_p, num_classes,\n pretrained_embeddings=None, freeze_embeddings=False,\n padding_idx=0):\n super(InterpretableCNN, self).__init__()\n\n # Filter sizes\n self.filter_sizes = filter_sizes\n\n # Initialize embeddings\n if pretrained_embeddings is None:\n self.embeddings = nn.Embedding(\n embedding_dim=embedding_dim, num_embeddings=vocab_size,\n padding_idx=padding_idx)\n else:\n pretrained_embeddings = torch.from_numpy(pretrained_embeddings).float()\n self.embeddings = nn.Embedding(\n embedding_dim=embedding_dim, num_embeddings=vocab_size,\n padding_idx=padding_idx, _weight=pretrained_embeddings)\n\n # Freeze embeddings or not\n if freeze_embeddings:\n self.embeddings.weight.requires_grad = False\n\n # Conv weights\n self.conv = nn.ModuleList(\n [nn.Conv1d(in_channels=embedding_dim,\n out_channels=num_filters,\n kernel_size=f) for f in filter_sizes])\n\n # FC weights\n self.dropout = nn.Dropout(dropout_p)\n self.fc1 = nn.Linear(num_filters*len(filter_sizes), hidden_dim)\n self.fc2 = nn.Linear(hidden_dim, num_classes)\n\n def forward(self, inputs, channel_first=False):\n\n # Embed\n x_in, = inputs\n x_in = self.embeddings(x_in)\n\n # Rearrange input so num_channels is in dim 1 (N, C, L)\n if not channel_first:\n x_in = x_in.transpose(1, 2)\n\n # Conv outputs\n z = []\n max_seq_len = x_in.shape[2]\n for i, f in enumerate(self.filter_sizes):\n # `SAME` padding\n padding_left = int((self.conv[i].stride[0]*(max_seq_len-1) - max_seq_len + self.filter_sizes[i])/2)\n padding_right = int(math.ceil((self.conv[i].stride[0]*(max_seq_len-1) - max_seq_len + self.filter_sizes[i])/2))\n\n # Conv + pool\n _z = self.conv[i](F.pad(x_in, (padding_left, padding_right)))\n z.append(_z.cpu().numpy())\n\n return z\n
1D global max-pooling would extract the highest value from each of our num_filters for each filter_size. We could also follow this same approach to figure out which n-gram is most relevant but notice in the heatmap above that many filters don't have much variance. To mitigate this, this paper uses threshold values to determine which filters to use for interpretability. But to keep things simple, let's extract which tokens' filter outputs were extracted via max-pooling the most frequently.
sample_index = 0\nprint (f\"Original text:\\n{text}\")\nprint (f\"\\nPreprocessed text:\\n{tokenizer.sequences_to_texts(X)[0]}\")\nprint (\"\\nMost important n-grams:\")\n# Process conv outputs for each unique filter size\nfor i, filter_size in enumerate(FILTER_SIZES):\n\n # Identify most important n-gram (excluding last token)\n popular_indices = collections.Counter([np.argmax(conv_output) \\\n for conv_output in conv_outputs[i]])\n\n # Get corresponding text\n start = popular_indices.most_common(1)[-1][0]\n n_gram = \" \".join([token for token in tokens[start:start+filter_size]])\n print (f\"[{filter_size}-gram]: {n_gram}\")\n
\nOriginal text:\nThe final tennis tournament starts next week.\n\nPreprocessed text:\nfinal tennis tournament starts next week\n\nMost important n-grams:\n[1-gram]: tennis\n[2-gram]: tennis tournament\n[3-gram]: final tennis tournament\n
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Embeddings - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
Use inputs \\(X\\) to predict the output \\(\\hat{y}\\) using a linear model. The model will be a line of best fit that minimizes the distance between the predicted (model's output) and target (ground truth) values. Training data \\((X, y)\\) is used to train the model and learn the weights \\(W\\) using gradient descent.
Advantages:
Computationally simple.
Highly interpretable.
Can account for continuous and categorical features.
Disadvantages:
The model will perform well only when the data is linearly separable (for classification).
Miscellaneous:
You can also use linear regression for binary classification tasks where if the predicted continuous value is above a threshold, it belongs to a certain class. But we will cover better techniques for classification in future lessons and will focus on linear regression for continuous regression tasks only.
We're going to generate some simple dummy data to apply linear regression on. It's going to create roughly linear data (y = 3.5X + noise); the random noise is added to create realistic data that doesn't perfectly align in a line. Our goal is to have the model converge to a similar linear equation (there will be slight variance since we added some noise).
import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n
SEED = 1234\nNUM_SAMPLES = 50\n
# Set seed for reproducibility\nnp.random.seed(SEED)\n
# Generate synthetic data\ndef generate_data(num_samples):\n\"\"\"Generate dummy data for linear regression.\"\"\"\n X = np.array(range(num_samples))\n random_noise = np.random.uniform(-10, 20, size=num_samples)\n y = 3.5*X + random_noise # add some noise\n return X, y\n
# Generate random (linear) data\nX, y = generate_data(num_samples=NUM_SAMPLES)\ndata = np.vstack([X, y]).T\nprint (data[:5])\n
Now that we have our data prepared, we'll first implement linear regression using just NumPy. This will let us really understand the underlying operations.
# Determine means and stds\nX_mean = np.mean(X_train)\nX_std = np.std(X_train)\ny_mean = np.mean(y_train)\ny_std = np.std(y_train)\n
We need to treat the validation and test sets as if they were hidden datasets. So we only use the train set to determine the mean and std to avoid biasing our training process.
# Check (means should be ~0 and std should be ~1)\n# Check (means should be ~0 and std should be ~1)\nprint (f\"mean: {np.mean(X_test, axis=0)[0]:.1f}, std: {np.std(X_test, axis=0)[0]:.1f}\")\nprint (f\"mean: {np.mean(y_test, axis=0)[0]:.1f}, std: {np.std(y_test, axis=0)[0]:.1f}\")\n
Our goal is to learn a linear model \\(\\hat{y}\\) that models \\(y\\) given \\(X\\) using weights \\(W\\) and bias \\(b\\) \u2192 \\(\\hat{y} = XW + b\\)
Step 1: Randomly initialize the model's weights \\(W\\).
INPUT_DIM = X_train.shape[1] # X is 1-dimensional\nOUTPUT_DIM = y_train.shape[1] # y is 1-dimensional\n
Step 3: Compare the predictions \\(\\hat{y}\\) with the actual target values \\(y\\) using the objective (cost) function to determine the loss \\(J\\). A common objective function for linear regression is mean squared error (MSE). This function calculates the difference between the predicted and target values and squares it.
The gradient is the derivative, or the rate of change of a function. It's a vector that points in the direction of greatest increase of a function. For example the gradient of our loss function (\\(J\\)) with respect to our weights (\\(W\\)) will tell us how to change \\(W\\) so we can maximize \\(J\\). However, we want to minimize our loss so we subtract the gradient from \\(W\\).
The learning rate \\(\\alpha\\) is a way to control how much we update the weights by. If we choose a small learning rate, it may take a long time for our model to train. However, if we choose a large learning rate, we may overshoot and our training will never converge. The specific learning rate depends on our data and the type of models we use but it's typically good to explore in the range of \\([1e^{-8}, 1e^{-1}]\\). We'll explore learning rate update strategies in later lessons.
To keep the code simple, we're not calculating and displaying the validation loss after each epoch here. But in later lessons, the performance on the validation set will be crucial in influencing the learning process (learning rate, when to stop training, etc.).
Now we're ready to see how well our trained model will perform on our test (hold-out) data split. This will be our best measure on how well the model would perform on the real world, given that our dataset's distribution is close to unseen data.
Since we standardized our inputs and outputs, our weights were fit to those standardized values. So we need to unstandardize our weights so we can compare it to our true weight (3.5).
Note that both \\(X\\) and \\(y\\) were standardized.
This time we'll use scikit-learn's StandardScaler to standardize our data.
\n
from sklearn.preprocessing import StandardScaler\n
\n
# Standardize the data (mean=0, std=1) using training data\nX_scaler = StandardScaler().fit(X_train)\ny_scaler = StandardScaler().fit(y_train)\n
\n
# Apply scaler on training and test data\nX_train = X_scaler.transform(X_train)\ny_train = y_scaler.transform(y_train).ravel().reshape(-1, 1)\nX_val = X_scaler.transform(X_val)\ny_val = y_scaler.transform(y_val).ravel().reshape(-1, 1)\nX_test = X_scaler.transform(X_test)\ny_test = y_scaler.transform(y_test).ravel().reshape(-1, 1)\n
\n
# Check (means should be ~0 and std should be ~1)\nprint (f\"mean: {np.mean(X_test, axis=0)[0]:.1f}, std: {np.std(X_test, axis=0)[0]:.1f}\")\nprint (f\"mean: {np.mean(y_test, axis=0)[0]:.1f}, std: {np.std(y_test, axis=0)[0]:.1f}\")\n
When we implemented linear regression with just NumPy, we used batch gradient descent to update our weights (used entire training set). But there are actually many different gradient descent optimization algorithms to choose from and it depends on the situation. However, the ADAM optimizer has become a standard algorithm for most cases.
Linear regression offers the great advantage of being highly interpretable. Each feature has a coefficient which signifies its importance/impact on the output variable y. We can interpret our coefficient as follows: by increasing X by 1 unit, we increase y by \\(W\\) (~3.65) units.\n
Regularization helps decrease overfitting. Below is L2 regularization (ridge regression). There are many forms of regularization but they all work to reduce overfitting in our models. With L2 regularization, we are penalizing large weight values by decaying them because having large weights will lead to preferential bias with the respective inputs and we want the model to work with all the inputs and not just a select few. There are also other types of regularization like L1 (lasso regression) which is useful for creating sparse models where some feature coefficients are zeroed out, or elastic which combines L1 and L2 penalties.
\n
Regularization is not just for linear regression. You can use it to regularize any model's weights including the ones we will look at in future lessons.
\n\\[ J(\\theta) = \\frac{1}{2}\\sum_{i}(X_iW - y_i)^2 + \\frac{\\lambda}{2}W^TW \\]\n\\[ \\frac{\\partial{J}}{\\partial{W}} = X (\\hat{y} - y) + \\lambda W \\]\n\\[ W = W - \\alpha\\frac{\\partial{J}}{\\partial{W}} \\]\n
Regularization didn't make a difference in performance with this specific example because our data is generated from a perfect linear equation but for large realistic data, regularization can help our model generalize well.
\n
To cite this content, please use:
\n
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Linear regression - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
Logistic regression is an extension on linear regression (both are generalized linear methods). We will still learn to model a line (plane) that models \\(y\\) given \\(X\\). Except now we are dealing with classification problems as opposed to regression problems so we'll be predicting probability distributions as opposed to discrete values. We'll be using the softmax operation to normalize our logits (\\(XW\\)) to derive probabilities.
Our goal is to learn a logistic model \\(\\hat{y}\\) that models \\(y\\) given \\(X\\).
Variable Description \\(N\\) total numbers of samples \\(C\\) number of classes \\(\\hat{y}\\) predictions \\(\\in \\mathbb{R}^{NXC}\\) \\(X\\) inputs \\(\\in \\mathbb{R}^{NXD}\\) \\(W\\) weights \\(\\in \\mathbb{R}^{DXC}\\)
(*) bias term (\\(b\\)) excluded to avoid crowding the notations
This function is known as the multinomial logistic regression or the softmax classifier. The softmax classifier will use the linear equation (\\(z=XW\\)) and normalize it (using the softmax function) to produce the probability for class y given the inputs.
Objectives:
Predict the probability of class \\(y\\) given the inputs \\(X\\). The softmax classifier normalizes the linear outputs to determine class probabilities.
Advantages:
Can predict class probabilities given a set on inputs.
Disadvantages:
Sensitive to outliers since objective is to minimize cross entropy loss. Support vector machines (SVMs) are a good alternative to counter outliers.
Miscellaneous:
Softmax classifier is widely in neural network architectures as the last layer since it produces class probabilities.
We'll used some synthesized data to train our models on. The task is to determine whether a tumor will be benign (harmless) or malignant (harmful) based on leukocyte (white blood cells) count and blood pressure. Note that this is a synthetic dataset that has no clinical relevance.
import matplotlib.pyplot as plt\nimport pandas as pd\nfrom pandas.plotting import scatter_matrix\n
SEED = 1234\n
# Set seed for reproducibility\nnp.random.seed(SEED)\n
# Read from CSV to Pandas DataFrame\nurl = \"https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/tumors.csv\"\ndf = pd.read_csv(url, header=0) # load\ndf = df.sample(frac=1).reset_index(drop=True) # shuffle\ndf.head()\n
We want to split our dataset so that each of the three splits has the same distribution of classes so that we can train and evaluate properly. We can easily achieve this by telling scikit-learn's train_test_split function what to stratify on.
You'll notice that our class labels are text. We need to encode them into integers so we can use them in our models. We could scikit-learn's LabelEncoder to do this but we're going to write our own simple label encoder class so we can see what's happening under the hood.
import itertools\n
class LabelEncoder(object):\n\"\"\"Label encoder for tag labels.\"\"\"\n def __init__(self, class_to_index={}):\n self.class_to_index = class_to_index or {} # mutable defaults ;)\n self.index_to_class = {v: k for k, v in self.class_to_index.items()}\n self.classes = list(self.class_to_index.keys())\n\n def __len__(self):\n return len(self.class_to_index)\n\n def __str__(self):\n return f\"<LabelEncoder(num_classes={len(self)})>\"\n\n def fit(self, y):\n classes = np.unique(y)\n for i, class_ in enumerate(classes):\n self.class_to_index[class_] = i\n self.index_to_class = {v: k for k, v in self.class_to_index.items()}\n self.classes = list(self.class_to_index.keys())\n return self\n\n def encode(self, y):\n encoded = np.zeros((len(y)), dtype=int)\n for i, item in enumerate(y):\n encoded[i] = self.class_to_index[item]\n return encoded\n\n def decode(self, y):\n classes = []\n for i, item in enumerate(y):\n classes.append(self.index_to_class[item])\n return classes\n\n def save(self, fp):\n with open(fp, \"w\") as fp:\n contents = {'class_to_index': self.class_to_index}\n json.dump(contents, fp, indent=4, sort_keys=False)\n\n @classmethod\n def load(cls, fp):\n with open(fp, \"r\") as fp:\n kwargs = json.load(fp=fp)\n return cls(**kwargs)\n
We also want to calculate our class weights, which are useful for weighting the loss function during training. It tells the model to focus on samples from an under-represented class. The loss section below will show how to incorporate these weights.
# Class weights\ncounts = np.bincount(y_train)\nclass_weights = {i: 1.0/count for i, count in enumerate(counts)}\nprint (f\"counts: {counts}\\nweights: {class_weights}\")\n
We need to standardize our data (zero mean and unit variance) so a specific feature's magnitude doesn't affect how the model learns its weights. We're only going to standardize the inputs X because our outputs y are class values.
from sklearn.preprocessing import StandardScaler\n
# Standardize the data (mean=0, std=1) using training data\nX_scaler = StandardScaler().fit(X_train)\n
# Apply scaler on training and test data (don't standardize outputs for classification)\nX_train = X_scaler.transform(X_train)\nX_val = X_scaler.transform(X_val)\nX_test = X_scaler.transform(X_test)\n
# Check (means should be ~0 and std should be ~1)\nprint (f\"X_test[0]: mean: {np.mean(X_test[:, 0], axis=0):.1f}, std: {np.std(X_test[:, 0], axis=0):.1f}\")\nprint (f\"X_test[1]: mean: {np.mean(X_test[:, 1], axis=0):.1f}, std: {np.std(X_test[:, 1], axis=0):.1f}\")\n
Now that we have our data prepared, we'll first implement logistic regression using just NumPy. This will let us really understand the underlying operations. It's normal to find the math and code in this section slightly complex. You can still read each of the steps to build intuition for when we implement this using PyTorch.
Our goal is to learn a logistic model \\(\\hat{y}\\) that models \\(y\\) given \\(X\\).
We are going to use multinomial logistic regression even though our task only involves two classes because you can generalize the softmax classifier to any number of classes.
Step 2: Feed inputs \\(X\\) into the model to receive the logits (\\(z=XW\\)). Apply the softmax operation on the logits to get the class probabilities \\(\\hat{y}\\) in one-hot encoded form. For example, if there are three classes, the predicted class probabilities could look like [0.3, 0.3, 0.4].
Step 3: Compare the predictions \\(\\hat{y}\\) (ex. [0.3, 0.3, 0.4]) with the actual target values \\(y\\) (ex. class 2 would look like [0, 0, 1]) with the objective (cost) function to determine loss \\(J\\). A common objective function for logistics regression is cross-entropy loss.
Step 4: Calculate the gradient of loss \\(J(\\theta)\\) w.r.t to the model weights. Let's assume that our classes are mutually exclusive (a set of inputs could only belong to one class).
Step 5: Update the weights \\(W\\) using a small learning rate \\(\\alpha\\). The updates will penalize the probability for the incorrect classes (j) and encourage a higher probability for the correct class (y).
# Training and test accuracy\ntrain_acc = np.mean(np.equal(y_train, pred_train))\ntest_acc = np.mean(np.equal(y_test, pred_test))\nprint (f\"train acc: {train_acc:.2f}, test acc: {test_acc:.2f}\")\n
\ntrain acc: 0.98, test acc: 0.94\n
def plot_multiclass_decision_boundary(model, X, y, savefig_fp=None):\n\"\"\"Plot the multiclass decision boundary for a model that accepts 2D inputs.\n Credit: https://cs231n.github.io/neural-networks-case-study/\n\n Arguments:\n model {function} -- trained model with function model.predict(x_in).\n X {numpy.ndarray} -- 2D inputs with shape (N, 2).\n y {numpy.ndarray} -- 1D outputs with shape (N,).\n \"\"\"\n # Axis boundaries\n x_min, x_max = X[:, 0].min() - 0.1, X[:, 0].max() + 0.1\n y_min, y_max = X[:, 1].min() - 0.1, X[:, 1].max() + 0.1\n xx, yy = np.meshgrid(np.linspace(x_min, x_max, 101),\n np.linspace(y_min, y_max, 101))\n\n # Create predictions\n x_in = np.c_[xx.ravel(), yy.ravel()]\n y_pred = model.predict(x_in)\n y_pred = np.argmax(y_pred, axis=1).reshape(xx.shape)\n\n # Plot decision boundary\n plt.contourf(xx, yy, y_pred, cmap=plt.cm.Spectral, alpha=0.8)\n plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.RdYlBu)\n plt.xlim(xx.min(), xx.max())\n plt.ylim(yy.min(), yy.max())\n\n # Plot\n if savefig_fp:\n plt.savefig(savefig_fp, format=\"png\")\n
We'll compute accuracy as we train our model because just looking the loss value isn't super intuitive to look at. We'll look at other metrics (precision, recall, f1) in the evaluation section below.
# Accuracy (could've also used our own accuracy function)\ntrain_acc = accuracy_score(y_train, pred_train)\ntest_acc = accuracy_score(y_test, pred_test)\nprint (f\"train acc: {train_acc:.2f}, test acc: {test_acc:.2f}\")\n
\ntrain acc: 0.98, test acc: 0.98\n
We can also evaluate our model on other meaningful metrics such as precision and recall. These are especially useful when there is data imbalance present.
Variable Description \\(TP\\) # of samples truly predicted to be positive and were positive \\(TN\\) # of samples truly predicted to be negative and were negative \\(FP\\) # of samples falsely predicted to be positive but were negative \\(FN\\) # of samples falsely predicted to be negative but were positive
import json\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import precision_recall_fscore_support\n
# Predict\ny_infer = F.softmax(model(torch.Tensor(X_infer)), dim=1)\nprob, _class = y_infer.max(dim=1)\nlabel = label_encoder.decode(_class.detach().numpy())[0]\nprint (f\"The probability that you have a {label} tumor is {prob.detach().numpy()[0]*100.0:.0f}%\")\n
\nThe probability that you have a benign tumor is 93%\n
Our goal is to learn a model \\(\\hat{y}\\) that models \\(y\\) given \\(X\\) . You'll notice that neural networks are just extensions of the generalized linear methods we've seen so far but with non-linear activation functions since our data will be highly non-linear.
Variable Description \\(N\\) total numbers of samples \\(D\\) number of features \\(H\\) number of hidden units \\(C\\) number of classes \\(W_1\\) 1st layer weights \\(\\in \\mathbb{R}^{DXH}\\) \\(z_1\\) outputs from first layer \\(\\in \\mathbb{R}^{NXH}\\) \\(f\\) non-linear activation function \\(a_1\\) activations from first layer \\(\\in \\mathbb{R}^{NXH}\\) \\(W_2\\) 2nd layer weights \\(\\in \\mathbb{R}^{HXC}\\) \\(z_2\\) outputs from second layer \\(\\in \\mathbb{R}^{NXC}\\) \\(\\hat{y}\\) prediction \\(\\in \\mathbb{R}^{NXC}\\)
(*) bias term (\\(b\\)) excluded to avoid crowding the notations
Objective:
Predict the probability of class \\(y\\) given the inputs \\(X\\). Non-linearity is introduced to model the complex, non-linear data.
Advantages:
Can model non-linear patterns in the data really well.
Disadvantages:
Overfits easily.
Computationally intensive as network increases in size.
Not easily interpretable.
Miscellaneous:
Future neural network architectures that we'll see use the MLP as a modular unit for feed forward operations (affine transformation (XW) followed by a non-linear operation).
In the previous lesson we wrote our own label encoder class to see the inner functions but this time we'll use scikit-learn LabelEncoder class which does the same operations as ours.
# Class weights\ncounts = np.bincount(y_train)\nclass_weights = {i: 1.0/count for i, count in enumerate(counts)}\nprint (f\"counts: {counts}\\nweights: {class_weights}\")\n
We need to standardize our data (zero mean and unit variance) so a specific feature's magnitude doesn't affect how the model learns its weights. We're only going to standardize the inputs X because our outputs y are class values.
from sklearn.preprocessing import StandardScaler\n
# Standardize the data (mean=0, std=1) using training data\nX_scaler = StandardScaler().fit(X_train)\n
# Apply scaler on training and test data (don't standardize outputs for classification)\nX_train = X_scaler.transform(X_train)\nX_val = X_scaler.transform(X_val)\nX_test = X_scaler.transform(X_test)\n
# Check (means should be ~0 and std should be ~1)\nprint (f\"X_test[0]: mean: {np.mean(X_test[:, 0], axis=0):.1f}, std: {np.std(X_test[:, 0], axis=0):.1f}\")\nprint (f\"X_test[1]: mean: {np.mean(X_test[:, 1], axis=0):.1f}, std: {np.std(X_test[:, 1], axis=0):.1f}\")\n
Before we get to our neural network, we're going to motivate non-linear activation functions by implementing a generalized linear model (logistic regression). We'll see why linear models (with linear activations) won't suffice for our dataset.
import torch\n
# Set seed for reproducibility\ntorch.manual_seed(SEED)\n
Using the generalized linear method (logistic regression) yielded poor results because of the non-linearity present in our data yet our activation functions were linear. We need to use an activation function that can allow our model to learn and map the non-linearity in our data. There are many different options so let's explore a few.
# Fig size\nplt.figure(figsize=(12,3))\n\n# Data\nx = torch.arange(-5., 5., 0.1)\n\n# Sigmoid activation (constrain a value between 0 and 1.)\nplt.subplot(1, 3, 1)\nplt.title(\"Sigmoid activation\")\ny = torch.sigmoid(x)\nplt.plot(x.numpy(), y.numpy())\n\n# Tanh activation (constrain a value between -1 and 1.)\nplt.subplot(1, 3, 2)\ny = torch.tanh(x)\nplt.title(\"Tanh activation\")\nplt.plot(x.numpy(), y.numpy())\n\n# Relu (clip the negative values to 0)\nplt.subplot(1, 3, 3)\ny = F.relu(x)\nplt.title(\"ReLU activation\")\nplt.plot(x.numpy(), y.numpy())\n\n# Show plots\nplt.show()\n
The ReLU activation function (\\(max(0,z)\\)) is by far the most widely used activation function for neural networks. But as you can see, each activation function has its own constraints so there are circumstances where you'll want to use different ones. For example, if we need to constrain our outputs between 0 and 1, then the sigmoid activation is the best choice.
In some cases, using a ReLU activation function may not be sufficient. For instance, when the outputs from our neurons are mostly negative, the activation function will produce zeros. This effectively creates a \"dying ReLU\" and a recovery is unlikely. To mitigate this effect, we could lower the learning rate or use alternative ReLU activations, ex. leaky ReLU or parametric ReLU (PReLU), which have a small slope for negative neuron outputs.
Now let's create our multilayer perceptron (MLP) which is going to be exactly like the logistic regression model but with the activation function to map the non-linearity in our data.
It's normal to find the math and code in this section slightly complex. You can still read each of the steps to build intuition for when we implement this using PyTorch.
Our goal is to learn a model \\(\\hat{y}\\) that models \\(y\\) given \\(X\\). You'll notice that neural networks are just extensions of the generalized linear methods we've seen so far but with non-linear activation functions since our data will be highly non-linear.
Step 3: Compare the predictions \\(\\hat{y}\\) (ex. [0.3, 0.3, 0.4]) with the actual target values \\(y\\) (ex. class 2 would look like [0, 0, 1]) with the objective (cost) function to determine loss \\(J\\). A common objective function for classification tasks is cross-entropy loss.
Step 5: Update the weights \\(W\\) using a small learning rate \\(\\alpha\\). The updates will penalize the probability for the incorrect classes (\\(j\\)) and encourage a higher probability for the correct class (\\(y\\)).
# Predict\ny_infer = F.softmax(model(torch.Tensor(X_infer)), dim=1)\nprob, _class = y_infer.max(dim=1)\nlabel = label_encoder.inverse_transform(_class.detach().numpy())[0]\nprint (f\"The probability that you have {label} is {prob.detach().numpy()[0]*100.0:.0f}%\")\n
So far we have been initializing weights with small random values but this isn't optimal for convergence during training. The objective is to initialize the appropriate weights such that our activations (outputs of layers) don't vanish (too small) or explode (too large), as either of these situations will hinder convergence. We can do this by sampling the weights uniformly from a bound distribution (many that take into account the precise activation function used) such that all activations have unit variance.
You may be wondering why we don't do this for every forward pass and that's a great question. We'll look at more advanced strategies that help with optimization like batch normalization, etc. in future lessons. Meanwhile you can check out other initializers here.
from torch.nn import init\n
class MLP(nn.Module):\n def __init__(self, input_dim, hidden_dim, num_classes):\n super(MLP, self).__init__()\n self.fc1 = nn.Linear(input_dim, hidden_dim)\n self.fc2 = nn.Linear(hidden_dim, num_classes)\n\n def init_weights(self):\n init.xavier_normal(self.fc1.weight, gain=init.calculate_gain(\"relu\"))\n\n def forward(self, x_in):\n z = F.relu(self.fc1(x_in)) # ReLU activation function added!\n z = self.fc2(z)\n return z\n
A great technique to have our models generalize (perform well on test data) is to increase the size of your data but this isn't always an option. Fortunately, there are methods like regularization and dropout that can help create a more robust model.
Dropout is a technique (used only during training) that allows us to zero the outputs of neurons. We do this for dropout_p% of the total neurons in each layer and it changes every batch. Dropout prevents units from co-adapting too much to the data and acts as a sampling strategy since we drop a different set of neurons each time.
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
DROPOUT_P = 0.1 # percentage of weights that are dropped each pass\n
Though neural networks are great at capturing non-linear relationships they are highly susceptible to overfitting to the training data and failing to generalize on test data. Just take a look at the example below where we generate completely random data and are able to fit a model with \\(2*N*C + D\\) (where N = # of samples, C = # of classes and D = input dimension) hidden units. The training performance is good (~70%) but the overfitting leads to very poor test performance. We'll be covering strategies to tackle overfitting in future lessons.
It's important that we experiment, starting with simple models that underfit (high bias) and improve it towards a good fit. Starting with simple models (linear/logistic regression) let's us catch errors without the added complexity of more sophisticated models (neural networks).
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Neural networks - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/foundations/notebooks/","title":"Working in Notebooks","text":""},{"location":"courses/foundations/notebooks/#set-up","title":"Set up","text":"
Click on this link to open the accompanying notebook for this lesson or create a blank one on Google Colab.
Sign into your Google account to start using the notebook. If you don't want to save your work, you can skip the steps below. If you do not have access to Google, you can follow along using Jupyter Lab.
If you do want to save your work, click the COPY TO DRIVE button on the toolbar. This will open a new notebook in a new tab. Rename this new notebook by removing the words Copy of from the title (change Copy of 01_Notebooks to 1_Notebooks).
Alternatives to Google Colab
Alternatively, you can run these notebooks locally by using JupyterLab. You should first set up a directory for our project, create a virtual environment and install jupyterlab.
Click on a desired location in the notebook and create the cell by clicking on the \u2795 TEXT (located in the top left corner).
Once you create the cell, click on it and type the following text inside it:
### This is a header\nHello world!\n
"},{"location":"courses/foundations/notebooks/#run-a-cell","title":"Run a cell","text":"
Once you type inside the cell, press the SHIFT and RETURN (enter key) together to run the cell.
"},{"location":"courses/foundations/notebooks/#edit-a-cell","title":"Edit a cell","text":"
To edit a cell, double click on it and make any changes.
"},{"location":"courses/foundations/notebooks/#move-a-cell","title":"Move a cell","text":"
Move a cell up and down by clicking on the cell and then pressing the \u2b06 and \u2b07 button on the top right of the cell.
"},{"location":"courses/foundations/notebooks/#delete-a-cell","title":"Delete a cell","text":"
Delete the cell by clicking on it and pressing the trash can button \ud83d\uddd1\ufe0f on the top right corner of the cell. Alternatively, you can also press \u2318/Ctrl + M + D.
Repeat the steps above to create and edit a code cell. You can create a code cell by clicking on the \u2795 CODE (located in the top left corner).
Once you've created the code cell, double click on it, type the following inside it and then press Shift + Enter to execute the code.
print (\"Hello world!\")\n
\nHello world!\n
These are the basic concepts we'll need to use these notebooks but we'll learn few more tricks in subsequent lessons.
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Notebooks - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/foundations/numpy/","title":"NumPy for Machine Learning","text":""},{"location":"courses/foundations/numpy/#set-up","title":"Set up","text":"
First we'll import the NumPy package and set seeds for reproducibility so that we can receive the exact same results every time.
import numpy as np\n
# Set seed for reproducibility\nnp.random.seed(seed=1234)\n
We can extract specific values from our tensors using indexing.
Keep in mind that when indexing the row and column, indices start at 0. And like indexing with lists, we can use negative indices as well (where -1 is the last item).
One of the most common NumPy operations we\u2019ll use in machine learning is matrix multiplication using the dot product. Suppose we wanted to take the dot product of two matrices with shapes [2 X 3] and [3 X 2]. We take the rows of our first matrix (2) and the columns of our second matrix (2) to determine the dot product, giving us an output of [2 X 2]. The only requirement is that the inside dimensions match, in this case the first matrix has 3 columns and the second matrix has 3 rows.
# Sum across a dimension\nx = np.array([[1,2],[3,4]])\nprint (x)\nprint (\"sum all: \", np.sum(x)) # adds all elements\nprint (\"sum axis=0: \", np.sum(x, axis=0)) # sum across rows\nprint (\"sum axis=1: \", np.sum(x, axis=1)) # sum across columns\n
What happens when we try to do operations with tensors with seemingly incompatible shapes? Their dimensions aren\u2019t compatible as is but how does NumPy still gives us the right result? This is where broadcasting comes in. The scalar is broadcast across the vector so that they have compatible shapes.
How can we fix this? We need to be careful to ensure that a is the same shape as b if we don't want this unintentional broadcasting behavior.
a = a.reshape(-1, 1)\na.shape # (3, 1)\nc = a + b\nc.shape # (3, 1)\nprint (c)\n
\narray([[ 6],\n [ 8],\n [10]])\n
This kind of unintended broadcasting happens more often then you'd think because this is exactly what happens when we create an array from a list. So we need to ensure that we apply the proper reshaping before using it for any operations.
We often need to change the dimensions of our tensors for operations like the dot product. If we need to switch two dimensions, we can transpose the tensor.
# Transposing\nx = np.array([[1,2,3], [4,5,6]])\nprint (\"x:\\n\", x)\nprint (\"x.shape: \", x.shape)\ny = np.transpose(x, (1,0)) # flip dimensions at index 0 and 1\nprint (\"y:\\n\", y)\nprint (\"y.shape: \", y.shape)\n
Sometimes, we'll need to alter the dimensions of the matrix. Reshaping allows us to transform a tensor into different permissible shapes. Below, our reshaped tensor has the same number of values as the original tensor. (1X6 = 2X3). We can also use -1 on a dimension and NumPy will infer the dimension based on our input tensor.
The way reshape works is by looking at each dimension of the new tensor and separating our original tensor into that many units. So here the dimension at index 0 of the new tensor is 2 so we divide our original tensor into 2 units, and each of those has 3 values.
Unintended reshaping
Though reshaping is very convenient to manipulate tensors, we must be careful of its pitfalls as well. Let's look at the example below. Suppose we have x, which has the shape [2 X 3 X 4].
Instead, if we transpose the tensor and then do a reshape, we get our desired tensor. Transpose allows us to put our two vectors that we want to combine together and then we use reshape to join them together. And as a general rule, we should always get our dimensions together before reshaping to combine them.
This becomes difficult when we're dealing with weight tensors with random values in many machine learning tasks. So a good idea is to always create a dummy example like this when you\u2019re unsure about reshaping. Blindly going by the tensor shape can lead to lots of issues downstream.
Check out Dask for scaling NumPy workflows with minimal change to existing code.
\n
To cite this content, please use:
\n
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { NumPy - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/foundations/pandas/","title":"Pandas for Machine Learning","text":""},{"location":"courses/foundations/pandas/#set-up","title":"Set up","text":"
First we'll import the NumPy and Pandas libraries and set seeds for reproducibility. We'll also download the dataset we'll be working with to disk.
import numpy as np\nimport pandas as pd\n
# Set seed for reproducibility\nnp.random.seed(seed=1234)\n
We're going to work with the Titanic dataset which has data on the people who embarked the RMS Titanic in 1912 and whether they survived the expedition or not. It's a very common and rich dataset which makes it very apt for exploratory data analysis with Pandas.
Let's load the data from the CSV file into a Pandas dataframe. The header=0 signifies that the first row (0th index) is a header row which contains the names of each column in our dataset.
# Read from CSV to Pandas DataFrame\nurl = \"https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/titanic.csv\"\ndf = pd.read_csv(url, header=0)\n
# First few items\ndf.head(3)\n
pclass name sex age sibsp parch ticket fare cabin embarked survived 0 1 Allen, Miss. Elisabeth Walton female 29.0000 0 0 24160 211.3375 B5 S 1 1 1 Allison, Master. Hudson Trevor male 0.9167 1 2 113781 151.5500 C22 C26 S 1 2 1 Allison, Miss. Helen Loraine female 2.0000 1 2 113781 151.5500 C22 C26 S 0
pclass name sex age sibsp parch ticket fare cabin embarked survived 14 1 Barkworth, Mr. Algernon Henry Wilson male 80.0 0 0 27042 30.0000 A23 S 1 61 1 Cavendish, Mrs. Tyrell William (Julia Florence... female 76.0 1 0 19877 78.8500 C46 S 1 1235 3 Svensson, Mr. Johan male 74.0 0 0 347060 7.7750 NaN S 0 135 1 Goldschmidt, Mr. George B male 71.0 0 0 PC 17754 34.6542 A5 C 0 9 1 Artagaveytia, Mr. Ramon male 71.0 0 0 PC 17609 49.5042 NaN C 0"},{"location":"courses/foundations/pandas/#grouping","title":"Grouping","text":"
We can also get statistics across our features for certain groups. Here we wan to see the average of our continuous features based on whether the passenger survived or not.
After exploring, we can clean and preprocess our dataset.
Be sure to check out our entire lesson focused on preprocessing in our MLOps course.
# Rows with at least one NaN value\ndf[pd.isnull(df).any(axis=1)].head()\n
pclass name sex age sibsp parch ticket fare cabin embarked survived 9 1 Artagaveytia, Mr. Ramon male 71.0 0 0 PC 17609 49.5042 NaN C 0 13 1 Barber, Miss. Ellen \"Nellie\" female 26.0 0 0 19877 78.8500 NaN S 1 15 1 Baumann, Mr. John D male NaN 0 0 PC 17318 25.9250 NaN S 0 23 1 Bidois, Miss. Rosalie female 42.0 0 0 PC 17757 227.5250 NaN C 1 25 1 Birnbaum, Mr. Jakob male 25.0 0 0 13905 26.0000 NaN C 0
# Drop rows with Nan values\ndf = df.dropna() # removes rows with any NaN values\ndf = df.reset_index() # reset's row indexes in case any rows were dropped\ndf.head()\n
index pclass name sex age sibsp parch ticket fare cabin embarked survived 0 0 1 Allen, Miss. Elisabeth Walton female 29.0000 0 0 24160 211.3375 B5 S 1 1 1 1 Allison, Master. Hudson Trevor male 0.9167 1 2 113781 151.5500 C22 C26 S 1 2 2 1 Allison, Miss. Helen Loraine female 2.0000 1 2 113781 151.5500 C22 C26 S 0 3 3 1 Allison, Mr. Hudson Joshua Creighton male 30.0000 1 2 113781 151.5500 C22 C26 S 0 4 4 1 Allison, Mrs. Hudson J C (Bessie Waldo Daniels) female 25.0000 1 2 113781 151.5500 C22 C26 S 0
# Dropping multiple columns\ndf = df.drop([\"name\", \"cabin\", \"ticket\"], axis=1) # we won't use text features for our initial basic models\ndf.head()\n
index pclass sex age sibsp parch fare embarked survived 0 0 1 female 29.0000 0 0 211.3375 S 1 1 1 1 male 0.9167 1 2 151.5500 S 1 2 2 1 female 2.0000 1 2 151.5500 S 0 3 3 1 male 30.0000 1 2 151.5500 S 0 4 4 1 female 25.0000 1 2 151.5500 S 0
We're now going to use feature engineering to create a column called family_size. We'll first define a function called get_family_size that will determine the family size using the number of parents and siblings.
# Lambda expressions to create new features\ndef get_family_size(sibsp, parch):\n family_size = sibsp + parch\n return family_size\n
Once we define the function, we can use lambda to apply that function on each row (using the numbers of siblings and parents in each row to determine the family size for each row).
When working with very large datasets, our Pandas DataFrames can become very large and it can be very slow or impossible to operate on them. This is where packages that can distribute workloads or run on more efficient hardware can come in handy.
Dask: parallel computing to scale packages like Numpy, Pandas and scikit-learn on one/multiple machines.
cuDF: efficient dataframe loading and computation on a GPU.
And, of course, we can combine these together (Dask-cuDF) to operate on partitions of a dataframe on the GPU.
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Pandas - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/foundations/python/","title":"Python for Machine Learning","text":""},{"location":"courses/foundations/python/#variables","title":"Variables","text":"
Variables are containers for holding data and they're defined by a name and value.
Here we use the variable name x in our examples but when you're working on a specific task, be sure to be explicit (ex. first_name) when creating variables (applies to functions, classes, etc. as well).
We can change the value of a variable by simply assigning a new value to it.
# Variables can be used with each other\na = 1\nb = 2\nc = a + b\nprint (c)\n
\n3\n
Know your types!
We should always know what types of variables we're dealing with so we can do the right operations with them. Here's a common mistake that can happen if we're using the wrong variable type.
# int variables\na = 5\nb = 3\nprint (a + b)\n
Show answer
\n8\n
# string variables\na = \"5\"\nb = \"3\"\nprint (a + b)\n
Lists are an ordered, mutable (changeable) collection of values that are comma separated and enclosed by square brackets. A list can be comprised of many different types of variables. Below is a list with an integer, string and a float:
# Creating a list\nx = [3, \"hello\", 1.2]\nprint (x)\n
\n[3, 'hello', 1.2]\n
# Length of a list\nlen(x)\n
\n3\n
We can add to a list by using the append function:
# Adding to a list\nx.append(7)\nprint (x)\nprint (len(x))\n
\n[3, 'hello', 1.2, 7]\n4\n
and just as easily replace existing items:
# Replacing items in a list\nx[1] = \"bye\"\nprint (x)\n
\n[3, 'bye', 1.2, 7]\n
and perform operations with lists:
# Operations\ny = [2.4, \"world\"]\nz = x + y\nprint (z)\n
Indexing and slicing from lists allow us to retrieve specific values within lists. Note that indices can be positive (starting from 0) or negative (-1 and lower, where -1 is the last item in the list).
# Indexing\nx = [3, \"hello\", 1.2]\nprint (\"x[0]: \", x[0])\nprint (\"x[1]: \", x[1])\nprint (\"x[-1]: \", x[-1]) # the last item\nprint (\"x[-2]: \", x[-2]) # the second to last item\n
# Slicing\nprint (\"x[:]: \", x[:]) # all indices\nprint (\"x[1:]: \", x[1:]) # index 1 to the end of the list\nprint (\"x[1:2]: \", x[1:2]) # index 1 to index 2 (not including index 2)\nprint (\"x[:-1]: \", x[:-1]) # index 0 to last index (not including last index)\n
What happens if we try to index beyond the length of a list?
x = [3, \"hello\", 1.2]\nprint (x[:100])\nprint (len(x[:100]))\n
Show answer
\n[3, 'hello', 1.2]\n3\n
Though this does produce results, we should always explicitly use the length of the list to index items from it to avoid incorrect assumptions for downstream processes."},{"location":"courses/foundations/python/#dictionaries","title":"Dictionaries","text":"
Dictionaries are an unordered, mutable collection of key-value pairs. You can retrieve values based on the key and a dictionary cannot have two of the same keys.
But of course, there is pretty much a way to do accomplish anything with Python. For example, even though native dictionaries are unordered, we can leverage the OrderedDict data structure to change that (useful if we want to iterate through keys in a certain order, etc.).
We can use if statements to conditionally do something. The conditions are defined by the words if, elif (which stands for else if) and else. We can have as many elif statements as we want. The indented code below each condition is the code that will execute if the condition is True.
# If statement\nx = 4\nif x < 1:\n score = \"low\"\nelif x <= 4: # elif = else if\n score = \"medium\"\nelse:\n score = \"high\"\nprint (score)\n
\nmedium\n
# If statement with a boolean\nx = True\nif x:\n print (\"it worked\")\n
A for loop can iterate over a collection of values (lists, tuples, dictionaries, etc.) The indented code is executed for each item in the collection of values.
# For loop\nveggies = [\"carrots\", \"broccoli\", \"beans\"]\nfor veggie in veggies:\n print (veggie)\n
\ncarrots\nbroccoli\nbeans\n
When the loop encounters the break command, the loop will terminate immediately. If there were more items in the list, they will not be processed.
# `break` from a for loop\nveggies = [\"carrots\", \"broccoli\", \"beans\"]\nfor veggie in veggies:\n if veggie == \"broccoli\":\n break\n print (veggie)\n
\ncarrots\n
When the loop encounters the continue command, the loop will skip all other operations for that item in the list only. If there were more items in the list, the loop will continue normally.
# `continue` to the next iteration\nveggies = [\"carrots\", \"broccoli\", \"beans\"]\nfor veggie in veggies:\n if veggie == \"broccoli\":\n continue\n print (veggie)\n
We can combine our knowledge of lists and for loops to leverage list comprehensions to create succinct code.
# For loop\nx = [1, 2, 3, 4, 5]\ny = []\nfor item in x:\n if item > 2:\n y.append(item)\nprint (y)\n
\n[3, 4, 5]\n
# List comprehension\ny = [item for item in x if item > 2]\nprint (y)\n
\n[3, 4, 5]\n
List comprehension for nested for loops
For the nested for loop below, which list comprehension is correct?
# Nested for loops\nwords = [[\"Am\", \"ate\", \"ATOM\", \"apple\"], [\"bE\", \"boy\", \"ball\", \"bloom\"]]\nsmall_words = []\nfor letter_list in words:\n for word in letter_list:\n if len(word) < 3:\n small_words.append(word.lower())\nprint (small_words)\n
\n['am', 'be']\n
[word.lower() if len(word) < 3 for word in letter_list for letter_list in words]
[word.lower() for word in letter_list for letter_list in words if len(word) < 3]
[word.lower() for letter_list in words for word in letter_list if len(word) < 3]
Show answer
Python syntax is usually very straight forward, so the correct answer involves just directly copying the statements from the nested for loop from top to bottom!
[word.lower() if len(word) < 3 for word in letter_list for letter_list in words]
[word.lower() for word in letter_list for letter_list in words if len(word) < 3]
[word.lower() for letter_list in words for word in letter_list if len(word) < 3]
Functions are a way to modularize reusable pieces of code. They're defined by the keyword def which stands for definition and they can have the following components.
# Define the function\ndef add_two(x):\n\"\"\"Increase x by 2.\"\"\"\n x += 2\n return x\n
Here are the components that may be required when we want to use the function. we need to ensure that the function name and the input parameters match with how we defined the function above.
# Use the function\nscore = 0\nnew_score = add_two(x=score)\nprint (new_score)\n
\n2\n
A function can have as many input parameters and outputs as we want.
# Function with multiple inputs\ndef join_name(first_name, last_name):\n\"\"\"Combine first name and last name.\"\"\"\n joined_name = first_name + \" \" + last_name\n return joined_name\n
# Use the function\nfirst_name = \"Goku\"\nlast_name = \"Mohandas\"\njoined_name = join_name(\n first_name=first_name, last_name=last_name)\nprint (joined_name)\n
\nGoku Mohandas\n
We can be even more explicit with our function definitions by specifying the types of our input and output arguments. We cover this in our documentation lesson because the typing information is automatically leveraged to create very intuitive documentation.
It's good practice to always use keyword argument when using a function so that it's very clear what input variable belongs to what function input parameter. On a related note, you will often see the terms *args and **kwargs which stand for arguments and keyword arguments. You can extract them when they are passed into a function. The significance of the * is that any number of arguments and keyword arguments can be passed into the function.
def f(*args, **kwargs):\n x = args[0]\n y = kwargs.get(\"y\")\n print (f\"x: {x}, y: {y}\")\n
Classes are object constructors and are a fundamental component of object oriented programming in Python. They are composed of a set of functions that define the class and it's operations.
Classes can be customized with magic methods like __init__ and __str__, to enable powerful operations. These are also known as dunder methods (ex. dunder init), which stands for double underscores due to the leading and trailing underscores.
The __init__ function is used when an instance of the class is initialized.
# Creating the class\nclass Pet(object):\n\"\"\"Class object for a pet.\"\"\"\n\n def __init__(self, species, name):\n\"\"\"Initialize a Pet.\"\"\"\n self.species = species\n self.name = name\n
# Creating an instance of a class\nmy_dog = Pet(species=\"dog\",\n name=\"Scooby\")\nprint (my_dog)\nprint (my_dog.name)\n
\n<__main__.Pet object at 0x7fe487e9c358>\nScooby\n
The print (my_dog) command printed something not so relevant to us. Let's fix that with the __str__ function.
# Creating the class\n# Creating the class\nclass Pet(object):\n\"\"\"Class object for a pet.\"\"\"\n\n def __init__(self, species, name):\n\"\"\"Initialize a Pet.\"\"\"\n self.species = species\n self.name = name\n\n def __str__(self):\n\"\"\"Output when printing an instance of a Pet.\"\"\"\n return f\"{self.species} named {self.name}\"\n
# Creating an instance of a class\nmy_dog = Pet(species=\"dog\",\n name=\"Scooby\")\nprint (my_dog)\nprint (my_dog.name)\n
\ndog named Scooby\nScooby\n
We'll be exploring additional built-in functions in subsequent notebooks (like __len__, __iter__ and __getitem__, etc.) but if you're curious, here is a tutorial on more magic methods.
Besides these magic functions, classes can also have object functions.
# Creating the class\nclass Pet(object):\n\"\"\"Class object for a pet.\"\"\"\n\n def __init__(self, species, name):\n\"\"\"Initialize a Pet.\"\"\"\n self.species = species\n self.name = name\n\n def __str__(self):\n\"\"\"Output when printing an instance of a Pet.\"\"\"\n return f\"{self.species} named {self.name}\"\n\n def change_name(self, new_name):\n\"\"\"Change the name of your Pet.\"\"\"\n self.name = new_name\n
# Creating an instance of a class\nmy_dog = Pet(species=\"dog\", name=\"Scooby\")\nprint (my_dog)\nprint (my_dog.name)\n
\ndog named Scooby\nScooby\n
# Using a class's function\nmy_dog.change_name(new_name=\"Scrappy\")\nprint (my_dog)\nprint (my_dog.name)\n
We can also build classes on top of one another using inheritance, which allows us to inherit all the properties and methods from another class (the parent).
class Dog(Pet):\n def __init__(self, name, breed):\n super().__init__(species=\"dog\", name=name)\n self.breed = breed\n\n def __str__(self):\n return f\"A {self.breed} doggo named {self.name}\"\n
Notice how we inherited the initialized variables from the parent Pet class like species and name. We also inherited functions such as change_name().
Which function is executed?
Which function is executed if the parent and child functions have functions with the same name?
Show answer
As you can see, both our parent class (Pet) and the child class (Dog) have different __str__ functions defined but share the same function name. The child class inherits everything from the parent classes but when there is conflict between function names, the child class' functions take precedence and overwrite the parent class' functions.
There are two important decorator methods to know about when it comes to classes: @classmethod and @staticmethod. We'll learn about decorators in the next section below but these specific methods pertain to classes so we'll cover them here.
class Dog(Pet):\n def __init__(self, name, breed):\n super().__init__(species=\"dog\", name=name)\n self.breed = breed\n\n def __str__(self):\n return f\"{self.breed} named {self.name}\"\n\n @classmethod\n def from_dict(cls, d):\n return cls(name=d[\"name\"], breed=d[\"breed\"])\n\n @staticmethod\n def is_cute(breed):\n return True # all animals are cute!\n
A @classmethod allows us to create class instances by passing in the uninstantiated class itself (cls). This is a great way to create (or load) classes from objects (ie. dictionaries).
Recall that functions allow us to modularize code and reuse them. However, we'll often want to add some functionality before or after the main function executes and we may want to do this for many different functions. Instead of adding more code to the original function, we can use decorators!
decorators: augment a function with pre/post-processing. Decorators wrap around the main function and allow us to operate on the inputs and or outputs.
Suppose we have a function called operations which increments the input value x by 1.
def operations(x):\n\"\"\"Basic operations.\"\"\"\n x += 1\n return x\n
operations(x=1)\n
\n2\n
Now let's say we want to increment our input x by 1 before and after the operations function executes and, to illustrate this example, let's say the increments have to be separate steps. Here's how we would do it by changing the original code:
def operations(x):\n\"\"\"Basic operations.\"\"\"\n x += 1\n x += 1\n x += 1\n return x\n
operations(x=1)\n
\n4\n
We were able to achieve what we want but we now increased the size of our operations function and if we want to do the same incrementing for any other function, we have to add the same code to all of those as well ... not very efficient. To solve this, let's create a decorator called add which increments x by 1 before and after the main function f executes.
"},{"location":"courses/foundations/python/#creating-a-decorator","title":"Creating a decorator","text":"
The decorator function accepts a function f which is the function we wish to wrap around, in our case, it's operations(). The output of the decorator is its wrapper function which receives the arguments and keyword arguments passed to function f.
Inside the wrapper function, we can:
extract the input parameters passed to function f.
make any changes we want to the function inputs.
function f is executed
make any changes to the function outputs
wrapper function returns some value(s), which is what the decorator returns as well since it returns wrapper.
# Decorator\ndef add(f):\n def wrapper(*args, **kwargs):\n\"\"\"Wrapper function for @add.\"\"\"\n x = kwargs.pop(\"x\") # .get() if not altering x\n x += 1 # executes before function f\n x = f(*args, **kwargs, x=x)\n x += 1 # executes after function f\n return x\n return wrapper\n
We can use this decorator by simply adding it to the top of our main function preceded by the @ symbol.
@add\ndef operations(x):\n\"\"\"Basic operations.\"\"\"\n x += 1\n return x\n
operations(x=1)\n
\n4\n
Suppose we wanted to debug and see what function actually executed with operations().
operations.__name__, operations.__doc__\n
\n('wrapper', 'Wrapper function for @add.')\n
The function name and docstring are not what we're looking for but it appears this way because the wrapper function is what was executed. In order to fix this, Python offers functools.wraps which carries the main function's metadata.
from functools import wraps\n
# Decorator\ndef add(f):\n @wraps(f)\n def wrap(*args, **kwargs):\n\"\"\"Wrapper function for @add.\"\"\"\n x = kwargs.pop(\"x\")\n x += 1\n x = f(*args, **kwargs, x=x)\n x += 1\n return x\n return wrap\n
@add\ndef operations(x):\n\"\"\"Basic operations.\"\"\"\n x += 1\n return x\n
operations.__name__, operations.__doc__\n
\n('operations', 'Basic operations.')\n
Awesome! We were able to decorate our main function operation() to achieve the customization we wanted without actually altering the function. We can reuse our decorator for other functions that may need the same customization!
This was a dummy example to show how decorators work but we'll be using them heavily during our MLOps lessons. A simple scenario would be using decorators to create uniform JSON responses from each API endpoint without including the bulky code in each endpoint.
Decorators allow for customized operations before and after the main function's execution but what about in between? Suppose we want to conditionally/situationally do some operations. Instead of writing a whole bunch of if-statements and make our functions bulky, we can use callbacks!
callbacks: conditional/situational processing within the function.
Our callbacks will be classes that have functions with key names that will execute at various periods during the main function's execution. The function names are up to us but we need to invoke the same callback functions within our main function.
We can pass in as many callbacks as we want and because they have appropriately named functions, they will be invoked at the appropriate times.
def operations(x, callbacks=[]):\n\"\"\"Basic operations.\"\"\"\n for callback in callbacks:\n callback.at_start(x)\n x += 1\n for callback in callbacks:\n callback.at_end(x)\n return x\n
x = 1\ntracker = x_tracker(x=x)\noperations(x=x, callbacks=[tracker])\n
\n2\n
tracker.history\n
\n[1, 2]\n
What's the difference compared to a decorator?
It seems like we've just done some operations before and after the function's main process? Isn't that what a decorator is for?
Show answer
With callbacks, it's easier to keep track of objects since it's all defined in a separate callback class. It's also now possible to interact with our function, not just before or after but throughout the entire process! Imagine a function with:
multiple processes where we want to execute operations in between them
execute operations repeatedly when loops are involved in functions
"},{"location":"courses/foundations/python/#putting-it-all-together","title":"Putting it all together","text":"
decorators + callbacks = powerful customization before, during and after the main function\u2019s execution without increasing its complexity. We will be using this duo to create powerful ML training scripts that are highly customizable in future lessons.
from functools import wraps\n
# Decorator\ndef add(f):\n @wraps(f)\n def wrap(*args, **kwargs):\n\"\"\"Wrapper function for @add.\"\"\"\n x = kwargs.pop(\"x\") # .get() if not altering x\n x += 1 # executes before function f\n x = f(*args, **kwargs, x=x)\n # can do things post function f as well\n return x\n return wrap\n
# Main function\n@add\ndef operations(x, callbacks=[]):\n\"\"\"Basic operations.\"\"\"\n for callback in callbacks:\n callback.at_start(x)\n x += 1\n for callback in callbacks:\n callback.at_end(x)\n return x\n
x = 1\ntracker = x_tracker(x=x)\noperations(x=x, callbacks=[tracker])\n
\n3\n
tracker.history\n
\n[1, 2, 3]\n
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Python - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
We'll first cover some basics with PyTorch such as creating tensors and converting from common data structures (lists, arrays, etc.) to tensors.\n
# Creating a random tensor\nx = torch.randn(2, 3) # normal distribution (rand(2,3) -> uniform distribution)\nprint(f\"Type: {x.type()}\")\nprint(f\"Size: {x.shape}\")\nprint(f\"Values: \\n{x}\")\n
# Dimensional operations\nx = torch.randn(2, 3)\nprint(f\"Values: \\n{x}\")\ny = torch.sum(x, dim=0) # add each row's value for every column\nprint(f\"Values: \\n{y}\")\nz = torch.sum(x, dim=1) # add each columns's value for every row\nprint(f\"Values: \\n{z}\")\n
We can determine gradients (rate of change) of our tensors with respect to their constituents using gradient bookkeeping. The gradient is a vector that points in the direction of greatest increase of a function. We'll be using gradients in the next lesson to determine how to change our weights to affect a particular objective function (ex. loss).
# Tensors with gradient bookkeeping\nx = torch.rand(3, 4, requires_grad=True)\ny = 3*x + 2\nz = y.mean()\nz.backward() # z has to be scalar\nprint(f\"x: \\n{x}\")\nprint(f\"x.grad: \\n{x.grad}\")\n
We also load our tensors onto the GPU for parallelized computation using CUDA (a parallel computing platform and API from Nvidia).\n
# Is CUDA available?\nprint (torch.cuda.is_available())\n
\n
\nFalse\n
\n\n
If False (CUDA is not available), let's change that by following these steps: Go to Runtime > Change runtime type > Change Hardware accelerator to GPU > Click Save\n
import torch\n
\n
# Is CUDA available now?\nprint (torch.cuda.is_available())\n
\n
\nTrue\n
\n
# Set device\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint (device)\n
\n
\ncuda\n
\n
x = torch.rand(2,3)\nprint (x.is_cuda)\nx = torch.rand(2,3).to(device) # Tensor is stored on the GPU\nprint (x.is_cuda)\n
\n
\nFalse\nTrue\n
\n\n
To cite this content, please use:
\n
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { PyTorch - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
So far we've processed inputs as whole (ex. applying filters across the entire input to extract features) but we can also process our inputs sequentially. For example we can think of each token in our text as an event in time (timestep). We can process each timestep, one at a time, and predict the class after the last timestep (token) has been processed. This is very powerful because the model now has a meaningful way to account for the sequential order of tokens in our sequence and predict accordingly.
$$ \\text{RNN forward pass for a single time step } X_t $$:
\\[ h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h) \\]
Variable Description \\(N\\) batch size \\(E\\) embeddings dimension \\(H\\) # of hidden units \\(W_{hh}\\) RNN weights \\(\\in \\mathbb{R}^{HXH}\\) \\(h_{t-1}\\) previous timestep's hidden state \\(\\in in \\mathbb{R}^{NXH}\\) \\(W_{xh}\\) input weights \\(\\in \\mathbb{R}^{EXH}\\) \\(X_t\\) input at time step \\(t \\in \\mathbb{R}^{NXE}\\) \\(b_h\\) hidden units bias \\(\\in \\mathbb{R}^{HX1}\\) \\(h_t\\) output from RNN for timestep \\(t\\)
Objective:
Process sequential data by accounting for the current input and also what has been learned from previous inputs.
Advantages:
Account for order and previous inputs in a meaningful way.
Conditioned generation for generating sequences.
Disadvantages:
Each time step's prediction depends on the previous prediction so it's difficult to parallelize RNN operations.
Processing long sequences can yield memory and computation issues.
Interpretability is difficult but there are few techniques that use the activations from RNNs to see what parts of the inputs are processed.
Miscellaneous:
Architectural tweaks to make RNNs faster and interpretable is an ongoing area of research.
title category 0 Sharon Accepts Plan to Reduce Gaza Army Operation... World 1 Internet Key Battleground in Wildlife Crime Fight Sci/Tech 2 July Durable Good Orders Rise 1.7 Percent Business 3 Growing Signs of a Slowing on Wall Street Business 4 The New Faces of Reality TV World"},{"location":"courses/foundations/recurrent-neural-networks/#preprocessing","title":"Preprocessing","text":"
We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc.
\n[nltk_data] Downloading package stopwords to /root/nltk_data...\n[nltk_data] Package stopwords is already up-to-date!\n['i', 'me', 'my', 'myself', 'we']\n
def preprocess(text, stopwords=STOPWORDS):\n\"\"\"Conditional preprocessing on our text unique to our task.\"\"\"\n # Lower\n text = text.lower()\n\n # Remove stopwords\n pattern = re.compile(r\"\\b(\" + r\"|\".join(stopwords) + r\")\\b\\s*\")\n text = pattern.sub(\"\", text)\n\n # Remove words in parenthesis\n text = re.sub(r\"\\([^)]*\\)\", \"\", text)\n\n # Spacing and filters\n text = re.sub(r\"([-;;.,!?<=>])\", r\" \\1 \", text)\n text = re.sub(\"[^A-Za-z0-9]+\", \" \", text) # remove non alphanumeric chars\n text = re.sub(\" +\", \" \", text) # remove multiple spaces\n text = text.strip()\n\n return text\n
# Sample\ntext = \"Great week for the NYSE!\"\npreprocess(text=text)\n
\ngreat week nyse\n
# Apply to dataframe\npreprocessed_df = df.copy()\npreprocessed_df.title = preprocessed_df.title.apply(preprocess)\nprint (f\"{df.title.values[0]}\\n\\n{preprocessed_df.title.values[0]}\")\n
\nSharon Accepts Plan to Reduce Gaza Army Operation, Haaretz Says\n\nsharon accepts plan reduce gaza army operation haaretz says\n
Warning
If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before applying those operations. This is because we cannot apply any knowledge gained from the test set accidentally (data leak) during preprocessing/training. However for global preprocessing steps like the function above where we aren't learning anything from the data itself, we can perform before splitting the data.
# Class weights\ncounts = np.bincount(y_train)\nclass_weights = {i: 1.0/count for i, count in enumerate(counts)}\nprint (f\"counts: {counts}\\nweights: {class_weights}\")\n
class Tokenizer(object):\n def __init__(self, char_level, num_tokens=None,\n pad_token=\"<PAD>\", oov_token=\"<UNK>\",\n token_to_index=None):\n self.char_level = char_level\n self.separator = \"\" if self.char_level else \" \"\n if num_tokens: num_tokens -= 2 # pad + unk tokens\n self.num_tokens = num_tokens\n self.pad_token = pad_token\n self.oov_token = oov_token\n if not token_to_index:\n token_to_index = {pad_token: 0, oov_token: 1}\n self.token_to_index = token_to_index\n self.index_to_token = {v: k for k, v in self.token_to_index.items()}\n\n def __len__(self):\n return len(self.token_to_index)\n\n def __str__(self):\n return f\"<Tokenizer(num_tokens={len(self)})>\"\n\n def fit_on_texts(self, texts):\n if not self.char_level:\n texts = [text.split(\" \") for text in texts]\n all_tokens = [token for text in texts for token in text]\n counts = Counter(all_tokens).most_common(self.num_tokens)\n self.min_token_freq = counts[-1][1]\n for token, count in counts:\n index = len(self)\n self.token_to_index[token] = index\n self.index_to_token[index] = token\n return self\n\n def texts_to_sequences(self, texts):\n sequences = []\n for text in texts:\n if not self.char_level:\n text = text.split(\" \")\n sequence = []\n for token in text:\n sequence.append(self.token_to_index.get(\n token, self.token_to_index[self.oov_token]))\n sequences.append(np.asarray(sequence))\n return sequences\n\n def sequences_to_texts(self, sequences):\n texts = []\n for sequence in sequences:\n text = []\n for index in sequence:\n text.append(self.index_to_token.get(index, self.oov_token))\n texts.append(self.separator.join([token for token in text]))\n return texts\n\n def save(self, fp):\n with open(fp, \"w\") as fp:\n contents = {\n \"char_level\": self.char_level,\n \"oov_token\": self.oov_token,\n \"token_to_index\": self.token_to_index\n }\n json.dump(contents, fp, indent=4, sort_keys=False)\n\n @classmethod\n def load(cls, fp):\n with open(fp, \"r\") as fp:\n kwargs = json.load(fp=fp)\n return cls(**kwargs)\n
Warning
It's important that we only fit using our train data split because during inference, our model will not always know every token so it's important to replicate that scenario with our validation and test splits as well.
# Sample of tokens\nprint (take(5, tokenizer.token_to_index.items()))\nprint (f\"least freq token's freq: {tokenizer.min_token_freq}\") # use this to adjust num_tokens\n
We'll need to do 2D padding to our tokenized text.
def pad_sequences(sequences, max_seq_len=0):\n\"\"\"Pad sequences to max length in sequence.\"\"\"\n max_seq_len = max(max_seq_len, max(len(sequence) for sequence in sequences))\n padded_sequences = np.zeros((len(sequences), max_seq_len))\n for i, sequence in enumerate(sequences):\n padded_sequences[i][:len(sequence)] = sequence\n return padded_sequences\n
Inputs to RNNs are sequential like text or time-series.
BATCH_SIZE = 64\nEMBEDDING_DIM = 100\n
# Input\nsequence_size = 8 # words per input\nx = torch.rand((BATCH_SIZE, sequence_size, EMBEDDING_DIM))\nseq_lens = torch.randint(high=sequence_size, size=(BATCH_SIZE, ))\nprint (x.shape)\nprint (seq_lens.shape)\n
\ntorch.Size([64, 8, 100])\ntorch.Size([1, 64])\n
$$ \\text{RNN forward pass for a single time step } X_t $$:
\\[ h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h) \\]
Variable Description \\(N\\) batch size \\(E\\) embeddings dimension \\(H\\) # of hidden units \\(W_{hh}\\) RNN weights \\(\\in \\mathbb{R}^{HXH}\\) \\(h_{t-1}\\) previous timestep's hidden state \\(\\in in \\mathbb{R}^{NXH}\\) \\(W_{xh}\\) input weights \\(\\in \\mathbb{R}^{EXH}\\) \\(X_t\\) input at time step \\(t \\in \\mathbb{R}^{NXE}\\) \\(b_h\\) hidden units bias \\(\\in \\mathbb{R}^{HX1}\\) \\(h_t\\) output from RNN for timestep \\(t\\)
At the first time step, the previous hidden state \\(h_{t-1}\\) can either be a zero vector (unconditioned) or initialized (conditioned). If we are conditioning the RNN, the first hidden state \\(h_0\\) can belong to a specific condition or we can concat the specific condition to the randomly initialized hidden vectors at each time step. More on this in the subsequent notebooks on RNNs.
# Forward pass through RNN\nx = x.permute(1, 0, 2) # RNN needs batch_size to be at dim 1\n\n# Loop through the inputs time steps\nhiddens = []\nfor t in range(sequence_size):\n hidden_t = rnn_cell(x[t], hidden_t)\n hiddens.append(hidden_t)\nhiddens = torch.stack(hiddens)\nhiddens = hiddens.permute(1, 0, 2) # bring batch_size back to dim 0\nprint (hiddens.size())\n
\ntorch.Size([64, 8, 128])\n
# We also could've used a more abstracted layer\nx = torch.rand((BATCH_SIZE, sequence_size, EMBEDDING_DIM))\nrnn = nn.RNN(EMBEDDING_DIM, RNN_HIDDEN_DIM, batch_first=True)\nout, h_n = rnn(x) # h_n is the last hidden state\nprint (\"out: \", out.shape)\nprint (\"h_n: \", h_n.shape)\n
In our model, we want to use the RNN's output after the last relevant token in the sentence is processed. The last relevant token doesn't refer the <PAD> tokens but to the last actual word in the sentence and its index is different for each input in the batch. This is why we included a seq_lens tensor in our batches.
def gather_last_relevant_hidden(hiddens, seq_lens):\n\"\"\"Extract and collect the last relevant\n hidden state based on the sequence length.\"\"\"\n seq_lens = seq_lens.long().detach().cpu().numpy() - 1\n out = []\n for batch_index, column_index in enumerate(seq_lens):\n out.append(hiddens[batch_index, column_index])\n return torch.stack(out)\n
# Get the last relevant hidden state\ngather_last_relevant_hidden(hiddens=out, seq_lens=seq_lens).squeeze(0).shape\n
\ntorch.Size([64, 128])\n
There are many different ways to use RNNs. So far we've processed our inputs one timestep at a time and we could either use the RNN's output at each time step or just use the final input timestep's RNN output. Let's look at a few other possibilities.
While our simple RNNs so far are great for sequentially processing our inputs, they have quite a few disadvantages. They commonly suffer from exploding or vanishing gradients as a result using the same set of weights (\\(W_{xh}\\) and \\(W_{hh}\\)) with each timestep's input. During backpropagation, this can cause gradients to explode (>1) or vanish (<1). If you multiply any number greater than 1 with itself over and over, it moves towards infinity (exploding gradients) and similarly, If you multiply any number less than 1 with itself over and over, it moves towards zero (vanishing gradients). To mitigate this issue, gated RNNs were devised to selectively retain information. If you're interested in learning more of the specifics, this post is a must-read.
There are two popular types of gated RNNs: Long Short-term Memory (LSTMs) units and Gated Recurrent Units (GRUs).
When deciding between LSTMs and GRUs, empirical performance is the best factor but in general GRUs offer similar performance with less complexity (less weights).
Understanding LSTM Networks - Chris Olah
# Input\nsequence_size = 8 # words per input\nx = torch.rand((BATCH_SIZE, sequence_size, EMBEDDING_DIM))\nprint (x.shape)\n
We can also have RNNs that process inputs from both directions (first token to last token and vice versa) and combine their outputs. This architecture is known as a bidirectional RNN.
Notice that the output for each sample at each timestamp has size 256 (double the RNN_HIDDEN_DIM). This is because this includes both the forward and backward directions from the BiRNN.
def get_probability_distribution(y_prob, classes):\n\"\"\"Create a dict of class probabilities from an array.\"\"\"\n results = {}\n for i, class_ in enumerate(classes):\n results[class_] = np.float64(y_prob[i])\n sorted_results = {k: v for k, v in sorted(\n results.items(), key=lambda item: item[1], reverse=True)}\n return sorted_results\n
Transformers are a very popular architecture that leverage and extend the concept of self-attention to create very useful representations of our input data for a downstream task.
advantages:
better representation for our input tokens via contextual embeddings where the token representation is based on the specific neighboring tokens using self-attention.
sub-word tokens, as opposed to character tokens, since they can hold more meaningful representation for many of our keywords, prefixes, suffixes, etc.
attend (in parallel) to all the tokens in our input, as opposed to being limited by filter spans (CNNs) or memory issues from sequential processing (RNNs).
disadvantages:
computationally intensive
required large amounts of data (mitigated using pretrained models)
Attention Is All You Need"},{"location":"courses/foundations/transformers/#set-up","title":"Set up","text":"
Let's set our seed and device for our main task.
import numpy as np\nimport pandas as pd\nimport random\nimport torch\nimport torch.nn as nn\n
title category 0 Sharon Accepts Plan to Reduce Gaza Army Operation... World 1 Internet Key Battleground in Wildlife Crime Fight Sci/Tech 2 July Durable Good Orders Rise 1.7 Percent Business 3 Growing Signs of a Slowing on Wall Street Business 4 The New Faces of Reality TV World
# Reduce data size (too large to fit in Colab's limited memory)\ndf = df[:10000]\nprint (len(df))\n
We're going to clean up our input data first by doing operations such as lower text, removing stop (filler) words, filters using regular expressions, etc.
\n[nltk_data] Downloading package stopwords to /root/nltk_data...\n[nltk_data] Package stopwords is already up-to-date!\n['i', 'me', 'my', 'myself', 'we']\n
def preprocess(text, stopwords=STOPWORDS):\n\"\"\"Conditional preprocessing on our text unique to our task.\"\"\"\n # Lower\n text = text.lower()\n\n # Remove stopwords\n pattern = re.compile(r\"\\b(\" + r\"|\".join(stopwords) + r\")\\b\\s*\")\n text = pattern.sub(\"\", text)\n\n # Remove words in parenthesis\n text = re.sub(r\"\\([^)]*\\)\", \"\", text)\n\n # Spacing and filters\n text = re.sub(r\"([-;;.,!?<=>])\", r\" \\1 \", text)\n text = re.sub(\"[^A-Za-z0-9]+\", \" \", text) # remove non alphanumeric chars\n text = re.sub(\" +\", \" \", text) # remove multiple spaces\n text = text.strip()\n\n return text\n
# Sample\ntext = \"Great week for the NYSE!\"\npreprocess(text=text)\n
\ngreat week nyse\n
# Apply to dataframe\npreprocessed_df = df.copy()\npreprocessed_df.title = preprocessed_df.title.apply(preprocess)\nprint (f\"{df.title.values[0]}\\n\\n{preprocessed_df.title.values[0]}\")\n
\nSharon Accepts Plan to Reduce Gaza Army Operation, Haaretz Says\n\nsharon accepts plan reduce gaza army operation haaretz says\n
Warning
If you have preprocessing steps like standardization, etc. that are calculated, you need to separate the training and test set first before applying those operations. This is because we cannot apply any knowledge gained from the test set accidentally (data leak) during preprocessing/training. However for global preprocessing steps like the function above where we aren't learning anything from the data itself, we can perform before splitting the data.
# Class weights\ncounts = np.bincount([label_encoder.class_to_index[class_] for class_ in y_train])\nclass_weights = {i: 1.0/count for i, count in enumerate(counts)}\nprint (f\"counts: {counts}\\nweights: {class_weights}\")\n
The most popular type of self-attention is scaled dot-product attention from the widely-cited Attention is all you need paper. This type of attention involves projecting our encoded input sequences onto three matrices, queries (Q), keys (K) and values (V), whose weights we learn.
\\[ Q = XW_q \\text{ where } W_q \\in \\mathbb{R}^{HXd_q} \\] \\[ K = XW_k \\text{ where } W_k \\in \\mathbb{R}^{HXd_k} \\] \\[ V = XW_v \\text{ where } W_v \\in \\mathbb{R}^{HXd_v} \\] \\[ attention (Q, K, V) = softmax( \\frac{Q K^{T}}{\\sqrt{d_k}} ) V \\in \\mathbb{R}^{MXd_v} \\]
Variable Description \\(X\\) encoded inputs \\(\\in \\mathbb{R}^{NXMXH}\\) \\(N\\) batch size \\(M\\) max sequence length in the batch \\(H\\) hidden dim, model dim, etc. \\(W_q\\) query weights \\(\\in \\mathbb{R}^{HXd_q}\\) \\(W_k\\) key weights \\(\\in \\mathbb{R}^{HXd_k}\\) \\(W_v\\) value weights \\(\\in \\mathbb{R}^{HXd_v}\\)
Instead of applying self-attention only once across the entire encoded input, we can also separate the input and apply self-attention in parallel (heads) to each input section and concatenate them. This allows the different head to learn unique representations while maintaining the complexity since we split the input into smaller subspaces.
Variable Description \\(h\\) number of attention heads \\(W_O\\) multi-head attention weights \\(\\in \\mathbb{R}^{hd_vXH}\\) \\(H\\) hidden dim (or dimension of the model \\(d_{model}\\))
With self-attention, we aren't able to account for the sequential position of our input tokens. To address this, we can use positional encoding to create a representation of the location of each token with respect to the entire sequence. This can either be learned (with weights) or we can use a fixed function that can better extend to create positional encoding for lengths during inference that were not observed during training.
Variable Description \\(pos\\) position of the token \\((1...M)\\) \\(i\\) hidden dim \\((1..H)\\)
This effectively allows us to represent each token's relative position using a fixed function for very large sequences. And because we've constrained the positional encodings to have the same dimensions as our encoded inputs, we can simply concatenate them before feeding them into the multi-head attention heads.
And here's how it all fits together! It's an end-to-end architecture that creates these contextual representations and uses an encoder-decoder architecture to predict the outcomes (one-to-one, many-to-one, many-to-many, etc.) Due to the complexity of the architecture, they require massive amounts of data for training without overfitting, however, they can be leveraged as pretrained models to finetune with smaller datasets that are similar to the larger set it was initially trained on.
Attention Is All You Need
We're not going to the implement the Transformer from scratch but we will use the Hugging Face library to do so in the training lesson!
We're going to use a pretrained BertModel to act as a feature extractor. We'll only use the encoder to receive sequential and pooled outputs (is_decoder=False is default).
class Transformer(nn.Module):\n def __init__(self, transformer, dropout_p, embedding_dim, num_classes):\n super(Transformer, self).__init__()\n self.transformer = transformer\n self.dropout = torch.nn.Dropout(dropout_p)\n self.fc1 = torch.nn.Linear(embedding_dim, num_classes)\n\n def forward(self, inputs):\n ids, masks = inputs\n seq, pool = self.transformer(input_ids=ids, attention_mask=masks)\n z = self.dropout(pool)\n z = self.fc1(z)\n return z\n
We decided to work with the pooled output, but we could have just as easily worked with the sequential output (encoder representation for each sub-token) and applied a CNN (or other decoder options) on top of it.
def get_probability_distribution(y_prob, classes):\n\"\"\"Create a dict of class probabilities from an array.\"\"\"\n results = {}\n for i, class_ in enumerate(classes):\n results[class_] = np.float64(y_prob[i])\n sorted_results = {k: v for k, v in sorted(\n results.items(), key=lambda item: item[1], reverse=True)}\n return sorted_results\n
Now we're ready to start the MLOps course to learn how to apply all this foundational modeling knowledge to responsibly develop, deploy and maintain production machine learning applications.
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Transformers - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/foundations/utilities/","title":"Utilities for Machine Learning","text":""},{"location":"courses/foundations/utilities/#set-up","title":"Set up","text":"
We're having to set a lot of seeds for reproducibility now, so let's wrap it all up in a function.
import numpy as np\nimport pandas as pd\nimport random\nimport torch\nimport torch.nn as nn\n
Next we'll define a LabelEncoder to encode our text labels into unique indices. We're not going to use scikit-learn's LabelEncoder anymore because we want to be able to save and load our instances the way we want to.
import itertools\n
class LabelEncoder(object):\n\"\"\"Label encoder for tag labels.\"\"\"\n def __init__(self, class_to_index={}):\n self.class_to_index = class_to_index or {} # mutable defaults ;)\n self.index_to_class = {v: k for k, v in self.class_to_index.items()}\n self.classes = list(self.class_to_index.keys())\n\n def __len__(self):\n return len(self.class_to_index)\n\n def __str__(self):\n return f\"<LabelEncoder(num_classes={len(self)})>\"\n\n def fit(self, y):\n classes = np.unique(y)\n for i, class_ in enumerate(classes):\n self.class_to_index[class_] = i\n self.index_to_class = {v: k for k, v in self.class_to_index.items()}\n self.classes = list(self.class_to_index.keys())\n return self\n\n def encode(self, y):\n encoded = np.zeros((len(y)), dtype=int)\n for i, item in enumerate(y):\n encoded[i] = self.class_to_index[item]\n return encoded\n\n def decode(self, y):\n classes = []\n for i, item in enumerate(y):\n classes.append(self.index_to_class[item])\n return classes\n\n def save(self, fp):\n with open(fp, \"w\") as fp:\n contents = {'class_to_index': self.class_to_index}\n json.dump(contents, fp, indent=4, sort_keys=False)\n\n @classmethod\n def load(cls, fp):\n with open(fp, \"r\") as fp:\n kwargs = json.load(fp=fp)\n return cls(**kwargs)\n
# Class weights\ncounts = np.bincount(y_train)\nclass_weights = {i: 1.0/count for i, count in enumerate(counts)}\nprint (f\"counts: {counts}\\nweights: {class_weights}\")\n
We need to standardize our data (zero mean and unit variance) so a specific feature's magnitude doesn't affect how the model learns its weights. We're only going to standardize the inputs X because our outputs y are class values. We're going to compose our own StandardScaler class so we can easily save and load it later during inference.
# Standardize the data (mean=0, std=1) using training data\nX_scaler = StandardScaler()\nX_scaler.fit(X_train)\n
# Apply scaler on training and test data (don't standardize outputs for classification)\nX_train = X_scaler.scale(X_train)\nX_val = X_scaler.scale(X_val)\nX_test = X_scaler.scale(X_test)\n
# Check (means should be ~0 and std should be ~1)\nprint (f\"X_test[0]: mean: {np.mean(X_test[:, 0], axis=0):.1f}, std: {np.std(X_test[:, 0], axis=0):.1f}\")\nprint (f\"X_test[1]: mean: {np.mean(X_test[:, 1], axis=0):.1f}, std: {np.std(X_test[:, 1], axis=0):.1f}\")\n
We're going to place our data into a Dataset and use a DataLoader to efficiently create batches for training and evaluation.
import torch\n
# Seed seed for reproducibility\ntorch.manual_seed(SEED)\n
class Dataset(torch.utils.data.Dataset):\n def __init__(self, X, y):\n self.X = X\n self.y = y\n\n def __len__(self):\n return len(self.y)\n\n def __str__(self):\n return f\"<Dataset(N={len(self)})>\"\n\n def __getitem__(self, index):\n X = self.X[index]\n y = self.y[index]\n return [X, y]\n\n def collate_fn(self, batch):\n\"\"\"Processing on a batch.\"\"\"\n # Get inputs\n batch = np.array(batch)\n X = np.stack(batch[:, 0], axis=0)\n y = batch[:, 1]\n\n # Cast\n X = torch.FloatTensor(X.astype(np.float32))\n y = torch.LongTensor(y.astype(np.int32))\n\n return X, y\n\n def create_dataloader(self, batch_size, shuffle=False, drop_last=False):\n return torch.utils.data.DataLoader(\n dataset=self, batch_size=batch_size, collate_fn=self.collate_fn,\n shuffle=shuffle, drop_last=drop_last, pin_memory=True)\n
We don't really need the collate_fn here but we wanted to make it transparent because we will need it when we want to do specific processing on our batch (ex. padding).
\nDatasets:\n Train dataset: <Dataset(N=1050)>\n Val dataset: <Dataset(N=225)>\n Test dataset: <Dataset(N=225)>\nSample point:\n X: [-1.47355106 -1.67417243]\n y: 0\n
So far, we used batch gradient descent to update our weights. This means that we calculated the gradients using the entire training dataset. We also could've updated our weights using stochastic gradient descent (SGD) where we pass in one training example one at a time. The current standard is mini-batch gradient descent, which strikes a balance between batch and SGD, where we update the weights using a mini-batch of n (BATCH_SIZE) samples. This is where the DataLoader object comes in handy.
So far we've been running our operations on the CPU but when we have large datasets and larger models to train, we can benefit by parallelizing tensor operations on a GPU. In this notebook, you can use a GPU by going to Runtime > Change runtime type > Select GPU in the Hardware accelerator dropdown. We can what device we're using with the following line of code:
# Set CUDA seeds\ntorch.cuda.manual_seed(SEED)\ntorch.cuda.manual_seed_all(SEED) # multi-GPU\n
# Device configuration\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint (device)\n
So far we've been writing training loops that train only using the train data split and then we perform evaluation on our test set. But in reality, we would follow this process:
Train using mini-batches on one epoch of the train data split.
Evaluate loss on the validation split and use it to adjust hyperparameters (ex. learning rate).
After training ends (via stagnation in improvements, desired performance, etc.), evaluate your trained model on the test (hold-out) data split.
We'll create a Trainer class to keep all of these processes organized.
The first function in the class is train_step which will train the model using batches from one epoch of the train data split.
def train_step(self, dataloader):\n\"\"\"Train step.\"\"\"\n # Set model to train mode\n self.model.train()\n loss = 0.0\n\n # Iterate over train batches\n for i, batch in enumerate(dataloader):\n\n # Step\n batch = [item.to(self.device) for item in batch] # Set device\n inputs, targets = batch[:-1], batch[-1]\n self.optimizer.zero_grad() # Reset gradients\n z = self.model(inputs) # Forward pass\n J = self.loss_fn(z, targets) # Define loss\n J.backward() # Backward pass\n self.optimizer.step() # Update weights\n\n # Cumulative Metrics\n loss += (J.detach().item() - loss) / (i + 1)\n\n return loss\n
Next we'll define the eval_step which will be used for processing both the validation and test data splits. This is because neither of them require gradient updates and display the same metrics.
def eval_step(self, dataloader):\n\"\"\"Validation or test step.\"\"\"\n # Set model to eval mode\n self.model.eval()\n loss = 0.0\n y_trues, y_probs = [], []\n\n # Iterate over val batches\n with torch.inference_mode():\n for i, batch in enumerate(dataloader):\n\n # Step\n batch = [item.to(self.device) for item in batch] # Set device\n inputs, y_true = batch[:-1], batch[-1]\n z = self.model(inputs) # Forward pass\n J = self.loss_fn(z, y_true).item()\n\n # Cumulative Metrics\n loss += (J - loss) / (i + 1)\n\n # Store outputs\n y_prob = F.softmax(z).cpu().numpy()\n y_probs.extend(y_prob)\n y_trues.extend(y_true.cpu().numpy())\n\n return loss, np.vstack(y_trues), np.vstack(y_probs)\n
The final function is the predict_step which will be used for inference. It's fairly similar to the eval_step except we don't calculate any metrics. We pass on the predictions which we can use to generate our performance scores.
def predict_step(self, dataloader):\n\"\"\"Prediction step.\"\"\"\n # Set model to eval mode\n self.model.eval()\n y_probs = []\n\n # Iterate over val batches\n with torch.inference_mode():\n for i, batch in enumerate(dataloader):\n\n # Forward pass w/ inputs\n inputs, targets = batch[:-1], batch[-1]\n z = self.model(inputs)\n\n # Store outputs\n y_prob = F.softmax(z).cpu().numpy()\n y_probs.extend(y_prob)\n\n return np.vstack(y_probs)\n
As our model starts to optimize and perform better, the loss will reduce and we'll need to make smaller adjustments. If we keep using a fixed learning rate, we'll be overshooting back and forth. Therefore, we're going to add a learning rate scheduler to our optimizer to adjust our learning rate during training. There are many schedulers schedulers to choose from but a popular one is ReduceLROnPlateau which reduces the learning rate when a metric (ex. validation loss) stops improving. In the example below we'll reduce the learning rate by a factor of 0.1 (factor=0.1) when our metric of interest (self.scheduler.step(val_loss)) stops decreasing (mode=\"min\") for three (patience=3) straight epochs.
We should never train our models for an arbitrary number of epochs but instead we should have explicit stopping criteria (even if you are bootstrapped by compute resources). Common stopping criteria include when validation performance stagnates for certain # of epochs (patience), desired performance is reached, etc.
There are lots of other utilities to cover, such as:
Tokenizers to convert text to sequence of indices
Various encoders to represent our data
Padding to ensure uniform data shapes
Experiment tracking to visualize and keep track of all experiments
Hyperparameter optimization to tune our parameters (layers, learning rate, etc.)
and many more!
We'll explore these as we require them in future lessons including some in our MLOps course!
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Utilities - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/api/","title":"APIs for Model Serving","text":""},{"location":"courses/mlops/api/#intuition","title":"Intuition","text":"
Our CLI application made it much easier to interact with our models, especially for fellow team members who may not want to delve into the codebase. But there are several limitations to serving our models with a CLI:
users need access to the terminal, codebase, virtual environment, etc.
CLI outputs on the terminal are not exportable
To address these issues, we're going to develop an application programming interface (API) that will anyone to interact with our application with a simple request.
The end user may not directly interact with our API but may use UI/UX components that send requests to our it.
APIs allow different applications to communicate with each other in real-time. But when it comes to serving predictions, we need to first decide if we'll do that in batches or real-time, which is entirely based on the feature space (finite vs. unbound).
We can make batch predictions on a finite set of inputs which are then written to a database for low latency inference. When a user or downstream process sends an inference request in real-time, cached results from the database are returned.
\u2705\u00a0 generate and cache predictions for very fast inference for users.
\u2705\u00a0 the model doesn't need to be spun up as it's own service since it's never used in real-time.
\u274c\u00a0 predictions can become stale if user develops new interests that aren\u2019t captured by the old data that the current predictions are based on.
\u274c\u00a0 input feature space must be finite because we need to generate all the predictions before they're needed for real-time.
Batch serving tasks
What are some tasks where batch serving is ideal?
Show answer
Recommend content that existing users will like based on their viewing history. However, new users may just receive some generic recommendations based on their explicit interests until we process their history the next day. And even if we're not doing batch serving, it might still be useful to cache very popular sets of input features (ex. combination of explicit interests leads to certain recommended content) so that we can serve those predictions faster.
We can also serve live predictions, typically through a request to an API with the appropriate input data.
\u2705\u00a0 can yield more up-to-date predictions which may yield a more meaningful user experience, etc.
\u274c\u00a0 requires managed microservices to handle request traffic.
\u274c\u00a0 requires real-time monitoring since input space in unbounded, which could yield erroneous predictions.
In this lesson, we'll create the API required to enable real-time serving. The interactions in our situation involve the client (users, other applications, etc.) sending a request (ex. prediction request) with the appropriate inputs to the server (our application with a trained model) and receiving a response (ex. prediction) in return.
Parts of the URI Description scheme protocol definition domain address of the website port endpoint path location of the resource query string parameters to identify resources anchor location on webpage Parts of the path Description /models collection resource of all models/models/{modelID} single resource from the models collection modelId path parameters filter query parameter"},{"location":"courses/mlops/api/#method","title":"Method","text":"
The method is the operation to execute on the specific resource defined by the URI. There are many possible methods to choose from, but the four below are the most popular, which are often referred to as CRUD because they allow you to Create, Read, Update and Delete.
GET: get a resource.
POST: create or update a resource.
PUT/PATCH: create or update a resource.
DELETE: delete a resource.
Note
You could use either the POST or PUT request method to create and modify resources but the main difference is that PUT is idempotent which means you can call the method repeatedly and it'll produce the same state every time. Whereas, calling POST multiple times can result in creating multiple instance and so changes the overall state each time.
POST /models/<new_model> -d {} # error since we haven't created the `new_model` resource yet\nPOST /models -d {} # creates a new model based on information provided in data\nPOST /models/<existing_model> -d {} # updates an existing model based on information provided in data\n\nPUT /models/<new_model> -d {} # creates a new model based on information provided in data\nPUT /models/<existing_model> -d {} # updates an existing model based on information provided in data\n
We can use cURL to execute our API calls with the following options:
curl --help\n
\nUsage: curl [options...] \n-X, --request HTTP method (ie. GET)\n-H, --header headers to be sent to the request (ex. authentication)\n-d, --data data to POST, PUT/PATCH, DELETE (usually JSON)\n...\n\n\n
For example, if we want to GET all models, our cURL command would look like this:\n
Headers contain information about a certain event and are usually found in both the client's request as well as the server's response. It can range from what type of format they'll send and receive, authentication and caching info, etc.\n
The body contains information that may be necessary for the request to be processed. It's usually a JSON object sent during POST, PUT/PATCH, DELETE request methods.
The response we receive from our server is the result of the request we sent. The response also includes headers and a body which should include the proper HTTP status code as well as explicit messages, data, etc.
We may also want to include other metadata in the response such as model version, datasets used, etc. Anything that the downstream consumer may be interested in or metadata that might be useful for inspection.
\n
There are many HTTP status codes to choose from depending on the situation but here are the most common options:
\n
\nCode\nDescription\n200 OK\nmethod operation was successful.\n201 CREATED\nPOST or PUT method successfully created a resource.\n202 ACCEPTED\nthe request was accepted for processing (but processing may not be done).\n400 BAD REQUEST\nserver cannot process the request because of a client side error.\n401 UNAUTHORIZED\nyou're missing required authentication.\n403 FORBIDDEN\nyou're not allowed to do this operation.\n404 NOT FOUND\nthe resource you're looking for was not found.\n500 INTERNAL SERVER ERROR\nthere was a failure somewhere in the system process.\n501 NOT IMPLEMENTED\nthis operation on the resource doesn't exist yet.\n
When designing our API, there are some best practices to follow:
\n
\n
URI paths, messages, etc. should be as explicit as possible. Avoid using cryptic resource names, etc.
\n
Use nouns, instead of verbs, for naming resources. The request method already accounts for the verb (\u2705\u00a0 GET /users not \u274c\u00a0 GET /get_users).
\n
Plural nouns (\u2705\u00a0 GET /users/{userId} not \u274c\u00a0 GET /user/{userID}).
\n
Use dashes in URIs for resources and path parameters but use underscores for query parameters (GET /nlp-models/?find_desc=bert).
\n
Return appropriate HTTP and informative messages to the user.
We're going to define our API in a separate app directory because, in the future, we may have additional packages like tagifai and we don't want our app to be attached to any one package. Inside our app directory, we'll create the follow scripts:
We're going to use FastAPI as our framework to build our API service. There are plenty of other framework options out there such as Flask, Django and even non-Python based options like Node, Angular, etc. FastAPI combines many of the advantages across these frameworks and is maturing quickly and becoming more widely adopted. It's notable advantages include:
\n
\n
development in Python
\n
highly performant
\n
data validation via pydantic
\n
autogenerated documentation
\n
dependency injection
\n
security via OAuth2
\n
\n
pip install fastapi==0.78.0\n
\n
# Add to requirements.txt\nfastapi==0.78.0\n
\n
Your choice of framework also depends on your team's existing systems and processes. However, with the wide adoption of microservices, we can wrap our specific application in any framework we choose and expose the appropriate resources so all other systems can easily communicate with it.
The first step is to initialize our API in our api.py script` by defining metadata like the title, description and version:
\n
# app/api.py\nfrom fastapi import FastAPI\n\n# Define application\napp = FastAPI(\n title=\"TagIfAI - Made With ML\",\n description=\"Classify machine learning projects.\",\n version=\"0.1\",\n)\n
\n
Our first endpoint is going to be a simple one where we want to show that everything is working as intended. The path for the endpoint will just be / (when a user visit our base URI) and it'll be a GET request. This simple endpoint is often used as a health check to ensure that our application is indeed up and running properly.
We let our application know that the endpoint is at / through the path operation decorator in line 4 and we return a JSON response with the 200 OK HTTP status code.
\n
In our actual api.py script, you'll notice that even our index function looks different. Don't worry, we're slowly adding components to our endpoints and justifying them along the way.
We're using Uvicorn, a fast ASGI server that can run asynchronous code in a single process to launch our application.
\n
pip install uvicorn==0.17.6\n
\n
# Add to requirements.txt\nuvicorn==0.17.6\n
\n
We can launch our application with the following command:
\n
uvicorn app.api:app \\ # location of app (`app` directory > `api.py` script > `app` object)\n--host 0.0.0.0 \\ # localhost\n--port 8000 \\ # port 8000\n--reload \\ # reload every time we update\n--reload-dir tagifai \\ # only reload on updates to `tagifai` directory\n--reload-dir app # and the `app` directory\n
\n
\nINFO: Will watch for changes in these directories: ['/Users/goku/Documents/madewithml/mlops/app', '/Users/goku/Documents/madewithml/mlops/tagifai']\nINFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)\nINFO: Started reloader process [57609] using statreload\nINFO: Started server process [57611]\nINFO: Waiting for application startup.\nINFO: Application startup complete.\n
\n\n
Notice that we only reload on changes to specific directories, as this is to avoid reloading on files that won't impact our application such as log files, etc.
\n
If we want to manage multiple uvicorn workers to enable parallelism in our application, we can use Gunicorn in conjunction with Uvicorn. This will usually be done in a production environment where we'll be dealing with meaningful traffic. We've included a app/gunicorn.py script with the customizable configuration and we can launch all the workers with the follow command:\n
Now that we have our application running, we can submit our GET request using several different methods:
\n
\n
Visit the endpoint on a browser at http://localhost:8000/
\n
cURL\n
curl -X GET http://localhost:8000/\n
\n
Access endpoints via code. Here we show how to do it with the requests library in Python but it can be done with most popular languages. You can even use an online tool to convert your cURL commands into code!\n
In our GET \\ request's response above, there was not a whole lot of information about the actual request, but it's useful to have details such as URL, timestamp, etc. But we don't want to do this individually for each endpoint, so let's use decorators to automatically add relevant metadata to our responses
We're passing in a Request instance in line 10 so we can access information like the request method and URL. Therefore, our endpoint functions also need to have this Request object as an input argument. Once we receive the results from our endpoint function f, we can append the extra details and return a more informative response. To use this decorator, we just have to wrap our functions accordingly.
There are also some built-in decorators we should be aware of. We've already seen the path operation decorator (ex. @app.get(\"/\")) which defines the path for the endpoint as well as other attributes. There is also the events decorator (@app.on_event()) which we can use to startup and shutdown our application. For example, we use the (@app.on_event(\"startup\")) event to load the artifacts for the model to use for inference. The advantage of doing this as an event is that our service won't start until this is complete and so no requests will be prematurely processed and cause errors. Similarly, we can perform shutdown events with (@app.on_event(\"shutdown\")), such as saving logs, cleaning, etc.
When we define an endpoint, FastAPI automatically generates some documentation (adhering to OpenAPI standards) based on the it's inputs, typing, outputs, etc. We can access the Swagger UI for our documentation by going to /docs endpoints on any browser while the api is running.
\n
Click on an endpoint > Try it out > Execute to see what the server's response will look like. Since this was a GET request without any inputs, our request body was empty but for other method's we'll need to provide some information (we'll illustrate this when we do a POST request).
\n
Notice that our endpoint is organized under sections in the UI. We can use tags when defining our endpoints in the script:\n
Notice that we're passing an optional query parameter filter here to indicate the subset of performance we care about. We can include this parameter in our GET request like so:
Our next endpoint will be to GET the arguments used to train the model. This time, we're using a path parameter args, which is a required field in the URI.
\n
@app.get(\"/args/{arg}\", tags=[\"Arguments\"])\n@construct_response\ndef _arg(request: Request, arg: str) -> Dict:\n\"\"\"Get a specific parameter's value used for the run.\"\"\"\n response = {\n \"message\": HTTPStatus.OK.phrase,\n \"status-code\": HTTPStatus.OK,\n \"data\": {\n arg: vars(artifacts[\"args\"]).get(arg, \"\"),\n },\n }\n return response\n
\n
We can perform our GET request like so, where the param is part of the request URI's path as opposed to being part of it's query string.\n
Now it's time to define our endpoint for prediction. We need to consume the inputs that we want to classify and so we need to define the schema that needs to be followed when defining those inputs.
Here we're defining a PredictPayload object as a List of Text objects called texts. Each Text object is a string that defaults to None and must have a minimum length of 1 character.
\n
Note
\n
We could've just defined our PredictPayload like so:\n
class PredictPayload(BaseModel):\n texts: List[str] = Query(None, min_length=1)\n
\nBut we wanted to create very explicit schemas in case we want to incorporate more validation or add additional parameters in the future.\n
We can now use this payload in our predict endpoint:
\n
from app.schemas import PredictPayload\nfrom tagifai import predict\n\n@app.post(\"/predict\", tags=[\"Prediction\"])\n@construct_response\ndef _predict(request: Request, payload: PredictPayload) -> Dict:\n\"\"\"Predict tags for a list of texts.\"\"\"\n texts = [item.text for item in payload.texts]\n predictions = predict.predict(texts=texts, artifacts=artifacts)\n response = {\n \"message\": HTTPStatus.OK.phrase,\n \"status-code\": HTTPStatus.OK,\n \"data\": {\"predictions\": predictions},\n }\n return response\n
\n
We need to adhere to the PredictPayload schema when we want to user our /predict endpoint:
\n
curl -X 'POST' 'http://0.0.0.0:8000/predict' \\\n-H 'accept: application/json' \\\n-H 'Content-Type: application/json' \\\n-d '{\n \"texts\": [\n {\"text\": \"Transfer learning with transformers for text classification.\"},\n {\"text\": \"Generative adversarial networks for image generation.\"}\n ]\n }'\n
\n
\n{\n \"message\":\"OK\",\n \"method\":\"POST\",\n \"status-code\":200,\n \"timestamp\":\"2022-05-25T12:23:34.381614\",\n \"url\":\"http://0.0.0.0:8001/predict\",\n \"data\":{\n \"predictions\":[\n {\n \"input_text\":\"Transfer learning with transformers for text classification.\",\n \"predicted_tag\":\"natural-language-processing\"\n },\n {\n \"input_text\":\"Generative adversarial networks for image generation.\",\n \"predicted_tag\":\"computer-vision\"\n }\n ]\n }\n}\n
We're using pydantic's BaseModel object here because it offers built-in validation for all of our schemas. In our case, if a Text instance is less than 1 character, then our service will return the appropriate error message and code:
We can also add custom validation on a specific entity by using the @validator decorator, like we do to ensure that list of texts is not empty.
\n
class PredictPayload(BaseModel):\n texts: List[Text]\n\n@validator(\"texts\")\ndef list_must_not_be_empty(cls, value):\nif not len(value):\nraise ValueError(\"List of texts to classify cannot be empty.\")\nreturn value\n
Lastly, we can add a schema_extra object under a Config class to depict what an example PredictPayload should look like. When we do this, it automatically appears in our endpoint's documentation (click Try it out).
\n
class PredictPayload(BaseModel):\n texts: List[Text]\n\n @validator(\"texts\")\n def list_must_not_be_empty(cls, value):\n if not len(value):\n raise ValueError(\"List of texts to classify cannot be empty.\")\n return value\n\nclass Config:\nschema_extra = {\n\"example\": {\n\"texts\": [\n{\"text\": \"Transfer learning with transformers for text classification.\"},\n{\"text\": \"Generative adversarial networks in both PyTorch and TensorFlow.\"},\n]\n}\n}\n
To make our API a standalone product, we'll need to create and manage a database for our users and resources. These users will have credentials which they will use for authentication and use their privileges to be able to communicate with our service. And of course, we can display a rendered frontend to make all of this seamless with HTML forms, buttons, etc. This is exactly how the old MWML platform was built and we leveraged FastAPI to deliver high performance for 500K+ daily service requests.
\n
If you are building a product, then I highly recommending forking this generation template to get started. It includes the backbone architecture you need for your product:
\n
\n
Databases (models, migrations, etc.)
\n
Authentication via JWT
\n
Asynchronous task queue with Celery
\n
Customizable frontend via Vue JS
\n
Docker integration
\n
so much more!
\n
\n
However, for the majority of ML developers, thanks to the wide adoption of microservices, we don't need to do all of this. A well designed API service that can seamlessly communicate with all other services (framework agnostic) will fit into any process and add value to the overall product. Our main focus should be to ensure that our service is working as it should and constantly improve, which is exactly what the next cluster of lessons will focus on (testing and monitoring)
Besides wrapping our models as separate, scalable microservices, we can also have a purpose-built model server to host our models. Model servers provide a registry with an API layer to seamlessly inspect, update, serve, rollback, etc. multiple versions of models. They also offer automatic scaling to meet throughput and latency needs. Popular options include BentoML, MLFlow, TorchServe, RedisAI, Nvidia Triton Inference Server, etc.
\n
Model servers are experiencing a lot of adoption for their ability to standardize the model deployment and serving processes across the team -- enabling seamless upgrades, validation and integration.
\n
Upcoming live cohorts
\n
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.\n\n
\n Learn more\n\n
To cite this content, please use:
\n
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { APIs for Model Serving - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
We'll often want to increase the size and diversity of our training data split through data augmentation. It involves using the existing samples to generate synthetic, yet realistic, examples.
Split the dataset. We want to split our dataset first because many augmentation techniques will cause a form of data leak if we allow the generated samples to be placed across different data splits.
For example, some augmentation involves generating synonyms for certain key tokens in a sentence. If we allow the generated sentences from the same origin sentence to go into different splits, we could be potentially leaking samples with nearly identical embedding representations across our different splits.
Augment the training split. We want to apply data augmentation on only the training set because our validation and testing splits should be used to provide an accurate estimate on actual data points.
Inspect and validate. It's useless to augment just for the same of increasing our training sample size if the augmented data samples are not probable inputs that our model could encounter in production.
The exact method of data augmentation depends largely on the type of data and the application. Here are a few ways different modalities of data can be augmented:
Data Augmentation with Snorkel
General: normalization, smoothing, random noise, synthetic oversampling (SMOTE), etc.
Natural language processing (NLP): substitutions (synonyms, tfidf, embeddings, masked models), random noise, spelling errors, etc.
While the transformations on some data modalities, such as images, are easy to inspect and validate, others may introduce silent errors. For example, shifting the order of tokens in text can significantly alter the meaning (\u201cthis is really cool\u201d \u2192 \u201cis this really cool\u201d). Therefore, it\u2019s important to measure the noise that our augmentation policies will introduce and do have granular control over the transformations that take place.
# Load tokenizers and transformers\nsubstitution = naw.ContextualWordEmbsAug(model_path=\"distilbert-base-uncased\", action=\"substitute\")\ninsertion = naw.ContextualWordEmbsAug(model_path=\"distilbert-base-uncased\", action=\"insert\")\ntext = \"Conditional image generation using Variational Autoencoders and GANs.\"\n
# Substitutions\nsubstitution.augment(text)\n
\nhierarchical risk mapping using variational signals and gans.\n
Substitution doesn't seem like a great idea for us because there are certain keywords that provide strong signal for our tags so we don't want to alter those. Also, note that these augmentations are NOT deterministic and will vary every time we run them. Let's try insertion...
# Insertions\ninsertion.augment(text)\n
\nautomated conditional inverse image generation algorithms using multiple variational autoencoders and gans.\n
A little better but still quite fragile and now it can potentially insert key words that can influence false positive tags to appear. Maybe instead of substituting or inserting new tokens, let's try simply swapping machine learning related keywords with their aliases. We'll use Snorkel's transformation functions to easily achieve this.
# Flatten dict\nflattened_aliases = {}\nfor tag, aliases in aliases_by_tag.items():\n tag = replace_dash(x=tag)\n if len(aliases):\n flattened_aliases[tag] = aliases\n for alias in aliases:\n _aliases = aliases + [tag]\n _aliases.remove(alias)\n flattened_aliases[alias] = _aliases\n
print (flattened_aliases[\"natural language processing\"])\nprint (flattened_aliases[\"nlp\"])\n
\n['nlp', 'nlproc']\n['nlproc', 'natural language processing']\n
For now we'll use tags and aliases as they are in aliases_by_tag but we could account for plurality of tags using the inflect package or apply stemming before replacing aliases, etc.
# We want to match with the whole word only\nprint (\"gan\" in \"This is a gan.\")\nprint (\"gan\" in \"This is gandalf.\")\n
Now let's use snorkel's transformation_function to systematically apply this transformation to our data.
from snorkel.augmentation import transformation_function\n
@transformation_function()\ndef swap_aliases(x):\n\"\"\"Swap ML keywords with their aliases.\"\"\"\n # Find all matches\n matches = []\n for i, tag in enumerate(flattened_aliases):\n match = find_word(tag, x.text)\n if match:\n matches.append(match)\n # Swap a random match with a random alias\n if len(matches):\n match = random.choice(matches)\n tag = x.text[match.start():match.end()]\n x.text = f\"{x.text[:match.start()]}{random.choice(flattened_aliases[tag])}{x.text[match.end():]}\"\n return x\n
# Swap\nfor i in range(3):\n sample_df = pd.DataFrame([{\"text\": \"a survey of reinforcement learning for nlp tasks.\"}])\n sample_df.text = sample_df.text.apply(clean_text, lower=True, stem=False)\n print (swap_aliases(sample_df.iloc[0]).text)\n
# Undesired behavior (needs contextual insight)\nfor i in range(3):\n sample_df = pd.DataFrame([{\"text\": \"Autogenerate your CV to apply for jobs using NLP.\"}])\n sample_df.text = sample_df.text.apply(clean_text, lower=True, stem=False)\n print (swap_aliases(sample_df.iloc[0]).text)\n
\nautogenerate vision apply jobs using nlp\nautogenerate cv apply jobs using natural language processing\nautogenerate cv apply jobs using nlproc\n
Now we'll define a augmentation policy to apply our transformation functions with certain rules (how many samples to generate, whether to keep the original data point, etc.)
from snorkel.augmentation import ApplyOnePolicy, PandasTFApplier\n
text tags 0 laplacian pyramid reconstruction refinement se... computer-vision 1 extract stock sentiment news headlines project... natural-language-processing 2 big bad nlp database collection 400 nlp datasets... natural-language-processing 2 big bad natural language processing database c... natural-language-processing 2 big bad nlproc database collection 400 nlp dat... natural-language-processing
len(train_df), len(train_df_augmented)\n
\n(668, 913)\n
For now, we'll skip the data augmentation because it's quite fickle and empirically it doesn't improvement performance much. But we can see how this can be very effective once we can control what type of vocabulary to augment on and what exactly to augment with.
Warning
Regardless of what method we use, it's important to validate that we're not just augmenting for the sake of augmentation. We can do this by executing any existing data validation tests and even creating specific tests to apply on augmented data.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Data Augmentation - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/cicd/","title":"CI/CD for Machine Learning","text":""},{"location":"courses/mlops/cicd/#intuition","title":"Intuition","text":"
In the previous lesson, we learned how to manually execute our ML workloads with Jobs and Services. However, we want to be able to automatically execute these workloads when certain events occur (new data, performance regressions, elapsed time, etc.) to ensure that our models are always up to date and increasing in quality. In this lesson, we'll learn how to create continuous integration and delivery (CI/CD) pipelines to achieve an application that is capable of continual learning.
We're going to use GitHub Actions to create our CI/CD pipelines. GitHub Actions allow us to define workflows that are triggered by events (pull request, push, etc.) and execute a series of actions.
Our GitHub Actions are defined under our repository's .github/workflows directory where we have workflows for documentation (documentation.yaml), workloads (workloads.yaml) to train/validate a model and a final workflow for serving our model (serve.yaml). Let's start by understanding the structure of a workflow.
Workflows are triggered by an event, which can be something that occurs on an event (like a push or pull request), schedule (cron), manually and many more. In our application, our workloads workflow is triggered on a pull request to the main branch and then our serve workflow and documentation workflows are triggered on a push to the main branch.
Each of our workflows only have one job but if we had multiple, the jobs would all run in parallel. If we wanted to create dependent jobs, where if a particular job fails all it's dependent jobs will be skipped, then we'd use the needs keyword.
Each job contains a series of steps which are executed in order. Each step has a name, as well as actions to use from the GitHub Action marketplace and/or commands we want to run. For example, here's a look at one of the steps in our workloads job inside our workloads.yaml workflow:
# .github/workflows/testing.yml\njobs:\nworkloads:\nruns-on: ubuntu-22.04\nsteps:\n...\n# Run workloads\n- name: Workloads\nrun: |\nexport ANYSCALE_HOST=$\\{\\{ secrets.ANYSCALE_HOST \\}\\}\nexport ANYSCALE_CLI_TOKEN=$\\{\\{ secrets.ANYSCALE_CLI_TOKEN \\}\\}\nanyscale jobs submit deploy/jobs/workloads.yaml --wait\n...\n
Now that we understand the basic components of a GitHub Actions workflow, let's take a closer look at each of our workflows. Most of our workflows will require access to our Anyscale credentials so we'll start by setting those up. We can set these secrets for our repository under the Settings tab.
And our first workflow will be our workloads workflow which will be triggered on a pull request to the main branch. This means that we'll need to push our local code changes to Git and then submit a pull request to the main branch. But in order to push our code to GitHub, we'll need to first authenticate with our credentials before pushing to our repository:
git config --global user.name $GITHUB_USERNAME\ngit config --global user.email you@example.com # <-- CHANGE THIS to your email\ngit add .\ngit commit -m \"\" # <-- CHANGE THIS to your message\ngit push origin dev\n
Now you will be prompted to enter your username and password (personal access token). Follow these steps to get personal access token: New GitHub personal access token \u2192 Add a name \u2192 Toggle repo and workflow \u2192 Click Generate token (scroll down) \u2192 Copy the token and paste it when prompted for your password.
Note that we should be on a dev branch, which we set up in our setup lesson. If you're not, go ahead and run git checkout -b dev first.
And when any of our GitHub Actions workflows execute, we will be able to view them under the Actions tab of our repository. Here we'll find all the workflows that have been executed and we can inspect each one to see the details of the execution.
Our workloads workflow is triggered on a pull request to the main branch. It contains a single job that runs our model development workloads with an Anyscale Job. The steps in this job are as follows:
We start by configuring our AWS credentials so that we can push/pull from our S3 buckets. Recall that we store our model registry and results in S3 buckets so we need to be able to access them. We created an IAM role for this course so that only certain repositories can access our S3 buckets.
Next, we checkout our repository code and install our Python dependencies so that we can execute our Anyscale Job.
# Set up dependencies\n- uses: actions/checkout@v3\n- uses: actions/setup-python@v4\nwith:\npython-version: '3.10.11'\ncache: 'pip'\n- run: python3 -m pip install anyscale==0.5.128 typer==0.9.0\n
Next, we can run our Anyscale Job but note that since this will be running on a GitHub hosted runner, we need to export our Anyscale credentials first (which we already set up earlier on our repository).
# Run workloads\n- name: Workloads\nrun: |\nexport ANYSCALE_HOST=$\\{\\{ secrets.ANYSCALE_HOST \\}\\}\nexport ANYSCALE_CLI_TOKEN=$\\{\\{ secrets.ANYSCALE_CLI_TOKEN \\}\\}\nanyscale jobs submit deploy/jobs/workloads.yaml --wait\n
Recall that our Anyscale Job in the previous step saves our model registry and results to S3 buckets. So in this step, we'll read the artifacts from S3 (from our unique path using our GitHub username) and save them locally on our GitHub runner. We have a small utility script called .github/workflows/json_to_md.py to convert our JSON results to markdown tables that we can comment on our PR.
We use a GitHub Action from the marketplace to comment our results markdown tables on our PR.
# Comment results to PR\n- name: Comment training results on PR\nuses: thollander/actions-comment-pull-request@v2\nwith:\nfilePath: results/training_results.md\n- name: Comment evaluation results on PR\nuses: thollander/actions-comment-pull-request@v2\nwith:\nfilePath: results/evaluation_results.md\n
So when this workloads workflow completes, we'll have a comment on our PR (example) with our training and evaluation results. We can now collaboratively analyze the details and decide if we want to merge the PR.
Tip
We could easily extend this by retrieving evaluation results from our currently deployed model in production as well. Recall that we defined a /evaluate/ endpoint for our service that expects a dataset location and returns the evaluation results. And we can submit this request as a step in our workflow and save the results to a markdown table that we can comment on our PR.
It contains a single job that serves our model with Anyscale Services. The steps in this job are as follows:
We start by configuring our AWS credentials so that we can push/pull from our S3 buckets. Recall that we store our model registry and results in S3 buckets so we need to be able to access them.
Next, we checkout our repository code and install our Python dependencies so that we can execute our Anyscale Job.
# Set up dependencies\n- uses: actions/checkout@v3\n- uses: actions/setup-python@v4\nwith:\npython-version: '3.10.11'\ncache: 'pip'\n- run: python3 -m pip install anyscale==0.5.128 typer==0.9.0\n
Next, we can run our Anyscale Service but note that since this will be running on a GitHub hosted runner, we need to export our Anyscale credentials first (which we already set up earlier on our repository).
# Run workloads\n- name: Workloads\nrun: |\nexport ANYSCALE_HOST=$\\{\\{ secrets.ANYSCALE_HOST \\}\\}\nexport ANYSCALE_CLI_TOKEN=$\\{\\{ secrets.ANYSCALE_CLI_TOKEN \\}\\}\nanyscale service rollout --service-config-file deploy/services/serve_model.yaml\n
So when this serve workflow completes, our model will be deployed to production and we can start making inference requests with it.
Note
The anyscale service rollout command will update our existing service (if there was already one running) without changing the SECRET_TOKEN or SERVICE_ENDPOINT. So this means that our downstream applications that were making inference requests to our service can continue to do so without any changes.
Our documentation workflow is also triggered on a push to the main branch. It contains a single job that builds our docs. The steps in this job are as follows:
We checkout our repository code and install our Python dependencies so that we can build our documentation.
# Set up dependencies\n- uses: actions/checkout@v3\n- uses: actions/setup-python@v4\nwith:\npython-version: '3.10.11'\ncache: 'pip'\n- run: python3 -m pip install mkdocs==1.4.2 mkdocstrings==0.21.2 \"mkdocstrings[python]>=0.18\"\n
And with that, we're able to automatically update our ML application when ever we make changes to the code and want to trigger a new deployment. We have fully control because we can decide not to trigger an event (ex. push to main branch) if we're not satisfied with the results of our model development workloads. We can easily extend this to include other events (ex. new data, performance regressions, etc.) to trigger our workflows, as well as, integrate with more functionality around orchestration (ex. Prefect, Kubeflow, etc.), monitoring, etc.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { CI/CD workflows - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
In the previous lesson, we organized our code from our notebook into individual Python scripts. We moved our functions and classes into their respective scripts and also created new workload functions to execute the main ML workloads (ex. train_model function from madewithml/train.py script). We now want to enable users to execute these workloads from the terminal without having to know anything about our code itself.
One way to execute these workloads is to import the functions in the Python script and execute them one at a time:
from madewithml import train\ntrain.train_model(experiment_name=\"llm\", ...)\n
Caution: Don't forget to run export PYTHONPATH=$PYTHONPATH:$PWD in your terminal to ensure that Python can find the modules in our project.
While this may seem simple, it still requires us to import packages, identify the input arguments, etc. Therefore, another alternative is to place the main function call under a if __name__ == \"__main__\" conditional so that it's only executed when we run the script directly. Here we can pass in the input arguments directly into the function in the code.
However, the limitation here is that we can't choose which function from a particular script to execute. We have to set the one we want to execute under the if __name__ == \"__main__\" conditional. It's also very rigid since we have to set the input argument values in the code, unless we use a library like argparse.
# madewithml/serve.py\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--run_id\", help=\"run ID to use for serving.\")\n parser.add_argument(\"--threshold\", type=float, default=0.9, help=\"threshold for `other` class.\")\n args = parser.parse_args()\n ray.init()\n serve.run(ModelDeployment.bind(run_id=args.run_id, threshold=args.threshold))\n
Which we can call from the terminal like so (note that --threshold is optional since it has a default value):
python madewithml/serve.py --run_id $RUN_ID\n
We use argparse in our madewithml/serve.py script because it's the only workload in the script and it's a one-line function call (serve.run()).
Compared to using functions or the __main__ conditional, a much better user experience would be to execute these workloads from the terminal. In this lesson, we'll learn how to build a command-line interface (CLI) so that execute our main ML workloads.
We're going to create our CLI using Typer. It's as simple as initializing the app and then adding the appropriate decorator to each function operation we wish to use as a CLI command in our script:
You may notice that our function inputs have a lot of information besides just the input name. We'll cover typing (str, List, etc.) in our documentation lesson but for now, just know that Annotated allows us to specify metadata about the input argument's type and details about the (required) option (typer.Option).
We make all of our input arguments optional so that we can explicitly define them in our CLI commands (ex. --experiment-name).
We can also add some helpful information about the input parameter (with typer.Option(help=\"...\")) and a default value (ex. None).
With our CLI commands defined and our input arguments enriched, we can execute our workloads. Let's start by executing our train_model function by assuming that we don't know what the required input parameters are. Instead of having to look in the code, we can just do the following:
python madewithml/train.py --help\n
\nUsage: train.py [OPTIONS]\nMain train function to train our model as a distributed workload.\n
We can follow this helpful message to execute our workload with the appropriate inputs.
When developing an application, there are a lot of technical decisions and results (preprocessing, performance, etc.) that are integral to our system. How can we effectively communicate this to other developers and business stakeholders? One option is a Jupyter notebook but it's often cluttered with code and isn't very easy for non-technical team members to access and run. We need to create a dashboard that can be accessed without any technical prerequisites and effectively communicates key findings. It would be even more useful if our dashboard was interactive such that it provides utility even for the technical developers.
There are some great tooling options, such as Dash, Gradio, Streamlit, Tableau, Looker, etc. for creating dashboards to deliver data oriented insights. Traditionally, interactive dashboards were exclusively created using front-end programming languages such as HTML Javascript, CSS, etc. However, given that many developers working in machine learning are using Python, the tooling landscape has evolved to bridge this gap. These tools now allow ML developers to create interactive dashboards and visualizations in Python while offering full customization via HTML, JS, and CSS. We'll be using Streamlit to create our dashboards because of it's intuitive API, sharing capabilities and increasing community adoption.
Before we create a dashboard for our specific application, we need to learn about the different Streamlit components. Instead of going through them all in this lesson, take a few minutes and go through the API reference. It's quite short and we promise you'll be amazed at how many UI components (styled text, latex, tables, plots, etc.) you can create using just Python. We'll explore the different components in detail as they apply to creating different interactions for our specific dashboard below.
In this section, we'll display the performance of from our latest trained model. Again, we're going to keep it simple but we could also overlay more information such as improvements or regressions from previous deployments by accessing the model store.
st.header(\"\ud83d\udcca Performance\")\nperformance_fp = Path(config.CONFIG_DIR, \"performance.json\")\nperformance = utils.load_dict(filepath=performance_fp)\nst.text(\"Overall:\")\nst.write(performance[\"overall\"])\ntag = st.selectbox(\"Choose a tag: \", list(performance[\"class\"].keys()))\nst.write(performance[\"class\"][tag])\ntag = st.selectbox(\"Choose a slice: \", list(performance[\"slices\"].keys()))\nst.write(performance[\"slices\"][tag])\n
With the inference section, we want to be able to quickly predict with the latest trained model.
st.header(\"\ud83d\ude80 Inference\")\ntext = st.text_input(\"Enter text:\", \"Transfer learning with transformers for text classification.\")\nrun_id = st.text_input(\"Enter run ID:\", open(Path(config.CONFIG_DIR, \"run_id.txt\")).read())\nprediction = main.predict_tag(text=text, run_id=run_id)\nst.write(prediction)\n
Tip
Our dashboard is quite simple but we can also more comprehensive dashboards that reflect some of the core topics we covered in our machine learning canvas.
Display findings from our labeling, EDA and preprocessing stages of development.
View false +/- interactively and connect with annotation pipelines so that changes to the data can be reviewed and incorporated.
Compare performances across multiple releases to visualize improvements/regressions over time (using model store, git tags, etc.)
Sometimes we may have views that involve computationally heavy operations, such as loading data or model artifacts. It's best practice to cache these operations by wrapping them as a separate function with the @st.cache decorator. This calls for Streamlit to cache the function by the specific combination of it's inputs to deliver the respective outputs when the function is invoked with the same inputs.
We have several different options for deploying and managing our Streamlit dashboard. We could use Streamlit's sharing feature (beta) which allows us to seamlessly deploy dashboards straight from GitHub. Our dashboard will continue to stay updated as we commit changes to our repository. Another option is to deploy the Streamlit dashboard along with our API service. We can use docker-compose to spin up a separate container or simply add it to the API service's Dockerfile's ENTRYPOINT with the appropriate ports exposed. The later might be ideal, especially if your dashboard isn't meant to be public and it you want added security, performance, etc.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Dashboard - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/data-engineering/","title":"Data Engineering for Machine Learning","text":""},{"location":"courses/mlops/data-engineering/#intuition","title":"Intuition","text":"
So far we've had the convenience of using local CSV files as data source but in reality, our data can come from many disparate sources. Additionally, our processes around transforming and testing our data should ideally be moved upstream so that many different downstream processes can benefit from them. Our ML use case being just one among the many potential downstream applications. To address these shortcomings, we're going to learn about the fundamentals of data engineering and construct a modern data stack that can scale and provide high quality data for our applications.
View the data-engineering repository for all the code.
At a high level, we're going to:
Extract and Load data from sources to destinations.
Transform data for downstream applications.
This process is more commonly known as ELT, but there are variants such as ETL and reverse ETL, etc. They are all essentially the same underlying workflows but have slight differences in the order of data flow and where data is processed and stored.
Utility and simplicity
It can be enticing to set up a modern data stack in your organization, especially with all the hype. But it's very important to motivate utility and adding additional complexity:
Start with a use case that we already have data sources for and has direct impact on the business' bottom line (ex. user churn).
Start with the simplest infrastructure (source \u2192 database \u2192 report) and add complexity (in infrastructure, performance and team) as needed.
Before we start working with our data, it's important to understand the different types of systems that our data can live in. So far in this course we've worked with files, but there are several types of data systems that are widely adopted in industry for different purposes.
A data lake is a flat data management system that stores raw objects. It's a great option for inexpensive storage and has the capability to hold all types of data (unstructured, semi-structured and structured). Object stores are becoming the standard for data lakes with default options across the popular cloud providers. Unfortunately, because data is stored as objects in a data lake, it's not designed for operating on structured data.
Popular data lake options include Amazon S3, Azure Blob Storage, Google Cloud Storage, etc.
Another popular storage option is a database (DB), which is an organized collection of structured data that adheres to either:
relational schema (tables with rows and columns) often referred to as a Relational Database Management System (RDBMS) or SQL database.
non-relational (key/value, graph, etc.), often referred to as a non-relational database or NoSQL database.
A database is an online transaction processing (OLTP) system because it's typically used for day-to-day CRUD (create, read, update, delete) operations where typically information is accessed by rows. However, they're generally used to store data from one application and is not designed to hold data from across many sources for the purpose of analytics.
Popular database options include PostgreSQL, MySQL, MongoDB, Cassandra, etc.
A data warehouse (DWH) is a type of database that's designed for storing structured data from many different sources for downstream analytics and data science. It's an online analytical processing (OLAP) system that's optimized for performing operations across aggregating column values rather than accessing specific rows.
Popular data warehouse options include SnowFlake, Google BigQuery, Amazon RedShift, Hive, etc.
"},{"location":"courses/mlops/data-engineering/#extract-and-load","title":"Extract and load","text":"
The first step in our data pipeline is to extract data from a source and load it into the appropriate destination. While we could construct custom scripts to do this manually or on a schedule, an ecosystem of data ingestion tools have already standardized the entire process. They all come equipped with connectors that allow for extraction, normalization, cleaning and loading between sources and destinations. And these pipelines can be scaled, monitored, etc. all with very little to no code.
Popular data ingestion tools include Fivetran, Airbyte, Stitch, etc.
We're going to use the open-source tool Airbyte to create connections between our data sources and destinations. Let's set up Airbyte and define our data sources. As we progress in this lesson, we'll set up our destinations and create connections to extract and load data.
Ensure that we have Docker installed, but if not, download it here. For Windows users, be sure to have these configurations enabled.
In a parent directory, outside our project directory for the MLOps course, execute the following commands to load the Airbyte repository locally and launch the service.
Our data sources we want to extract from can be from anywhere. They could come from 3rd party apps, files, user click streams, physical devices, data lakes, databases, data warehouses, etc. But regardless of the source of our data, they type of data should fit into one of these categories:
structured: organized data stored in an explicit structure (ex. tables)
semi-structured: data with some structure but no formal schema or data types (web pages, CSV, JSON, etc.)
unstructured: qualitative data with no formal structure (text, images, audio, etc.)
For our application, we'll define two data sources:
projects.csv: data containing projects with their ID, create date, title and description.
tags.csv: labels for each of project IDs in projects.csv
Ideally, these data assets would be retrieved from a database that contains projects that we extracted and perhaps another database that stores labels from our labeling team's workflows. However, for simplicity we'll use CSV files to demonstrate how to define a data source.
"},{"location":"courses/mlops/data-engineering/#define-file-source-in-airbyte","title":"Define file source in Airbyte","text":"
We'll start our ELT process by defining the data source in Airbyte:
On our Airbyte UI, click on Sources on the left menu. Then click the + New source button on the top right corner.
Click on the Source type dropdown and choose File. This will open a view to define our file data source.
Once we know the source we want to extract data from, we need to decide the destination to load it. The choice depends on what our downstream applications want to be able to do with the data. And it's also common to store data in one location (ex. data lake) and move it somewhere else (ex. data warehouse) for specific processing.
"},{"location":"courses/mlops/data-engineering/#set-up-google-bigquery","title":"Set up Google BigQuery","text":"
Our destination will be a data warehouse since we'll want to use the data for downstream analytical and machine learning applications. We're going to use Google BigQuery which is free under Google Cloud's free tier for up to 10 GB storage and 1TB of queries (which is significantly more than we'll ever need for our purpose).
Log into your Google account and then head over to Google CLoud. If you haven't already used Google Cloud's free trial, you'll have to sign up. It's free and you won't be autocharged unless you manually upgrade your account. Once the trial ends, we'll still have the free tier which is more than plenty for us.
Go to the Google BigQuery page and click on the Go to console button.
We can create a new project by following these instructions which will lead us to the create project page.
Project name: made-with-ml # Google will append a unique ID to the end of it\nLocation: No organization\n
Once the project has been created, refresh the page and we should see it (along with few other default projects from Google).
Most cloud providers will allow us to do everything via console but also programmatically via API, Python, etc. For example, we manually create a project but we could've also done so with code as shown here.
"},{"location":"courses/mlops/data-engineering/#define-bigquery-destination-in-airbyte","title":"Define BigQuery destination in Airbyte","text":"
Next, we need to establish the connection between Airbyte and BigQuery so that we can load the extracted data to the destination. In order to authenticate our access to BigQuery with Airbyte, we'll need to create a service account and generate a secret key. This is basically creating an identity with certain access that we can use for verification. Follow these instructions to create a service and generate the key file (JSON). Note down the location of this file because we'll be using it throughout this lesson. For example ours is /Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json.
On our Airbyte UI, click on Destinations on the left menu. Then click the + New destination button on the top right corner.
Click on the Destination type dropdown and choose BigQuery. This will open a view to define our file data source.
Name: BigQuery\nDefault Dataset ID: mlops_course # where our data will go inside our BigQuery project\nProject ID: made-with-ml-XXXXXX # REPLACE this with your Google BiqQuery Project ID\nCredentials JSON: SERVICE-ACCOUNT-KEY.json # REPLACE this with your service account JSON location\nDataset location: US # select US or EU, all other options will not be compatible with dbt later\n
Click the Set up destination button and our data destination will be tested and saved.
So we've set up our data sources (public CSV files) and destination (Google BigQuery data warehouse) but they haven't been connected yet. To create the connection, we need to think about a few aspects.
How often do we want to extract data from the sources and load it into the destination?
batch: extracting data in batches, usually following a schedule (ex. daily) or when an event of interest occurs (ex. new data count)
streaming: extracting data in a continuous stream (using tools like Kafka, Kinesis, etc.)
Micro-batch
As we keep decreasing the time between batch ingestion (ex. towards 0), do we have stream ingestion? Not exactly. Batch processing is deliberately deciding to extract data from a source at a given interval. As that interval becomes <15 minutes, it's referred to as a micro-batch (many data warehouses allow for batch ingestion every 5 minutes). However, with stream ingestion, the extraction process is continuously on and events will keep being ingested.
Start simple
In general, it's a good idea to start with batch ingestion for most applications and slowly add the complexity of streaming ingestion (and additional infrastructure). This was we can prove that downstream applications are finding value from the data source and evolving to streaming later should only improve things.
We'll learn more about the different system design implications of batch vs. stream in our systems design lesson.
"},{"location":"courses/mlops/data-engineering/#connecting-file-source-to-bigquery-destination","title":"Connecting File source to BigQuery destination","text":"
Now we're ready to create the connection between our sources and destination:
On our Airbyte UI, click on Connections on the left menu. Then click the + New connection button on the top right corner.
Under Select a existing source, click on the Source dropdown and choose Projects and click Use existing source.
Under Select a existing destination, click on the Destination dropdown and choose BigQuery and click Use existing destination.
Click the Set up connection button and our connection will be tested and saved.
Repeat the same for our Tags source with the same BigQuery destination.
Notice that our sync mode is Full refresh | Overwrite which means that every time we sync data from our source, it'll overwrite the existing data in our destination. As opposed to Full refresh | Append which will add entries from the source to bottom of the previous syncs.
\nid\n created_on\n title\n description\n 0\n 6\n 2020-02-20 06:43:18\n Comparison between YOLO and RCNN on real world...\n Bringing theory to experiment is cool. We can ...\n 1\n 7\n 2020-02-20 06:47:21\n Show, Infer & Tell: Contextual Inference for C...\n The beauty of the work lies in the way it arch...\n 2\n 9\n 2020-02-24 16:24:45\n Awesome Graph Classification\n A collection of important graph embedding, cla...\n 3\n 15\n 2020-02-28 23:55:26\n Awesome Monte Carlo Tree Search\n A curated list of Monte Carlo tree search papers...\n 4\n 19\n 2020-03-03 13:54:31\n Diffusion to Vector\n Reference implementation of Diffusion2Vec (Com...\n
With the advent of cheap storage and cloud SaaS options to manage them, it's become a best practice to store raw data into data lakes. This allows for storage of raw, potentially unstructured, data without having to justify storage with downstream applications. When we do need to transform and process the data, we can move it to a data warehouse so can perform those operations efficiently.
Once we've extracted and loaded our data, we need to transform the data so that it's ready for downstream applications. These transformations are different from the preprocessing we've seen before but are instead reflective of business logic that's agnostic to downstream applications. Common transformations include defining schemas, filtering, cleaning and joining data across tables, etc. While we could do all of these things with SQL in our data warehouse (save queries as tables or views), dbt delivers production functionality around version control, testing, documentation, packaging, etc. out of the box. This becomes crucial for maintaining observability and high quality data workflows.
Popular transformation tools include dbt, Matillion, custom jinja templated SQL, etc.
Note
In addition to data transformations, we can also process the data using large-scale analytics engines like Spark, Flink, etc.
Now we're ready to transform our data in our data warehouse using dbt. We'll be using a developer account on dbt Cloud (free), which provides us with an IDE, unlimited runs, etc.
We'll learn how to use the dbt-core in our orchestration lesson. Unlike dbt Cloud, dbt core is completely open-source and we can programmatically connect to our data warehouse and perform transformations.
Create a free account and verify it.
Go to https://cloud.getdbt.com/ to get set up.
Click continue and choose BigQuery as the database.
Click Upload a Service Account JSON file and upload our file to autopopulate everything.
Click the Test > Continue.
Click Managed repository and name it dbt-transforms (or anything else you want).
Click Create > Continue > Skip and complete.
This will open the project page and click >_ Start Developing button.
This will open the IDE where we can click \ud83d\uddc2 initialize your project.
Now we're ready to start developing our models:
Click the \u00b7\u00b7\u00b7 next to the models directory on the left menu.
Click New folder called models/labeled_projects.
Create a New file under models/labeled_projects called labeled_projects.sql.
Repeat for another file under models/labeled_projects called schema.yml.
Inside our models/labeled_projects/labeled_projects.sql file we'll create a view that joins our project data with the appropriate tags. This will create the labeled data necessary for downstream applications such as machine learning models. Here we're joining based on the matching id between the projects and tags:
-- models/labeled_projects/labeled_projects.sql\nSELECT p.id, created_on, title, description, tag\nFROM `made-with-ml-XXXXXX.mlops_course.projects` p -- REPLACE\nLEFT JOIN `made-with-ml-XXXXXX.mlops_course.tags` t -- REPLACE\nON p.id = t.id\n
We can view the queried results by clicking the Preview button and view the data lineage as well.
Inside our models/labeled_projects/schema.yml file we'll define the schemas for each of the features in our transformed data. We also define several tests that each feature should pass. View the full list of dbt tests but note that we'll use Great Expectations for more comprehensive tests when we orchestrate all these data workflows in our orchestration lesson.
# models/labeled_projects/schema.yml\n\nversion: 2\n\nmodels:\n- name: labeled_projects\ndescription: \"Tags for all projects\"\ncolumns:\n- name: id\ndescription: \"Unique ID of the project.\"\ntests:\n- unique\n- not_null\n- name: title\ndescription: \"Title of the project.\"\ntests:\n- not_null\n- name: description\ndescription: \"Description of the project.\"\ntests:\n- not_null\n- name: tag\ndescription: \"Labeled tag for the project.\"\ntests:\n- not_null\n
At the bottom of the IDE, we can execute runs based on the transformations we've defined. We'll run each of the following commands and once they finish, we can see the transformed data inside our data warehouse.
dbt run\ndbt test\n
Once these commands run successfully, we're ready to move our transformations to a production environment where we can insert this view in our data warehouse.
There is so much more to dbt so be sure to check out their official documentation to really customize any workflows. And be sure to check out our orchestration lesson where we'll programmatically create and execute our dbt transformations.
Hopefully we created our data stack for the purpose of gaining some actionable insight about our business, users, etc. Because it's these use cases that dictate which sources of data we extract from, how often and how that data is stored and transformed. Downstream applications of our data typically fall into one of these categories:
data analytics: use cases focused on reporting trends, aggregate views, etc. via charts, dashboards, etc.for the purpose of providing operational insight for business stakeholders.
\ud83d\udee0\u00a0 Popular tools: Tableau, Looker, Metabase, Superset, etc.
machine learning: use cases centered around using the transformed data to construct predictive models (forecasting, personalization, etc.).
While it's very easy to extract data from our data warehouse:
pip install google-cloud-bigquery==1.21.0\n
from google.cloud import bigquery\nfrom google.oauth2 import service_account\n\n# Replace these with your own values\nproject_id = \"made-with-ml-XXXXXX\" # REPLACE\nSERVICE_ACCOUNT_KEY_JSON = \"/Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json\" # REPLACE\n\n# Establish connection\ncredentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_KEY_JSON)\nclient = bigquery.Client(credentials= credentials, project=project_id)\n\n# Query data\nquery_job = client.query(\"\"\"\n SELECT *\n FROM mlops_course.labeled_projects\"\"\")\nresults = query_job.result()\nresults.to_dataframe().head()\n
id created_on title description tag 0 1994.0 2020-07-29 04:51:30 Understanding the Effectivity of Ensembles in ... The report explores the ideas presented in Dee... computer-vision 1 1506.0 2020-06-19 06:26:17 Using GitHub Actions for MLOps & Data Science A collection of resources on how to facilitate... mlops 2 807.0 2020-05-11 02:25:51 Introduction to Machine Learning Problem Framing This course helps you frame machine learning (... mlops 3 1204.0 2020-06-05 22:56:38 Snaked: Classifying Snake Species using Images Proof of concept that it is possible to identi... computer-vision 4 1706.0 2020-07-04 11:05:28 PokeZoo A deep learning based web-app developed using ... computer-vision
Warning
Check out our notebook where we extract the transformed data from our data warehouse. We do this in a separate notebook because it requires the google-cloud-bigquery package and until dbt loosens it's Jinja versioning constraints... it'll have to be done in a separate environment. However, downstream applications are typically analytics or ML applications which have their own environments anyway so these conflicts are not inhibiting.
many of the analytics (ex. dashboards) and machine learning solutions (ex. feature stores) allow for easy connection to our data warehouses so that workflows can be triggered when an event occurs or on a schedule. We're going to take this a step further in the next lesson where we'll use a central orchestration platform to control all these workflows.
Analytics first, then ML
It's a good idea for the first several applications to be analytics and reporting based in order to establish a robust data stack. These use cases typically just involve displaying data aggregations and trends, as opposed to machine learning systems that involve additional complex infrastructure and workflows.
When we create complex data workflows like this, observability becomes a top priority. Data observability is the general concept of understanding the condition of data in our system and it involves:
data quality: testing and monitoring our data quality after every step (schemas, completeness, recency, etc.).
data lineage: mapping the where data comes from and how it's being transformed as it moves through our pipelines.
discoverability: enabling discovery of the different data sources and features for downstream applications.
privacy + security: are the different data assets treated and restricted appropriately amongst the applications?
Popular observability tools include Monte Carlo, Bigeye, etc.
The data stack ecosystem to create the robust data workflows is growing and maturing. However, it can be overwhelming when it comes to choosing the best tooling options, especially as needs change over time. Here are a few important factors to consider when making a tooling decision in this space:
What is the cost per time per employee? Some of the tooling options can rack up quite the bill!
Does the tool have the proper connectors to integrate with our data sources and the rest of the stack?
Does the tool fit with our team's technical aptitude (SQL, Spark, Python, etc.)?
What kind of support does the tool offer (enterprise, community, etc.)?
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Data engineering - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/data-stack/","title":"Data Stack for Machine Learning","text":""},{"location":"courses/mlops/data-stack/#intuition","title":"Intuition","text":"
So far we've had the convenience of using local CSV files as data source but in reality, our data can come from many disparate sources. Additionally, our processes around transforming and testing our data should ideally be moved upstream so that many different downstream processes can benefit from them. Our ML use case being just one among the many potential downstream applications. To address these shortcomings, we're going to learn about the fundamentals of data engineering and construct a modern data stack that can scale and provide high quality data for our applications.
View the data-engineering repository for all the code.
At a high level, we're going to:
Extract and Load data from sources to destinations.
Transform data for downstream applications.
This process is more commonly known as ELT, but there are variants such as ETL and reverse ETL, etc. They are all essentially the same underlying workflows but have slight differences in the order of data flow and where data is processed and stored.
Utility and simplicity
It can be enticing to set up a modern data stack in your organization, especially with all the hype. But it's very important to motivate utility and adding additional complexity:
Start with a use case that we already have data sources for and has direct impact on the business' bottom line (ex. user churn).
Start with the simplest infrastructure (source \u2192 database \u2192 report) and add complexity (in infrastructure, performance and team) as needed.
Before we start working with our data, it's important to understand the different types of systems that our data can live in. So far in this course we've worked with files, but there are several types of data systems that are widely adopted in industry for different purposes.
A data lake is a flat data management system that stores raw objects. It's a great option for inexpensive storage and has the capability to hold all types of data (unstructured, semi-structured and structured). Object stores are becoming the standard for data lakes with default options across the popular cloud providers. Unfortunately, because data is stored as objects in a data lake, it's not designed for operating on structured data.
Popular data lake options include Amazon S3, Azure Blob Storage, Google Cloud Storage, etc.
Another popular storage option is a database (DB), which is an organized collection of structured data that adheres to either:
relational schema (tables with rows and columns) often referred to as a Relational Database Management System (RDBMS) or SQL database.
non-relational (key/value, graph, etc.), often referred to as a non-relational database or NoSQL database.
A database is an online transaction processing (OLTP) system because it's typically used for day-to-day CRUD (create, read, update, delete) operations where typically information is accessed by rows. However, they're generally used to store data from one application and is not designed to hold data from across many sources for the purpose of analytics.
Popular database options include PostgreSQL, MySQL, MongoDB, Cassandra, etc.
A data warehouse (DWH) is a type of database that's designed for storing structured data from many different sources for downstream analytics and data science. It's an online analytical processing (OLAP) system that's optimized for performing operations across aggregating column values rather than accessing specific rows.
Popular data warehouse options include SnowFlake, Google BigQuery, Amazon RedShift, Hive, etc.
"},{"location":"courses/mlops/data-stack/#extract-and-load","title":"Extract and load","text":"
The first step in our data pipeline is to extract data from a source and load it into the appropriate destination. While we could construct custom scripts to do this manually or on a schedule, an ecosystem of data ingestion tools have already standardized the entire process. They all come equipped with connectors that allow for extraction, normalization, cleaning and loading between sources and destinations. And these pipelines can be scaled, monitored, etc. all with very little to no code.
Popular data ingestion tools include Fivetran, Airbyte, Stitch, etc.
We're going to use the open-source tool Airbyte to create connections between our data sources and destinations. Let's set up Airbyte and define our data sources. As we progress in this lesson, we'll set up our destinations and create connections to extract and load data.
Ensure that we still have Docker installed from our Docker lesson but if not, download it here. For Windows users, be sure to have these configurations enabled.
In a parent directory, outside our project directory for the MLOps course, execute the following commands to load the Airbyte repository locally and launch the service.
Our data sources we want to extract from can be from anywhere. They could come from 3rd party apps, files, user click streams, physical devices, data lakes, databases, data warehouses, etc. But regardless of the source of our data, they type of data should fit into one of these categories:
structured: organized data stored in an explicit structure (ex. tables)
semi-structured: data with some structure but no formal schema or data types (web pages, CSV, JSON, etc.)
unstructured: qualitative data with no formal structure (text, images, audio, etc.)
For our application, we'll define two data sources:
projects.csv: data containing projects with their ID, create date, title and description.
tags.csv: labels for each of project IDs in projects.csv
Ideally, these data assets would be retrieved from a database that contains projects that we extracted and perhaps another database that stores labels from our labeling team's workflows. However, for simplicity we'll use CSV files to demonstrate how to define a data source.
"},{"location":"courses/mlops/data-stack/#define-file-source-in-airbyte","title":"Define file source in Airbyte","text":"
We'll start our ELT process by defining the data source in Airbyte:
On our Airbyte UI, click on Sources on the left menu. Then click the + New source button on the top right corner.
Click on the Source type dropdown and choose File. This will open a view to define our file data source.
Once we know the source we want to extract data from, we need to decide the destination to load it. The choice depends on what our downstream applications want to be able to do with the data. And it's also common to store data in one location (ex. data lake) and move it somewhere else (ex. data warehouse) for specific processing.
"},{"location":"courses/mlops/data-stack/#set-up-google-bigquery","title":"Set up Google BigQuery","text":"
Our destination will be a data warehouse since we'll want to use the data for downstream analytical and machine learning applications. We're going to use Google BigQuery which is free under Google Cloud's free tier for up to 10 GB storage and 1TB of queries (which is significantly more than we'll ever need for our purpose).
Log into your Google account and then head over to Google CLoud. If you haven't already used Google Cloud's free trial, you'll have to sign up. It's free and you won't be autocharged unless you manually upgrade your account. Once the trial ends, we'll still have the free tier which is more than plenty for us.
Go to the Google BigQuery page and click on the Go to console button.
We can create a new project by following these instructions which will lead us to the create project page.
Project name: made-with-ml # Google will append a unique ID to the end of it\nLocation: No organization\n
Once the project has been created, refresh the page and we should see it (along with few other default projects from Google).
Most cloud providers will allow us to do everything via console but also programmatically via API, Python, etc. For example, we manually create a project but we could've also done so with code as shown here.
"},{"location":"courses/mlops/data-stack/#define-bigquery-destination-in-airbyte","title":"Define BigQuery destination in Airbyte","text":"
Next, we need to establish the connection between Airbyte and BigQuery so that we can load the extracted data to the destination. In order to authenticate our access to BigQuery with Airbyte, we'll need to create a service account and generate a secret key. This is basically creating an identity with certain access that we can use for verification. Follow these instructions to create a service and generate the key file (JSON). Note down the location of this file because we'll be using it throughout this lesson. For example ours is /Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json.
On our Airbyte UI, click on Destinations on the left menu. Then click the + New destination button on the top right corner.
Click on the Destination type dropdown and choose BigQuery. This will open a view to define our file data source.
Name: BigQuery\nDefault Dataset ID: mlops_course # where our data will go inside our BigQuery project\nProject ID: made-with-ml-XXXXXX # REPLACE this with your Google BiqQuery Project ID\nCredentials JSON: SERVICE-ACCOUNT-KEY.json # REPLACE this with your service account JSON location\nDataset location: US # select US or EU, all other options will not be compatible with dbt later\n
Click the Set up destination button and our data destination will be tested and saved.
So we've set up our data sources (public CSV files) and destination (Google BigQuery data warehouse) but they haven't been connected yet. To create the connection, we need to think about a few aspects.
How often do we want to extract data from the sources and load it into the destination?
batch: extracting data in batches, usually following a schedule (ex. daily) or when an event of interest occurs (ex. new data count)
streaming: extracting data in a continuous stream (using tools like Kafka, Kinesis, etc.)
Micro-batch
As we keep decreasing the time between batch ingestion (ex. towards 0), do we have stream ingestion? Not exactly. Batch processing is deliberately deciding to extract data from a source at a given interval. As that interval becomes <15 minutes, it's referred to as a micro-batch (many data warehouses allow for batch ingestion every 5 minutes). However, with stream ingestion, the extraction process is continuously on and events will keep being ingested.
Start simple
In general, it's a good idea to start with batch ingestion for most applications and slowly add the complexity of streaming ingestion (and additional infrastructure). This was we can prove that downstream applications are finding value from the data source and evolving to streaming later should only improve things.
We'll learn more about the different system design implications of batch vs. stream in our systems design lesson.
"},{"location":"courses/mlops/data-stack/#connecting-file-source-to-bigquery-destination","title":"Connecting File source to BigQuery destination","text":"
Now we're ready to create the connection between our sources and destination:
On our Airbyte UI, click on Connections on the left menu. Then click the + New connection button on the top right corner.
Under Select a existing source, click on the Source dropdown and choose Projects and click Use existing source.
Under Select a existing destination, click on the Destination dropdown and choose BigQuery and click Use existing destination.
Click the Set up connection button and our connection will be tested and saved.
Repeat the same for our Tags source with the same BigQuery destination.
Notice that our sync mode is Full refresh | Overwrite which means that every time we sync data from our source, it'll overwrite the existing data in our destination. As opposed to Full refresh | Append which will add entries from the source to bottom of the previous syncs.
\nid\n created_on\n title\n description\n 0\n 6\n 2020-02-20 06:43:18\n Comparison between YOLO and RCNN on real world...\n Bringing theory to experiment is cool. We can ...\n 1\n 7\n 2020-02-20 06:47:21\n Show, Infer & Tell: Contextual Inference for C...\n The beauty of the work lies in the way it arch...\n 2\n 9\n 2020-02-24 16:24:45\n Awesome Graph Classification\n A collection of important graph embedding, cla...\n 3\n 15\n 2020-02-28 23:55:26\n Awesome Monte Carlo Tree Search\n A curated list of Monte Carlo tree search papers...\n 4\n 19\n 2020-03-03 13:54:31\n Diffusion to Vector\n Reference implementation of Diffusion2Vec (Com...\n
With the advent of cheap storage and cloud SaaS options to manage them, it's become a best practice to store raw data into data lakes. This allows for storage of raw, potentially unstructured, data without having to justify storage with downstream applications. When we do need to transform and process the data, we can move it to a data warehouse so can perform those operations efficiently.
Once we've extracted and loaded our data, we need to transform the data so that it's ready for downstream applications. These transformations are different from the preprocessing we've seen before but are instead reflective of business logic that's agnostic to downstream applications. Common transformations include defining schemas, filtering, cleaning and joining data across tables, etc. While we could do all of these things with SQL in our data warehouse (save queries as tables or views), dbt delivers production functionality around version control, testing, documentation, packaging, etc. out of the box. This becomes crucial for maintaining observability and high quality data workflows.
Popular transformation tools include dbt, Matillion, custom jinja templated SQL, etc.
Note
In addition to data transformations, we can also process the data using large-scale analytics engines like Spark, Flink, etc.
Now we're ready to transform our data in our data warehouse using dbt. We'll be using a developer account on dbt Cloud (free), which provides us with an IDE, unlimited runs, etc.
We'll learn how to use the dbt-core in our orchestration lesson. Unlike dbt Cloud, dbt core is completely open-source and we can programmatically connect to our data warehouse and perform transformations.
Create a free account and verify it.
Go to https://cloud.getdbt.com/ to get set up.
Click continue and choose BigQuery as the database.
Click Upload a Service Account JSON file and upload our file to autopopulate everything.
Click the Test > Continue.
Click Managed repository and name it dbt-transforms (or anything else you want).
Click Create > Continue > Skip and complete.
This will open the project page and click >_ Start Developing button.
This will open the IDE where we can click \ud83d\uddc2 initialize your project.
Now we're ready to start developing our models:
Click the \u00b7\u00b7\u00b7 next to the models directory on the left menu.
Click New folder called models/labeled_projects.
Create a New file under models/labeled_projects called labeled_projects.sql.
Repeat for another file under models/labeled_projects called schema.yml.
Inside our models/labeled_projects/labeled_projects.sql file we'll create a view that joins our project data with the appropriate tags. This will create the labeled data necessary for downstream applications such as machine learning models. Here we're joining based on the matching id between the projects and tags:
-- models/labeled_projects/labeled_projects.sql\nSELECT p.id, created_on, title, description, tag\nFROM `made-with-ml-XXXXXX.mlops_course.projects` p -- REPLACE\nLEFT JOIN `made-with-ml-XXXXXX.mlops_course.tags` t -- REPLACE\nON p.id = t.id\n
We can view the queried results by clicking the Preview button and view the data lineage as well.
Inside our models/labeled_projects/schema.yml file we'll define the schemas for each of the features in our transformed data. We also define several tests that each feature should pass. View the full list of dbt tests but note that we'll use Great Expectations for more comprehensive tests when we orchestrate all these data workflows in our orchestration lesson.
# models/labeled_projects/schema.yml\n\nversion: 2\n\nmodels:\n- name: labeled_projects\ndescription: \"Tags for all projects\"\ncolumns:\n- name: id\ndescription: \"Unique ID of the project.\"\ntests:\n- unique\n- not_null\n- name: title\ndescription: \"Title of the project.\"\ntests:\n- not_null\n- name: description\ndescription: \"Description of the project.\"\ntests:\n- not_null\n- name: tag\ndescription: \"Labeled tag for the project.\"\ntests:\n- not_null\n
At the bottom of the IDE, we can execute runs based on the transformations we've defined. We'll run each of the following commands and once they finish, we can see the transformed data inside our data warehouse.
dbt run\ndbt test\n
Once these commands run successfully, we're ready to move our transformations to a production environment where we can insert this view in our data warehouse.
There is so much more to dbt so be sure to check out their official documentation to really customize any workflows. And be sure to check out our orchestration lesson where we'll programmatically create and execute our dbt transformations.
Hopefully we created our data stack for the purpose of gaining some actionable insight about our business, users, etc. Because it's these use cases that dictate which sources of data we extract from, how often and how that data is stored and transformed. Downstream applications of our data typically fall into one of these categories:
data analytics: use cases focused on reporting trends, aggregate views, etc. via charts, dashboards, etc.for the purpose of providing operational insight for business stakeholders.
\ud83d\udee0\u00a0 Popular tools: Tableau, Looker, Metabase, Superset, etc.
machine learning: use cases centered around using the transformed data to construct predictive models (forecasting, personalization, etc.).
While it's very easy to extract data from our data warehouse:
pip install google-cloud-bigquery==1.21.0\n
from google.cloud import bigquery\nfrom google.oauth2 import service_account\n\n# Replace these with your own values\nproject_id = \"made-with-ml-XXXXXX\" # REPLACE\nSERVICE_ACCOUNT_KEY_JSON = \"/Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json\" # REPLACE\n\n# Establish connection\ncredentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_KEY_JSON)\nclient = bigquery.Client(credentials= credentials, project=project_id)\n\n# Query data\nquery_job = client.query(\"\"\"\n SELECT *\n FROM mlops_course.labeled_projects\"\"\")\nresults = query_job.result()\nresults.to_dataframe().head()\n
id created_on title description tag 0 1994.0 2020-07-29 04:51:30 Understanding the Effectivity of Ensembles in ... The report explores the ideas presented in Dee... computer-vision 1 1506.0 2020-06-19 06:26:17 Using GitHub Actions for MLOps & Data Science A collection of resources on how to facilitate... mlops 2 807.0 2020-05-11 02:25:51 Introduction to Machine Learning Problem Framing This course helps you frame machine learning (... mlops 3 1204.0 2020-06-05 22:56:38 Snaked: Classifying Snake Species using Images Proof of concept that it is possible to identi... computer-vision 4 1706.0 2020-07-04 11:05:28 PokeZoo A deep learning based web-app developed using ... computer-vision
Warning
Check out our notebook where we extract the transformed data from our data warehouse. We do this in a separate notebook because it requires the google-cloud-bigquery package and until dbt loosens it's Jinja versioning constraints... it'll have to be done in a separate environment. However, downstream applications are typically analytics or ML applications which have their own environments anyway so these conflicts are not inhibiting.
many of the analytics (ex. dashboards) and machine learning solutions (ex. feature stores) allow for easy connection to our data warehouses so that workflows can be triggered when an event occurs or on a schedule. We're going to take this a step further in the next lesson where we'll use a central orchestration platform to control all these workflows.
Analytics first, then ML
It's a good idea for the first several applications to be analytics and reporting based in order to establish a robust data stack. These use cases typically just involve displaying data aggregations and trends, as opposed to machine learning systems that involve additional complex infrastructure and workflows.
When we create complex data workflows like this, observability becomes a top priority. Data observability is the general concept of understanding the condition of data in our system and it involves:
data quality: testing and monitoring our data quality after every step (schemas, completeness, recency, etc.).
data lineage: mapping the where data comes from and how it's being transformed as it moves through our pipelines.
discoverability: enabling discovery of the different data sources and features for downstream applications.
privacy + security: are the different data assets treated and restricted appropriately amongst the applications?
Popular observability tools include Monte Carlo, Bigeye, etc.
The data stack ecosystem to create the robust data workflows is growing and maturing. However, it can be overwhelming when it comes to choosing the best tooling options, especially as needs change over time. Here are a few important factors to consider when making a tooling decision in this space:
What is the cost per time per employee? Some of the tooling options can rack up quite the bill!
Does the tool have the proper connectors to integrate with our data sources and the rest of the stack?
Does the tool fit with our team's technical aptitude (SQL, Spark, Python, etc.)?
What kind of support does the tool offer (enterprise, community, etc.)?
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Data Stack for Machine Learning - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/distributed-data/","title":"Distributed Data Processing","text":""},{"location":"courses/mlops/distributed-data/#intuition","title":"Intuition","text":"
So far we've performed our data processing operations on a single machine. Our dataset was able to fit into a single Pandas DataFrame and we were able to perform our operations in a single Python process. But what if our dataset was too large to fit into a single machine? We would need to distribute our data processing operations across multiple machines. And with the increasing trend in ML for larger unstructured datasets and larger models (LLMs), we can quickly outgrow our single machine constraints and will need to go distributed.
Note
Our dataset is intentionally small for this course so that we can quickly execute the code. But with our distributed set up in this lesson, we can easily switch to a mcuh larger dataset and the code will continue to execute perfectly. And if we add more compute resources, we can scale our data processing operations to be even faster with no changes to our code.
There are many frameworks for distributed computing, such as Ray, Dask, Modin, Spark, etc. All of these are great options but for our application we want to choose a framework that is will allow us to scale our data processing operations with minimal changes to our existing code and all in Python. We also want to choose a framework that will integrate well when we want to distributed our downstream workloads (training, tuning, serving, etc.).
To address these needs, we'll be using Ray, a distributed computing framework that makes it easy to scale your Python applications. It's a general purpose framework that can be used for a variety of applications but we'll be using it for our data processing operations first (and more later). And it also has great integrations with the previously mentioned distributed data processing frameworks (Dask, Modin, Spark).
The only setup we have to do is set Ray to preserve order when acting on our data. This is important for ensuring reproducible and deterministic results.
We'll start by ingesting our dataset. Ray has a range of input/output functions that supports all major data formats and sources.
# Data ingestion\nds = ray.data.read_csv(DATASET_LOC)\nds = ds.random_shuffle(seed=1234)\nds.take(1)\n
\n[{'id': 2166,\n 'created_on': datetime.datetime(2020, 8, 17, 5, 19, 41),\n 'title': 'Pix2Pix',\n 'description': 'Tensorflow 2.0 Implementation of the paper Image-to-Image Translation using Conditional GANs by Philip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei A. Efros.',\n 'tag': 'computer-vision'}]\n
Next, we'll split our dataset into our training and validation splits. Ray has a built-in train_test_split function but we're using a modified version so that we can stratify our split based on the tag column.
And finally, we're ready to preprocess our data splits. One of the advantages of using Ray is that we won't have to change anything to our original Pandas-based preprocessing function we implemented in the previous lesson. Instead, we can use it directly with Ray's map_batches utility to map our preprocessing function across batches in our data in a distributed manner.
# Mapping\ntags = train_ds.unique(column=\"tag\")\nclass_to_index = {tag: i for i, tag in enumerate(tags)}\n
The last step in achieving reproducibility is to deploy our versioned code and artifacts in a reproducible environment. This goes well beyond the virtual environment we configured for our Python applications because there are system-level specifications (operating system, required implicit packages, etc.) we aren't capturing. We want to be able to encapsulate all the requirements we need so that there are no external dependencies that would prevent someone else from reproducing our exact application.
There are actually quite a few solutions for system-level reproducibility (VMs, container engines, etc.) but the Docker container engine is by far the most popular for several key advantages:
reproducibility via Dockerfile with explicit instructions to deploy our application in a specific system.
isolation via containers as to not affect other applications that may also run on the same underlying operating system.
and many more advantages including size (no separate OS needed for each application), speed, Docker Hub, etc.
We're going to use Docker to deploy our application locally in an isolated, reproducible and scalable fashion. Once we do this, any machine with the Docker engine installed can reproduce our work. However, there is so much more to Docker, which you can explore in the docs, that goes beyond what we'll need.
Before we install Docker, let's take a look at how the container engine works on top our operating system, which can be our local hardware or something managed on the cloud.
The Docker container engine is responsible for spinning up configured containers, which contains our application and it's dependencies (binaries, libraries, etc.). The container engine is very efficient in that it doesn't need to create a separate operating system for each containerized application. This also means that our containers can share the system's resources via the Docker engine.
Now we're ready to install Docker based on our operating system. Once installed, we can start the Docker Desktop which will allow us to create and deploy our containerized applications.
The first step is to build a docker image which has the application and all it's specified dependencies. We can create this image using a Dockerfile which outlines a set of instructions. These instructions essentially build read-only image layers on top of each other to construct our entire image. Let's take a look at our application's Dockerfile and the image layers it creates.
The first line we'll write in our Dockerfile specifies the base image we want to pull FROM. Here we want to use the base image for running Python based applications and specifically for Python 3.7 with the slim variant. Since we're only deploying a Python application, this slim variant with minimal packages satisfies our requirements while keeping the size of the image layer low.
# Base image\nFROM python:3.7-slim\n
Next we're going to install our application dependencies. First, we'll COPY the required files from our local file system so we can use them for installation. Alternatively, if we were running on some remote infrastructure, we could've pulled from a remote git host. Once we have our files, we can install the packages required to install our application's dependencies using the RUN command. Once we're done using the packages, we can remove them to keep our image layer's size to a minimum.
Since our application (API) requires PORT 8000 to be open, we need to specify in our Dockerfile to expose it.
# Export ports\nEXPOSE 8000\n
The final step in building our image is to specify the executable to be run when a container is built from our image. For our application, we want to launch our API with gunicorn since this Dockerfile may be used to deploy our service to production at scale.
There are many more commands available for us to use in the Dockerfile, such as using environment variables (ENV) and arguments (ARG), command arguments (CMD), specifying volumes (VOLUME), setting the working directory (WORKDIR) and many more, all of which you can explore through the official docs.
Once we're done composing the Dockerfile, we're ready to build our image using the build command which allows us to add a tag and specify the location of the Dockerfile to use.
docker build -t tagifai:latest -f Dockerfile .\n
We can inspect all built images and their attributes like so:
docker images\n
\nREPOSITORY TAG IMAGE ID CREATED SIZE\ntagifai latest 02c88c95dd4c 23 minutes ago 2.57GB\n
We can also remove any or all images based on their unique IDs.
docker rmi <IMAGE_ID> # remove an image\ndocker rmi $(docker images -a -q) # remove all images\n
Once we've built our image, we're ready to run a container using that image with the run command which allows us to specify the image, port forwarding, etc.
docker run -p 8000:8000 --name tagifai tagifai:latest\n
Once we have our container running, we can use the API thanks for the port we're sharing (8000):
curl -X 'POST' \\\n'http://localhost:8000/predict' \\\n-H 'accept: application/json' \\\n-H 'Content-Type: application/json' \\\n-d '{\n \"texts\": [\n {\n \"text\": \"Transfer learning with transformers for text classification.\"\n }\n ]\n}'\n
We can inspect all containers (running or stopped) like so:
docker ps # running containers\ndocker ps -a # stopped containers\n
\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\nee5f1b08abd5 tagifai:latest \"gunicorn -c config\u2026\" 19 minutes ago Created 0.0.0.0:8000->8000/tcp tagifai\n
We can also stop and remove any or all containers based on their unique IDs:
docker stop <CONTAINER_ID> # stop a running container\ndocker rm <CONTAINER_ID> # remove a container\ndocker stop $(docker ps -a -q) # stop all containers\ndocker rm $(docker ps -a -q) # remove all containers\n
If our application required multiple containers for different services (API, database, etc.) then we can bring them all up at once using the docker compose functionality and scale and manage them using a container orchestration system like Kubernetes (K8s). If we're specifically deploying ML workflows, we can use a toolkit like KubeFlow to help us manage and scale.
In the event that we run into errors while building our image layers, a very easy way to debug the issue is to run the container with the image layers that have been build so far. We can do this by only including the commands that have ran successfully so far (and all COPY statements) in the Dockerfile. And then we need to rebuild the image (since we altered the Dockerfile) and run the container:
Once we have our container running, we can use our application as we would on our local machine but now it's reproducible on any operating system that can run the Docker container engine. We've covered just what we need from Docker to deploy our application but there is so much more to Docker, which you can explore in the docs.
This Dockerfile is commonly the end artifact a data scientist or ML engineer delivers to their DevOps teams to deploy and scale their services, with a few changes:
data assets would be pulled from a remote storage location (ex. S3).
model artifacts would be loaded from a remote model registry.
code would be loaded from a remote repository (ex. GitHub) via git clone.
All of these changes would involve using the proper credentials (via encrypted secrets and can even be automatically deployed via CI/CD workflows. But, of course, there are subsequent responsibilities such as monitoring.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Docker - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
Code tells you how, comments tell you why. -- Jeff Atwood
We can really improve the quality of our codebase by documenting it to make it easier for others (and our future selves) to easily navigate and extend it. We know our code base best the moment we finish writing it but fortunately documenting it will allow us to quickly get back to that familiar state of mind. Documentation can mean many different things to developers, so let's define the most common components:
comments: short descriptions as to why a piece of code exists.
typing: specification of a function's inputs and outputs' data types, providing information pertaining to what a function consumes and produces.
docstrings: meaningful descriptions for functions and classes that describe overall utility, arguments, returns, etc.
docs: rendered webpage that summarizes all the functions, classes, workflows, examples, etc.
It's important to be as explicit as possible with our code. We've already discussed choosing explicit names for variables, functions but another way we can be explicit is by defining the types for our function's inputs and outputs by using the typing library.
So far, our functions have looked like this:
def some_function(a, b):\n return c\n
But we can incorporate so much more information using typing:
from typing import List\ndef some_function(a: List, b: int = 0) -> np.ndarray:\n return c\n
Here we've defined:
input parameter a is a list
input parameter b is an integer with default value 0
output parameter c is a NumPy array
There are many other data types that we can work with, including List, Set, Dict, Tuple, Sequence and more, as well as included types such as int, float, etc. You can also use types from packages we install (ex. np.ndarray) and even from our own defined classes (ex. LabelEncoder).
Starting from Python 3.9+, common types are built in so we don't need to import them with from typing import List, Set, Dict, Tuple, Sequence anymore.
We can make our code even more explicit by adding docstrings to describe overall utility, arguments, returns, exceptions and more. Let's take a look at an example:
from typing import List\ndef some_function(a: List, b: int = 0) -> np.ndarray:\n\"\"\"Function description.\n\n ```python\n c = some_function(a=[], b=0)\n print (c)\n ```\n <pre>\n [[1 2]\n [3 4]]\n </pre>\n\n Args:\n a (List): description of `a`.\n b (int, optional): description of `b`. Defaults to 0.\n\n Raises:\n ValueError: Input list is not one-dimensional.\n\n Returns:\n np.ndarray: Description of `c`.\n\n \"\"\"\n return c\n
Let's unpack the different parts of this function's docstring:
[Line 3]: Summary of the overall utility of the function.
[Lines 5-12]: Example of how to use our function.
[Lines 14-16]: Description of the function's input arguments.
[Lines 18-19]: Any exceptions that may be raised in the function.
[Lines 21-22]: Description of the function's output(s).
We'll render these docstrings in the docs section below to produce this:
Take a look at the docstrings of different functions and classes in our repository.
# madewithml/data.py\nfrom typing import List\n\ndef clean_text(text: str, stopwords: List = STOPWORDS) -> str:\n\"\"\"Clean raw text string.\n Args:\n text (str): Raw text to clean.\n stopwords (List, optional): list of words to filter out. Defaults to STOPWORDS.\n\n Returns:\n str: cleaned text.\n \"\"\"\n pass\n
Tip
If using Visual Studio Code, be sure to use the Python Docstrings Generator extension so you can type \"\"\" under a function and then hit the Shift key to generate a template docstring. It will autofill parts of the docstring using the typing information and even exception in your code!
So we're going through all this effort of including typing and docstrings to our functions but it's all tucked away inside our scripts. What if we can collect all this effort and automatically surface it as documentation? Well that's exactly what we'll do with the following open-source packages \u2192 final result here.
We'll start by overwriting the default index.md file in our docs directory with information specific to our project: index.md
## Documentation\n- [madewithml](madewithml/config.md): documentation for functions and classes.\n\n## Course\nLearn how to combine machine learning with software engineering to design, develop, deploy and iterate on production ML applications.\n\n- Lessons: [https://madewithml.com/](https://madewithml.com/#course)\n- Code: [GokuMohandas/Made-With-ML](https://github.com/GokuMohandas/Made-With-ML)\n
Next we'll create documentation files for each script in our madewithml directory:
It's helpful to have the docs directory structure mimic our project's structure as much as possible.
Next we'll add madewithml.<SCRIPT_NAME> to each file under docs/madewithml. This will populate the file with information about the functions and classes (using their docstrings) from madewithml/<SCRIPT_NAME>.py thanks to the mkdocstrings plugin.
Be sure to check out the complete list of mkdocs plugins.
# docs/madewithml/data.md\n::: madewithml.data\n
Finally, we'll add some configurations to our mkdocs.yml file that mkdocs automatically created:
site_name: Made With ML\nsite_url: https://madewithml.com/\nrepo_url: https://github.com/GokuMohandas/Made-With-ML/\nnav:\n- Home: index.md\n- madewithml:\n- data: madewithml/data.md\n- models: madewithml/models.md\n- train: madewithml/train.md\n- tune: madewithml/tune.md\n- evaluate: madewithml/evaluate.md\n- predict: madewithml/predict.md\n- serve: madewithml/serve.md\n- utils: madewithml/utils.md\ntheme: readthedocs\nplugins:\n- mkdocstrings\nwatch:\n- . # reload docs for any file changes\n
Serve our documentation locally:
python3 -m mkdocs serve\n
This will serve our docs at http://localhost:8000/:
We can easily serve our documentation for free using GitHub pages for public repositories as wells as private documentation for private repositories. And we can even host it on a custom domain (ex. company's subdomain).
Be sure to check out the auto-generated documentation page for our repository. We'll learn how to automatically generate and update this docs page every time we make changes to our codebase later in our CI/CD lesson.
In the next lesson, we'll learn how to style and format our codebase in a consistent manner.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Documentation - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
Evaluation is an integral part of modeling and it's one that's often glossed over. We'll often find evaluation to involve simply computing the accuracy or other global metrics but for many real work applications, a much more nuanced evaluation process is required. However, before evaluating our model, we always want to:
be clear about what metrics we are prioritizing
be careful not to over optimize on any one metric because it may mean you're compromising something else
Let's start by setting up our metrics dictionary that we'll fill in as we go along and all the data we'll need for evaluation: grounds truth labels (y_test, predicted labels (y_pred) and predicted probabilities (y_prob).
id created_on title description tag text prediction 0 19 2020-03-03 13:54:31 Diffusion to Vector Reference implementation of Diffusion2Vec (Com... other Diffusion to Vector Reference implementation o... other 1 26 2020-03-07 23:11:58 Graph Wavelet Neural Network A PyTorch implementation of \"Graph Wavelet Neu... other Graph Wavelet Neural Network A PyTorch impleme... other 2 44 2020-03-08 00:32:58 Capsule Graph Neural Network A PyTorch implementation of \"Capsule Graph Neu... other Capsule Graph Neural Network A PyTorch impleme... other 3 80 2020-03-20 05:59:32 NeRF: Neural Radiance Fields Representing scenes as neural radiance fields ... computer-vision NeRF: Neural Radiance Fields Representing scen... computer-vision 4 84 2020-03-20 15:18:43 Mention Classifier Category prediction model\\r\\nThis repo contain... natural-language-processing Mention Classifier Category prediction model\\r... natural-language-processing"},{"location":"courses/mlops/evaluation/#coarse-grained","title":"Coarse-grained","text":"
While we were developing our models, our evaluation process involved computing the coarse-grained metrics such as overall precision, recall and f1 metrics.
True positives (TP): we correctly predicted class X.
False positives (FP): we incorrectly predicted class X but it was another class.
True negatives (TN): we correctly predicted that it's wasn't the class X.
False negatives (FN): we incorrectly predicted that it wasn't the class X but it was.
The precision_recall_fscore_support() function from scikit-learn has an input parameter called average which has the following options below. We'll be using the different averaging methods for different metric granularities.
None: metrics are calculated for each unique class.
binary: used for binary classification tasks where the pos_label is specified.
micro: metrics are calculated using global TP, FP, and FN.
macro: per-class metrics which are averaged without accounting for class imbalance.
weighted: per-class metrics which are averaged by accounting for class imbalance.
samples: metrics are calculated at the per-sample level.
Inspecting these coarse-grained, overall metrics is a start but we can go deeper by evaluating the same fine-grained metrics at the categorical feature levels.
Besides just inspecting the metrics for each class, we can also identify the true positives, false positives and false negatives. Each of these will give us insight about our model beyond what the metrics can provide.
True positives (TP): learn about where our model performs well.
False positives (FP): potentially identify samples which may need to be relabeled.
False negatives (FN): identify the model's less performant areas to oversample later.
It's a good to have our FP/FN samples feed back into our annotation pipelines in the event we want to fix their labels and have those changes be reflected everywhere.
# TP, FP, FN samples\ntag = \"natural-language-processing\"\nindex = preprocessor.class_to_index[tag]\ntp, fp, fn = [], [], []\nfor i, true in enumerate(y_test):\n pred = y_pred[i]\n if index==true==pred:\n tp.append(i)\n elif index!=true and index==pred:\n fp.append(i)\n elif index==true and index!=pred:\n fn.append(i)\n
# Samples\nnum_samples = 3\ncm = [(tp, \"True positives\"), (fp, \"False positives\"), (fn, \"False negatives\")]\nfor item in cm:\n if len(item[0]):\n print (f\"\\n=== {item[1]} ===\")\n for index in item[0][:num_samples]:\n print (f\"{test_df.iloc[index].text}\")\n print (f\" true: {test_df.tag[index]}\")\n print (f\" pred: {test_df.prediction[index]}\\n\")\n
\n=== True positives ===\nMention Classifier Category prediction model\nThis repo contains AllenNLP model for prediction of Named Entity categories by its mentions.\n true: natural-language-processing\n pred: natural-language-processing\n\nFinetune: Scikit-learn Style Model Finetuning for NLP Finetune is a library that allows users to leverage state-of-the-art pretrained NLP models for a wide variety of downstream tasks.\n true: natural-language-processing\n pred: natural-language-processing\n\nFinetuning Transformers with JAX + Haiku Walking through a port of the RoBERTa pre-trained model to JAX + Haiku, then fine-tuning the model to solve a downstream task.\n true: natural-language-processing\n pred: natural-language-processing\n\n\n=== False positives ===\nHow Docker Can Help You Become A More Effective Data Scientist A look at Docker from the perspective of a data scientist.\n true: mlops\n pred: natural-language-processing\n\nTransfer Learning & Fine-Tuning With Keras Your 100% up-to-date guide to transfer learning & fine-tuning with Keras.\n true: computer-vision\n pred: natural-language-processing\n\nExploratory Data Analysis on MS COCO Style Datasets A Simple Toolkit to do exploratory data analysis on MS COCO style formatted datasets.\n true: computer-vision\n pred: natural-language-processing\n\n\n=== False negatives ===\nThe Unreasonable Effectiveness of Recurrent Neural Networks A close look at how RNNs are able to perform so well.\n true: natural-language-processing\n pred: other\n\nMachine Learning Projects This Repo contains projects done by me while learning the basics. All the familiar types of regression, classification, and clustering methods have been used.\n true: natural-language-processing\n pred: other\n\nBERT Distillation with Catalyst How to distill BERT with Catalyst.\n true: natural-language-processing\n pred: mlops\n\n
Tip
It's a really good idea to do this kind of analysis using our rule-based approach to catch really obvious labeling errors.
While the confusion-matrix sample analysis was a coarse-grained process, we can also use fine-grained confidence based approaches to identify potentially mislabeled samples. Here we\u2019re going to focus on the specific labeling quality as opposed to the final model predictions.
Simple confidence based techniques include identifying samples whose:
Categorical
prediction is incorrect (also indicate TN, FP, FN)
confidence score for the correct class is below a threshold
confidence score for an incorrect class is above a threshold
standard deviation of confidence scores over top N samples is low
different predictions from same model using different parameters
Continuous
difference between predicted and ground-truth values is above some %
# Tag to inspect\ntag = \"natural-language-processing\"\nindex = class_to_index[tag]\nindices = np.where(y_test==index)[0]\n
# Confidence score for the correct class is below a threshold\nlow_confidence = []\nmin_threshold = 0.5\nfor i in indices:\n prob = y_prob[i][index]\n if prob <= 0.5:\n low_confidence.append({\n \"text\": f\"{test_df.iloc[i].text}\",\n \"true\": test_df.tag[i],\n \"pred\": test_df.prediction[i],\n \"prob\": prob})\n
low_confidence[0:3]\n
\n[{'text': 'The Unreasonable Effectiveness of Recurrent Neural Networks A close look at how RNNs are able to perform so well.',\n 'true': 'natural-language-processing',\n 'pred': 'other',\n 'prob': 0.0023471832},\n {'text': 'Machine Learning Projects This Repo contains projects done by me while learning the basics. All the familiar types of regression, classification, and clustering methods have been used.',\n 'true': 'natural-language-processing',\n 'pred': 'other',\n 'prob': 0.0027675298},\n {'text': 'BERT Distillation with Catalyst How to distill BERT with Catalyst.',\n 'true': 'natural-language-processing',\n 'pred': 'mlops',\n 'prob': 0.37908182}]\n
But these are fairly crude techniques because neural networks are easily overconfident and so their confidences cannot be used without calibrating them.
Modern (large) neural networks result in higher accuracies but are over confident.On Calibration of Modern Neural Networks
Assumption: \u201cthe probability associated with the predicted class label should reflect its ground truth correctness likelihood.\u201d
Reality: \u201cmodern (large) neural networks are no longer well-calibrated\u201d
Solution: apply temperature scaling (extension of Platt scaling) on model outputs
Recent work on confident learning (cleanlab) focuses on identifying noisy labels (with calibration), which can then be properly relabeled and used for training.
id created_on title description tag prediction 165 2137 2020-08-13 02:10:03 Unpopular Opinion - Data Scientists Should Be ... I believe data scientists can be more effectiv... mlops natural-language-processing 154 1976 2020-07-27 14:12:03 Close-Domain fine-tuning for table detection In this project, we show the benefits of using... computer-vision natural-language-processing 16 264 2020-04-06 21:33:32 The Unreasonable Effectiveness of Recurrent Ne... A close look at how RNNs are able to perform s... natural-language-processing other 103 1459 2020-06-16 03:06:10 SuperGlue: Learning Feature Matching with Grap... SuperGlue, a neural network that matches two s... other computer-vision 112 1524 2020-06-20 10:42:25 Machine Learning Projects This Repo contains projects done by me while l... natural-language-processing other
Not all of these are necessarily labeling errors but situations where the predicted probabilities were not so confident. Therefore, it will be useful to attach the predicted outcomes along side results. This way, we can know if we need to relabel, upsample, etc. as mitigation strategies to improve our performance.
The operations in this section can be applied to entire labeled dataset to discover labeling errors via confidence learning.
Just inspecting the overall and class metrics isn't enough to deploy our new version to production. There may be key slices of our dataset that we need to do really well on:
Target / predicted classes (+ combinations)
Features (explicit and implicit)
Metadata (timestamps, sources, etc.)
Priority slices / experience (minority groups, large users, etc.)
An easy way to create and evaluate slices is to define slicing functions.
from snorkel.slicing import PandasSFApplier\nfrom snorkel.slicing import slice_dataframe\nfrom snorkel.slicing import slicing_function\n
@slicing_function()\ndef nlp_llm(x):\n\"\"\"NLP projects that use LLMs.\"\"\"\n nlp_project = \"natural-language-processing\" in x.tag\n llm_terms = [\"transformer\", \"llm\", \"bert\"]\n llm_project = any(s.lower() in x.text.lower() for s in llm_terms)\n return (nlp_project and llm_project)\n
@slicing_function()\ndef short_text(x):\n\"\"\"Projects with short titles and descriptions.\"\"\"\n return len(x.text.split()) < 8 # less than 8 words\n
Here we're using Snorkel's slicing_function to create our different slices. We can visualize our slices by applying this slicing function to a relevant DataFrame using slice_dataframe.
text tag 12 Finetuning Transformers with JAX + Haiku Walki... natural-language-processing 19 Question Answering with a Fine-Tuned BERT What... natural-language-processing 29 BertViz Tool for visualizing attention in the ... natural-language-processing 30 The Transformer Family This post presents how ... natural-language-processing 31 Pruning Bert to Accelerate Inference After pre... natural-language-processing
text tag 75 NLPAug Data augmentation for NLP natural-language-processing 123 Offline Reinforcement Learning Challenges, alg... other 127 Image Classifier Pure JavaScript Image Classifier computer-vision 132 imgaug Image augmentation for machine learning... computer-vision 140 QSVM Quantum SVM for sentiment analysis natural-language-processing
We can define even more slicing functions and create a slices record array using the PandasSFApplier. The slices array has N (# of data points) items and each item has S (# of slicing functions) items, indicating whether that data point is part of that slice. Think of this record array as a masking layer for each slicing function on our data.
To calculate metrics for our slices, we could use snorkel.analysis.Scorer but we've implemented a version that will work for multiclass or multilabel scenarios.
Slicing can help identify sources of bias in our data. For example, our model has most likely learned to associated algorithms with certain applications such as CNNs used for computer vision or transformers used for NLP projects. However, these algorithms are not being applied beyond their initial use cases. We\u2019d need ensure that our model learns to focus on the application over algorithm. This could be learned with:
enough data (new or oversampling incorrect predictions)
masking the algorithm (using text matching heuristics)
Besides just comparing predicted outputs with ground truth values, we can also inspect the inputs to our models. What aspects of the input are more influential towards the prediction? If the focus is not on the relevant features of our input, then we need to explore if there is a hidden pattern we're missing or if our model has learned to overfit on the incorrect features. We can use techniques such as SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations) to inspect feature importance. On a high level, these techniques learn which features have the most signal by assessing the performance in their absence. These inspections can be performed on a global level (ex. per-class) or on a local level (ex. single prediction).
from lime.lime_text import LimeTextExplainer\nfrom sklearn.pipeline import make_pipeline\n
LimeTextExplainer.explain_instance function requires a classifier_fn that takes in a list of strings and outputs the predicted probabilities.
Besides just looking at metrics, we also want to conduct some behavioral sanity tests. Behavioral testing is the process of testing input data and expected outputs while treating the model as a black box. They don't necessarily have to be adversarial in nature but more along the types of perturbations we'll see in the real world once our model is deployed. A landmark paper on this topic is Beyond Accuracy: Behavioral Testing of NLP Models with CheckList which breaks down behavioral testing into three types of tests:
invariance: Changes should not affect outputs.
# INVariance via verb injection (changes should not affect outputs)\ntokens = [\"revolutionized\", \"disrupted\"]\ntexts = [f\"Transformers applied to NLP have {token} the ML field.\" for token in tokens]\n[preprocessor.index_to_class[y_prob.argmax()] for y_prob in classifier_fn(texts=texts)]\n
# DIRectional expectations (changes with known outputs)\ntokens = [\"text classification\", \"image classification\"]\ntexts = [f\"ML applied to {token}.\" for token in tokens]\n[preprocessor.index_to_class[y_prob.argmax()] for y_prob in classifier_fn(texts=texts)]\n
minimum functionality: Simple combination of inputs and expected outputs.
# Minimum Functionality Tests (simple input/output pairs)\ntokens = [\"natural language processing\", \"mlops\"]\ntexts = [f\"{token} is the next big wave in machine learning.\" for token in tokens]\n[preprocessor.index_to_class[y_prob.argmax()] for y_prob in classifier_fn(texts=texts)]\n
\n['natural-language-processing', 'mlops']\n
We'll learn how to systematically create tests in our testing lesson.
Once we've evaluated our model's ability to perform on a static dataset we can run several types of online evaluation techniques to determine performance on actual production data. It can be performed using labels or, in the event we don't readily have labels, proxy signals.
manually label a subset of incoming data to evaluate periodically.
asking the initial set of users viewing a newly categorized content if it's correctly classified.
allow users to report misclassified content by our model.
And there are many different experimentation strategies we can use to measure real-time performance before committing to replace our existing version of the system.
AB testing involves sending production traffic to our current system (control group) and the new version (treatment group) and measuring if there is a statistical difference between the values for two metrics. There are several common issues with AB testing such as accounting for different sources of bias, such as the novelty effect of showing some users the new system. We also need to ensure that the same users continue to interact with the same systems so we can compare the results without contamination.
In many cases, if we're simply trying to compare the different versions for a certain metric, AB testing can take while before we reach statical significance since traffic is evenly split between the different groups. In this scenario, multi-armed bandits will be a better approach since they continuously assign traffic to the better performing version.
Canary tests involve sending most of the production traffic to the currently deployed system but sending traffic from a small cohort of users to the new system we're trying to evaluate. Again we need to make sure that the same users continue to interact with the same system as we gradually roll out the new system.
Shadow testing involves sending the same production traffic to the different systems. We don't have to worry about system contamination and it's very safe compared to the previous approaches since the new system's results are not served. However, we do need to ensure that we're replicating as much of the production system as possible so we can catch issues that are unique to production early on. But overall, shadow testing is easy to monitor, validate operational consistency, etc.
What can go wrong?
If shadow tests allow us to test our updated system without having to actually serve the new results, why doesn't everyone adopt it?
Show answer
With shadow deployment, we'll miss out on any live feedback signals (explicit/implicit) from our users since users are not directly interacting with the product using our new version.
We also need to ensure that we're replicating as much of the production system as possible so we can catch issues that are unique to production early on. This is rarely possible because, while your ML system may be a standalone microservice, it ultimately interacts with an intricate production environment that has many dependencies.
"},{"location":"courses/mlops/evaluation/#capability-vs-alignment","title":"Capability vs. alignment","text":"
We've seen the many different metrics that we'll want to calculate when it comes to evaluating our model but not all metrics mean the same thing. And this becomes very important when it comes to choosing the \"best\" model(s).
capability: the ability of our model to perform a task, measured by the objective function we optimize for (ex. log loss)
alignment: desired behavior of our model, measure by metrics that are not differentiable or don't account for misclassifications and probability differences (ex. accuracy, precision, recall, etc.)
While capability (ex. loss) and alignment (ex. accuracy) metrics may seem to be aligned, their differences can indicate issues in our data:
\u2193 accuracy, \u2191 loss = large errors on lots of data (worst case)
\u2193 accuracy, \u2193 loss = small errors on lots of data, distributions are close but tipped towards misclassifications (misaligned)
\u2191 accuracy, \u2191 loss = large errors on some data (incorrect predictions have very skewed distributions)
\u2191 accuracy, \u2193 loss = no/few errors on some data (best case)
So far, we've been training and evaluating our different baselines but haven't really been tracking these experiments. We'll fix this but defining a proper process for experiment tracking which we'll use for all future experiments (including hyperparameter optimization). Experiment tracking is the process of managing all the different experiments and their components, such as parameters, metrics, models and other artifacts and it enables us to:
Organize all the necessary components of a specific experiment. It's important to have everything in one place and know where it is so you can use them later.
Reproduce past results (easily) using saved experiments.
Log iterative improvements across time, data, ideas, teams, etc.
There are many options for experiment tracking but we're going to use MLFlow (100% free and open-source) because it has all the functionality we'll need. We can run MLFlow on our own servers and databases so there are no storage cost / limitations, making it one of the most popular options and is used by Microsoft, Facebook, Databricks and others. There are also several popular options such as a Comet ML (used by Google AI, HuggingFace, etc.), Neptune (used by Roche, NewYorker, etc.), Weights and Biases (used by Open AI, Toyota Research, etc.). These are fully managed solutions that provide features like dashboards, reports, etc.
In this course, our MLflow artifact and backend store will both be on our local machine. In a production setting, these would be remote such as S3 for the artifact store and a database service (ex. PostgreSQL RDS) as our backend store.
While we could use MLflow directly to log metrics, artifacts and parameters:
# Example mlflow calls\nmlflow.log_metrics({\"train_loss\": train_loss, \"val_loss\": val_loss}, step=epoch)\nmlflow.log_artifacts(dir)\nmlflow.log_params(config)\n
We'll instead use Ray to integrate with MLflow. Specifically we'll use the MLflowLoggerCallback which will automatically log all the necessary components of our experiments to the location specified in our MLFLOW_TRACKING_URI. We of course can still use MLflow directly if we want to log something that's not automatically logged by the callback. And if we're using other experiment trackers, Ray has integrations for those as well.
With our updated RunConfig, with the MLflow callback, we can now train our model and all the necessary components will be logged to MLflow. This is the exact same training workflow we've been using so far from the training lesson.
We're going to use the search_runs function from the MLflow python API to identify the best run in our experiment so far (we' only done one run so far so it will be the run from above).
Once we're done training, we can use the MLflow dashboard to visualize our results. To do so, we'll use the mlflow server command to launch the MLflow dashboard and navigate to the experiment we just created.
mlflow server -h 0.0.0.0 -p 8080 --backend-store-uri /tmp/mlflow/\n
View the dashboard
AnyscaleLocal
If you're on Anyscale Workspaces, then we need to first expose the port of the MLflow server. Run the following command on your Anyscale Workspace terminal to generate the public URL to your MLflow server.
After inspection and once we've identified an experiment that we like, we can load the model for evaluation and inference.
from ray.air import Result\nfrom urllib.parse import urlparse\n
We're going to create a small utility function that uses an MLflow run's artifact path to load a Ray Result object. We'll then use the Result object to load the best checkpoint.
def get_best_checkpoint(run_id):\n artifact_dir = urlparse(mlflow.get_run(run_id).info.artifact_uri).path # get path from mlflow\n results = Result.from_path(artifact_dir)\n return results.best_checkpoints[0][0]\n
With a particular run's best checkpoint, we can load the model from it and use it.
# Evaluate on test split\nbest_checkpoint = get_best_checkpoint(run_id=best_run.run_id)\npredictor = TorchPredictor.from_checkpoint(best_checkpoint)\nperformance = evaluate(ds=test_ds, predictor=predictor)\nprint (json.dumps(performance, indent=2))\n
# Predict on sample\ntitle = \"Transfer learning with transformers\"\ndescription = \"Using transformers for transfer learning on text classification tasks.\"\nsample_df = pd.DataFrame([{\"title\": title, \"description\": description, \"tag\": \"other\"}])\npredict_with_proba(df=sample_df, predictor=predictor)\n
In the next lesson we'll learn how to tune our models and use our MLflow dashboard to compare the results.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Tracking - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/exploratory-data-analysis/","title":"Exploratory Data Analysis (EDA)","text":""},{"location":"courses/mlops/exploratory-data-analysis/#intuition","title":"Intuition","text":"
Exploratory data analysis (EDA) to understand the signals and nuances of our dataset. It's a cyclical process that can be done at various points of our development process (before/after labeling, preprocessing, etc. depending on how well the problem is defined. For example, if we're unsure how to label or preprocess our data, we can use EDA to figure it out.
We're going to start our project with EDA, a vital (and fun) process that's often misconstrued. Here's how to think about EDA:
not just to visualize a prescribed set of plots (correlation matrix, etc.).
goal is to convince yourself that the data you have is sufficient for the task.
use EDA to answer important questions and to make it easier to extract insight
not a one time process; as your data grows, you want to revisit EDA to catch distribution shifts, anomalies, etc.
Let's answer a few key questions using EDA.
from collections import Counter\nimport matplotlib.pyplot as plt\nimport seaborn as sns; sns.set_theme()\nimport warnings; warnings.filterwarnings(\"ignore\")\nfrom wordcloud import WordCloud, STOPWORDS\n
We can then separate the tags and from their respective counts and plot them using Plotly.
# Plot tag frequencies\ntags, tag_counts = zip(*all_tags.most_common())\nplt.figure(figsize=(10, 3))\nax = sns.barplot(x=list(tags), y=list(tag_counts))\nax.set_xticklabels(tags, rotation=0, fontsize=8)\nplt.title(\"Tag distribution\", fontsize=14)\nplt.ylabel(\"# of projects\", fontsize=12)\nplt.show()\n
We do have some data imbalance but it's not too bad. If we did want to account for this, there are many strategies, including over-sampling less frequent classes and under-sampling popular classes, class weights in the loss function, etc.
Is there enough signal in the title and description that's unique to each tag? This is important to know because we want to verify our initial hypothesis that the project's title and description are high quality features for predicting the tag. And to visualize this, we're going to use a wordcloud. We also use a jupyter widget, which you can view in the notebook, to interactively select a tag and see the wordcloud for that tag.
# Most frequent tokens for each tag\ntag=\"natural-language-processing\"\nplt.figure(figsize=(10, 3))\nsubset = df[df.tag==tag]\ntext = subset.title.values\ncloud = WordCloud(\n stopwords=STOPWORDS, background_color=\"black\", collocations=False,\n width=500, height=300).generate(\" \".join(text))\nplt.axis(\"off\")\nplt.imshow(cloud)\n
Looks like the title text feature has some good signal for the respective classes and matches our intuition. We can repeat this for the description text feature as well and see similar quality signals. This information will become useful when we decide how to use our features for modeling.
There's a lot more exploratory data analysis that we can do but for now we've answered our questions around our class distributions and the quality of our text features. In the next lesson we'll preprocess our dataset in preparation for model training.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Exploration - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/feature-store/","title":"Feature Store","text":""},{"location":"courses/mlops/feature-store/#what-is-a-feature-store","title":"What is a feature store","text":"
Let's motivate the need for a feature store by chronologically looking at what challenges developers face in their current workflows. Suppose we had a task where we needed to predict something for an entity (ex. user) using their features.
Duplication: feature development in isolation (for each unique ML application) can lead to duplication of efforts (setting up ingestion pipelines, feature engineering, etc.).
Solution: create a central feature repository where the entire team contributes maintained features that anyone can use for any application.
Skew: we may have different pipelines for generating features for training and serving which can introduce skew through the subtle differences.
Solution: create features using a unified pipeline and store them in a central location that the training and serving pipelines pull from.
Values: once we set up our data pipelines, we need to ensure that our input feature values are up-to-date so we aren't working with stale data, while maintaining point-in-time correctness so we don't introduce data leaks.
Solution: retrieve input features for the respective outcomes by pulling what's available when a prediction would be made.
Point-in-time correctness refers to mapping the appropriately up-to-date input feature values to an observed outcome at \\(t_{n+1}\\). This involves knowing the time (\\(t_n\\)) that a prediction is needed so we can collect feature values (\\(X\\)) at that time.
When actually constructing our feature store, there are several core components we need to have to address these challenges:
data ingestion: ability to ingest data from various sources (databases, data warehouse, etc.) and keep them updated.
feature definitions: ability to define entities and corresponding features
historical features: ability to retrieve historical features to use for training.
online features: ability to retrieve features from a low latency origin for inference.
Each of these components is fairly easy to set up but connecting them all together requires a managed service, SDK layer for interactions, etc. Instead of building from scratch, it's best to leverage one of the production-ready, feature store options such as Feast, Hopsworks, Tecton, Rasgo, etc. And of course, the large cloud providers have their own feature store options as well (Amazon's SageMaker Feature Store, Google's Vertex AI, etc.)
Tip
We highly recommend that you explore this lesson after completing the previous lessons since the topics (and code) are iteratively developed. We did, however, create the feature-store repository for a quick overview with an interactive notebook.
Not all machine learning platforms require a feature store. In fact, our use case is a perfect example of a task that does not benefit from a feature store. All of our data points are independent, stateless, from client-side and there is no entity that has changing features over time. The real utility of a feature store shines when we need to have up-to-date features for an entity that we continually generate predictions for. For example, a user's behavior (clicks, purchases, etc.) on an e-commerce platform or the deliveries a food runner recently made in the last hour, etc.
"},{"location":"courses/mlops/feature-store/#when-do-i-need-a-feature-store","title":"When do I need a feature store?","text":"
To answer this question, let's revisit the main challenges that a feature store addresses:
Duplication: if we don't have too many ML applications/models, we don't really need to add the additional complexity of a feature store to manage transformations. All the feature transformations can be done directly inside the model processing or as a separate function. We could even organize these transformations in a separate central repository for other team members to use. But this quickly becomes difficult to use because developers still need to know which transformations to invoke and which are compatible with their specific models, etc.
Note
Additionally, if the transformations are compute intensive, then they'll incur a lot of costs by running on duplicate datasets across different applications (as opposed to having a central location with upt-o-date transformed features).
Skew: similar to duplication of efforts, if our transformations can be tied to the model or as a standalone function, then we can just reuse the same pipelines to produce the feature values for training and serving. But this becomes complex and compute intensive as the number of applications, features and transformations grow.
Value: if we aren't working with features that need to be computed server-side (batch or streaming), then we don't have to worry about concepts like point-in-time, etc. However, if we are, a feature store can allow us to retrieve the appropriate feature values across all data sources without the developer having to worry about using disparate tools for different sources (batch, streaming, etc.)
We're going to create a feature repository at the root of our project. Feast will create a configuration file for us and we're going to add an additional features.py file to define our features.
Traditionally, the feature repository would be it's own isolated repository that other services will use to read/write features from.
We're going to configure the locations for our registry and online store (SQLite) in our feature_store.yaml file.
registry: contains information about our feature repository, such as data sources, feature views, etc. Since it's in a DB, instead of a Python file, it can very quickly be accessed in production.
online store: DB (SQLite for local) that stores the (latest) features for defined entities to be used for online inference.
If all our feature definitions look valid, Feast will sync the metadata about Feast objects to the registry. The registry is a tiny database storing most of the same information you have in the feature repository. This step is necessary because the production feature serving infrastructure won't be able to access Python files in the feature repository at run time, but it will be able to efficiently and securely read the feature definitions from the registry.
When we run Feast locally, the offline store is effectively represented via Pandas point-in-time joins. Whereas, in production, the offline store can be something more robust like Google BigQuery, Amazon RedShift, etc.
We'll go ahead and paste this into our features/feature_store.yaml file (the notebook cell is automatically do this):
The first step is to establish connections with our data sources (databases, data warehouse, etc.). Feast requires it's data sources to either come from a file (Parquet), data warehouse (BigQuery) or data stream (Kafka / Kinesis). We'll convert our generated features file from the DataOps pipeline (features.json) into a Parquet file, which is a column-major data format that allows fast feature retrieval and caching benefits (contrary to row-major data formats such as CSV where we have to traverse every single row to collect feature values).
Next, we need to define the main entity that each data point pertains to. In our case, each project has a unique ID with features such as text and tags.
# Define an entity\nproject = Entity(\n name=\"id\",\n value_type=ValueType.INT64,\n description=\"project id\",\n)\n
Finally, we're ready to create a FeatureView that loads specific features (features), of various value types, from a data source (input) for a specific period of time (ttl).
# Define a Feature View for each project\nproject_details_view = FeatureView(\n name=\"project_details\",\n entities=[\"id\"],\nttl=Duration(\nseconds=(datetime.today() - datetime.strptime(START_TIME, \"%Y-%m-%d\")).days * 24 * 60 * 60\n ),\nfeatures=[\nFeature(name=\"text\", dtype=ValueType.STRING),\n Feature(name=\"tag\", dtype=ValueType.STRING),\n ],\n online=True,\ninput=project_details,\ntags={},\n)\n
So let's go ahead and define our feature views by moving this code into our features/features.py script (the notebook cell is automatically do this):
Show code
from datetime import datetime\nfrom pathlib import Path\n\nfrom feast import Entity, Feature, FeatureView, ValueType\nfrom feast.data_source import FileSource\nfrom google.protobuf.duration_pb2 import Duration\n\n\n# Read data\nSTART_TIME = \"2020-02-17\"\nproject_details = FileSource(\n path=\"/content/data/features.parquet\",\n event_timestamp_column=\"created_on\",\n)\n\n# Define an entity for the project\nproject = Entity(\n name=\"id\",\n value_type=ValueType.INT64,\n description=\"project id\",\n)\n\n# Define a Feature View for each project\n# Can be used for fetching historical data and online serving\nproject_details_view = FeatureView(\n name=\"project_details\",\n entities=[\"id\"],\n ttl=Duration(\n seconds=(datetime.today() - datetime.strptime(START_TIME, \"%Y-%m-%d\")).days * 24 * 60 * 60\n ),\n features=[\n Feature(name=\"text\", dtype=ValueType.STRING),\n Feature(name=\"tag\", dtype=ValueType.STRING),\n ],\n online=True,\n input=project_details,\n tags={},\n)\n
Once we've defined our feature views, we can apply it to push a version controlled definition of our features to the registry for fast access. It will also configure our registry and online stores that we've defined in our feature_store.yaml.
cd features\nfeast apply\n
\nRegistered entity id\nRegistered feature view project_details\nDeploying infrastructure for project_details\n
Once we've registered our feature definition, along with the data source, entity definition, etc., we can use it to fetch historical features. This is done via joins using the provided timestamps using pandas for our local setup or BigQuery, Hive, etc. as an offline DB for production.
import pandas as pd\nfrom feast import FeatureStore\n
event_timestamp id project_details__text project_details__tag 0 2022-06-23 00:00:00+00:00 6 Comparison between YOLO and RCNN on real world... computer-vision 1 2022-06-23 00:00:00+00:00 7 Show, Infer & Tell: Contextual Inference for C... computer-vision 2 2022-06-23 00:00:00+00:00 9 Awesome Graph Classification A collection of i... graph-learning"},{"location":"courses/mlops/feature-store/#materialize","title":"Materialize","text":"
For online inference, we want to retrieve features very quickly via our online store, as opposed to fetching them from slow joins. However, the features are not in our online store just yet, so we'll need to materialize them first.
cd features\nCURRENT_TIME=$(date -u +\"%Y-%m-%dT%H:%M:%S\")\nfeast materialize-incremental $CURRENT_TIME\n
\nMaterializing 1 feature views to 2022-06-23 19:16:05+00:00 into the sqlite online store.\nproject_details from 2020-02-17 19:16:06+00:00 to 2022-06-23 19:16:05+00:00:\n100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 955/955 [00:00<00:00, 10596.97it/s]\n
This has moved the features for all of our projects into the online store since this was first time materializing to the online store. When we subsequently run the materialize-incremental command, Feast keeps track of previous materializations and so we'll only materialize the new data since the last attempt.
{'id': [6],\n 'project_details__tag': ['computer-vision'],\n 'project_details__text': ['Comparison between YOLO and RCNN on real world videos Bringing theory to experiment is cool. We can easily train models in colab and find the results in minutes.']}\n
The feature store we implemented above assumes that our task requires batch processing. This means that inference requests on specific entity instances can use features that have been materialized from the offline store. Note that they may not be the most recent feature values for that entity.
Application data is stored in a database and/or a data warehouse, etc. And it goes through the necessary pipelines to be prepared for downstream application (analytics, machine learning, etc.).
These features are written to the offline store which can then be used to retrieve historical training data to train a model with. In our local set up, this was join via Pandas DataFrame joins for given timestamps and entity IDs but in a production setting, something like Google BigQuery or Hive would receive the feature requests.
Once we have our training data, we can start the workflows to optimize, train and validate a model.
We can incrementally materialize features to the online store so that we can retrieve an entity's feature values with low latency. In our local set up, this was join via SQLite for a given set of entities but in a production setting, something like Redis or DynamoDB would be used.
These online features are passed on to the deployed model to generate predictions that would be used downstream.
Warning
Had our entity (projects) had features that change over time, we would materialize them to the online store incrementally. But since they don't, this would be considered over engineering but it's important to know how to leverage a feature store for entities with changing features over time.
Some applications may require stream processing where we require near real-time feature values to deliver up-to-date predictions at low latency. While we'll still utilize an offline store for retrieving historical data, our application's real-time event data will go directly through our data streams to an online store for serving. An example where stream processing would be needed is when we want to retrieve real-time user session behavior (clicks, purchases) in an e-commerce platform so that we can recommend the appropriate items from our catalog.
Real-time event data enters our running data streams (Kafka / Kinesis, etc.) where they can be processed to generate features.
These features are written to the online store which can then be used to retrieve online features for serving at low latency. In our local set up, this was join via SQLite for a given set of entities but in a production setting, something like Redis or DynamoDB would be used.
Streaming features are also written from the data stream to the batch data source (data warehouse, db, etc.) to be processed for generating training data later on.
Historical data will be validated and used to generate features for training a model. This cadence for how often this happens depends on whether there are data annotation lags, compute constraints, etc.
There are a few more components we're not visualizing here such as the unified ingestion layer (Spark), that connects data from the varied data sources (warehouse, DB, etc.) to the offline/online stores, or low latency serving (<10 ms). We can read more about all of these in the official Feast Documentation, which also has guides to set up a feature store with Feast with AWS, GCP, etc.
Though we could continue to version our training data with DVC whenever we release a version of the model, it might not be necessary. When we pull data from source or compute features, should they save the data itself or just the operations?
Version the data
This is okay if (1) the data is manageable, (2) if our team is small/early stage ML or (3) if changes to the data are infrequent.
But what happens as data becomes larger and larger and we keep making copies of it.
Version the operations
We could keep snapshots of the data (separate from our projects) and provided the operations and timestamp, we can execute operations on those snapshots of the data to recreate the precise data artifact used for training. Many data systems use time-travel to achieve this efficiently.
But eventually this also results in data storage bulk. What we need is an append-only data source where all changes are kept in a log instead of directly changing the data itself. So we can use the data system with the logs to produce versions of the data as they were without having to store separate snapshots of the the data itself.
Regardless of the choice above, feature stores are very useful here. Instead of coupling data pulls and feature compute with the time of modeling, we can separate these two processes so that features are up-to-date when we need them. And we can still achieve reproducibility via efficient point-in-time correctness, low latency snapshots, etc. This essentially creates the ability to work with any version of the dataset at any point in time.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Feature Store - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
Whether we're working individually or with a team, it's important that we have a system to track changes to our projects so that we can revert to previous versions and so that others can reproduce our work and contribute to it. Git is a distributed versional control system that allows us do exactly this. Git runs locally on our computer and it keeps track of our files and their histories. To enable collaboration with others, we can use a remote host (GitHub, GitLab, BitBucket, etc.) to host our files and their histories. We'll use git to push our local changes and pull other's changes to and from the remote host.
Git is traditionally used to store and version small files <100MB (scripts, READMEs, etc.), however, we can still version large artifacts (datasets, model weights, etc.) using text pointers pointing to blob stores. These pointers will contain information such as where the asset is located, it's specific contents/version (ex. via hashing), etc.
Initialize a local repository (.git directory) to track our files:
git init\n
\nInitialized empty Git repository in /Users/goku/Documents/madewithml/MLOps/.git/\n
We can see what files are untracked or yet to be committed:
git status\n
\nOn branch main\n\nNo commits yet\n\nUntracked files:\n (use \"git add ...\" to include in what will be committed)\n .flake8\n .vscode/\n Makefile\n ..."},{"location":"courses/mlops/git/#gitignore","title":".gitignore","text":"
We can see that we have some files that we don't want to push to a remote host, such as our virtual environment, logs, large data files, etc. We can create a .gitignore file to make sure we aren't checking in these files.
For now, we're going to add data to our .gitignore file as well but this means that others will not be able to produce the same data assets when they pull from our remote host. To address this, we'll push pointers to our data files in our versioning lesson so that the data too can be reproduced exactly as we have it locally.
\n
Tip
\n
Check out our project's .gitignore for a more complete example that also includes lots of other system artifacts that we would normally not want to push to a remote repository. Our complete .gitignore file is based on GitHub's Python template and we're using a Mac, so we added the relevant global file names as well.
\n
If we run git status now, we should no longer see the files we've defined in our .gitignore file.
"},{"location":"courses/mlops/git/#add-to-stage","title":"Add to stage","text":"
Next, we'll add our work from the working directory to the staging area.
\n
\n
We can add one file at a time:\n
git add <filename>\n
\n
We can add all files at once:\n
git add .\n
\n
\n
Now running git status will show us all the staged files:
\n
git status\n
\n
\nOn branch main\n\nNo commits yet\n\nChanges to be committed:\n (use \"git rm --cached ...\" to unstage)\n new file: .flake8\n new file: .gitignore\n new file: Makefile\n ..."},{"location":"courses/mlops/git/#commit-to-repo","title":"Commit to repo","text":"
Now we're ready to commit the files in the staging area to the local repository. The default branch (a version of our project) will be called main.
The commit requires a message indicating what changes took place. We can use git commit --amend to edit the commit message if needed. If we do a git status check we'll see that there is nothing else to commit from our staging area.
\n
git status\n
\n
\nOn branch main\nnothing to commit, working tree clean\n
"},{"location":"courses/mlops/git/#push-to-remote","title":"Push to remote","text":"
Now we're ready to push the updates from our local repository to a remote repository. Start by creating an account on GitHub (or any other remote repository) and follow the instructions to create a remote repository (it can be private or public). Inside our local repository, we're going to set our username and email credentials so that we can push changes from our local to the remote repository.
\n
# Set credentials via terminal\ngit config --global user.name <USERNAME>\ngit config --global user.email <EMAIL>\n
\nWe can quickly validate that we set the proper credentials like so:\n
Next, we need to establish the connection between our local and remote repositories:
\n
# Push to remote\ngit remote add origin https://github.com/<USERNAME>/<REPOSITORY_NAME>.git\ngit push -u origin main # pushing the contents of our local repo to the remote repo\n# origin signifies the remote repository\n
<REMOTE_REPO_URL> is the location of the remote repo (ex. https://github.com/GokuMohandas/Made-With-ML).
\n
<PATH_TO_PROJECT_DIR> is the name of the local directory you want to clone the project into.
\n
"},{"location":"courses/mlops/git/#create-a-branch","title":"Create a branch","text":"
When we want to add or change something, such as adding a feature, fixing a bug, etc., it's always a best practice to create a separate branch before developing. This is especially crucial when working with a team so we can cleanly merge our work with the main branch after discussions and reviews.
\n
We'll start by creating a new branch:
\n
git checkout -b <NEW_BRANCH_NAME>\n
\n
We can see all the branches we've created with the following command where the * indicates our current branch:
\n
git branch\n
\n
\n* convnet\nmain\n
\n\n
We can easily switch between existing branches using:
\n
git checkout <BRANCH_NAME>\n
\n
Once we're in a branch, we can make changes to our project and commit those changes.
\n
git add .\ngit commit -m \"update model to a convnet\"\ngit push origin convnet\n
\n
Note that we are pushing this branch to our remote repository, which doesn't yet exist there, so GitHub will create it accordingly.
When we push our new branch to the remote repository, we'll need to create a pull request (PR) to merge with another branch (ex. our main branch in this case). When merging our work with another branch (ex. main), it's called a pull request because we're requesting the branch to pull our committed work. We can create the pull request using steps outlined here: Creating a pull request.
\n
Note
\n
We can merge branches and resolve conflicts using git CLI commands but it's preferred to use the online interface because we can easily visualize the changes, have discussion with teammates, etc.\n
Once we accepted the pull request, our main branch is now updated with our changes. However, the update only happened on the remote repository so we should pull those changes to our local main branch as well.
Once we're done working with a branch, we can delete it to prevent our repository from cluttering up. We can easily delete both the local and remote versions of the branch with the following commands:\n
So far, the workflows for integrating our iterative development has been very smooth but in a collaborative setting, we may need to resolve conflicts. Let's say there are two branches (a and b) that were created from the main branch. Here's what we're going to try and simulate:
\n\n
Developer A and B fork the main branch to make some changes
\n
Developer A makes a change and submits a PR to the main branch.
\n
Developer B makes a change to the same line as Developer A and submits a PR to main.
\n
We have a merge conflict now since both developers altered the same line.
\n
Choose which version of the code to keep and resolve the conflict.
\n\n
When we try to merge the second PR, we have to resolve the conflicts between this new PR and what already exists in the main branch.
\n
We can resolve the conflict by choosing which content (current main which merged with the a branch or this b branch) to keep and delete the other one. Then we can merge the PR successfully and update our local main branch.
\n
<<<<<< BRANCH_A\n<CHANGES FROM BRANCH A>\n======\n<CHANGES FROM BRANCH B>\n>>>>>> BRANCH_B\n
\n
Once the conflicts have been resolved and we merge the PR, we can update our local repository to reflect the decisions.
\n
git checkout main\ngit pull origin main\n
\n
Note
\n
We only have a conflict because both branches were forked from a previous version of the main branch and they both happened to alter the same content. Had we created one branch first and then updated main before creating the second branch, we wouldn't have any conflicts. But in a collaborative setting, different developers may fork off the same version of the branch anytime.
\n
A few more important commands to know include rebase and stash.
If we want to see the log of all our commits, we can do so using the log command. We can also do the same by inspecting specific branch histories on the Git online interface.
\n
# Log\ngit log\ngit log --oneline # short version\n
\n
\n704d99c (HEAD -> main) added project files\n
\n\n
Commit IDs are 40 characters long but we can represent them with the first few (seven digits is the default for a Git SHA). If there is ambiguity, Git will notify us and we can simply add more of the commit ID.
If we want to know the difference between two commits, we can use the diff command.
\n
# Diff\ngit diff # all changes between current working tree and previous commit\ngit diff <COMMIT_A> <COMMIT_B> # diff b/w two commits\ngit diff <COMMIT_A>:<PATH_TO_FILE> <COMMIT_B>:<PATH_TO_FILE> # file diff b/w two commits\n
Now if we already made the commit but haven't pushed to remote yet, we can reset to the previous commit by moving the branch pointer to that commit. Note that this will undo all changes made since the previous commit.\n
# Reset\ngit reset <PREVIOUS_COMMIT_ID> # or HEAD^\n
\n
HEAD is a quick way to refer to the previous commit. Both HEAD and any previous commit ID can be accompanied with a ^ or ~ symbol which acts as a relative reference. ^n refers to the nth parent of the commit while ~n refers to the nth grandparent. Of course we can always just explicitly use commit IDs but these short hands can come in handy for quick checks without doing git log to retrieve commit IDs."},{"location":"courses/mlops/git/#revert","title":"Revert","text":"
But instead of moving the branch pointer to a previous commit, we can continue to move forward by adding a new commit to revert certain previous commits.
Sometimes we may want to temporarily switch back to a previous commit just to explore or commit some changes. It's best practice to do this in a separate branch and if we want to save our changes, we need to create a separate PR. Note that if you do checkout a previous commit and submit a PR, you may override the commits in between.\n
There so many different works to work with git and sometimes it can became quickly unruly when fellow developers follow different practices. Here are a few, widely accepted, best practices when it comes to working with commits and branches.
Leverage git tags to mark significant release commits. We can create tags either through the terminal or the online remote interface and this can be done to previous commits as well (in case we forgot).
\n
# Tags\ngit tag # view all existing tags\ngit tag -a <TAG_NAME> -m \"SGD\" # create a tag\ngit checkout -b <BRANCH_NAME> <TAG_NAME> # checkout a specific tag\ngit tag -d <TAG_NAME> # delete local tag\ngit push origin --delete <TAG_NAME> # delete remote tag\ngit fetch --all --tags # fetch all tags from remote\n
\n
Tag names usually adhere to version naming conventions, such as v1.4.2 where the numbers indicate major, minor and bug changes from left to right.
\n
Upcoming live cohorts
\n
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.\n\n
\n Learn more\n\n
To cite this content, please use:
\n
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Git - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/jobs-and-services/","title":"Jobs and Services","text":""},{"location":"courses/mlops/jobs-and-services/#intuition","title":"Intuition","text":"
Our ML workloads have been responsible for everything from data ingestion to model validation:
We can execute these workloads as standalone CLI commands:
With all of our ML workloads implemented (and tested), we're ready to go to production. In this lesson, we'll learn how to convert our ML workloads from CLI commands into a scalable, fault-tolerant and reproducible workflow.
We'll combine our ML workloads up to (and including) model validation into a workflow.
This workflow will then produce model artifacts, which will be saved to our model registry.
And finally, we can serve that model behind an API endpoint to use in production.
Since we have our CLI commands for our ML workloads, we could just execute them one-by-one on our local machine or Workspace. But for efficiency, we're going to combine them all into one script. We'll organize this under a workloads.sh bash script inside our deploy/jobs directory. Here the workloads are very similar to our CLI commands but we have some additional steps to print and save the logs from each of our workloads. For example, our data validation workload looks like this:
At the end of our workloads.sh script, we save our model registry (with our saved model artifacts) and the results from the different workloads to S3. We'll use these artifacts and results later on when we deploy our model as a Service.
If you're doing this lesson on your local laptop, you'll have to add the proper AWS credentials and up the S3 buckets for our workloads script to run successfully.
If you don't want to set up all of this yourself, we highly recommend joining our upcoming live cohort where we'll provide an environment with all of this infrastructure already set up for you so that you just focused on the machine learning."},{"location":"courses/mlops/jobs-and-services/#configuration","title":"Configuration","text":"
Now that we have our single script to execute all workloads, we can execute it with one command (./deploy/jobs/workloads.sh). But even better way is to use Anyscale Jobs to get features like automatic failure handling, email alerts and persisted logs all out of the box for our workloads. And with our cluster_env.yaml, cluster_compute.yaml and workloads.sh files, we can create the configuration for our Anyscale Job with an workloads.yaml file:
Line 3: name of our Anyscale Project (we're organizing it all under the same madewithml project we used for our Workspace setup)
Line 4: name of our cluster environment
Line 5: name of our compute configuration
Line 6-10: runtime environment for our Anyscale Job. The runtime_env here specifies that we should upload our current working_dir to an S3 bucket so that all of our workers when we execute an Anyscale Job have access to the code to use. We also set some environment variables that our workloads will have access to.
Line 11: entrypoint for our Anyscale Job. This is the command that will be executed when we submit our Anyscale Job.
Line 12: maximum number of retries for our Anyscale Job. If our Anyscale Job fails, it will automatically retry up to this number of times.
Warning
Be sure to update the $GITHUB_USERNAME slots inside our deploy/jobs/workloads.yaml configuration to your own GitHub username. This is used to save your model registry and results to a unique path on our shared S3 bucket (s3://madewithml).
Because we're using the exact same cluster environment and compute configuration, what worked during development will work in production. This is a huge benefit of using Anyscale Jobs because we don't have to worry about any environment discrepanices when we deploy our workloads to production. This makes going to production much easier and faster!
And now we can execute our Anyscale Job in one line:
anyscale job submit deploy/jobs/workloads.yaml\n
\nAuthenticating\n\nOutput\n(anyscale +8.8s) Maximum uptime is disabled for clusters launched by this job.\n(anyscale +8.8s) Job prodjob_zqj3k99va8a5jtd895u3ygraup has been successfully submitted. Current state of job: PENDING.\n(anyscale +8.8s) Query the status of the job with `anyscale job list --job-id prodjob_zqj3k99va8a5jtd895u3ygraup`.\n(anyscale +8.8s) Get the logs for the job with `anyscale job logs --job-id prodjob_zqj3k99va8a5jtd895u3ygraup --follow`.\n(anyscale +8.8s) View the job in the UI at https://console.anyscale.com/jobs/prodjob_zqj3k99va8a5jtd895u3ygraup\n(anyscale +8.8s) Use --follow to stream the output of the job when submitting a job.\n
Tip
When we run anyscale cli commands inside our Workspaces, we automatically have our credentials set up for us. But if we're running anyscale cli commands on our local laptop, we'll have to set up the appropriate credentials.
Since we use the exact same cluster (environment and compute) for production as we did for development, we're significantly less likely to run into the environment discrepancy issues that typically arise when going from development to production. However, there can always be small issues that arize from missing credentials, etc. We can easily debug our Anyscale Jobs by inspecting the jobs: Jobs page > choose job > View console logs at the bottom > View Ray workers logs > paste command > Open job-logs directory > View job-driver-raysubmit_XYZ.log. Alternatively, we can also run our Anyscale Job as a Workspace by clicking on the Duplicate as Workspace button the top of a particular Job's page.
After we execute our Anyscale Job, we will have saved our model artifacts to a particular location. We'll now use Anyscale Services to pull from this location to serve our models in production behind a scalable rest endpoint
Similar to Anyscale Jobs, we'll start by creating a serve_model.py and a serve_model.yaml configuration:
# deploy/services/serve_model.py\n\nimport os\nimport subprocess\nfrom madewithml.config import MODEL_REGISTRY # NOQA: E402\nfrom madewithml.serve import ModelDeployment # NOQA: E402\n\n# Copy from S3\ngithub_username = os.environ.get(\"GITHUB_USERNAME\")\nsubprocess.check_output([\"aws\", \"s3\", \"cp\", f\"s3://madewithml/{github_username}/mlflow/\", str(MODEL_REGISTRY), \"--recursive\"])\nsubprocess.check_output([\"aws\", \"s3\", \"cp\", f\"s3://madewithml/{github_username}/results/\", \"./\", \"--recursive\"])\n\n# Entrypoint\nrun_id = [line.strip() for line in open(\"run_id.txt\")][0]\nentrypoint = ModelDeployment.bind(run_id=run_id, threshold=0.9)\n\n# Inference\ndata = {\"query\": \"What is the default batch size for map_batches?\"}\nresponse = requests.post(\"http://127.0.0.1:8000/query\", json=data)\nprint(response.json())\n\n\n# Inference\ndata = {\"query\": \"What is the default batch size for map_batches?\"}\nresponse = requests.post(\"http://127.0.0.1:8000/query\", json=data)\nprint(response.json())\n
In this script, we first pull our previously saved artifacts from our S3 bucket to our local storage and then define the entrypoint for our model.
Tip
Recall that we have the option to scale when we define our service inside out madewithml/serve.py script. And we can scale our compute configuration to meet those demands.
Line 3: name of our Anyscale Project (we're organizing it all under the same madewithml project we used for our Workspace setup)
Line 4: name of our cluster environment
Line 5: name of our compute configuration
Line 6-12: serving configuration that specifies our entry point and details about the working directory, environment variables, etc.
Line 13: rollout strategy for our Anyscale Service. We can either rollout a new version of our service or replace the existing version with the new one.
Warning
Be sure to update the $GITHUB_USERNAME slots inside our deploy/services/serve_model.yaml configuration to your own GitHub username. This is used to pull model artifacts and results from our shared S3 bucket (s3://madewithml).
And now we can execute our Anyscale Service in one line:
# Rollout service\nanyscale service rollout -f deploy/services/serve_model.yaml\n
\nAuthenticating\n\nOutput\n(anyscale +7.3s) Service service2_xwmyv1wcm3i7qan2sahsmybymw has been deployed. Service is transitioning towards: RUNNING.\n(anyscale +7.3s) View the service in the UI at https://console.anyscale.com/services/service2_xwmyv1wcm3i7qan2sahsmybymw\n
Note
If we chose the ROLLOUT strategy, we get a canary rollout (increasingly serving traffic to the new version of our service) by default.
Once our service is up and running, we can query it:
# Query\ncurl -X POST -H \"Content-Type: application/json\" -H \"Authorization: Bearer $SECRET_TOKEN\" -d '{\n \"title\": \"Transfer learning with transformers\",\n \"description\": \"Using transformers for transfer learning on text classification tasks.\"\n}' $SERVICE_ENDPOINT/predict/\n
And we can just as easily rollback to a previous version of our service or terminate it altogether:
# Rollback (to previous version of the Service)\nanyscale service rollback -f $SERVICE_CONFIG --name $SERVICE_NAME\n\n# Terminate\nanyscale service terminate --name $SERVICE_NAME\n
Once we rollout our service, we have several different dashboards that we can use to monitor our service. We can access these dashboards by going to the Services page > choose service > Click the Dashboard button (top right corner) > Ray Dashboard. Here we'll able to see the logs from our Service, metrics, etc.
On the same Dashboard button, we also have a Metrics option that will take us to a Grafana Dashboard. This view has a lot more metrics on incoming requests, latency, errors, etc.
Serving our models may not always work as intended. Even if our model serving logic is correct, there are external dependencies that could causes errors --- such as our model not being stored where it should be, trouble accessing our model registry, etc. For all these cases and more, it's very important to know how to be able to debug our Services.
Services page > choose service > Go to Resource usage section > Click on the cluster link (cluster_for_service_XYZ) > Ray logs (tab at bottom) > paste command > Open worker-XYZ directory > View combined_worker.log
The combination of using Workspaces for development and Job & Services for production make it extremely easy and fast to make the transition. The cluster environment and compute configurations are the exact same so the code that's executing runs on the same conditions. However, we may sometimes want to scale up our production compute configurations to execute Jobs faster or meet the availability/latency demands for our Services. We could address this by creating a new compute configuration:
# Compute config\nexport CLUSTER_COMPUTE_NAME=\"madewithml-cluster-compute-prod\"\nanyscale cluster-compute create deploy/cluster_compute_prod.yaml --name $CLUSTER_COMPUTE_NAME # uses new config with prod compute requirements\n
or by using a one-off configuration to specify the compute changes, where instead of pointing to a previously existing compute configuration, we can define it directly in our Jobs/Services yaml configuration:
And with that, we're able to completely productionize our ML workloads! We have a working service that we can use to make predictions using our trained model. However, what happens when we receive new data or our model's performance regresses over time? With our current approach here, we have to manually execute our Jobs and Services again to udpate our application. In the next lesson, we'll learn how to automate this process with CI/CD workflows that execute our Jobs and Services based on an event (e.g. new data).
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Jobs & Services - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/labeling/","title":"Data Labeling","text":""},{"location":"courses/mlops/labeling/#what-is-data-labeling","title":"What is data labeling","text":"
Labeling (or annotation) is the process of identifying the inputs and outputs that are worth modeling (not just what could be modeled).
use objective as a guide to determine the necessary signals.
explore creating new signals (via combining features, collecting new data, etc.).
iteratively add more features to justify complexity and effort.
It's really important to get our labeling workflows in place before we start performing downstream tasks such as data augmentation, model training, etc.
Warning
Be careful not to include features in the dataset that will not be available during prediction, causing data leaks.
What else can we learn?
It's not just about identifying and labeling our initial dataset. What else can we learn from it?
Show answer
It's also the phase where we can use our deep understanding of the problem to:
- augment the training data split\n- enhance using auxiliary datasets\n- simplify using constraints\n- remove noisy samples\n- improve the labeling process\n
Regardless of whether we have a custom labeling platform or we choose a generalized platform, the process of labeling and all it's related workflows (QA, data import/export, etc.) follow a similar approach.
identify natural labels you may already have (ex. time-series)
consult with domain experts to ensure you're labeling the appropriate signals
decide on the appropriate labels (and hierarchy) for your task
[WHERE] Design the labeling interface:
intuitive, data modality dependent and quick (keybindings are a must!)
avoid option paralysis by allowing the labeler to dig deeper or suggesting likely labels
measure and resolve inter-labeler discrepancy
[HOW] Compose labeling instructions:
examples of each labeling scenario
course of action for discrepancies
Multi-label text classification for our task using Prodigy (labeling + QA)"},{"location":"courses/mlops/labeling/#workflow-setup","title":"Workflow setup","text":"
Establish data pipelines:
[IMPORT] new data for annotation
[EXPORT] annotated data for QA, testing, modeling, etc.
Create a quality assurance (QA) workflow:
separate from labeling workflow (no bias)
communicates with labeling workflow to escalate errors
For the purpose of this course, our data is already labeled, so we'll perform a basic version of ELT (extract, load, transform) to construct the labeled dataset.
In our data-stack and orchestration lessons, we'll construct a modern data stack and programmatically deliver high quality data via DataOps workflows.
projects.csv: projects with id, created time, title and description.
tags.csv: labels (tag category) for the projects by id.
Recall that our objective was to classify incoming content so that the community can discover them easily. These data assets will act as the training data for our first model.
We'll start by extracting data from our sources (external CSV files). Traditionally, our data assets will be stored, versioned and updated in a database, warehouse, etc. We'll learn more about these different data systems later, but for now, we'll load our data as a stand-alone CSV file.
\nid\n created_on\n title\n description\n 0\n 6\n 2020-02-20 06:43:18\n Comparison between YOLO and RCNN on real world...\n Bringing theory to experiment is cool. We can ...\n 1\n 7\n 2020-02-20 06:47:21\n Show, Infer & Tell: Contextual Inference for C...\n The beauty of the work lies in the way it arch...\n 2\n 9\n 2020-02-24 16:24:45\n Awesome Graph Classification\n A collection of important graph embedding, cla...\n 3\n 15\n 2020-02-28 23:55:26\n Awesome Monte Carlo Tree Search\n A curated list of Monte Carlo tree search papers...\n 4\n 19\n 2020-03-03 13:54:31\n Diffusion to Vector\n Reference implementation of Diffusion2Vec (Com...\n
We'll also load the labels (tag category) for our projects.
Apply basic transformations to create our labeled dataset.
# Join projects and tags\ndf = pd.merge(projects, tags, on=\"id\")\ndf.head()\n
\nid\n created_on\n title\n description\n tag\n 0\n 6\n 2020-02-20 06:43:18\n Comparison between YOLO and RCNN on real world...\n Bringing theory to experiment is cool. We can ...\n computer-vision\n 1\n 7\n 2020-02-20 06:47:21\n Show, Infer & Tell: Contextual Inference for C...\n The beauty of the work lies in the way it arch...\n computer-vision\n 2\n 9\n 2020-02-24 16:24:45\n Awesome Graph Classification\n A collection of important graph embedding, cla...\n graph-learning\n 3\n 15\n 2020-02-28 23:55:26\n Awesome Monte Carlo Tree Search\n A curated list of Monte Carlo tree search papers...\n reinforcement-learning\n 4\n 19\n 2020-03-03 13:54:31\n Diffusion to Vector\n Reference implementation of Diffusion2Vec (Com...\n graph-learning\n
df = df[df.tag.notnull()] # remove projects with no tag\n
We could have used the user provided tags as our labels but what if the user added a wrong tag or forgot to add a relevant one. To remove this dependency on the user to provide the gold standard labels, we can leverage labeling tools and platforms. These tools allow for quick and organized labeling of the dataset to ensure its quality. And instead of starting from scratch and asking our labeler to provide all the relevant tags for a given project, we can provide the author's original tags and ask the labeler to add / remove as necessary. The specific labeling tool may be something that needs to be custom built or leverages something from the ecosystem.
As our platform grows, so too will our dataset and labeling needs so it's imperative to use the proper tooling that supports the workflows we'll depend on.
MedCAT: a medical concept annotation tool that can extract information from Electronic Health Records (EHRs) and link it to biomedical ontologies like SNOMED-CT and UMLS.
Generalized labeling solutions
What criteria should we use to evaluate what labeling platform to use?
Show answer
It's important to pick a generalized platform that has all the major labeling features for your data modality with the capability to easily customize the experience.
how easy is it to connect to our data sources (DB, QA, etc.)?
how easy was it to make changes (new features, labeling paradigms)?
how securely is our data treated (on-prem, trust, etc.)
However, as an industry trend, this balance between generalization and specificity is difficult to strike. So many teams put in the upfront effort to create bespoke labeling platforms or used industry specific, niche, labeling tools.
Even with a powerful labeling tool and established workflows, it's easy to see how involved and expensive labeling can be. Therefore, many teams employ active learning to iteratively label the dataset and evaluate the model.
Label a small, initial dataset to train a model.
Ask the trained model to predict on some unlabeled data.
Determine which new data points to label from the unlabeled data based on:
entropy over the predicted class probabilities
samples with lowest predicted, calibrated, confidence (uncertainty sampling)
discrepancy in predictions from an ensemble of trained models
Repeat until the desired performance is achieved.
This can be significantly more cost-effective and faster than labeling the entire dataset.
Active Learning Literature Survey"},{"location":"courses/mlops/labeling/#libraries_1","title":"Libraries","text":"
modAL: a modular active learning framework for Python.
libact: pool-based active learning in Python.
ALiPy: active learning python toolbox, which allows users to conveniently evaluate, compare and analyze the performance of active learning methods.
If we had samples that needed labeling or if we simply wanted to validate existing labels, we can use weak supervision to generate labels as opposed to hand labeling all of them. We could utilize weak supervision via labeling functions to label our existing and new data, where we can create constructs based on keywords, pattern expressions, knowledge bases, etc. And we can add to the labeling functions over time and even mitigate conflicts amongst the different labeling functions. We'll use these labeling functions to create and evaluate slices of our data in the evaluation lesson.
from snorkel.labeling import labeling_function\n\n@labeling_function()\ndef contains_tensorflow(text):\n condition = any(tag in text.lower() for tag in (\"tensorflow\", \"tf\"))\n return \"tensorflow\" if condition else None\n
An easy way to validate our labels (before modeling) is to use the aliases in our auxillary datasets to create labeling functions for the different classes. Then we can look for false positives and negatives to identify potentially mislabeled samples. We'll actually implement a similar kind of inspection approach, but using a trained model as a heuristic, in our dashboards lesson.
Labeling isn't just a one time event or something we repeat identically. As new data is available, we'll want to strategically label the appropriate samples and improve slices of our data that are lacking in quality. Once new data is labeled, we can have workflows that are triggered to start the (re)training process to deploy a new version of our system.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Data Labeling - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/logging/","title":"Logging for ML Systems","text":""},{"location":"courses/mlops/logging/#intuition","title":"Intuition","text":"
Logging is the process of tracking and recording key events that occur in our applications for the purpose of inspection, debugging, etc. They're a whole lot more powerful than print statements because they allow us to send specific pieces of information to specific locations with custom formatting, shared interfaces, etc. This makes logging a key proponent in being able to surface insightful information from the internal processes of our application.
There are a few overarching concepts to be aware of:
Logger: emits the log messages from our application.
Handler: sends log records to a specific location.
Formatter: formats and styles the log records.
There is so much more to logging such as filters, exception logging, etc. but these basics will allows us to do everything we need for our application.
Before we create our specialized, configured logger, let's look at what logged messages look like by using the basic configuration.
import logging\nimport sys\n\n# Create super basic logger\nlogging.basicConfig(stream=sys.stdout, level=logging.DEBUG)\n\n# Logging levels (from lowest to highest priority)\nlogging.debug(\"Used for debugging your code.\")\nlogging.info(\"Informative messages from your code.\")\nlogging.warning(\"Everything works but there is something to be aware of.\")\nlogging.error(\"There's been a mistake with the process.\")\nlogging.critical(\"There is something terribly wrong and process may terminate.\")\n
\nDEBUG:root:Used for debugging your code.\nINFO:root:Informative messages from your code.\nWARNING:root:Everything works but there is something to be aware of.\nERROR:root:There's been a mistake with the process.\nCRITICAL:root:There is something terribly wrong and process may terminate.\n
These are the basic levels of logging, where DEBUG is the lowest priority and CRITICAL is the highest. We defined our logger using basicConfig to emit log messages to stdout (ie. our terminal console), but we also could've written to any other stream or even a file. We also defined our logging to be sensitive to log messages starting from level DEBUG. This means that all of our logged messages will be displayed since DEBUG is the lowest level. Had we made the level ERROR, then only ERROR and CRITICAL log message would be displayed.
import logging\nimport sys\n\n# Create super basic logger\nlogging.basicConfig(stream=sys.stdout, level=logging.ERROR)\n# Logging levels (from lowest to highest priority)\nlogging.debug(\"Used for debugging your code.\")\nlogging.info(\"Informative messages from your code.\")\nlogging.warning(\"Everything works but there is something to be aware of.\")\nlogging.error(\"There's been a mistake with the process.\")\nlogging.critical(\"There is something terribly wrong and process may terminate.\")\n
\nERROR:root:There's been a mistake with the process.\nCRITICAL:root:There is something terribly wrong and process may terminate.\n
[Lines 6-11]: define two different Formatters (determine format and style of log messages), minimal and detailed, which use various LogRecord attributes to create a formatting template for log messages.
[Lines 12-35]: define the different Handlers (details about location of where to send log messages):
console: sends log messages (using the minimal formatter) to the stdout stream for messages above level DEBUG (ie. all logged messages).
info: send log messages (using the detailed formatter) to logs/info.log (a file that can be up to 1 MB and we'll backup the last 10 versions of it) for messages above level INFO.
error: send log messages (using the detailed formatter) to logs/error.log (a file that can be up to 1 MB and we'll backup the last 10 versions of it) for messages above level ERROR.
[Lines 36-40]: attach our different handlers to our root Logger.
We chose to use a dictionary to configure our logger but there are other ways such as Python script, configuration file, etc. Click on the different options below to expand and view the respective implementation.
import logging\nimport logging.config\nfrom rich.logging import RichHandler\n\n# Use config file to initialize logger\nlogging.config.fileConfig(Path(CONFIG_DIR, \"logging.config\"))\nlogger = logging.getLogger()\nlogger.handlers[0] = RichHandler(markup=True) # set rich handler\n
We can load our logger configuration dict like so:
# madewithml/config.py\nimport logging\n\n# Logger\nlogging_config = {...}\nlogging.config.dictConfig(logging_config)\nlogger = logging.getLogger()\n\n# Sample messages (note that we use configured `logger` now)\nlogger.debug(\"Used for debugging your code.\")\nlogger.info(\"Informative messages from your code.\")\nlogger.warning(\"Everything works but there is something to be aware of.\")\nlogger.error(\"There's been a mistake with the process.\")\nlogger.critical(\"There is something terribly wrong and process may terminate.\")\n
\nDEBUG Used for debugging your code. config.py:71\nINFO Informative messages from your code. config.py:72\nWARNING Everything works but there is something to be aware of. config.py:73\nERROR There's been a mistake with the process. config.py:74\nCRITICAL There is something terribly wrong and process may terminate. config.py:75\n
Our logged messages become stored inside the respective files in our logs directory:
from config import logger\nlogger.info(\"\u2705 Training complete!\")\n
All of our log messages are at the INFO level but while developing we may have had to use DEBUG levels and we also add some ERROR or CRITICAL log messages if our system behaves in an unintended manner.
what: log all the necessary details you want to surface from our application that will be useful during development and afterwards for retrospective inspection.
where: a best practice is to not clutter our modular functions with log statements. Instead we should log messages outside of small functions and inside larger workflows. For example, there are no log messages inside any of our scripts except the main.py and train.py files. This is because these scripts use the smaller functions defined in the other scripts (data.py, evaluate.py, etc.). If we ever feel that we the need to log within our other functions, then it usually indicates that the function needs to be broken down further.
When it comes to saving our logs, we could simply upload our logs to a cloud blog storage (ex. S3 or Google Cloud Storage). Or for a more production-grade logging option, we could consider the Elastic stack.
In the next lesson, we'll learn how to document our code and automatically generate high quality docs for our application.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Logging - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
Throughout our development so far, there are so many different commands to keep track of. To help organize everything, we're going to use a Makefile which is a automation tool that organizes our commands. We'll start by create this file in our project's root directory.
touch Makefile\n
At the top of our Makefile we need to specify the shell environment we want all of our commands to execute in:
Inside our Makefile, we'll be creating a list of rules. These rules have a target which can sometimes have prerequisites that need to be met (can be other targets) and on the next line a Tab followed by a recipe which specifies how to create the target.
# Makefile\ntarget: prerequisites\n<TAB> recipe\n
For example, if we wanted to create a rule for styling our files, we would add the following to our Makefile:
# Styling\nstyle:\n black .\n flake8\n python3 -m isort .\n
Tabs vs. spaces
Makefiles require that indention be done with a , instead of spaces where we'll receive an error:
\nMakefile:: *** missing separator. Stop.\n\nLuckily, editors like VSCode automatically change indentation to tabs even if other files use spaces."},{"location":"courses/mlops/makefile/#targets","title":"Targets","text":"
We can execute any of the rules by typing make <target> in the terminal:
A Makefile is called as such because traditionally the targets are supposed to be files we can make. However, Makefiles are also commonly used as command shortcuts, which can lead to confusion when a Makefile target and a file share the same name! For example if we have a file called venv (which we do) and a target in your Makefile called venv, when you run make venv we'll get this message:
\n
$ make venv\n
\n
\nmake: `venv' is up to date.\n
\n\n
In this situation, this is the intended behavior because if a virtual environment already exists, then we don't want ot make that target again. However, sometimes, we'll name our targets and want them to execute whether it exists as an actual file or not. In these scenarios, we want to define a PHONY target in our makefile by adding this line above the target:\n
.PHONY: <target_name>\n
\n
Most of the rules in our Makefile will require the PHONY target because we want them to execute even if there is a file sharing the target's name.
\n
# Styling\n.PHONY: style\nstyle:\n black .\n flake8\n isort .\n
Before we make a target, we can attach prerequisites to them. These can either be file targets that must exist or PHONY target commands that need to be executed prior to making this target. For example, we'll set the style target as a prerequisite for the clean target so that all files are formatted appropriately prior to cleaning them.
We can also set and use variables inside our Makefile to organize all of our rules.
\n
\n
\n
We can set the variables directly inside the Makefile. If the variable isn't defined in the Makefile, then it would default to any environment variable with the same name.\n
# Set variable\nMESSAGE := \"hello world\"\n\n# Use variable\ngreeting:\n @echo ${MESSAGE}\n
\n
\n
\n
We can also use variables passed in when executing the rule like so (ensure that the variable is not overridden inside the Makefile):\n
Each line in a recipe for a rule will execute in a separate sub-shell. However for certain recipes such as activating a virtual environment and loading packages, we want to execute all steps in one shell. To do this, we can add the .ONESHELL special target above any target.
However this is only available in Make version 3.82 and above and most Macs currently use version 3.81. You can either update to the current version or chain your commands with && as we did previously:
The last thing we'll add to our Makefile (for now at least) is a help target to the very top. This rule will provide an informative message for this Makefile's capabilities:
Even though we've trained and thoroughly evaluated our model, the real work begins once we deploy to production. This is one of the fundamental differences between traditional software engineering and ML development. Traditionally, with rule based, deterministic, software, the majority of the work occurs at the initial stage and once deployed, our system works as we've defined it. But with machine learning, we haven't explicitly defined how something works but used data to architect a probabilistic solution. This approach is subject to natural performance degradation over time, as well as unintended behavior, since the data exposed to the model will be different from what it has been trained on. This isn't something we should be trying to avoid but rather understand and mitigate as much as possible. In this lesson, we'll understand the short comings from attempting to capture performance degradation in order to motivate the need for drift detection.
Tip
We highly recommend that you explore this lesson after completing the previous lessons since the topics (and code) are iteratively developed. We did, however, create the monitoring-ml repository for a quick overview with an interactive notebook.
The first step to insure that our model is performing well is to ensure that the actual system is up and running as it should. This can include metrics specific to service requests such as latency, throughput, error rates, etc. as well as infrastructure utilization such as CPU/GPU utilization, memory, etc.
Fortunately, most cloud providers and even orchestration layers will provide this insight into our system's health for free through a dashboard. In the event we don't, we can easily use Grafana, Datadog, etc. to ingest system performance metrics from logs to create a customized dashboard and set alerts.
Unfortunately, just monitoring the system's health won't be enough to capture the underlying issues with our model. So, naturally, the next layer of metrics to monitor involves the model's performance. These could be quantitative evaluation metrics that we used during model evaluation (accuracy, precision, f1, etc.) but also key business metrics that the model influences (ROI, click rate, etc.).
It's usually never enough to just analyze the cumulative performance metrics across the entire span of time since the model has been deployed. Instead, we should also inspect performance across a period of time that's significant for our application (ex. daily). These sliding metrics might be more indicative of our system's health and we might be able to identify issues faster by not obscuring them with historical data.
\ud83d\udc49 \u00a0 Follow along interactive notebook in the monitoring-ml repository as we implement the concepts below.
import matplotlib.pyplot as plt\nimport numpy as np\nimport seaborn as sns\nsns.set_theme()\n
# Cumulative f1\ncumulative_f1 = [np.mean(hourly_f1[:n]) for n in range(1, len(hourly_f1)+1)]\nprint (f\"Average cumulative f1 on the last day: {np.mean(cumulative_f1[-24:]):.1f}\")\n
\nAverage cumulative f1 on the last day: 93.7\n
# Sliding f1\nwindow_size = 24\nsliding_f1 = np.convolve(hourly_f1, np.ones(window_size)/window_size, mode=\"valid\")\nprint (f\"Average sliding f1 on the last day: {np.mean(sliding_f1[-24:]):.1f}\")\n
We may need to monitor metrics at various window sizes to catch performance degradation as soon as possible. Here we're monitoring the overall f1 but we can do the same for slices of data, individual classes, etc. For example, if we monitor the performance on a specific tag, we may be able to quickly catch new algorithms that were released for that tag (ex. new transformer architecture).
We may not always have the ground-truth outcomes available to determine the model's performance on production inputs. This is especially true if there is significant lag or annotation is required on the real-world data. To mitigate this, we could:
devise an approximate signal that can help us estimate the model's performance. For example, in our tag prediction task, we could use the actual tags that an author attributes to a project as the intermediary labels until we have verified labels from an annotation pipeline.
label a small subset of our live dataset to estimate performance. This subset should try to be representative of the various distributions in the live data.
However, approximate signals are not always available for every situation because there is no feedback on the ML system\u2019s outputs or it\u2019s too delayed. For these situations, a recent line of research relies on the only component that\u2019s available in all situations: the input data.
Mandoline: Model Evaluation under Distribution Shift
The core idea is to develop slicing functions that may potentially capture the ways our data may experience distribution shift. These slicing functions should capture obvious slices such as class labels or different categorical feature values but also slices based on implicit metadata (hidden aspects of the data that are not explicit feature columns). These slicing functions are then applied to our labeled dataset to create matrices with the corresponding labels. The same slicing functions are applied to our unlabeled production data to approximate what the weighted labels would be. With this, we can determine the approximate performance! The intuition here is that we can better approximate performance on our unlabeled dataset based on the similarity between the labeled slice matrix and unlabeled slice matrix. A core dependency of this assumption is that our slicing functions are comprehensive enough that they capture the causes of distributional shift.
Warning
If we wait to catch the model decay based on the performance, it may have already caused significant damage to downstream business pipelines that are dependent on it. We need to employ more fine-grained monitoring to identify the sources of model drift prior to actual performance degradation.
We need to first understand the different types of issues that can cause our model's performance to decay (model drift). The best way to do this is to look at all the moving pieces of what we're trying to model and how each one can experience drift.
Entity Description Drift \\(X\\) inputs (features) data drift \\(\\rightarrow P(X) \\neq P_{ref}(X)\\) \\(y\\) outputs (ground-truth) target drift \\(\\rightarrow P(y) \\neq P_{ref}(y)\\) \\(P(y \\vert X)\\) actual relationship between \\(X\\) and \\(y\\) concept drift \\(\\rightarrow P(y \\vert X) \\neq P_{ref}(y \\vert X)\\)
Data drift, also known as feature drift or covariate shift, occurs when the distribution of the production data is different from the training data. The model is not equipped to deal with this drift in the feature space and so, it's predictions may not be reliable. The actual cause of drift can be attributed to natural changes in the real-world but also to systemic issues such as missing data, pipeline errors, schema changes, etc. It's important to inspect the drifted data and trace it back along it's pipeline to identify when and where the drift was introduced.
Warning
Besides just looking at the distribution of our input data, we also want to ensure that the workflows to retrieve and process our input data is the same during training and serving to avoid training-serving skew. However, we can skip this step if we retrieve our features from the same source location for both training and serving, ie. from a feature store.
Data drift can occur in either continuous or categorical features.
As data starts to drift, we may not yet notice significant decay in our model's performance, especially if the model is able to interpolate well. However, this is a great opportunity to potentially retrain before the drift starts to impact performance.
Besides just the input data changing, as with data drift, we can also experience drift in our outcomes. This can be a shift in the distributions but also the removal or addition of new classes with categorical tasks. Though retraining can mitigate the performance decay caused target drift, it can often be avoided with proper inter-pipeline communication about new classes, schema changes, etc.
Besides the input and output data drifting, we can have the actual relationship between them drift as well. This concept drift renders our model ineffective because the patterns it learned to map between the original inputs and outputs are no longer relevant. Concept drift can be something that occurs in various patterns:
gradually over a period of time
abruptly as a result of an external event
periodically as a result of recurring events
All the different types of drift we discussed can can occur simultaneously which can complicated identifying the sources of drift.
Now that we've identified the different types of drift, we need to learn how to locate and how often to measure it. Here are the constraints we need to consider:
reference window: the set of points to compare production data distributions with to identify drift.
test window: the set of points to compare with the reference window to determine if drift has occurred.
Since we're dealing with online drift detection (ie. detecting drift in live production data as opposed to past batch data), we can employ either a fixed or sliding window approach to identify our set of points for comparison. Typically, the reference window is a fixed, recent subset of the training data while the test window slides over time.
Scikit-multiflow provides a toolkit for concept drift detection techniques directly on streaming data. The package offers windowed, moving average functionality (including dynamic preprocessing) and even methods around concepts like gradual concept drift.
We can also compare across various window sizes simultaneously to ensure smaller cases of drift aren't averaged out by large window sizes.
The first form of measurement can be rule-based such as validating expectations around missing values, data types, value ranges, etc. as we did in our data testing lesson. The difference now is that we'll be validating these expectations on live production data.
Our task may involve univariate (1D) features that we will want to monitor. While there are many types of hypothesis tests we can use, a popular option is the Kolmogorov-Smirnov (KS) test.
The KS test determines the maximum distance between two distribution's cumulative density functions. Here, we'll measure if there is any drift on the size of our input text feature between two different data subsets.
Tip
While text is a direct feature in our task, we can also monitor other implicit features such as % of unknown tokens in text (need to maintain a training vocabulary), etc. While they may not be used for our machine learning model, they can be great indicators for detecting drift.
Similarly, for categorical data (input features, targets, etc.), we can apply the Pearson's chi-squared test to determine if a frequency of events in production is consistent with a reference distribution.
We're creating a categorical variable for the # of tokens in our text feature but we could very very apply it to the tag distribution itself, individual tags (binary), slices of tags, etc.
from alibi_detect.cd import ChiSquareDrift\n
# Reference\ndf.token_count = df.num_tokens.apply(lambda x: \"small\" if x <= 10 else (\"medium\" if x <=25 else \"large\"))\nref = df.token_count[0:200].to_numpy()\nplt.hist(ref, alpha=0.75, label=\"reference\")\nplt.legend()\n
As we can see, measuring drift is fairly straightforward for univariate data but difficult for multivariate data. We'll summarize the reduce and measure approach outlined in the following paper: Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift.
We vectorized our text using tf-idf (to keep modeling simple), which has high dimensionality and is not semantically rich in context. However, typically with text, word/char embeddings are used. So to illustrate what drift detection on multivariate data would look like, let's represent our text using pretrained embeddings.
Be sure to refer to our embeddings and transformers lessons to learn more about these topics. But note that detecting drift on multivariate text embeddings is still quite difficult so it's typically more common to use these methods applied to tabular features or images.
We'll start by loading the tokenizer from a pretrained model.
Next, we'll load the pretrained model's weights and use the TransformerEmbedding object to extract the embeddings from the hidden state (averaged across tokens).
from alibi_detect.models.pytorch import TransformerEmbedding\n
# Embedding layer\nemb_type = \"hidden_state\"\nlayers = [-x for x in range(1, 9)] # last 8 layers\nembedding_layer = TransformerEmbedding(model_name, emb_type, layers)\n
Now we need to use a dimensionality reduction method to reduce our representations dimensions into something more manageable (ex. 32 dim) so we can run our two-sample tests on to detect drift. Popular options include:
Principle component analysis (PCA): orthogonal transformations that preserve the variability of the dataset.
Autoencoders (AE): networks that consume the inputs and attempt to reconstruct it from an lower dimensional space while minimizing the error. These can either be trained or untrained (the Failing loudly paper recommends untrained).
Black box shift detectors (BBSD): the actual model trained on the training data can be used as a dimensionality reducer. We can either use the softmax outputs (multivariate) or the actual predictions (univariate).
import torch\nimport torch.nn as nn\n
# Device\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(device)\n
We can wrap all of the operations above into one preprocessing function that will consume input text and produce the reduced representation.
from alibi_detect.cd.pytorch import preprocess_drift\nfrom functools import partial\n
# Preprocessing with the reducer\nmax_len = 100\nbatch_size = 32\npreprocess_fn = partial(preprocess_drift, model=reducer, tokenizer=tokenizer,\n max_len=max_len, batch_size=batch_size, device=device)\n
"},{"location":"courses/mlops/monitoring/#maximum-mean-discrepancy-mmd","title":"Maximum Mean Discrepancy (MMD)","text":"
After applying dimensionality reduction techniques on our multivariate data, we can use different statistical tests to calculate drift. A popular option is Maximum Mean Discrepancy (MMD), a kernel-based approach that determines the distance between two distributions by computing the distance between the mean embeddings of the features from both distributions.
So far we've applied our drift detection methods on offline data to try and understand what reference window sizes should be, what p-values are appropriate, etc. However, we'll need to apply these methods in the online production setting so that we can catch drift as easy as possible.
Many monitoring libraries and platforms come with online equivalents for their detection methods.
Typically, reference windows are large so that we have a proper benchmark to compare our production data points to. As for the test window, the smaller it is, the more quickly we can catch sudden drift. Whereas, a larger test window will allow us to identify more subtle/gradual drift. So it's best to compose windows of different sizes to regularly monitor.
As data starts to flow in, we can use the detector to predict drift at every point. Our detector should detect drift sooner in our drifter dataset than in our normal data.
def simulate_production(test_window):\n i = 0\n online_mmd_drift_detector.reset()\n for text in test_window:\n result = online_mmd_drift_detector.predict(text)\n is_drift = result[\"data\"][\"is_drift\"]\n if is_drift:\n break\n else:\n i += 1\n print (f\"{i} steps\")\n
There are also several considerations around how often to refresh both the reference and test windows. We could base in on the number of new observations or time without drift, etc. We can also adjust the various thresholds (ERT, window size, etc.) based on what we learn about our system through monitoring.
With drift, we're comparing a window of production data with reference data as opposed to looking at any one specific data point. While each individual point may not be an anomaly or outlier, the group of points may cause a drift. The easiest way to illustrate this is to imagine feeding our live model the same input data point repeatedly. The actual data point may not have anomalous features but feeding it repeatedly will cause the feature distribution to change and lead to drift.
Unfortunately, it's not very easy to detect outliers because it's hard to constitute the criteria for an outlier. Therefore the outlier detection task is typically unsupervised and requires a stochastic streaming algorithm to identify potential outliers. Luckily, there are several powerful libraries such as PyOD, Alibi Detect, WhyLogs (uses Apache DataSketches), etc. that offer a suite of outlier detection functionality (largely for tabular and image data for now).
Typically, outlier detection algorithms fit (ex. via reconstruction) to the training set to understand what normal data looks like and then we can use a threshold to predict outliers. If we have a small labeled dataset with outliers, we can empirically choose our threshold but if not, we can choose some reasonable tolerance.
When we identify outliers, we may want to let the end user know that the model's response may not be reliable. Additionally, we may want to remove the outliers from the next training set or further inspect them and upsample them in case they're early signs of what future distributions of incoming features will look like.
It's not enough to just be able to measure drift or identify outliers but to also be able to act on it. We want to be able to alert on drift, inspect it and then act on it.
Once we've identified outliers and/or measured statistically significant drift, we need to a devise a workflow to notify stakeholders of the issues. A negative connotation with monitoring is fatigue stemming from false positive alerts. This can be mitigated by choosing the appropriate constraints (ex. alerting thresholds) based on what's important to our specific application. For example, thresholds could be:
fixed values/range for situations where we're concretely aware of expected upper/lower bounds.
if percentage_unk_tokens > 5%:\n trigger_alert()\n
forecasted thresholds dependent on previous inputs, time, etc.
if current_f1 < forecast_f1(current_time):\n trigger_alert()\n
appropriate p-values for different drift detectors (\u2193 p-value = \u2191 confident that the distributions are different).
from alibi_detect.cd import KSDrift\nlength_drift_detector = KSDrift(reference, p_val=0.01)\n
Once we have our carefully crafted alerting workflows in place, we can notify stakeholders as issues arise via email, Slack, PageDuty, etc. The stakeholders can be of various levels (core engineers, managers, etc.) and they can subscribe to the alerts that are relevant for them.
With these pieces of information, we can work backwards from the alert towards identifying the root cause of the issue. Root cause analysis (RCA) is an important first step when it comes to monitoring because we want to prevent the same issue from impacting our system again. Often times, many alerts are triggered but they maybe all actually be caused by the same underlying issue. In this case, we'd want to intelligently trigger just one alert that pinpoints the core issue. For example, let's say we receive an alert that our overall user satisfaction ratings are reducing but we also receive another alert that our North American users also have low satisfaction ratings. Here's the system would automatically assess for drift in user satisfaction ratings across many different slices and aggregations to discover that only users in a specific area are experiencing the issue but because it's a popular user base, it ends up triggering all aggregate downstream alerts as well!
There are many different ways we can act to drift based on the situation. An initial impulse may be to retrain our model on the new data but it may not always solve the underlying issue.
ensure all data expectations have passed.
confirm no data schema changes.
retrain the model on the new shifted dataset.
move the reference window to more recent data or give it more weight.
determine if outliers are potentially valid data points.
Since detecting drift and outliers can involve compute intensive operations, we need a solution that can execute serverless workloads on top of our event data streams (ex. Kafka). Typically these solutions will ingest payloads (ex. model's inputs and outputs) and can trigger monitoring workloads. This allows us to segregate the resources for monitoring from our actual ML application and scale them as needed.
When it actually comes to implementing a monitoring system, we have several options, ranging from fully managed to from-scratch. Several popular managed solutions are Arize, Arthur, Fiddler, Gantry, Mona, WhyLabs, etc., all of which allow us to create custom monitoring views, trigger alerts, etc. There are even several great open-source solutions such as EvidentlyAI, TorchDrift, WhyLogs, etc.
We'll often notice that monitoring solutions are offered as part of the larger deployment option such as Sagemaker, TensorFlow Extended (TFX), TorchServe, etc. And if we're already working with Kubernetes, we could use KNative or Kubeless for serverless workload management. But we could also use a higher level framework such as KFServing or Seldon core that natively use a serverless framework like KNative.
An overview of unsupervised drift detection methods
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
Monitoring and explainability of models in production
Detecting and Correcting for Label Shift with Black Box Predictors
Outlier and anomaly pattern detection on data streams
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Monitoring - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/orchestration/","title":"Orchestration for Machine Learning","text":""},{"location":"courses/mlops/orchestration/#intuition","title":"Intuition","text":"
So far we've implemented our DataOps (ELT, validation, etc.) and MLOps (optimization, training, evaluation, etc.) workflows as Python function calls. This has worked well since our dataset is static and small. But happens when we need to:
schedule these workflows as new data arrives?
scale these workflows as our data grows?
share these workflows to downstream applications?
monitor these workflows?
We'll need to break down our end-to-end ML pipeline into individual workflows that can be orchestrated as needed. There are several tools that can help us so this such as Airflow, Prefect, Dagster, Luigi, Orchest and even some ML focused options such as Metaflow, Flyte, KubeFlow Pipelines, Vertex pipelines, etc. We'll be creating our workflows using AirFlow for its:
wide adoption of the open source platform in industry
Python based software development kit (SDK)
ability to run locally and scale easily
maturity over the years and part of the apache ecosystem
We'll be running Airflow locally but we can easily scale it by running on a managed cluster platform where we can run Python, Hadoop, Spark, etc. on large batch processing jobs (AWS EMR, Google Cloud's Dataproc, on-prem hardware, etc.).
Before we create our specific pipelines, let's understand and implement Airflow's overarching concepts that will allow us to \"author, schedule, and monitor workflows\".
Separate repository
Our work in this lesson will live in a separate repository so create a new directory (outside our mlops-course repository) called data-engineering. All the work in this lesson can be found in our data-engineering repository.
To install and run Airflow, we can either do so locally or with Docker. If using docker-compose to run Airflow inside Docker containers, we'll want to allocate at least 4 GB in memory.
# Configurations\nexport AIRFLOW_HOME=${PWD}/airflow\nAIRFLOW_VERSION=2.3.3\nPYTHON_VERSION=\"$(python --version | cut -d \" \" -f 2 | cut -d \".\" -f 1-2)\"\nCONSTRAINT_URL=\"https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt\"\n\n# Install Airflow (may need to upgrade pip)\npip install \"apache-airflow==${AIRFLOW_VERSION}\" --constraint \"${CONSTRAINT_URL}\"\n\n# Initialize DB (SQLite by default)\nairflow db init\n
This will create an airflow directory with the following components:
We're going to edit the airflow.cfg file to best fit our needs:
# Inside airflow.cfg\nenable_xcom_pickling = True # needed for Great Expectations airflow provider\nload_examples = False # don't clutter webserver with examples\n
And we'll perform a reset to implement these configuration changes.
airflow db reset -y\n
Now we're ready to initialize our database with an admin user, which we'll use to login to access our workflows in the webserver.
# We'll be prompted to enter a password\nairflow users create \\\n--username admin \\\n--firstname FIRSTNAME \\\n--lastname LASTNAME \\\n--role Admin \\\n--email EMAIL\n
The webserver allows us to run and inspect workflows, establish connections to external data storage, manager users, etc. through a UI. Similarly, we could also use Airflow's REST API or Command-line interface (CLI) to perform the same operations. However, we'll be using the webserver because it's convenient to visually inspect our workflows.
We'll explore the different components of the webserver as we learn about Airflow and implement our workflows.
Next, we need to launch our scheduler, which will execute and monitor the tasks in our workflows. The schedule executes tasks by reading from the metadata database and ensures the task has what it needs to finish running. We'll go ahead and execute the following commands on the separate terminal window:
# Launch scheduler (in separate terminal)\nsource venv/bin/activate\nexport AIRFLOW_HOME=${PWD}/airflow\nexport OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES\nairflow scheduler\n
As our scheduler reads from the metadata database, the executor determines what worker processes are necessary for the task to run to completion. Since our default database SQLlite, which can't support multiple connections, our default executor is the Sequential Executor. However, if we choose a more production-grade database option such as PostgresSQL or MySQL, we can choose scalable Executor backends Celery, Kubernetes, etc. For example, running Airflow with Docker uses PostgresSQL as the database and so uses the Celery Executor backend to run tasks in parallel.
Workflows are defined by directed acyclic graphs (DAGs), whose nodes represent tasks and edges represent the data flow relationship between the tasks. Direct and acyclic implies that workflows can only execute in one direction and a previous, upstream task cannot run again once a downstream task has started.
DAGs can be defined inside Python workflow scripts inside the airflow/dags directory and they'll automatically appear (and continuously be updated) on the webserver. Before we start creating our DataOps and MLOps workflows, we'll learn about Airflow's concepts via an example DAG outlined in airflow/dags/example.py. Execute the following commands in a new (3rd) terminal window:
Inside each workflow script, we can define some default arguments that will apply to all DAGs within that workflow.
# Default DAG args\ndefault_args = {\n \"owner\": \"airflow\",\n}\n
Typically, our DAGs are not the only ones running in an Airflow cluster. However, it can be messy and sometimes impossible to execute different workflows when they require different resources, package versions, etc. For teams with multiple projects, it\u2019s a good idea to use something like the KubernetesPodOperator to execute each job using an isolated docker image.
We can initialize DAGs with many parameters (which will override the same parameters in default_args) and in several different ways:
using a with statement
from airflow import DAG\n\nwith DAG(\n dag_id=\"example\",\n description=\"Example DAG\",\n default_args=default_args,\n schedule_interval=None,\n start_date=days_ago(2),\n tags=[\"example\"],\n) as example:\n # Define tasks\n pass\n
There are many parameters that we can initialize our DAGs with, including a start_date and a schedule_interval. While we could have our workflows execute on a temporal cadence, many ML workflows are initiated by events, which we can map using sensors and hooks to external databases, file systems, etc.
Tasks are the operations that are executed in a workflow and are represented by nodes in a DAG. Each task should be a clearly defined single operation and it should be idempotent, which means we can execute it multiple times and expect the same result and system state. This is important in the event we need to retry a failed task and don't have to worry about resetting the state of our system. Like DAGs, there are several different ways to implement tasks:
using the task decorator
from airflow.decorators import dag, task\nfrom airflow.utils.dates import days_ago\n\n@dag(\n dag_id=\"example\",\n description=\"Example DAG with task decorators\",\n default_args=default_args,\n schedule_interval=None,\n start_date=days_ago(2),\n tags=[\"example\"],\n)\ndef example():\n @task\n def task_1():\n return 1\n @task\n def task_2(x):\n return x+1\n
Though the graphs are directed, we can establish certain trigger rules for each task to execute on conditional successes or failures of the parent tasks.
The first method of creating tasks involved using Operators, which defines what exactly the task will be doing. Airflow has many built-in Operators such as the BashOperator or PythonOperator, which allow us to execute bash and Python commands respectively.
There are also many other Airflow native Operators (email, S3, MySQL, Hive, etc.), as well as community maintained provider packages (Kubernetes, Snowflake, Azure, AWS, Salesforce, Tableau, etc.), to execute tasks specific to certain platforms or tools.
We can also create our own custom Operators by extending the BashOperator class.
Once we've defined our tasks using Operators or as decorated functions, we need to define the relationships between them (edges). The way we define the relationships depends on how our tasks were defined:
When we use task decorators, we can see how values can be passed between tasks. But, how can we pass values when using Operators? Airflow uses XComs (cross communications) objects, defined with a key, value, timestamp and task_id, to push and pull values between tasks. When we use decorated functions, XComs are being used under the hood but it's abstracted away, allowing us to pass values amongst Python functions seamlessly. But when using Operators, we'll need to explicitly push and pull the values as we need it.
We can also view our XComs on the webserver by going to Admin >> XComs:
Warning
The data we pass between tasks should be small (metadata, metrics, etc.) because Airflow's metadata database is not equipped to hold large artifacts. However, if we do need to store and use the large results of our tasks, it's best to use an external data storage (blog storage, model registry, etc.) and perform heavy processing using Spark or inside data systems like a data warehouse.
Our DAG is initially paused since we specified dags_are_paused_at_creation = True inside our airflow.cfg configuration, so we'll have to manually execute this DAG by clicking on it > unpausing it (toggle) > triggering it (button). To view the logs for any of the tasks in our DAG run, we can click on the task > Log.
Note
We could also use Airflow's REST API (will configured authorization) or Command-line interface (CLI) to inspect and trigger workflows (and a whole lot more). Or we could even use the trigger_dagrun Operator to trigger DAGs from within another workflow.
# CLI to run dags\nairflow dags trigger <DAG_ID>\n
Had we specified a start_date and schedule_interval when defining the DAG, it would have have automatically executed at the appropriate times. For example, the DAG below will have started two days ago and will be triggered at the start of every day.
Depending on the start_date and schedule_interval, our workflow should have been triggered several times and Airflow will try to catchup to the current time. We can avoid this by setting catchup=False when defining the DAG. We can also set this configuration as part of the default arguments:
However, if we did want to run particular runs in the past, we can manually backfill what we need.
We could also specify a cron expression for our schedule_interval parameter or even use cron presets.
Airflow's Scheduler will run our workflows one schedule_interval from the start_date. For example, if we want our workflow to start on 01-01-1983 and run @daily, then the first run will be immediately after 01-01-1983T11:59.
While it may make sense to execute many data processing workflows on a scheduled interval, machine learning workflows may require more nuanced triggers. We shouldn't be wasting compute by running executing our workflows just in case we have new data. Instead, we can use sensors to trigger workflows when some external condition is met. For example, we can initiate data processing when a new batch of annotated data appears in a database or when a specific file appears in a file system, etc.
There's so much more to Airflow (monitoring, Task groups, smart senors, etc.) so be sure to explore them as you need them by using the official documentation.
Now that we've reviewed Airflow's major concepts, we're ready to create the DataOps workflows. It's the exact same workflow we defined in our data stack lesson -- extract, load and transform -- but this time we'll be doing everything programmatically and orchestrating it with Airflow.
We'll start by creating the script where we'll define our workflows:
"},{"location":"courses/mlops/orchestration/#extract-and-load","title":"Extract and load","text":"
We're going to use the Airbyte connections we set up in our data-stack lesson but this time we're going to programmatically trigger the data syncs with Airflow. First, let's ensure that Airbyte is running on a separate terminal in it's repository:
git clone https://github.com/airbytehq/airbyte.git # skip if already create in data-stack lesson\ncd airbyte\ndocker-compose up\n
Next, let's install the required packages and establish the connection between Airbyte and Airflow:
The string in the CONNECTION_ID position is the connection's id.
We can trigger our DAG right now and view the extracted data be loaded into our BigQuery data warehouse but we'll continue developing and execute our DAG once the entire DataOps workflow has been defined.
The specific process of where and how we extract our data can be bespoke but what's important is that we have validation at every step of the way. We'll once again use Great Expectations, as we did in our testing lesson, to validate our extracted and loaded data before transforming it.
With the Airflow concepts we've learned so far, there are many ways to use our data validation library to validate our data. Regardless of what data validation tool we use (ex. Great Expectations, TFX, AWS Deequ, etc.) we could use the BashOperator, PythonOperator, etc. to run our tests. However, Great Expectations has a Airflow Provider package to make it even easier to validate our data. This package contains a GreatExpectationsOperator which we can use to execute specific checkpoints as tasks.
But first, before we can create our tests, we need to define a new datasource within Great Expectations for our Google BigQuery data warehouse. This will require several packages and exports:
What data would you like Great Expectations to connect to?\n 1. Files on a filesystem (for processing with Pandas or Spark)\n2. Relational database (SQL) \ud83d\udc48\n
What are you processing your files with?\n1. MySQL\n2. Postgres\n3. Redshift\n4. Snowflake\n5. BigQuery \ud83d\udc48\n6. other - Do you have a working SQLAlchemy connection string?\n
This will open up an interactive notebook where we can fill in the following details:
Next, we can create a suite of expectations for our data assets:
great_expectations suite new\n
How would you like to create your Expectation Suite?\n 1. Manually, without interacting with a sample batch of data (default)\n2. Interactively, with a sample batch of data \ud83d\udc48\n 3. Automatically, using a profiler\n
Select a datasource\n 1. dwh \ud83d\udc48\n
Which data asset (accessible by data connector \"default_inferred_data_connector_name\") would you like to use?\n 1. mlops_course.projects \ud83d\udc48\n 2. mlops_course.tags\n
Name the new Expectation Suite [mlops.projects.warning]: projects\n
This will open up an interactive notebook where we can define our expectations. Repeat the same for creating a suite for our tags data asset as well.
Expectations for mlops_course.projects
Table expectations
# data leak\nvalidator.expect_compound_columns_to_be_unique(column_list=[\"title\", \"description\"])\n
Once we've validated our extracted and loaded data, we're ready to transform it. Our DataOps workflows are not specific to any particular downstream application so the transformation must be globally relevant (ex. cleaning missing data, aggregation, etc.). Just like in our data stack lesson, we're going to use dbt to transform our data. However, this time, we're going to do everything programmatically using the open-source dbt-core package.
In the root of our data-engineering repository, initialize our dbt directory with the following command:
dbt init dbf_transforms\n
Which database would you like to use?\n[1] bigquery \ud83d\udc48\n
We'll prepare our dbt models as we did using the dbt Cloud IDE in the previous lesson.
cd dbt_transforms\nrm -rf models/example\nmkdir models/labeled_projects\ntouch models/labeled_projects/labeled_projects.sql\ntouch models/labeled_projects/schema.yml\n
and add the following code to our model files:
-- models/labeled_projects/labeled_projects.sql\nSELECT p.id, created_on, title, description, tag\nFROM `made-with-ml-XXXXXX.mlops_course.projects` p -- REPLACE\nLEFT JOIN `made-with-ml-XXXXXX.mlops_course.tags` t -- REPLACE\nON p.id = t.id\n
# models/labeled_projects/schema.yml\n\nversion: 2\n\nmodels:\n- name: labeled_projects\ndescription: \"Tags for all projects\"\ncolumns:\n- name: id\ndescription: \"Unique ID of the project.\"\ntests:\n- unique\n- not_null\n- name: title\ndescription: \"Title of the project.\"\ntests:\n- not_null\n- name: description\ndescription: \"Description of the project.\"\ntests:\n- not_null\n- name: tag\ndescription: \"Labeled tag for the project.\"\ntests:\n- not_null\n
And we can use the BashOperator to execute our dbt commands like so:
While we developed locally, we could just as easily use Airflow\u2019s dbt cloud provider to connect to our dbt cloud and use the different operators to schedule jobs. This is recommended for production because we can design jobs with proper environment, authentication, schemas, etc.
Connect Airflow with dbt Cloud:
Go to Admin > Connections > +
Connection ID: dbt_cloud_default\nConnection Type: dbt Cloud\nAccount ID: View in URL of https://cloud.getdbt.com/\nAPI Token: View in https://cloud.getdbt.com/#/profile/api/\n
from airflow.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperator\ntransform = DbtCloudRunJobOperator(\n task_id=\"transform\",\n job_id=118680, # Go to dbt UI > click left menu > Jobs > Transform > job_id in URL\n wait_for_termination=True, # wait for job to finish running\n check_interval=10, # check job status\n timeout=300, # max time for job to execute\n)\n
And of course, we'll want to validate our transformations beyond dbt's built-in methods, using great expectations. We'll create a suite and checkpoint as we did above for our projects and tags data assets.
great_expectations suite new # for mlops_course.labeled_projects\n
Expectations for mlops_course.labeled_projects
Table expectations
# data leak\nvalidator.expect_compound_columns_to_be_unique(column_list=[\"title\", \"description\"])\n
Now we have our entire DataOps DAG define and executing it will prepare our data from extraction to loading to transformation (and with validation at every step of the way) for downstream applications.
Typically we'll use sensors to trigger workflows when a condition is met or trigger them directly from the external source via API calls, etc. For our ML use cases, this could be at regular intervals or when labeling or monitoring workflows trigger retraining, etc.
Once we have our data prepared, we're ready to create one of the many potential downstream applications that will depend on it. Let's head back to our mlops-course project and follow the same set up instructions for Airflow (you can stop the Airflow webserver and scheduler from our data-engineering project since we'll reuse PORT 8000).
# Airflow webserver\nsource venv/bin/activate\nexport AIRFLOW_HOME=${PWD}/airflow\nexport GOOGLE_APPLICATION_CREDENTIALS=/Users/goku/Downloads/made-with-ml-XXXXXX-XXXXXXXXXXXX.json # REPLACE\nairflow webserver --port 8080\n# Go to http://localhost:8080\n
We already had an tagifai.elt_data function defined to prepare our data but if we want to leverage the data inside our data warehouse, we'll want to connect to it.
Next, we'll use Great Expectations to validate our data. Even though we've already validated our data, it's a best practice to test for data quality whenever there is a hand-off of data from one place to another. We've already created a checkpoint for our labeled_projects in our testing lesson so we'll just leverage that inside our MLOps DAG.
And with that we have our MLOps workflow defined that uses the prepared data from our DataOps workflow. At this point, we can add additional tasks for offline/online evaluation, deployment, etc. with the same process as above.
The DataOps and MLOps workflows connect to create an ML system that's capable of continually learning. Such a system will guide us with when to update, what exactly to update and how to update it (easily).
We use the word continual (repeat with breaks) instead of continuous (repeat without interruption / intervention) because we're not trying to create a system that will automatically update with new incoming data without human intervention.
Our production system is live and monitored. When an event of interest occurs (ex. drift), one of several events needs to be triggered:
continue: with the currently deployed model without any updates. However, an alert was raised so it should analyzed later to reduce false positive alerts.
improve: by retraining the model to avoid performance degradation causes by meaningful drift (data, target, concept, etc.).
inspect: to make a decision. Typically expectations are reassessed, schemas are reevaluated for changes, slices are reevaluated, etc.
rollback: to a previous version of the model because of an issue with the current deployment. Typically these can be avoided using robust deployment strategies (ex. dark canary).
If we need to improve on the existing version of the model, it's not just the matter of fact of rerunning the model creation workflow on the new dataset. We need to carefully compose the training data in order to avoid issues such as catastrophic forgetting (forget previously learned patterns when presented with new data).
labeling: new incoming data may need to be properly labeled before being used (we cannot just depend on proxy labels).
active learning: we may not be able to explicitly label every single new data point so we have to leverage active learning workflows to complete the labeling process.
QA: quality assurance workflows to ensure that labeling is accurate, especially for known false positives/negatives and historically poorly performing slices of data.
augmentation: increasing our training set with augmented data that's representative of the original dataset.
sampling: upsampling and downsampling to address imbalanced data slices.
evaluation: creation of an evaluation dataset that's representative of what the model will encounter once deployed.
Once we have the proper dataset for retraining, we can kickoff the workflows to update our system!
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Orchestration for Machine Learning - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
Before performing a commit to our local repository, there are a lot of items on our mental todo list, ranging from styling, formatting, testing, etc. And it's very easy to forget some of these steps, especially when we want to \"push to quick fix\". To help us manage all these important steps, we can use pre-commit hooks, which will automatically be triggered when we try to perform a commit. These hooks can ensure that certain rules are followed or specific actions are executed successfully and if any of them fail, the commit will be aborted.
We define our pre-commit hooks via a .pre-commit-config.yaml configuration file. We can either create our yaml configuration from scratch or use the pre-commit CLI to create a sample configuration which we can add to.
Inside the sample configuration, we can see that pre-commit has added some default hooks from it's repository. It specifies the location of the repository, version as well as the specific hook ids to use. We can read about the function of these hooks and add even more by exploring pre-commit's built-in hooks. Many of them also have additional arguments that we can configure to customize the hook.
Be sure to explore the many other built-in hooks because there are some really useful ones that we use in our project. For example, check-merge-conflict to see if there are any lingering merge conflict strings or detect-aws-credentials if we accidentally left our credentials exposed in a file, and so much more.
And we can also exclude certain files from being processed by the hooks by using the optional exclude key. There are many other optional keys we can configure for each hook ID.
Besides pre-commit's built-in hooks, there are also many custom, 3rd party popular hooks that we can choose from. For example, if we want to apply formatting checks with Black as a hook, we can leverage Black's pre-commit hook.
This specific hook is defined under a .pre-commit-hooks.yaml inside Black's repository, as are other custom hooks under their respective package repositories.
We can also create our own local hooks without configuring a separate .pre-commit-hooks.yaml. Here we're defining two pre-commit hooks, test-non-training and clean, to run some commands that we've defined in our Makefile. Similarly, we can run any entry command with arguments to create hooks very quickly.
Our pre-commit hooks will automatically execute when we try to make a commit. We'll be able to see if each hook passed or failed and make any changes. If any of the hooks fail, we have to fix the errors ourselves or, in many instances, reformatting will occur automatically.
Though pre-commit hooks are meant to run before (pre) a commit, we can manually trigger all or individual hooks on all or a set of files.
# Run\npre-commit run --all-files # run all hooks on all files\npre-commit run <HOOK_ID> --all-files # run one hook on all files\npre-commit run --files <PATH_TO_FILE> # run all hooks on a file\npre-commit run <HOOK_ID> --files <PATH_TO_FILE> # run one hook on a file\n
It is highly not recommended to skip running any of the pre-commit hooks because they are there for a reason. But for some highly urgent, world saving commits, we can use the no-verify flag.
# Commit without hooks\ngit commit -m <MESSAGE> --no-verify\n
Highly recommend not doing this because no commit deserves to be force pushed no matter how \"small\" your change was. If you accidentally did this and want to clear the cache, run pre-commit run --all-files and execute the commit message operation again.
In our .pre-commit-config.yaml configuration files, we've had to specify the versions for each of the repositories so we can use their latest hooks. Pre-commit has an autoupdate CLI command which will update these versions as they become available.
# Autoupdate\npre-commit autoupdate\n
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Pre-commit - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
Our data could reside in many different places (databases, files, etc.) and exist in different formats (CSV, JSON, Parquet, etc.). For our application, we'll load the data from a CSV file to a Pandas DataFrame using the read_csv function.
Here is a quick refresher on the Pandas library.
import pandas as pd\n
# Data ingestion\nDATASET_LOC = \"https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/dataset.csv\"\ndf = pd.read_csv(DATASET_LOC)\ndf.head()\n
id created_on title description tag 0 6 2020-02-20 06:43:18 Comparison between YOLO and RCNN on real world... Bringing theory to experiment is cool. We can ... computer-vision 1 7 2020-02-20 06:47:21 Show, Infer & Tell: Contextual Inference for C... The beauty of the work lies in the way it arch... computer-vision 2 9 2020-02-24 16:24:45 Awesome Graph Classification A collection of important graph embedding, cla... other 3 15 2020-02-28 23:55:26 Awesome Monte Carlo Tree Search A curated list of Monte Carlo tree search pape... other 4 25 2020-03-07 23:04:31 AttentionWalk A PyTorch Implementation of \"Watch Your Step: ... other
In our data engineering lesson we'll look at how to continually ingest data from more complex sources (ex. data warehouses)
Next, we need to split our training dataset into train and val data splits.
Use the train split to train the model.
Here the model will have access to both inputs (features) and outputs (labels) to optimize its internal weights.
After each iteration (epoch) through the training split, we will use the val split to determine the model's performance.
Here the model will not use the labels to optimize its weights but instead, we will use the validation performance to optimize training hyperparameters such as the learning rate, etc.
Finally, we will use a separate holdout test dataset to determine the model's performance after training.
This is our best measure of how the model may behave on new, unseen data that is from a similar distribution to our training dataset.
Tip
For our application, we will have a training dataset to split into train and val splits and a separate testing dataset for the test set. While we could have one large dataset and split that into the three splits, it's a good idea to have a separate test dataset. Over time, our training data may grow and our test splits will look different every time. This will make it difficult to compare models against other models and against each other.
We can view the class counts in our dataset by using the pandas.DataFrame.value_counts function:
from sklearn.model_selection import train_test_split\n
For our multi-class task (where each project has exactly one tag), we want to ensure that the data splits have similar class distributions. We can achieve this by specifying how to stratify the split by using the stratify keyword argument with sklearn's train_test_split() function.
Creating proper data splits
What are the criteria we should focus on to ensure proper data splits?
Show answer
the dataset (and each data split) should be representative of data we will encounter
equal distributions of output values across all splits
shuffle your data if it's organized in a way that prevents input variance
avoid random shuffles if your task can suffer from data leaks (ex. time-series)
Before we view our validation split's class counts, recall that our validation split is only test_size of the entire dataset. So we need to adjust the value counts so that we can compare it to the training split's class counts.
Data preprocessing can be categorized into two types of processes: preparation and transformation. We'll explore common preprocessing techniques and then we'll preprocess our dataset.
Warning
Certain preprocessing steps are global (don't depend on our dataset, ex. lower casing text, removing stop words, etc.) and others are local (constructs are learned only from the training split, ex. vocabulary, standardization, etc.). For the local, dataset-dependent preprocessing steps, we want to ensure that we split the data first before preprocessing to avoid data leaks.
Performing SQL joins with existing data tables to organize all the relevant data you need into one view. This makes working with our dataset a whole lot easier.
SELECT * FROM A\nINNER JOIN B on A.id == B.id\n
Warning
We need to be careful to perform point-in-time valid joins to avoid data leaks. For example, if Table B may have features for objects in Table A that were not available at the time inference would have been needed.
First, we'll have to identify the rows with missing values and once we do, there are several approaches to dealing with them.
omit samples with missing values (if only a small subset are missing it)
# Drop a row (sample) by index\ndf.drop([4, 10, ...])\n# Conditionally drop rows (samples)\ndf = df[df.value > 0]\n# Drop samples with any missing feature\ndf = df[df.isnull().any(axis=1)]\n
omit the entire feature (if too many samples are missing the value)
# Drop a column (feature)\ndf.drop([\"A\"], axis=1)\n
fill in missing values for features (using domain knowledge, heuristics, etc.)
# Fill in missing values with mean\ndf.A = df.A.fillna(df.A.mean())\n
may not always seem \"missing\" (ex. 0, null, NA, etc.)
# Replace zeros to NaNs\nimport numpy as np\ndf.A = df.A.replace({\"0\": np.nan, 0: np.nan})\n
similarity: similar to count vectorization but based on similarities in tokens
and many more!
We'll often was to retrieve feature values for an entity (user, item, etc.) over time and reuse the same features across different projects. To ensure that we're retrieving the proper feature values and to avoid duplication of efforts, we can use a feature store.
Curse of dimensionality
What can we do if a feature has lots of unique values but enough data points for each unique value (ex. URL as a feature)?
Show answer
We can encode our data with hashing or using it's attributes instead of the exact entity itself. For example, representing a user by their location and favorites as opposed to using their user ID or representing a webpage with it's domain as opposed to the exact url. This methods effectively decrease the total number of unique feature values and increase the number of data points for each.
We can combine existing input features to create new meaningful signal for helping the model learn. However, there's usually no simple way to know if certain feature combinations will help or not without empirically experimenting with the different combinations. Here, we could use a project's title and description separately as features but we'll combine them to create one input feature.
Since we're dealing with text data, we can apply some common text preprocessing operations. Here, we'll be using Python's built-in regular expressions library re and the Natural Language Toolkit nltk.
def clean_text(text, stopwords=STOPWORDS):\n\"\"\"Clean raw text string.\"\"\"\n # Lower\n text = text.lower()\n\n # Remove stopwords\n pattern = re.compile(r'\\b(' + r\"|\".join(stopwords) + r\")\\b\\s*\")\n text = pattern.sub('', text)\n\n # Spacing and filters\n text = re.sub(r\"([!\\\"'#$%&()*\\+,-./:;<=>?@\\\\\\[\\]^_`{|}~])\", r\" \\1 \", text) # add spacing\n text = re.sub(\"[^A-Za-z0-9]+\", \" \", text) # remove non alphanumeric chars\n text = re.sub(\" +\", \" \", text) # remove multiple spaces\n text = text.strip() # strip white space at the ends\n text = re.sub(r\"http\\S+\", \"\", text) # remove links\n\n return text\n
Note
We could definitely try and include emojis, punctuations, etc. because they do have a lot of signal for the task but it's best to simplify the initial feature set we use to just what we think are the most influential and then we can slowly introduce other features and assess utility.
Once we're defined our function, we can apply it to each row in our dataframe via pandas.DataFrame.apply.
# Apply to dataframe\noriginal_df = df.copy()\ndf.text = df.text.apply(clean_text)\nprint (f\"{original_df.text.values[0]}\\n{df.text.values[0]}\")\n
\nComparison between YOLO and RCNN on real world videos Bringing theory to experiment is cool. We can easily train models in colab and find the results in minutes.\ncomparison yolo rcnn real world videos bringing theory experiment cool easily train models colab find results minutes\n
Warning
We'll want to introduce less frequent features as they become more frequent or encode them in a clever way (ex. binning, extract general attributes, common n-grams, mean encoding using other feature values, etc.) so that we can mitigate the feature value dimensionality issue until we're able to collect more data.
We'll wrap up our cleaning operation by removing columns (pandas.DataFrame.drop) and rows with null tag values (pandas.DataFrame.dropna).
text tag 0 comparison yolo rcnn real world videos bringin... computer-vision 1 show infer tell contextual inference creative ... computer-vision 2 awesome graph classification collection import... other 3 awesome monte carlo tree search curated list m... other 4 attentionwalk pytorch implementation watch ste... other"},{"location":"courses/mlops/preprocessing/#encoding_1","title":"Encoding","text":"
We need to encode our data into numerical values so that our models can process them. We'll start by encoding our text labels into unique indices.
# Label to index\ntags = train_df.tag.unique().tolist()\nnum_classes = len(tags)\nclass_to_index = {tag: i for i, tag in enumerate(tags)}\nclass_to_index\n
text tag 0 comparison yolo rcnn real world videos bringin... 2 1 show infer tell contextual inference creative ... 2 2 awesome graph classification collection import... 3 3 awesome monte carlo tree search curated list m... 3 4 attentionwalk pytorch implementation watch ste... 3
We'll also want to be able to decode our predictions back into text labels. We can do this by creating an index_to_class dictionary and using that to convert encoded labels back into text labels.
def decode(indices, index_to_class):\n return [index_to_class[index] for index in indices]\n
index_to_class = {v:k for k, v in class_to_index.items()}\ndecode(df.head()[\"tag\"].values, index_to_class=index_to_class)\n
Next we'll encode our text as well. Instead of using a random dictionary, we'll use a tokenizer that was used for a pretrained LLM (scibert) to tokenize our text. We'll be fine-tuning this exact model later when we train our model.
Here is a quick refresher on attention and Transformers.
import numpy as np\nfrom transformers import BertTokenizer\n
The tokenizer will convert our input text into a list of token ids and a list of attention masks. The token ids are the indices of the tokens in the vocabulary. The attention mask is a binary mask indicating the position of the token indices so that the model can attend to them (and ignore the pad tokens).
# Bert tokenizer\ntokenizer = BertTokenizer.from_pretrained(\"allenai/scibert_scivocab_uncased\", return_dict=False)\ntext = \"Transfer learning with transformers for text classification.\"\nencoded_inputs = tokenizer([text], return_tensors=\"np\", padding=\"longest\") # pad to longest item in batch\nprint (\"input_ids:\", encoded_inputs[\"input_ids\"])\nprint (\"attention_mask:\", encoded_inputs[\"attention_mask\"])\nprint (tokenizer.decode(encoded_inputs[\"input_ids\"][0]))\n
\ninput_ids: [[ 102 2268 1904 190 29155 168 3267 2998 205 103]]\nattention_mask: [[1 1 1 1 1 1 1 1 1 1]]\n[CLS] transfer learning with transformers for text classification. [SEP]\n
Note that we use padding=\"longest\" in our tokenizer function to pad our inputs to the longest item in the batch. This becomes important when we use batches of inputs later and want to create a uniform input size, where shorted text sequences will be padded with zeros to meet the length of the longest input in the batch.
We'll wrap our tokenization into a tokenize function that we can use to tokenize batches of our data.
We'll wrap up by combining all of our preprocessing operations into function. This way we can easily apply it to different datasets (training, inference, etc.)
Before we start developing any machine learning models, we need to first motivate and design our application. While this is a technical course, this initial product design process is extremely crucial for creating great products. We'll focus on the product design aspects of our application in this lesson and the systems design aspects in the next lesson.
The template below is designed to guide machine learning product development. It involves both the product and systems design (next lesson) aspects of our application:
Product design (What & Why) \u2192 Systems design (How)
\ud83d\udc49 \u00a0 Download a PDF of the ML canvas to use for your own products \u2192 ml-canvas.pdf (right click the link and hit \"Save Link As...\")
Propose the value we can create through a product-centric approach:
product: what needs to be built to help our users reach their goals?
alleviates: how will the product reduce pains?
advantages: how will the product create gains?
Our task
We will build a platform that helps machine learning developers and researchers stay up-to-date on ML content. We'll do this by discovering and categorizing content from popular sources (Reddit, Twitter, etc.) and displaying it on our platform. For simplicity, assume that we already have a pipeline that delivers ML content from popular sources to our platform. We will just focus on developing the ML service that can correctly categorize the content.
product: a service that discovers and categorizes ML content from popular sources.
alleviates: display categorized content for users to discover.
advantages: when users visit our platform to stay up-to-date on ML content, they don't waste time searching for that content themselves in the noisy internet.
How feasible is our solution and do we have the required resources to deliver it (data, $, team, etc.)?
Our task
We have a dataset with ML content that has been labeled. We'll need to assess if it has the necessary signals to meet our objectives.
Sample data point
{\n\"id\": 443,\n\"created_on\": \"2020-04-10 17:51:39\",\n\"title\": \"AllenNLP Interpret\",\n\"description\": \"A Framework for Explaining Predictions of NLP Models\",\n\"tag\": \"natural-language-processing\"\n}\n
Now that we've set up the product design requirements for our ML service, let's move on to the systems design requirements in the next lesson.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Product - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/scripting/","title":"Moving from Notebooks to Scripts","text":""},{"location":"courses/mlops/scripting/#intuition","title":"Intuition","text":"
In this lesson, we'll discuss how to migrate and organize code from our notebook to Python scripts. We'll be using VSCode in this course, but feel free to use any editor you feel comfortable with.
Notebooks have been great so far for development. They're interactive, stateful (don't have to rerun code), and allow us to visualize outputs. However, when we want to a develop quality codebase, we need to move to scripts. Here are some reasons why:
stateless: when we run code in a notebook, it's automatically saved to the global state (memory). This is great for experimentation because code and variables will be readily available across different cells. However, this can be very problematic as well because there can be hidden state that we're not aware of. Scripts, on the other hand, are stateless and we have to explicitly pass variables to functions and classes.
linear: in notebooks, the order in which we execute cells matter. This can be problematic because we can easily execute cells out of order. Scripts, on the other hand, are linear and we have to explicitly execute code for each workload.
testing: As we'll see in our testing lesson, it's significantly easier to compose and run tests on scripts, as opposed to Jupyter notebooks. This is crucial for ensuring that we have quality code that works as expected.
It's always a good idea to start organizing our scripts with a README.md file. This is where we can organize all of the instructions necessary to walkthrough our codebase. Our README has information on how to set up our environment, how to run our scripts, etc.
The contents of the README.md file is what everyone will see when they visit your repository on GitHub. So, it's a good idea to keep it updated with the latest information.
Let's start by moving our code from notebooks to scripts. We're going to start by creating the different files and directories that we'll need for our project. The exact number and name of these scripts is entirely up to us, however, it's best to organize and choose names that relate to a specific workload. For example, data.py will have all of our data related functions and classes. And we can also have scripts for configurations (config.py), shared utilities (utils.py), etc.
Don't worry about the contents in these files that aren't from our notebooks just yet or if our code looks significantly more documented. We'll be taking a closer look at those in the respective lessons.
"},{"location":"courses/mlops/scripting/#functions-and-classes","title":"Functions and classes","text":"
Once we have these ready, we can start moving code from our notebooks to the appropriate scripts. It should intuitive in which script a particular function or class belongs to. If not, we need to rethink how the names of our scripts. For example, train.py has functions from our notebook such as train_step, val_step, train_loop_per_worker, etc.
These code cells are not part of a function or class, so we need to wrap them around a function so that we can easily execute that workload. For example, all of this training logic is wrapped inside a train_model function in train.py that has all the required inputs to execute the workload:
In addition to our core workload scripts, recall that we also have a config.py script. This file will include all of the setup and configuration that all/most of our workloads depend on. For example, setting up our model registry:
We wouldn't have configurations like our ScalingConfig here because that's specific to our training workload. The config.py script is for configurations that are shared across different workloads.
Similarly, we also have a utils.py script to include components that will be reused across different scripts. It's a good idea to organize these shared components here as opposed to the core scripts to avoid circular dependency conflicts (two scripts call on functions from each other). Here is an example of one of our utility functions, set_seeds, that's used in both our train.py and tune.py scripts.
Recall in our setup lesson that we initialized Ray inside our notebooks. We still need to initialize Ray before executing our ML workloads via scripts but we can decide to do this only for the scripts with Ray dependent workloads. For example, at the bottom of our train.py script, we have:
Now that we've set up our scripts, we can start executing them from the command line. In the next lesson, we'll learn how to do this with command-line interfaces (CLI).
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Scripting - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
In this lesson, we're going to serve the machine learning models that we have developed so that we can use them to make predictions on unseen data. And we want to be able to serve our models in a scalable and robust manner so it can deliver high throughput (handle many requests) and low latency (quickly respond to each request). In an effort to be comprehensive, we will implement both batch inference (offline) and online inference (real-time), though we will focus on the latter in the remaining lessons as it's more appropriate for our application.
There are many frameworks to choose from when it comes to model serving, such as Ray Serve, Nvidia Triton, HuggingFace, Bento ML, etc. When choosing between these frameworks, we want to choose the option that will allow us to:
Pythonic: we don't want to learn a new framework to be able to serve our models.
framework agnostic: we want to be able to serve models from all frameworks (PyTorch, TensorFlow, etc.)
scale: (auto)scaling our service should be as easy as changing a configuration.
composition: combine multiple models and business logic into our service.
integrations: integrate with popular API frameworks like FastAPI.
To address all of these requirements (and more), we will be using Ray Serve to create our service. While we'll be specifically using it's integration with FastAPI, there are many other integrations you might want to explore based on your stack (LangChain, Kubernetes, etc.).
We will first implement batch inference (or offline inference), which is when we make predictions on a large batch of data. This is useful when we don't need to serve a model's prediction on input data as soon as the input data is received. For example, our service can be used to make predictions once at the end of every day on the batches of content collected throughout the day. This can be more efficient than making predictions on each content individually if we don't need that kind of low latency.
Let's take a look at our how we can easily implement batch inference with Ray Serve. We'll start with some setup and load the best checkpoint from our training run.
Next, we'll define a Predictor class that will load the model from our checkpoint and then define the __call__ method that will be used to make predictions on our input data.
To do batch inference, we'll be using the map_batches functionality. We previously used map_batches to map (or apply) a preprocessing function across batches (chunks) of our data. We're now using the same concept to apply our predictor across batches of our inference data.
Note that best_checkpoint as a keyword argument to our Predictor class so that we can load the model from that checkpoint. We can pass this in via the fn_constructor_kwargs argument in our map_batches function.
While we can achieve batch inference at scale, many models will need to be served in an real-time manner where we may need to deliver predictions for many incoming requests (high throughput) with low latency. We want to use online inference for our application over batch inference because we want to quickly categorize content as they are received/submitted to our platform so that the community can discover them quickly.
from fastapi import FastAPI\nfrom ray import serve\nimport requests\nfrom starlette.requests import Request\n
We'll start by defining our FastAPI application which involves initializing a predictor (and preprocessor) from the best checkpoint for a particular run (specified by run_id). We'll also define a predict function that will be used to make predictions on our input data.
class ModelDeployment:\n\n def __init__(self, run_id):\n\"\"\"Initialize the model.\"\"\"\n self.run_id = run_id\n mlflow.set_tracking_uri(MLFLOW_TRACKING_URI) # so workers have access to model registry\n best_checkpoint = get_best_checkpoint(run_id=run_id)\n self.predictor = TorchPredictor.from_checkpoint(best_checkpoint)\n self.preprocessor = self.predictor.get_preprocessor()\n\n @app.post(\"/predict/\")\n async def _predict(self, request: Request):\n data = await request.json()\n df = pd.DataFrame([{\"title\": data.get(\"title\", \"\"), \"description\": data.get(\"description\", \"\"), \"tag\": \"\"}])\n results = predict_with_proba(df=df, predictor=self.predictor)\n return {\"results\": results}\n
async def refers to an asynchronous function (when we call the function we don't have to wait for the function to complete executing). The await keyword is used inside an asynchronous function to wait for the completion of the request.json() operation.
We can now combine our FastAPI application with Ray Serve by simply wrapping our application with the serve.ingress decorator. We can further wrap all of this with the serve.deployment decorator to define our deployment configuration (ex. number of replicas, compute resources, etc.). These configurations allow us to easily scale our service as needed.
Now let's run our service and perform some real-time inference.
# Run service\nsorted_runs = mlflow.search_runs(experiment_names=[experiment_name], order_by=[\"metrics.val_loss ASC\"])\nrun_id = sorted_runs.iloc[0].run_id\nserve.run(ModelDeployment.bind(run_id=run_id))\n
\nStarted detached Serve instance in namespace \"serve\".\nDeployment 'default_ModelDeployment:IcuFap' is ready at `http://127.0.0.1:8000/`. component=serve deployment=default_ModelDeployment\nRayServeSyncHandle(deployment='default_ModelDeployment')\n
# Query\ntitle = \"Transfer learning with transformers\"\ndescription = \"Using transformers for transfer learning on text classification tasks.\"\njson_data = json.dumps({\"title\": title, \"description\": description})\nrequests.post(\"http://127.0.0.1:8000/predict/\", data=json_data).json()\n
The issue with neural networks (and especially LLMs) is that they are notoriously overconfident. For every input, they will always make some prediction. And to account for this, we have an other class but that class only has projects that are not in our accepted tags but are still machine learning related nonetheless. Here's what happens when we input complete noise as our input:
To make our service a bit more robust, let's add some custom logic to predict the other class if the probability of the predicted class is below a certain threshold probability.
@serve.deployment(route_prefix=\"/\", num_replicas=\"1\", ray_actor_options={\"num_cpus\": 8, \"num_gpus\": 0})\n@serve.ingress(app)\nclass ModelDeploymentRobust:\n\n def __init__(self, run_id, threshold=0.9):\n\"\"\"Initialize the model.\"\"\"\n self.run_id = run_id\n self.threshold = threshold\n mlflow.set_tracking_uri(MLFLOW_TRACKING_URI) # so workers have access to model registry\n best_checkpoint = get_best_checkpoint(run_id=run_id)\n self.predictor = TorchPredictor.from_checkpoint(best_checkpoint)\n self.preprocessor = self.predictor.get_preprocessor()\n\n @app.post(\"/predict/\")\n async def _predict(self, request: Request):\n data = await request.json()\n df = pd.DataFrame([{\"title\": data.get(\"title\", \"\"), \"description\": data.get(\"description\", \"\"), \"tag\": \"\"}])\n results = predict_with_proba(df=df, predictor=self.predictor)\n\n # Apply custom logic\n for i, result in enumerate(results):\n pred = result[\"prediction\"]\n prob = result[\"probabilities\"]\n if prob[pred] < self.threshold:\n results[i][\"prediction\"] = \"other\"\n\n return {\"results\": results}\n
Tip
It's easier to incorporate custom logic instead of altering the model itself. This way, we won't have to collect new data. change the model's architecture or retrain it. This also makes it really easy to change the custom logic as our product specifications may change (clean separation of product and machine learning).
# Run service\nserve.run(ModelDeploymentRobust.bind(run_id=run_id, threshold=0.9))\n
\nStarted detached Serve instance in namespace \"serve\".\nDeployment 'default_ModelDeploymentRobust:RTbrNg' is ready at `http://127.0.0.1:8000/`. component=serve deployment=default_ModelDeploymentRobust\nRayServeSyncHandle(deployment='default_ModelDeploymentRobust')\n
Now let's see how we perform on the same random noise with our custom logic incorporate into the service.
In this lesson, we'll setup the development environment that we'll be using in all of our lessons. We'll have instructions for both local laptop and remote scalable clusters (Anyscale). While everything will work locally on your laptop, you can sign up to join one of our upcoming live cohorts where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day \u2192 sign up here.
We'll start with defining our cluster, which refers to a group of servers that come together to form one system. Our clusters will have a head node that manages the cluster and it will be connected to a set of worker nodes that will execute workloads for us. These clusters can be fixed in size or autoscale based on our application's compute needs, which makes them highly scalable and performant. We'll create our cluster by defining a compute configuration and an environment.
We'll start by defining our cluster environment which will specify the software dependencies that we'll need for our workloads.
\ud83d\udcbb Local
Your personal laptop will need to have Python installed and we highly recommend using Python 3.10. You can use a tool like pyenv (mac) or pyenv-win (windows) to easily download and switch between Python versions.
pyenv install 3.10.11 # install\npyenv global 3.10.11 # set default\n
Once we have our Python version, we can create a virtual environment to install our dependencies. We'll download our Python dependencies after we clone our repository from git shortly.
Our cluster environment will be defined inside a cluster_env.yaml file. Here we specify some details around our base image (anyscale/ray:2.6.0-py310-cu118) that has our Python version, GPU dependencies, etc.
We could specify any python packages inside pip_packages or conda_packages but we're going to use a requirements.txt file to load our dependencies under post_build_cmds.
Next, we'll define our compute configuration, which will specify our hardware dependencies (head and worker nodes) that we'll need for our workloads.
\ud83d\udcbb Local
Your personal laptop (single machine) will act as the cluster, where one CPU will be the head node and some of the remaining CPU will be the worker nodes (no GPUs required). All of the code in this course will work in any personal laptop though it will be slower than executing the same workloads on a larger cluster.
\ud83d\ude80 Anyscale
Our cluster compute will be defined inside a cluster_compute.yaml file. Here we specify some details around where our compute resources will come from (cloud computing platform like AWS), types of nodes and their counts, etc.
Our worker nodes will be GPU-enabled so we can train our models faster and we set min_workers to 0 so that we can autoscale these workers only when they're needed (up to a maximum of max_workers). This will help us significantly reduce our compute costs without having to manage the infrastructure ourselves.
With our compute and environment defined, we're ready to create our cluster workspace. This is where we'll be developing our ML application on top of our compute, environment and storage.
\ud83d\udcbb Local
Your personal laptop will need to have an interactive development environment (IDE) installed, such as VS Code. For bash commands in this course, you're welcome to use the terminal on VSCode or a separate one.
\ud83d\ude80 Anyscale
We're going to launch an Anyscale Workspace to do all of our development in. Workspaces allow us to use development tools such as VSCode, Jupyter notebooks, web terminal, etc. on top of our cluster compute, environment and storage. This create an \"infinite laptop\" experience that feels like a local laptop experience but on a powerful, scalable cluster.
We have the option to create our Workspace using a CLI but we're going to create it using the web UI (you will receive the required credentials during the cohort). On the UI, we can fill in the following information:
- Workspace name: `madewithml`\n- Project: `madewithml`\n- Cluster environment name: `madewithml-cluster-env`\n# Toggle `Select from saved configurations`\n- Compute config: `madewithml-cluster-compute`\n- Click on the **Start** button to launch the Workspace\n
We have already saved created our Project, cluster environment and compute config so we can select them from the dropdowns but we could just as easily create new ones / update these using the CLI.
With our development workspace all set up, we're ready to start developing. We'll start by following these instructions to create a repository:
Create a new repository
name it Made-With-ML
Toggle Add a README file (very important as this creates a main branch)
Scroll down and click Create repository
Now we're ready to clone the Made With ML repository's contents from GitHub inside our madewithml directory.
export GITHUB_USERNAME=\"YOUR_GITHUB_UESRNAME\" # <-- CHANGE THIS to your username\ngit clone https://github.com/GokuMohandas/Made-With-ML.git .\ngit remote set-url origin https://github.com/$GITHUB_USERNAME/Made-With-ML.git\ngit checkout -b dev\nexport PYTHONPATH=$PYTHONPATH:$PWD # so we can import modules from our scripts\n
\ud83d\udcbb Local
Recall that we created our virtual environment earlier but didn't actually load any Python dependencies yet. We'll clone our repository and then install the packages using the requirements.txt file.
python3 -m pip install -r requirements.txt\n
Caution: make sure that we're installing our Python packages inside our virtual environment.
\ud83d\ude80 Anyscale
Our environment with the appropriate Python version and libraries is already all set for us through the cluster environment we used when setting up our Anyscale Workspace. But if we want to install additional Python packages as we develop, we need to do pip install with the --user flag inside our Workspaces (via terminal) to ensure that our head and all worker nodes receive the package. And then we should also add it to our requirements file so it becomes part of the cluster environment build process next time.
Now we're ready to launch our Jupyter notebook to interactively develop our ML application.
\ud83d\udcbb Local
We already installed jupyter through our requirements.txt file in the previous step, so we can just launch it.
jupyter lab notebooks/madewithml.ipynb\n
\ud83d\ude80 Anyscale
Click on the Jupyter icon \u00a0\u00a0 at the top right corner of our Anyscale Workspace page and this will open up our JupyterLab instance in a new tab. Then navigate to the notebooks directory and open up the madewithml.ipynb notebook.
We'll be using Ray to scale and productionize our ML application. Ray consists of a core distributed runtime along with libraries for scaling ML workloads and has companies like OpenAI, Spotify, Netflix, Instacart, Doordash + many more using it to develop their ML applications. We're going to start by initializing Ray inside our notebooks:
We can also view our cluster resources to view the available compute resources:
ray.cluster_resources()\n
\ud83d\udcbb Local
If you are running this on a local laptop (no GPU), use the CPU count from ray.cluster_resources() to set your resources. For example if your machine has 10 CPUs:
num_workers = 6 # prefer to do a few less than total available CPU (1 for head node + 1 for background tasks)\nresources_per_worker={\"CPU\": 1, \"GPU\": 0}\n
\ud83d\ude80 Anyscale
On our Anyscale Workspace, the ray.cluster_resources() command will produce:
These cluster resources only reflect our head node (1 m5.2xlarge). But recall earlier in our compute configuration that we also added g4dn.xlarge worker nodes (each has 1 GPU and 4 CPU) to our cluster. But because we set min_workers=0, our worker nodes will autoscale ( up to max_workers) as they're needed for specific workloads (ex. training). So we can set the # of workers and resources by worker based on this insight:
Head on over to the next lesson, where we'll motivate the specific application that we're trying to build from a product and systems design perspective. And after that, we're ready to start developing!
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Setup - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/splitting/","title":"Splitting a Dataset for Machine Learning","text":""},{"location":"courses/mlops/splitting/#intuition","title":"Intuition","text":"
To determine the efficacy of our models, we need to have an unbiased measuring approach. To do this, we split our dataset into training, validation, and testing data splits.
Use the training split to train the model.
Here the model will have access to both inputs and outputs to optimize its internal weights.
After each loop (epoch) of the training split, we will use the validation split to determine model performance.
Here the model will not use the outputs to optimize its weights but instead, we will use the performance to optimize training hyperparameters such as the learning rate, etc.
After training stops (epoch(s)), we will use the testing split to perform a one-time assessment of the model.
This is our best measure of how the model may behave on new, unseen data. Note that training stops when the performance improvement is not significant or any other stopping criteria that we may have specified.
Creating proper data splits
What are the criteria we should focus on to ensure proper data splits?
Show answer
the dataset (and each data split) should be representative of data we will encounter
equal distributions of output values across all splits
shuffle your data if it's organized in a way that prevents input variance
avoid random shuffles if your task can suffer from data leaks (ex. time-series)
We need to clean our data first before splitting, at least for the features that splitting depends on. So the process is more like: preprocessing (global, cleaning) \u2192 splitting \u2192 preprocessing (local, transformations).
For our multi-class task (each input has one label), we want to ensure that each data split has similar class distributions. We can achieve this by specifying how to stratify the split by adding the stratify keyword argument.
# Get counts for each class\ncounts = {}\ncounts[\"train_counts\"] = {tag: label_encoder.decode(y_train).count(tag) for tag in label_encoder.classes}\ncounts[\"val_counts\"] = {tag: label_encoder.decode(y_val).count(tag) for tag in label_encoder.classes}\ncounts[\"test_counts\"] = {tag: label_encoder.decode(y_test).count(tag) for tag in label_encoder.classes}\n
computer-vision mlops natural-language-processing other train 249 55 272 92 val 53 12 58 20 test 54 12 58 20
It's hard to compare these because our train and test proportions are different. Let's see what the distribution looks like once we balance it out. What do we need to multiply our test ratio by so that we have the same amount as our train ratio?
# Adjust counts across splits\nfor k in counts[\"val_counts\"].keys():\n counts[\"val_counts\"][k] = int(counts[\"val_counts\"][k] * \\\n (train_size/val_size))\nfor k in counts[\"test_counts\"].keys():\n counts[\"test_counts\"][k] = int(counts[\"test_counts\"][k] * \\\n (train_size/test_size))\n
computer-vision mlops natural-language-processing other train 249 55 272 92 val 247 56 270 93 test 252 56 270 93
We can see how much deviance there is in our naive data splits by computing the standard deviation of each split's class counts from the mean (ideal split).
If we had a multi-label classification task, then we would've applied iterative stratification via the skmultilearn library, which essentially splits each input into subsets (where each label is considered individually) and then it distributes the samples starting with fewest \"positive\" samples and working up to the inputs that have the most labels.
from skmultilearn.model_selection import IterativeStratification\ndef iterative_train_test_split(X, y, train_size):\n\"\"\"Custom iterative train test split which\n 'maintains balanced representation with respect\n to order-th label combinations.'\n \"\"\"\n stratifier = IterativeStratification(\n n_splits=2, order=1, sample_distribution_per_fold=[1.0-train_size, train_size, ])\n train_indices, test_indices = next(stratifier.split(X, y))\n X_train, y_train = X[train_indices], y[train_indices]\n X_test, y_test = X[test_indices], y[test_indices]\n return X_train, X_test, y_train, y_test\n
Iterative stratification essentially creates splits while \"trying to maintain balanced representation with respect to order-th label combinations\". We used to an order=1 for our iterative split which means we cared about providing representative distribution of each tag across the splits. But we can account for higher-order label relationships as well where we may care about the distribution of label combinations.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Splitting a Dataset for Machine Learning - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/styling/","title":"Styling and Formatting Code","text":""},{"location":"courses/mlops/styling/#intuition","title":"Intuition","text":"
Code is read more often than it is written. -- Guido Van Rossum (author of Python)
When we write a piece of code, it's almost never the last time we see it or the last time it's edited. So we need to explain what's going on (via documentation) and make it easy to read. One of the easiest ways to make code more readable is to follow consistent style and formatting conventions. There are many options when it comes to Python style conventions to adhere to, but most are based on PEP8 conventions. Different teams follow different conventions and that's perfectly alright. The most important aspects are:
consistency: everyone follows the same standards.
automation: formatting should be largely effortless after initial configuration.
We will be using a very popular blend of style and formatting conventions that makes some very opinionated decisions on our behalf (with configurable options).
Black: an in-place reformatter that (mostly) adheres to PEP8.
isort: sorts and formats import statements inside Python scripts.
flake8: a code linter with stylistic conventions that adhere to PEP8.
Before we can properly use these tools, we'll have to configure them because they may have some discrepancies amongst them since they follow slightly different conventions that extend from PEP8.
To configure Black, we could just pass in options using the CLI method, but it's much cleaner to do this through our pyproject.toml file.
# Black formatting\n[tool.black]\nline-length = 150\ninclude = '\\.pyi?$'\nexclude = '''\n/(\n .eggs # exclude a few common directories in the\n | .git # root of the project\n | .hg\n | .mypy_cache\n | .tox\n | venv\n | _build\n | buck-out\n | build\n | dist\n )/\n'''\n
Here we're telling Black what our maximum line length should and to include and exclude certain file extensions.
The pyproject.toml was created to establish a more human-readable configuration file that is meant to replace a setup.py or setup.cfg file and is increasingly adopted by many open-source libraries.
Lastly, we'll set up flake8 by also adding it's configuration details to out pyproject.toml file.
[tool.flake8]\nexclude = \"venv\"\nignore = [\"E501\", \"W503\", \"E226\"]\n# E501: Line too long\n# W503: Line break occurred before binary operator\n# E226: Missing white space around arithmetic operator\n
Here we're including an ignore option to ignore certain flake8 rules so everything works with our Black and isort configurations. And besides defining configuration options here, which are applied globally, we can also choose to specifically ignore certain conventions on a line-by-line basis. Here is an example of how we utilize this:
# madewithml/config.py\nimport pretty_errors # NOQA: F401 (imported but unused)\n
By placing the # NOQA: <error-code> on a line, we're telling flake8 to do NO Quality Assurance for that particular error on this line.
Remembering these three lines to style our code is a bit cumbersome so it's a good idea to create a Makefile. This file can be used to define a set of commands that can be executed with a single command. Here's what our Makefile looks like:
Notice that the clean command depends on the style command (clean: style), which means that style will be executed first before clean is executed.
.PHONY
As the name suggests, a Makefile is typically used to make a file, where if a file with the name already exists, then the commands below won't be executed. But we're using it in a way where we want to execute some commands with a single alias. Therefore, the .PHONY: $FILENAME lines indicate that even if there is a file called $FILENAME, go ahead and execute the commands below anyway.
In the next lesson on pre-commit we'll learn how to automatically execute this formatting whenever we make changes to our code.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Styling - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/systems-design/","title":"Machine Learning Systems Design","text":""},{"location":"courses/mlops/systems-design/#overview","title":"Overview","text":"
In the previous lesson, we covered the product design process for our ML application. In this lesson, we'll cover the systems design process where we'll learn how to design the ML system that will address our product objectives.
The template below is designed to guide machine learning product development. It involves both the product and systems design aspects of our application:
Product design (What & Why) \u2192 Systems design (How)
\ud83d\udc49 \u00a0 Download a PDF of the ML canvas to use for your own products \u2192 ml-canvas.pdf (right click the link and hit \"Save Link As...\")
Describe the training and production (batches/streams) sources of data.
id created_on title description tag 0 6 2020-02-20 06:43:18 Comparison between YOLO and RCNN on real world ... Bringing theory to experiment is cool. We can ... computer-vision 1 89 2020-03-20 18:17:31 Rethinking Batch Normalization in Transformers We found that NLP batch statistics exhibit large ... natural-language-processing 2 1274 2020-06-10 05:21:00 Getting Machine Learning to Production Machine learning is hard and there are a lot, a lot of ... mlops 4 19 2020-03-03 13:54:31 Diffusion to Vector Reference implementation of Diffusion2Vec ... other
Our task
training:
access to training data and testing (holdout) data.
was there sampling of any kind applied to create this dataset?
are we introducing any data leaks?
production:
access to batches or real-time streams of ML content from various sources
how can we trust that this stream only has data that is consistent with what we have historically seen?
Assumption Reality Reason All of our incoming data is only machine learning related (no spam). We would need a filter to remove spam content that's not ML related. To simplify our ML task, we will assume all the data is ML content.
Describe the labeling process (ingestions, QA, etc.) and how we decided on the features and labels.
Our task
Labels: categories of machine learning (for simplification, we've restricted the label space to the following tags: natural-language-processing, computer-vision, mlops and other).
Features: text features (title and description) that describe the content.
Assumption Reality Reason Content can only belong to one category (multiclass). Content can belong to more than one category (multilabel). For simplicity and many libraries don't support or complicate multilabel scenarios.
One of the hardest challenges with ML systems is tying our core objectives, many of which may be qualitative, with quantitative metrics that our model can optimize towards.
Our task
For our task, we want to have both high precision and recall, so we'll optimize for f1 score (weighted combination of precision and recall). We'll determine these metrics for the overall dataset, as well as specific classes or slices of data.
True positives (TP): we correctly predicted class X.
False positives (FP): we incorrectly predicted class X but it was another class.
True negatives (TN): we correctly predicted that it's wasn't the class X.
False negatives (FN): we incorrectly predicted that it wasn't the class X but it was.
It entirely depends on the specific task. For example, in an email spam detector, precision is very important because it's better than we some spam then completely miss an important email. Overtime, we need to iterate on our solution so all evaluation metrics improve but it's important to know which one's we can't comprise on from the get-go.
Online evaluation ensures that our model continues to perform well in production and can be performed using labels or, in the event we don't readily have labels, proxy signals.
Our task
manually label a subset of incoming data to evaluate periodically.
asking the initial set of users viewing a newly categorized content if it's correctly classified.
allow users to report misclassified content by our model.
It's important that we measure real-time performance before committing to replace our existing version of the system.
Internal canary rollout, monitoring for proxy/actual performance, etc.
Rollout to the larger internal team for more feedback.
A/B rollout to a subset of the population to better understand UX, utility, etc.
Not all releases have to be high stakes and external facing. We can always include internal releases, gather feedback and iterate until we\u2019re ready to increase the scope.
While the specific methodology we employ can differ based on the problem, there are core principles we always want to follow:
End-to-end utility: the end result from every iteration should deliver minimum end-to-end utility so that we can benchmark iterations against each other and plug-and-play with the system.
Manual before ML: try to see how well a simple rule-based system performs before moving onto more complex ones.
Augment vs. automate: allow the system to supplement the decision making process as opposed to making the actual decision.
Internal vs. external: not all early releases have to be end-user facing. We can use early versions for internal validation, feedback, data collection, etc.
Thorough: every approach needs to be well tested (code, data + models) and evaluated, so we can objectively benchmark different approaches.
Our task
creating a gold-standard labeled dataset that is representative of the problem space.
rule-based text matching approaches to categorize content.
predict labels (probabilistic) from content title and description.
Assumption Reality Reason Solution needs to involve ML due to unstructured data and ineffectiveness of rule-based systems for this task. An iterative approach where we start with simple rule-based solutions and slowly add complexity. This course is about responsibly delivering value with ML, so we'll jump to it right away.
Utility in starting simple
Some of the earlier, simpler, approaches may not deliver on a certain performance objective. What are some advantages of still starting simple?
Show answer
get internal feedback on end-to-end utility.
perform A/B testing to understand UI/UX design.
deployed locally to start generating more data required for more complex approaches.
We can use our models to make batch predictions on a finite set of inputs which are then written to a database for low latency inference. When a user or downstream service makes an inference request, cached results from the database are returned. In this scenario, our trained model can directly be loaded and used for inference in the code. It doesn't have to be served as a separate service.
\u2705\u00a0 generate and cache predictions for very fast inference for users.
\u2705\u00a0 the model doesn't need to be spun up as it's own service since it's never used in real-time.
\u274c\u00a0 predictions can become stale if user develops new interests that aren\u2019t captured by the old data that the current predictions are based on.
Batch serving tasks
What are some tasks where batch serving is ideal?
Show answer
Recommend content that existing users will like based on their viewing history. However, new users may just receive some generic recommendations based on their explicit interests until we process their history the next day. And even if we're not doing batch serving, it might still be useful to cache very popular sets of input features (ex. combination of explicit interests leads to certain recommended content) so that we can serve those predictions faster.
We can also serve real-time predictions where input features are fed to the model to retrieve predictions. In this scenario, our model will need to be served as a separate service (ex. api endpoint) that can handle incoming requests.
\u2705\u00a0 can yield more up-to-date predictions which may yield a more meaningful user experience, etc.
\u274c\u00a0 requires managed microservices to handle request traffic.
\u274c\u00a0 requires real-time monitoring since input space in unbounded, which could yield erroneous predictions.
Online inference tasks
In our example task for batch inference above, how can online inference significantly improve content recommendations?
Show answer
With batch processing, we generate content recommendations for users offline using their history. These recommendations won't change until we process the batch the next day using the updated user features. But what is the user's taste significantly changes during the day (ex. user is searching for horror movies to watch). With real-time serving, we can use these recent features to recommend highly relevant content based on the immediate searches.
Our task
For our task, we'll be serving our model as a separate service to handle real-time requests. We want to be able to perform online inference so that we can quickly categorize ML content as they become available. However, we will also demonstrate how to do batch inference for the sake of completeness.
How do we receive feedback on our system and incorporate it into the next iteration? This can involve both human-in-the-loop feedback as well as automatic feedback via monitoring, etc.
Our task
enforce human-in-loop checks when there is low confidence in classifications.
allow users to report issues related to misclassification.
Always return to the value proposition
While it's important to iterate and optimize on our models, it's even more important to ensure that our ML systems are actually making an impact. We need to constantly engage with our users to iterate on why our ML system exists and how it can be made better.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Systems - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/testing/","title":"Testing Machine Learning Systems: Code, Data and Models","text":""},{"location":"courses/mlops/testing/#intuition","title":"Intuition","text":"
In this lesson, we'll learn how to test code, data and machine learning models to construct a machine learning system that we can reliably iterate on. Tests are a way for us to ensure that something works as intended. We're incentivized to implement tests and discover sources of error as early in the development cycle as possible so that we can decrease downstream costs and wasted time. Once we've designed our tests, we can automatically execute them every time we change or add to our codebase.
Tip
We highly recommend that you explore this lesson after completing the previous lessons since the topics (and code) are iteratively developed. We did, however, create the testing-ml repository for a quick overview with an interactive notebook.
"},{"location":"courses/mlops/testing/#types-of-tests","title":"Types of tests","text":"
There are four majors types of tests which are utilized at different points in the development cycle:
Unit tests: tests on individual components that each have a single responsibility (ex. function that filters a list).
Integration tests: tests on the combined functionality of individual components (ex. data processing).
System tests: tests on the design of a system for expected outputs given inputs (ex. training, inference, etc.).
Acceptance tests: tests to verify that requirements have been met, usually referred to as User Acceptance Testing (UAT).
Regression tests: tests based on errors we've seen before to ensure new changes don't reintroduce them.
While ML systems are probabilistic in nature, they are composed of many deterministic components that can be tested in a similar manner as traditional software systems. The distinction between testing ML systems begins when we move from testing code to testing the data and models.
There are many other types of functional and non-functional tests as well, such as smoke tests (quick health checks), performance tests (load, stress), security tests, etc. but we can generalize all of these under the system tests above.
"},{"location":"courses/mlops/testing/#how-should-we-test","title":"How should we test?","text":"
The framework to use when composing tests is the Arrange Act Assert methodology.
Arrange: set up the different inputs to test on.
Act: apply the inputs on the component we want to test.
Assert: confirm that we received the expected output.
Cleaning is an unofficial fourth step to this methodology because it's important to not leave remnants of a previous test which may affect subsequent tests. We can use packages such as pytest-randomly to test against state dependency by executing tests randomly.
In Python, there are many tools, such as unittest, pytest, etc. that allow us to easily implement our tests while adhering to the Arrange Act Assert framework. These tools come with powerful built-in functionality such as parametrization, filters, and more, to test many conditions at scale.
"},{"location":"courses/mlops/testing/#what-should-we-test","title":"What should we test?","text":"
When arranging our inputs and asserting our expected outputs, what are some aspects of our inputs and outputs that we should be testing for?
inputs: data types, format, length, edge cases (min/max, small/large, etc.)
outputs: data types, formats, exceptions, intermediary and final outputs
\ud83d\udc49 \u00a0We'll cover specific details pertaining to what to test for regarding our data and models below.
Regardless of the framework we use, it's important to strongly tie testing into the development process.
atomic: when creating functions and classes, we need to ensure that they have a single responsibility so that we can easily test them. If not, we'll need to split them into more granular components.
compose: when we create new components, we want to compose tests to validate their functionality. It's a great way to ensure reliability and catch errors early on.
reuse: we should maintain central repositories where core functionality is tested at the source and reused across many projects. This significantly reduces testing efforts for each new project's code base.
regression: we want to account for new errors we come across with a regression test so we can ensure we don't reintroduce the same errors in the future.
coverage: we want to ensure 100% coverage for our codebase. This doesn't mean writing a test for every single line of code but rather accounting for every single line.
automate: in the event we forget to run our tests before committing to a repository, we want to auto run tests when we make changes to our codebase. We'll learn how to do this locally using pre-commit hooks and remotely via GitHub actions in subsequent lessons.
Note that we aren't testing evaluate.py and serve.py because it involves complicated testing that's based on the data and models. We'll be testing these components as part of our integration tests when we test our system end-to-end.
We'll start by testing our code and we'll use pytest as our testing framework for it's powerful builtin features such as parametrization, fixtures, markers and more.
Pytest expects tests to be organized under a tests directory by default. However, we can also add to our existing pyproject.toml file to configure any other test directories as well. Once in the directory, pytest looks for python scripts starting with tests_*.py but we can configure it to read any other file patterns as well.
Let's see what a sample test and it's results look like. Assume we have a simple function that decodes a list of indices into their respective classes using a dictionary mapping.
# madewithml/predict.py\ndef decode(indices: Iterable[Any], index_to_class: Dict) -> List:\n return [index_to_class[index] for index in indices]\n
To test this function, we can use assert statements to map inputs with expected outputs. The statement following the word assert must return True.
We can also have assertions about exceptions like we do in lines 6-8 where all the operations under the with statement are expected to raise the specified exception.
We can execute our tests above using several different levels of granularity:
python3 -m pytest # all tests\npython3 -m pytest tests/code # tests under a directory\npython3 -m pytest tests/code/test_predict.py # tests for a single file\npython3 -m pytest tests/code/test_predict.py::test_decode # tests for a single function\n
Running our specific test above would produce the following output:
It's important to test for the variety of inputs and expected outputs that we outlined above and to never assume that a test is trivial. In our example above, it's important that we test for both \"apple\" and \"Apple\" in the event that our function didn't account for casing!
So far, in our tests, we've had to create individual assert statements to validate different combinations of inputs and expected outputs. However, there's a bit of redundancy here because the inputs always feed into our functions as arguments and the outputs are compared with our expected outputs. To remove this redundancy, pytest has the @pytest.mark.parametrize decorator which allows us to represent our inputs and outputs as parameters.
Parametrization allows us to reduce redundancy inside test functions but what about reducing redundancy across different test functions? For example, suppose that different test functions all have a common component (ex. preprocessor). Here, we can use pytest's builtin fixture, which is a function that is executed before the test function. Let's rewrite our test_fit_transform function from above using a fixture:
All of our test scripts know to look inside a conftest.py script in the same directory for any fixtures. Note that the name of the fixture and the input argument to our function have to be the same.
Fixture scopes
Fixtures can have different scopes depending on how we want to use them. For example our df fixture has the module scope because we don't want to keep recreating it after every test but, instead, we want to create it just once for all the tests in our module (tests/test_data.py).
function: fixture is destroyed after every test. [default]
class: fixture is destroyed after the last test in the class.
module: fixture is destroyed after the last test in the module (script).
package: fixture is destroyed after the last test in the package.
session: fixture is destroyed after the last test of the session.
We've been able to execute our tests at various levels of granularity (all tests, script, function, etc.) but we can create custom granularity by using markers. We've already used one type of marker (parametrize) but there are several other builtin markers as well. For example, the skipif marker allows us to skip execution of a test if a condition is met. For example, supposed we only wanted to test training our model if a GPU is available:
@pytest.mark.skipif(\n not torch.cuda.is_available(),\n reason=\"Full training tests require a GPU.\"\n)\ndef test_training():\n pass\n
We can also create our own custom markers with the exception of a few reserved marker names.
We can execute them by using the -m flag which requires a (case-sensitive) marker expression like below:
pytest -m \"training\" # runs all tests marked with `training`\npytest -m \"not training\" # runs all tests besides those marked with `training`\n
Tip
The proper way to use markers is to explicitly list the ones we've created in our pyproject.toml file. Here we can specify that all markers must be defined in this file with the --strict-markers flag and then declare our markers (with some info about them) in our markers list:
Once we do this, we can view all of our existing list of markers by executing pytest --markers and we'll receive an error when we're trying to use a new marker that's not defined here."},{"location":"courses/mlops/testing/#coverage","title":"Coverage","text":"
As we're developing tests for our application's components, it's important to know how well we're covering our code base and to know if we've missed anything. We can use the Coverage library to track and visualize how much of our codebase our tests account for. With pytest, it's even easier to use this package thanks to the pytest-cov plugin.
python3 -m pytest tests/code --cov madewithml --cov-report html --disable-warnings\n
Here we're asking to run all tests under tests/code and to check for coverage for all the code in our madewithml directory. When we run this, we'll see the tests from our tests directory executing while the coverage plugin is keeping tracking of which lines in our application are being executed. Once our tests are done, we can view the generated report either through the terminal:
but a more interactive way is to view it through the htmlcov/index.html on a browser. Here we can click on individual files to see which parts were not covered by any tests.
Warning
Though we have 100% coverage, this does not mean that our application is perfect. Coverage only indicates that a piece of code executed in a test, not necessarily that every part of it was tested, let alone thoroughly tested. Therefore, coverage should never be used as a representation of correctness. However, it is very useful to maintain coverage at 100% so we can know when new functionality has yet to be tested. In our CI/CD lesson, we'll see how to use GitHub actions to make 100% coverage a requirement when pushing to specific branches.
Sometimes it doesn't make sense to write tests to cover every single line in our application yet we still want to account for these lines so we can maintain 100% coverage. We have two levels of purview when applying exclusions:
Excusing lines by adding this comment # pragma: no cover, <MESSAGE>
if results_fp: # pragma: no cover, saving results\n utils.save_dict(d, results_fp)\n
Excluding files by specifying them in our pyproject.toml configuration:
The main point is that we were able to add justification to these exclusions through comments so our team can follow our reasoning.
Now that we have a foundation for testing traditional software, let's dive into testing our data and models in the context of machine learning systems.
So far, we've used unit and integration tests to test the functions that interact with our data but we haven't tested the validity of the data itself. We're going to use the great expectations library to test what our data is expected to look like. It's a library that allows us to create expectations as to what our data should look like in a standardized way. It also provides modules to seamlessly connect with backend data sources such as local file systems, S3, databases, etc. Let's explore the library by implementing the expectations we'll need for our application.
\ud83d\udc49 \u00a0 Follow along interactive notebook in the testing-ml repository as we implement the concepts below.
First we'll load the data we'd like to apply our expectations on. We can load our data from a variety of sources (filesystem, database, cloud etc.) which we can then wrap around a Dataset module (Pandas / Spark DataFrame, SQLAlchemy). Since multiple data tests may want access to this data, we'll create a fixture for it.
# tests/data/conftest.py\nimport great_expectations as ge\nimport pandas as pd\nimport pytest\n\n@pytest.fixture(scope=\"module\")\ndef df(request):\n dataset_loc = request.config.getoption(\"--dataset-loc\")\n df = ge.dataset.PandasDataset(pd.read_csv(dataset_loc))\n return df\n
When it comes to creating expectations as to what our data should look like, we want to think about our entire dataset and all the features (columns) within it.
Each of these expectations will create an output with details about success or failure, expected and observed values, expectations raised, etc. For example, the expectation df.expect_column_values_to_be_of_type(column=\"title\", type_=\"str\") would produce the following if successful:
and if we have a failed expectation (ex. df.expect_column_values_to_be_of_type(column=\"title\", type_=\"int\")), we'd receive this output(notice the counts and examples for what caused the failure):
{\n\"success\": false,\n\"exception_info\": {\n\"raised_exception\": false,\n\"exception_traceback\": null,\n\"exception_message\": null\n},\n\"expectation_config\": {\n\"meta\": {},\n\"kwargs\": {\n\"column\": \"title\",\n\"type_\": \"int\",\n\"result_format\": \"BASIC\"\n},\n\"expectation_type\": \"_expect_column_values_to_be_of_type__map\"\n},\n\"result\": {\n\"element_count\": 955,\n\"missing_count\": 0,\n\"missing_percent\": 0.0,\n\"unexpected_count\": 955,\n\"unexpected_percent\": 100.0,\n\"unexpected_percent_nonmissing\": 100.0,\n\"partial_unexpected_list\": [\n\"How to Deal with Files in Google Colab: What You Need to Know\",\n\"Machine Learning Methods Explained (+ Examples)\",\n\"OpenMMLab Computer Vision\",\n\"...\",\n]\n},\n\"meta\": {}\n}\n
There are just a few of the different expectations that we can create. Be sure to explore all the expectations, including custom expectations. Here are some other popular expectations that don't pertain to our specific dataset but are widely applicable:
feature value relationships with other feature values \u2192 expect_column_pair_values_a_to_be_greater_than_b
We've added a --dataset-loc flag to pytest by specifying in our tests/data/conftest.py script. This allows us to pass in the dataset location as an argument to our tests.
We're keeping things simple by using our expectations with pytest but Great expectations also has a lot more functionality around connecting to data sources, Checkpoints to execute suites across various parts of the pipeline, data docs to generate reports, etc.
While we're validating our datasets inside our machine learning applications, in most production scenarios, the data validation happens much further upstream. Our dataset may not be used just for our specific application and may actually be feeding into many other downstream application (ML and otherwise). Therefore, it's a great idea to execute these data validation tests as up stream as possible so that downstream applications can reliably use the data.
Learn more about different data systems in our data engineering lesson if you're not familiar with them.
We want to write tests iteratively while we're developing our training pipelines so we can catch errors quickly. This is especially important because, unlike traditional software, ML systems can run to completion without throwing any exceptions / errors but can produce incorrect systems. We also want to catch errors quickly to save on time and compute.
Behavioral testing is the process of testing input data and expected outputs while treating the model as a black box (model agnostic evaluation). They don't necessarily have to be adversarial in nature but more along the types of perturbations we may expect to see in the real world once our model is deployed. A landmark paper on this topic is Beyond Accuracy: Behavioral Testing of NLP Models with CheckList which breaks down behavioral testing into three types of tests:
invariance: Changes should not affect outputs.
# INVariance via verb injection (changes should not affect outputs)\nget_label(text=\"Transformers applied to NLP have revolutionized machine learning.\", predictor=predictor)\nget_label(text=\"Transformers applied to NLP have disrupted machine learning.\", predictor=predictor)\n
# DIRectional expectations (changes with known outputs)\nget_label(text=\"ML applied to text classification.\", predictor=predictor)\nget_label(text=\"ML applied to image classification.\", predictor=predictor)\nget_label(text=\"CNNs for text classification.\", predictor=predictor)\n
minimum functionality: Simple combination of inputs and expected outputs.
# Minimum Functionality Tests (simple input/output pairs)\nget_label(text=\"Natural language processing is the next big wave in machine learning.\", predictor=predictor)\nget_label(text=\"MLOps is the next big wave in machine learning.\", predictor=predictor)\nget_label(text=\"This is about graph neural networks.\", predictor=predictor)\n
And we can convert these tests into proper parameterized tests by first defining from fixtures in our tests/model/conftest.py and our tests/model/utils.py scripts:
# tests/model/conftest.py\nimport pytest\nfrom ray.train.torch.torch_predictor import TorchPredictor\nfrom madewithml import predict\n\ndef pytest_addoption(parser):\n parser.addoption(\"--run-id\", action=\"store\", default=None, help=\"Run ID of model to use.\")\n\n\n@pytest.fixture(scope=\"module\")\ndef run_id(request):\n return request.config.getoption(\"--run-id\")\n\n\n@pytest.fixture(scope=\"module\")\ndef predictor(run_id):\n best_checkpoint = predict.get_best_checkpoint(run_id=run_id)\n predictor = TorchPredictor.from_checkpoint(best_checkpoint)\n return predictor\n
And now, we can use these components to create our behavioral tests:
# tests/model/test_behavioral.py\n@pytest.mark.parametrize(\n \"input_a, input_b, label\",\n [\n (\n \"Transformers applied to NLP have revolutionized machine learning.\",\n \"Transformers applied to NLP have disrupted machine learning.\",\n \"natural-language-processing\",\n ),\n ],\n)\ndef test_invariance(input_a, input_b, label, predictor):\n\"\"\"INVariance via verb injection (changes should not affect outputs).\"\"\"\n label_a = utils.get_label(text=input_a, predictor=predictor)\n label_b = utils.get_label(text=input_b, predictor=predictor)\n assert label_a == label_b == label\n
# tests/model/test_behavioral.py\n@pytest.mark.parametrize(\n \"input, label\",\n [\n (\n \"ML applied to text classification.\",\n \"natural-language-processing\",\n ),\n (\n \"ML applied to image classification.\",\n \"computer-vision\",\n ),\n (\n \"CNNs for text classification.\",\n \"natural-language-processing\",\n ),\n ],\n)\ndef test_directional(input, label, predictor):\n\"\"\"DIRectional expectations (changes with known outputs).\"\"\"\n prediction = utils.get_label(text=input, predictor=predictor)\n assert label == prediction\n
# tests/model/test_behavioral.py\n@pytest.mark.parametrize(\n \"input, label\",\n [\n (\n \"Natural language processing is the next big wave in machine learning.\",\n \"natural-language-processing\",\n ),\n (\n \"MLOps is the next big wave in machine learning.\",\n \"mlops\",\n ),\n (\n \"This is about graph neural networks.\",\n \"other\",\n ),\n ],\n)\ndef test_mft(input, label, predictor):\n\"\"\"Minimum Functionality Tests (simple input/output pairs).\"\"\"\n prediction = utils.get_label(text=input, predictor=predictor)\n assert label == prediction\n
"},{"location":"courses/mlops/testing/#testing-vs-monitoring","title":"Testing vs. monitoring","text":"
We'll conclude by talking about the similarities and distinctions between testing and monitoring. They're both integral parts of the ML development pipeline and depend on each other for iteration. Testing is assuring that our system (code, data and models) passes the expectations that we've established offline. Whereas, monitoring involves that these expectations continue to pass online on live production data while also ensuring that their data distributions are comparable to the reference window (typically subset of training data) through \\(t_n\\). When these conditions no longer hold true, we need to inspect more closely (retraining may not always fix our root problem).
With monitoring, there are quite a few distinct concerns that we didn't have to consider during testing since it involves (live) data we have yet to see.
features and prediction distributions (drift), typing, schema mismatches, etc.
determining model performance (rolling and window metrics on overall and slices of data) using indirect signals (since labels may not be readily available).
in situations with large data, we need to know which data points to label and upsample for training.
identifying anomalies and outliers.
We'll cover all of these concepts in much more depth (and code) in our monitoring lesson.
Now that we have our data prepared, we can start training our models to optimize on our objective. Ideally, we would start with the simplest possible baseline and slowly add complexity to our models:
Start with a random (chance) model.
Since we have four classes, we may expect a random model to be correct around 25% of the time but recall that not all of our classes have equal counts.
Develop a rule-based approach using if-else statements, regular expressions, etc.
We could build a list of common words for each class and if a word in the input matches a word in the list, we can predict that class.
Slowly add complexity by addressing limitations and motivating representations and model architectures.
We could start with a simple term frequency (TF-IDF) mode and then move onto embeddings with CNNs, RNNs, Transformers, etc.
Weigh tradeoffs (performance, latency, size, etc.) between performant baselines.
Revisit and iterate on baselines as your dataset grows and new model architectures are developed.
We're going to skip straight to step 3 of developing a complex model because this task involves unstructured data and rule-based systems are not well suited for this. And with the increase adoption of large language models (LLMs) as a proven model architecture for NLP tasks, we'll fine-tune a pretrained LLM on our dataset.
Iterate on the data
Instead of using a fixed dataset and iterating on the models, we could keep the model constant and iterate on the dataset. This is useful to improve the quality of our datasets.
remove or fix data samples (false positives & negatives)
With the rapid increase in data (unstructured) and model sizes (ex. LLMs), it's becoming increasingly difficult to train models on a single machine. We need to be able to distribute our training across multiple machines in order to train our models in a reasonable amount of time. And we want to be able to do this without having to:
set up a cluster by individually (and painstakingly) provisioning compute resources (CPU, GPU, etc.)
writing complex code to distribute our training across multiple machines
worry about communication and resource utilization between our different distributed compute resources
worry about fault tolerance and recovery from our large training workloads
To address all of these concerns, we'll be using Ray Train here in order to create a training workflow that can scale across multiple machines. While there are many options to choose from for distributed training, such as Pytorch Distributed Data Parallel (DDP), Horovod, etc., none of them allow us to scale across different machines with ease and do so with minimal changes to our single-machine training code as Ray does.
Primer on distributed training
With distributed training, there will be a head node that's responsible for orchestrating the training process. While the worker nodes that will be responsible for training the model and communicating results back to the head node. From a user's perspective, Ray abstracts away all of this complexity and we can simply define our training functionality with minimal changes to our code (as if we were training on a single machine).
In this lesson, we're going to be fine-tuning a pretrained large language model (LLM) using our labeled dataset. The specific class of LLMs we'll be using is called BERT. Bert models are encoder-only models and are the gold-standard for supervised NLP tasks. However, you may be wondering how do all the (much larger) LLM, created for generative applications, fare (GPT 4, Falcon 40B, Llama 2, etc.)?
We chose the smaller BERT model for our course because it's easier to train and fine-tune. However, the workflow for fine-tuning the larger LLMs are quite similar as well. They do require much more compute but Ray abstracts away the scaling complexities involved with that.
Note
All the code for this section can be found in our separate benchmarks.ipynb notebook.
We'll first load the our training and inference data into dataframes.
import pandas as pd\n
# Load training data\nDATASET_LOC = \"https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/dataset.csv\"\ntrain_df = pd.read_csv(DATASET_LOC)\ntrain_df.head()\n
id created_on title description tag 0 6 2020-02-20 06:43:18 Comparison between YOLO and RCNN on real world... Bringing theory to experiment is cool. We can ... computer-vision 1 7 2020-02-20 06:47:21 Show, Infer & Tell: Contextual Inference for C... The beauty of the work lies in the way it arch... computer-vision 2 9 2020-02-24 16:24:45 Awesome Graph Classification A collection of important graph embedding, cla... other 3 15 2020-02-28 23:55:26 Awesome Monte Carlo Tree Search A curated list of Monte Carlo tree search pape... other 4 25 2020-03-07 23:04:31 AttentionWalk A PyTorch Implementation of \"Watch Your Step: ... other
We'll define a few utility functions to make the OpenAI request and to store our predictions. While we could perform batch prediction by loading samples until the context length is reached, we'll just perform one at a time since it's not too many data points and we can have fully deterministic behavior (if you insert new data, etc.). We'll also added some reliability in case we overload the endpoints with too many request at once.
import json\nfrom collections import Counter\nimport matplotlib.pyplot as plt\nimport seaborn as sns; sns.set_theme()\nfrom sklearn.metrics import precision_recall_fscore_support\nimport time\nfrom tqdm import tqdm\n
We'll first define what a sample call to the OpenAI endpoint looks like. We'll pass in: - system_content that has information about how the LLM should behave. - assistant_content for any additional context it should have for answering our questions. - user_content that has our message or query to the LLM. - model should specify which specific model we want to send our request to.
We can pass all of this information in through the openai.ChatCompletion.create function to receive our response.
# Query OpenAI endpoint\nsystem_content = \"you only answer in rhymes\" # system content (behavior)\nassistant_content = \"\" # assistant content (context)\nuser_content = \"how are you\" # user content (message)\nresponse = openai.ChatCompletion.create(\n model=\"gpt-3.5-turbo-0613\",\n messages=[\n {\"role\": \"system\", \"content\": system_content},\n {\"role\": \"assistant\", \"content\": assistant_content},\n {\"role\": \"user\", \"content\": user_content},\n ],\n)\nprint (response.to_dict()[\"choices\"][0].to_dict()[\"message\"][\"content\"])\n
\nI'm doing just fine, so glad you ask,\nRhyming away, up to the task.\nHow about you, my dear friend?\nTell me how your day did ascend.\n
Now, let's create a function that can predict tags for a given sample.
# Get tag\nmodel = \"gpt-3.5-turbo-0613\"\nsystem_context = f\"\"\"\n You are a NLP prediction service that predicts the label given an input's title and description.\n You must choose between one of the following labels for each input: {tags}.\n Only respond with the label name and nothing else.\n \"\"\"\nassistant_content = \"\"\nuser_context = \"Transfer learning with transformers: Using transformers for transfer learning on text classification tasks.\"\ntag = get_tag(model=model, system_content=system_context, assistant_content=assistant_content, user_content=user_context)\nprint (tag)\n
\nnatural-language-processing\n
Next, let's create a function that can predict tags for a list of inputs.
# List of dicts w/ {title, description} (just the first 3 samples for now)\nsamples = test_df[[\"title\", \"description\"]].to_dict(orient=\"records\")[:3]\nsamples\n
\n[{'title': 'Diffusion to Vector',\n 'description': 'Reference implementation of Diffusion2Vec (Complenet 2018) built on Gensim and NetworkX. '},\n {'title': 'Graph Wavelet Neural Network',\n 'description': 'A PyTorch implementation of \"Graph Wavelet Neural Network\" (ICLR 2019) '},\n {'title': 'Capsule Graph Neural Network',\n 'description': 'A PyTorch implementation of \"Capsule Graph Neural Network\" (ICLR 2019).'}]\n
def get_predictions(inputs, model, system_content, assistant_content=\"\"):\n y_pred = []\n for item in tqdm(inputs):\n # Convert item dict to string\n user_content = str(item)\n\n # Get prediction\n predicted_tag = get_tag(\n model=model, system_content=system_content,\n assistant_content=assistant_content, user_content=user_content)\n\n # If error, try again after pause (repeatedly until success)\n while predicted_tag is None:\n time.sleep(30) # could also do exponential backoff\n predicted_tag = get_tag(\n model=model, system_content=system_content,\n assistant_content=assistant_content, user_content=user_content)\n\n # Add to list of predictions\n y_pred.append(predicted_tag)\n\n return y_pred\n
# Get predictions for a list of inputs\nget_predictions(inputs=samples, model=model, system_content=system_context)\n
Next we'll define a function that can clean our predictions in the event that it's not the proper format or has hallucinated a tag outside of our expected tags.
def clean_predictions(y_pred, tags, default=\"other\"):\n for i, item in enumerate(y_pred):\n if item not in tags: # hallucinations\n y_pred[i] = default\n if item.startswith(\"'\") and item.endswith(\"'\"): # GPT 4 likes to places quotes\n y_pred[i] = item[1:-1]\n return y_pred\n
Tip
Open AI has now released function calling and custom instructions which is worth exploring to avoid this manual cleaning.
Next, we'll define a function that will plot our ground truth labels and predictions.
We'll start with zero-shot learning which involves providing the model with the system_content that tells it how to behave but no examples of the behavior (no assistant_content).
system_content = f\"\"\"\n You are a NLP prediction service that predicts the label given an input's title and description.\n You must choose between one of the following labels for each input: {tags}.\n Only respond with the label name and nothing else.\n \"\"\"\n
Now, we'll be adding a assistant_context with a few samples from our training data for each class. The intuition here is that we're giving the model a few examples (few-shot learning) of what each class looks like so that it can learn to generalize better.
# Create additional context with few samples from each class\nnum_samples = 2\nadditional_context = []\ncols_to_keep = [\"title\", \"description\", \"tag\"]\nfor tag in tags:\n samples = train_df[cols_to_keep][train_df.tag == tag][:num_samples].to_dict(orient=\"records\")\n additional_context.extend(samples)\nadditional_context\n
\n[{'title': 'Comparison between YOLO and RCNN on real world videos',\n 'description': 'Bringing theory to experiment is cool. We can easily train models in colab and find the results in minutes.',\n 'tag': 'computer-vision'},\n {'title': 'Show, Infer & Tell: Contextual Inference for Creative Captioning',\n 'description': 'The beauty of the work lies in the way it architects the fundamental idea that humans look at the overall image and then individual pieces of it.\\r\\n',\n 'tag': 'computer-vision'},\n {'title': 'Awesome Graph Classification',\n 'description': 'A collection of important graph embedding, classification and representation learning papers with implementations.',\n 'tag': 'other'},\n {'title': 'Awesome Monte Carlo Tree Search',\n 'description': 'A curated list of Monte Carlo tree search papers with implementations. ',\n 'tag': 'other'},\n {'title': 'Rethinking Batch Normalization in Transformers',\n 'description': 'We found that NLP batch statistics exhibit large variance throughout training, which leads to poor BN performance.',\n 'tag': 'natural-language-processing'},\n {'title': 'ELECTRA: Pre-training Text Encoders as Discriminators',\n 'description': 'PyTorch implementation of the electra model from the paper: ELECTRA - Pre-training Text Encoders as Discriminators Rather Than Generators',\n 'tag': 'natural-language-processing'},\n {'title': 'Pytest Board',\n 'description': 'Continuous pytest runner with awesome visualization.',\n 'tag': 'mlops'},\n {'title': 'Debugging Neural Networks with PyTorch and W&B',\n 'description': 'A closer look at debugging common issues when training neural networks.',\n 'tag': 'mlops'}]\n
# Add assistant context\nassistant_content = f\"\"\"Here are some examples with the correct labels: {additional_context}\"\"\"\nprint (assistant_content)\n
\nHere are some examples with the correct labels: [{'title': 'Comparison between YOLO and RCNN on real world videos', ... 'description': 'A closer look at debugging common issues when training neural networks.', 'tag': 'mlops'}]\n
Tip
We could increase the number of samples by increasing the context length. We could also retrieve better few-shot samples by extracting examples from the training data that are similar to the current sample (ex. similar unique vocabulary).
As we can see, few shot learning performs better than it's respective zero shot counter part. GPT 4 has had considerable improvements in reducing hallucinations but for our supervised task this comes at an expense of high precision but lower recall and f1 scores. When GPT 4 is not confident, it would rather predict other.
So far, we've only been using closed-source models from OpenAI. While these are currently the gold-standard, there are many open-source models that are rapidly catching up (Falcon 40B, Llama 2, etc.). Before we see how these models perform on our task, let's first consider a few reasons why we should care about open-source models.
data ownership: you can serve your models and pass data to your models, without having to share it with a third-party API endpoint.
fine-tune: with access to our model's weights, we can actually fine-tune them, as opposed to experimenting with fickle prompting strategies.
optimization: we have full freedom to optimize our deployed models for inference (ex. quantization, pruning, etc.) to reduce costs.
And we can plot these on a bar plot to compare them visually.
# Transform data into a new dictionary with four keys\nby_model_and_context = {}\nfor context_type, models_data in performance.items():\n for model, metrics in models_data.items():\n key = f\"{model}_{context_type}\"\n by_model_and_context[key] = metrics\n
# Extracting the model names and the metric values\nmodels = list(by_model_and_context.keys())\nmetrics = list(by_model_and_context[models[0]].keys())\n\n# Plotting the bar chart with metric scores on top of each bar\nfig, ax = plt.subplots(figsize=(10, 4))\nwidth = 0.2\nx = range(len(models))\n\nfor i, metric in enumerate(metrics):\n metric_values = [by_model_and_context[model][metric] for model in models]\n ax.bar([pos + width * i for pos in x], metric_values, width, label=metric)\n # Displaying the metric scores on top of each bar\n for pos, val in zip(x, metric_values):\n ax.text(pos + width * i, val, f'{val:.3f}', ha='center', va='bottom', fontsize=9)\n\nax.set_xticks([pos + width for pos in x])\nax.set_xticklabels(models, rotation=0, ha='center', fontsize=8)\nax.set_ylabel('Performance')\nax.set_title('GPT Benchmarks')\nax.legend(loc='upper left', bbox_to_anchor=(1, 1))\n\nplt.tight_layout()\nplt.show()\n
Our best model is GPT 4 with few shot learning at an f1 score of ~93%. We will see, in the rest of the course, how fine-tuning an LLM with a proper training dataset to change the actual weights of the last N layers (as opposed to the hard prompt tuning here) will yield similar/slightly better results to GPT 4 (at a fraction of the model size and inference costs).
However, the best system might actually be a combination of using these few-shot hard prompt LLMs alongside fine-tuned LLMs. For example, our fine-tuned LLMs in the course will perform well when the test data is similar to the training data (similar distributions of vocabulary, etc.) but may not perform well on out of distribution. Whereas, these hard prompted LLMs, by themselves or augmented with additional context (ex. arXiv plugins in our case), could be used when our primary fine-tuned model is not so confident.
We'll define a set_seeds function that will set the seeds for reproducibility across our libraries (np.random.seed, random.seed, torch.manual_seed and torch.cuda.manual_seed). We'll also set the behavior for some torch backends to ensure deterministic results when we run our workloads on GPUs.
When working with very large datasets, it's a good idea to limit the number of samples in our dataset so that we can execute our code quickly and iterate on bugs, etc. This is why we have a num_samples input argument in our load_data function (None = no limit, all samples).
We'll also define a custom preprocessor class that we'll to conveniently preprocess our dataset but also to save/load for later. When defining a preprocessor, we'll need to define a _fit method to learn how to fit to our dataset and a _transform_{pandas|numpy} method to preprocess the dataset using any components from the _fit method. We can either define a _transform_pandas method to apply our preprocessing to a Pandas DataFrame or a _transform_numpy method to apply our preprocessing to a NumPy array. We'll define the _transform_pandas method since our preprocessing function expects a batch of data as a Pandas DataFrame.
class CustomPreprocessor(Preprocessor):\n\"\"\"Custom preprocessor class.\"\"\"\n def _fit(self, ds):\n tags = ds.unique(column=\"tag\")\n self.class_to_index = {tag: i for i, tag in enumerate(tags)}\n self.index_to_class = {v:k for k, v in self.class_to_index.items()}\n def _transform_pandas(self, batch): # could also do _transform_numpy\n return preprocess(batch, class_to_index=self.class_to_index)\n
Now we're ready to start defining our model architecture. We'll start by loading a pretrained LLM and then defining the components needed for fine-tuning it on our dataset. Our pretrained LLM here is a transformer-based model that has been pretrained on a large corpus of scientific text called scibert.
If you're not familiar with transformer-based models like LLMs, be sure to check out the attention and Transformers lessons.
import torch.nn as nn\nfrom transformers import BertModel\n
We can load our pretrained model by using the from_pretrained` method.
Once our model is loaded, we can tokenize an input text, convert it to torch tensors and pass it through our model to get a sequence and pooled representation of the text.
# Sample\ntext = \"Transfer learning with transformers for text classification.\"\nbatch = tokenizer([text], return_tensors=\"np\", padding=\"longest\")\nbatch = {k:torch.tensor(v) for k,v in batch.items()} # convert to torch tensors\nseq, pool = llm(input_ids=batch[\"input_ids\"], attention_mask=batch[\"attention_mask\"])\nnp.shape(seq), np.shape(pool)\n
We're going to use this pretrained model to represent our input text features and add additional layers (linear classifier) on top of it for our specific classification task. In short, the pretrained LLM will process the tokenized text and return a sequence (one representation after each token) and pooled (combined) representation of the text. We'll use the pooled representation as input to our final fully-connection layer (fc1) to result in a vector of size num_classes (number of classes) that we can use to make predictions.
We can iterate through our dataset in batches however we may have batches of different sizes. Recall that our tokenizer padded the inputs to the longest item in the batch (padding=\"longest\"). However, our batches for training will be smaller than our large data processing batches and so our batches here may have inputs with different lengths. To address this, we're going to define a custom collate_fn to repad the items in our training batches.
from ray.train.torch import get_device\n
Our pad_array function will take an array of arrays and pad the inner arrays to the longest length.
def pad_array(arr, dtype=np.int32):\n max_len = max(len(row) for row in arr)\n padded_arr = np.zeros((arr.shape[0], max_len), dtype=dtype)\n for i, row in enumerate(arr):\n padded_arr[i][:len(row)] = row\n return padded_arr\n
And our collate_fn will take a batch of data to pad them and convert them to the appropriate PyTorch tensor types.
Next, we'll implement set the necessary utility functions for distributed training.
from ray.air import Checkpoint, session\nfrom ray.air.config import CheckpointConfig, DatasetConfig, RunConfig, ScalingConfig\nimport ray.train as train\nfrom ray.train.torch import TorchCheckpoint, TorchTrainer\nimport torch.nn.functional as F\n
We'll start by defining what one step (or iteration) of training looks like. This will be a function that takes in a batch of data, a model, a loss function, and an optimizer. It will then perform a forward pass, compute the loss, and perform a backward pass to update the model's weights. And finally, it will return the loss.
Note: We're using the ray.data.iter_torch_batches method instead of torch.utils.data.DataLoader to create a generator that will yield batches of data. In fact, this is the only line that's different from a typical PyTorch training loop and the actual training workflow remains untouched. Ray supports many other ways to load/consume data for different frameworks as well.
The validation step is quite similar to the training step but we don't need to perform a backward pass or update the model's weights.
def eval_step(ds, batch_size, model, num_classes, loss_fn):\n\"\"\"Eval step.\"\"\"\n model.eval()\n loss = 0.0\n y_trues, y_preds = [], []\nds_generator = ds.iter_torch_batches(batch_size=batch_size, collate_fn=collate_fn)\nwith torch.inference_mode():\n for i, batch in enumerate(ds_generator):\n z = model(batch)\n targets = F.one_hot(batch[\"targets\"], num_classes=num_classes).float() # one-hot (for loss_fn)\n J = loss_fn(z, targets).item()\n loss += (J - loss) / (i + 1)\n y_trues.extend(batch[\"targets\"].cpu().numpy())\n y_preds.extend(torch.argmax(z, dim=1).cpu().numpy())\n return loss, np.vstack(y_trues), np.vstack(y_preds)\n
Next, we'll define the train_loop_per_worker which defines the overall training loop for each worker. It's important that we include operations like loading the datasets, models, etc. so that each worker will have its own copy of these objects. Ray takes care of combining all the workers' results at the end of each iteration, so from the user's perspective, it's the exact same as training on a single machine!
The only additional lines of code we need to add compared to a typical PyTorch training loop are the following:
session.get_dataset_shard(\"train\") and session.get_dataset_shard(\"val\") to load the data splits (session.get_dataset_shard).
model = train.torch.prepare_model(model) to prepare the torch model for distributed execution (train.torch.prepare_model).
batch_size_per_worker = batch_size // session.get_world_size() to adjust the batch size for each worker (session.get_world_size).
session.report(metrics, checkpoint=checkpoint) to report metrics and save our model checkpoint (session.report).
All the other lines of code are the same as a typical PyTorch training loop!
Our dataset doesn't suffer from horrible class imbalance, but if it did, we could easily account for it through our loss function. There are also other strategies such as over-sampling less frequent classes and under-sampling popular classes.
# Class weights\nbatch_counts = []\nfor batch in train_ds.iter_torch_batches(batch_size=256, collate_fn=collate_fn):\n batch_counts.append(np.bincount(batch[\"targets\"].cpu().numpy()))\ncounts = [sum(count) for count in zip(*batch_counts)]\nclass_weights = np.array([1.0/count for i, count in enumerate(counts)])\nclass_weights_tensor = torch.Tensor(class_weights).to(get_device())\n\n# Training components\nloss_fn = nn.BCEWithLogitsLoss(weight=class_weights_tensor)\n...\n
Next we'll define our scaling configuration (ScalingConfig) that will specify how we want to scale our training workload. We specify the number of workers (num_workers), whether to use GPU or not (use_gpu), the resources per worker (resources_per_worker) and how much CPU each worker is allowed to use (_max_cpu_fraction_per_node).
_max_cpu_fraction_per_node=0.8 indicates that 20% of CPU is reserved for non-training workloads that our workers will do such as data preprocessing (which we do prior to training anyway).
Next, we'll define our CheckpointConfig which will specify how we want to checkpoint our model. Here we will just save one checkpoint (num_to_keep) based on the checkpoint with the minval_loss. We'll also configure a RunConfig which will specify the name of our run and where we want to save our checkpoints.
We'll be naming our experiment llm and saving our results to ~/ray_results, so a sample directory structure for our trained models would look like this:
The TorchTrainer_ objects are the individuals runs in this experiment and each one will have the following contents:
/home/ray/ray_results/TorchTrainer_fd40a_00000_0_2023-07-20_18-14-50/\n\u251c\u2500\u2500 checkpoint_000009/ # we only save one checkpoint (the best)\n\u251c\u2500\u2500 events.out.tfevents.1689902160.ip-10-0-49-200\n\u251c\u2500\u2500 params.json\n\u251c\u2500\u2500 params.pkl\n\u251c\u2500\u2500 progress.csv\n\u2514\u2500\u2500 result.json\n
There are several other configs that we could set with Ray (ex. failure handling) so be sure to check them out here.
Stopping criteria
While we'll just let our experiments run for a certain number of epochs and stop automatically, our RunConfig accepts an optional stopping criteria (stop) which determines the conditions our training should stop for. It's entirely customizable and common examples include a certain metric value, elapsed time or even a custom class.
Calling materialize here is important because it will cache the preprocessed data in memory. This will allow us to train our model without having to reprocess the data each time.
Because we've preprocessed the data prior to training, we can use the fit=False and transform=False flags in our dataset config. This will allow us to skip the preprocessing step during training.
We'll pass all of our functions and configs to the TorchTrainer class to start training. Ray supports a wide variety of framework Trainers so if you're using other frameworks, you can use the corresponding Trainer class instead.
While our model is training, we can inspect our Ray dashboard to observe how our compute resources are being utilized.
\ud83d\udcbb Local
We can inspect our Ray dashboard by opening http://127.0.0.1:8265 on a browser window. Click on Cluster on the top menu bar and then we will be able to see a list of our nodes (head and worker) and their utilizations.
\ud83d\ude80 Anyscale
On Anyscale Workspaces, we can head over to the top right menu and click on \ud83d\udee0\ufe0f Tools \u2192 Ray Dashboard and this will open our dashboard on a new tab. Click on Cluster on the top menu bar and then we will be able to see a list of our nodes (head and worker) and their utilizations.
Learn about all the other observability features on the Ray Dashboard through this video.
Now that we've trained our model, we can evaluate it on a separate holdout test set. We'll cover the topic of evaluation much more extensively in our evaluation lesson but for now we'll calculate some quick overall metrics.
from ray.train.torch import TorchPredictor\nfrom sklearn.metrics import precision_recall_fscore_support\n
We'll define a function that can take in a dataset and a predictor and return the performance metrics.
Load the predictor and preprocessor from the best checkpoint:
Now let's load our trained model for inference on new data. We'll create a few utility functions to format the probabilities into a dictionary for each class and to return predictions for each item in a dataframe.
import pandas as pd\n
def format_prob(prob, index_to_class):\n d = {}\n for i, item in enumerate(prob):\n d[index_to_class[i]] = item\n return d\n
def predict_with_proba(df, predictor):\n preprocessor = predictor.get_preprocessor()\n z = predictor.predict(data=df)[\"predictions\"]\n y_prob = torch.tensor(np.stack(z)).softmax(dim=1).numpy()\n results = []\n for i, prob in enumerate(y_prob):\n tag = decode([z[i].argmax()], preprocessor.index_to_class)[0]\n results.append({\"prediction\": tag, \"probabilities\": format_prob(prob, preprocessor.index_to_class)})\n return results\n
We'll load our predictor from the best checkpoint and load it's preprocessor.
And now we're ready to apply our model to new data. We'll create a sample dataframe with a title and description and then use our predict_with_proba function to get the predictions. Note that we use a placeholder value for tag since our input dataframe will automatically be preprocessed (and it expects a value in the tag column).
# Predict on sample\ntitle = \"Transfer learning with transformers\"\ndescription = \"Using transformers for transfer learning on text classification tasks.\"\nsample_df = pd.DataFrame([{\"title\": title, \"description\": description, \"tag\": \"other\"}])\npredict_with_proba(df=sample_df, predictor=predictor)\n
Distributed training strategies are great for when our data or models are too large for training but there are additional strategies to make the models itself smaller for serving. The following model compression techniques are commonly used to reduce the size of the model:
Pruning: remove weights (unstructured) or entire channels (structured) to reduce the size of the network. The objective is to preserve the model\u2019s performance while increasing its sparsity.
Quantization: reduce the memory footprint of the weights by reducing their precision (ex. 32 bit to 8 bit). We may loose some precision but it shouldn\u2019t affect performance too much.
Distillation: training smaller networks to \u201cmimic\u201d larger networks by having it reproduce the larger network\u2019s layers\u2019 outputs.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Training - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
Hyperparameter tuning is the process of discovering a set of performant parameter values for our model. It can be a computationally involved process depending on the number of parameters, search space and model architectures. Hyperparameters don't just include the model's parameters but could also include parameters related to preprocessing, splitting, etc. When we look at all the different parameters that can be tuned, it quickly becomes a very large search space. However, just because something is a hyperparameter doesn't mean we need to tune it.
It's absolutely acceptable to fix some hyperparameters (ex. using lower cased text [lower=True] during preprocessing).
You can initially just tune a small, yet influential, subset of hyperparameters that you believe will yield great results.
We want to optimize our hyperparameters so that we can understand how each of them affects our objective. By running many trials across a reasonable search space, we can determine near ideal values for our different parameters.
There are many options for hyperparameter tuning (Ray tune, Optuna, Hyperopt, etc.). We'll be using Ray Tune with it's HyperOpt integration for it's simplicity and general popularity. Ray Tune also has a wide variety of support for many other tune search algorithms (Optuna, Bayesian, etc.).
There are many factors to consider when performing hyperparameter tuning. We'll be conducting a small study where we'll tune just a few key hyperparameters across a few trials. Feel free to include additional parameters and to increase the number trials in the tuning experiment.
# Number of trials (small sample)\nnum_runs = 2\n
We'll start with some the set up, data and model prep as we've done in previous lessons.
from ray import tune\nfrom ray.tune import Tuner\nfrom ray.tune.schedulers import AsyncHyperBandScheduler\nfrom ray.tune.search import ConcurrencyLimiter\nfrom ray.tune.search.hyperopt import HyperOptSearch\n
We can think of tuning as training across different combinations of parameters. For this, we'll need to define several configurations around when to stop tuning (stopping criteria), how to define the next set of parameters to train with (search algorithm) and even the different values that the parameters can take (search space).
We'll start by defining our CheckpointConfig and RunConfig as we did for training:
Next, we're going to set the initial parameter values and the search algorithm (HyperOptSearch) for our tuning experiment. We're also going to set the maximum number of trials that can be run concurrently (ConcurrencyLimiter) based on the compute resources we have.
It's a good idea to start with some initial parameter values that you think might be reasonable. This can help speed up the tuning process and also guarantee at least one experiment that will perform decently well.
Next, we're going to define the parameter search space by choosing the parameters, their distribution and range of values. Depending on the parameter type, we have many different distributions to choose from.
Next, we're going to define a scheduler to prune unpromising trials. We'll be using AsyncHyperBandScheduler (ASHA), which is a very popular and aggressive early-stopping algorithm. Due to our aggressive scheduler, we'll set a grace_period to allow the trials to run for at least a few epochs before pruning and a maximum of max_t epochs.
# Scheduler\nscheduler = AsyncHyperBandScheduler(\n max_t=train_loop_config[\"num_epochs\"], # max epoch (<time_attr>) per trial\n grace_period=5, # min epoch (<time_attr>) per trial\n)\n
And on our MLflow dashboard, we can create useful plots like a parallel coordinates plot to visualize the different hyperparameters and their values across the different trials.
# Predict on sample\ntitle = \"Transfer learning with transformers\"\ndescription = \"Using transformers for transfer learning on text classification tasks.\"\nsample_df = pd.DataFrame([{\"title\": title, \"description\": description, \"tag\": \"other\"}])\npredict_with_proba(df=sample_df, predictor=predictor)\n
Now that we're tuned our model, in the next lesson, we're going to perform a much more intensive evaluation on our model compared to just viewing it's overall metrics on a test set.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Tuning - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
"},{"location":"courses/mlops/versioning/","title":"Versioning Code, Data and Models","text":""},{"location":"courses/mlops/versioning/#intuition","title":"Intuition","text":"
In this lesson, we're going to learn how to version our code, data and models to ensure reproducible behavior in our ML systems. It's imperative that we can reproduce our results and track changes to our system so we can debug and improve our application. Without it, it would be difficult to share our work, recreate our models in the event of system failures and fallback to previous versions in the event of regressions.
To version our code, we'll be using git, which is a widely adopted version control system. In fact, when we cloned our repository in the setup lesson, we pulled code from a git repository that we had prepared for you.
We can then make changes to the code and Git, which is running locally on our computer, will keep track of our files and it's versions as we add and commit our changes. But it's not enough to just version our code locally, we need to push our work to a central location that can be pulled by us and others we want to grant access to. This is where remote repositories like GitHub, GitLab, BitBucket, etc. provide a remote location to hold our versioned code in.
Here's a simplified workflow for how we version our code using GitHub:
While Git is ideal for saving our code, it's not ideal for saving artifacts like our datasets (especially unstructured data like text or images) and models. Also, recall that Git stores every version of our files and so large files that change frequently can very quickly take up space. So instead, it would be ideal if we can save locations (pointers) to these large artifacts in our code as opposed to the artifacts themselves. This way, we can version the locations of our artifacts and pull them as they're needed.
While we're saving our dataset on GitHub for easy course access (and because our dataset is small), in a production setting, we would use a remote blob storage like S3 or a data warehouse like Snowflake. There are also many tools available for versioning our data, such as GitLFS, Dolt, Pachyderm, DVC, etc. With any of these solutions, we would be pointing to our remote storage location and versioning the pointer locations (ex. S3 bucket path) to our data instead of the data itself.
In a production setting, these would be remote such as S3 for the artifact store and a database service (ex. PostgreSQL RDS) as our backend store. This way, our models can be versioned and others, with the appropriate access credentials, can pull the model artifacts and deploy them.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
Learn more
To cite this content, please use:
@article{madewithml,\nauthor = {Goku Mohandas},\ntitle = { Versioning - Made With ML },\nhowpublished = {\\url{https://madewithml.com/}},\nyear = {2023}\n}\n
This content will be coming soon! Be sure to subscribe and follow us on Twitter and LinkedIn for updates and tips.
"},{"location":"misc/confirmation/","title":"\ud83d\udcec Check your email","text":""},{"location":"misc/confirmation/#_1","title":"\ud83d\udcec Check your email","text":"
Thank you for subscribing to our newsletter! A confirmation email was sent to the email address you provided. Please click on the confirmation button in the email to complete your subscription. If you don\u2019t see it within a few minutes, be sure to check your promotions/spam/junk folder (and mark it Not junk so you receive our future emails).
We created Made With ML to educate and enable the community to responsibly develop, deploy and maintain production machine learning applications. While there are many facets to this mission, at its core are the relationships with teams who share this mission. We want to work together to help the community discover and use the best tools for their context and start the flywheel to make the tools even better.
A few numbers that reflect the 100% organic traction we've had over the past few years and the need it's filled in the community for bringing ML to production:
#1 MLOps repository on GitHub (30K+ stars)
40K+ subscribers to our community newsletter
a highly recommended industry resource (testimonials, Twitter, LinkedIn)
1.5M+ organic monthly website traffic (50K+ MAU)
10K+ followers on Twitter & LinkedIn (w/ great organic traction on every post)
top organic SEO for common/popular search terms (mlops lessons, testing ml, experiment tracking, monitoring ml, data stack ml, etc.)
All of our lessons focus on first principles when approaching a concept. This helps develop the foundational understanding to be able to adapt to any stack. But to really solidify the understanding, we implement everything in code within the context of an end-to-end project. This helps understand the implicit value a tool provides and develop the decision making framework for constructing the appropriate stack. And because the community adopts what they've learned for their own use cases in industry, it's imperative that we use tools that can offer that enterprise maturity.
Your product will be deeply integrated into the MLOps course, where thousands of developers everyday will use and assess your product for their industry contexts. All of this visibility and traction is invaluable for industry adoption, standing out in the competitive landscape and using the feedback to improve the product.
We also have many downstream projects in progress to add more value on top of the content (async video course, private community, university labs, corporate onboarding, talent platform).
After you've applied and been accepted to the next cohort, copy and paste this email template below to send to your manager for reimbursement for the course. Feel free to add any additional details as you see fit.
Subject: Reimbursement for Made With ML's MLOps Course
Hi,
Hope you're doing well. I was recently accepted into Made With ML's MLOps course, which is an interactive project-based course on MLOps fundamentals. The course costs $1,250 but the value I'll gain for myself and our team/company will pay for the course almost immediately. I added some key information about the course below and would love to get this career development opportunity reimbursed.
Course page: https://madewithml.com/
"},{"location":"misc/reimbursement/#what-is-the-course","title":"What is the course?","text":"
An interactive project-based course to learn and apply the fundamentals of MLOps. I'll be learning to combine machine learning with software engineering best practices which I want to extend to build and improve our own systems. This course brings all of the MLOps best practices into one place, allowing me to quickly (and properly) learn it. And best of all, the course can be done before and after work, so it won't be interfering during work hours.
Here's a quick look at the curriculum:
"},{"location":"misc/reimbursement/#whos-teaching-the-course","title":"Who's teaching the course?","text":"
The course is from Made With ML, one of the top ML repositories on GitHub (30K+ stars) with a growing community (30K+) and is a highly recommended resource used by industry. Their content not only covers MLOps concepts but they go deep into actually implementing everything with production quality code.
"},{"location":"misc/reimbursement/#how-will-this-help-me","title":"How will this help me?","text":"
I'll be learning the foundation I need to responsibly develop ML systems. This includes producing clean, production-grade code, testing my work, understanding MLOps (experiment management, monitoring, systems design, etc.) and data engineering (data stack, orchestration, feature stores) concepts.
"},{"location":"misc/reimbursement/#how-will-this-help-our-company","title":"How will this help our company?","text":"
What I learn will directly translate to better quality ML systems in our products. I'll also be able to engage in conversations with peers and management as we traverse this space to build what's right for us. And, most important of all, I'll be able to pass on what I learn as I collaborate with others in our team so we're all working towards building reliable ML systems.
Send me an email at goku@madewithml.com to say hi, a bit about yourself and what you're currently learning or working on. I personally respond to all emails and always love to meet people from the community.
Upcoming live cohorts
Sign up for our upcoming live cohort, where we'll provide live lessons + QA, compute (GPUs) and community to learn everything in one day.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
at the top right corner of our Anyscale Workspace page and this will open up our JupyterLab instance in a new tab. Then navigate to the 
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+