-
Notifications
You must be signed in to change notification settings - Fork 0
/
search.xml
185 lines (87 loc) · 106 KB
/
search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>行人身份识别</title>
<link href="/2023/08/06/%E8%A1%8C%E4%BA%BA%E8%BA%AB%E4%BB%BD%E8%AF%86%E5%88%AB/"/>
<url>/2023/08/06/%E8%A1%8C%E4%BA%BA%E8%BA%AB%E4%BB%BD%E8%AF%86%E5%88%AB/</url>
<content type="html"><![CDATA[<h2 id="1-赛题描述"><a href="#1-赛题描述" class="headerlink" title="1 赛题描述"></a>1 赛题描述</h2><h3 id="1-1-赛题背景"><a href="#1-1-赛题背景" class="headerlink" title="1.1 赛题背景"></a>1.1 赛题背景</h3><p>机器虽然被大量用到农业生产中，但人还是不可或缺的因素。通过农民身份识别，可以真实客观地记录农民的状态，为农场管理和农产品追溯提供真实的客观数据；较之直接存储视频，可以有效地降低存储空间；自动识别也比人工监管，大幅度提高效率，减少人工成本。</p><h3 id="1-2-赛事任务"><a href="#1-2-赛事任务" class="headerlink" title="1.2 赛事任务"></a>1.2 赛事任务</h3><p>农民身份识别需要对农民进行分类，本次大赛提供了中国农业大学实验室视频制成的图像序列。提供了<strong>25名农民</strong>身份，每个身份包含10段视频制成的图像序列，选手需要对图像序列进行预处理，打标签，并对农民进行身份识别。参赛选手先对图像进行预处理，并制作样本，对图像中的农民进行识别。选手需要自行训练模型，并上传自己训练好的模型和权重。</p><p>这里实际上就是图像分类的问题，因为虽然是人物身份识别，但是所有的身份已经确定，仅包含在这25个人之中，因此实际上是图像分类的问题。</p><h3 id="1-3-评价指标"><a href="#1-3-评价指标" class="headerlink" title="1.3 评价指标"></a>1.3 评价指标</h3><p>本模型依据提交的结果文件，采用Macro-F1进行评价，其中Macro-F1的一种定义如下：</p><p><img src="/2023/08/06/%E8%A1%8C%E4%BA%BA%E8%BA%AB%E4%BB%BD%E8%AF%86%E5%88%AB/image-20230806212241289.png" alt="image-20230806212241289"></p><p><img src="/2023/08/06/%E8%A1%8C%E4%BA%BA%E8%BA%AB%E4%BB%BD%E8%AF%86%E5%88%AB/79069068-f7cebf00-7ce8-11ea-948d-d90d2a40fba5.png" alt="79069068-f7cebf00-7ce8-11ea-948d-d90d2a40fba5"></p><p>在“Training algorithms for linear text classifiers（ Lewis, David D., et al. “Training algorithms for linear text classifiers.” SIGIR. Vol. 96. 1996.）”中，作者指出，第一种方式对错误的分布不太敏感。，Macro-F1应当是所有类中F1-score的平均值，即第二种方式才是Macro-F1的计算方式，因此我们使用第二种计算方式。</p><h2 id="2-任务分析"><a href="#2-任务分析" class="headerlink" title="2 任务分析"></a>2 任务分析</h2><p> <img src="/2023/08/06/%E8%A1%8C%E4%BA%BA%E8%BA%AB%E4%BB%BD%E8%AF%86%E5%88%AB/image-20230809144608958.png" alt="image-20230809144608958"></p><ol><li><p>可以观察到，这里的图像都是上下顶格的，考虑到边缘在CNN卷积的过程中会存在padding的情况，我们可以考试适当放缩图片，为边缘补充zero padding，保证图像效果。</p></li><li><p>测试集的评测标准是 macro-F1，但是似乎在两次实验中都没有发现使用F1有显著好于使用acc的情况，似乎说明了macro-F1在验证集的评价能力并没有显著高于测试集？</p></li></ol><h2 id="3-代码"><a href="#3-代码" class="headerlink" title="3 代码"></a>3 代码</h2><h3 id="1-数据读取"><a href="#1-数据读取" class="headerlink" title="1. 数据读取"></a>1. 数据读取</h3><pre><code class="python"># 数据读取# 训练集train_loader = paddle.io.DataLoader( XunFeiDataset(train_path[:-1000], train_label[:-1000], A.Compose([ A.RandomRotate90(), A.Resize(256, 256), A.RandomCrop(224, 224), A.HorizontalFlip(p=0.5), A.RandomContrast(p=0.5), A.RandomBrightnessContrast(p=0.5), A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)) ]) ), batch_size=30, shuffle=True, num_workers=0)# 验证集val_loader = paddle.io.DataLoader( XunFeiDataset(train_path[-1000:], train_label[-1000:], A.Compose([ A.Resize(256, 256), A.RandomCrop(224, 224), A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)) ]) ), batch_size=30, shuffle=False, num_workers=0)# 测试集test_loader = paddle.io.DataLoader( XunFeiDataset(test_path, [0] * len(test_path), A.Compose([ A.Resize(256, 256), A.RandomCrop(224, 224), A.HorizontalFlip(p=0.5), A.RandomContrast(p=0.5), A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)) ]) ), batch_size=2, shuffle=False, num_workers=0)</code></pre><p>观察到这里是固定采用最后一千张作为验证集，因此可能会存在问题，这里我们在数据集开始划分的时候进行随机shuffle，保证数据集的划分更加具有随机性，并且是独立同分布的。</p><p><strong>要注意，shuffle的话需要保证标签一起shuffle，之前改代码的时候遇到了这个问题，改了好久都没反应过来。（</strong></p><h3 id="2-模型建立"><a href="#2-模型建立" class="headerlink" title="2. 模型建立"></a>2. 模型建立</h3><ol><li>保存最优模型，防止产生过拟合现象。</li></ol><pre><code class="python"># 模型训练def train(train_loader, model, criterion, optimizer): model.train() train_loss = 0.0 for i, data in enumerate(train_loader()): input, target = data input = paddle.to_tensor(input) target = paddle.to_tensor(target) output = model(input) loss = criterion(output, target) loss.backward() optimizer.step() optimizer.clear_grad() if i % 100 == 0: print('Train loss', float(loss)) train_loss += float(loss) return train_loss/len(train_loader)# 模型验证 def validate(val_loader, model, criterion): model.eval() val_acc = 0.0 for i, data in enumerate(val_loader()): input, target = data input = paddle.to_tensor(input) target = paddle.to_tensor(target) output = model(input) loss = criterion(output, target) # 这里是因为使用了cutmix，所以target的label也是软标签形式 # 其实我有点不确定这里是不是该用acc val_acc += float((output.argmax(1) == target.argmax(1)).sum()) return val_acc / len(val_loader.dataset) # 模型预测def predict(test_loader, model, criterion): model.eval() val_acc = 0.0 test_pred = [] for i, data in enumerate(test_loader()): input, target = data input = paddle.to_tensor(input) target = paddle.to_tensor(target) output = model(input) test_pred.append(output.data.cpu().numpy()) return np.vstack(test_pred)for _ in range(100): train_loss = train(train_loader, model, criterion, optimizer) val_acc = validate(val_loader, model, criterion) train_acc = validate(train_loader, model, criterion) print(train_loss, train_acc, val_acc) if val_acc > best_acc: best_acc = val_acc best_dict = model.state_dict() print(' EPOCH {} Best accuracy is {:0.6f}'.format(_, best_acc))</code></pre><ol start="2"><li><p>采用学习率的梯度下降，在最小值点提供更小的学习率，防止不收敛。之前同学们提到的0.2-0.5突然震荡回0.8就可能是这种情况。这里我的损失收敛到了0.05-0.12左右。使用了paddle的stepdecay策略，梯度下降学习率。</p><p> <img src="/2023/08/06/%E8%A1%8C%E4%BA%BA%E8%BA%AB%E4%BB%BD%E8%AF%86%E5%88%AB/image-20230806223356737.png" alt="image-20230806223356737"></p><pre><code class="python">model = XunFeiNet()model = modelcriterion = nn.CrossEntropyLoss(soft_label=True)scheduler = paddle.optimizer.lr.StepDecay(learning_rate=1e-3, step_size=3, gamma=0.9, verbose=False)optimizer = paddle.optimizer.AdamW(parameters=model.parameters(), learning_rate=0.001)</code></pre></li><li><p>使用cutmix增强，防止类别不均衡的问题。</p><pre><code class="python">def rand_bbox(size, lam): if len(size) == 4: W = size[2] H = size[3] elif len(size) == 3: W = size[0] H = size[1] else: raise Exception cut_rat = np.sqrt(1. - lam) cut_w = int(W * cut_rat) cut_h = int(H * cut_rat) # uniform cx = np.random.randint(W) cy = np.random.randint(H) bbx1 = np.clip(cx - cut_w // 2, 0, W) bby1 = np.clip(cy - cut_h // 2, 0, H) bbx2 = np.clip(cx + cut_w // 2, 0, W) bby2 = np.clip(cy + cut_h // 2, 0, H) return bbx1, bby1, bbx2, bby2# 自定义数据集# 带有图片缓存的逻辑DATA_CACHE = {}class XunFeiDataset(Dataset): def __init__(self, img_path, img_label, transform=None): self.img_path = img_path self.img_label = img_label if transform is not None: self.transform = transform else: self.transform = None def __getitem__(self, index): if self.img_path[index] in DATA_CACHE: img = DATA_CACHE[self.img_path[index]] else: img = cv2.imread(self.img_path[index]) DATA_CACHE[self.img_path[index]] = img label = paddle.zeros([25]) label[self.img_label[index]] = 1 # ------------------------------ CutMix ------------------------------------------ prob = 20 if random.randint(0, 99) < prob: rand_index = random.randint(0, len(self.img_path) - 1) if self.img_path[rand_index] in DATA_CACHE: rand_image = DATA_CACHE[self.img_path[rand_index]] else: rand_image = cv2.imread(self.img_path[rand_index]) DATA_CACHE[self.img_path[index]] = rand_image lam = np.random.beta(1,1) bbx1, bby1, bbx2, bby2 = rand_bbox(img.shape, lam) img[bbx1:bbx2, bby1:bby2, :] = rand_image[bbx1:bbx2, bby1:bby2, :] # masks[bbx1:bbx2, bby1:bby2] = rand_masks[bbx1:bbx2, bby1:bby2] lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (img.shape[1] * img.shape[0])) rand_label = paddle.zeros([25]) rand_label[self.img_label[rand_index]] = 1 label = label * lam + rand_label * (1. - lam) # --------------------------------- CutMix --------------------------------------- # print(f"[INFO]: label is {label}") if self.transform is not None: img = self.transform(image = img)['image'] img = img.transpose([2,0,1]) return img, label def __len__(self): return len(self.img_path)</code></pre></li><li><p>采用预训练Resnet50_64x4d作为骨架，修改全连接层为最后为25类。同时可以使用VGG，densenet等分别得到预测结果，最后使用集成的方法。我这里使用了简单投票，其实可以考虑更有效的集成方法。</p><pre><code class="python">class XunFeiNet(paddle.nn.Layer): def __init__(self): super(XunFeiNet, self).__init__() model = models.resnet101(True) model.avgpool = paddle.nn.AdaptiveAvgPool2D(1) model.fc = nn.Linear(512, 25) self.resnet = model def forward(self, img): out = self.resnet(img) return out class XunFeiNet(paddle.nn.Layer): def __init__(self): super(XunFeiNet, self).__init__() self.net = paddle.hub.load('PaddlePaddle/PaddleClas:develop', 'resnext50_64x4d', source='gitee', force_reload=False, pretrained=True) self.linear = nn.Linear(1000, 25) def forward(self, x): x = self.net(x) x = self.linear(x) return x</code></pre></li><li><p>防止过拟合的几种方法</p><blockquote><p>早停：即在验证集上的性能不再提升时，停止训练。这样可以防止模型在训练集上过度学习。<br>正则化：在损失函数中加入L1或者L2正则项，防止模型权重过大。<br>Dropout：随机丢弃一些节点的输出，可以增加模型的鲁棒性。<br>数据增强：通过对训练数据进行变换，从而创建新的训练样本，使模型更容易泛化到未见过的数据。<br>使用更小的网络结构：减少模型的复杂度，避免过拟合。</p></blockquote></li><li><p>TTA 集成学习的思路较为简单，若某一个预测器件的准确率大于50%，那么很多个这样的预测器对结果投票得出的结果准确率将会更高。类似于模型集成，增加了鲁棒性，在测试时通过各种数据增广，在得到结果后再综合得到输出。</p></li></ol>]]></content>
<tags>
<tag> Datawhale-Related </tag>
</tags>
</entry>
<entry>
<title>Baseline阅读与修改</title>
<link href="/2023/07/21/Baseline%E9%98%85%E8%AF%BB%E4%B8%8E%E4%BF%AE%E6%94%B9/"/>
<url>/2023/07/21/Baseline%E9%98%85%E8%AF%BB%E4%B8%8E%E4%BF%AE%E6%94%B9/</url>
<content type="html"><![CDATA[<h2 id="Baseline部分"><a href="#Baseline部分" class="headerlink" title="Baseline部分"></a>Baseline部分</h2><h3 id="数据集处理"><a href="#数据集处理" class="headerlink" title="数据集处理"></a>数据集处理</h3><h4 id="数据分析"><a href="#数据分析" class="headerlink" title="数据分析"></a>数据分析</h4><pre><code class="python">sizeSet = set()for path in trainPath: image = nib.load(path) imgShape = image.dataobj.shape if imgShape not in sizeSet: sizeSet.add(imgShape)print(sizeSet)</code></pre><p>可以发现数据中尺寸总体只有以下几种形式：</p><p>{(168, 168, 82, 1), (128, 128, 47, 1), (256, 256, 207, 1), (256, 256, 81, 1), (400, 400, 109, 1), (128, 128, 540, 1), (128, 128, 63, 1), (168, 168, 81, 1)}，因此在建立数据集时要注意尺寸设置。</p><pre><code class="python">### Import related ###...# 路径获取trainPath = glob.glob('./脑PET图像分析和疾病预测挑战赛公开数据/Train/*/*')testPath = glob.glob('./脑PET图像分析和疾病预测挑战赛公开数据/Test/*')np.random.shuffle(trainPath)np.random.shuffle(testPath)# 因为图片数量较少，防止反复解码带来时间损失，将解码图片存入DATA_CACHE = {}class XunFeiDataset(Dataset): def __init__(self, imgPath, transform=None): self.imgPath = imgPath if transform is not None: self.transform = transform else: self.transform = None def __getitem__(self, index): if self.imgPath[index] in DATA_CACHE: img = DATA_CACHE[self.imgPath[index]] else: # nib load 读入图片 img = nib.load(self.imgPath[index]) # dataobj 读为numpy.ndarray，最后一维为灰度图，可以去掉，从而转为二维卷积 img = img.dataobj[:,:,:, 0] DATA_CACHE[self.imgPath[index]] = img # 随机选择一些通道，防止通道不均，random.choice指变为50个通道， # 每个通道都是随机抽取原来img.shape[-1]其中的一个通道得到，可以重复 idx = np.random.choice(range(img.shape[-1]), 50) img = img[:, :, idx] img = img.astype(np.float32) if self.transform is not None: img = self.transform(image = img)['image'] # 转换数据格式为C, H, W img = img.transpose([2,0,1]) ########################################## # 注意，这里long()很重要，不然会遇到报错 # ########################################## return img,torch.from_numpy(np.array(int('NC' in self.imgPath[index]))).long() def __len__(self): return len(self.imgPath) import albumentations as A############################################ 如果使用windows系统，num_workers置0最稳妥############################################train_loader = torch.utils.data.DataLoader( XunFeiDataset(trainPath[:-10], A.Compose([ A.RandomRotate90(), A.RandomCrop(120, 120), A.HorizontalFlip(p=0.5), A.RandomContrast(p=0.5), A.RandomBrightnessContrast(p=0.5), ]) ), batch_size=2, shuffle=True, num_workers=0, pin_memory=False)val_loader = torch.utils.data.DataLoader( XunFeiDataset(trainPath[-10:], A.Compose([ A.RandomCrop(120, 120), ]) ), batch_size=2, shuffle=False, num_workers=0, pin_memory=False)test_loader = torch.utils.data.DataLoader( XunFeiDataset(testPath, A.Compose([ A.RandomCrop(128, 128), A.HorizontalFlip(p=0.5), A.RandomContrast(p=0.5), ]) ), batch_size=2, shuffle=False, num_workers=0, pin_memory=False)</code></pre><p>这里有两个容易报错的点</p><ol><li>数据类型不匹配</li></ol><pre><code class="python">RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Int'</code></pre><p>对应上面数据类型不匹配的问题，使用 <em>.long()</em> 转换。</p><ol start="2"><li><p>多线程未能正确退出。</p><pre><code class="python">raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from eRuntimeError: DataLoader worker (pid(s) 14652) exited unexpectedly.</code></pre><p>对应上面 num_workers 设置，因为数据量较小，其实设置num_workers=0即可。</p><p><strong>注：本条未验证正确性</strong>：如果数据量较大的情况下要加上</p><pre><code class="python">if name == '__main__':</code></pre></li></ol><h3 id="模型建立"><a href="#模型建立" class="headerlink" title="模型建立"></a>模型建立</h3><h5 id="2D-ResNet"><a href="#2D-ResNet" class="headerlink" title="2D-ResNet"></a>2D-ResNet</h5><pre><code class="python">class XunFeiNet(nn.Module): def __init__(self): super(XunFeiNet, self).__init__() model = models.resnet18(True) model.conv1 = torch.nn.Conv2d(50, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) model.avgpool = nn.AdaptiveAvgPool2d(1) model.fc = nn.Linear(512, 2) self.resnet = model def forward(self, img): out = self.resnet(img) return out model = XunFeiNet()model = model.to('cuda')criterion = nn.CrossEntropyLoss().cuda()optimizer = torch.optim.AdamW(model.parameters(), 0.001)</code></pre><p>这里没什么好说的，使用torch预先设定的模型，并自己修改一些层。</p><h5 id="3D-ResNet"><a href="#3D-ResNet" class="headerlink" title="3D-ResNet"></a>3D-ResNet</h5><p>具体内容可以参见<a href="#%E9%99%84%E5%BD%95%EF%BC%9AA">附录：A</a>。注意这里使用3D-ResNet的时候要保留灰度通道，reshape改为</p><pre><code class="python">img = img.transpose([3,2,0,1])</code></pre><p>或在使用时 unsqueeze 即可。</p><pre><code class="python">model = ResNet3D(Bottleneck, [3, 8, 36, 3], num_classes=2, shortcut_type='B', no_cuda=False, include_top=True) # resnet 152model = model.to('cuda')</code></pre><h3 id="模型训练与验证"><a href="#模型训练与验证" class="headerlink" title="模型训练与验证"></a>模型训练与验证</h3><pre><code class="python">def train(train_loader, model, criterion, optimizer): model.train() trainLoss = 0.0 probAll = [] labelAll = [] for i, (input, target) in enumerate(train_loader): input = input.cuda(non_blocking=True) target = target.cuda(non_blocking=True) output = model(input) # 求每一行的最大值索引 probAll.extend(np.argmax(output.detach().cpu().numpy(),axis=1)) # 将标签也记录 labelAll.extend(target.cpu().numpy()) loss = criterion(output, target) optimizer.zero_grad() loss.backward() optimizer.step() if i % 20 == 0: print(loss.item()) trainLoss += loss.item() # 计算F1-score print("F1-Score:{:.4f}".format(f1_score(labelAll,probAll))) return trainLoss/len(train_loader) def validate(val_loader, model, criterion): model.eval() valAcc = 0.0 probAll = [] labelAll = [] with torch.no_grad(): for i, (input, target) in enumerate(val_loader): input = input.cuda() target = target.cuda() # 计算输出 output = model(input) # 求每一行的最大值索引 probAll.extend(np.argmax(output.detach().cpu().numpy(),axis=1)) # 将标签也记录 labelAll.extend(target.cpu().numpy()) loss = criterion(output, target) valAcc += (output.argmax(1) == target).sum().item() # 因为最终比赛使用F1-Score作为评价标准，所以本地也要使用F1-Score。 ###################################################################### # 注意：因为线上有提交限制，所以一定要建立好本地分数和线上分数的对应关系# ###################################################################### score = f1_score(labelAll,probAll) # print("val F1-Score:{:.4f}".format(score)) return (valAcc / len(val_loader.dataset)), scoreimport copybestDict = NonebestF1 = 0for _ in range(12): trainLoss = train(train_loader, model, criterion, optimizer) valAcc, scoreVal = validate(val_loader, model, criterion) trainAcc, scoreTrain = validate(train_loader, model, criterion) # 根据输出结果保存模型参数 if scoreVal > bestF1: # 使用deepcopy建立副本，避免浅拷贝问题 bestDict = copy.deepcopy(model.state_dict()) bestF1 = scoreVal print(f"[INFO]: Model saved. F1-score {scoreVal}") print(trainLoss, trainAcc, valAcc, scoreTrain, scoreVal )print(f"[INFO]: Training finished. F1-score {bestF1}")</code></pre><h3 id="模型预测与提交"><a href="#模型预测与提交" class="headerlink" title="模型预测与提交"></a>模型预测与提交</h3><pre><code class="python">def predict(test_loader, model, criterion): model.eval() val_acc = 0.0 test_pred = [] with torch.no_grad(): for i, (input, target) in enumerate(test_loader): input = input.cuda() target = target.cuda() output = model(input) test_pred.append(output.data.cpu().numpy()) return np.vstack(test_pred) pred = Nonefor _ in range(10): if pred is None: pred = predict(test_loader, model, criterion) else: pred += predict(test_loader, model, criterion) ################################################### 如果是windows系统，这里要换成\\，第一个\代表转义 ####################################################submit = pd.DataFrame( { 'uuid': [int(x.split('\\')[-1][:-4]) for x in testPath], 'label': pred.argmax(1)})submit['label'] = submit['label'].map({1:'NC', 0: 'MCI'})submit = submit.sort_values(by='uuid')submit.to_csv('submit2.csv', index=None)</code></pre><h2 id="被提问到的配置问题"><a href="#被提问到的配置问题" class="headerlink" title="被提问到的配置问题"></a>被提问到的配置问题</h2><h3 id="Set-ExecutionPolicy"><a href="#Set-ExecutionPolicy" class="headerlink" title="Set-ExecutionPolicy"></a>Set-ExecutionPolicy</h3><p>在Win11系统中，如果出现如下命令没有反应的情况</p><pre><code class="powershell">Set-ExecutionPolicy -scope CurrentUser Remotesigned</code></pre><p>可以尝试输入</p><pre><code class="powershell">Get-ExecutionPolicy -List</code></pre><p>看一下ExecutionPolicy是否正确修改。</p><h3 id="Jupyter-notebook-运行无反应或-conda-init-失败"><a href="#Jupyter-notebook-运行无反应或-conda-init-失败" class="headerlink" title="Jupyter notebook 运行无反应或 conda init 失败"></a>Jupyter notebook 运行无反应或 conda init 失败</h3><p>因为 Onedrive 可能会开启自动备份或者 Windows 用户名为中文时系统路径，所以可能会因为中文字符原因产生问题。</p><h3 id="GBK-编码问题"><a href="#GBK-编码问题" class="headerlink" title="GBK 编码问题"></a>GBK 编码问题</h3><p>可能是因为中文字符原因，可以设置EncodingOutput</p><pre><code class="powershell">code $Profile</code></pre><p>输入 <em>[System.Console]::OutputEncoding=[System.Text.Encoding]::GetEncoding(65001)</em> 即可，修改为65001 UTF-8编码。</p><h2 id="一些小Trick"><a href="#一些小Trick" class="headerlink" title="一些小Trick"></a>一些小Trick</h2><h3 id="Test-Time-Augment（TTA）"><a href="#Test-Time-Augment（TTA）" class="headerlink" title="Test-Time-Augment（TTA）"></a>Test-Time-Augment（TTA）</h3><pre><code class="python">import ttach as ttamodel = ResNet_3d(Bottleneck, [3, 8, 36, 3], num_classes=2, shortcut_type='B', no_cuda=False, include_top=True)model = model.to('cuda')model.load_state_dict(best_dict)transforms = tta.Compose( [ tta.HorizontalFlip(), A.RandomContrast(p=0.5), A.RandomBrightnessContrast(p=0.5), tta.VerticalFlip(), # tta.Scale(scales=[1, 2, 4]), # tta.Multiply(factors=[0.9, 1, 1.1]), ])model_tta = tta.ClassificationTTAWrapper(model, transforms)def predict(test_loader, model, criterion): model.eval() val_acc = 0.0 test_pred = [] with torch.no_grad(): for i, (input, target) in enumerate(test_loader): input = input.unsqueeze(1).cuda() target = target.cuda() output = model(input) test_pred.append(output.data.cpu().numpy()) return np.vstack(test_pred) import timepred = Nonefor _ in range(10): if pred is None: pred = predict(test_loader, model, criterion) else: pred += predict(test_loader, model, criterion) submit = pd.DataFrame( { 'uuid': [int(x.split('/')[-1].split("\\")[-1][:-4]) for x in testPath], 'label': pred.argmax(1)})submit['label'] = submit['label'].map({1:'NC', 0: 'MCI'})submit = submit.sort_values(by='uuid')submit.to_csv('submit-tta-pre.csv', index=None)</code></pre><p>TTA是一种偏工程的方法，类似于模型集成，增加了鲁棒性，在测试时通过各种数据增广，在得到结果后再综合得到输出。</p><h2 id="附录：A"><a href="#附录：A" class="headerlink" title="附录：A"></a><span id="附录：A">附录：A</span></h2><pre><code class="python">import torchimport torch.nn as nnimport torch.nn.functional as Ffrom torch.autograd import Variableimport mathfrom functools import partialdef conv3x3x3(in_planes, out_planes, stride=1, dilation=1): # 3x3x3 convolution with padding return nn.Conv3d( in_planes, out_planes, kernel_size=3, dilation=dilation, stride=stride, padding=dilation, bias=False)def downsample_basic_block(x, planes, stride, no_cuda=False): out = F.avg_pool3d(x, kernel_size=1, stride=stride) zero_pads = torch.Tensor( out.size(0), planes - out.size(1), out.size(2), out.size(3), out.size(4)).zero_() if not no_cuda: if isinstance(out.data, torch.cuda.FloatTensor): zero_pads = zero_pads.cuda() out = Variable(torch.cat([out.data, zero_pads], dim=1)) return outclass BasicBlock(nn.Module): expansion = 1 def __init__(self, inplanes, planes, stride=1, dilation=1, downsample=None): super(BasicBlock, self).__init__() self.conv1 = conv3x3x3(inplanes, planes, stride=stride, dilation=dilation) self.bn1 = nn.BatchNorm3d(planes) self.relu = nn.ReLU(inplace=True) self.conv2 = conv3x3x3(planes, planes, dilation=dilation) self.bn2 = nn.BatchNorm3d(planes) self.downsample = downsample self.stride = stride self.dilation = dilation def forward(self, x): residual = x out = self.conv1(x) out = self.bn1(out) out = self.relu(out) out = self.conv2(out) out = self.bn2(out) if self.downsample is not None: residual = self.downsample(x) out += residual out = self.relu(out) return outclass Bottleneck(nn.Module): expansion = 4 def __init__(self, inplanes, planes, stride=1, dilation=1, downsample=None): super(Bottleneck, self).__init__() self.conv1 = nn.Conv3d(inplanes, planes, kernel_size=1, bias=False) self.bn1 = nn.BatchNorm3d(planes) self.conv2 = nn.Conv3d( planes, planes, kernel_size=3, stride=stride, dilation=dilation, padding=dilation, bias=False) self.bn2 = nn.BatchNorm3d(planes) self.conv3 = nn.Conv3d(planes, planes * 4, kernel_size=1, bias=False) self.bn3 = nn.BatchNorm3d(planes * 4) self.relu = nn.ReLU(inplace=True) self.downsample = downsample self.stride = stride self.dilation = dilation def forward(self, x): residual = x out = self.conv1(x) out = self.bn1(out) out = self.relu(out) out = self.conv2(out) out = self.bn2(out) out = self.relu(out) out = self.conv3(out) out = self.bn3(out) if self.downsample is not None: residual = self.downsample(x) out += residual out = self.relu(out) return outclass ResNet_3d(nn.Module): def __init__(self, block, layers, num_classes=1000, shortcut_type='B', no_cuda = False, include_top=True): super(ResNet_3d, self).__init__() self.inplanes = 64 self.no_cuda = no_cuda self.include_top = include_top self.conv1 = nn.Conv3d( 1, 64, kernel_size=7, stride=(2, 2, 2), padding=(3, 3, 3), bias=False) self.bn1 = nn.BatchNorm3d(64) self.relu = nn.ReLU(inplace=True) self.maxpool = nn.MaxPool3d(kernel_size=(3, 3, 3), stride=2, padding=1) self.layer1 = self._make_layer(block, 64, layers[0], shortcut_type) self.layer2 = self._make_layer( block, 128, layers[1], shortcut_type, stride=2) self.layer3 = self._make_layer( block, 256, layers[2], shortcut_type, stride=2) self.layer4 = self._make_layer( block, 512, layers[3], shortcut_type, stride=2) if self.include_top: self.avgpool = nn.AdaptiveAvgPool3d((1, 1, 1)) # output size = (1, 1)自适应 self.fc = nn.Linear(512 * block.expansion, num_classes) for m in self.modules(): if isinstance(m, nn.Conv3d): m.weight = nn.init.kaiming_normal(m.weight, mode='fan_out') elif isinstance(m, nn.BatchNorm3d): m.weight.data.fill_(1) m.bias.data.zero_() def _make_layer(self, block, planes, blocks, shortcut_type, stride=1, dilation=1): downsample = None if stride != 1 or self.inplanes != planes * block.expansion: if shortcut_type == 'A': downsample = partial( downsample_basic_block, planes=planes * block.expansion, stride=stride, no_cuda=self.no_cuda) else: downsample = nn.Sequential( nn.Conv3d( self.inplanes, planes * block.expansion, kernel_size=1, stride=stride, bias=False), nn.BatchNorm3d(planes * block.expansion)) layers = [] layers.append(block(self.inplanes, planes, stride=stride, downsample=downsample)) self.inplanes = planes * block.expansion for i in range(1, blocks): layers.append(block(self.inplanes, planes)) return nn.Sequential(*layers) def forward(self, x): x = self.conv1(x) x = self.bn1(x) x = self.relu(x) x = self.maxpool(x) x = self.layer1(x) x = self.layer2(x) x = self.layer3(x) x = self.layer4(x) if self.include_top: x = self.avgpool(x) x = torch.flatten(x, 1) x = self.fc(x) return x</code></pre>]]></content>
<tags>
<tag> Datawhale-Related </tag>
</tags>
</entry>
<entry>
<title>服务器相关</title>
<link href="/2023/06/01/%E6%9C%8D%E5%8A%A1%E5%99%A8%E7%9B%B8%E5%85%B3/"/>
<url>/2023/06/01/%E6%9C%8D%E5%8A%A1%E5%99%A8%E7%9B%B8%E5%85%B3/</url>
<content type="html"><![CDATA[<h2 id="关于配置"><a href="#关于配置" class="headerlink" title="关于配置"></a>关于配置</h2><p>最近经常有迁移服务器的需求，但是又嫌弃每次配置比较麻烦，就把配置存在了本地，使用脚本统一配置。</p><h4 id="PowerShell-Scripting"><a href="#PowerShell-Scripting" class="headerlink" title="PowerShell Scripting"></a>PowerShell Scripting</h4><p>开始的想法是写 PowerShell Scripting，就了解了一下在编写PowerShell脚本，在PowerShell脚本中，可以使用 *param *声明参数，如下：</p><pre><code class="powershell">param($args1, $args2, $args3=default)</code></pre><p>一旦使用这样的方式，所有的参数都必须声明在这参数组里。</p><p>但是问题是 Windows 里面没有 ssh-copy-id 这个函数，只能自己定义：</p><pre><code class="powershell"># copyid.ps1function ssh-copy-id([string]$userAtMachine, $args){ $publicKey = "$ENV:USERPROFILE" + "/.ssh/id_rsa.pub" if (!(Test-Path "$publicKey")){ Write-Error "ERROR: failed to open ID file '$publicKey': No such file" } else { # Write-Output $userAtMachine & cat "$publicKey" | ssh $args $userAtMachine -tt "umask 077; test -d .ssh || mkdir .ssh ; cat >> .ssh/authorized_keys || exit 1" }}ssh-copy-id $args[0]</code></pre><h4 id="Python-脚本"><a href="#Python-脚本" class="headerlink" title="Python 脚本"></a>Python 脚本</h4><h5 id="服务器文件传输"><a href="#服务器文件传输" class="headerlink" title="服务器文件传输"></a>服务器文件传输</h5><p>首先是处理 config 文件以及通过 os.system 将需要的内容传到服务器上</p><pre><code class="python"># settmux.py: python settmux.py abbr addr [port]import sysimport osfrom subprocess import callimport paramiko, timeimport platformos.system('cp /mnt/c/Users/China/.ssh/config /root/.ssh/')print("[INFO]: Moving config in to /root/.ssh/")# Args processingusername = "name"args = sys.argvprint(f"[INFO]: Getting {len(args)} args: {args}")if len(args) == 3: port = "22" _, abbr, addr = argselse: _, abbr, addr, port = args# Get config file and processingwith open("/root/.ssh/config", 'r') as f: OCCUPIED = False context = f.read() context = context.split("\n")with open("/root/.ssh/config", 'a') as f: for line in context: line = line.split("#")[0].strip() if line == "": continue kind = line.split(" ")[0] if kind == 'HostName': config_addr = line.split(" ")[1] if addr == config_addr: print("[INFO]: Already have config") OCCUPIED = True if not OCCUPIED: f.write(f"\n\nHost {abbr}\n HostName {addr}\n Port {port}\n User {username}")# Using platform to check the platformif platform.system().lower() == 'windows': os.system(f".\ssh-copy-id {abbr}")elif platform.system().lower() == 'linux': os.system(f"ssh-copy-id -i /mnt/c/Users/China/.ssh/id_rsa.pub {abbr}")# SCP all the needed filesos.system(f"scp -r ./serverconfig/.tmux/ {username}@{abbr}:~/.tmux")os.system(f"scp ./serverconfig/.bashrc {username}@{abbr}:~/")os.system(f"scp ./serverconfig/.tmux.conf {username}@{abbr}:~/")</code></pre><p>首先处理参数，然后根据 config 文件获得内容，注意，（就目前我的知识水平而言）在打开文件的时候无法同时进入可读之前内容与可添加的模式，虽然存在”a+”的扩展模式，但由于”a”模式下光标将位于文件末尾，因此需要打开两次。</p><p>通过在不同平台下的判断将 <em>id_rsa.pub</em> 上传到服务器之后就可以通过 Host 免密登陆了。</p><h5 id="服务器文件执行"><a href="#服务器文件执行" class="headerlink" title="服务器文件执行"></a>服务器文件执行</h5><p>然后是在本地执行服务器端的配置文件生效的操作。这里我了解到paramiko库，所以使用了一下</p><pre><code class="python">import sysimport osfrom subprocess import callimport paramiko, timefrom collections import defaultdict# Reading configswith open("/root/.ssh/config", 'r') as f: OCCUPIED = False context = f.read() context = context.split("\n")config = defaultdict(dict)for line in context: line = line.split("#")[0].strip() if line == "": continue kind = line.split(" ")[0] if kind == 'Host': temp_name = line.split(" ")[1] elif kind == 'Port': config[temp_name]['port'] = line.split(" ")[1] elif 'jump' in temp_name: continue elif kind == 'HostName': config[temp_name]['hostname'] = line.split(" ")[1] elif kind == 'User': config[temp_name]['user'] = line.split(" ")[1] else: print("[INFO]: Not known args")print(f"[INFO]: CONFIG is")for conf in config: print(conf, config[conf])username = "name"abbr = sys.argv[1]addr = config[abbr]['hostname']port = config[abbr]['port'] if 'port' in config[abbr] else '22'# Using the private key to log in without a passwordpkey='/root/.ssh/id_rsa'key=paramiko.RSAKey.from_private_key_file(pkey)ssh = paramiko.SSHClient()ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())ssh.connect(addr, port, username, pkey=key)# Setting tmux configstdin, stdout, stderr = ssh.exec_command('tmux new -s Tiberius')stdin, stdout, stderr = ssh.exec_command('tmux source-file ~/.tmux.conf')out = stdout.read().decode('utf8')err = stderr.read().decode('utf8')result = [out, err]print(f"[INFO]: result of command {result}") # Downloading Anacondastdin, stdout, stderr = ssh.exec_command('wget https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/Anaconda3-5.3.1-Linux-x86_64.sh')out = stdout.read().decode('utf8')err = stderr.read().decode('utf8')result = [out, err]print(f"[INFO]: result of command {result}") ssh.close()</code></pre><p>通过私钥配置免密登录，配置后设置tmux的操作以及下载anaconda，随后只需要scp将环境传输到新的anaconda上即可。</p><h4 id="显示Git分支"><a href="#显示Git分支" class="headerlink" title="显示Git分支"></a>显示Git分支</h4><p><a href="https://zhuanlan.zhihu.com/p/133291342">如何在Linux下显示当前git分支 - 知乎 (zhihu.com)</a></p>]]></content>
<tags>
<tag> Interesting Things </tag>
</tags>
</entry>
<entry>
<title>交叉熵损失函数和KL散度</title>
<link href="/2023/03/28/%E4%BA%A4%E5%8F%89%E7%86%B5%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0%E5%92%8CKL%E6%95%A3%E5%BA%A6/"/>
<url>/2023/03/28/%E4%BA%A4%E5%8F%89%E7%86%B5%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0%E5%92%8CKL%E6%95%A3%E5%BA%A6/</url>
<content type="html"><![CDATA[<p>当助教的时候遇到一个同学提出了这个问题</p><p><img src="/2023/03/28/%E4%BA%A4%E5%8F%89%E7%86%B5%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0%E5%92%8CKL%E6%95%A3%E5%BA%A6/image-20230601122451507.png" alt="image-20230601122451507"></p><p>关于这个问题，stackexchange上有类似的回答</p><p><img src="/2023/03/28/%E4%BA%A4%E5%8F%89%E7%86%B5%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0%E5%92%8CKL%E6%95%A3%E5%BA%A6/image-20230601122837830.png" alt="image-20230601122837830"></p><p>他解释的是关于工程相关的，意思是说不能直接得到分布，在 minibatch 采样的时候交叉熵更稳定</p><p>但实际上在我的印象中，显著性任务或一些与分布相关的任务例如 t-SNE 同样也会使用KL散度作为损失函数。KL可以用来计算损失，但是计算比较麻烦，一般假设训练集分布等价于真实值分布，而且训练集分布是给定不变的，所以这种情况下，KL散度等价于交叉熵，而且交叉熵计算比较好算，所以就用交叉熵了。</p>]]></content>
<tags>
<tag> Interesting Things </tag>
</tags>
</entry>
<entry>
<title>数字图像处理大作业2——空间滤波</title>
<link href="/2022/12/06/DIPAssignment/"/>
<url>/2022/12/06/DIPAssignment/</url>
<content type="html"><![CDATA[<h2 id="a-噪声定义文件-noise-py"><a href="#a-噪声定义文件-noise-py" class="headerlink" title="a) 噪声定义文件 noise.py"></a>a) 噪声定义文件 noise.py</h2><pre><code class="python">from abc import ABCfrom typing import Tupleimport numpy as npfrom functools import singledispatchmethodimport cv2# from IPython import embed# from numpy.typing import ArrayLike__all__ = [ "NoiseGenerator", "UniformNoiseGenerator", "NormalNoiseGenerator", "GaussianNoiseGenerator", "SaltPepperNoiseGenerator",]class NoiseGenerator: """The base distribution (without uniform noise).""" __slots__ = [] def __init__(self): pass @singledispatchmethod def generate_noise(self): """Generating noise which should be implemented""" raise NotImplementedError def add_noise(self, image: np.ndarray) -> np.ndarray: """Adding noise to image""" image_shape = image.shape return image + self.generate_noise(image_shape=image_shape)</code></pre><ol><li> __all__定义将定义的所有类，从而更好的供后续使用，定义噪声基类与虚函数。</li></ol><ul><li>这里本来想用 python 3.8 提供的 @singledispatchmethod 装饰器，在抽象基类里定义两个虚函数，然后在继承的时候重载函数及调用方式，但是发现如果是继承的类好像不能直接继承（？）因为会报没有这个方法的错，只能把 @singledispatchmethod 写在类内部？不知道能否有人解答我的疑惑。</li></ul><pre><code class="python">class UniformNoiseGenerator(NoiseGenerator, ABC): """The base distribution with uniform noise.""" __slots__ = ['min_val', 'max_val'] def __init__(self, min_val: float = 0, max_val: float = 1): super(UniformNoiseGenerator, self).__init__() self.min_val = min_val self.max_val = max_val def generate_uniform_noise(self, min_val: float = 0, max_val: float = 1, image_shape: Tuple[int, int] = None) -> np.ndarray: """Generating noise :param: image_shape: Image shape of generated noise :return: output_noise: ArrayLike noise or single noise """ if min_val is None and max_val is None: min_val = self.min_val max_val = self.max_val assert (min_val < max_val) # assertion distribution_range = max_val - min_val # getting range of Uniform(0, b - a) moving_factor = distribution_range - max_val # getting move of Uniform(a, b) from Uniform(0, b - a) if image_shape is None: # just generate one noise return np.random.rand() * distribution_range - moving_factor return np.random.rand(*image_shape) * distribution_range - moving_factor # ArrayLike output def add_noise(self, image: np.ndarray) -> np.ndarray: """Adding noise to image""" image_shape = image.shape return image + self.generate_uniform_noise(image_shape=image_shape)</code></pre><ol start="2"><li><p>定义均匀噪声类，使用numpy.random.rand生成0，1之间的均匀噪声，同时利用平移和放缩的到任意范围的均匀噪声。</p><ul><li><p>由于前面定义的的是抽象类，因此后面继承的时候需要同时继承 ABC 抽象类。</p></li><li><p>感觉这里写的有些问题，可能应该写成</p><pre><code class="python">class Noise(abc.ABCMeta): @abc.abstractmethod def read(self): pass</code></pre><p>不太确定。</p></li><li><p>用__slots__减少类初始化和定义时的时间消耗。</p></li><li><p>numpy.random.rand(*image_shape) 时利用了拆包，类似于 *args 的用法</p></li></ul></li></ol><pre><code class="python">class NormalNoiseGenerator(UniformNoiseGenerator, ABC): """The base distribution with additive i.i.d. uniform noise.""" def __init__(self): super(NormalNoiseGenerator, self).__init__() def generate_normal_noise(self, image_shape: Tuple[int, int]) -> np.ndarray: """Generating Normal noise""" uniform_noise_1 = self.generate_uniform_noise(0, 1, image_shape) uniform_noise_2 = self.generate_uniform_noise(0, 1, image_shape) normal_noise = np.cos(2.0 * np.pi * uniform_noise_1) * np.sqrt(-2.0 * np.log(uniform_noise_2)) return normal_noise def add_noise(self, image: np.ndarray) -> np.ndarray: """Adding noise to image""" image_shape = image.shape return image + self.generate_normal_noise(image_shape=image_shape)</code></pre><ol start="3"><li>继承均匀噪声类，使用 Box-Muller 变换得到标准正态分布的噪声。<ul><li> Box-Muller 的核心就是通过极坐标变换将正态分布的指数变换为极坐标的辐角的均匀分布。</li></ul></li></ol><pre><code class="python">class GaussianNoiseGenerator(NormalNoiseGenerator, ABC): """Gaussian distribution with reparameterization trick.""" def __init__(self): super(GaussianNoiseGenerator, self).__init__() def generate_gaussian_noise(self, image_shape: Tuple[int, int], mean: float = 0, var: float = 1) -> np.ndarray: """ Generating Normal noise :param image_shape: Shape of input image :param mean: Mean of gaussian :param var: Variance of gaussian :return: gaussian_noise: Output noise with same size """ normal_noise = self.generate_normal_noise(image_shape) gaussian_noise = normal_noise * np.sqrt(var) + mean return gaussian_noise def add_all_channel_noise(self, image: np.ndarray, mean: float = 0, var: float = 1) -> np.ndarray: """Adding gaussian noise to image with SAME noise in each channel :param image: input image :param mean: Mean of gaussian :param var: Variance of gaussian :return: output_img: Output image with same size adding gaussian noise """ image_shape = image.shape input_img = image / 255 gaussian_noise = np.expand_dims(self.generate_gaussian_noise(image_shape[:2], mean, var), -1) gaussian_noise = gaussian_noise.repeat(3, axis=-1) output_img = input_img + gaussian_noise output_img[input_img > 1] = 1 output_img[input_img < 0] = 0 return output_img * 255 def add_channel_wise_noise(self, image: np.ndarray, mean: float = 0, var: float = 1) -> np.ndarray: """Adding gaussian noise to image with DIFFERENT noise in each channel :param image: input image :param mean: Mean of gaussian :param var: Variance of gaussian :return: output_img: Output image with same size adding channel-wise gaussian noise """ image_shape = image.shape input_img = image / 255 gaussian_noise = self.generate_gaussian_noise(image_shape, mean, var) output_img = input_img + gaussian_noise output_img[input_img > 1] = 1 output_img[input_img < 0] = 0 return output_img * 255</code></pre><ol start="4"><li><p>通过重参数化技巧(re-parameterization trick)，从标准正态分布噪声生成带有均值和方差的高斯噪声分布。同时设置两种加噪声方式：分通道添加或整体添加。</p><ul><li><p><strong>注意</strong>，这里如果想直接使用 cv2 或其他方式直接可视化，有两种方式，</p><ul><li><p>一种是除 255 变换到 [0, 1] 之间，</p></li><li><p>第二种是通过 uint8 的方式转化为整值。</p><p>如果直接可视化的话，会产生过亮的情况，因为会自动判定为 0-1 从而被 clip 成高亮图像。</p></li></ul></li></ul></li></ol><pre><code class="python">class SaltPepperNoiseGenerator(UniformNoiseGenerator, ABC): """Gaussian distribution with additive i.i.d. uniform noise.""" def __init__(self): super(SaltPepperNoiseGenerator, self).__init__() def generate_saltpepper_noise(self, image_shape: Tuple[int, int]) -> np.ndarray: """ :param: prob_1: Probability of "Salt" noise. :param: prob_2: Probability of "Pepper" noise. :param: image_shape: Shape of input image. :return: saltpepper_noise: ArrayLike output salt pepper noise. """ uniform_noise = self.generate_uniform_noise(0, 1, image_shape) saltpepper_noise = uniform_noise.copy() return saltpepper_noise def add_saltpepper_noise(self, prob_1: float, prob_2: float, image: np.ndarray) -> np.ndarray: """Adding noise to image""" output_img = image.copy() image_shape = image.shape saltpepper_noise = self.generate_saltpepper_noise(image_shape[:2]) output_img[saltpepper_noise > 1 - prob_1] = 255 output_img[saltpepper_noise < prob_2] = 0 return output_img</code></pre><ol start="5"><li>继承均匀噪声类，生成均匀噪声，根据均匀噪声的值确定为原始图、胡椒噪声或原始噪声。</li></ol><pre><code class="python">if __name__ == '__main__': uniform_noise_generator = UniformNoiseGenerator() print(uniform_noise_generator.generate_uniform_noise(2, 4, (2, 3)), uniform_noise_generator.generate_uniform_noise()) normal_noise_generator = NormalNoiseGenerator() print(normal_noise_generator.generate_normal_noise((2, 3))) gaussian_noise_generator = GaussianNoiseGenerator() print(gaussian_noise_generator.generate_gaussian_noise((2, 3))) input_image = cv2.imread('./test_image/test3.jpg', 1) print(type(input_image)) cv2.namedWindow('input_image', cv2.WINDOW_AUTOSIZE) cv2.imshow('input_image', input_image) # saltpepper_noise_generator = SaltPepperNoiseGenerator() # tar = saltpepper_noise_generator.add_saltpepper_noise(0.12, 0.1, input_image) # cv2.imshow('saltpepper_noise', tar) gaussian_noise_generator = GaussianNoiseGenerator() tar2 = gaussian_noise_generator.add_all_channel_noise(input_image, 0, 0.05) tar2_channel_wise = gaussian_noise_generator.add_channel_wise_noise(input_image, 0, 0.05) cv2.imshow('gaussian_noise', tar2) cv2.imshow('gaussian_noise_channel', tar2_channel_wise) cv2.waitKey(0)</code></pre><h2 id="b-空间滤波器定义文件-spatial-filter-py"><a href="#b-空间滤波器定义文件-spatial-filter-py" class="headerlink" title="b) 空间滤波器定义文件 spatial_filter.py"></a>b) 空间滤波器定义文件 spatial_filter.py</h2><pre><code class="python">from typing import Tuple, List, Optionalimport numpy as npimport loggingimport sysimport functoolsimport cv2from IPython import embedlevel = logging.DEBUGfmt = "[%(levelname)s] %(asctime)s - %(message)s"logging.basicConfig(level=level, format=fmt)def get_median(noised_image: np.ndarray, x_min: int, x_max: int, y_min: int, y_max: int, c: int) -> float: median_area = noised_image[x_min:x_max + 1, y_min:y_max + 1, c].flatten() sorted_area = sorted(median_area) n = len(median_area) median = sorted_area[n // 2 + 1] if n % 2 else (sorted_area[n // 2] + sorted_area[n // 2 + 1]) / 2 return mediandef get_adaptive_median(noised_image: np.ndarray, x: int, y: int, h: int, w: int, c: int, max_size: int = None) -> float: median_area = noised_image[max(0, x - h):x + h + 1, max(0, y - w):y + w + 1, c].flatten() max_val, min_val = np.max(median_area), np.min(median_area) sorted_area = sorted(median_area) n = len(median_area) median = sorted_area[n // 2 + 1] if n % 2 else (int(sorted_area[n // 2]) + int(sorted_area[n // 2 + 1])) // 2 if min_val < median < max_val: return noised_image[x][y][c] if min_val < noised_image[x][y][c] < max_val else median elif h < max_size and w < max_size: return get_adaptive_median(noised_image, x, y, h + 1, w + 1, c, max_size) else: return noised_image[x][y][c] if min_val < noised_image[x][y][c] < max_val else mediandef get_mean(noised_image: np.ndarray, x_min: int, x_max: int, y_min: int, y_max: int) -> np.ndarray: return np.mean(noised_image[x_min:x_max + 1, y_min:y_max + 1], axis=(0, 1))</code></pre><ol><li>定义静态函数，完成计算规定kernel内的均值，中值与自适应中值。<ul><li>我有一个坏习惯，经常使用 print debug（虽然我现在还是这样），但我尝试使用 log 来输出。</li></ul></li></ol><pre><code class="python">class SpatialFilter: """ This is description which will be showed The Class of Padding Adder """ instance = None # There should be ONLY ONE Spatial Filter once __slots__ = ['noised_image', 'kernel_size'] def __new__(cls, *args): if cls.instance is None: # Once no adder exists cls.instance = super().__new__(cls) return cls.instance def __init__(self, noised_image: np.ndarray, kernel_size: int = 3): self.noised_image = noised_image self.kernel_size = kernel_size def reverse_image(self, image: np.ndarray, reflect_size: int = None, axis: int = 0) -> np.ndarray: """ reverse image in one axis for reflect padding :param: image: Input image for reflection :param: reflect_size: The size of reflect area edge + 1 :param: axis: The axis that will be reflected :return: reversed_img: Reversed image at axis """ if reflect_size is None: reflect_size = self.kernel_size logging.info(f"No kernel size input, automatically use size = {self.kernel_size}") if axis == 0: reflect_area_left = image[reflect_size - 1::-1] reflect_area_right = image[-1 * reflect_size:][::-1] else: reflect_area_left = image[:, reflect_size - 1::-1] reflect_area_right = image[:, -1 * reflect_size:][:, ::-1] reversed_img = np.concatenate([reflect_area_left, image, reflect_area_right], axis=axis) return reversed_img def add_reflect_padding(self, image: np.ndarray, kernel_size: int = None) -> np.ndarray: """ Adding reflect padding :param: image: Input image for reflection :param: kernel_size: The size of reflect kernel :return: reversed_img: Padded image. """ if kernel_size is None: kernel_size = self.kernel_size logging.info(f"No kernel size input, automatically use size = {self.kernel_size}") horizontal_flip = self.reverse_image(image, kernel_size, 0) reversed_img = self.reverse_image(horizontal_flip, kernel_size, 1) return reversed_img def add_zero_padding(self, image: np.ndarray, kernel_size: int = None): """ Adding zero padding :param: image: Input image for padding :param: kernel_size: The size of padding kernel :return: output_img: Padded image. """ if kernel_size is None: kernel_size = self.kernel_size logging.info(f"No kernel size input, automatically use size = {self.kernel_size}") if len(image.shape) > 2: h, w, c = image.shape output_img = np.zeros((h + 2 * kernel_size, w + 2 * kernel_size, c)) else: h, w = image.shape output_img = np.zeros((h + 2 * kernel_size, w + 2 * kernel_size)) output_img[kernel_size:h + kernel_size, kernel_size:w + kernel_size] = image return output_img</code></pre><ol start="2"><li><p>定义Spatial Filter类，同时，由于防止复用时覆盖掉特性，以及在要求的三个函数中不必每次初始化类，因此重载__new__函数，保证只需要使用同一个Spatial Filter，同时实现将图像翻转的功能，供后续对称padding使用。</p></li><li><p>定义两种padding方式，对称填充和零填充。</p></li><li><p>对输入参数做预处理，填充未给出的值。</p><ul><li>（主要是 pycharm 提示，duplicate too long）所以我就写进一个函数了。</li></ul></li></ol><pre><code class="python"> def _preprocess_args(self, noised_image: np.ndarray = None, kernel_size: int = None, padding_func: Optional = None, padding_type: str = None, max_size: int = None): assert (padding_type is None or padding_func is None) assert (kernel_size % 2) if noised_image is None: noised_image = self.noised_image if kernel_size is None: kernel_size = self.kernel_size if max_size is None: max_size = self.kernel_size + 4 if padding_func is not None: padded_image = padding_func(noised_image, (kernel_size - 1) // 2) elif padding_type is not None: padding_func = getattr(sys.modules[__name__].SpatialFilter, "add_%s_padding" % padding_type) padded_image = padding_func(self, noised_image, (kernel_size - 1) // 2) else: padded_image = self.add_zero_padding(noised_image, (kernel_size - 1) // 2) depth = (kernel_size - 1) // 2 c_img = 1 if len(padded_image.shape) > 2: h_img, w_img, c_img = noised_image.shape else: h_img, w_img = noised_image.shape output_img = padded_image.copy() return noised_image, padded_image, kernel_size, padding_func, max_size, depth, h_img, w_img, c_img, output_img</code></pre><ol start="5"><li>实现中值滤波与均值滤波。</li></ol><pre><code class="python"> def mean_filter(self, noised_image: np.ndarray = None, kernel_size: int = None, padding_func: Optional = None, padding_type: str = None): """ Mean filter Support two type of padding: 1. define it yourself and put it with padding_func 2. use the predefined type :param: noised_image: Input image for padding :param: kernel_size: The size of padding kernel :param: padding_func: Padding function defined by yourself :param: padding_type: Type of padding, use in ["reflect", "zero"] :return: output_img: Output image after filter """ noised_image, padded_image, kernel_size, padding_func, _, depth, h_img, w_img, c_img, output_img = \ self._preprocess_args(noised_image, kernel_size, padding_func, padding_type) for h in range(depth, h_img + depth): for w in range(depth, w_img + depth): output_img[h, w, :] = get_mean(padded_image, h - depth, h + depth, w - depth, w + depth) output_img = output_img[depth:h_img + depth, depth:w_img + depth] return np.uint8(output_img) def median_filter(self, noised_image: np.ndarray = None, kernel_size: int = None, padding_func: Optional = None, padding_type: str = None): noised_image, padded_image, kernel_size, padding_func, _, depth, h_img, w_img, c_img, output_img = \ self._preprocess_args(noised_image, kernel_size, padding_func, padding_type) for h in range(depth, h_img + depth): for w in range(depth, w_img + depth): for c in range(c_img): output_img[h, w, c] = get_median(padded_image, h - depth, h + depth, w - depth, w + depth, c) output_img = output_img[depth:h_img + depth, depth:w_img + depth] return np.uint8(output_img) def adaptive_median_filter(self, noised_image: np.ndarray = None, kernel_size: int = None, max_size: int = None, padding_func: Optional = None, padding_type: str = None): noised_image, padded_image, kernel_size, padding_func, _, depth, h_img, w_img, c_img, output_img = \ self._preprocess_args(noised_image, kernel_size, padding_func, padding_type) for h in range(depth, h_img + depth): for w in range(depth, w_img + depth): for c in range(c_img): output_img[h, w, c] = get_adaptive_median(padded_image, h, w, depth, depth, c, max_size) output_img = output_img[depth:h_img + depth, depth:w_img + depth] return np.uint8(output_img)</code></pre><ol start="6"><li>实现自适应中值滤波。</li></ol><h2 id="c-测量指标文件-metric-py"><a href="#c-测量指标文件-metric-py" class="headerlink" title="c) 测量指标文件 metric.py"></a>c) 测量指标文件 metric.py</h2><ol><li>由于并非任务要求，因此这里使用了库函数。<ul><li>本来想用 sys.modules 调用变量，后来才知道类和函数是存在 sys.modules 里的，变量是存在 locals() 或者 globals() 里。</li></ul></li></ol><pre><code class="python"># The implementation of digital image processing assignment 2.# Random Noise and Spatial Filter# Metrics# author: leafy# 2022-12-5# last_modified: 2022-12-5from itertools import productimport mathimport numpy as npfrom skimage.metrics import structural_similarityimport cv2def mean_squared_error(img1: np.ndarray, img2: np.ndarray) -> float: mse = np.mean((img1 / 1.0 - img2 / 1.0) ** 2) return float(mse)def peak_signal_noise_ratio(mse: float) -> float: if mse < 1.0e-10: return 100 return 10 * math.log10(255.0 ** 2 / mse)# Just borrow it from skimage# will implement soon(for assignment request)def compare(img1, img2): mse = mean_squared_error(img1, img2) psnr = peak_signal_noise_ratio(mse) ssim = structural_similarity(img1, img2, multichannel=True) # print('PSNR：{}，SSIM：{}，MSE：{}'.format(psnr, ssim, mse)) return psnr, ssim, mseif __name__ == "__main__": for i in range(1, 5): globals()[f"input_image{i}"] = cv2.imread(f'./result/input_image_{i}.png') # Recording gaussian noise images for i in range(1, 5): for types in ["channel", "full"]: globals()[f"gaussian_img_{types}_{i}"] = cv2.imread(f'./result/gaussian_img_{types}_{i}.png') # recording saltpepper noise images for i in range(1, 5): globals()[f"sp_img_full_{i}"] = cv2.imread(f'./result/sp_img_full_{i}.png', ) for i in range(1, 5): globals()[f"low_sp_img_full_{i}"] = cv2.imread(f'./result/low_sp_img_full_{i}.png') for i in range(1, 5): for f_type, noise in product(["mean", "median", "median_adaptive"], ["gaussian", "sp", "low_sp"]): # for noise in ["gaussian", "sp", "low_sp"]: for types in ["channel", "full"]: if "sp" in noise: types = "full" for pad in ["_reflect", ""]: print(f"Reading images in ./result/{f_type}_{noise}_{types}{pad}_{i}.png") globals()[f"{f_type}_{noise}_{types}{pad}_{i}"] = cv2.imread( f'./result/{f_type}_{noise}_{types}{pad}_{i}.png') for f_type, noise in product(["mean", "median", "median_adaptive", "no_filter"], ["gaussian", "sp", "low_sp"]): for types in ["channel", "full"]: if "sp" in noise and types == "channel": continue for pad in ["_reflect", ""]: cnt_psnr, cnt_ssim, cnt_mse = 0, 0, 0 if f_type == "no_filter": filt = "" img = "img_" else: filt = f_type + "_" img = "" if f_type == "no_filter" and pad != "": continue for i in range(1, 5): cur_psnr, cur_ssim, cur_mse = compare(globals()[f"input_image{i}"], globals()[f"{filt}{noise}_{img}{types}{pad}_{i}"]) for metric in ["psnr", "ssim", "mse"]: globals()[f"cnt_{metric}"] += globals()[f"cur_{metric}"] cnt_psnr, cnt_ssim, cnt_mse = cnt_psnr / 4, cnt_ssim / 4, cnt_mse / 4 print(f"Difference between {filt}{noise}_{img}{types}{pad}.png and input is " f"\n{cnt_psnr} \n{cnt_ssim} \n{cnt_mse}")</code></pre><ol start="2"><li>完成了所有图像的指标测试。</li></ol><h2 id="d-主文件（五个函数及主函数调用）"><a href="#d-主文件（五个函数及主函数调用）" class="headerlink" title="d) 主文件（五个函数及主函数调用）"></a>d) 主文件（五个函数及主函数调用）</h2><pre><code class="python">import osfrom typing import Tupleimport cv2import numpy as npimport sysfrom itertools import productfrom noise import GaussianNoiseGenerator, SaltPepperNoiseGeneratorfrom spatial_filter import SpatialFilterfrom metric import comparefrom IPython import embeddef generate_gaussian_noise(input_img: np.ndarray, mean: float, var: float) -> Tuple[np.ndarray, np.ndarray]: """ :param: input_img: Input image. :param: mean: Mean of gaussian noise. :param: var: Variance of gaussian noise. :return: output_image: Output image with gaussian noise added. output_img, output_img_channel_wise """ inner_gaussian_noise_generator = GaussianNoiseGenerator() output_img = inner_gaussian_noise_generator.add_all_channel_noise(input_img, mean, var) output_img_channel_wise = inner_gaussian_noise_generator.add_channel_wise_noise(input_img, mean, var) return output_img, output_img_channel_wisedef generate_saltpepper_noise(prob_1: float, prob_2: float, input_img: np.ndarray) -> np.ndarray: """ :param: prob_1: prob_1 of saltpepper noise. :param: prob_2: prob_2 of saltpepper noise. :param: input_img: Input image. :return: output_image: Output image with saltpepper noise added. """ saltpepper_noise_generator = SaltPepperNoiseGenerator() output_img = saltpepper_noise_generator.add_saltpepper_noise(prob_1, prob_2, input_img) return output_imgdef mean_filter(noised_img: np.ndarray, kernel_size: int = None) -> Tuple[np.ndarray, np.ndarray]: """ :param: input_img: Input image. :param: kernel_size: kernel size of filter. :return: output_image: Output filtered image. output_img_zero, output_img_reflect """ sp_filter = SpatialFilter(noised_img) output_img_zero = sp_filter.mean_filter(noised_img, kernel_size=kernel_size, padding_type="zero") output_img_reflect = sp_filter.mean_filter(noised_img, kernel_size=kernel_size, padding_type="reflect") return output_img_zero, output_img_reflectdef median_filter(noised_img: np.ndarray, kernel_size: int = None) -> Tuple[np.ndarray, np.ndarray]: """ :param: input_img: Input image. :param: kernel_size: kernel size of filter. :return: output_image: Output filtered image. output_img_zero, output_img_reflect """ sp_filter = SpatialFilter(noised_img) output_img_zero = sp_filter.median_filter(noised_img, kernel_size=kernel_size, padding_type="zero") output_img_reflect = sp_filter.median_filter(noised_img, kernel_size=kernel_size, padding_type="reflect") return output_img_zero, output_img_reflectdef median_adaptive_filter(noised_img: np.ndarray, kernel_size: int = None, max_size: int = None) -> Tuple[np.ndarray, np.ndarray]: """ :param: input_img: Input image. :param: kernel_size: kernel size of filter. :param: max_size: Max size of adaptive filter. :return: output_image: Output filtered image. output_img_zero, output_img_reflect """ sp_filter = SpatialFilter(noised_img) output_img_zero = sp_filter.adaptive_median_filter(noised_img, kernel_size=kernel_size, max_size=max_size, padding_type="zero") output_img_reflect = sp_filter.adaptive_median_filter(noised_img, kernel_size=kernel_size, max_size=max_size, padding_type="reflect") return output_img_zero, output_img_reflect</code></pre><ol><li><p>两种噪声的生成。</p></li><li><p>三种滤波结果，同时包括对称填充和零填充。</p></li><li><p>主函数——图像读入与加噪声，添加了高斯噪声、高概率椒盐噪声和低概率椒盐噪声。</p></li></ol><pre><code class="python">def main(test_dir: str = "./test_image"): # Reading images input_images = os.listdir(test_dir) for idx, path in input_images: locals()[f"input_image{idx}"] = cv2.imread(os.path.join(test_dir, path), 1) # Recording input images for i in range(1, 5): cv2.imwrite(f'./result/input_image_{i}.png', locals()[f"input_image{i}"]) # Recording gaussian noise images for i in range(1, 5): locals()[f"gaussian_img_full_{i}"], locals()[f"gaussian_img_channel_{i}"] = \ generate_gaussian_noise(locals()[f"input_image{i}"], 0, 0.05) for types in ["channel", "full"]: cv2.imwrite(f'./result/gaussian_img_{types}_{i}.png', locals()[f"gaussian_img_{types}_{i}"]) # recording saltpepper noise images for i in range(1, 5): locals()[f"sp_img_full_{i}"] = generate_saltpepper_noise(0.1, 0.1, locals()[f"input_image{i}"]) cv2.imwrite(f'./result/sp_img_full_{i}.png', locals()[f"sp_img_full_{i}"]) for i in range(1, 5): locals()[f"low_sp_img_full_{i}"] = generate_saltpepper_noise(0.01, 0.01, locals()[f"input_image{i}"]) cv2.imwrite(f'./result/low_sp_img_full_{i}.png', locals()[f"low_sp_img_full_{i}"]) # Filtering and recording the output images print(locals().keys()) for i in range(1, 5): for f_type, noise in product(["mean", "median", "median_adaptive"], ["gaussian", "sp", "low_sp"]): # for noise in ["gaussian", "sp", "low_sp"]: for types in ["channel", "full"]: if "sp" in noise and types == "channel": continue print(f"Generating images {f_type}_{noise}_{types}_{i}") if f_type == "median_adaptive": locals()[f"{f_type}_{noise}_{types}_{i}"], locals()[f"{f_type}_{noise}_{types}_reflect_{i}"] = \ getattr(sys.modules[__name__], f"{f_type}_filter")(locals()[f"{noise}_img_{types}_{i}"], kernel_size=3, max_size=7) else: locals()[f"{f_type}_{noise}_{types}_{i}"], locals()[f"{f_type}_{noise}_{types}_reflect_{i}"] = \ getattr(sys.modules[__name__], f"{f_type}_filter")(locals()[f"{noise}_img_{types}_{i}"], kernel_size=3) for pad in ["_reflect", ""]: print(f"Saving images in ./result/{f_type}_{noise}_{types}{pad}_{i}.png") cv2.imwrite(f'./result/{f_type}_{noise}_{types}{pad}_{i}.png', locals()[f"{f_type}_{noise}_{types}{pad}_{i}"]) for f_type, noise in product(["mean", "median", "median_adaptive", "no_filter"], ["gaussian", "sp", "low_sp"]): for types in ["channel", "full"]: if "sp" in noise and types == "channel": continue for pad in ["_reflect", ""]: cnt_psnr, cnt_ssim, cnt_mse = 0, 0, 0 if f_type == "no_filter": filt = "" img = "img_" else: filt = f_type + "_" img = "" if f_type == "no_filter" and pad != "": continue for i in range(1, 5): cur_psnr, cur_ssim, cur_mse = compare(globals()[f"input_image{i}"], globals()[f"{filt}{noise}_{img}{types}{pad}_{i}"]) for metric in ["psnr", "ssim", "mse"]: globals()[f"cnt_{metric}"] += globals()[f"cur_{metric}"] cnt_psnr, cnt_ssim, cnt_mse = cnt_psnr / 4, cnt_ssim / 4, cnt_mse / 4 print(f"Difference between {filt}{noise}_{img}{types}{pad}.png and input is " f"\n{cnt_psnr} \n{cnt_ssim} \n{cnt_mse}")if __name__ == '__main__': # input_image3 = cv2.imread('test3.jpg', 1) # a, b = generate_gaussian_noise(input_image3, 0, 0.05) # cv2.imshow('gaussian_noise', a) # cv2.imshow('gaussian_noise_channel', b) # cv2.waitKey(0) main()</code></pre><ol start="4"><li>主函数——滤波与指标计算。</li></ol>]]></content>
<tags>
<tag> Assignment </tag>
</tags>
</entry>
<entry>
<title>类的方法与属性</title>
<link href="/2022/11/29/Method/"/>
<url>/2022/11/29/Method/</url>
<content type="html"><![CDATA[<p>故事的开始来自今天发现本月5号忘记了leetcode每日一题打卡。为了保证本人的第一次月打卡徽章，我打算补打这个卡。很快我发现这是一道并不困难的“困难”题，因此我决定写一个很Coooooooool的函数，题目是这样的：</p><blockquote><p>给你一个以字符串形式表述的 布尔表达式（boolean） expression，返回该式的运算结果。</p><p>有效的表达式需遵循以下约定：</p><ul><li>“t”，运算结果为 True</li><li>“f”，运算结果为 False</li><li>“!(expr)”，运算过程为对内部表达式 expr 进行逻辑 非的运算（NOT）</li><li>“&(expr1,expr2,…)”，运算过程为对 2 个或以上内部表达式 expr1, expr2, … 进行逻辑 与的运算（AND）</li><li>“|(expr1,expr2,…)”，运算过程为对 2 个或以上内部表达式 expr1, expr2, … 进行逻辑 或的运算（OR）</li></ul><p>来源：力扣（LeetCode）<br>链接：<a href="https://leetcode.cn/problems/parsing-a-boolean-expression">https://leetcode.cn/problems/parsing-a-boolean-expression</a><br>著作权归领扣网络所有。商业转载请联系官方授权，非商业转载请注明出处。</p></blockquote><p>这个是我写的代码。</p><pre><code class="python">from collections import dequeDEBUG = Falsedef boolean(x: str) -> bool or str: if x not in "tf": return x return True if x == "t" else Falseclass BoolExpression: def __repr__(self): return f"A Bool Expression parser" def cal_symbol_result(self, x1: bool, grammar_symbol: str, x2: bool = None) -> bool: assert (grammar_symbol in "!|&") # calculate in three types of char if DEBUG: print(f"checking {x1}, {x2}") if grammar_symbol == "!": return not x1 elif grammar_symbol == "&": return x1 & x2 if x2 != None else x1 else: return x1 | x2 if x2 != None else x1 def parse_bool_expr(self, expression: str) -> bool: # print(expression) grammar_stack = deque() char_stack = deque() for expression_char in expression: # iter through expression char if DEBUG: print(f"showing {expression_char}, {grammar_stack}, {char_stack}") if expression_char in "!&|": # if char is grammar sign grammar_stack.append(expression_char) elif expression_char == ")": # or end of the expression cur_grammar_char = grammar_stack.pop() cur_variable_char = char_stack.pop() cur_result = None while cur_variable_char != "(": # iter through all expression need to be calculate if DEBUG: print(f"popping {cur_variable_char}, {cur_grammar_char}") cur_result = self.cal_symbol_result(cur_variable_char, cur_grammar_char, cur_result) if DEBUG: print(f"getting {cur_result}") cur_variable_char = char_stack.pop() if DEBUG: print(f"popping {cur_variable_char}, remaining {char_stack}") char_stack.append(cur_result) elif expression_char != ",": # or some other chars char_stack.append(boolean(expression_char)) # print(char_stack) assert (len(char_stack) == 1) return char_stack[0]</code></pre><p>其实我一开始是想把boolean这个函数写进类里的，但是为了省去写self.boolean的调用，我绞尽脑汁回想起了曾经见过的一个装饰器@staticmethod。我清晰地记得似乎这个方法可以不用写self，但很快，事实告诉我并非如此，我依然要使用self.boolean去调用。显然我贫瘠的记忆并不足以支持我继续干想，动手试一试的想法很快涌入了我的脑海。那就去试一试学一学吧，我想。</p><h2 id="类的方法"><a href="#类的方法" class="headerlink" title="类的方法"></a>类的方法</h2><p>因此，今天的内容，就从这里开始。首先我们要提到的是装饰器带来的类方法。凭借我的记忆和一些简单的查找，我找到了三种方法：@property, @classmethod, @staticmethod</p><pre><code class="python">class Solution: """ This is description which will be showed The class of solution """ def add(self) -> int: return self.a + self.b @property def add1(self) -> int: return self.a + self.b @classmethod def add2_1(cls): print("add2_1") @classmethod def add2(cls) -> str: print(f"self.instance is {cls.__name__}") print(cls.a) return cls.__name__ @staticmethod def add3(a: int) -> int: return a + 1 def __repr__(self): return f"a solution class with {self.a}, {self.b}"</code></pre><h3 id="property"><a href="#property" class="headerlink" title="@property"></a>@property</h3><p><strong>@property</strong> 将函数包装为类的属性，这个属性是直接获取属性，使用和类的普通属性完全相同。</p><pre><code class="python">In [7]: Solution.add1Out[7]: <property at 0x2812ab37778>In [8]: s.add1Out[8]: 5</code></pre><p><strong>个人觉得主要用途是用于在类中创建只读的属性，即将私有属性安全化</strong>，例如本例中，如果我们不希望实例对象可以修改 _add1 属性，则可以在不定义 setattr 的情况下将 add1 变为只读属性。</p><h3 id="classmethod"><a href="#classmethod" class="headerlink" title="@classmethod"></a>@classmethod</h3><p><strong>@classmethod</strong> 将函数变为类的方法，该方法只能调用类的属性，而不能调用实例的属性，因此无法完成 self.a + self.b 的操作，只能使用类的属性</p><pre><code class="python">In [1]: s.add2()self.instance is Solution<member 'a' of 'Solution' objects>Out[1]: 'Solution'</code></pre><h3 id="staticmethod"><a href="#staticmethod" class="headerlink" title="@staticmethod"></a>@staticmethod</h3><p><strong>@staticmethod</strong> 确实可以少写self，很遗憾，是在定义的时候，是在定义 add3 时，无需写self，因为它不只是实例的方法，它是类的方法，我们可以随意调用。</p><pre><code class="python">In [3]: s.add3(1)Out[3]: 2In [4]: Solution.add3(1)Out[4]: 2</code></pre><h3 id="Summary"><a href="#Summary" class="headerlink" title="Summary"></a>Summary</h3><p>简单看一下这三个函数的区别：</p><pre><code class="python">In [13]: Solution.addOut[13]: <function __main__.Solution.add(self) -> int>In [14]: Solution.add1Out[14]: <property at 0x1cf238e7728>In [15]: Solution.add2Out[15]: <bound method Solution.add2 of <class '__main__.Solution'>>In [16]: Solution.add3Out[16]: <function __main__.Solution.add3(a: int) -> int>----------------------------------------------------------------------------------In [9]: s.addOut[9]: <bound method Solution.add of a solution class with 2, 3>In [10]: s.add1Out[10]: 5In [11]: s.add2Out[11]: <bound method Solution.add2 of <class '__main__.Solution'>>In [12]: s.add3Out[12]: <function __main__.Solution.add3(a: int) -> int></code></pre><h2 id="类的属性"><a href="#类的属性" class="headerlink" title="类的属性"></a>类的属性</h2><p>这一部分我想说的是类的属性，其实也没什么特别难的，可以通过 dir 自己查到类的方法</p><pre><code class="python">In [17]: dir(s)Out[17]: ['__annotations__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_add1', 'a', 'add', 'add1', 'add2', 'add2_1', 'add3', 'b', 'instance']</code></pre><p>这里还是上面这个类，我们添加一些方法的重载。</p><pre><code class="python">class Solution: """ This is description which will be showed The class of solution """ __name__: str = "Sub solution" __slots__ = ["a", "b", "_add1"] instance = 0 def __new__(cls, *args, **kwargs): if not cls.instance: cls.instance = super().__new__(cls) return cls.instance def __init__(self, a: int, b: int): self.a = a self.b = b self._add1 = 5 def _in_slots(self, attr) -> bool: for cls in type(self).__mro__: if attr in getattr(cls, '__slots__', []): return True return False def __setattr__(self, attribute: 'attribute', item: int): if attribute == "_add1": print("PLEASE DO NOT CHANGE add1") return if attribute == "a" and item > 100: print(f"self.a do not support assign over 100") if self._in_slots(attribute): object.__setattr__(self, attribute, item) return # print("SET ERROR, try for subclass") raise AttributeError(f"Something has gone wrong. {attribute} may not be assigned") def add(self) -> int: return self.a + self.b @property def add1(self) -> int: return self.a + self.b @classmethod def add2_1(cls): print("add2_1") @classmethod def add2(cls) -> str: print(f"self.instance is {cls.__name__}") print(cls.a) return cls.__name__ @staticmethod def add3(a: int) -> int: return a + 1 def __repr__(self): return f"a solution class with {self.a}, {self.b}"</code></pre><p>如果我们这里实现了 __slot__ 那么类将不需要 __dict__ 从而可以节省内存，并且固定类的属性。当事先知道类的属性的时候，可以__slots__来节省内存以及获得更快的属性读取。<strong>注意不应当把防止创造__slots__之外的新属性作为使用__slots__的原因，可以使用decorators以及getters，setters来实现属性控制，使类只接受实现的属性。</strong></p><pre><code class="python">In [1]: s.a = 101self.a do not support assign over 100In [2]: s._add1 = 10PLEASE DO NOT CHANGE add1In [3]: s.a = 5In [4]: sOut[4]: a solution class with 5, 3In [5]: s.a = 4In [6]: sOut[6]: a solution class with 4, 3</code></pre><h3 id="annotations"><a href="#annotations" class="headerlink" title="__annotations__"></a>__annotations__</h3><p><strong>这个属性可以看到类的注释，当然函数也有这个属性。</strong></p><pre><code class="python">In [18]: s.__annotations__Out[18]: {'__name__': str}In [19]: s.__init__.__annotations__Out[19]: {'a': int, 'b': int}</code></pre><h3 id="setattr"><a href="#setattr" class="headerlink" title="__setattr__"></a>__setattr__</h3><p><strong>实现了该方法后，在对属性赋值时会调用，因此可以在函数里限制属性赋值。</strong><br>可以禁止不允许的赋值。</p><h3 id="getattribute"><a href="#getattribute" class="headerlink" title="__getattribute__"></a>__getattribute__</h3><p><strong>实现该方法可以对类的方法调用进行额外的包装，例如对类的函数调用实现监听</strong>，详情见：<a href="https://blog.csdn.net/weixin_36179862/article/details/102829018">python给类的所有方法加上装饰器</a>。</p><blockquote><pre><code class="python">class ChiseAipFace(AipFace): #AipFace为百度的人脸识别http封装的类。 def __getattribute__(self, item): """建议对item进行一下判断，不要全局增加""" ret = super().__getattribute__(item) if type( ret) == "<class 'method'>": # 类里面的成员分为三种，method（类方法和实例方法），function（实例方法），int,str...（变量成员），具体需要的时候还是通过type进行判断或者直接通过item来判断 def res(*args, **kwargs): retu = None t = 0 for i in range(10): if i > 0: t = time.time() retu = ret(*args, **kwargs) if retu['error_code'] == 18:#主要解决瞬时并发超限的问题，通过随机值将超限的并发随机在之后的一段时间里面进行接口访问。 time.sleep(random.random() * i * 5) else: if t: logger.warning("接口访问延时18:" + str(time.time() - t) + ",name:" + item) return retu logger.error("接口失败18:" + item) return retu return res else: return ret</code></pre></blockquote><h3 id="new-与-init"><a href="#new-与-init" class="headerlink" title="__new__ 与 __init__"></a>__new__ 与 __init__</h3><p>new是对类的实例化，因此__new__接收 cls 对象，并实例化后返回实例化的对象；而__init__是给实例化的对象赋予属性，新词接受的是 self 实例属性。</p><p><strong>这里的__new__主要是用于唯一的对象，即一个类同一时刻只能实例化一个对象（例如音乐播放器或垃圾箱）</strong></p><h2 id="Fun-Fact"><a href="#Fun-Fact" class="headerlink" title="Fun Fact"></a>Fun Fact</h2><pre><code class="python">In [2]: type(s)Out[2]: __main__.SolutionIn [3]: type(type(s))Out[3]: typeIn [4]: type.__mro__Out[4]: (type, object)In [5]: object.__mro__Out[5]: (object,)</code></pre><p>type 是创建类（包括它本身）的类，而 object 是创建实例的类，为继承的类提供了内置方法。<br>但是 __mro__ 显示 type 继承了 type 和 object 而 object 继承了 object。</p>]]></content>
</entry>
<entry>
<title>写代码时记录的小技巧</title>
<link href="/2022/11/25/tricks-in-writing/"/>
<url>/2022/11/25/tricks-in-writing/</url>
<content type="html"><![CDATA[<!-- wait for replacing: \tricks-in-writing\ -> --><p>本文是记录自己写代码时候遇到的一些感觉比较有用/有意思的技巧，怕自己忘了记录一下。如果有想要分享的思路/纠正改进的地方欢迎交流。</p><h3 id="如何输出结果-Verbose-Trick"><a href="#如何输出结果-Verbose-Trick" class="headerlink" title="如何输出结果 (Verbose Trick)"></a>如何输出结果 (Verbose Trick)</h3><h4 id="有关类的输出"><a href="#有关类的输出" class="headerlink" title="有关类的输出"></a>有关类的输出</h4><ol><li><p>如果想在<strong>模型的 forward 中输出相关变量</strong>，但又不希望反复输出影响观看效果，可以在类的私有变量中设定 self.verbose = True，并在首次输出后修改 Self.verbose = False 即可。</p></li><li><p>如果想<strong>输出类的信息</strong>，可以通过实现类的__str__ 方法或 __repr__方法，其中</p><ol><li>__repr__ 所返回的字符串应该准确、无歧义，并且尽可能表达出如何 用代码创建出这个被打印的对象。</li><li>而__str__ 在 str() 函数被使用，或是在用 print 函数打印一个对象的时候才被调用的，并且它返回的字符串对终端用户更友好。 <strong>__str__ 方法未实现时，会转而调用__repr__方法，因此如果只想实现一个时，可以实现__repr__方法。</strong><br><img src="/2022/11/25/tricks-in-writing/%E5%B1%8F%E5%B9%95%E6%88%AA%E5%9B%BE_20221125_133515.png" alt="\_\_repr__"></li></ol></li><li><p>当我们想让模型<strong>输出多个东西</strong>时，很容易存在记混顺序或是别的什么情况，这时我们可以使用Collection中的具名元组namedtuple来存储记录的属性. </p><pre><code class="python"> DiscOutSplit = collections.namedtuple( "DiscOutSplit", ["d_real", "d_fake", "d_real_bit", "d_fake_bit"]) EntropyInfo = collections.namedtuple( "EntropyInfo", "noisy_out quantized_out nbpp qbpp", )</code></pre><p> 代码中展示了两种定义方法，通过列表或空格分隔的str</p></li></ol><h4 id="格式化输出"><a href="#格式化输出" class="headerlink" title="格式化输出"></a>格式化输出</h4><ol><li><p>Python 3.6 支持了简明的 f 字符串输出，可以支持多行表达式，表达式运算，哈希取值等多种运算。</p><ol><li>直接使用</li></ol><pre><code class="python">print("The {add_a} + {add_b} = {add_out}")</code></pre><ol start="2"><li>表达式运算</li></ol><pre><code class="python">bags = 3apples_in_bag = 12print(f'The result of add is {add_a + add_b}')</code></pre><ol start="3"><li>哈希取值</li></ol><pre><code class="python">user = {'name': 'leafy', 'occupation': 'student'}print(f"{user['name']} is a {user['occupation']}")</code></pre><ol start="4"><li>多行表达式</li></ol><pre><code class="python">name = 'leafy'occupation = 'student'msg = ( f'Name: {name}\n' f'Occupation: {occupation}')print(msg)</code></pre><ol start="5"><li>函数调用</li></ol><pre><code class="python"> print("The {add_a} + {add_b} = {add(add_a, add_b)}")</code></pre><ol start="6"><li>实例对象调用（对象所在类必须定义了表达方法 __repr__ 或 __str__）</li></ol><pre><code class="python"> class User: def __init__(self, name, occupation): self.name = name self.occupation = occupation def __repr__(self): return f"{self.name} is a {self.occupation}" u = User('leafy', 'student') print(f'{u}')</code></pre><ol start="7"><li>格式化字符串</li></ol><pre><code class="python">for x in range(0, 20, 2): print(f'{x:02} {x*x:3} {x*x*x:4}')</code></pre><p> <img src="/2022/11/25/tricks-in-writing/%E5%B1%8F%E5%B9%95%E6%88%AA%E5%9B%BE_20221125_142605.png" alt="geshihua"></p><pre><code class="python"># hexadecimalprint(f"{a:x}")# octalprint(f"{a:o}")# scientificprint(f"{a:e}")</code></pre><ol start="8"><li>右对齐</li></ol><pre><code class="python">s1 = 'Fnatic's2 = 'TPA's3 = 'SKT T1's4 = 'SSW'print(f'{s1:>6}')print(f'{s2:>6}')print(f'{s3:>6}')print(f'{s4:>6}')</code></pre><h4 id="切片拆包"><a href="#切片拆包" class="headerlink" title="切片拆包"></a>切片拆包</h4><p>例子来自 Fluent Python (“Fluent Python by Luciano Ramalho (O’Reilly). Copyright 2015 Luciano Ramalho, 978-1-491- 94600-8.”)<br><img src="/2022/11/25/tricks-in-writing/%E5%B1%8F%E5%B9%95%E6%88%AA%E5%9B%BE_20221125_143146.png" alt="slice"></p></li></ol><p>目前就想到这么多，想到了会继续补充</p>]]></content>
</entry>
<entry>
<title>如何在网页中添加自己的图标</title>
<link href="/2022/11/22/%E5%A6%82%E4%BD%95%E5%9C%A8%E7%BD%91%E9%A1%B5%E4%B8%AD%E6%B7%BB%E5%8A%A0%E8%87%AA%E5%B7%B1%E7%9A%84%E5%9B%BE%E6%A0%87/"/>
<url>/2022/11/22/%E5%A6%82%E4%BD%95%E5%9C%A8%E7%BD%91%E9%A1%B5%E4%B8%AD%E6%B7%BB%E5%8A%A0%E8%87%AA%E5%B7%B1%E7%9A%84%E5%9B%BE%E6%A0%87/</url>
<content type="html"><![CDATA[<p>  因为在制作自己的个人主页的时候遇到了Hexo主题没有提供对应图标的问题，就查看了一下Hexo主题是如何添加图标的。发现主要的方法是直接修改fonts文件夹下的iconfont.svg文件。<a href="https://blog.csdn.net/csdn_inside/article/details/89159803">修改yilia theme下的font文件</a>，这个也刚好是同学blog使用的主题，学习了一下这个主题，明白了这类图标是通过字体库的形式导入的。<br></p><p><img src="/2022/11/22/%E5%A6%82%E4%BD%95%E5%9C%A8%E7%BD%91%E9%A1%B5%E4%B8%AD%E6%B7%BB%E5%8A%A0%E8%87%AA%E5%B7%B1%E7%9A%84%E5%9B%BE%E6%A0%87/Claudia_github.jpg" alt="Claudia_github"></p><p>  但我使用的Claudia-theme并没有类似的source-src文件夹，因此需要找到这些图标是如何导入的。分析widget-sns.pug文件，发现他通过i.iconfont.icon-xxxx导入。自然的，我们可以去对应文件中寻找iconfont是如何被定义的。<br></p><p><img src="/2022/11/22/%E5%A6%82%E4%BD%95%E5%9C%A8%E7%BD%91%E9%A1%B5%E4%B8%AD%E6%B7%BB%E5%8A%A0%E8%87%AA%E5%B7%B1%E7%9A%84%E5%9B%BE%E6%A0%87/Claudia_pug.jpg" alt="Claudia_pug"></p><p>  在对应的scss文件中寻找到了结果，iconfont通过font-class的css文件在线链接导入，并将其中的sns-container设定为.iconfont。而链接中的alicdn说明导入的图标素材来自阿里的iconfont网站图标库。</p><p>  因此在官方文档中学习了一下<a href="https://www.iconfont.cn/help/detail?spm=a313x.7781069.1998910419.15&helptype=code">如何使用这个图标库</a>。并通过学习的<a href="#jump">Font-class方式</a>，将在线css文件import导入，使用了自制的图标库。<br></p><h2 id="icon-font使用方法"><a href="#icon-font使用方法" class="headerlink" title="icon-font使用方法"></a>icon-font使用方法</h2><p>根据官方文档，iconfont一共有三种使用方式: Unicode 引入，Font-class引入和symbol引用。其中symbol引用是官方最推荐的方法。但是由于主题作者使用的是第二种，因此我延续了第二种使用方式。<br></p><h3 id="1-Unicode引入"><a href="#1-Unicode引入" class="headerlink" title="1. Unicode引入"></a>1. Unicode引入</h3><p>第一步：拷贝项目下面生成的font-face<br></p><blockquote><p>@font-face {font-family: ‘iconfont’;<br><br>  src: url(‘iconfont.eot’);<br><br>  src: url(‘iconfont.eot?#iefix’) format(‘embedded-opentype’),<br><br>  url(‘iconfont.woff’) format(‘woff’),<br><br>  url(‘iconfont.ttf’) format(‘truetype’),<br><br>  url(‘iconfont.svg#iconfont’) format(‘svg’);<br><br>}<br></p></blockquote><p>第二步：定义使用iconfont的样式<br></p><blockquote><p>.iconfont{<br><br>  font-family:”iconfont” !important;<br><br>  font-size:16px;font-style:normal;<br><br>  -webkit-font-smoothing: antialiased;<br><br>  -webkit-text-stroke-width: 0.2px;<br><br>  -moz-osx-font-smoothing: grayscale;}<br></p></blockquote><p>第三步：挑选相应图标并获取字体编码，应用于页面</p><blockquote><p><code><i class="iconfont">&#x33;</i></code></p></blockquote><h3 id="2-Font-class引入"><a href="#2-Font-class引入" class="headerlink" title="2. Font-class引入"></a><span id="jump">2. Font-class引入</span></h3><p>第一步：拷贝项目下面生成的fontclass代码：</p><blockquote><p>//at.alicdn.com/t/font_8d5l8fzk5b87iudi.css</p></blockquote><p>第二步：挑选相应图标并获取类名，应用于页面：</p><blockquote><p><code><i class="iconfont icon-xxx"></i></code></p></blockquote><p>本文中就使用了这一步的方法，但是在base.scss文件中@import了对应的css链接，从而在iconfont类下使用<br><img src="/2022/11/22/%E5%A6%82%E4%BD%95%E5%9C%A8%E7%BD%91%E9%A1%B5%E4%B8%AD%E6%B7%BB%E5%8A%A0%E8%87%AA%E5%B7%B1%E7%9A%84%E5%9B%BE%E6%A0%87/alicdn.jpg" alt="alicdn"></p><h3 id="3-symbol引用"><a href="#3-symbol引用" class="headerlink" title="3. symbol引用"></a>3. symbol引用</h3><blockquote><p>这是一种全新的使用方式，应该说这才是未来的主流，也是平台目前推荐的用法。相关介绍可以参考这篇文章 这种用法其实是做了一个svg的集> 合，与上面两种相比具有如下特点：</p><ul><li>支持多色图标了，不再受单色限制。</li><li>通过一些技巧，支持像字体那样，通过font-size,color来调整样式。</li><li>兼容性较差，支持 ie9+,及现代浏览器。</li><li>浏览器渲染svg的性能一般，还不如png。</li></ul></blockquote><p>第一步：拷贝项目下面生成的symbol代码：<br></p><blockquote><p>//at.alicdn.com/t/font_8d5l8fzk5b87iudi.js</p></blockquote><p>第二步：加入通用css代码（引入一次就行）：</p><pre><code><style type="text/css"> .icon { width: 1em; height: 1em; vertical-align: -0.15em; fill: currentColor; overflow: hidden; }</style></code></pre><p>第三步：挑选相应图标并获取类名，应用于页面：</p><pre><code><svg class="icon" aria-hidden="true"> <use xlink:href="#icon-xxx"></use></svg></code></pre>]]></content>
<tags>
<tag> 博客构建 </tag>
</tags>
</entry>
<entry>
<title>Test Bug</title>
<link href="/2022/11/19/test-bug/"/>
<url>/2022/11/19/test-bug/</url>
<content type="html"><![CDATA[<h1 id="Here-is-Title"><a href="#Here-is-Title" class="headerlink" title="Here is Title"></a>Here is Title</h1><h2 id="And-Here-is-sub-title"><a href="#And-Here-is-sub-title" class="headerlink" title="And Here is sub title"></a>And Here is sub title</h2><pre><code class="python"># want to and some code hereprint("Hello Python")</code></pre><h4 id="To-do-List"><a href="#To-do-List" class="headerlink" title="To-do List"></a>To-do List</h4><ul><li><input disabled="" type="checkbox"> work</li><li><input checked="" disabled="" type="checkbox"> play</li></ul>]]></content>
<tags>
<tag> 博客构建 </tag>
</tags>
</entry>
<entry>
<title>Hello World</title>
<link href="/2022/11/18/hello-world/"/>
<url>/2022/11/18/hello-world/</url>
<content type="html"><![CDATA[<p>Welcome to <a href="https://hexo.io/">Hexo</a>! This is your very first post. Check <a href="https://hexo.io/docs/">documentation</a> for more info. If you get any problems when using Hexo, you can find the answer in <a href="https://hexo.io/docs/troubleshooting.html">troubleshooting</a> or you can ask me on <a href="https://github.com/hexojs/hexo/issues">GitHub</a>.</p><h2 id="Quick-Start"><a href="#Quick-Start" class="headerlink" title="Quick Start"></a>Quick Start</h2><h3 id="Create-a-new-post"><a href="#Create-a-new-post" class="headerlink" title="Create a new post"></a>Create a new post</h3><pre><code class="bash">$ hexo new "My New Post"</code></pre><p>More info: <a href="https://hexo.io/docs/writing.html">Writing</a></p><h3 id="Run-server"><a href="#Run-server" class="headerlink" title="Run server"></a>Run server</h3><pre><code class="bash">$ hexo server</code></pre><p>More info: <a href="https://hexo.io/docs/server.html">Server</a></p><h3 id="Generate-static-files"><a href="#Generate-static-files" class="headerlink" title="Generate static files"></a>Generate static files</h3><pre><code class="bash">$ hexo generate</code></pre><p>More info: <a href="https://hexo.io/docs/generating.html">Generating</a></p><h3 id="Deploy-to-remote-sites"><a href="#Deploy-to-remote-sites" class="headerlink" title="Deploy to remote sites"></a>Deploy to remote sites</h3><pre><code class="bash">$ hexo deploy</code></pre><p>More info: <a href="https://hexo.io/docs/one-command-deployment.html">Deployment</a></p>]]></content>
<tags>
<tag> 博客构建 </tag>
</tags>
</entry>
</search>