-
Notifications
You must be signed in to change notification settings - Fork 0
/
search.xml
4620 lines (4313 loc) · 663 KB
/
search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>八大排序算法的 Python 实现</title>
<url>/2018/03/14/8-sort-algorithm/</url>
<content><![CDATA[<!-- toc -->
<h2 id="插入排序"><a href="#插入排序" class="headerlink" title="插入排序"></a>插入排序</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">insert_sort</span>(<span class="params">lists</span>):</span></span><br><span class="line"> count = <span class="built_in">len</span>(lists)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, count):</span><br><span class="line"> key = lists[i]</span><br><span class="line"> j = i - <span class="number">1</span></span><br><span class="line"> <span class="keyword">while</span> j >= <span class="number">0</span>:</span><br><span class="line"> <span class="keyword">if</span> lists[j] > key:</span><br><span class="line"> lists[j + <span class="number">1</span>] = lists[j]</span><br><span class="line"> lists[j] = key</span><br><span class="line"> j -= <span class="number">1</span></span><br><span class="line"> <span class="keyword">return</span> lists</span><br></pre></td></tr></table></figure>
<a id="more"></a>
<p><img src="/images/Insertion_sort_animation.gif" class="lazyload" data-srcset="/images/Insertion_sort_animation.gif" srcset="data:image/png;base64,666" alt="insert_sort"></p>
<h2 id="希尔排序"><a href="#希尔排序" class="headerlink" title="希尔排序"></a>希尔排序</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">shell_sort</span>(<span class="params">lists</span>):</span></span><br><span class="line"> count = <span class="built_in">len</span>(lists)</span><br><span class="line"> step = <span class="number">2</span></span><br><span class="line"> group = count // step</span><br><span class="line"> <span class="keyword">while</span> group > <span class="number">0</span>:</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">0</span>, group):</span><br><span class="line"> j = i + group</span><br><span class="line"> <span class="keyword">while</span> j < count:</span><br><span class="line"> k = j - group</span><br><span class="line"> key = lists[j]</span><br><span class="line"> <span class="keyword">while</span> k >= <span class="number">0</span>:</span><br><span class="line"> <span class="keyword">if</span> lists[k] > key:</span><br><span class="line"> lists[k + group] = lists[k]</span><br><span class="line"> lists[k] = key</span><br><span class="line"> k -= group</span><br><span class="line"> j += group</span><br><span class="line"> group //= step</span><br><span class="line"> <span class="keyword">return</span> lists</span><br></pre></td></tr></table></figure>
<p><img src="/images/Sorting_shellsort_anim.gif" class="lazyload" data-srcset="/images/Sorting_shellsort_anim.gif" srcset="data:image/png;base64,666" alt="shell_sort"></p>
<h2 id="冒泡排序"><a href="#冒泡排序" class="headerlink" title="冒泡排序"></a>冒泡排序</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">bubble_sort</span>(<span class="params">lists</span>):</span></span><br><span class="line"> count = <span class="built_in">len</span>(lists)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">0</span>, count):</span><br><span class="line"> <span class="keyword">for</span> j <span class="keyword">in</span> <span class="built_in">range</span>(i + <span class="number">1</span>, count):</span><br><span class="line"> <span class="keyword">if</span> lists[i] > lists[j]:</span><br><span class="line"> lists[i], lists[j] = lists[j], lists[i]</span><br><span class="line"> <span class="keyword">return</span> lists</span><br></pre></td></tr></table></figure>
<p><img src="/images/Bubble_sort_animation.gif" class="lazyload" data-srcset="/images/Bubble_sort_animation.gif" srcset="data:image/png;base64,666" alt="bubble_sort"></p>
<h2 id="快速排序"><a href="#快速排序" class="headerlink" title="快速排序"></a>快速排序</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">qs = <span class="keyword">lambda</span> xs: ((<span class="built_in">len</span>(xs) <= <span class="number">1</span> <span class="keyword">and</span> [xs]) <span class="keyword">or</span> [qs([x <span class="keyword">for</span> x <span class="keyword">in</span> xs[<span class="number">1</span>:] <span class="keyword">if</span> x < xs[</span><br><span class="line"> <span class="number">0</span>]]) + [xs[<span class="number">0</span>]] + qs([x <span class="keyword">for</span> x <span class="keyword">in</span> xs[<span class="number">1</span>:] <span class="keyword">if</span> x >= xs[<span class="number">0</span>]])])[<span class="number">0</span>]</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">quick_sort</span>(<span class="params">lists, left=<span class="number">0</span>, right=<span class="number">9</span></span>):</span></span><br><span class="line"> <span class="keyword">if</span> left >= right:</span><br><span class="line"> <span class="keyword">return</span> lists</span><br><span class="line"> key = lists[left]</span><br><span class="line"> low = left</span><br><span class="line"> high = right</span><br><span class="line"> <span class="keyword">while</span> left < right:</span><br><span class="line"> <span class="keyword">while</span> left < right <span class="keyword">and</span> lists[right] >= key:</span><br><span class="line"> right -= <span class="number">1</span></span><br><span class="line"> lists[left] = lists[right]</span><br><span class="line"> <span class="keyword">while</span> left < right <span class="keyword">and</span> lists[left] <= key:</span><br><span class="line"> left += <span class="number">1</span></span><br><span class="line"> lists[right] = lists[left]</span><br><span class="line"> lists[right] = key</span><br><span class="line"> quick_sort(lists, low, left - <span class="number">1</span>)</span><br><span class="line"> quick_sort(lists, left + <span class="number">1</span>, high)</span><br><span class="line"> <span class="keyword">return</span> lists</span><br></pre></td></tr></table></figure>
<p><img src="/images/Sorting_quicksort_anim.gif" class="lazyload" data-srcset="/images/Sorting_quicksort_anim.gif" srcset="data:image/png;base64,666" alt="quick_sort"></p>
<h2 id="选择排序"><a href="#选择排序" class="headerlink" title="选择排序"></a>选择排序</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">select_sort</span>(<span class="params">lists</span>):</span></span><br><span class="line"> count = <span class="built_in">len</span>(lists)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">0</span>, count):</span><br><span class="line"> <span class="built_in">min</span> = i</span><br><span class="line"> <span class="keyword">for</span> j <span class="keyword">in</span> <span class="built_in">range</span>(i + <span class="number">1</span>, count):</span><br><span class="line"> <span class="keyword">if</span> lists[<span class="built_in">min</span>] > lists[j]:</span><br><span class="line"> <span class="built_in">min</span> = j</span><br><span class="line"> lists[<span class="built_in">min</span>], lists[i] = lists[i], lists[<span class="built_in">min</span>]</span><br><span class="line"> <span class="keyword">return</span> lists</span><br></pre></td></tr></table></figure>
<p><img src="/images/Selection_sort_animation.gif" class="lazyload" data-srcset="/images/Selection_sort_animation.gif" srcset="data:image/png;base64,666" alt="select_sort"></p>
<h2 id="堆排序"><a href="#堆排序" class="headerlink" title="堆排序"></a>堆排序</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">adjust_heap</span>(<span class="params">lists, i, size</span>):</span></span><br><span class="line"> lchild = <span class="number">2</span> * i + <span class="number">1</span></span><br><span class="line"> rchild = <span class="number">2</span> * i + <span class="number">2</span></span><br><span class="line"> <span class="built_in">max</span> = i</span><br><span class="line"> <span class="keyword">if</span> i < size // <span class="number">2</span>:</span><br><span class="line"> <span class="keyword">if</span> lchild < size <span class="keyword">and</span> lists[lchild] > lists[<span class="built_in">max</span>]:</span><br><span class="line"> <span class="built_in">max</span> = lchild</span><br><span class="line"> <span class="keyword">if</span> rchild < size <span class="keyword">and</span> lists[rchild] > lists[<span class="built_in">max</span>]:</span><br><span class="line"> <span class="built_in">max</span> = rchild</span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">max</span> != i:</span><br><span class="line"> lists[<span class="built_in">max</span>], lists[i] = lists[i], lists[<span class="built_in">max</span>]</span><br><span class="line"> adjust_heap(lists, <span class="built_in">max</span>, size)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">build_heap</span>(<span class="params">lists, size</span>):</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">0</span>, (size // <span class="number">2</span>))[::-<span class="number">1</span>]:</span><br><span class="line"> adjust_heap(lists, i, size)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">heap_sort</span>(<span class="params">lists</span>):</span></span><br><span class="line"> size = <span class="built_in">len</span>(lists)</span><br><span class="line"> build_heap(lists, size)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">0</span>, size)[::-<span class="number">1</span>]:</span><br><span class="line"> lists[<span class="number">0</span>], lists[i] = lists[i], lists[<span class="number">0</span>]</span><br><span class="line"> adjust_heap(lists, <span class="number">0</span>, i)</span><br><span class="line"> <span class="keyword">return</span> lists</span><br></pre></td></tr></table></figure>
<p><img src="/images/Sorting_heapsort_anim.gif" class="lazyload" data-srcset="/images/Sorting_heapsort_anim.gif" srcset="data:image/png;base64,666" alt="heap_sort"></p>
<h2 id="归并排序"><a href="#归并排序" class="headerlink" title="归并排序"></a>归并排序</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">merge</span>(<span class="params">left, right</span>):</span></span><br><span class="line"> i, j = <span class="number">0</span>, <span class="number">0</span></span><br><span class="line"> result = []</span><br><span class="line"> <span class="keyword">while</span> i < <span class="built_in">len</span>(left) <span class="keyword">and</span> j < <span class="built_in">len</span>(right):</span><br><span class="line"> <span class="keyword">if</span> left[i] <= right[j]:</span><br><span class="line"> result.append(left[i])</span><br><span class="line"> i += <span class="number">1</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> result.append(right[j])</span><br><span class="line"> j += <span class="number">1</span></span><br><span class="line"> result += left[i:]</span><br><span class="line"> result += right[j:]</span><br><span class="line"> <span class="keyword">return</span> result</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">merge_sort</span>(<span class="params">lists</span>):</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(lists) <= <span class="number">1</span>:</span><br><span class="line"> <span class="keyword">return</span> lists</span><br><span class="line"> num = <span class="built_in">len</span>(lists) // <span class="number">2</span></span><br><span class="line"> left = merge_sort(lists[:num])</span><br><span class="line"> right = merge_sort(lists[num:])</span><br><span class="line"> <span class="keyword">return</span> merge(left, right)</span><br></pre></td></tr></table></figure>
<p><img src="/images/Merge_sort_animation2.gif" class="lazyload" data-srcset="/images/Merge_sort_animation2.gif" srcset="data:image/png;base64,666" alt="merge_sort"></p>
<h2 id="基数排序"><a href="#基数排序" class="headerlink" title="基数排序"></a>基数排序</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> math</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">radix_sort</span>(<span class="params">lists, radix=<span class="number">10</span></span>):</span></span><br><span class="line"> k = <span class="built_in">int</span>(math.ceil(math.log(<span class="built_in">max</span>(lists), radix)))</span><br><span class="line"> bucket = [[] <span class="keyword">for</span> _ <span class="keyword">in</span> <span class="built_in">range</span>(radix)]</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, k + <span class="number">1</span>):</span><br><span class="line"> <span class="keyword">for</span> j <span class="keyword">in</span> lists:</span><br><span class="line"> bucket[j // (radix**(i - <span class="number">1</span>)) % radix].append(j)</span><br><span class="line"> <span class="keyword">del</span> lists[:]</span><br><span class="line"> <span class="keyword">for</span> z <span class="keyword">in</span> bucket:</span><br><span class="line"> lists += z</span><br><span class="line"> <span class="keyword">del</span> z[:]</span><br><span class="line"> <span class="keyword">return</span> lists</span><br></pre></td></tr></table></figure>
<p><img src="/images/radix_sort.gif" class="lazyload" data-srcset="/images/radix_sort.gif" srcset="data:image/png;base64,666" alt="merge_sort"></p>
<h2 id="测试"><a href="#测试" class="headerlink" title="测试"></a>测试</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> random</span><br><span class="line">original_test = <span class="built_in">list</span>(random.randint(<span class="number">1</span>, <span class="number">100</span>) <span class="keyword">for</span> _ <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>))</span><br><span class="line">print(<span class="string">"原始列表： %s"</span> % original_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 插入排序</span></span><br><span class="line">insert_test = insert_sort(original_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 希尔排序</span></span><br><span class="line">shell_test = shell_sort(original_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 冒泡排序</span></span><br><span class="line">bubble_test = bubble_sort(original_test)</span><br><span class="line"></span><br><span class="line">快速排序</span><br><span class="line">quick_test = quick_sort(original_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 直接选择排序</span></span><br><span class="line">select_test = select_sort(original_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 堆排序</span></span><br><span class="line">heap_test = heap_sort(original_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 归并排序</span></span><br><span class="line">merge_test = merge_sort(original_test)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 基数排序</span></span><br><span class="line">radix_test = radix_sort(original_test)</span><br><span class="line"></span><br><span class="line">print(<span class="string">"插入排序： %s"</span> % insert_test)</span><br><span class="line">print(<span class="string">"希尔排序： %s"</span> % shell_test)</span><br><span class="line">print(<span class="string">"冒泡排序： %s"</span> % bubble_test)</span><br><span class="line">print(<span class="string">"快速排序： %s"</span> % quick_test)</span><br><span class="line">print(<span class="string">"直接选择排序：%s"</span> % select_test)</span><br><span class="line">print(<span class="string">"堆排序： %s"</span> % heap_test)</span><br><span class="line">print(<span class="string">"归并排序： %s"</span> % merge_test)</span><br><span class="line">print(<span class="string">"基数排序： %s"</span> % radix_test)</span><br><span class="line">print(<span class="string">"快速排序： %s"</span> % qs(original_test))</span><br></pre></td></tr></table></figure>
<p>本文来自：<a href="http://python.jobbole.com/82270">八大排序算法的 Python 实现</a></p>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
</tags>
</entry>
<entry>
<title>Asyncio 笔记</title>
<url>/2018/11/11/asyncio-tutorial/</url>
<content><![CDATA[<!-- toc -->
<!-- markdownlint-disable MD033 -->
<blockquote><p>并发是指一次处理多件事。
并行是指一次做多件事。
二者不同，但是有联系。
一个关于结构，一个关于执行。
并发用于制定方案，用来解决可能（但未必）并行的问题。</p>
<p align="right">——Rob Pike
Go 语言的创造者之一</p></blockquote>
<a id="more"></a>
<hr>
<h2 id="异步版-hello-world"><a href="#异步版-hello-world" class="headerlink" title="异步版 hello-world"></a>异步版 <code>hello-world</code></h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span>():</span></span><br><span class="line"> print(<span class="string">'hello'</span>)</span><br><span class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">.1</span>)</span><br><span class="line"> print(<span class="string">'world'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># python3.7 提供</span></span><br><span class="line">asyncio.run(main())</span><br><span class="line"></span><br><span class="line"><span class="comment"># main 函数是个协程，直接运行不会执行操作</span></span><br><span class="line"><span class="comment"># main() --> <coroutine object main at 0x109be6d48></span></span><br></pre></td></tr></table></figure>
<h2 id="运行协程的三种方式"><a href="#运行协程的三种方式" class="headerlink" title="运行协程的三种方式"></a>运行协程的三种方式</h2><ul>
<li><code>asyncio.run()</code></li>
<li>使用 <code>await</code> 关键字</li>
</ul>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">say_after</span>(<span class="params">delay, what</span>):</span></span><br><span class="line"> <span class="keyword">await</span> asyncio.sleep(delay)</span><br><span class="line"> print(what)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span>():</span></span><br><span class="line"> print(<span class="string">f"started at <span class="subst">{time.strftime(<span class="string">'%X'</span>)}</span>"</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">await</span> say_after(<span class="number">1</span>, <span class="string">'hello'</span>)</span><br><span class="line"> <span class="keyword">await</span> say_after(<span class="number">2</span>, <span class="string">'world'</span>)</span><br><span class="line"> print(<span class="string">f"finished at <span class="subst">{time.strftime(<span class="string">'%X'</span>)}</span>"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">asyncio.run(main())</span><br></pre></td></tr></table></figure>
<ul>
<li>使用 <code>asyncio.create_task()</code> 创建一个 <code>Task</code> 对象</li>
</ul>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 直接对上面的 main 函数进行修改</span></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span>():</span></span><br><span class="line"> t1 = asyncio.create_task(say_after(<span class="number">1</span>, <span class="string">'hello'</span>))</span><br><span class="line"> t2 = asyncio.create_task(say_after(<span class="number">2</span>, <span class="string">'world'</span>))</span><br><span class="line"></span><br><span class="line"> print(<span class="string">f"started at <span class="subst">{time.strftime(<span class="string">'%X'</span>)}</span>"</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">await</span> t1</span><br><span class="line"> <span class="keyword">await</span> t2</span><br><span class="line"></span><br><span class="line"> print(<span class="string">f"finished at <span class="subst">{time.strftime(<span class="string">'%X'</span>)}</span>"</span>)</span><br></pre></td></tr></table></figure>
<h2 id="Awaitable-对象"><a href="#Awaitable-对象" class="headerlink" title="Awaitable 对象"></a>Awaitable 对象</h2><p><code>awaitable</code> 对象是指可以在 <code>await</code> 表达式中使用的对象。</p>
<p><code>coroutines</code>, <code>Tasks</code> 和 <code>Futures</code> 是 <code>awaitable</code> 对象。</p>
<ul>
<li>协程 <code>coroutines</code></li>
</ul>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">nested</span>():</span></span><br><span class="line"> <span class="keyword">return</span> <span class="number">42</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span>():</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># 这中方式不会执行 nested 函数</span></span><br><span class="line"> nested()</span><br><span class="line"></span><br><span class="line"> <span class="comment"># await</span></span><br><span class="line"> print(<span class="keyword">await</span> nested())</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">asyncio.run(main())</span><br></pre></td></tr></table></figure>
<ul>
<li>任务 <code>Tasks</code></li>
</ul>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">nested</span>():</span></span><br><span class="line"> <span class="keyword">return</span> <span class="number">42</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span>():</span></span><br><span class="line"> t = asyncio.create_task(nested())</span><br><span class="line"> <span class="keyword">await</span> t</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">asyncio.run(main())</span><br></pre></td></tr></table></figure>
<ul>
<li>期物 <code>Futures</code></li>
</ul>
<h2 id="并发执行-Tasks"><a href="#并发执行-Tasks" class="headerlink" title="并发执行 Tasks"></a>并发执行 Tasks</h2><p>使用 <code>asyncio.gather</code> 并发执行 <code>Tasks</code></p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">factorial</span>(<span class="params">name, number</span>):</span></span><br><span class="line"> f = <span class="number">1</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">2</span>, number + <span class="number">1</span>):</span><br><span class="line"> print(<span class="string">f"Task <span class="subst">{name}</span>: Compute factorial(<span class="subst">{i}</span>)..."</span>)</span><br><span class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">1</span>)</span><br><span class="line"> f *= i</span><br><span class="line"> print(<span class="string">f"Task <span class="subst">{name}</span>: factorial(<span class="subst">{number}</span>) = <span class="subst">{f}</span>"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span>():</span></span><br><span class="line"> <span class="keyword">await</span> asyncio.gather(</span><br><span class="line"> factorial(<span class="string">'A'</span>, <span class="number">2</span>),</span><br><span class="line"> factorial(<span class="string">'B'</span>, <span class="number">3</span>),</span><br><span class="line"> factorial(<span class="string">'C'</span>, <span class="number">4</span>),</span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">asyncio.run(main())</span><br></pre></td></tr></table></figure>
<h2 id="线程和协程的对比"><a href="#线程和协程的对比" class="headerlink" title="线程和协程的对比"></a>线程和协程的对比</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 线程版以动画形式显示文本旋转指针</span></span><br><span class="line"><span class="keyword">import</span> threading</span><br><span class="line"><span class="keyword">import</span> itertools</span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Signal</span>:</span></span><br><span class="line"> go = <span class="literal">True</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">spin</span>(<span class="params">msg, signal</span>):</span></span><br><span class="line"> write, flush = sys.stdout.write, sys.stdout.flush</span><br><span class="line"> <span class="keyword">for</span> char <span class="keyword">in</span> itertools.cycle(<span class="string">'|/-\\'</span>):</span><br><span class="line"> status = char + <span class="string">' '</span> + msg</span><br><span class="line"> write(status)</span><br><span class="line"> flush()</span><br><span class="line"> write(<span class="string">'\x08'</span> * <span class="built_in">len</span>(status))</span><br><span class="line"> time.sleep(<span class="number">.1</span>)</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> signal.go:</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"> write(<span class="string">' '</span> * <span class="built_in">len</span>(status) + <span class="string">'\x08'</span> * <span class="built_in">len</span>(status))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">slow_funtion</span>():</span></span><br><span class="line"> time.sleep(<span class="number">3</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="number">42</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">supervisor</span>():</span></span><br><span class="line"> signal = Signal()</span><br><span class="line"> spinner = threading.Thread(</span><br><span class="line"> target=spin,</span><br><span class="line"> args=(<span class="string">'thinking!'</span>, signal))</span><br><span class="line"> print(<span class="string">'spinner object: '</span>, spinner)</span><br><span class="line"> spinner.start()</span><br><span class="line"> result = slow_funtion()</span><br><span class="line"> signal.go = <span class="literal">False</span></span><br><span class="line"> spinner.join()</span><br><span class="line"> <span class="keyword">return</span> result</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span>():</span></span><br><span class="line"> result = supervisor()</span><br><span class="line"> print(<span class="string">'Answer: '</span>, result)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> main()</span><br></pre></td></tr></table></figure>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 协程版以动画形式显示文本旋转指针</span></span><br><span class="line"><span class="keyword">import</span> asyncio</span><br><span class="line"><span class="keyword">import</span> itertools</span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">spin</span>(<span class="params">msg</span>):</span></span><br><span class="line"> write, flush = sys.stdout.write, sys.stdout.flush</span><br><span class="line"> <span class="keyword">for</span> char <span class="keyword">in</span> itertools.cycle(<span class="string">'|/-\\'</span>):</span><br><span class="line"> status = char + <span class="string">' '</span> + msg</span><br><span class="line"> write(status)</span><br><span class="line"> flush()</span><br><span class="line"> write(<span class="string">'\x08'</span> * <span class="built_in">len</span>(status))</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">.1</span>)</span><br><span class="line"> <span class="keyword">except</span> asyncio.CancelledError:</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"> write(<span class="string">' '</span> * <span class="built_in">len</span>(status) + <span class="string">'\x08'</span> * <span class="built_in">len</span>(status))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">slow_funtion</span>():</span></span><br><span class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">3</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="number">42</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">supervisor</span>():</span></span><br><span class="line"> spinner = asyncio.create_task(spin(<span class="string">'thinking!'</span>))</span><br><span class="line"> print(<span class="string">'spinner object: '</span>, spinner)</span><br><span class="line"> result = <span class="keyword">await</span> slow_funtion()</span><br><span class="line"> spinner.cancel()</span><br><span class="line"> <span class="keyword">return</span> result</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 一般执行方式</span></span><br><span class="line">loop = asyncio.get_event_loop()</span><br><span class="line">result = loop.run_until_complete(supervisor())</span><br><span class="line">loop.close()</span><br><span class="line">print(<span class="string">'Answer: '</span>, result)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># python3.7 执行方式</span></span><br><span class="line">asyncio.run(supervisor())</span><br></pre></td></tr></table></figure>
<ul>
<li>Task 对象像是实现协作式多任务的库（如 <code>gevent</code>）中的绿色线程（<code>green thread</code>）。</li>
<li>Task 对象用于驱动协程，Thread 对象用于调用可调用对象。</li>
<li>Task 对象不由自己手动实例化，而是由 <code>asyncio.create_task</code> 方法获取。</li>
<li>获取的 Task 对象已经排定了运行时间，而 Thread 实例需要调用 <code>start</code> 方法运行。</li>
<li>异步版 <code>slow_funtion</code> 是协程，由 <code>await</code> （就是 <code>yield from</code>）驱动。</li>
<li>终止线程需要借助外部变量 <code>go</code>,终止 Task 可以调用 <code>Task.cancel()</code> 实例方法，在协程内部抛出 <code>CancelledError</code> 异常，协程内部也可以捕获这个异常，处理终止请求。</li>
</ul>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
</tags>
</entry>
<entry>
<title>CentOS 离线安装 MySQL</title>
<url>/2018/09/06/centos-install-mysql/</url>
<content><![CDATA[<p>记录一下 CentOS 离线安装 MySQL 并配置多实列主从复制的过程，<br>如果有旧版 Mariadb，先卸载旧版 Mariadb。</p>
<a id="more"></a>
<!-- toc -->
<h2 id="卸载系统自带的-Mariadb"><a href="#卸载系统自带的-Mariadb" class="headerlink" title="卸载系统自带的 Mariadb"></a>卸载系统自带的 Mariadb</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">rpm -qa|grep mariadb <span class="comment"># 查询出已安装的 mariadb</span></span><br><span class="line">rpm -e --nodeps filename <span class="comment"># 上面列出的所有文件</span></span><br><span class="line">rm -f /etc/my.cnf <span class="comment"># 删除配置文件</span></span><br></pre></td></tr></table></figure>
<h2 id="创建-mysql-用户组"><a href="#创建-mysql-用户组" class="headerlink" title="创建 mysql 用户组"></a>创建 mysql 用户组</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">groupadd mysql</span><br><span class="line">useradd -g mysql mysql</span><br></pre></td></tr></table></figure>
<h2 id="下载安装包"><a href="#下载安装包" class="headerlink" title="下载安装包"></a>下载安装包</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.12-linux-glibc2.12-x86_64.tar.xz</span><br></pre></td></tr></table></figure>
<h2 id="解压文件到目录-usr-local"><a href="#解压文件到目录-usr-local" class="headerlink" title="解压文件到目录 /usr/local"></a>解压文件到目录 /usr/local</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">cp mysql-8.0.12-linux-glibc2.12-x86_64.tar.xz /usr/<span class="built_in">local</span>/mysql-8.0.12-linux-glibc2.12-x86_64.tar.xz</span><br><span class="line"><span class="built_in">cd</span> /usr/<span class="built_in">local</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># xz 结尾的是经过两层压缩的压缩包</span></span><br><span class="line"><span class="comment"># 先解压 xz</span></span><br><span class="line">xz -d your_file_name.tar.xz</span><br><span class="line"></span><br><span class="line"><span class="comment"># 再解压 tar</span></span><br><span class="line">tar -xvf your_file_name.tar</span><br><span class="line"></span><br><span class="line"><span class="comment"># 或者直接解压</span></span><br><span class="line">tar xvJf your_file_name.tar.xz</span><br><span class="line"></span><br><span class="line">tar xvJf mysql-8.0.12-linux-glibc2.12-x86_64.tar.xz</span><br><span class="line">mv mysql-8.0.12-linux-glibc2.12-x86_64 mysql</span><br></pre></td></tr></table></figure>
<h2 id="配置-etc-my-cnf"><a href="#配置-etc-my-cnf" class="headerlink" title="配置 /etc/my.cnf"></a>配置 /etc/my.cnf</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">[mysqld_multi]</span><br><span class="line">mysqld = /usr/<span class="built_in">local</span>/mysql/bin/mysqld_safe</span><br><span class="line">mysqladmin = /usr/<span class="built_in">local</span>/mysql/bin/mysqladmin</span><br><span class="line">user = root</span><br><span class="line"></span><br><span class="line"><span class="comment"># The MySQL server</span></span><br><span class="line"><span class="comment">###############################################################################</span></span><br><span class="line">[mysqld1]</span><br><span class="line">port =33306</span><br><span class="line">datadir =/data/mysqldata/data1</span><br><span class="line">socket =/var/lib/mysql/mysql1.sock</span><br><span class="line">pid-file =/var/lib/mysql/mysql1.pid</span><br><span class="line">user =mysql</span><br><span class="line">server_id =33306</span><br><span class="line">log_bin =/data/mysqldata/data1/mysql-bin</span><br><span class="line">log_bin_index =/data/mysqldata/data1/mysql-bin.index</span><br><span class="line">expire_logs_days =10 <span class="comment"># 按需设置过期时间，表示保留最近10天的日志</span></span><br><span class="line"><span class="comment">###############################################################################</span></span><br><span class="line">[mysqld2]</span><br><span class="line">port =33307</span><br><span class="line">datadir =/data/mysqldata/data2</span><br><span class="line">socket =/var/lib/mysql/mysql2.sock</span><br><span class="line">pid-file =/var/lib/mysql/mysql2.pid</span><br><span class="line">user =mysql</span><br><span class="line">server_id =33307</span><br><span class="line">log_bin =/data/mysqldata/data2/mysql-bin</span><br><span class="line">log_bin_index =/data/mysqldata/data2/mysql-bin.index</span><br><span class="line">expire_logs_days =10 <span class="comment"># 按需设置过期时间，表示保留最近10天的日志</span></span><br><span class="line"><span class="comment">###############################################################################</span></span><br></pre></td></tr></table></figure>
<h2 id="初始化数据目录"><a href="#初始化数据目录" class="headerlink" title="初始化数据目录"></a>初始化数据目录</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">mkdir /var/lib/mysql</span><br><span class="line">chown -R mysql:mysql /var/lib/mysql</span><br><span class="line"></span><br><span class="line">mkdir /data/mysqldata</span><br><span class="line">chown -R mysql:mysql /data/mysqldata</span><br><span class="line"></span><br><span class="line">mkdir /data/mysqldata/data1</span><br><span class="line">chown -R mysql:mysql /data/mysqldata/data1</span><br><span class="line"></span><br><span class="line">mkdir /data/mysqldata/data2</span><br><span class="line">chown -R mysql:mysql /data/mysqldata/data2</span><br><span class="line"></span><br><span class="line"><span class="built_in">cd</span> /usr/<span class="built_in">local</span>/mysql/bin</span><br><span class="line"></span><br><span class="line"><span class="comment"># 这种初始化数据库目录的方式过时了</span></span><br><span class="line">./mysql_install_db --basedir=/usr/<span class="built_in">local</span>/mysql --datadir=/data/mysqldata/data1 --user=mysql</span><br><span class="line">./mysql_install_db --basedir=/usr/<span class="built_in">local</span>/mysql --datadir=/data/mysqldata/data2 --user=mysql</span><br><span class="line"></span><br><span class="line"><span class="comment"># 新的初始化数据库目录方式，会在终端打印一个临时登入密码。</span></span><br><span class="line">./mysqld --initialize --basedir=/usr/<span class="built_in">local</span>/mysql --datadir=/data/mysqldata/data1 --user=mysql --console</span><br><span class="line">./mysqld --initialize --basedir=/usr/<span class="built_in">local</span>/mysql --datadir=/data/mysqldata/data2 --user=mysql --console</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="启动实例"><a href="#启动实例" class="headerlink" title="启动实例"></a>启动实例</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line"><span class="built_in">cd</span> /usr/<span class="built_in">local</span>/mysql/bin</span><br><span class="line">mysqld_multi start 1</span><br><span class="line">mysqld_multi start 2</span><br><span class="line">mysqld_multi report</span><br></pre></td></tr></table></figure>
<h2 id="主库创建同步账号"><a href="#主库创建同步账号" class="headerlink" title="主库创建同步账号"></a>主库创建同步账号</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 用刚才初始化数据库目录生成的临时密码登入</span></span><br><span class="line">./mysql -S /var/lib/mysql/mysql1.sock -p your-password</span><br><span class="line"></span><br><span class="line"><span class="comment"># 如果忘记密码，可以在 my.cnf 的 mysqld 块中添加一行 skip-grant-tables = 1</span></span><br><span class="line"><span class="comment"># 可以进行无密码登入，修改成功之后去掉这一行然后重启数据库。</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 修改 root 密码</span></span><br><span class="line">USE mysql;</span><br><span class="line">UPDATE user SET authentication_string = PASSWORD(<span class="string">'new-password'</span>), password_expired = <span class="string">'N'</span>, password_last_changed = NOW() WHERE user = <span class="string">'root'</span>;</span><br><span class="line">FLUSH PRIVILEGES;</span><br><span class="line"></span><br><span class="line"><span class="comment"># 授权账户使得局域网内的机器可以访问数据库</span></span><br><span class="line">GRANT ALL PRIVILEGES ON *.* TO <span class="string">'root'</span>@<span class="string">'%'</span> IDENTIFIED BY <span class="string">'new-password'</span> WITH GRANT OPTION;</span><br><span class="line"></span><br><span class="line"><span class="comment"># 创建一个同步账户</span></span><br><span class="line">CREATE USER <span class="string">'repl'</span>@<span class="string">'%'</span> IDENTIFIED BY <span class="string">'repl-password'</span>;</span><br><span class="line">GRANT REPLICATION SLAVE ON *.* TO <span class="string">'repl'</span>@<span class="string">'%'</span>;</span><br><span class="line"></span><br><span class="line"><span class="comment"># 查看状态</span></span><br><span class="line">SHOW MASTER STATUS;</span><br><span class="line">SHOW BINARY LOGS;</span><br><span class="line"></span><br><span class="line"><span class="comment"># 如果忘记设置日志过期时间，可以进入数据库进行全局设置，并手动清理过期日志</span></span><br><span class="line"><span class="comment"># 不要在数据库目录进行删除日志，这样会使得数据库日志索引不一致，导致自动清理失效</span></span><br><span class="line">SET GLOBAL EXPIRE_LOGS_DAYS = 10;</span><br><span class="line">FLUSH LOGS; <span class="comment"># 触发日志清理，一般是在有新的日志生成的时候触发检查一次。</span></span><br><span class="line">SHOW BINARY LOGS;</span><br><span class="line"></span><br><span class="line"><span class="comment"># 也可以手动删除某个日志之前的所有日志</span></span><br><span class="line">PURGE BINARY LOGS TO <span class="string">'mysql-bin.000080'</span>; <span class="comment"># 删除 80 之前的日志</span></span><br><span class="line">SHOW BINARY LOGS;</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="从库配置"><a href="#从库配置" class="headerlink" title="从库配置"></a>从库配置</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 修改从库的配置文件</span></span><br><span class="line">server-id =2</span><br><span class="line">relay-log =/dbdata/data/relay-log</span><br><span class="line">relay-log-index =/dbdata/data/relay-log.index</span><br><span class="line"></span><br><span class="line"><span class="comment"># 进入数据库执行</span></span><br><span class="line">CHANGE MASTER TO</span><br><span class="line">MASTER_HOST=‘master_host_name’, <span class="comment"># 主库的主机名</span></span><br><span class="line">MASTER_PORT=port_number <span class="comment"># 主库的端口号</span></span><br><span class="line">MASTER_USER=‘replication_user_name’, <span class="comment"># 复制的数据库用户名</span></span><br><span class="line">MASTER_PASSWORD=‘replication_password’, <span class="comment"># 复制的用户密码</span></span><br><span class="line">MASTER_LOG_FILE=‘recorded_log_file_name’, <span class="comment"># 主库的日志文件名</span></span><br><span class="line">MASTER_LOG_POS=recorded_log_position; <span class="comment"># 主库的日志文件位置</span></span><br><span class="line"></span><br><span class="line">start slave;</span><br><span class="line"></span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>MySQL</tag>
<tag>CentOS</tag>
</tags>
</entry>
<entry>
<title>Centos 安装 PhantomJS</title>
<url>/2018/11/21/centos-install-phantomjs/</url>
<content><![CDATA[<p>PhantomJS 已经不再开发了，Seleniumn 也警告使用 PhantomJS 是过时的，推荐使用 headless 版的 Chrome 或者 Firefox，但是有时候需要用到，够用就行，而且在 Linux 下安装也相对简单。</p>
<a id="more"></a>
<!-- toc -->
<h2 id="安装-fontconfig-依赖"><a href="#安装-fontconfig-依赖" class="headerlink" title="安装 fontconfig 依赖"></a>安装 fontconfig 依赖</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">yum install -y fontconfig freetype freetype-devel fontconfig-devel libstdc++</span><br></pre></td></tr></table></figure>
<h2 id="下载-PhantomJS-并解压"><a href="#下载-PhantomJS-并解压" class="headerlink" title="下载 PhantomJS 并解压"></a>下载 PhantomJS 并解压</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 安装到此目录</span></span><br><span class="line"><span class="built_in">cd</span> /usr/<span class="built_in">local</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 下载</span></span><br><span class="line">wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2</span><br><span class="line"></span><br><span class="line"><span class="comment"># 解压</span></span><br><span class="line">tar -jxvf phantomjs-2.1.1-linux-x86_64.tar.bz2</span><br><span class="line"></span><br><span class="line"><span class="comment"># 重命名</span></span><br><span class="line">mv phantomjs-2.1.1-linux-x86_64 phantomjs</span><br><span class="line"></span><br><span class="line"><span class="comment"># 添加软链接</span></span><br><span class="line">ln -s /usr/<span class="built_in">local</span>/phantomjs/bin/phantomjs /usr/bin/phantomjs</span><br><span class="line"></span><br><span class="line"><span class="comment"># 验证</span></span><br><span class="line">phantomjs --version</span><br></pre></td></tr></table></figure>
<h2 id="用-Sselenium-驱动-PhantomJS"><a href="#用-Sselenium-驱动-PhantomJS" class="headerlink" title="用 Sselenium 驱动 PhantomJS"></a>用 Sselenium 驱动 PhantomJS</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> selenium <span class="keyword">import</span> webdriver</span><br><span class="line"><span class="keyword">from</span> selenium.webdriver.common.desired_capabilities <span class="keyword">import</span> DesiredCapabilities</span><br><span class="line"></span><br><span class="line"><span class="comment"># 更改浏览器头</span></span><br><span class="line">dcap = <span class="built_in">dict</span>(DesiredCapabilities.PHANTOMJS)</span><br><span class="line">dcap[<span class="string">"phantomjs.page.settings.userAgent"</span>] = <span class="string">"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36"</span></span><br><span class="line">driver = webdriver.PhantomJS(desired_capabilities=dcap)</span><br><span class="line">driver.set_page_load_timeout(<span class="number">10</span>)</span><br><span class="line">driver.set_script_timeout(<span class="number">10</span>)</span><br><span class="line">driver.get(<span class="string">"https://www.baidu.com"</span>)</span><br></pre></td></tr></table></figure>
<h2 id="推荐使用的-Chrome-用法"><a href="#推荐使用的-Chrome-用法" class="headerlink" title="推荐使用的 Chrome 用法"></a>推荐使用的 Chrome 用法</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> selenium <span class="keyword">import</span> webdriver</span><br><span class="line"><span class="keyword">from</span> selenium.webdriver.chrome.options <span class="keyword">import</span> Options</span><br><span class="line"></span><br><span class="line"><span class="comment"># 无界面浏览器</span></span><br><span class="line">options = Options()</span><br><span class="line">options.add_argument(<span class="string">'headless'</span>)</span><br><span class="line">options.add_argument(<span class="string">'disable-gpu'</span>)</span><br><span class="line">options.add_argument(<span class="string">'window-size=1200x600'</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 禁用 javascript</span></span><br><span class="line">prefs = {<span class="string">'profile.managed_default_content_settings.javascript'</span>: <span class="number">2</span>}</span><br><span class="line">options.add_experimental_option(<span class="string">"prefs"</span>, prefs)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 禁止弹出式窗口</span></span><br><span class="line">prefs = {<span class="string">"profile.default_content_setting_values.notifications"</span>: <span class="number">2</span>}</span><br><span class="line">options.add_experimental_option(<span class="string">"prefs"</span>, prefs)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 禁用图片</span></span><br><span class="line">prefs = {<span class="string">'profile.managed_default_content_settings.images'</span>: <span class="number">2</span>}</span><br><span class="line">options.add_experimental_option(<span class="string">"prefs"</span>, prefs)</span><br><span class="line"></span><br><span class="line">driver = webdriver.Chrome(chrome_options=options)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 执行JS</span></span><br><span class="line">driver.execute_script(<span class="string">'window.scrollTo(0, 0)'</span>) <span class="comment"># scroll to top</span></span><br><span class="line">driver.execute_script(<span class="string">'window.scrollTo(0, document.body.scrollHeight)'</span>) <span class="comment"># end</span></span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
<tag>CentOS</tag>
</tags>
</entry>
<entry>
<title>CentOS 编译安装 Python3</title>
<url>/2018/03/12/centos-install-python3/</url>
<content><![CDATA[<p>记录一下 CentOS 编译安装 Python3 的过程。</p>
<a id="more"></a>
<!-- toc -->
<h2 id="安装系统相关依赖"><a href="#安装系统相关依赖" class="headerlink" title="安装系统相关依赖"></a>安装系统相关依赖</h2><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># https://github.com/pyenv/pyenv/wiki#suggested-build-environment</span></span><br><span class="line">yum install -y zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-static openssl-devel xz xz-devel libffi-devel findutils gcc</span><br><span class="line"></span><br><span class="line"><span class="comment"># 安装终端会话管理工具以便编译时喝茶</span></span><br><span class="line">yum install -y screen tmux</span><br><span class="line"></span><br><span class="line"><span class="comment"># 安装mysqlclient驱动所需的依赖</span></span><br><span class="line">yum install -y mysql-devel</span><br><span class="line"></span><br><span class="line"><span class="comment"># 遇到mysql_config问题</span></span><br><span class="line"><span class="built_in">export</span> PATH=<span class="variable">$PATH</span>:/usr/<span class="built_in">local</span>/mysql/bin</span><br><span class="line"></span><br><span class="line"><span class="comment"># 遇到动态链接库问题</span></span><br><span class="line"><span class="built_in">export</span> LD_LIBRARY_PATH=/usr/<span class="built_in">local</span>/mysql/lib</span><br><span class="line"></span><br><span class="line"><span class="comment"># 遇gcc编译错误(ld:cannot found -lc)</span></span><br><span class="line">yum install -y glibc-devel</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="下载-python3-包"><a href="#下载-python3-包" class="headerlink" title="下载 python3 包"></a>下载 python3 包</h2><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">wget https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz</span><br></pre></td></tr></table></figure>
<h2 id="解压到当前目录"><a href="#解压到当前目录" class="headerlink" title="解压到当前目录"></a>解压到当前目录</h2><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">tar -zxvf Python-3.9.6.tgz</span><br></pre></td></tr></table></figure>
<h2 id="进入生成的目录进行配置"><a href="#进入生成的目录进行配置" class="headerlink" title="进入生成的目录进行配置"></a>进入生成的目录进行配置</h2><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">./configure --prefix=/usr/<span class="built_in">local</span>/python3 --enable-loadable-sqlite-extensions --enable-optimizations</span><br></pre></td></tr></table></figure>
<h2 id="编译安装"><a href="#编译安装" class="headerlink" title="编译安装"></a>编译安装</h2><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">make && make install</span><br></pre></td></tr></table></figure>
<h2 id="添加软连接"><a href="#添加软连接" class="headerlink" title="添加软连接"></a>添加软连接</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">ln -s /usr/<span class="built_in">local</span>/python3/bin/python3 /usr/bin/python3</span><br></pre></td></tr></table></figure>
<h2 id="其他"><a href="#其他" class="headerlink" title="其他"></a>其他</h2><p><code>yum</code> 搜索可用包</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">yum search python3 | grep devel</span><br></pre></td></tr></table></figure>
<p>一键更新 <code>python</code> 包</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line">python3 -m pip list --outdated --format=freeze | grep -v <span class="string">'^\-e'</span> | cut -d = -f 1 | xargs -n1 pip install -U</span><br></pre></td></tr></table></figure>
<p>切换豆瓣源</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line"># 编辑 .pip/pip.conf 添加如下内容</span><br><span class="line"></span><br><span class="line">[global]</span><br><span class="line">index-url = https://pypi.douban.com/simple</span><br><span class="line">trusted-host = pypi.douban.com</span><br><span class="line"></span><br><span class="line">[list]</span><br><span class="line">format = columns</span><br></pre></td></tr></table></figure>
<p>离线安装python包</p>
<figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 首先离线下载依赖包，比如 `tensorflow-cpu`，同时会下载所有的依赖项</span></span><br><span class="line">python3 -m pip download -d packages tensorflow-cpu</span><br><span class="line"></span><br><span class="line"><span class="comment"># 下载和目标机器对应架构的依赖包</span></span><br><span class="line"><span class="comment"># https://pip.pypa.io/en/stable/cli/pip_download/</span></span><br><span class="line">python3 -m pip download \</span><br><span class="line"> --only-binary=:all: \</span><br><span class="line"> --platform manylinux1_x86_64 --platform linux_x86_64 --platform any \</span><br><span class="line"> --python-version 36 \</span><br><span class="line"> --implementation cp \</span><br><span class="line"> --abi cp36m --abi cp36 --abi abi3 --abi none \</span><br><span class="line"> tensorflow-cpu</span><br><span class="line"></span><br><span class="line"><span class="comment"># 离线安装，指定寻找包的目录为 packages</span></span><br><span class="line">python3 -m pip install --no-index --find-links=packages -r requirements.txt</span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
<tag>CentOS</tag>
</tags>
</entry>
<entry>
<title>Python 处理 Csv 文件常见错误</title>
<url>/2018/12/03/csv-error/</url>
<content><![CDATA[<p>在用 Python 处理 csv 文件时遇到2个错误，记录下处理方法。</p>
<a id="more"></a>
<!-- toc -->
<h2 id="字段包含-NULL-值"><a href="#字段包含-NULL-值" class="headerlink" title="字段包含 NULL 值"></a>字段包含 NULL 值</h2><p>csv 文件中字段包含 NULL 值会出错，解决方法是读取文件时把 NULL 值替换为空字符串。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> csv</span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> <span class="built_in">open</span>(<span class="string">'test.csv'</span>, <span class="string">'rt'</span>, encoding=<span class="string">'utf-8'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> fc = csv.DictReader((line.replace(<span class="string">'\0'</span>, <span class="string">''</span>) <span class="keyword">for</span> line <span class="keyword">in</span> f))</span><br><span class="line"> <span class="comment"># do something with fc</span></span><br></pre></td></tr></table></figure>
<h2 id="OverflowError-and-maxInt"><a href="#OverflowError-and-maxInt" class="headerlink" title="OverflowError and maxInt"></a>OverflowError and maxInt</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> csv</span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"></span><br><span class="line">maxInt = sys.maxsize</span><br><span class="line">decrement = <span class="literal">True</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">while</span> decrement:</span><br><span class="line"> <span class="comment"># decrease the maxInt value by factor 10</span></span><br><span class="line"> <span class="comment"># as long as the OverflowError occurs.</span></span><br><span class="line"></span><br><span class="line"> decrement = <span class="literal">False</span></span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> csv.field_size_limit(maxInt)</span><br><span class="line"> <span class="keyword">except</span> OverflowError:</span><br><span class="line"> maxInt = <span class="built_in">int</span>(maxInt / <span class="number">10</span>)</span><br><span class="line"> decrement = <span class="literal">True</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">with</span> <span class="built_in">open</span>(<span class="string">'test.csv'</span>, <span class="string">'rt'</span>, encoding=<span class="string">'utf-8'</span>) <span class="keyword">as</span> f:</span><br><span class="line"> fc = csv.DictReader(f)</span><br><span class="line"> <span class="comment"># do something with fc</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
</tags>
</entry>
<entry>
<title>几种启动 Flask 应用的方式</title>
<url>/2018/03/12/flask-start-up/</url>
<content><![CDATA[<p>记录几种启动 Flask 应用的方式</p>
<a id="more"></a>
<!-- toc -->
<p>首先写一个简单的 <code>index.py</code></p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</span><br><span class="line"></span><br><span class="line">app = Flask(__name__)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route(<span class="params"><span class="string">'/'</span>, methods=[<span class="string">'GET'</span>]</span>)</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span>():</span></span><br><span class="line"> <span class="keyword">return</span> <span class="string">'hello, world!'</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="最简单的启动方式"><a href="#最简单的启动方式" class="headerlink" title="最简单的启动方式"></a>最简单的启动方式</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> app.run()</span><br></pre></td></tr></table></figure>
<p>这只能用于开发模式，可以设置<code>debug=True</code>开启调试模式，并且这是单线程同步的。</p>
<h2 id="用-tornado-驱动-flask"><a href="#用-tornado-驱动-flask" class="headerlink" title="用 tornado 驱动 flask"></a>用 <code>tornado</code> 驱动 <code>flask</code></h2><p>写一个<code>server.py</code>，并将上面<code>index.py</code>中的启动代码去掉，终端运行<code>python server.py</code>。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> tornado.wsgi <span class="keyword">import</span> WSGIContainer</span><br><span class="line"><span class="keyword">from</span> tornado.httpserver <span class="keyword">import</span> HTTPServer</span><br><span class="line"><span class="keyword">from</span> tornado.ioloop <span class="keyword">import</span> IOLoop</span><br><span class="line"><span class="keyword">from</span> index <span class="keyword">import</span> app</span><br><span class="line">http_server = HTTPServer(WSGIContainer(app))</span><br><span class="line">http_server.listen(<span class="number">5000</span>) <span class="comment"># flask默认的端口</span></span><br><span class="line">IOLoop.instance().start()</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>这也是同步的，同一时间只能处理一个请求，可以写个简单的接口测试一下。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</span><br><span class="line"></span><br><span class="line">app = Flask(__name__)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route(<span class="params"><span class="string">'/'</span>, methods=[<span class="string">'GET'</span>]</span>)</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span>():</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> <span class="string">'hello, world!'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route(<span class="params"><span class="string">'/test'</span>, methods=[<span class="string">'GET'</span>]</span>)</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test</span>():</span></span><br><span class="line"> <span class="keyword">for</span> n <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>):</span><br><span class="line"> print(n)</span><br><span class="line"> time.sleep(<span class="number">2</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> <span class="string">'hello'</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>用postman同时发起5个请求，后台按顺序打印0-9，5个请求是一个一个执行的。</p>
<h2 id="用-twisted-驱动-flask"><a href="#用-twisted-驱动-flask" class="headerlink" title="用 twisted 驱动 flask"></a>用 <code>twisted</code> 驱动 <code>flask</code></h2><p>这个可以同时处理多个请求。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># -*- coding: utf-8 -*-</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> time</span><br><span class="line"><span class="keyword">from</span> flask <span class="keyword">import</span> Flask</span><br><span class="line"></span><br><span class="line">app = Flask(__name__)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route(<span class="params"><span class="string">'/'</span>, methods=[<span class="string">'GET'</span>]</span>)</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">hello</span>():</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> <span class="string">'hello, world!'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="meta">@app.route(<span class="params"><span class="string">'/test'</span>, methods=[<span class="string">'GET'</span>]</span>)</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">test</span>():</span></span><br><span class="line"> <span class="keyword">for</span> n <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>):</span><br><span class="line"> print(n)</span><br><span class="line"> time.sleep(<span class="number">2</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> <span class="string">'hello'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">"__main__"</span>:</span><br><span class="line"> reactor_args = {}</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">run_twisted_wsgi</span>():</span></span><br><span class="line"> <span class="keyword">from</span> twisted.internet <span class="keyword">import</span> reactor</span><br><span class="line"> <span class="keyword">from</span> twisted.web.server <span class="keyword">import</span> Site</span><br><span class="line"> <span class="keyword">from</span> twisted.web.wsgi <span class="keyword">import</span> WSGIResource</span><br><span class="line"></span><br><span class="line"> resource = WSGIResource(reactor, reactor.getThreadPool(), app)</span><br><span class="line"> site = Site(resource)</span><br><span class="line"> reactor.listenTCP(<span class="number">5000</span>, site)</span><br><span class="line"> reactor.run(**reactor_args)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> app.debug:</span><br><span class="line"> <span class="comment"># Disable twisted signal handlers in development only.</span></span><br><span class="line"> reactor_args[<span class="string">'installSignalHandlers'</span>] = <span class="number">0</span></span><br><span class="line"> <span class="comment"># Turn on auto reload.</span></span><br><span class="line"> <span class="keyword">import</span> werkzeug.serving</span><br><span class="line"> run_twisted_wsgi = werkzeug.serving.run_with_reloader(run_twisted_wsgi)</span><br><span class="line"></span><br><span class="line"> run_twisted_wsgi()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> app.run()</span><br><span class="line"></span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Flask</tag>
</tags>
</entry>
<entry>
<title>Python 通过 Thrift 操作 Hbase</title>
<url>/2018/11/14/hbase-thrift/</url>
<content><![CDATA[<!-- toc -->
<p>记录 Python 通过 Thrift 操作 Hbase 的通用操作方法。</p>
<a id="more"></a>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> thrift.transport <span class="keyword">import</span> TSocket</span><br><span class="line"><span class="keyword">from</span> thrift.protocol <span class="keyword">import</span> TBinaryProtocol</span><br><span class="line"><span class="keyword">from</span> thrift.transport <span class="keyword">import</span> TTransport</span><br><span class="line"><span class="keyword">from</span> elasticsearch <span class="keyword">import</span> Elasticsearch</span><br><span class="line"><span class="keyword">from</span> hbase <span class="keyword">import</span> Hbase</span><br><span class="line"></span><br><span class="line"><span class="comment"># Connect to HBase Thrift server</span></span><br><span class="line">transport = TTransport.TBufferedTransport(TSocket.TSocket(<span class="string">'localhost'</span>, <span class="number">9090</span>))</span><br><span class="line">protocol = TBinaryProtocol.TBinaryProtocolAccelerated(transport)</span><br><span class="line"></span><br><span class="line"><span class="comment"># Create and open the client connection</span></span><br><span class="line">client = Hbase.Client(protocol)</span><br><span class="line">transport.<span class="built_in">open</span>()</span><br><span class="line"></span><br><span class="line"><span class="comment"># Connect to Elasticsearch server</span></span><br><span class="line">es = Elasticsearch(<span class="string">'localhost'</span>,</span><br><span class="line"> http_auth=(<span class="string">'username'</span>, <span class="string">'password'</span>), port=<span class="string">'9200'</span>,</span><br><span class="line"> timeout=<span class="number">30</span>, max_retries=<span class="number">10</span>, retry_on_timeout=<span class="literal">True</span></span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fetch_one</span>(<span class="params">index, doc_type, body, size=<span class="number">1</span></span>):</span></span><br><span class="line"> <span class="string">"""查询es获取第一条匹配的数据</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Arguments:</span></span><br><span class="line"><span class="string"> index {str} -- 索引</span></span><br><span class="line"><span class="string"> doc_type {str} -- 类型</span></span><br><span class="line"><span class="string"> body {dict} -- 查询语句</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Keyword Arguments:</span></span><br><span class="line"><span class="string"> size {int} -- 返回数量 (default: {1})</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Returns:</span></span><br><span class="line"><span class="string"> dict -- 一条数据，没有结果返回 None</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"></span><br><span class="line"> res = es.search(index=index, doc_type=doc_type,</span><br><span class="line"> scroll=<span class="string">'2m'</span>, body=body,</span><br><span class="line"> size=size)</span><br><span class="line"> hits = res[<span class="string">'hits'</span>][<span class="string">'hits'</span>]</span><br><span class="line"> <span class="keyword">return</span> hits[<span class="number">0</span>] <span class="keyword">if</span> hits <span class="keyword">else</span> <span class="literal">None</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fetch_all</span>(<span class="params">index, doc_type, body, size=<span class="number">100</span></span>):</span></span><br><span class="line"> <span class="string">"""查询es获取所有匹配的结果</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Arguments:</span></span><br><span class="line"><span class="string"> index {str} -- 索引</span></span><br><span class="line"><span class="string"> doc_type {str} -- 类型</span></span><br><span class="line"><span class="string"> body {dict} -- 查询语句</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Keyword Arguments:</span></span><br><span class="line"><span class="string"> size {int} -- 返回数量 (default: {100})</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Returns:</span></span><br><span class="line"><span class="string"> list -- 结果集</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"></span><br><span class="line"> res = es.search(index=index, doc_type=doc_type,</span><br><span class="line"> scroll=<span class="string">'2m'</span>, body=body,</span><br><span class="line"> size=size)</span><br><span class="line"> <span class="keyword">return</span> res[<span class="string">'hits'</span>][<span class="string">'hits'</span>]</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">build_term</span>(<span class="params">field, value</span>):</span></span><br><span class="line"> <span class="string">"""term</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Arguments:</span></span><br><span class="line"><span class="string"> field {str} -- 字段</span></span><br><span class="line"><span class="string"> value {str} -- 值</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Returns:</span></span><br><span class="line"><span class="string"> dict -- 查询语句</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"></span><br><span class="line"> body = {</span><br><span class="line"> <span class="string">"query"</span>: {</span><br><span class="line"> <span class="string">"term"</span>: {</span><br><span class="line"> field: {</span><br><span class="line"> <span class="string">"value"</span>: value</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> body</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">build_terms</span>(<span class="params">field, values</span>):</span></span><br><span class="line"> <span class="string">"""terms</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Arguments:</span></span><br><span class="line"><span class="string"> field {str} -- 字段</span></span><br><span class="line"><span class="string"> values {list} -- 列表</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Returns:</span></span><br><span class="line"><span class="string"> dict -- 查询语句</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"></span><br><span class="line"> body = {</span><br><span class="line"> <span class="string">"query"</span>: {</span><br><span class="line"> <span class="string">"terms"</span>: {</span><br><span class="line"> field: values</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> body</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_row_with_columns</span>(<span class="params">table_name, rowkey, columns</span>):</span></span><br><span class="line"> <span class="string">"""根据 rowkey 从 hbase 获取一条数据</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Arguments:</span></span><br><span class="line"><span class="string"> table_name {str} -- 表名</span></span><br><span class="line"><span class="string"> rowkey {str} -- rowkey</span></span><br><span class="line"><span class="string"> attributes {list} -- 属性列表</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Returns:</span></span><br><span class="line"><span class="string"> dict -- 一条数据，没有则返回None</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> table_name = table_name.encode()</span><br><span class="line"> rowkey = rowkey.encode()</span><br><span class="line"> columns = [(<span class="string">'0:'</span> + c).encode() <span class="keyword">for</span> c <span class="keyword">in</span> columns]</span><br><span class="line"> res = client.getRowWithColumns(table_name, rowkey, columns, <span class="literal">None</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> res:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">None</span></span><br><span class="line"> d = {</span><br><span class="line"> k.decode().split(<span class="string">':'</span>)[<span class="number">1</span>]: v.value.decode()</span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> res[<span class="number">0</span>].columns.items()</span><br><span class="line"> }</span><br><span class="line"> d[<span class="string">'rowkey'</span>] = res[<span class="number">0</span>].row.decode()</span><br><span class="line"> <span class="keyword">return</span> d</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">get_rows_with_columns</span>(<span class="params">table_name, rowkeys, columns</span>):</span></span><br><span class="line"> <span class="string">"""根据 rowkeys 从 hbase 获取所有匹配的数据</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Arguments:</span></span><br><span class="line"><span class="string"> table_name {str} -- 表名</span></span><br><span class="line"><span class="string"> rowkeys {list} -- rowkey 列表</span></span><br><span class="line"><span class="string"> columns {list} -- 指定返回字段</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> Returns:</span></span><br><span class="line"><span class="string"> list -- 数据结果集</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"> data = []</span><br><span class="line"> table_name = table_name.encode()</span><br><span class="line"> rowkeys = [k.encode() <span class="keyword">for</span> k <span class="keyword">in</span> rowkeys]</span><br><span class="line"> columns = [(<span class="string">'0:'</span> + c).encode() <span class="keyword">for</span> c <span class="keyword">in</span> columns]</span><br><span class="line"> res = client.getRowsWithColumns(table_name, rowkeys, columns, <span class="literal">None</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> r <span class="keyword">in</span> res:</span><br><span class="line"> d = {</span><br><span class="line"> k.decode().split(<span class="string">':'</span>)[<span class="number">1</span>]: v.value.decode()</span><br><span class="line"> <span class="keyword">for</span> k, v <span class="keyword">in</span> r.columns.items()</span><br><span class="line"> }</span><br><span class="line"> d[<span class="string">'rowkey'</span>] = r.row.decode()</span><br><span class="line"> data.append(d)</span><br><span class="line"> <span class="keyword">return</span> data</span><br><span class="line"></span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
</tags>
</entry>
<entry>
<title>Postgresql Tutorial</title>
<url>/2021/03/14/postgresql-tutorial/</url>
<content><![CDATA[<p>psql 数据库名 –连接数据库<br>select rolname,rolpassword from pg_authid;–查看用户名密码<br>select usename,passwd from pg_shadow;–查看用户名密码<br>select version(); – 查看版本<br>select current_database();–查看当前数据库<br>\l –查看所有数据库<br>\dt –查看表<br>\password username –修改密码<br>\password –设置密码。<br>? –查看psql命令列表。<br>\c [database_name] –连接其他数据库，切换数据库。<br>\conninfo –列出当前数据库和连接的信息。<br>\d –列出当前数据库的所有表格。<br>\d [table_name] –列出某一张表格的结构。<br>\du –列出所有用户。<br>\e –打开文本编辑器。<br>help –帮助<br>\h –查看SQL命令的解释，比如\h select。<br>\q –退出</p>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>PostgreSQL</tag>
</tags>
</entry>
<entry>
<title>Python 脚本自动重载</title>
<url>/2018/03/09/python-reloader/</url>
<content><</span><br><span class="line"> signal.signal(signal.SIGTERM, <span class="keyword">lambda</span> *args: sys.exit(<span class="number">0</span>))</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> <span class="keyword">if</span> os.environ.get(<span class="string">'WERKZEUG_RUN_MAIN'</span>) == <span class="string">'true'</span>:</span><br><span class="line"> t = threading.Thread(target=main_func, args=())</span><br><span class="line"> t.setDaemon(<span class="literal">True</span>)</span><br><span class="line"> t.start()</span><br><span class="line"> reloader.run()</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> sys.exit(reloader.restart_with_reloader())</span><br><span class="line"> <span class="keyword">except</span> KeyboardInterrupt:</span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>用法如下</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span>():</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> print(<span class="string">"hello, world!"</span>)</span><br><span class="line"> time.sleep(<span class="number">2</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> run_with_reloader(main)</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>但是执行函数不能传参，修改 run_with_reloader 如下</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">run_with_reloader</span>(<span class="params">main_func, args=(<span class="params"></span>), kwargs=<span class="literal">None</span>,</span></span></span><br><span class="line"><span class="function"><span class="params"> extra_files=<span class="literal">None</span>, interval=<span class="number">1</span>,</span></span></span><br><span class="line"><span class="function"><span class="params"> reloader_type=<span class="string">'auto'</span></span>):</span></span><br><span class="line"> <span class="string">"""Run the given function in an independent python interpreter."""</span></span><br><span class="line"> <span class="keyword">import</span> os</span><br><span class="line"> <span class="keyword">import</span> sys</span><br><span class="line"> <span class="keyword">import</span> signal</span><br><span class="line"> <span class="keyword">import</span> threading</span><br><span class="line"> <span class="keyword">from</span> werkzeug._reloader <span class="keyword">import</span> reloader_loops</span><br><span class="line"> reloader = reloader_loops[reloader_type](extra_files, interval)</span><br><span class="line"> signal.signal(signal.SIGTERM, <span class="keyword">lambda</span> *args: sys.exit(<span class="number">0</span>))</span><br><span class="line"> <span class="keyword">try</span>:</span><br><span class="line"> <span class="keyword">if</span> os.environ.get(<span class="string">'WERKZEUG_RUN_MAIN'</span>) == <span class="string">'true'</span>:</span><br><span class="line"> t = threading.Thread(target=main_func, args=args, kwargs=kwargs)</span><br><span class="line"> t.setDaemon(<span class="literal">True</span>)</span><br><span class="line"> t.start()</span><br><span class="line"> reloader.run()</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> sys.exit(reloader.restart_with_reloader())</span><br><span class="line"> <span class="keyword">except</span> KeyboardInterrupt:</span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>然后就能传参了</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> time</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span>(<span class="params">name, age</span>):</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> print(<span class="string">"hello, "</span>, name, <span class="string">'!'</span>)</span><br><span class="line"> time.sleep(<span class="number">2</span>)</span><br><span class="line"> print(age)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</span><br><span class="line"> run_with_reloader(main, args=(<span class="string">'foo'</span>, <span class="number">20</span>))</span><br><span class="line"></span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
</tags>
</entry>
<entry>
<title>RDS for MySQL 备份文件恢复到自建数据库</title>
<url>/2018/09/13/rds-for-mysql/</url>
<content><![CDATA[<p>云数据库MySQL版使用开源软件Percona Xtrabackup对数据库进行备份，所以您可以使用该软件将云数据库MySQL的备份文件恢复到自建数据库中，本文将介绍详细的操作步骤。</p>
<a id="more"></a>
<!-- toc -->
<h2 id="软件说明"><a href="#软件说明" class="headerlink" title="软件说明"></a>软件说明</h2><ol>
<li>MySQL 5.6.41</li>
<li>Percona XtraBackup 2.2.9</li>
<li><a href="http://oss.aliyuncs.com/aliyunecs/rds_backup_extract.sh?spm=a2c4g.11186623.2.6.3f4d5cf8Ze2ycZ&file=rds_backup_extract.sh">rds_backup_extract.sh</a></li>
</ol>
<h2 id="解压数据库备份文件"><a href="#解压数据库备份文件" class="headerlink" title="解压数据库备份文件"></a>解压数据库备份文件</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">unzip -P密码 mysql_data_backup.tar.gz.zip</span><br><span class="line">bash rds_backup_extract -f mysql_data_backup.tar.gz -C /data/mysql/data</span><br></pre></td></tr></table></figure>
<h2 id="修改配置文件-backup-my-cnf-如下"><a href="#修改配置文件-backup-my-cnf-如下" class="headerlink" title="修改配置文件 backup-my.cnf 如下"></a>修改配置文件 backup-my.cnf 如下</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line"><span class="comment"># This MySQL options file was generated by innobackupex.</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># The MySQL server</span></span><br><span class="line">[mysqld]</span><br><span class="line">innodb_checksum_algorithm=innodb</span><br><span class="line"><span class="comment"># innodb_log_checksum_algorithm=innodb</span></span><br><span class="line">innodb_data_file_path=ibdata1:200M:autoextend</span><br><span class="line">innodb_log_files_in_group=2</span><br><span class="line">innodb_log_file_size=1572864000</span><br><span class="line"><span class="comment"># innodb_fast_checksum=false</span></span><br><span class="line"><span class="comment"># innodb_page_size=16384</span></span><br><span class="line"><span class="comment"># innodb_log_block_size=512</span></span><br><span class="line">innodb_undo_directory=.</span><br><span class="line">innodb_undo_tablespaces=0</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># rds_encrypt_data=false</span></span><br><span class="line"><span class="comment"># innodb_encrypt_algorithm=aes_128_ecb</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="修改文件属主"><a href="#修改文件属主" class="headerlink" title="修改文件属主"></a>修改文件属主</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">chown -R mysql:mysql /data/mysql/data</span><br></pre></td></tr></table></figure>
<h2 id="恢复数据文件"><a href="#恢复数据文件" class="headerlink" title="恢复数据文件"></a>恢复数据文件</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">chmod 400 /data/mysql/data/backup-my.cnf</span><br><span class="line">innobackupex --defaults-file=/data/mysql/data/backup-my.cnf --apply-log /data/mysql/data</span><br></pre></td></tr></table></figure>
<h2 id="启动数据库并登入验证"><a href="#启动数据库并登入验证" class="headerlink" title="启动数据库并登入验证"></a>启动数据库并登入验证</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">mysqld_safe --defaults-file=/data/mysql/data/backup-my.cnf --user=mysql --datadir=/data/mysql/data --skip-grant-tables &</span><br></pre></td></tr></table></figure>
<h2 id="新建用户"><a href="#新建用户" class="headerlink" title="新建用户"></a>新建用户</h2><p>恢复完成后，表 mysql.user 中是不包含 RDS 中创建的用户，需要新建，新建用户前请执行如下 SQL：</p>
<figure class="highlight sql"><table><tr><td class="code"><pre><span class="line"><span class="keyword">delete</span> <span class="keyword">from</span> mysql.db <span class="keyword">where</span> <span class="keyword">user</span><><span class="string">'root'</span> <span class="keyword">and</span> <span class="keyword">char_length</span>(<span class="keyword">user</span>)><span class="number">0</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">delete</span> <span class="keyword">from</span> mysql.tables_priv <span class="keyword">where</span> <span class="keyword">user</span><><span class="string">'root'</span> <span class="keyword">and</span> <span class="keyword">char_length</span>(<span class="keyword">user</span>)><span class="number">0</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">flush</span> <span class="keyword">privileges</span>;</span><br></pre></td></tr></table></figure>
<h2 id="参考链接"><a href="#参考链接" class="headerlink" title="参考链接"></a>参考链接</h2><p><a href="https://help.aliyun.com/knowledge_detail/41817.html">阿里云数据备份/恢复</a></p>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>MySQL</tag>
<tag>CentOS</tag>
</tags>
</entry>
<entry>
<title>Selenium 在 Ubuntu 服务器上的使用</title>
<url>/2018/08/13/selenium-linux/</url>
<content><![CDATA[<h2 id="安装-chrome"><a href="#安装-chrome" class="headerlink" title="安装 chrome"></a>安装 chrome</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -</span><br><span class="line"><span class="built_in">echo</span> <span class="string">'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main'</span> | sudo tee /etc/apt/sources.list.d/google-chrome.list</span><br><span class="line">sudo apt-get update</span><br><span class="line">sudo apt-get install google-chrome-stable</span><br></pre></td></tr></table></figure>
<a id="more"></a>
<!-- toc -->
<h2 id="安装-chromedriver"><a href="#安装-chromedriver" class="headerlink" title="安装 chromedriver"></a>安装 chromedriver</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">wget -N https://chromedriver.storage.googleapis.com/2.41/chromedriver_linux64.zip</span><br><span class="line">unzip chromedriver_linux64.zip</span><br><span class="line">chmod +x chromedriver</span><br><span class="line">cp chromedriver /usr/bin/</span><br></pre></td></tr></table></figure>
<h2 id="安装-Xvfb"><a href="#安装-Xvfb" class="headerlink" title="安装 Xvfb"></a>安装 Xvfb</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">sudo apt-get -y install xvfb gtk2-engines-pixbuf</span><br><span class="line">sudo apt-get -y install xfonts-cyrillic xfonts-100dpi xfonts-75dpi xfonts-base xfonts-scalable</span><br><span class="line"><span class="comment"># 截图功能，可选</span></span><br><span class="line">sudo apt-get -y install imagemagick x11-apps</span><br><span class="line">Xvfb -ac :99 -screen 0 1280x1024x16 & <span class="built_in">export</span> DISPLAY=:99</span><br></pre></td></tr></table></figure>
<h2 id="测试脚本"><a href="#测试脚本" class="headerlink" title="测试脚本"></a>测试脚本</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> selenium <span class="keyword">import</span> webdriver</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">chrome_options = webdriver.ChromeOptions()</span><br><span class="line">chrome_options.add_argument(<span class="string">'--headless'</span>)</span><br><span class="line">chrome_options.add_argument(<span class="string">'--disable-gpu'</span>)</span><br><span class="line">driver = webdriver.Chrome(chrome_options=chrome_options,executable_path=<span class="string">'/usr/bin/chromedriver'</span>)</span><br><span class="line">driver.get(<span class="string">"https://www.baidu.com"</span>)</span><br><span class="line">print(driver.title)</span><br><span class="line">driver.quit()</span><br></pre></td></tr></table></figure>
<h2 id="遇到的问题"><a href="#遇到的问题" class="headerlink" title="遇到的问题"></a>遇到的问题</h2><p><code>ubuntu server 18.04</code> 虽然内置 <code>python3</code> 版本，但是没有 <code>pip</code><br>在 <code>/etc/apt/sources.list</code> 添加下列源</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">deb http://cn.archive.ubuntu.com/ubuntu bionic main multiverse restricted universe</span><br><span class="line">deb http://cn.archive.ubuntu.com/ubuntu bionic-updates main multiverse restricted universe</span><br><span class="line">deb http://cn.archive.ubuntu.com/ubuntu bionic-security main multiverse restricted universe</span><br><span class="line">deb http://cn.archive.ubuntu.com/ubuntu bionic-proposed main multiverse restricted universe</span><br></pre></td></tr></table></figure>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">sudo apt-get update</span><br><span class="line">sudo apt-get install python3-pip</span><br></pre></td></tr></table></figure>
<h2 id="再用-pip-安装-selenium"><a href="#再用-pip-安装-selenium" class="headerlink" title="再用 pip 安装 selenium"></a>再用 pip 安装 selenium</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">pip3 install selenium</span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
</tags>
</entry>
<entry>
<title>Screen 使用教程</title>
<url>/2018/03/13/screen-usage/</url>
<content><![CDATA[<p>GNU Screen是一款由GNU计划开发的用于命令行终端切换的自由软件。用户可以通过该软件同时连接多个本地或远程的命令行会话，并在其间自由切换。</p>
<a id="more"></a>
<!-- toc -->
<h2 id="安装-screen"><a href="#安装-screen" class="headerlink" title="安装 screen"></a>安装 screen</h2><figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">yum install -y screen</span><br></pre></td></tr></table></figure>
<h2 id="screen-常用命令"><a href="#screen-常用命令" class="headerlink" title="screen 常用命令"></a>screen 常用命令</h2><p>新建一个Screen Session</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">screen -S session_name</span><br></pre></td></tr></table></figure>
<p>将当前Screen Session放到后台</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">CTRL + A + D</span><br></pre></td></tr></table></figure>
<p>唤起一个Screen Session</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">screen -r session_name</span><br></pre></td></tr></table></figure>
<p>分享一个Screen Session</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">screen -x session_name</span><br></pre></td></tr></table></figure>
<p>终止一个Screen Session</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line"><span class="built_in">exit</span></span><br><span class="line">or</span><br><span class="line">CTRL + D</span><br></pre></td></tr></table></figure>
<p>默认显示一屏的内容，要查看之前内容，如下操作：</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">Ctrl + A ESC</span><br></pre></td></tr></table></figure>
<p>列表所有的会话</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">screen -ls</span><br></pre></td></tr></table></figure>
<p>进入某个会话</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">screen -r session_name</span><br></pre></td></tr></table></figure>
<p>如果进不去，则</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">screen -d session_name</span><br></pre></td></tr></table></figure>
<p>再</p>
<figure class="highlight sh"><table><tr><td class="code"><pre><span class="line">screen -r session_name</span><br></pre></td></tr></table></figure>
<p><code>ctrl + A + N</code> 切换窗口</p>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>CentOS</tag>
</tags>
</entry>
<entry>
<title>数据科学中的各种相似性度量及其实现</title>
<url>/2021/06/10/similarity-measures-in-python/</url>
<content><![CDATA[<!-- toc -->
<h2 id="欧氏距离-Euclidean-Distance"><a href="#欧氏距离-Euclidean-Distance" class="headerlink" title="欧氏距离(Euclidean Distance)"></a>欧氏距离(Euclidean Distance)</h2><p>欧氏距离（也称欧几里得度量）指在 m 维空间中两个点之间的真实距离，或者向量的自然长度（即该点到原点的距离）。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> math <span class="keyword">import</span> *</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">euclidean_distance</span>(<span class="params">x, y</span>):</span></span><br><span class="line"> <span class="comment"># 方法一</span></span><br><span class="line"> <span class="comment"># d = sqrt(sum(pow(a - b, 2) for a, b in zip(x, y)))</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># 方法二</span></span><br><span class="line"> x, y = np.array(x), np.array(y)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># d = np.linalg.norm(x - y, ord=2)</span></span><br><span class="line"></span><br><span class="line"> d = np.sqrt(np.<span class="built_in">sum</span>(np.square(x - y)))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> d</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">x = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">y = [<span class="number">6</span>, <span class="number">7</span>, <span class="number">8</span>, <span class="number">9</span>, <span class="number">10</span>]</span><br><span class="line">euclidean_distance(x, y) <span class="comment"># 11.180339887498949</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> scipy.spatial.distance <span class="keyword">import</span> euclidean</span><br><span class="line">euclidean(x, y) <span class="comment"># 11.180339887498949</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<a id="more"></a>
<h2 id="标准化欧氏距离-Standardized-Euclidean-Distance"><a href="#标准化欧氏距离-Standardized-Euclidean-Distance" class="headerlink" title="标准化欧氏距离(Standardized Euclidean Distance)"></a>标准化欧氏距离(Standardized Euclidean Distance)</h2><p>将各个分量都“标准化”到均值、方差相等所得出的距离。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> scipy.spatial.distance <span class="keyword">import</span> seuclidean</span><br><span class="line">seuclidean([<span class="number">1</span>, <span class="number">0</span>, <span class="number">0</span>], [<span class="number">0</span>, <span class="number">1</span>, <span class="number">0</span>], [<span class="number">0.1</span>, <span class="number">0.1</span>, <span class="number">0.1</span>]) <span class="comment"># 4.4721359549995796</span></span><br><span class="line">seuclidean([<span class="number">1</span>, <span class="number">0</span>, <span class="number">0</span>], [<span class="number">0</span>, <span class="number">1</span>, <span class="number">0</span>], [<span class="number">1</span>, <span class="number">0.1</span>, <span class="number">0.1</span>]) <span class="comment"># 3.3166247903553998</span></span><br><span class="line">seuclidean([<span class="number">1</span>, <span class="number">0</span>, <span class="number">0</span>], [<span class="number">0</span>, <span class="number">1</span>, <span class="number">0</span>], [<span class="number">10</span>, <span class="number">0.1</span>, <span class="number">0.1</span>]) <span class="comment"># 3.1780497164141406</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="马氏距离-Mahalanobis-Distance"><a href="#马氏距离-Mahalanobis-Distance" class="headerlink" title="马氏距离(Mahalanobis Distance)"></a>马氏距离(Mahalanobis Distance)</h2><p>马哈拉诺比斯距离是由印度统计学家马哈拉诺比斯提出的，表示数据的协方差距离。它是一种有效的计算两个未知样本集的相似度的方法。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> scipy.spatial.distance <span class="keyword">import</span> mahalanobis</span><br><span class="line">iv = [[<span class="number">1</span>, <span class="number">0.5</span>, <span class="number">0.5</span>], [<span class="number">0.5</span>, <span class="number">1</span>, <span class="number">0.5</span>], [<span class="number">0.5</span>, <span class="number">0.5</span>, <span class="number">1</span>]]</span><br><span class="line">mahalanobis([<span class="number">1</span>, <span class="number">0</span>, <span class="number">0</span>], [<span class="number">0</span>, <span class="number">1</span>, <span class="number">0</span>], iv) <span class="comment"># 1.0</span></span><br><span class="line">mahalanobis([<span class="number">0</span>, <span class="number">2</span>, <span class="number">0</span>], [<span class="number">0</span>, <span class="number">1</span>, <span class="number">0</span>], iv) <span class="comment"># 1.0</span></span><br><span class="line">mahalanobis([<span class="number">2</span>, <span class="number">0</span>, <span class="number">0</span>], [<span class="number">0</span>, <span class="number">1</span>, <span class="number">0</span>], iv) <span class="comment"># 1.7320508075688772</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="编辑距离"><a href="#编辑距离" class="headerlink" title="编辑距离"></a>编辑距离</h2><p>编辑距离是针对二个字符串的差异程度的量化量测，量测方式是看至少需要多少次的处理才能将一个字符串变成另一个字符串。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> Levenshtein</span><br><span class="line"></span><br><span class="line">s1 = <span class="string">'Hello, World!'</span></span><br><span class="line">s2 = <span class="string">'I am very happy!'</span></span><br><span class="line"></span><br><span class="line">Levenshtein.distance(s1, s2) <span class="comment"># 14</span></span><br><span class="line">Levenshtein.ratio(s1, s2) <span class="comment"># 0.20689655172413793</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="曼哈顿距离-Manhattan-Distance"><a href="#曼哈顿距离-Manhattan-Distance" class="headerlink" title="曼哈顿距离(Manhattan Distance)"></a>曼哈顿距离(Manhattan Distance)</h2><p>在欧几里得空间的固定直角坐标系上两点所形成的线段对轴产生的投影的距离总和。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> math <span class="keyword">import</span> *</span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">manhattan_distance</span>(<span class="params">x, y</span>):</span></span><br><span class="line"> <span class="comment"># 方法一</span></span><br><span class="line"> <span class="comment"># d = sum(abs(a - b) for a, b in zip(x, y))</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># 方法二</span></span><br><span class="line"> x, y = np.array(x), np.array(y)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># d = np.linalg.norm(x - y, ord=1)</span></span><br><span class="line"></span><br><span class="line"> d = np.<span class="built_in">sum</span>(np.<span class="built_in">abs</span>(x - y))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> d</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">x = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">y = [<span class="number">6</span>, <span class="number">7</span>, <span class="number">8</span>, <span class="number">9</span>, <span class="number">10</span>]</span><br><span class="line">manhattan_distance(x, y) <span class="comment"># 25</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> scipy.spatial.distance <span class="keyword">import</span> cityblock</span><br><span class="line">cityblock(x, y) <span class="comment"># 25</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="切比雪夫距离-Chebyshev-Distance"><a href="#切比雪夫距离-Chebyshev-Distance" class="headerlink" title="切比雪夫距离(Chebyshev Distance)"></a>切比雪夫距离(Chebyshev Distance)</h2><p>数学上，切比雪夫距离或是L∞度量是向量空间中的一种度量，二个点之间的距离定义为其各座标数值差的最大值。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">chebyshev_distance</span>(<span class="params">x, y</span>):</span></span><br><span class="line"> x, y = np.array(x), np.array(y)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># d = np.linalg.norm(x - y, ord=np.inf)</span></span><br><span class="line"></span><br><span class="line"> d = np.<span class="built_in">abs</span>(x - y).<span class="built_in">max</span>()</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> d</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">x = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">y = [<span class="number">6</span>, <span class="number">7</span>, <span class="number">8</span>, <span class="number">9</span>, <span class="number">10</span>]</span><br><span class="line">chebyshev_distance(x, y) <span class="comment"># 5</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> scipy.spatial.distance <span class="keyword">import</span> chebyshev</span><br><span class="line">chebyshev(x, y) <span class="comment"># 5</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="闵可夫斯基距离-Minkowski-Distance"><a href="#闵可夫斯基距离-Minkowski-Distance" class="headerlink" title="闵可夫斯基距离(Minkowski Distance)"></a>闵可夫斯基距离(Minkowski Distance)</h2><p>明氏距离又叫做明可夫斯基距离，是欧氏空间中的一种测度，被看做是欧氏距离和曼哈顿距离的一种推广。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> math <span class="keyword">import</span> *</span><br><span class="line"><span class="keyword">from</span> decimal <span class="keyword">import</span> Decimal</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">nth_root</span>(<span class="params">value, n_root</span>):</span></span><br><span class="line"> root_value = <span class="number">1</span> / <span class="built_in">float</span>(n_root)</span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">round</span>(Decimal(value)**Decimal(root_value), <span class="number">3</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">minkowski_distance</span>(<span class="params">x, y, p_value</span>):</span></span><br><span class="line"> d = nth_root(<span class="built_in">sum</span>(<span class="built_in">pow</span>(<span class="built_in">abs</span>(a - b), p_value) <span class="keyword">for</span> a, b <span class="keyword">in</span> <span class="built_in">zip</span>(x, y)), p_value)</span><br><span class="line"> <span class="keyword">return</span> d</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">x = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">y = [<span class="number">6</span>, <span class="number">7</span>, <span class="number">8</span>, <span class="number">9</span>, <span class="number">10</span>]</span><br><span class="line">minkowski_distance(x, y, <span class="number">1</span>) <span class="comment"># Decimal('25.000')</span></span><br><span class="line">minkowski_distance(x, y, <span class="number">2</span>) <span class="comment"># Decimal('11.180')</span></span><br><span class="line">minkowski_distance(x, y, <span class="number">3</span>) <span class="comment"># Decimal('8.550')</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">minkowski_distance</span>(<span class="params">x, y, <span class="built_in">ord</span>=<span class="number">1</span></span>):</span></span><br><span class="line"> <span class="comment"># ord = 1 一范数</span></span><br><span class="line"> <span class="comment"># ord = 2 二范数</span></span><br><span class="line"> <span class="comment"># ord = np.inf 无穷范数</span></span><br><span class="line"> d = np.linalg.norm(x - y, <span class="built_in">ord</span>=<span class="built_in">ord</span>)</span><br><span class="line"> <span class="keyword">return</span> d</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> scipy.spatial.distance <span class="keyword">import</span> minkowski</span><br><span class="line">minkowski(x, y, p=<span class="number">1</span>) <span class="comment"># 25.0</span></span><br><span class="line">minkowski(x, y, p=<span class="number">2</span>) <span class="comment"># 11.180339887498949</span></span><br><span class="line">minkowski(x, y, p=<span class="number">3</span>) <span class="comment"># 8.549879733383484</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="汉明距离-Hamming-Distance"><a href="#汉明距离-Hamming-Distance" class="headerlink" title="汉明距离(Hamming Distance)"></a>汉明距离(Hamming Distance)</h2><p>在信息论中，两个等长字符串之间的汉明距离是两个字符串对应位置的不同字符的个数。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> numpy <span class="keyword">as</span> np</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">hamming_distance</span>(<span class="params">x, y</span>):</span></span><br><span class="line"> x, y = np.array(x), np.array(y)</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 方法一</span></span><br><span class="line"> <span class="comment"># d = len(np.nonzero(x - y)[0])</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># 方法二</span></span><br><span class="line"> d = np.shape(np.nonzero(x - y)[<span class="number">0</span>])[<span class="number">0</span>]</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> d</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">x = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">y = [<span class="number">6</span>, <span class="number">7</span>, <span class="number">8</span>, <span class="number">9</span>, <span class="number">10</span>]</span><br><span class="line">hamming_distance(x, y) <span class="comment"># 5</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> scipy.spatial.distance <span class="keyword">import</span> hamming</span><br><span class="line">hamming(x, y) <span class="comment"># 1.0</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="余弦相似度-Cosine-Similarity"><a href="#余弦相似度-Cosine-Similarity" class="headerlink" title="余弦相似度(Cosine Similarity)"></a>余弦相似度(Cosine Similarity)</h2><p>余弦相似性通过测量两个向量的夹角的余弦值来度量它们之间的相似性。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">cosine_distance</span>(<span class="params">x, y</span>):</span></span><br><span class="line"> x, y = np.array(x), np.array(y)</span><br><span class="line"></span><br><span class="line"> d = np.dot(x, y) / (np.linalg.norm(x) * np.linalg.norm(y))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> d</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">x = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">y = [<span class="number">6</span>, <span class="number">7</span>, <span class="number">8</span>, <span class="number">9</span>, <span class="number">10</span>]</span><br><span class="line">cosine_distance(x, y) <span class="comment"># 0.9649505047327671</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> sklearn.metrics.pairwise <span class="keyword">import</span> cosine_similarity</span><br><span class="line">x, y = np.array(x), np.array(y)</span><br><span class="line">cosine_similarity(x.reshape(<span class="number">1</span>, -<span class="number">1</span>), y.reshape(<span class="number">1</span>, -<span class="number">1</span>)) <span class="comment"># array([[0.9649505]])</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="皮尔森相关系数-Pearson-Correlation-Coefficient"><a href="#皮尔森相关系数-Pearson-Correlation-Coefficient" class="headerlink" title="皮尔森相关系数(Pearson Correlation Coefficient)"></a>皮尔森相关系数(Pearson Correlation Coefficient)</h2><p>用于度量两个变量X和Y之间的相关程度，其值介于-1与1之间。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> scipy.stats <span class="keyword">import</span> pearsonr</span><br><span class="line">x = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line">y = [<span class="number">10</span>, <span class="number">9</span>, <span class="number">2.5</span>, <span class="number">6</span>, <span class="number">4</span>]</span><br><span class="line">coe, pv = pearsonr(x, y) <span class="comment"># coe = -0.7426106572325057, pv = 0.1505558088534455</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="杰卡德相似系数-Jaccard-Similarity-Coefficient-及杰卡德距离-Jaccard-Distance"><a href="#杰卡德相似系数-Jaccard-Similarity-Coefficient-及杰卡德距离-Jaccard-Distance" class="headerlink" title="杰卡德相似系数(Jaccard Similarity Coefficient)及杰卡德距离(Jaccard Distance)"></a>杰卡德相似系数(Jaccard Similarity Coefficient)及杰卡德距离(Jaccard Distance)</h2><p>Jaccard 相似指数用来度量两个集合之间的相似性，它被定义为两个集合交集的元素个数除以并集的元素个数。</p>
<p>Jaccard 距离用来度量两个集合之间的差异性，它是 Jaccard 的相似系数的补集，被定义为 1 减去 Jaccard 相似系数。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> scipy.spatial.distance <span class="keyword">import</span> jaccard</span><br><span class="line"></span><br><span class="line">x = np.array([<span class="number">1</span>, <span class="number">1</span>, <span class="number">0</span>, <span class="number">1</span>, <span class="number">0</span>, <span class="number">1</span>, <span class="number">0</span>, <span class="number">0</span>, <span class="number">1</span>])</span><br><span class="line">y = np.array([<span class="number">0</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">0</span>, <span class="number">0</span>, <span class="number">0</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>])</span><br><span class="line">jaccard(x, y) <span class="comment"># 0.75</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
</tags>
</entry>
<entry>
<title>Session 、Cookie 和 JWT 详解</title>
<url>/2021/04/17/session-cookie-jwt/</url>
<content><![CDATA[<!-- toc -->
<h2 id="什么是-Cookie"><a href="#什么是-Cookie" class="headerlink" title="什么是 Cookie"></a>什么是 Cookie</h2><p>HTTP Cookie（也叫 Web Cookie 或浏览器 Cookie）是服务器发送到用户浏览器并保存在本地的一小块数据，它会在浏览器下次向同一服务器再发起请求时被携带并发送到服务器上。通常，它用于告知服务端两个请求是否来自同一浏览器，如保持用户的登录状态。Cookie 使基于无状态的HTTP协议记录稳定的状态信息成为了可能。</p>
<p>Cookie 主要用于以下三个方面：</p>
<ol>
<li>会话状态管理（如用户登录状态、购物车、游戏分数或其它需要记录的信息）</li>
<li>个性化设置（如用户自定义设置、主题等）</li>
<li>浏览器行为跟踪（如跟踪分析用户行为等）</li>
</ol>
<a id="more"></a>
<h2 id="什么是-Session"><a href="#什么是-Session" class="headerlink" title="什么是 Session"></a>什么是 Session</h2><p>Session 代表着服务器和客户端一次会话的过程。Session 对象存储特定用户会话所需的属性及配置信息。这样，当用户在应用程序的 Web 页之间跳转时，存储在 Session 对象中的变量将不会丢失，而是在整个用户会话中一直存在下去。当客户端关闭会话，或者 Session 超时失效时会话结束。</p>
<h2 id="Session-和-Cookie-的区别"><a href="#Session-和-Cookie-的区别" class="headerlink" title="Session 和 Cookie 的区别"></a>Session 和 Cookie 的区别</h2><ol>
<li>作用范围不同，Cookie 保存在客户端（浏览器），Session 保存在服务器端。</li>
<li>存取方式的不同，Cookie 只能保存 ASCII，Session 可以存任意数据类型，一般情况下我们可以在 Session 中保持一些常用变量信息，比如说 UserId 等。</li>
<li>有效期不同，Cookie 可设置为长时间保持，比如我们经常使用的默认登录功能，Session 一般失效时间较短，客户端关闭或者 Session 超时都会失效。</li>
<li>隐私策略不同，Cookie 存储在客户端，比较容易遭到不法获取，早期有人将用户的登录名和密码存储在 Cookie 中导致信息被窃取；Session 存储在服务端，安全性相对 Cookie 要好一些。</li>
<li>存储大小不同， 单个 Cookie 保存的数据不能超过 4K，Session 可存储数据远高于 Cookie。</li>
<li>Cookie 是客户端技术，Session 是服务端技术，Session 借助 Cookie 存储 sessionid。</li>
<li>如果浏览器禁用 Cookie，可以使用 POST 提交 sessionid 或者使用 JWT 技术。</li>
</ol>
<h2 id="分布式-Session"><a href="#分布式-Session" class="headerlink" title="分布式 Session"></a>分布式 Session</h2><p>在互联网公司为了可以支撑更大的流量，后端往往需要多台服务器共同来支撑前端用户请求，那如果用户在 A 服务器登录了，第二次请求跑到服务 B 就会出现登录失效问题。</p>
<p>分布式 Session 一般会有以下几种解决方案：</p>
<ol>
<li>Nginx ip_hash 策略，服务端使用 Nginx 代理，每个请求按访问 IP 的 hash 分配，这样来自同一 IP 固定访问一个后台服务器，避免了在服务器 A 创建 Session，第二次分发到服务器 B 的现象。</li>
<li>Session 复制，任何一个服务器上的 Session 发生改变（增删改），该节点会把这个 Session 的所有内容序列化，然后广播给所有其它节点。</li>
<li>共享 Session，服务端无状态话，将用户的 Session 等信息使用缓存中间件来统一管理，保障分发到每一个服务器的响应结果都一致。</li>
</ol>
<h2 id="跨域请求"><a href="#跨域请求" class="headerlink" title="跨域请求"></a>跨域请求</h2><p>同源策略/SOP（Same origin policy）是一种约定，由 Netscape 公司 1995年引入浏览器，它是浏览器最核心也最基本的安全功能，如果缺少了同源策略，浏览器很容易受到 XSS、CSFR 等攻击。所谓同源是指”协议+域名+端口”三者相同，即便两个不同的域名指向同一个 ip 地址，也非同源。</p>
<p>解决方案：</p>
<ol>
<li>Nginx 代理</li>
<li>Jsonp 跨域</li>
<li>JWT 认证</li>
</ol>
<h2 id="JWT-数据结构"><a href="#JWT-数据结构" class="headerlink" title="JWT 数据结构"></a>JWT 数据结构</h2><h3 id="Encoded"><a href="#Encoded" class="headerlink" title="Encoded"></a>Encoded</h3><figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c</span><br></pre></td></tr></table></figure>
<h3 id="Decoded"><a href="#Decoded" class="headerlink" title="Decoded"></a>Decoded</h3><h4 id="Header-头部"><a href="#Header-头部" class="headerlink" title="Header (头部)"></a>Header (头部)</h4><figure class="highlight json"><table><tr><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"alg"</span>: <span class="string">"HS256"</span>,</span><br><span class="line"> <span class="attr">"typ"</span>: <span class="string">"JWT"</span>,</span><br><span class="line"> <span class="attr">"kid"</span>: <span class="string">"Key ID"</span>,</span><br><span class="line"> <span class="attr">"cty"</span>: <span class="string">"Content Type"</span>,</span><br><span class="line"> <span class="attr">"enc"</span>: <span class="string">"Encrypt Algorithm"</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="Payload-负载"><a href="#Payload-负载" class="headerlink" title="Payload (负载)"></a>Payload (负载)</h4><figure class="highlight json"><table><tr><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"iss"</span>: <span class="string">"issuer(签发人)"</span>,</span><br><span class="line"> <span class="attr">"exp"</span>: <span class="string">"expiration time(过期时间)"</span>,</span><br><span class="line"> <span class="attr">"sub"</span>: <span class="string">"subject(主题)"</span>,</span><br><span class="line"> <span class="attr">"aud"</span>: <span class="string">"audience(受众)"</span>,</span><br><span class="line"> <span class="attr">"nbf"</span>: <span class="string">"Not Before(生效时间)"</span>,</span><br><span class="line"> <span class="attr">"iat"</span>: <span class="string">"Issued At(签发时间)"</span>,</span><br><span class="line"> <span class="attr">"jti"</span>: <span class="string">"JWT ID(编号)"</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="Signature-签名"><a href="#Signature-签名" class="headerlink" title="Signature (签名)"></a>Signature (签名)</h4><figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">HMACSHA256(</span><br><span class="line"> base64UrlEncode(header) + "." +</span><br><span class="line"> base64UrlEncode(payload),</span><br><span class="line"></span><br><span class="line">your-256-bit-secret</span><br><span class="line"></span><br><span class="line">) secret base64 encoded</span><br></pre></td></tr></table></figure>
<p>JWT = Header.Payload.Signature</p>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>JWT</tag>
<tag>Session</tag>
<tag>Cookie</tag>
</tags>
</entry>
<entry>
<title>Sqlalchemy 基本用法</title>
<url>/2019/05/12/sqlalchemy-usage/</url>
<content><![CDATA[<p>Sqlalchemy 基本用法</p>
<a id="more"></a>
<!-- toc -->
<h2 id="通用导入"><a href="#通用导入" class="headerlink" title="通用导入"></a>通用导入</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> sqlalchemy <span class="keyword">import</span> create_engine</span><br><span class="line"><span class="keyword">from</span> sqlalchemy.orm <span class="keyword">import</span> scoped_session, sessionmaker</span><br><span class="line"><span class="keyword">from</span> sqlalchemy <span class="keyword">import</span> Column, Integer, String, ForeignKey, Boolean</span><br><span class="line"><span class="keyword">from</span> sqlalchemy.orm <span class="keyword">import</span> relationship</span><br><span class="line"><span class="keyword">from</span> sqlalchemy.ext.declarative <span class="keyword">import</span> declarative_base</span><br><span class="line"></span><br><span class="line">engine = create_engine(<span class="string">'sqlite:///test.db'</span>, echo=<span class="literal">True</span>)</span><br><span class="line">Base = declarative_base()</span><br><span class="line">db_session = scoped_session(sessionmaker(bind=engine))</span><br><span class="line">Base.query = db_session.query_property()</span><br></pre></td></tr></table></figure>
<h2 id="一对一"><a href="#一对一" class="headerlink" title="一对一"></a>一对一</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Parent</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'parent'</span></span><br><span class="line"></span><br><span class="line"> <span class="built_in">id</span> = Column(Integer, primary_key=<span class="literal">True</span>)</span><br><span class="line"> name = Column(String)</span><br><span class="line"> child_id = Column(Integer, ForeignKey(<span class="string">'child.id'</span>))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Child</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'child'</span></span><br><span class="line"></span><br><span class="line"> <span class="built_in">id</span> = Column(Integer, primary_key=<span class="literal">True</span>)</span><br><span class="line"> name = Column(String)</span><br><span class="line"> parent = relationship(<span class="string">'Parent'</span>, backref=<span class="string">'child'</span>, uselist=<span class="literal">False</span>)</span><br></pre></td></tr></table></figure>
<h2 id="一对多"><a href="#一对多" class="headerlink" title="一对多"></a>一对多</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># the one side</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Parent</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'parent'</span></span><br><span class="line"></span><br><span class="line"> <span class="built_in">id</span> = Column(Integer, primary_key=<span class="literal">True</span>)</span><br><span class="line"> name = Column(String)</span><br><span class="line"> <span class="comment"># children = relationship("Child", back_populates="parent")</span></span><br><span class="line"> children = relationship(<span class="string">"Child"</span>, backref=<span class="string">"parent"</span>, lazy=<span class="string">"dynamic"</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># the many side</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Child</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'child'</span></span><br><span class="line"></span><br><span class="line"> <span class="built_in">id</span> = Column(Integer, primary_key=<span class="literal">True</span>)</span><br><span class="line"> name = Column(String)</span><br><span class="line"> parent_id = Column(Integer, ForeignKey(<span class="string">'parent.id'</span>))</span><br><span class="line"> <span class="comment"># parent = relationship("Parent", back_populates="children")</span></span><br><span class="line"> <span class="comment"># parent = relationship("Parent", backref="children")</span></span><br></pre></td></tr></table></figure>
<h2 id="多对一"><a href="#多对一" class="headerlink" title="多对一"></a>多对一</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="comment"># the many side</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Parent</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'parent'</span></span><br><span class="line"></span><br><span class="line"> <span class="built_in">id</span> = Column(Integer, primary_key=<span class="literal">True</span>)</span><br><span class="line"> name = Column(String)</span><br><span class="line"> child_id = Column(Integer, ForeignKey(<span class="string">'child.id'</span>))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># the one side</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Child</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'child'</span></span><br><span class="line"></span><br><span class="line"> <span class="built_in">id</span> = Column(Integer, primary_key=<span class="literal">True</span>)</span><br><span class="line"> name = Column(String)</span><br><span class="line"> parents = relationship(<span class="string">'Parent'</span>, backref=<span class="string">'child'</span>, lazy=<span class="string">'dynamic'</span>)</span><br></pre></td></tr></table></figure>
<h2 id="多对多"><a href="#多对多" class="headerlink" title="多对多"></a>多对多</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Department</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'department'</span></span><br><span class="line"> <span class="built_in">id</span> = Column(Integer, primary_key=<span class="literal">True</span>)</span><br><span class="line"> name = Column(String)</span><br><span class="line"> employees = relationship(</span><br><span class="line"> <span class="string">'Employee'</span>,</span><br><span class="line"> secondary=<span class="string">'department_employee_link'</span></span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Employee</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'employee'</span></span><br><span class="line"> <span class="built_in">id</span> = Column(Integer, primary_key=<span class="literal">True</span>)</span><br><span class="line"> name = Column(String)</span><br><span class="line"> hired_on = Column(</span><br><span class="line"> DateTime,</span><br><span class="line"> default=func.now())</span><br><span class="line"> departments = relationship(</span><br><span class="line"> <span class="string">'Department'</span>,</span><br><span class="line"> secondary=<span class="string">'department_employee_link'</span></span><br><span class="line"> )</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">DepartmentEmployeeLink</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'department_employee_link'</span></span><br><span class="line"> department_id = Column(Integer, ForeignKey(<span class="string">'department.id'</span>),</span><br><span class="line"> primary_key=<span class="literal">True</span>)</span><br><span class="line"> department = relationship(<span class="string">'Department'</span>)</span><br><span class="line"> employee_id = Column(Integer, ForeignKey(<span class="string">'employee.id'</span>), primary_key=<span class="literal">True</span>)</span><br><span class="line"> employee = relationship(<span class="string">'Employee'</span>)</span><br></pre></td></tr></table></figure>
<h2 id="自身多对多"><a href="#自身多对多" class="headerlink" title="自身多对多"></a>自身多对多</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Follow</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'me_follow_you'</span></span><br><span class="line"></span><br><span class="line"> me_id = Column(Integer, ForeignKey(<span class="string">'users.id'</span>), primary_key=<span class="literal">True</span>)</span><br><span class="line"> me = relationship(<span class="string">'User'</span>, foreign_keys=[me_id])</span><br><span class="line"> you_id = Column(Integer, ForeignKey(<span class="string">'users.id'</span>), primary_key=<span class="literal">True</span>)</span><br><span class="line"> you = relationship(<span class="string">'User'</span>, foreign_keys=[you_id])</span><br><span class="line"> created = Column(DateTime(timezone=<span class="literal">True</span>), default=datetime.utcnow)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">User</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'users'</span></span><br><span class="line"></span><br><span class="line"> <span class="built_in">id</span> = Column(Integer, primary_key=<span class="literal">True</span>)</span><br><span class="line"> name = Column(String(<span class="number">64</span>))</span><br><span class="line"></span><br><span class="line"> <span class="comment"># stars=我关注的人 fans=我的粉丝</span></span><br><span class="line"> stars = relationship(<span class="string">'User'</span>,</span><br><span class="line"> secondary=<span class="string">'me_follow_you'</span>,</span><br><span class="line"> primaryjoin=<span class="string">'User.id==Follow.me_id'</span>,</span><br><span class="line"> secondaryjoin=<span class="string">'User.id==Follow.you_id'</span>,</span><br><span class="line"> backref=backref(<span class="string">'fans'</span>, lazy=<span class="string">'dynamic'</span>),</span><br><span class="line"> lazy=<span class="string">'dynamic'</span>)</span><br></pre></td></tr></table></figure>
<h2 id="自身一对一"><a href="#自身一对一" class="headerlink" title="自身一对一"></a>自身一对一</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Node</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'node'</span></span><br><span class="line"> <span class="built_in">id</span> = Column(Integer, primary_key=<span class="literal">True</span>)</span><br><span class="line"> parent_id = Column(Integer, ForeignKey(<span class="string">'node.id'</span>))</span><br><span class="line"> data = Column(String(<span class="number">50</span>))</span><br><span class="line"> parent = relationship(<span class="string">"Node"</span>, remote_side=[<span class="built_in">id</span>])</span><br></pre></td></tr></table></figure>
<h2 id="创建表"><a href="#创建表" class="headerlink" title="创建表"></a>创建表</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">init_db</span>():</span></span><br><span class="line"> <span class="keyword">from</span> sqlalchemy <span class="keyword">import</span> create_engine</span><br><span class="line"> engine = create_engine(<span class="string">'sqlite:///test.db'</span>, echo=<span class="literal">True</span>)</span><br><span class="line"> Base.metadata.create_all(engine)</span><br></pre></td></tr></table></figure>
<h2 id="backref-和-back-populates"><a href="#backref-和-back-populates" class="headerlink" title="backref 和 back_populates"></a>backref 和 back_populates</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line">Parent 下添加 `children = relationship(<span class="string">"Child"</span>, back_populates=<span class="string">"parent"</span>)`</span><br><span class="line"></span><br><span class="line">创建p1 = Parent()和c1 = Child()失败，原因是One <span class="keyword">or</span> more mappers failed to initialize，即back_populates必须在关系两端同时指定</span><br><span class="line"></span><br><span class="line">Parent下添加 `children = relationship(<span class="string">"Child"</span>, back_populates=<span class="string">"parent"</span>)`</span><br><span class="line">Child下添加 `parent = relationship(<span class="string">"Parent"</span>, back_populates=<span class="string">"children"</span>)`</span><br><span class="line"></span><br><span class="line">Parent Attribute:</span><br><span class="line">Parent.children Parent.<span class="built_in">id</span> Parent.metadata Parent.name Parent.query</span><br><span class="line"></span><br><span class="line">Child Attribute:</span><br><span class="line">Child.<span class="built_in">id</span> Child.metadata Child.name Child.parent Child.parent_id Child.query</span><br><span class="line"></span><br><span class="line">p1 = Parent()</span><br><span class="line">c1 = Child()</span><br><span class="line">c1.parent = p1 <span class="keyword">or</span> p1.children.append(c1)</span><br><span class="line"></span><br><span class="line">Parent下添加 `children = relationship(<span class="string">"Child"</span>, backref=<span class="string">"parent"</span>)`</span><br><span class="line"></span><br><span class="line">Parent Attribute:</span><br><span class="line">Parent.children Parent.<span class="built_in">id</span> Parent.metadata Parent.name Parent.query</span><br><span class="line"></span><br><span class="line">Child Attribute:</span><br><span class="line">Child.<span class="built_in">id</span> Child.metadata Child.name Child.parent_id Child.query</span><br><span class="line"></span><br><span class="line">p1 = Parent()</span><br><span class="line">c1 = Child()</span><br><span class="line">c1.parent = p1 <span class="keyword">or</span> p1.children.append(c1)</span><br><span class="line"></span><br><span class="line">可以看出使用backref时，实例化c1时会自动在c1对象上添加parent属性</span><br><span class="line"></span><br><span class="line">此后再检查:</span><br><span class="line"><span class="built_in">hasattr</span>(Child, <span class="string">'parent'</span>) // <span class="literal">True</span></span><br><span class="line"><span class="built_in">hasattr</span>(c1, <span class="string">'parent'</span>) // <span class="literal">True</span></span><br><span class="line"><span class="built_in">hasattr</span>(Parent, <span class="string">'children'</span>) // <span class="literal">True</span></span><br><span class="line"><span class="built_in">hasattr</span>(p1, <span class="string">'children'</span>) // <span class="literal">True</span></span><br><span class="line"></span><br><span class="line">Child 下添加 `parent = relationship(<span class="string">"Parent"</span>, backref=<span class="string">"children"</span>)` 情况和 <span class="number">3</span> 相同</span><br><span class="line">Parent下添加 `children = relationship(<span class="string">"Child"</span>, backref=<span class="string">"parent"</span>)`</span><br><span class="line">Child下添加 `parent = relationship(<span class="string">"Parent"</span>, backref=<span class="string">"children"</span>)`</span><br><span class="line"></span><br><span class="line">创建p1 = Parent()和c1 = Child()失败，原因是One <span class="keyword">or</span> more mappers failed to initialize</span><br><span class="line">因此两者只能使用其中之一</span><br><span class="line"></span><br><span class="line">lazy 指定如何加载相关记录，默认值是<span class="string">"select"</span></span><br><span class="line"> select 首次访问时按需加载</span><br><span class="line"> immediate 源对象加载后就加载</span><br><span class="line"> joined 加载记录,但使用联结</span><br><span class="line"> subquery 立即加载,但使用子查询</span><br><span class="line"> noload 永不加载</span><br><span class="line"> dynamic 不加载记录,但提供加载记录的查询</span><br><span class="line"></span><br><span class="line">lazy = <span class="string">"dynamic"</span>只能用于collections，不立即查询出结果集，而是提供一系列结果集的方法，可以基于结果集再次进行更精确的查找</span><br></pre></td></tr></table></figure>
<h2 id="default-和-server-default"><a href="#default-和-server-default" class="headerlink" title="default 和 server_default"></a>default 和 server_default</h2><ol>
<li>default 是在 ORM 层设置默认值，server_default 是在表结构上设置默认值</li>
<li>onupdate 在 ORM 层生效，server_onupdate 在数据库生效，在 MySQL 上 ON UPDATE 是MySQL在背后创建了 trigger，而在 PostgreSQL 上你必须手动创建 trigger</li>
</ol>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> datetime <span class="keyword">import</span> datetime</span><br><span class="line"><span class="keyword">from</span> sqlalchemy <span class="keyword">import</span> func, sql, text</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Record</span>(<span class="params">Base</span>):</span></span><br><span class="line"> __tablename__ = <span class="string">'records</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"> id = Column(Integer, primary_key=True)</span></span><br><span class="line"><span class="string"> name = Column(String(64), server_default=text('</span>name<span class="string">'))</span></span><br><span class="line"><span class="string"> created = Column(DateTime(timezone=True), default=datetime.utcnow)</span></span><br><span class="line"><span class="string"> # created = Column(DateTime(timezone=True), server_default=func.now())</span></span><br><span class="line"><span class="string"> # created = Column(DateTime(timezone=True), server_default=func.current_timestamp())</span></span><br><span class="line"><span class="string"> updated = Column(DateTime(timezone=True), server_default=func.current_timestamp(), onupdate=func.current_timestamp())</span></span><br><span class="line"><span class="string"> deleted = Column(Boolean, default=False)</span></span><br><span class="line"><span class="string"> # deleted = Column(Boolean, server_default=sql.expression.false())</span></span><br></pre></td></tr></table></figure>
<h2 id="为flask-sqlalchemy扩展BaseQuery方法"><a href="#为flask-sqlalchemy扩展BaseQuery方法" class="headerlink" title="为flask_sqlalchemy扩展BaseQuery方法"></a>为flask_sqlalchemy扩展BaseQuery方法</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">from</span> flask_sqlalchemy <span class="keyword">import</span> SQLAlchemy, BaseQuery</span><br><span class="line"><span class="keyword">from</span> sqlalchemy <span class="keyword">import</span> func</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">CustomQuery</span>(<span class="params">BaseQuery</span>):</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">count_all</span>(<span class="params">self</span>):</span></span><br><span class="line"> <span class="keyword">return</span> self.with_entities(func.count()).scalar()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">db = SQLAlchemy(query_class=CustomQuery)</span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Sqlalchemy</tag>
</tags>
</entry>
<entry>
<title>Instagram 的分片与 ID 设计</title>
<url>/2020/06/03/sharding-ids-at-instagram/</url>
<content><![CDATA[<!-- toc -->
<p>Instagram上有大量的数据，每分钟就有超过25张的图片和90个点赞。为了确保所有重要的数据都能被合理存储并且及时得被提取应用，我们对数据进行了分片（sharding）——也就是说，我们把数据放到多个桶（bucket）中，每个桶里都有一部分数据。</p>
<p>我们的应用服务器上运行的是Django， 后端数据库是PostgreSQL。对数据分片首先要决定是否要保留PostgreSQL作为主要的数据存储库，是否要采用其他的数据库。经过评估一些不同的数据库解决方案，我们最终确定最适合的方案是在PostgreSQL数据库集群上实现数据分片。</p>
<p>然而在把数据写到数据库之前，我们还要解决如何给数据（例如Instagram上发布的没一张图片）加上唯一标识符的问题。在单一数据库上的典型解法——使用数据库自带的自增主键功能——在当数据需要被同时插入到多个数据库时就不适用了。文章的下面就来讲讲我们是如何解决这个问题的</p>
<a id="more"></a>
<p>开始前，我们先列出系统所需要的所有重要的功能。</p>
<blockquote>
<ol>
<li>产生的数据ID需是可以按时间排序的。（比如对一列图片数据的ID进行排序，可以不需要提取太多图片本身的信息）</li>
<li>理想的ID是64位的。（这样索引更小，存储也更优，像Redis）</li>
<li>系统要尽量少的引用“可变动因素”——在很少工程师的情况下还可以扩张Instagram的很大一部分原因就是，我们相信简单易懂的方案。</li>
</ol>
</blockquote>
<h2 id="现有的解决方案"><a href="#现有的解决方案" class="headerlink" title="现有的解决方案"></a>现有的解决方案</h2><h3 id="由Web应用层生成ID"><a href="#由Web应用层生成ID" class="headerlink" title="由Web应用层生成ID"></a>由Web应用层生成ID</h3><p>这种方案将ID的生成完全交到应用层，而不是数据库。例如，MongoDB的ObjectId，就是12字节长并且在最前面加上时间戳进行编码。另一个流行的方案是使用UUIDs。</p>
<p>优点：</p>
<blockquote>
<ol>
<li>每个应用线程独立生成ID，最小的降低ID生成的失败和竞争。</li>
<li>如果用时间戳作为ID的起始部分，那么ID可以按时间排序。</li>
</ol>
</blockquote>
<p>缺点：</p>
<blockquote>
<ol>
<li>通常需要更多的存储空间（96位或者更多）来确保ID的合理唯一性。</li>
<li>一些UUID类型完全是随机的，无法排序。</li>
</ol>
</blockquote>
<h3 id="通过单独的服务产生ID"><a href="#通过单独的服务产生ID" class="headerlink" title="通过单独的服务产生ID"></a>通过单独的服务产生ID</h3><p>例如：Twitter的Snowflake，是一个Thrift服务，使用了Apache Zookeeper来协调各个结点并且产生64为的唯一ID。</p>
<p>优点：</p>
<blockquote>
<ol>
<li>Snowflake的ID只有64位，是UUID的一半。</li>
<li>可以放时间戳到ID头，从而可以按时间排序。</li>
<li>分布式系统保证了系统结点不会挂掉。</li>
</ol>
</blockquote>
<p>缺点：</p>
<blockquote>
<p>增加了复杂性，而且引入了更多的“可变动因素”（如ZooKeeper, Snowflake服务器）到系统构架中。</p>
</blockquote>
<h3 id="数据库票据（DB-Ticket）服务器"><a href="#数据库票据（DB-Ticket）服务器" class="headerlink" title="数据库票据（DB Ticket）服务器"></a>数据库票据（DB Ticket）服务器</h3><p>利用数据库自带的自增特性来确保唯一性。Flicker采用这一方法，不过还用了两台ticket数据库（一个生成偶数，一个生成奇数）来避免单点失败。</p>
<p>优点：</p>
<blockquote>
<p>数据库好理解，带有易预测的可扩张功能。</p>
</blockquote>
<p>缺点：</p>
<blockquote>
<ol>
<li>最终会出现数据写入的瓶颈（尽管Flicker称在高扩展下没有问题）。</li>
<li>需要管理多的两台服务器（或者EC2实例）。</li>
<li>如果单独使用数据库，会出现单点失效。如果使用多个数据库，则不能保证ID可以按时间排序。</li>
</ol>
</blockquote>
<p>在所有这些方案中，Twitter的snowflake是最接近的，但是生成ID所需的添加复杂性又和我们的目标冲突。我们的替换方案是采用概念上相近的方法，但是带到PostgreSQL内部实现。</p>
<h2 id="Instagram-的方案"><a href="#Instagram-的方案" class="headerlink" title="Instagram 的方案"></a>Instagram 的方案</h2><p>我们的分片系统是由上千个“逻辑”分片组成的，由代码映射到少量的物理分片。</p>
<p>通过这个方法，我们一开始用少数数据库实现，慢慢扩展到更多个数据库，只需要把部分逻辑分片从一台数据库转移到另一台数据库里，不需要重新把数据重新聚合。我们用到的PostgreSQL的schema的特性可以轻松的实现计划和管理。</p>
<p>Schema（不要和SQL单个表的schema搞混了）是PostgreSQL里的一个逻辑分组功能。每一个PostgreSQL数据库都有好几个schema，每一个schema都有一到多个表。表名在没个schema中都是唯一的，而不是每个数据库，默认情况下，PostgreSQL会把所有数据都放在一个叫“public”的schema中。</p>
<p>在我们的系统中每个“逻辑”分片都是一个PostgreSQL schema， 每个分片的表（比如照片的“点赞”功能）都存在每个schema中。</p>
<p>我们通过使用PL/PGSQL, PostgreSQL内部的编程语言，和PostgreSQL现有的自增功能来生成ID。</p>
<p>每个ID都由下面几个部分组成：</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line">41位的毫秒级时间（用于产生41年的ID）</span><br><span class="line">13位用来表示逻辑分片的ID</span><br><span class="line">10位的自增序列，模上1024， 意味着每个分片每毫秒可以产生1024个ID</span><br></pre></td></tr></table></figure>
<p>下面通过一个例子说明：比如说现在是2011年的九月九日，我们的纪元是从2011年的一月一号开始。从新纪元开始到此有1387263000毫秒，那么我们把这个数字左移41位来填满ID的头。</p>
<blockquote>
<p>id = 1387263000 <<(64 – 41)</p>
</blockquote>
<p>接下来， 我们拿来我们准备把这个数据插入的分片ID；如果我们用户ID是31341， 这个分片的ID是 31341%2000 → 1341。 我们接下来把下面13为填满：</p>
<blockquote>
<p>id |= 1341 << (64-41-13)</p>
</blockquote>
<p>最后， 我们用所剩的自增序列（每个schema中每个表中这个序列都是唯一的）来填满的后面的位数。 假设这张表中已经有了5000个ID； 我们下一个数据就是5001，我们把它模上1024得到：</p>
<blockquote>
<p>id |= （5001%1024）</p>
</blockquote>
<p>我们就得到了我们的ID， 我们把这个id作为insert中的RETURNING返回给应用层。</p>
<p>下面是是实现以上过程的PL/PGSQL代码（这里用的schema是insta5）</p>
<figure class="highlight sql"><table><tr><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">OR</span> <span class="keyword">REPLACE</span> <span class="keyword">FUNCTION</span> insta5.next_id(<span class="keyword">OUT</span> <span class="keyword">result</span> <span class="built_in">bigint</span>) <span class="keyword">AS</span> $$</span><br><span class="line"><span class="keyword">DECLARE</span></span><br><span class="line"> our_epoch <span class="built_in">bigint</span> := <span class="number">1314220021721</span>;</span><br><span class="line"> seq_id bigint;</span><br><span class="line"> now_millis bigint;</span><br><span class="line"> shard_id int := 5;</span><br><span class="line"><span class="keyword">BEGIN</span></span><br><span class="line"> <span class="keyword">SELECT</span> <span class="keyword">nextval</span>(<span class="string">'insta5.table_id_seq'</span>) %% <span class="number">1024</span> <span class="keyword">INTO</span> seq_id;</span><br><span class="line"> <span class="keyword">SELECT</span> <span class="keyword">FLOOR</span>(<span class="keyword">EXTRACT</span>(EPOCH <span class="keyword">FROM</span> clock_timestamp()) * <span class="number">1000</span>) <span class="keyword">INTO</span> now_millis;</span><br><span class="line"> result := (now_millis - our_epoch) << 23;</span><br><span class="line"> result := result | (shard_id << 10);</span><br><span class="line"> result := result | (seq_id);</span><br><span class="line"><span class="keyword">END</span>;</span><br><span class="line">$$ LANGUAGE PLPGSQL;</span><br></pre></td></tr></table></figure>
<p>要生成表示执行下面的部分：</p>
<figure class="highlight sql"><table><tr><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> insta5.our_table(</span><br><span class="line"> <span class="string">"id"</span> <span class="built_in">bigint</span> <span class="keyword">NOT</span> <span class="literal">NULL</span> <span class="keyword">DEFAULT</span> insta5.next_id(),</span><br><span class="line"> ... rest <span class="keyword">of</span> <span class="keyword">table</span> <span class="keyword">schema</span> ...</span><br><span class="line">)</span><br></pre></td></tr></table></figure>
<p>成了！我们得到了应用中唯一的主键（还有一个好处是，ID中包含了分片ID可以用来轻松映射）。我们已经把这个方法用在产品中，并且对结果感到非常满意。</p>
<p>本文来自：<a href="https://juejin.im/post/58de0676da2f60005fbec568">Instagram 的分片与 ID 设计</a></p>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Algorithm</tag>
</tags>
</entry>
<entry>
<title>Tmux 使用教程</title>
<url>/2020/05/29/tmux-tutorial/</url>
<content><![CDATA[<p>Tmux 是一个终端复用工具，和 screen 一样，screen 相对简单好使，tmux 更高级。</p>
<h2 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h2><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># CentOS</span></span><br><span class="line">yum install -y tmux</span><br><span class="line"></span><br><span class="line"><span class="comment"># MacOS</span></span><br><span class="line">brew install tmux</span><br></pre></td></tr></table></figure>
<a id="more"></a>
<!-- toc -->
<h2 id="基本操作"><a href="#基本操作" class="headerlink" title="基本操作"></a>基本操作</h2><figure class="highlight bash"><table><tr><td class="code"><pre><span class="line"><span class="comment"># 新建无名称会话</span></span><br><span class="line">tmux</span><br><span class="line"></span><br><span class="line"><span class="comment"># 新建会话</span></span><br><span class="line">tmux new -s demo</span><br><span class="line"></span><br><span class="line"><span class="comment"># 挂起会话</span></span><br><span class="line">tmux detach</span><br><span class="line"></span><br><span class="line"><span class="comment"># 默认进入第一个会话</span></span><br><span class="line">tmux a</span><br><span class="line"></span><br><span class="line"><span class="comment"># 进入名为demo的会话</span></span><br><span class="line">tmux a -t demo</span><br><span class="line"></span><br><span class="line"><span class="comment"># 关闭demo会话</span></span><br><span class="line">tmux kill-session -t demo</span><br><span class="line"></span><br><span class="line"><span class="comment"># 关闭服务器</span></span><br><span class="line">tmux kill-server</span><br><span class="line"></span><br><span class="line"><span class="comment"># 查看会话</span></span><br><span class="line">tmux list-session</span><br><span class="line">tmux ls</span><br><span class="line"></span><br><span class="line"><span class="comment"># 切换会话</span></span><br><span class="line">tmux switch -t 0 <span class="comment"># 使用会话编号</span></span><br><span class="line">tmux switch -t demo <span class="comment"># 使用会话名称</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 重命名会话</span></span><br><span class="line">tmux rename-session -t demo new-demo</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="系统指令"><a href="#系统指令" class="headerlink" title="系统指令"></a>系统指令</h2><table>
<thead>
<tr>
<th><strong>前缀</strong></th>
<th><strong>指令</strong></th>
<th><strong>描述</strong></th>
</tr>
</thead>
<tbody><tr>
<td>prefix</td>
<td>?</td>
<td>显示快捷键帮助文档</td>
</tr>
<tr>
<td>prefix</td>
<td>d</td>
<td>断开当前会话</td>
</tr>
<tr>
<td>prefix</td>
<td>D</td>
<td>选择要断开的会话</td>
</tr>
<tr>
<td>prefix</td>
<td>Ctrl+z</td>
<td>挂起当前会话</td>
</tr>
<tr>
<td>prefix</td>
<td>r</td>
<td>强制重载当前会话</td>
</tr>
<tr>
<td>prefix</td>
<td>s</td>
<td>显示会话列表用于选择并切换</td>
</tr>
<tr>
<td>prefix</td>
<td>:</td>
<td>进入命令行模式</td>
</tr>
<tr>
<td>prefix</td>
<td>[</td>
<td>进入复制模式，按q退出</td>
</tr>
<tr>
<td>prefix</td>
<td>]</td>
<td>粘贴复制模式中复制的文本</td>
</tr>
<tr>
<td>prefix</td>
<td>~</td>
<td>列出提示信息缓存</td>
</tr>
</tbody></table>
<h2 id="窗口指令"><a href="#窗口指令" class="headerlink" title="窗口指令"></a>窗口指令</h2><table>
<thead>
<tr>
<th><strong>前缀</strong></th>
<th><strong>指令</strong></th>
<th><strong>描述</strong></th>
</tr>
</thead>
<tbody><tr>
<td>prefix</td>
<td>c</td>
<td>新建窗口</td>
</tr>
<tr>
<td>prefix</td>
<td>&</td>
<td>关闭当前窗口（关闭前需输入y or n确认）</td>
</tr>
<tr>
<td>prefix</td>
<td>0-9</td>
<td>切换到指定窗口</td>
</tr>
<tr>
<td>prefix</td>
<td>p</td>
<td>切换到上一窗口</td>
</tr>
<tr>
<td>prefix</td>
<td>n</td>
<td>切换到下一窗口</td>
</tr>
<tr>
<td>prefix</td>
<td>w</td>
<td>打开窗口列表，用于且切换窗口</td>
</tr>
<tr>
<td>prefix</td>
<td>,</td>
<td>重命名当前窗口</td>
</tr>
<tr>
<td>prefix</td>
<td>.</td>
<td>修改当前窗口编号（适用于窗口重新排序）</td>
</tr>
<tr>
<td>prefix</td>
<td>f</td>
<td>快速定位到窗口（输入关键字匹配窗口名称）</td>
</tr>
</tbody></table>
<h2 id="面板指令"><a href="#面板指令" class="headerlink" title="面板指令"></a>面板指令</h2><table>
<thead>
<tr>
<th><strong>前缀</strong></th>
<th><strong>指令</strong></th>
<th><strong>描述</strong></th>
</tr>
</thead>
<tbody><tr>
<td>prefix</td>
<td>“</td>
<td>当前面板上下一分为二，下侧新建面板</td>
</tr>
<tr>
<td>prefix</td>
<td>%</td>
<td>当前面板左右一分为二，右侧新建面板</td>
</tr>
<tr>
<td>prefix</td>
<td>x</td>
<td>关闭当前面板（关闭前需输入y or n确认）</td>
</tr>
<tr>
<td>prefix</td>
<td>z</td>
<td>最大化当前面板，再重复一次按键后恢复正常（v1.8版本新增）</td>
</tr>
<tr>
<td>prefix</td>
<td>!</td>
<td>将当前面板移动到新的窗口打开（原窗口中存在两个及以上面板有效）</td>
</tr>
<tr>
<td>prefix</td>
<td>;</td>
<td>切换到最后一次使用的面板</td>
</tr>
<tr>
<td>prefix</td>
<td>q</td>
<td>显示面板编号，在编号消失前输入对应的数字可切换到相应的面板</td>
</tr>
<tr>
<td>prefix</td>
<td>{</td>
<td>向前置换当前面板</td>
</tr>
<tr>
<td>prefix</td>
<td>}</td>
<td>向后置换当前面板</td>
</tr>
<tr>
<td>prefix</td>
<td>Ctrl+o</td>
<td>顺时针旋转当前窗口中的所有面板</td>
</tr>
<tr>
<td>prefix</td>
<td>方向键</td>
<td>移动光标切换面板</td>
</tr>
<tr>
<td>prefix</td>
<td>o</td>
<td>选择下一面板</td>
</tr>
<tr>
<td>prefix</td>
<td>空格键</td>
<td>在自带的面板布局中循环切换</td>
</tr>
<tr>
<td>prefix</td>
<td>Alt+方向键</td>
<td>以5个单元格为单位调整当前面板边缘</td>
</tr>
<tr>
<td>prefix</td>
<td>Ctrl+方向键</td>
<td>以1个单元格为单位调整当前面板边缘（Mac下被系统快捷键覆盖）</td>
</tr>
<tr>
<td>prefix</td>
<td>t</td>
<td>显示时钟</td>
</tr>
</tbody></table>
<h2 id="自定义配置"><a href="#自定义配置" class="headerlink" title="自定义配置"></a>自定义配置</h2><p>编辑 ~/.tmux.conf 添加如下内容</p>
<figure class="highlight text"><table><tr><td class="code"><pre><span class="line"># prefix configuration</span><br><span class="line">set -g prefix C-a</span><br><span class="line">unbind C-b</span><br><span class="line">bind C-a send-prefix</span><br><span class="line"></span><br><span class="line"># split window</span><br><span class="line">unbind '"'</span><br><span class="line">bind - split-window -v -c '#{pane_current_path}'</span><br><span class="line">unbind %</span><br><span class="line">bind = split-window -h -c '#{pane_current_path}'</span><br><span class="line"></span><br><span class="line"># mouse on</span><br><span class="line">set-option -g mouse on</span><br><span class="line"></span><br><span class="line"># pane navigation</span><br><span class="line">bind -r k select-pane -U</span><br><span class="line">bind -r j select-pane -D</span><br><span class="line">bind -r h select-pane -L</span><br><span class="line">bind -r l select-pane -R</span><br><span class="line"></span><br><span class="line"># pane resizing</span><br><span class="line">bind -r ^k resize-pane -U 2</span><br><span class="line">bind -r ^j resize-pane -D 2</span><br><span class="line">bind -r ^h resize-pane -L 2</span><br><span class="line">bind -r ^l resize-pane -R 2</span><br><span class="line"></span><br><span class="line"># reload configuration</span><br><span class="line">bind r source-file ~/.tmux.conf \; display '~/.tmux.conf sourced'</span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>CentOS</tag>
</tags>
</entry>
<entry>
<title>二叉树</title>
<url>/2019/01/15/treenode/</url>
<content><![CDATA[<p><code>LeetCode</code> 二叉树题解汇总</p>
<a id="more"></a>
<!-- toc -->
<h2 id="二叉树定义"><a href="#二叉树定义" class="headerlink" title="二叉树定义"></a>二叉树定义</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">TreeNode</span>:</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span>(<span class="params">self, x</span>):</span></span><br><span class="line"> self.val = x</span><br><span class="line"> self.left = <span class="literal">None</span></span><br><span class="line"> self.right = <span class="literal">None</span></span><br></pre></td></tr></table></figure>
<h2 id="相同的树"><a href="#相同的树" class="headerlink" title="相同的树"></a>相同的树</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">is_same_tree</span>(<span class="params">p, q</span>):</span></span><br><span class="line"> <span class="keyword">if</span> p <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">not</span> q</span><br><span class="line"> <span class="keyword">if</span> q <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">not</span> p</span><br><span class="line"> <span class="keyword">return</span> p.val == q.val <span class="keyword">and</span> is_same_tree(p.left, q.left) <span class="keyword">and</span> is_same_tree(p.right, q.right)</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="对称的树"><a href="#对称的树" class="headerlink" title="对称的树"></a>对称的树</h2><figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">is_symmetric</span>(<span class="params">root</span>):</span></span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> root:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">True</span></span><br><span class="line"> <span class="keyword">return</span> symmetric(root.left, root.right)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">symmetric</span>(<span class="params">l1, l2</span>):</span></span><br><span class="line"> <span class="keyword">if</span> l1 <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">not</span> l2</span><br><span class="line"> <span class="keyword">if</span> l2 <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">not</span> l1</span><br><span class="line"> <span class="keyword">return</span> l1.val == l2.val <span class="keyword">and</span> symmetric(l1.left, l2.right) <span class="keyword">and</span> symmetric(l1.right, l2.left)</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="层次遍历"><a href="#层次遍历" class="headerlink" title="层次遍历"></a>层次遍历</h2><p>给定一个二叉树，返回其按层次遍历的节点值。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">add</span>(<span class="params">node, level, res</span>):</span></span><br><span class="line"> <span class="keyword">if</span> node <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(res) < level:</span><br><span class="line"> res.append([])</span><br><span class="line"> res[level - <span class="number">1</span>].append(node.val)</span><br><span class="line"> add(node.left, level + <span class="number">1</span>, res)</span><br><span class="line"> add(node.right, level + <span class="number">1</span>, res)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">level_order</span>(<span class="params">root</span>):</span></span><br><span class="line"> res = []</span><br><span class="line"> add(root, <span class="number">1</span>, res)</span><br><span class="line"> <span class="keyword">return</span> res</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="最大深度"><a href="#最大深度" class="headerlink" title="最大深度"></a>最大深度</h2><p>给定一个二叉树，找出其最大深度。</p>
<p>二叉树的深度为根节点到最远叶子节点的最长路径上的节点数。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">max_depth</span>(<span class="params">root</span>):</span></span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span> + <span class="built_in">max</span>(<span class="built_in">map</span>(max_depth, (root.left, root.right))) <span class="keyword">if</span> root <span class="keyword">else</span> <span class="number">0</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="最小深度"><a href="#最小深度" class="headerlink" title="最小深度"></a>最小深度</h2><p>给定一个二叉树，找出其最小深度。</p>
<p>最小深度是从根节点到最近叶子节点的最短路径上的节点数量。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">min_depth</span>(<span class="params">root</span>):</span></span><br><span class="line"> <span class="keyword">if</span> root <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span></span><br><span class="line"> <span class="keyword">if</span> root.left <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span> + min_depth(root.right)</span><br><span class="line"> <span class="keyword">if</span> root.right <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span> + min_depth(root.left)</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span> + <span class="built_in">min</span>(<span class="built_in">map</span>(min_depth, (root.left, root.right)))</span><br><span class="line"></span><br><span class="line"><span class="comment"># 更简洁的写法</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">min_depth</span>(<span class="params">root</span>):</span></span><br><span class="line"> <span class="keyword">if</span> root <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span></span><br><span class="line"> depth_under_root = <span class="built_in">map</span>(min_depth, (root.left, root.right))</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span> + (<span class="built_in">min</span>(depth_under_root) <span class="keyword">or</span> <span class="built_in">max</span>(depth_under_root))</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="将有序数组转化为二叉树"><a href="#将有序数组转化为二叉树" class="headerlink" title="将有序数组转化为二叉树"></a>将有序数组转化为二叉树</h2><p>将一个按照升序排列的有序数组，转换为一棵高度平衡二叉搜索树。</p>
<p>高度平衡二叉树是指一个二叉树每个节点的左右两个子树的高度差的绝对值不超过1。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">sorted_array_to_balanced_tree</span>(<span class="params">nums</span>):</span></span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> nums:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">None</span></span><br><span class="line"> mid = <span class="built_in">len</span>(nums) // <span class="number">2</span></span><br><span class="line"> root = TreeNode(nums[mid])</span><br><span class="line"> root.left = sorted_array_to_balanced_tree(nums[:mid])</span><br><span class="line"> root.right = sorted_array_to_balanced_tree(nums[mid + <span class="number">1</span>:])</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="平衡二叉树"><a href="#平衡二叉树" class="headerlink" title="平衡二叉树"></a>平衡二叉树</h2><p>一个二叉树每个节点的左右两个子树的高度差的绝对值不超过1。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">hight</span>(<span class="params">node</span>):</span></span><br><span class="line"> <span class="keyword">if</span> node <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0</span></span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span> + <span class="built_in">max</span>(<span class="built_in">map</span>(hight, (node.left, node.right)))</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">is_balanced</span>(<span class="params">root</span>):</span></span><br><span class="line"> <span class="keyword">if</span> root <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">True</span></span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">abs</span>(hight(root.left) - hight(root.right)) <= <span class="number">1</span> <span class="keyword">and</span> is_balanced(root.left) <span class="keyword">and</span> is_balanced(root.right)</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="路径总和"><a href="#路径总和" class="headerlink" title="路径总和"></a>路径总和</h2><p>给定一个二叉树和一个目标和，判断该树中是否存在根节点到叶子节点的路径，这条路径上所有节点值相加等于目标和。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">has_path_sum</span>(<span class="params">root, sums</span>):</span></span><br><span class="line"> <span class="keyword">if</span> root <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">False</span></span><br><span class="line"> <span class="keyword">if</span> root.left <span class="keyword">or</span> root.right:</span><br><span class="line"> <span class="keyword">return</span> has_path_sum(root.left, sums - root.val) <span class="keyword">or</span> has_path_sum(root.right, sums - root.val)</span><br><span class="line"> <span class="keyword">return</span> sums == root.val</span><br><span class="line"></span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
</tags>
</entry>
<entry>
<title>理解 Python 的元类</title>
<url>/2021/07/09/understand-python-metaclass/</url>
<content><![CDATA[<p>元类是一个深奥的OOP概念，几乎隐藏在所有Python代码之后。无论您是否知道，都在使用它们。在大多数情况下，您无需意识到这一点。大多数Python程序员很少（即使有的话）也不必考虑元类。</p>
<a id="more"></a>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line">print_line = <span class="keyword">lambda</span> x: print(<span class="string">'*'</span> * x)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># python中类的概念借鉴于Smalltalk，在大多数语言中类就是一段用于生成一个对象的代码。</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ObjectCreator</span>(<span class="params"><span class="built_in">object</span></span>):</span></span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">myobj = ObjectCreator()</span><br><span class="line">print(myobj)</span><br><span class="line">print_line(<span class="number">50</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 但是类在python中也是对象，每当你用class关键字申明一个类时，python就会执行它并创建一个对象。比如上面的代码，python在内存中创建了一个名为ObjectCreator的对象。</span></span><br><span class="line"><span class="comment"># 类对象本身也有创建对象（实例）的能力，这就是为什么它叫做类。</span></span><br><span class="line"><span class="comment"># 但是同时，类本身也是个对象，因此你可以把它赋给一个变量，拷贝，添加属性或者作为函数的参数。</span></span><br><span class="line">print(ObjectCreator) <span class="comment"># 你可以打印它，因为它是一个对象</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">echo</span>(<span class="params">obj</span>):</span></span><br><span class="line"> print(obj)</span><br><span class="line">echo(ObjectCreator) <span class="comment"># 你可以把类作为参数传递</span></span><br><span class="line">print(<span class="built_in">hasattr</span>(ObjectCreator, <span class="string">'new_attribute'</span>))</span><br><span class="line">ObjectCreator.new_attribute = <span class="string">'foo'</span> <span class="comment"># 你可以给它添加属性</span></span><br><span class="line">print(<span class="built_in">hasattr</span>(ObjectCreator, <span class="string">'new_attribute'</span>))</span><br><span class="line">print(ObjectCreator.new_attribute)</span><br><span class="line">ObjectCreatorMirror = ObjectCreator <span class="comment"># 你可以把它赋给一个变量</span></span><br><span class="line">print(ObjectCreatorMirror.new_attribute)</span><br><span class="line">print(ObjectCreatorMirror())</span><br><span class="line">print_line(<span class="number">50</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 因为类也是一个对象，你可以像任何其他对象一样动态地创建它。</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">choose_class</span>(<span class="params">name</span>):</span></span><br><span class="line"> <span class="keyword">if</span> name == <span class="string">'foo'</span>:</span><br><span class="line"> <span class="class"><span class="keyword">class</span> <span class="title">Foo</span>(<span class="params"><span class="built_in">object</span></span>):</span></span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"> <span class="keyword">return</span> Foo <span class="comment"># 注意这里返回类，而不是实例</span></span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> <span class="class"><span class="keyword">class</span> <span class="title">Bar</span>(<span class="params"><span class="built_in">object</span></span>):</span></span><br><span class="line"> <span class="keyword">pass</span></span><br><span class="line"> <span class="keyword">return</span> Bar</span><br><span class="line"></span><br><span class="line">MyClass = choose_class(<span class="string">'foo'</span>)</span><br><span class="line">print(MyClass) <span class="comment"># 函数返回的是一个类，而不是实例</span></span><br><span class="line">print(MyClass()) <span class="comment"># 你可以用这个类创建一个实例对象</span></span><br><span class="line">print_line(<span class="number">50</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 但这还不够动态，因为你还得自己手动编写整个类的代码。既然类是对象，那么肯定有什么东西来生成它，每当你使用class关键字时，python就自动创建这个对象，但是python也给你自己动手的机会，还记得type这个古老的函数吗？它返回这个对象的类型。</span></span><br><span class="line">print(<span class="built_in">type</span>(<span class="number">1</span>))</span><br><span class="line">print(<span class="built_in">type</span>(<span class="string">'1'</span>))</span><br><span class="line">print(<span class="built_in">type</span>(ObjectCreator))</span><br><span class="line">print(<span class="built_in">type</span>(ObjectCreator()))</span><br><span class="line">print_line(<span class="number">50</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># type还有另一种完全不同的能力，它也能够动态地创建类，type接受一个类地描述作为参数然后返回一个类。</span></span><br><span class="line"><span class="comment"># 比如下面这个类可以这样手动创建</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">class MyShinyClass(object):</span></span><br><span class="line"><span class="string"> pass</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">type(name of the class,</span></span><br><span class="line"><span class="string"> tuple of the parent class (for inheritance, can be empty),</span></span><br><span class="line"><span class="string"> dictionary containing attributes names and values)</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line">MyShinyClass = <span class="built_in">type</span>(<span class="string">'MyShinyClass'</span>, (), {}) <span class="comment"># 返回类对象</span></span><br><span class="line">print(MyShinyClass)</span><br><span class="line">print(MyShinyClass())</span><br><span class="line">print_line(<span class="number">50</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 再来个例子</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">class Foo(object):</span></span><br><span class="line"><span class="string"> bar = True</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="comment"># 可以写成</span></span><br><span class="line">Foo = <span class="built_in">type</span>(<span class="string">'Foo'</span>, (), {<span class="string">'bar'</span>: <span class="literal">True</span>})</span><br><span class="line"><span class="comment"># 然后我们像正常的类一样来使用它</span></span><br><span class="line">print(Foo)</span><br><span class="line">print(Foo.bar)</span><br><span class="line">f = Foo()</span><br><span class="line">print(f)</span><br><span class="line">print(f.bar)</span><br><span class="line"><span class="comment"># 我们还可以继承它</span></span><br><span class="line">FooChild = <span class="built_in">type</span>(<span class="string">'FooChild'</span>, (Foo,), {})</span><br><span class="line">print(FooChild)</span><br><span class="line">print(FooChild.bar) <span class="comment"># bar属性从Foo继承</span></span><br><span class="line"><span class="comment"># 在类中添加方法</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">echo_bar</span>(<span class="params">self</span>):</span></span><br><span class="line"> print(self.bar)</span><br><span class="line"></span><br><span class="line">FooChild = <span class="built_in">type</span>(<span class="string">'FooChild'</span>, (Foo,), {<span class="string">'echo_bar'</span>: echo_bar})</span><br><span class="line">print(<span class="built_in">hasattr</span>(Foo, <span class="string">'echo_bar'</span>))</span><br><span class="line">print(<span class="built_in">hasattr</span>(FooChild, <span class="string">'echo_bar'</span>))</span><br><span class="line">my_foo = FooChild()</span><br><span class="line">my_foo.echo_bar()</span><br><span class="line">print_line(<span class="number">50</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 综上，type就是元类，用于创建所有类的元类，是python内建的元类，你也可以创建自己的元类</span></span><br><span class="line"><span class="comment"># 不信你看</span></span><br><span class="line">age = <span class="number">35</span></span><br><span class="line">name = <span class="string">'fuyun'</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">foo</span>():</span> <span class="keyword">pass</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Bar</span>(<span class="params"><span class="built_in">object</span></span>):</span> <span class="keyword">pass</span></span><br><span class="line">print(age.__class__)</span><br><span class="line">print(age.__class__.__class__)</span><br><span class="line">print(name.__class__)</span><br><span class="line">print(name.__class__.__class__)</span><br><span class="line">print(foo.__class__)</span><br><span class="line">print(foo.__class__.__class__)</span><br><span class="line">print(Bar().__class__)</span><br><span class="line">print(Bar().__class__.__class__)</span><br><span class="line">print_line(<span class="number">50</span>)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 当你创建一个类的时候，你可以添加__metaclass__属性，python将会在类定义中寻找__metaclass__，如果找到了就用它来创建类，如果没有就用type来创建类。</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="string">class Foo(Bar):</span></span><br><span class="line"><span class="string"> pass</span></span><br><span class="line"><span class="string">"""</span></span><br><span class="line"><span class="comment"># 上面的代码python将会这样做，首先python会在Foo中找__metaclass__，找到就用它来创建类对象，如果没有就会从父类Bar中找__metaclass__，如果在任何父类中都找不到__metaclass__，就会在模块层次中找__metaclass__，都找不到那就用内置的type来创建类对象。</span></span><br><span class="line"><span class="comment"># 因此我们可以在__metaclass__中写一些用于创建类的代码，什么可以创建类？那就是type，或者任何使用到type或子类化type的东西。</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 元类的主要目的就是在创建类的时候能够自动改变类。</span></span><br><span class="line"><span class="comment"># 举个例子，编写一个元类，让类的属性都改成大写形式。</span></span><br><span class="line"><span class="comment"># 元类会自动将你通常传给'type'的参数作为自己的参数传入</span></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">upper_attr</span>(<span class="params">future_class_name, future_class_parents, future_class_attr</span>):</span></span><br><span class="line"> <span class="string">"""</span></span><br><span class="line"><span class="string"> 返回一个将属性列表变为大写字母的类对象</span></span><br><span class="line"><span class="string"> """</span></span><br><span class="line"></span><br><span class="line"> <span class="comment"># 选取所有不以'__'开头的属性，并把它们变成大写</span></span><br><span class="line"> uppercase_attr = {}</span><br><span class="line"> <span class="keyword">for</span> name, val <span class="keyword">in</span> future_class_attr.items():</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> name.startswith(<span class="string">'__'</span>):</span><br><span class="line"> uppercase_attr[name.upper()] = val</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> uppercase_attr[name] = val</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 用type创建类</span></span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">type</span>(future_class_name, future_class_parents, uppercase_attr)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Foo</span>(<span class="params">metaclass=upper_attr</span>):</span></span><br><span class="line"> <span class="comment"># __metaclass__ = upper_attr 这是python2的写法</span></span><br><span class="line"> bar = <span class="string">'foo'</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">print(<span class="built_in">hasattr</span>(Foo, <span class="string">'bar'</span>))</span><br><span class="line">print(<span class="built_in">hasattr</span>(Foo, <span class="string">'BAR'</span>))</span><br><span class="line">print(Foo().BAR)</span><br><span class="line">print_line(<span class="number">50</span>)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># type实际上是一个类，就像str和int一样，所以，你可以从type继承</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">UpperAttrMetaclass</span>(<span class="params"><span class="built_in">type</span></span>):</span></span><br><span class="line"> <span class="comment"># __new__ 是在__init__之前被调用的特殊方法</span></span><br><span class="line"> <span class="comment"># __new__是用来创建对象并返回它的方法</span></span><br><span class="line"> <span class="comment"># 而__init__只是用来将传入的参数初始化给对象</span></span><br><span class="line"> <span class="comment"># 你很少用到__new__，除非你希望能够控制对象的创建</span></span><br><span class="line"> <span class="comment"># 这里，创建的对象是类，我们希望能够自定义它，所以我们这里改写__new__</span></span><br><span class="line"> <span class="comment"># 如果你希望的话，你也可以在__init__中做些事情</span></span><br><span class="line"> <span class="comment"># 还有一些高级的用法会涉及到改写__call__特殊方法，但是我们这里不用</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__new__</span>(<span class="params">upperattr_metaclass, future_class_name,</span></span></span><br><span class="line"><span class="function"><span class="params"> future_class_parents, future_class_attr</span>):</span></span><br><span class="line"></span><br><span class="line"> uppercase_attr = {}</span><br><span class="line"> <span class="keyword">for</span> name, val <span class="keyword">in</span> future_class_attr.items():</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> name.startswith(<span class="string">'__'</span>):</span><br><span class="line"> uppercase_attr[name.upper()] = val</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> uppercase_attr[name] = val</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">type</span>(future_class_name, future_class_parents, uppercase_attr)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 但是这不是真正的面向对象(OOP)。我们直接调用了type，而且我们没有改写父类的new方法。</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">UpperAttrMetaclass</span>(<span class="params"><span class="built_in">type</span></span>):</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__new__</span>(<span class="params">upperattr_metaclass, future_class_name,</span></span></span><br><span class="line"><span class="function"><span class="params"> future_class_parents, future_class_attr</span>):</span></span><br><span class="line"></span><br><span class="line"> uppercase_attr = {}</span><br><span class="line"> <span class="keyword">for</span> name, val <span class="keyword">in</span> future_class_attr.items():</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> name.startswith(<span class="string">'__'</span>):</span><br><span class="line"> uppercase_attr[name.upper()] = val</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> uppercase_attr[name] = val</span><br><span class="line"></span><br><span class="line"> <span class="comment"># 重用 type.__new__ 方法</span></span><br><span class="line"> <span class="comment"># 这就是基本的OOP编程，没什么魔法</span></span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">type</span>.__new__(upperattr_metaclass, future_class_name,</span><br><span class="line"> future_class_parents, uppercase_attr)</span><br><span class="line"> <span class="comment"># 这里有个额外的参数upperattr_metaclass，类似于self，类方法的第一个参数总是代表当前实例。</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 为了便于理解上面的代码名字编的太长，实际产品中的代码应该是这样的</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">UpperAttrMetaclass</span>(<span class="params"><span class="built_in">type</span></span>):</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__new__</span>(<span class="params">cls, clsname, bases, dct</span>):</span></span><br><span class="line"></span><br><span class="line"> uppercase_attr = {}</span><br><span class="line"> <span class="keyword">for</span> name, val <span class="keyword">in</span> dct.items():</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> name.startswith(<span class="string">'__'</span>):</span><br><span class="line"> uppercase_attr[name.upper()] = val</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> uppercase_attr[name] = val</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">type</span>.__new__(cls, clsname, bases, uppercase_attr)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 我们还可以使用super方法使代码变得更清晰一点</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">UpperAttrMetaclass</span>(<span class="params"><span class="built_in">type</span></span>):</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__new__</span>(<span class="params">cls, clsname, bases, dct</span>):</span></span><br><span class="line"></span><br><span class="line"> uppercase_attr = {}</span><br><span class="line"> <span class="keyword">for</span> name, val <span class="keyword">in</span> dct.items():</span><br><span class="line"> <span class="keyword">if</span> <span class="keyword">not</span> name.startswith(<span class="string">'__'</span>):</span><br><span class="line"> uppercase_attr[name.upper()] = val</span><br><span class="line"> <span class="keyword">else</span>:</span><br><span class="line"> uppercase_attr[name] = val</span><br><span class="line"></span><br><span class="line"> <span class="comment"># return super(UpperAttrMetaclass, cls).__new__(cls, clsname, bases, uppercase_attr) 这是python2的写法</span></span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">super</span>().__new__(cls, clsname, bases, uppercase_attr)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># 元类要做的事就是拦截类的创建，修改一个类，返回修改之后的类。</span></span><br><span class="line"><span class="comment"># 当你需要动态修改类的时候，最好使用“monkey patching”或“装饰器”。</span></span><br><span class="line"></span><br></pre></td></tr></table></figure>
]]></content>
<categories>
<category>笔记</category>
</categories>
<tags>
<tag>Python3</tag>
</tags>
</entry>
<entry>
<title>理解 Python 的关键字 Yield</title>
<url>/2018/03/14/understand-yield/</url>
<content><![CDATA[<p>为了理解什么是yield,你必须理解什么是生成器。</p>
<p>在理解生成器之前，让我们先走近迭代。</p>
<p>当你建立了一个列表，你可以逐项地读取这个列表，这叫做一个可迭代对象。</p>
<a id="more"></a>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line">mylist = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>]</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> mylist:</span><br><span class="line"> print(i)</span><br></pre></td></tr></table></figure>
<p>所有你可以使用for…in…语法的叫做一个迭代器，列表，字符串，文件等等，你经常使用它们是因为你可以如你所愿的读取其中的元素，但是你把所有的值都存储到了内存中，如果你有大量数据的话这个方式并不是你想要的。</p>
<p>生成器是可以迭代的，但是你只可以读取它一次，因为它并不把所有的值放在内存中，它是实时地生成数据。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line">mygenerator = (x * x <span class="keyword">for</span> x <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">5</span>))</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> mygenerator:</span><br><span class="line"> print(i)</span><br></pre></td></tr></table></figure>
<p>你不可以再次迭代生成器</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> <span class="built_in">next</span>(mygenerator)</span><br><span class="line"><span class="keyword">except</span> StopIteration:</span><br><span class="line"> print(<span class="string">"停止迭代"</span>)</span><br></pre></td></tr></table></figure>
<p>yield 是一个类似 return 的关键字，只是这个函数返回的是个生成器。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">create_generator</span>():</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">5</span>):</span><br><span class="line"> <span class="keyword">yield</span> i * i</span><br></pre></td></tr></table></figure>
<p>如果函数内部使用 return，则返回 0。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line">mygenerator = create_generator()</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> mygenerator:</span><br><span class="line"> print(i)</span><br></pre></td></tr></table></figure>
<p>斐波拉契数列</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">fib</span>():</span></span><br><span class="line"> x, y = <span class="number">0</span>, <span class="number">1</span></span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> x, y = y, x + y</span><br><span class="line"> <span class="keyword">yield</span> x</span><br></pre></td></tr></table></figure>
<p>获取斐波拉契数列前10个</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> itertools</span><br><span class="line"><span class="built_in">list</span>(itertools.islice(fib(), <span class="number">10</span>))</span><br></pre></td></tr></table></figure>
<p>杨辉三角</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">triangle</span>():</span></span><br><span class="line"> a = [<span class="number">1</span>]</span><br><span class="line"> <span class="keyword">while</span> <span class="literal">True</span>:</span><br><span class="line"> <span class="keyword">yield</span> a</span><br><span class="line"> a = [<span class="built_in">sum</span>(i) <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">zip</span>([<span class="number">0</span>] + a, a + [<span class="number">0</span>])]</span><br></pre></td></tr></table></figure>
<p>输出前10行杨辉三角</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> pprint</span><br><span class="line"><span class="built_in">list</span>(itertools.islice(triangle(), <span class="number">10</span>))</span><br></pre></td></tr></table></figure>
<p>控制迭代器的穷尽</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Bank</span>():</span></span><br><span class="line"> crisis = <span class="literal">False</span> <span class="comment"># crisis是危机的意思</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">create_atm</span>(<span class="params">self</span>):</span></span><br><span class="line"> <span class="keyword">while</span> <span class="keyword">not</span> self.crisis:</span><br><span class="line"> <span class="keyword">yield</span> <span class="string">"$100"</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">bank = Bank() <span class="comment"># 创建一个银行</span></span><br><span class="line">corner_street_atm = bank.create_atm() <span class="comment"># 创建一个ATM机</span></span><br><span class="line">print([<span class="built_in">next</span>(corner_street_atm) <span class="keyword">for</span> _ <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">5</span>)])</span><br><span class="line">bank.crisis = <span class="literal">True</span> <span class="comment"># 危机来了</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> print(<span class="built_in">next</span>(corner_street_atm))</span><br><span class="line"><span class="keyword">except</span> StopIteration:</span><br><span class="line"> print(<span class="string">"corner_street_atm: no more money!"</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> wall_street_atm = bank.create_atm()</span><br><span class="line"> print(<span class="built_in">next</span>(wall_street_atm))</span><br><span class="line"><span class="keyword">except</span> StopIteration:</span><br><span class="line"> print(<span class="string">"wall_street_atm: no more money!"</span>)</span><br><span class="line"></span><br><span class="line">bank.crisis = <span class="literal">False</span> <span class="comment"># 问题是，即使改变crisis的值，ATM依然是空的</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line"> print(<span class="built_in">next</span>(corner_street_atm))</span><br><span class="line"><span class="keyword">except</span> StopIteration:</span><br><span class="line"> print(<span class="string">"crisis is %s, and still no more money!"</span> % bank.crisis)</span><br><span class="line"></span><br><span class="line"><span class="comment"># 重新创建一个ATM机，现在有钱了</span></span><br><span class="line">brand_new_atm = bank.create_atm()</span><br><span class="line">print([<span class="built_in">next</span>(brand_new_atm) <span class="keyword">for</span> _ <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">5</span>)])</span><br></pre></td></tr></table></figure>
<p><code>itertools</code> 模块包含了许多特殊的迭代方法</p>
<p>比赛中4匹马可能到达终点的先后顺序的可能情况</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="keyword">import</span> pprint</span><br><span class="line">horses = [<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>]</span><br><span class="line">races = itertools.permutations(horses)</span><br><span class="line">print(races)</span><br><span class="line">pprint.pprint(<span class="built_in">list</span>(races))</span><br></pre></td></tr></table></figure>
<p>一个实现了 <code>__iter__</code> 方法的对象是可迭代的，一个实现了 <code>__next__</code> 方法的对象是迭代器。</p>
<figure class="highlight python"><table><tr><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Fibs</span>():</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span>(<span class="params">self</span>):</span></span><br><span class="line"> self.a = <span class="number">0</span></span><br><span class="line"> self.b = <span class="number">1</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__next__</span>(<span class="params">self</span>):</span></span><br><span class="line"> self.a, self.b = self.b, self.a + self.b</span><br><span class="line"> <span class="keyword">return</span> self.a</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__iter__</span>(<span class="params">self</span>):</span></span><br><span class="line"> <span class="keyword">return</span> self</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">fibs = Fibs()</span><br><span class="line">print([<span class="built_in">next</span>(fibs) <span class="keyword">for</span> _ <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">10</span>)])</span><br></pre></td></tr></table></figure>
]]></content>