-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathatom.xml
973 lines (805 loc) · 74.3 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title><![CDATA[季米风]]></title>
<link href="http://www.fengjimin.com/atom.xml" rel="self"/>
<link href="http://www.fengjimin.com/"/>
<updated>2019-01-22T20:16:46+08:00</updated>
<id>http://www.fengjimin.com/</id>
<author>
<name><![CDATA[冯继敏]]></name>
<email><![CDATA[[email protected]]]></email>
</author>
<generator uri="http://octopress.org/">Octopress</generator>
<entry>
<title type="html"><![CDATA[python machine learning and text processing (2)]]></title>
<link href="http://www.fengjimin.com/blog/2019/01/22/python-machine-learning-and-text-processing-2/"/>
<updated>2019-01-22T16:44:00+08:00</updated>
<id>http://www.fengjimin.com/blog/2019/01/22/python-machine-learning-and-text-processing-2</id>
<content type="html"><![CDATA[<p>机器学习是个大课题，也是一种解决现实问题的思维模式。虽然单纯学习其算法不算太难，然而要有对解决实际问题的纯熟运用，却需要在理论与实践相结合方面下足功夫。python在这方面提供了足够便利的工具。</p>
<!--more-->
<h3>简介</h3>
<ol>
<li>监督学习
<ol>
<li>分类问题</li>
<li>回归问题</li>
</ol>
</li>
<li>非监督学习
<ol>
<li>聚类</li>
<li>降维</li>
</ol>
</li>
</ol>
<h3>步骤</h3>
<ol>
<li>收集数据
<ol>
<li><a href="https://data.world/crowdflower/disasters-on-social-media">Disasters on Social Media</a> 数据集</li>
</ol>
</li>
<li>数据清洗
<ol>
<li>去除不相关字符</li>
<li>拆分独立单词</li>
<li>去除不相关词语</li>
<li>转换成小写</li>
<li>多种拼法御特定表达绑定</li>
<li>词型还原</li>
</ol>
</li>
<li>特征提取
<ol>
<li>bag of words 词袋法
<ol>
<li><code>John is happy he is not hungary for apples. -> [0,2,1,1,1,1,1,1,1]</code></li>
<li>完全忽略单词顺序</li>
</ol>
</li>
<li>one-hot
<ol>
<li><code>John is happy he is not hungary for apples. -> [0,1,1,1,1,1,1,1,1]</code></li>
</ol>
</li>
<li>TF-IDF
<ol>
<li><img src="http://www.ruanyifeng.com/blogimg/asset/201303/bg2013031504.png" alt="TF" /></li>
<li><img src="http://www.ruanyifeng.com/blogimg/asset/201303/bg2013031506.png" alt="IDF" /></li>
</ol>
</li>
<li>word2vec
<ol>
<li>使用浅层双层神经网络，映射每一个词到一个向量，使之能表达词语之间的相互关系，以及上下文的关系。如：<code>V(queen) - V(woman) + V(man) = V(king)</code></li>
<li>训练算法
<ol>
<li>skip-grams: 由当前单词预测上下文</li>
<li>CBOW: 由上下文预测当前单词</li>
<li><img src="https://static.leiphone.com/uploads/new/article/740_740/201706/594b31d0920ef.png?imageMogr2/format/jpg/quality/90" alt="nn" /></li>
<li><img src="https://static.leiphone.com/uploads/new/article/740_740/201706/594b3267c64f4.png?imageMogr2/format/jpg/quality/90" alt="weights" /> 中间300weights就是car单词的词向量</li>
<li>句子嵌入的向量表示，可以采用将句子中词汇的word2vec平均化</li>
</ol>
</li>
</ol>
</li>
</ol>
</li>
<li>分类
<ol>
<li>数据集划分为训练集和测试集</li>
<li>分类算法：逻辑回归，SVM，贝叶斯公式，等等</li>
</ol>
</li>
<li>检验效果
<ol>
<li>混淆矩阵</li>
<li>precision, recall, f1_score..</li>
</ol>
</li>
</ol>
<h3>Scikit-learn库</h3>
<ol>
<li><p>算法包</p>
<table>
<thead>
<tr>
<th></th>
<th> 导入算法 </th>
<th> 代码 </th>
</tr>
</thead>
<tbody>
<tr>
<td></td>
<td> 广义线性模型 </td>
<td> from sklearn.linear_model import LinearRegression |</td>
</tr>
<tr>
<td></td>
<td> 逻辑回归 </td>
<td> rom sklearn.linear_model import LogisticRegression |</td>
</tr>
<tr>
<td></td>
<td> 支持向量机 </td>
<td> from sklearn.svm import SVC |</td>
</tr>
<tr>
<td></td>
<td> k近邻 </td>
<td> from sklearn.neighbors import KNeighborsClassifier |</td>
</tr>
<tr>
<td></td>
<td> 决策树 </td>
<td> from sklearn.tree import DecisionTreeClassifier |</td>
</tr>
<tr>
<td></td>
<td> 朴素贝叶斯 </td>
<td> from sklearn.naive_bayes import MultinomialNB |</td>
</tr>
</tbody>
</table>
</li>
<li><p>转换器（transformer）：用于数据预处理和数据转换</p>
<ol>
<li>transformer.fit</li>
<li>transformer.transform</li>
<li>transformer.fit_transform</li>
</ol>
</li>
<li><p>估计器（estimator）：用于分类和聚类</p>
<ol>
<li>estimator.fit</li>
<li>estimator.predict</li>
</ol>
</li>
<li><p>特征提取（sklearn.feature_extraction）</p>
<ol>
<li>sklearn.feature_extraction.text.CountVectorizer：将文本转换为每个词出现的个数的向量（词典法）</li>
<li>sklearn.feature_extraction.text.TfidfVectorizer：将文本转换为tfidf值的向量</li>
</ol>
</li>
<li><p>数据集分割（sklearn.cross_validation）</p>
<ol>
<li>train_test_split(X, y, test_size, random_state)</li>
</ol>
</li>
<li><p>效果评估（sklearn.metrics）</p>
<ol>
<li>confusion_matrix</li>
<li>accuracy_score</li>
<li>classification_report</li>
<li>precision_recall_fscore_support</li>
</ol>
</li>
</ol>
<h3>sklearn文本特征提取</h3>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">sklearn.feature_extraction.text</span> <span class="kn">import</span> <span class="n">CountVectorizer</span>
</span><span class='line'>
</span><span class='line'><span class="n">corpus</span> <span class="o">=</span> <span class="p">[</span><span class="s">"Hey lets get lunch :)"</span><span class="p">,</span>
</span><span class='line'> <span class="s">"Hey!!! I need a favor"</span><span class="p">]</span>
</span><span class='line'><span class="n">vectorize</span> <span class="o">=</span> <span class="n">CountVectorizer</span><span class="p">()</span>
</span><span class='line'><span class="n">vectorize</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
</span><span class='line'><span class="c">#构建 文档词频矩阵</span>
</span><span class='line'><span class="n">dtm</span> <span class="o">=</span> <span class="n">vectorize</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">sklearn.feature_extraction.text</span> <span class="kn">import</span> <span class="n">TfidfVectorizer</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">createDTM</span><span class="p">(</span><span class="n">corpus</span><span class="p">):</span>
</span><span class='line'> <span class="sd">"""构建文档词语矩阵"""</span>
</span><span class='line'> <span class="n">vectorize</span> <span class="o">=</span> <span class="n">TfidfVectorizer</span><span class="p">()</span>
</span><span class='line'> <span class="c">#注意fit_transform相当于fit之后又transform。</span>
</span><span class='line'> <span class="n">dtm</span> <span class="o">=</span> <span class="n">vectorize</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
</span><span class='line'> <span class="k">return</span> <span class="n">dtm</span>
</span><span class='line'>
</span><span class='line'><span class="n">corpus</span> <span class="o">=</span> <span class="p">[</span><span class="s">"Hey lets get lunch :)"</span><span class="p">,</span>
</span><span class='line'> <span class="s">"Hey!!! I need a favor"</span><span class="p">]</span>
</span><span class='line'><span class="n">createDTM</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
<h3>sklearn文本分类</h3>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">import</span> <span class="nn">re</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">pandas</span> <span class="kn">as</span> <span class="nn">pd</span>
</span><span class='line'><span class="kn">from</span> <span class="nn">sklearn.naive_bayes</span> <span class="kn">import</span> <span class="n">MultinomialNB</span>
</span><span class='line'><span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LogisticRegression</span>
</span><span class='line'><span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">normalize_numbers</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
</span><span class='line'> <span class="sd">"""</span>
</span><span class='line'><span class="sd"> 将数字特征统一替换为NUM</span>
</span><span class='line'><span class="sd"> """</span>
</span><span class='line'> <span class="k">return</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="s">r'\b\d+\b'</span><span class="p">,</span> <span class="s">'NUM'</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="n">dataset</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s">'data/20news-18828.csv'</span><span class="p">,</span> <span class="n">header</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s">','</span><span class="p">,</span> <span class="n">names</span><span class="o">=</span><span class="p">[</span><span class="s">'label'</span><span class="p">,</span> <span class="s">'text'</span><span class="p">])</span>
</span><span class='line'>
</span><span class='line'><span class="c">#依次导入X， y， test_size</span>
</span><span class='line'><span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">dataset</span><span class="o">.</span><span class="n">text</span><span class="p">,</span> <span class="n">dataset</span><span class="o">.</span><span class="n">label</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.3</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c">#CountVectorizer和TfidfVectorizer都有preprocessor参数，该参数传入预处理函数。</span>
</span><span class='line'><span class="c">#也含有停用词处理参数stop_words</span>
</span><span class='line'><span class="n">tv</span> <span class="o">=</span> <span class="n">TfidfVectorizer</span><span class="p">(</span><span class="n">preprocessor</span><span class="o">=</span><span class="n">normalize_numbers</span><span class="p">,</span> <span class="n">stop_words</span><span class="o">=</span><span class="s">'english'</span><span class="p">)</span>
</span><span class='line'><span class="n">X_train_tf</span> <span class="o">=</span> <span class="n">tv</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
</span><span class='line'><span class="n">X_test_tf</span> <span class="o">=</span> <span class="n">tv</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c"># 贝叶斯分类器</span>
</span><span class='line'><span class="n">estimator2</span> <span class="o">=</span> <span class="n">MultinomialNB</span><span class="p">()</span>
</span><span class='line'><span class="n">estimator2</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train_tf</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
</span><span class='line'><span class="n">predicted</span> <span class="o">=</span> <span class="n">estimator2</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test_tf</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="k">print</span><span class="p">(</span><span class="n">classification_report</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">predicted</span><span class="p">,</span> <span class="n">labels</span><span class="o">=</span><span class="n">dataset</span><span class="o">.</span><span class="n">label</span><span class="o">.</span><span class="n">unique</span><span class="p">()))</span>
</span><span class='line'>
</span><span class='line'><span class="c"># 逻辑回归分类器</span>
</span><span class='line'><span class="n">LR</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">()</span>
</span><span class='line'><span class="n">LR</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train_tf</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
</span><span class='line'><span class="n">predicted</span> <span class="o">=</span> <span class="n">LR</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test_tf</span><span class="p">)</span>
</span><span class='line'><span class="k">print</span><span class="p">(</span><span class="n">classification_report</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">predicted</span><span class="p">,</span> <span class="n">labels</span><span class="o">=</span><span class="n">dataset</span><span class="o">.</span><span class="n">label</span><span class="o">.</span><span class="n">unique</span><span class="p">()))</span>
</span></code></pre></td></tr></table></div></figure>
<h3>sklearn话题分析（聚类）</h3>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">import</span> <span class="nn">csv</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">jieba</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">re</span>
</span><span class='line'><span class="kn">from</span> <span class="nn">sklearn.cluster</span> <span class="kn">import</span> <span class="n">KMeans</span>
</span><span class='line'><span class="kn">from</span> <span class="nn">sklearn.feature_extraction.text</span> <span class="kn">import</span> <span class="n">TfidfVectorizer</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">pandas</span> <span class="kn">as</span> <span class="nn">pd</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">read_data</span><span class="p">(</span><span class="nb">file</span><span class="p">):</span>
</span><span class='line'> <span class="sd">"""</span>
</span><span class='line'><span class="sd"> 读取csv数据，返回含有字符串的列表数据。</span>
</span><span class='line'><span class="sd"> :param file: csv文件</span>
</span><span class='line'><span class="sd"> :return: 列表数据（列表中每个组成部分是字符串文本）</span>
</span><span class='line'><span class="sd"> """</span>
</span><span class='line'>
</span><span class='line'> <span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_excel</span><span class="p">(</span><span class="nb">file</span><span class="p">)</span>
</span><span class='line'> <span class="c">#从dataframe中读取 news_content 列的所有新闻信息</span>
</span><span class='line'> <span class="n">documents</span> <span class="o">=</span> <span class="n">df</span><span class="p">[</span><span class="s">'news_content'</span><span class="p">]</span>
</span><span class='line'> <span class="k">return</span> <span class="n">documents</span>
</span><span class='line'>
</span><span class='line'><span class="k">def</span> <span class="nf">cluster_analysis</span><span class="p">(</span><span class="n">best_k</span><span class="p">,</span> <span class="n">texts</span><span class="p">):</span>
</span><span class='line'> <span class="sd">"""</span>
</span><span class='line'><span class="sd"> 按照最佳聚类数k，将texts聚成k类。</span>
</span><span class='line'><span class="sd"> :param best_k: 聚类准确率最好的 k值</span>
</span><span class='line'><span class="sd"> :param texts: 列表数据（且列表中每个组成部分是字符串文本）</span>
</span><span class='line'><span class="sd"> :return:None</span>
</span><span class='line'>
</span><span class='line'><span class="sd"> """</span>
</span><span class='line'>
</span><span class='line'> <span class="n">documents</span> <span class="o">=</span> <span class="p">[</span><span class="s">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">jieba</span><span class="o">.</span><span class="n">lcut</span><span class="p">(</span><span class="n">text</span><span class="p">))</span> <span class="k">for</span> <span class="n">text</span> <span class="ow">in</span> <span class="n">texts</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'> <span class="n">tfidf</span> <span class="o">=</span> <span class="n">TfidfVectorizer</span><span class="p">(</span><span class="n">max_df</span><span class="o">=</span><span class="mf">0.5</span><span class="p">)</span>
</span><span class='line'> <span class="c"># 学习tfidf矩阵数据中的分布规律</span>
</span><span class='line'> <span class="n">tfidf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">documents</span><span class="p">)</span>
</span><span class='line'> <span class="n">matrix</span> <span class="o">=</span> <span class="n">tfidf</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">documents</span><span class="p">)</span>
</span><span class='line'> <span class="k">print</span><span class="p">(</span><span class="s">'文档向量化矩阵：'</span><span class="p">)</span>
</span><span class='line'> <span class="k">print</span><span class="p">(</span><span class="n">matrix</span><span class="o">.</span><span class="n">toarray</span><span class="p">())</span>
</span><span class='line'> <span class="c"># c初始化Kmeans估计器</span>
</span><span class='line'> <span class="n">estimator</span> <span class="o">=</span> <span class="n">KMeans</span><span class="p">(</span><span class="n">n_clusters</span><span class="o">=</span><span class="n">best_k</span><span class="p">)</span>
</span><span class='line'> <span class="n">estimator</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">matrix</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'> <span class="c">#机器给所有的文档打上的标签</span>
</span><span class='line'> <span class="c">#此处self.label_pred 为一维列表，列表长度等于 文档的数目</span>
</span><span class='line'> <span class="n">label_pred</span> <span class="o">=</span> <span class="n">estimator</span><span class="o">.</span><span class="n">labels_</span>
</span><span class='line'>
</span><span class='line'> <span class="c">#保存聚类算法的分类结果 到output文件夹中。</span>
</span><span class='line'> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'data/cluster_output.csv'</span><span class="p">,</span> <span class="s">'a+'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s">'gbk'</span><span class="p">,</span> <span class="n">newline</span><span class="o">=</span><span class="s">''</span><span class="p">)</span> <span class="k">as</span> <span class="n">csvf</span><span class="p">:</span>
</span><span class='line'> <span class="n">writer</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">writer</span><span class="p">(</span><span class="n">csvf</span><span class="p">)</span>
</span><span class='line'> <span class="n">writer</span><span class="o">.</span><span class="n">writerow</span><span class="p">((</span><span class="s">'news_content'</span><span class="p">,</span> <span class="s">'cluster'</span><span class="p">))</span>
</span><span class='line'> <span class="c">#将新闻与 无kmeans聚类结果（cluster）写入csv文件</span>
</span><span class='line'> <span class="k">for</span> <span class="n">idx</span><span class="p">,</span> <span class="n">text</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">texts</span><span class="p">):</span>
</span><span class='line'> <span class="k">try</span><span class="p">:</span>
</span><span class='line'> <span class="n">writer</span><span class="o">.</span><span class="n">writerow</span><span class="p">((</span><span class="n">text</span><span class="p">,</span> <span class="n">label_pred</span><span class="p">[</span><span class="n">idx</span><span class="p">]))</span>
</span><span class='line'> <span class="k">except</span><span class="p">:</span>
</span><span class='line'> <span class="k">continue</span>
</span><span class='line'>
</span><span class='line'><span class="n">texts</span> <span class="o">=</span> <span class="n">read_data</span><span class="p">(</span><span class="nb">file</span><span class="o">=</span><span class="s">'data/百度新闻.xlsx'</span><span class="p">)</span>
</span><span class='line'><span class="n">cluster_analysis</span><span class="p">(</span><span class="n">best_k</span><span class="o">=</span><span class="mi">19</span><span class="p">,</span> <span class="n">texts</span><span class="o">=</span><span class="n">texts</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
<h3>SVD计算消费者偏好</h3>
<ol>
<li>推荐算法比较常见的实现方式是协同过滤（Collaberative Filtering）。数据涉及到用户，产品，和产品评价三种信息。</li>
<li>有了用户评价矩阵，可以采用奇异值分解（SVD），获得降维后的用户向量，从而更准确的计算得到推荐信息</li>
<li>pip install scikit-surprise</li>
</ol>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">from</span> <span class="nn">surprise</span> <span class="kn">import</span> <span class="n">Reader</span><span class="p">,</span> <span class="n">Dataset</span>
</span><span class='line'><span class="kn">from</span> <span class="nn">surprise</span> <span class="kn">import</span> <span class="n">SVD</span><span class="p">,</span> <span class="n">evaluate</span>
</span><span class='line'>
</span><span class='line'><span class="c">#定义数据格式</span>
</span><span class='line'><span class="n">reader</span> <span class="o">=</span> <span class="n">Reader</span><span class="p">(</span><span class="n">line_format</span><span class="o">=</span><span class="s">'user item rating timestamp'</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s">'</span><span class="se">\t</span><span class="s">'</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c">#使用reader格式从u.data文件中读取数据</span>
</span><span class='line'><span class="n">data</span> <span class="o">=</span> <span class="n">Dataset</span><span class="o">.</span><span class="n">load_from_file</span><span class="p">(</span><span class="s">'data/u.data'</span><span class="p">,</span> <span class="n">reader</span><span class="o">=</span><span class="n">reader</span><span class="p">)</span>
</span><span class='line'><span class="n">data</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">n_folds</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c">#相当于scikit的机器学习算法的初始化</span>
</span><span class='line'><span class="n">svd</span> <span class="o">=</span> <span class="n">SVD</span><span class="p">()</span>
</span><span class='line'><span class="c">#相当于scikit中的score，模型评估</span>
</span><span class='line'><span class="n">evaluate</span><span class="p">(</span><span class="n">svd</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">measures</span><span class="o">=</span><span class="p">[</span><span class="s">'RMSE'</span><span class="p">,</span> <span class="s">'MAE'</span><span class="p">])</span>
</span><span class='line'>
</span><span class='line'><span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">build_full_trainset</span><span class="p">()</span>
</span><span class='line'><span class="n">svd</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</span><span class='line'><span class="n">svd</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="mi">186</span><span class="p">,</span> <span class="mi">302</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[python machine learning and text processing (1)]]></title>
<link href="http://www.fengjimin.com/blog/2019/01/22/python-machine-learning-and-text-processing-1/"/>
<updated>2019-01-22T15:32:00+08:00</updated>
<id>http://www.fengjimin.com/blog/2019/01/22/python-machine-learning-and-text-processing-1</id>
<content type="html"><![CDATA[<h3>前言</h3>
<p>近期在学习python和机器学习。其实本人大学时学过不少机器学习相关专业课，包括神经网络，计算视觉与图像处理，模式识别，自然语言处理（NLP）等。也做过相关研究，读了些paper。可以说有一定基础。只是工作之后没有特意往这个方向发展，不自觉走入了被各种业务需求埋没的境地。如今AI大热，自己也对这块兴趣盎然，因而特别想在理论和实践上弄通。</p>
<!--more-->
<p>若干年以前，因Ruby on Rails快速开发的特性，自己曾把ruby当作银弹，很痴迷于这门语言。也看了很多python与ruby谁更强大这方面的争论。当时觉得ruby的表达能力很强大且优美，近乎魔术，而python，光使用缩进语法这一点就觉得有点怪，因而并没有深入去学。虽然其解决问题有且只有一条最佳思路、还有make things explicit的理念听起来有些道理。然而世事难料，随着Rails渐趋冷静，ruby似乎也渐趋没落。而python，随着人工智能的兴起却在走向潮头。当然，自己现在来学python，已经没有了当初的门户之见，反倒很容易的见识到python简单易用，以及确实强大的方面。在大数据和人工智能的浪潮中，python能站在前沿，是有它的独到之处的。</p>
<p>本blog主要对python文本处理基础，以及机器学习相关的内容进行提纲挈领。其中文本处理包含了分词、自然语言统计、可视化、数据分析等python库。而机器学习，则包含其理论分野、实现步骤、基本机器学习算法和python库等。</p>
<p>以下为文本处理基础库相关内容。</p>
<h3>分词</h3>
<ol>
<li><p>jieba 中文分词库</p>
<ol>
<li>cut, lcut</li>
<li>load_userdict(file)</li>
<li>jieba.posseg.cut</li>
</ol>
</li>
<li><p>正则词频统计</p>
<ol>
<li><code>words = re.findall(r'[\u4e00-\u9fa5]+', test)</code> 匹配中文正则</li>
</ol>
</li>
</ol>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">wordset</span><span class="p">:</span>
</span><span class='line'> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">word</span><span class="p">)</span><span class="o">></span><span class="mi">1</span> <span class="p">:</span>
</span><span class='line'> <span class="n">freq</span> <span class="o">=</span> <span class="n">wordlist</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
</span><span class='line'> <span class="n">result</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">word</span><span class="p">,</span> <span class="n">freq</span><span class="p">))</span>
</span><span class='line'><span class="c">#[(w1,freq1),(w2,freq2)....]</span>
</span><span class='line'><span class="c">#对词频统计结果进行排序</span>
</span><span class='line'><span class="n">new_result</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">k</span><span class="p">:</span><span class="n">k</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">reverse</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
<h3>自然语言统计</h3>
<ol>
<li><p>nltk 自然语言处理库</p>
<ol>
<li><p>nltk.text.Text类</p>
<ol>
<li><code>text = Text(wordlist)</code></li>
<li><code>text.concordance(word='汪淼', width=20, lines=10)</code> 上下文</li>
<li><code>text.similar(word='叶文洁', num=10)</code>上下文最相关词</li>
<li><code>text.count(word='叶文洁')</code> 统计词频</li>
<li><code>text.dispersion_plot(words)</code> 离散图，横坐标表示词语的索引位置</li>
</ol>
</li>
<li><p>Ngrams的实现</p>
<ol>
<li><code>nltk.util.bigrams(seq), trigrams(seq), ngrams(seq, n), skipgrams(seq, n, k)</code></li>
</ol>
</li>
<li><p>nltk.probability</p>
<ol>
<li><p>FreqDist</p>
<ol>
<li><code>fdist = FreqDist(wordlist)</code></li>
<li><code>fdist.items()</code> 词频对</li>
<li><code>fdist.N()</code> 词语总数</li>
<li><code>fdist.max()</code> 词语总数</li>
<li><code>fdist.freq('三体')</code> 词频占比</li>
<li><code>fdist['三体']</code> 词频</li>
<li><code>fdist.plot(20,cumulative=True)</code> 前20个词画图</li>
</ol>
</li>
<li><p>ConditionFreqDist 带条件的频率分布类</p></li>
</ol>
</li>
</ol>
</li>
</ol>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">cfd</span> <span class="o">=</span> <span class="n">ConditionalFreqDist</span><span class="p">(</span>
</span><span class='line'> <span class="p">[(</span><span class="n">conditon1</span><span class="p">,</span> <span class="n">event1</span><span class="p">),(</span><span class="n">conditon2</span><span class="p">,</span> <span class="n">event1</span><span class="p">),(</span><span class="n">conditon1</span><span class="p">,</span> <span class="n">event2</span><span class="p">)</span><span class="o">...</span><span class="p">(</span><span class="n">conditon_n</span><span class="p">,</span> <span class="n">event_n</span><span class="p">)]</span>
</span><span class='line'> <span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">cfd</span> <span class="o">=</span> <span class="n">ConditionalFreqDist</span><span class="p">(</span>
</span><span class='line'> <span class="p">(</span><span class="nb">file</span><span class="p">,</span> <span class="n">word</span><span class="p">)</span>
</span><span class='line'> <span class="k">for</span> <span class="nb">file</span> <span class="ow">in</span> <span class="n">files</span>
</span><span class='line'> <span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">text_split</span><span class="p">(</span><span class="nb">file</span><span class="p">))</span>
</span><span class='line'>
</span><span class='line'> <span class="c">#计算前10大词在各类文本中的频数</span>
</span><span class='line'> <span class="n">cfd</span><span class="o">.</span><span class="n">tabulate</span><span class="p">(</span><span class="n">conditions</span><span class="o">=</span><span class="n">files</span><span class="p">,</span> <span class="n">samples</span><span class="o">=</span><span class="n">words</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
<h3>可视化</h3>
<ol>
<li><p>pyecharts – 动态交互可视化库</p>
<ol>
<li>Bar, Bar3D, Pie, Map, …</li>
</ol>
</li>
</ol>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="sd">"""</span>
</span><span class='line'><span class="sd">kwargs为通用配置项</span>
</span><span class='line'><span class="sd">"""</span>
</span><span class='line'><span class="n">add</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">x_axis</span><span class="p">,</span> <span class="n">y_axis</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span>
</span><span class='line'> <span class="n">grid3d_opacity</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
</span><span class='line'> <span class="n">grid3d_shading</span><span class="o">=</span><span class="s">'color'</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="kn">from</span> <span class="nn">pyecharts</span> <span class="kn">import</span> <span class="n">Bar3D</span>
</span><span class='line'>
</span><span class='line'><span class="n">bar3d</span> <span class="o">=</span> <span class="n">Bar3D</span><span class="p">(</span><span class="s">"3D 柱状图示例"</span><span class="p">,</span> <span class="n">width</span><span class="o">=</span><span class="mi">1200</span><span class="p">,</span> <span class="n">height</span><span class="o">=</span><span class="mi">600</span>
</span><span class='line'> <span class="o">...</span>
</span><span class='line'> <span class="o">...</span>
</span><span class='line'><span class="n">bar3d</span><span class="o">.</span><span class="n">add</span><span class="p">(</span>
</span><span class='line'> <span class="s">""</span><span class="p">,</span>
</span><span class='line'> <span class="n">x_axis</span><span class="p">,</span>
</span><span class='line'> <span class="n">y_axis</span><span class="p">,</span>
</span><span class='line'> <span class="p">[[</span><span class="n">d</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">d</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">d</span><span class="p">[</span><span class="mi">2</span><span class="p">]]</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">data</span><span class="p">],</span>
</span><span class='line'> <span class="n">is_visualmap</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
</span><span class='line'> <span class="n">visual_range</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">20</span><span class="p">],</span>
</span><span class='line'> <span class="n">visual_range_color</span><span class="o">=</span><span class="n">range_color</span><span class="p">,</span>
</span><span class='line'> <span class="n">grid3d_width</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span>
</span><span class='line'> <span class="n">grid3d_depth</span><span class="o">=</span><span class="mi">80</span><span class="p">,</span>
</span><span class='line'><span class="p">)</span>
</span><span class='line'><span class="n">bar3d</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="s">'html/3D_Bar.html'</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
<ol>
<li>. WordCloud</li>
</ol>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="sd">"""</span>
</span><span class='line'><span class="sd">name str图例名称</span>
</span><span class='line'><span class="sd">attr list属性名称</span>
</span><span class='line'><span class="sd">value list属性所对应的值</span>
</span><span class='line'><span class="sd">"""</span>
</span><span class='line'><span class="n">add</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">attr</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span>
</span><span class='line'> <span class="n">shape</span><span class="o">=</span><span class="s">"circle"</span><span class="p">,</span>
</span><span class='line'> <span class="n">word_gap</span><span class="o">=</span><span class="mi">20</span><span class="p">,</span>
</span><span class='line'> <span class="n">word_size_range</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
</span><span class='line'> <span class="n">rotate_step</span><span class="o">=</span><span class="mi">45</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="kn">from</span> <span class="nn">pyecharts</span> <span class="kn">import</span> <span class="n">WordCloud</span>
</span><span class='line'><span class="n">words</span> <span class="o">=</span> <span class="p">[</span><span class="s">'Python'</span><span class="p">,</span> <span class="s">'文本分析'</span><span class="p">,</span> <span class="s">'编程'</span><span class="p">]</span>
</span><span class='line'><span class="n">freqs</span> <span class="o">=</span> <span class="p">[</span><span class="mi">100</span><span class="p">,</span><span class="mi">200</span><span class="p">,</span><span class="mi">50</span><span class="p">]</span>
</span><span class='line'><span class="n">wordcloud</span> <span class="o">=</span> <span class="n">WordCloud</span><span class="p">(</span><span class="n">width</span><span class="o">=</span><span class="mi">1300</span><span class="p">,</span> <span class="n">height</span><span class="o">=</span><span class="mi">620</span><span class="p">)</span>
</span><span class='line'><span class="n">wordcloud</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s">""</span><span class="p">,</span> <span class="n">words</span><span class="p">,</span> <span class="n">freqs</span><span class="p">,</span> <span class="n">word_size_range</span><span class="o">=</span><span class="p">[</span><span class="mi">20</span><span class="p">,</span> <span class="mi">100</span><span class="p">])</span>
</span><span class='line'><span class="n">wordcloud</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="s">'html/wordcloud.html'</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
<h3>数据分析库</h3>
<ol>
<li><p>pandas</p>
<ol>
<li><p>Series</p>
<ol>
<li>Series(list), Series(dict)</li>
<li>notnull, s2>2, s2[s2>2]</li>
<li>数学运算</li>
<li>value_counts(normalize=False)</li>
<li>head, tail, sample</li>
<li>plot(kind=‘bar’)</li>
</ol>
</li>
<li><p>DataFrame</p>
<ol>
<li>列向量和行向量组成的数据结构</li>
</ol>
</li>
</ol>
</li>
</ol>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
</span><span class='line'><span class="kn">import</span> <span class="nn">pandas</span> <span class="kn">as</span> <span class="nn">pd</span>
</span><span class='line'><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">40</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="mi">8</span><span class="p">,</span><span class="mi">5</span><span class="p">),</span>
</span><span class='line'> <span class="n">index</span><span class="o">=</span><span class="p">[</span><span class="s">'one'</span><span class="p">,</span><span class="s">'two'</span><span class="p">,</span><span class="s">'three'</span><span class="p">,</span><span class="s">'four'</span><span class="p">,</span><span class="s">'five'</span><span class="p">,</span><span class="s">'six'</span><span class="p">,</span><span class="s">'seven'</span><span class="p">,</span><span class="s">'eight'</span><span class="p">],</span>
</span><span class='line'> <span class="n">columns</span><span class="o">=</span><span class="p">[</span><span class="s">'a'</span><span class="p">,</span><span class="s">'b'</span><span class="p">,</span><span class="s">'c'</span><span class="p">,</span><span class="s">'d'</span><span class="p">,</span><span class="s">'e'</span><span class="p">])</span>
</span></code></pre></td></tr></table></div></figure>
<pre><code> 3. `df.drop('one', axis='rows', inplace=False)`
4. `df['a']`, `df.a` - 索引列名称
5. `df.index` - 行索引
6. `df[:3]` - 行索引查询
7. `df.loc[['two','three'], ['a','d']]` - 子DataFrame查询
8. `df.loc[:, ['a','d']]`, `df[['a', 'd']]` - 所有行, a,d 列
9. `df.loc[['two','three'], :]` - 所有列
10. `df.describe()`
11. `df['colname'] = 10009` - 新建或更新列
12. 列方向进行**聚合运算**
</code></pre>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">df</span><span class="o">.</span><span class="n">agg</span><span class="p">({</span><span class="s">'colname1'</span><span class="p">:</span> <span class="p">[</span><span class="s">'mean'</span><span class="p">,</span> <span class="n">diyfunc</span><span class="p">],</span>
</span><span class='line'> <span class="s">'colname2'</span><span class="p">:</span> <span class="p">[</span><span class="s">'mean'</span><span class="p">,</span> <span class="s">'sum'</span><span class="p">]})</span>
</span></code></pre></td></tr></table></div></figure>
<pre><code> *. `df.agg('mean')`- 针对所有列的聚合操作
13. 针对值进行分组 - **groupby**
1. `df.set_index('date').groupby('name')['ext price'].resample("M").sum()` - 针对name列的值进行分组
2. `df.groupby(['name', pd.Grouper(key='date', freq='M')])['ext price'].sum()`
3. `df.groupby('violation_raw').search_conducted.agg(['mean', 'count'])`
14. `newdf=df+100`
15. `newdf>120`
16. `df[newdf>120]`
17. **时间操作**
1. pd.date_range() 生成一个时间段索引
1. `pd.date_range('2017/10/11 10:00:01', '20181030', freq='3M') `
2. pd.to_datetime(data) 将data序列数据转换为datetime类型Series
</code></pre>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">datatime</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">to_datetime</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</span><span class='line'><span class="n">datatime</span><span class="o">.</span><span class="n">dt</span><span class="o">.</span><span class="n">year</span>
</span><span class='line'><span class="n">datatime</span><span class="o">.</span><span class="n">dt</span><span class="o">.</span><span class="n">month</span><span class="o">.</span><span class="n">value_counts</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>
<pre><code> 3. 指定日期为索引
1.
</code></pre>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">tm_rng</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">date_range</span><span class="p">(</span><span class="s">'2017-12-31 12:00:00'</span><span class="p">,</span><span class="n">periods</span><span class="o">=</span><span class="mi">20</span><span class="p">,</span> <span class="n">freq</span><span class="o">=</span><span class="s">'5H'</span><span class="p">)</span>
</span><span class='line'><span class="n">tm_series</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">tm_rng</span><span class="p">)),</span> <span class="n">index</span><span class="o">=</span><span class="n">tm_rng</span><span class="p">)</span>
</span><span class='line'><span class="n">tm_series</span><span class="p">[</span><span class="s">'2017'</span><span class="p">]</span>
</span></code></pre></td></tr></table></div></figure>
<pre><code> 4. resample
1. `df.set_index('date').resample('2M')['ext price'].sum()`
18. 读取csv或excell文件为dataFrame
1. `df = pd.read_excel('data/sample-salesv3.xlsx', encoding='gbk')`
</code></pre>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[我是如何一步步走向职场死胡同的]]></title>
<link href="http://www.fengjimin.com/blog/2018/09/13/how-did-i-step-into-a-career-dead-end/"/>
<updated>2018-09-13T21:54:00+08:00</updated>
<id>http://www.fengjimin.com/blog/2018/09/13/how-did-i-step-into-a-career-dead-end</id>
<content type="html"><![CDATA[<h3>前言</h3>
<p>2018年，我从普通程序员变成了无业游民，不惑之年近在眼前，前方却好像找不到路了，是一个死胡同。10年以前，这一切都难以想象。我是怎么走到了这一步？</p>
<!--more-->
<p>我是一个内向、敏感、谨慎、细致的人，从小学到高中，总的来说，我是父母眼中争气的儿子，老师眼里不需要怎么操心的学生。家在农村，主要由于家里几个子女的教育，作为普通农民的父母负担沉重。我在2000年考入大学，入学不久又幸运拿到了新加坡教育部的本科奖学金，赴新留学，解了父母燃眉之急。</p>
<p>也许造就我人生前20年的顺风顺水的性格行为模式，同样也造就了我在大学以后越来越平庸，以及出社会在职场中不如意的种子。从小学到高考的12年，我的生活除了读书还是读书。在与同学老师的关系中，我是自我的、单一的。看不到人性的复杂，没有磨练基本的社会生存技能。同样，我也并不了解真正的自我，看不到我的缺点足以让我四处碰壁。我的心性也没有得到磨练，总之面对更广阔的世界，我太无知，太幼稚了。</p>
<p>然而我自认还是一个正常的人，也经历了相当的自我成长。在大学和职场中，也抓住了一些机会，然而却在不知不觉中，做了当时看起来也没什么大不了的抉择，令我一步步的走了下坡路。在这即将奔四的岁月关头，我想把我的经历略作分享。也许没有太多普适性，但也希望给来者一些借鉴。</p>
<h3>新加坡大学生活</h3>
<p>新加坡的大学生活由大半年的英语强化课程起步。当时比较自我封闭，除了同组同学，与其他人交流不多。总觉得那些所谓积极的人，很多充斥着傲慢和自大，难以交流。真正上了大学以后，自己选课，自习，住宿。除了几个熟悉的同学，跟外界的交流更少。没有了高考这种“终极考验”，很快自己就懈怠了，加之选了计算机专业，之前零基础。成绩出来，很难看，同时也很惶恐，怕拿不到荣誉学位，3年毕业。这在当时的我看来是挺丢人的一件事。好在最后决定分流的一学期，拿到了将将过的CAP，进了荣誉stream。结果又是为毕业论文挣扎。</p>
<p>主要问题是自己缺乏动手能力。国内小学到中学，衡量一切的标准是试卷，是分数。而学习计算机，动手能力才是关键。不能动手，所有学到的东西都是虚的，不扎实，是没有自信的。说到底还是自己缺乏自律，不能专注深入的学习。这一点，给了我很大影响，不仅仅是在大学里。</p>
<p>本科毕业时由于毕设的关系，获得了一个做嵌入式系统分析的Samarjit教授的提供的助研职位，做了大半年的GPU算法研究。后来忝列Terence教授门下，做图像、面部识别的研究，同时读研。同样由于水平、专注不够，只是忙于学业，没有什么成果。这段时间最大的收获是结识了实验室的优秀的同门，大部分来自国内名校。当时觉得自己跟他们的水平差距不大，今天回头看各自境遇差别，实际上差距巨大。</p>
<p>这就如同样是运行软件，底层操作系统存在巨大差异，结果是逻辑清晰条理分明的操作系统A，运行各种软件如鱼得水；而理不清关系，bug百出的操作系统B，运行一个简单的软件就已经勉为其难，叫苦连天了。虽然当时自己也过了PhD就读资格考试，但已经心生退意。加之当时自己本科毕业的同学这时也开始步入正轨，心理也开始失衡了。</p>
<p>10年过去了，回头看看学术这条路，反倒有可能是适合自己的路子。只是自己从大学开始，并未建立起良好的学习习惯。遇到困难挫折，没有谦卑的心态，必胜的信念，向导师、同门请教。说到底，面对做一个技术工作者所必须的基本技能，我没有选择正视、征服，而是逃避它。当时认为在工业界应该不需要深入的研究，更适合我。</p>
<h3>在新加坡的工作</h3>
<p>2008年由于家里亲人的变故，我错过了校园招聘季。回头找到第一份工作，是NCS（新加坡政府软件服务商）的一个开发职位，然而不在开发部门，是一个Infra部门，同事和领导都缺乏开发经验。对于缺乏工业界经验的我来说，这个起步并不理想。1年后，辞职又找了一个邮政领域的小公司（十几人规模）的开发职位。其实也是自己缺乏经验，只看到了他们能提供更好的薪水，而拒掉了一个更好的平台（Gemalto）的机会。不管如何，这份工作我憋足了劲要证明自己，学习和工作的劲头很足。做了好几个实际项目，包括邮包自动检测，消息中间件集成（ESB）等，得到美国大老板赏识，去了Boston一个月。然而却被越南同事忌恨，也不被印度小领导信任。1年多之后，再次选择跳槽去了PayPal。</p>
<p>在PayPal的工作算是自己在互联网领域的起步。其实在PayPal之前，自己已经建立起了软件开发的信心。然而在PayPal，我得到的教训是，做好一份工作，远远不止软件开发写代码那么简单。内部协调，需求迭代，跨部门协作，获取同事和领导信任、认可，这其中每一项，最重要的是沟通。而这沟通，并不仅仅是指你跟同事说了什么，还包括你表现出来了一切行为，被同事看在眼里，还意味着什么。我在Compliance部门，印度同事占大头、包括部门领导，还有4，5个东南亚越南、菲律宾、印尼、新加坡的同事，以及我还有另外一个来自中国的同事。一开始我表现出来的势头是专注于学习PayPal的开发技术、谦卑。然而这并不对某些印度同事和领导的胃口，也经常成了甩锅对象。接着，我的表现出了不满和不合作。到后来就干脆放弃获得认可的努力，随心所欲了。由于工作时间自由，我甚至常常打乒乓球，桌上足球，在这方面跟印度同事较劲。</p>
<p>当然以上总结的教训，其实当时我并没有认识到。由于一开始亲人的变故，我有个很迫切的念头，就是一定要回国。因此我把在新加坡的经历仅仅作为满足教育部奖学金合同、和赚钱的需求。当时为了省钱，我甚至一直住得很偏，上下班都得挤个把小时的地铁，导致几乎每天的工作的感觉都是累和负面的情绪。</p>
<p>于是在PayPal工作了两年后，我回国了。</p>
<h3>最初回国的几个月</h3>
<p>在新加坡工作的最后一段时间，我接触到了Ruby on Rails。当时觉得跟PayPal的前后端沉重的架构比起来，简直是神器。它提倡的约定高于配置（convention over configuration），不重复（DRY）的理念，让我有找到真理的感觉。我觉得好像创业梦想就在眼前了。有了RoR，何苦又累于公司内部各种消耗与内斗呢，起码接接活也能养活自己吧？</p>
<p>于是刚一回国，我通过高中同学接到一个实现分销会计系统的活。RoR搭起应用架构起来，快是快了，界面却简陋的很，用户体验也接近于没有。然而自己对UI设计、前端实现，发现还是有个巨大的鸿沟没办法填。项目做完了，也没信心做别的项目了。又拖了几个月希望想到创业好点子一鸣惊人，然而不管是点子、设计、还是实现、推广，实在没谱。只好又回到工业界。</p>
<p>教训是创业没有银弹，no silver bullet，不仅仅指软件开发。</p>
<h3>回国的第一份工作</h3>
<p>13年冬我到了上海加入平安付。当时由于支付宝和余额宝的影响，第三方支付和互联网金融热潮涌动。当时平安付起步半年多，已经发展到三四百人规模了。组织上主要有来自新加坡PayPal和X宝的两拨人领导。在平安付我真正体会到了第三方支付的业务和宏大的技术架构，这个架构主要源流于X宝。在技术上，相比当时的PayPal，X宝更引领潮流。毕竟PayPal创始于90年代，当时主要编程语言是c++。</p>
<p>公司管理并不完善，在开发和部署的持续集成上有所欠缺，消耗了很多不必要的人力。不可避免的来自新加坡PayPal和国内X宝的两拨人在管理理念上有巨大分歧。虽然我只是门户的开发小lead，也看到了两拨人的互斗。整体上评价，X宝系的人更aggressive，更具野心。而PayPal系，由于对国内公司文化和行事风格的不了解，更趋向于和谐和但求无过。包括我自己，也算卷入了大斗争下面的小漩涡。被某国内同事踩了几脚，像我这样只专注技术，沟通不畅的人，不踩我踩谁呢？</p>
<p>幸运的是，不管是在门户还是其他部门，我认识了一些优秀的国内同事和领导。从他们身上我能学习到一些在公司为人处事的方式。坚韧，能抓住重点，能从对方视角出发，表达自己的需求，这些都难能可贵。</p>
<p>斗争以PayPal系退出为结局。我也无心恋战。14年和在高校工作的女友在武汉结婚，回武汉似乎是必然选择。</p>
<h3>在武汉的工作</h3>
<p>15年中，我回到了武汉。刚开始加入了一个汽车资讯类小型互联网公司，公司外表看起来年轻活力，然而很快我便感受到其内在文化与我个性之间的不协调。其一是要尽快表现能出成果的能力与融入圈子的活力，否则你能很快感受到每个人的三六九等。换言之也算一种草莽习气吧。其二是让我深刻感受到自己不再年轻，而一堆的年轻同事他们的思维和圈子与我差距巨大。</p>
<p>仅仅三个月我从公司离职，内心备受打击。之后曾尝试转型到教育培训领域，然而通过试讲又发现自己似乎缺乏当讲师的“天分”，整个人状态太紧。此时认识了一个刚从1号店出来的朋友，推荐我去1号店。说是工作压力不大，基本上不怎么加班。虽然将信将疑，好歹算是救命稻草。</p>
<p>16年5月入职后发现确实如此。此时1号店的大股东是沃尔玛，平台和流程比较规范，让我找到了一些原来在PayPal和平安付的感觉。然而好景不长，很快传出要被京东收购的消息，到8月份尘埃落定。此时各种架构调整，山雨欲来风满楼。开始几个月仍然在做1号店业务，然而公司打卡等等制度在收紧。</p>
<p>自己仍然只专注我的技术工作，接手了一堆杂七杂八的活，好歹也在尽心尽力的干。越来越感受到被甩锅的次数见多，加班也越加常态化。等架构调整落定时，1号店文化成功转型为京东文化。站没站对位置，态度有没有到位是基本要求，做得多是不够的。</p>
<p>公平的说，1号店是好平台，京东的平台更好。开发、运营的组织架构专业、有效，是不输PayPal的。但是不像在PayPal、甚至在平安付，看不到让我觉得闪光的人性。无论能力大小，我只看到附庸和拍马，没有给真正的人性留下空间。也许只是我层次不够。不要错误理解，我的同事都是优秀的人。我只是说京东的文化没有给他们发挥个性的空间。Anyway，我再一次辞职。而这一次，我对我融入国内公司文化的能力，是前所未有的悲观了。</p>
<h3>总结</h3>
<p>从京东出来，我在武汉仍然参加了几次面试。深深的感觉到，属于我的职场机会不多了。就算给了机会，跟我的内心要求的还是有差距的。我做够了开发工作，开发维护的流程了如指掌，然而我体会不到任何有乐趣、激动人心的东西。低层次的开发工作只能陷于跟需求开发测试管理各方扯皮拉筋的泥潭，只能是别人升职加薪的炮灰。在国内IT职场语境下，早过了35岁的年纪，没到管理层，在职场已经进入死胡同了，更别提更高的层级了。总结经验教训：</p>
<ol>
<li>职场走管理路径需要更高情商，技术在其次</li>
<li>如果专注于技术，则需要做到深度专业化不可替代</li>
<li>创业需要机遇、资源、情商，天时地利人和</li>
<li>认清自我，找到自身深层次的问题</li>
<li>专注，找到属于自己的节奏；不要被外界带乱节奏</li>
</ol>
<p>回首职业生涯，缺乏明确的目标，走了无数弯路，于是几乎只能原地踏步。并不是没学到东西，只是涉及的领域太多，学的多而杂，深度不够。也许大部分的开发工作不需要太深入，却不利于我职业的发展。说到底，只有做到不可替代，才能战胜中年危机。如果一开始就目标明确，聚焦于算法，系统架构，安全，或者AI，会有不同……</p>
<p>另外就是由于没有建立自信，一直以来被其他人带乱节奏。同学中有做金融的金领，有创业成功翻身做主人的。每个人都有自己的素质的和境遇，都是不可复制的。保持开放的心态很重要，但决定是自己的，了解自己比去了解别人是如何成功的更重要。</p>
<p>无论如何，好的坏的，都是自己。这一二十年也并非一无是处，至少教训是深刻的。了解自己的目标，找到自己的定位，解决影响我最深的问题。也许走过去，就是一片晴天。</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[浅谈软件开发]]></title>
<link href="http://www.fengjimin.com/blog/2013/09/02/an-engineers-view-of-application-development/"/>
<updated>2013-09-02T20:47:00+08:00</updated>
<id>http://www.fengjimin.com/blog/2013/09/02/an-engineers-view-of-application-development</id>
<content type="html"><![CDATA[<blockquote><p>第一篇日志写于2013年。2015年的今日，经历一个充实的互联网业务的开发周期，回头看看，理解又有不同。说到底软件开发是定义问题，解决问题，二者合一。开发的过程绝不仅仅是写代码，测试，上线，支持。而是始终围绕其中的一个主题：用户、业务、产品、开发，各种角色不断沟通，提出各自理解，以各种形式如描叙、图表、代码、产品、UI、体验。。。这一切的一切，都是软件开发的一部分，每一个部分，都有同样的重要性，都可以遵循同样的方法论。最终的目标，是所有stakeholder达到一致的理解，从而从中得到自己想要的价值。</p></blockquote>
<!--more-->
<blockquote><p>另一方面，目前主流软件开发的模式，是面向对象编程，是基于OOP之上的设计模式。这是基础思想方法，是开发人员赖以搭高楼、赖以组装赛车的百宝箱。面向对象主要的思想来源，在于概念的不同抽象层次，对应了父类子类关系，既符合重用原则，也更符合人类思维。
当然，现实由于其复杂性，各种问题各种关连。从现实的问题中找到最基本最具重要性的维度来抽象，这需要一个匠人的直觉。这也是软件开发最有趣最激动人心的部分。没有武功秘籍，没有万能钥匙。真正有的，只存乎一心。。。</p></blockquote>
<p>从大学学编程开始，到如今学习工作的经验也有十多年了。记得大学毕业一、二年的时候和两个朋友讨论到同批不少同学转行去做金融，表达了羡慕之情，但那两个朋友异口同声的说其实我还蛮适合做IT的。姑且当作他们发现我身上适合搞IT做软件的特质吧。既然在这一行坚持了下来，也有必要做一个总结回顾，目的是从经验教训中找到规律，得到一个更清晰的视野。</p>
<p>软件开发，如同建房子、文艺创作，是创造的活动。任何创造的活动，需要工具，也需要想象力，完成一个复杂的项目则需要多人协作。创造，也是左脑和右脑协作的结果。左脑负责计算，逻辑推理，右脑负责情景想象，左右脑合作，才能做出符合需求、同时保证质量的产品。</p>
<p>关乎右脑想象力的方面，在建筑行业，想象力比较容易得到实物参照，一栋三维的建筑模型可以把成品外形模仿得八九不离十。而创作文学、绘画、音乐，是对现实的描叙或理念的抽象，似乎更关乎右脑记忆和情感的功能。而软件的设计，则是要满足用户的信息和功能的需求，这种需求难以有固化的实物参照，是一种概念的抽象和概念关系的梳理，且需要精确固化；它也不像文学和音乐可以有生动的现实做参照、表达的理念不需要精确。</p>
<p>另一方面关乎左脑的逻辑推理。一个建筑工程需要对工程涉及的各要素如人力、材料等进行必要测算，技术方面主要是结构力学和工程管理的应用。文艺的创作是对文字或绘画、音乐元素的组合，需要的是有对运用这些元素进行自我表达的感觉(Feel)。而软件的设计开发，需要把握编程语言，功能框架（函数库），运行机理，以及合理运用他们来表达清晰的概念关系，其最终产品不仅需要面向终端用户，也需要面向技术人员（包括自己），保证可维护和可扩展。</p>
<p>需求实现是应用开发的缘起。可以根据操作信息的不同类型、面向用户提供的功能进行分类，比如多媒体应用，网络应用，游戏应用，大数据应用，等等。在我多年应用开发的经验中，需求的理解是重要一环。其难点是需求与所运用技术的要求或限制的清晰区分。往往在思考需求的时候，缺乏经验的程序员会与所要运用的技术实现的要求和限制混为一谈。比如 Java deploy 的环境要求。只需要记住，所有的需求都是面向软件用户的。这种需求是完全独立于实现的语言或者框架的，可以用更高级的语言（甚至自然语言）来进行表达的。</p>
<p>大多数应用可以分为前端和后端来开发，也就可以运用MVC模式（视图-模型-控制器）。为保证维护性和可扩展性，需要使用DRY和正交设计的等等原则。本文和以后的系列博文，我将根据自己应用开发的经验，对不同需求的应用开发过程进行总结，尽量找到内在规律，推广到普遍软件开发的工作中，提高效率和质量。</p>
]]></content>
</entry>
</feed>