-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
470 lines (257 loc) · 397 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>DELIN</title>
<subtitle>TO LEARN WELL,TO LIVE WELL.</subtitle>
<link href="https://idelin.github.io/atom.xml" rel="self"/>
<link href="https://idelin.github.io/"/>
<updated>2023-04-19T09:23:36.232Z</updated>
<id>https://idelin.github.io/</id>
<author>
<name>DELIN</name>
</author>
<generator uri="https://hexo.io/">Hexo</generator>
<entry>
<title>剧集分享-《小太郎一个人生活》</title>
<link href="https://idelin.github.io/2022/08/26/%E5%89%A7%E9%9B%86%E5%88%86%E4%BA%AB-%E3%80%8A%E5%B0%8F%E5%A4%AA%E9%83%8E%E4%B8%80%E4%B8%AA%E4%BA%BA%E7%94%9F%E6%B4%BB%E3%80%8B/"/>
<id>https://idelin.github.io/2022/08/26/%E5%89%A7%E9%9B%86%E5%88%86%E4%BA%AB-%E3%80%8A%E5%B0%8F%E5%A4%AA%E9%83%8E%E4%B8%80%E4%B8%AA%E4%BA%BA%E7%94%9F%E6%B4%BB%E3%80%8B/</id>
<published>2022-08-26T08:00:55.000Z</published>
<updated>2023-04-19T09:23:36.232Z</updated>
<content type="html"><![CDATA[<h2 id="一、简介"><a href="#一、简介" class="headerlink" title="一、简介"></a>一、简介</h2><p>不红的漫画家狩野进（横山裕饰）在禁止带孩子入住的公寓里，过着自甘堕落的日子。有一天，隔壁的房间里突然搬来了一个5岁的孩子小太郎（川原瑛都），不知为什么只有他一个人住。本以为自己比谁都坚强，但偶尔也会流露出孩子气的内心想法，当与小太郎相遇时，包括狩野在内，住在同一栋公寓里的房客们也重新体会到了作为成年人的责任和体谅他人的心情的重要性，一点点地成长起来。</p><p><img src="https://jihulab.com/idelin/pic_bed/-/raw/main/e6c9d24ely1h5k8701y7mj209u0dwwg5.jpg" alt="小太郎一个人生活"></p><h2 id="二、资源"><a href="#二、资源" class="headerlink" title="二、资源"></a>二、资源</h2><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs wiki">链接: https://pan.baidu.com/s/1VOt3b66zMUP0Q4LXn54Aew?pwd=16S0 <br>提取码: 16S0<br></code></pre></td></tr></table></figure>]]></content>
<summary type="html"><h2 id="一、简介"><a href="#一、简介" class="headerlink" title="一、简介"></a>一、简介</h2><p>不红的漫画家狩野进（横山裕饰）在禁止带孩子入住的公寓里，过着自甘堕落的日子。有一天，隔壁的房间里突然搬来了一个5岁的孩子小太</summary>
<category term="生活" scheme="https://idelin.github.io/categories/%E7%94%9F%E6%B4%BB/"/>
<category term="电视剧" scheme="https://idelin.github.io/tags/%E7%94%B5%E8%A7%86%E5%89%A7/"/>
</entry>
<entry>
<title>数据平台产品全功能流程图</title>
<link href="https://idelin.github.io/2022/08/26/%E6%95%B0%E6%8D%AE%E5%B9%B3%E5%8F%B0%E4%BA%A7%E5%93%81%E5%85%A8%E5%8A%9F%E8%83%BD%E6%B5%81%E7%A8%8B%E5%9B%BE/"/>
<id>https://idelin.github.io/2022/08/26/%E6%95%B0%E6%8D%AE%E5%B9%B3%E5%8F%B0%E4%BA%A7%E5%93%81%E5%85%A8%E5%8A%9F%E8%83%BD%E6%B5%81%E7%A8%8B%E5%9B%BE/</id>
<published>2022-08-26T03:25:39.000Z</published>
<updated>2022-08-26T03:59:37.245Z</updated>
<content type="html"><![CDATA[<iframe id="embed_dom" name="embed_dom" frameborder="0" style="display:block;width:800px; height:400px;" src="https://www.processon.com/embed/61a43fee63768975f8547859"></iframe><p><strong>详细流程表述了数据平台类型的产品的全功能流程图</strong></p><p><strong>适用于 安全分析、资产管理、风控、数据分析 等产品。</strong></p>]]></content>
<summary type="html"><iframe id="embed_dom" name="embed_dom" frameborder="0" style="display:block;width:800px; height:400px;" src="https://www.processon.com/embe</summary>
<category term="架构" scheme="https://idelin.github.io/categories/%E6%9E%B6%E6%9E%84/"/>
<category term="流程图" scheme="https://idelin.github.io/tags/%E6%B5%81%E7%A8%8B%E5%9B%BE/"/>
<category term="架构图" scheme="https://idelin.github.io/tags/%E6%9E%B6%E6%9E%84%E5%9B%BE/"/>
</entry>
<entry>
<title>如何维护更新日志</title>
<link href="https://idelin.github.io/2022/08/25/%E5%A6%82%E4%BD%95%E7%BB%B4%E6%8A%A4%E6%9B%B4%E6%96%B0%E6%97%A5%E5%BF%97/"/>
<id>https://idelin.github.io/2022/08/25/%E5%A6%82%E4%BD%95%E7%BB%B4%E6%8A%A4%E6%9B%B4%E6%96%B0%E6%97%A5%E5%BF%97/</id>
<published>2022-08-25T11:58:46.000Z</published>
<updated>2023-04-19T09:23:36.243Z</updated>
<content type="html"><![CDATA[<h3 id="更新日志是什么？"><a href="#更新日志是什么？" class="headerlink" title="更新日志是什么？"></a>更新日志是什么？</h3><p>更新日志（Change Log）是一个由人工编辑，以时间为倒序的列表， 以记录一个项目中所有版本的显著变动。</p><h3 id="为何要提供更新日志？"><a href="#为何要提供更新日志？" class="headerlink" title="为何要提供更新日志？"></a>为何要提供更新日志？</h3><p>为了让用户和开发人员更简单明确的知晓项目在不同版本之间有哪些显著变动。</p><h3 id="哪些人需要更新日志？"><a href="#哪些人需要更新日志？" class="headerlink" title="哪些人需要更新日志？"></a>哪些人需要更新日志？</h3><p>人人需要更新日志。无论是消费者还是开发者，软件的最终用户都关心软件所包含什么。 当软件有所变动时，大家希望知道改动是为何、以及如何进行的。</p><h3 id="指导原则"><a href="#指导原则" class="headerlink" title="指导原则"></a>指导原则</h3><ul><li>记住日志是写给<em>人</em>的，而非机器。</li><li>每个版本都应该有独立的入口。</li><li>同类改动应该分组放置。</li><li>版本与章节应该相互对应。</li><li>新版本在前，旧版本在后。</li><li>应包括每个版本的发布日期。</li></ul><h3 id="变动类型"><a href="#变动类型" class="headerlink" title="变动类型"></a>变动类型</h3><ul><li><code>Added</code> 新添加的功能。</li><li><code>Changed</code> 对现有功能的变更。</li><li><code>Deprecated</code> 已经不建议使用，准备很快移除的功能。</li><li><code>Removed</code> 已经移除的功能。</li><li><code>Fixed</code> 对bug的修复</li><li><code>Security</code> 对安全的改进</li></ul><h3 id="如何减少维护更新日志的精力？"><a href="#如何减少维护更新日志的精力？" class="headerlink" title="如何减少维护更新日志的精力？"></a>如何减少维护更新日志的精力？</h3><p>在文档最上方提供 <code>Unreleased</code> 区块以记录即将发布的更新内容。</p><p>这样有两大意义：</p><ul><li>大家可以知道在未来版本中可能会有哪些变更</li><li>在发布新版本时，可以直接将<code>Unreleased</code>区块中的内容移动至新发 布版本的描述区块就可以了</li></ul><p>当然有，下面就是一些糟糕的方式。</p><h3 id="使用git日志"><a href="#使用git日志" class="headerlink" title="使用git日志"></a>使用git日志</h3><p>使用git日志作为更新日志是个非常不好的方式：git日志充满各种无意义的信息， 如合并提交、语焉不详的提交标题、文档更新等。</p><p>提交的目的是记录源码的演化。 一些项目会清理提交记录，一些则不会。</p><p>更新日志的目的则是记录重要的变更以供最终受众阅读，而记录范围通常涵盖多次提交。</p><p>但这是个重要的参考，特别是规范的 git 日志。</p><h3 id="无视即将弃用功能"><a href="#无视即将弃用功能" class="headerlink" title="无视即将弃用功能"></a>无视即将弃用功能</h3><p>当从一个版本升级至另一个时，人们应清楚（尽管痛苦）的知道哪些部分将出现问题。 应该允许先升级至一个列出哪些功能将会被弃用的版本，待去掉那些不再支持的部分后， 再升级至把那些弃用功能真正移除的版本。</p><p>即使其他什么都不做，也要在更新日志中列出derecations，removals以及其他重大变动。</p>]]></content>
<summary type="html"><h3 id="更新日志是什么？"><a href="#更新日志是什么？" class="headerlink" title="更新日志是什么？"></a>更新日志是什么？</h3><p>更新日志（Change Log）是一个由人工编辑，以时间为倒序的列表， 以记录一个项目中所有</summary>
<category term="开发" scheme="https://idelin.github.io/categories/%E5%BC%80%E5%8F%91/"/>
<category term="更新日志" scheme="https://idelin.github.io/tags/%E6%9B%B4%E6%96%B0%E6%97%A5%E5%BF%97/"/>
</entry>
<entry>
<title>Jks证书转crt&key</title>
<link href="https://idelin.github.io/2022/08/24/Jks%E8%AF%81%E4%B9%A6%E8%BD%ACcrt-key/"/>
<id>https://idelin.github.io/2022/08/24/Jks%E8%AF%81%E4%B9%A6%E8%BD%ACcrt-key/</id>
<published>2022-08-24T11:38:43.000Z</published>
<updated>2023-04-19T09:23:36.233Z</updated>
<content type="html"><![CDATA[<ol><li>jks -> p12<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">keytool -importkeystore -srckeystore server.key -srcstoretype JKS -deststoretype PKCS12 -destkeystore server.p12<br></code></pre></td></tr></table></figure></li></ol> <span id="more"></span><ol start="2"><li><p>p12 -> pem</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">openssl pkcs12 -in server.p12 -out server.pem <br></code></pre></td></tr></table></figure></li><li><p>p12 -> key、crt</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">openssl pkcs12 -in server.p12 -nocerts -out server.key<br>openssl pkcs12 -in server.p12 -nokeys -clcerts -out server.crt<br></code></pre></td></tr></table></figure></li><li><p>去除生成key文件时的口令，避免每次使用都要输入：</p></li></ol><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs shell">cp server.key server.key.org<br>openssl rsa -in server.key.org -out server.key<br></code></pre></td></tr></table></figure><ol start="5"><li>配置nginx.conf，在server块下面添加以下配置：</li></ol><figure class="highlight nginx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs nginx"><span class="hljs-attribute">ssl</span> <span class="hljs-literal">on</span>;<br><span class="hljs-attribute">ssl_certificate</span> /usr/local/nginx/conf/server.crt; //证书<br><span class="hljs-attribute">ssl_certificate_key</span> /usr/local/nginx/conf/server.key; //私钥<br></code></pre></td></tr></table></figure><ol start="6"><li>重新加载配置: <figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">/usr/local/nginx/sbin/nginx -s reload<br></code></pre></td></tr></table></figure></li></ol>]]></content>
<summary type="html"><ol>
<li>jks -&gt; p12<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">keytool -importkeystore -srckeystore server.key -srcstoretype JKS -deststoretype PKCS12 -destkeystore server.p12<br></code></pre></td></tr></table></figure></li>
</ol></summary>
<category term="https" scheme="https://idelin.github.io/categories/https/"/>
<category term="https" scheme="https://idelin.github.io/tags/https/"/>
<category term="证书" scheme="https://idelin.github.io/tags/%E8%AF%81%E4%B9%A6/"/>
</entry>
<entry>
<title>垃圾回收器G1</title>
<link href="https://idelin.github.io/2022/08/09/%E5%9E%83%E5%9C%BE%E5%9B%9E%E6%94%B6%E5%99%A8G1/"/>
<id>https://idelin.github.io/2022/08/09/%E5%9E%83%E5%9C%BE%E5%9B%9E%E6%94%B6%E5%99%A8G1/</id>
<published>2022-08-09T02:18:39.000Z</published>
<updated>2023-04-19T09:23:36.233Z</updated>
<content type="html"><![CDATA[<p>G1运作步骤：</p><ol><li>初始标记(stop the world事件CPU停顿只处理垃圾)；</li><li>并发标记(与用户线程并发执行)；</li><li>最终标记(stop the world事件，CPU停顿处理垃圾)；</li><li>筛选回收(stop the world事件根据用户期望的GC停顿时间回收)(注意：CMS在这一步不需要stop the world)（阿里问为何停顿时间可以设置，参考：<a href="https://www.cnblogs.com/aspirant/p/8663872.html">G1 垃圾收集器架构和如何做到可预测的停顿(阿里)</a></li></ol><span id="more"></span><p>与其他GC收集器相比，G1具备如下特点：</p><p>1、并行于并发：G1能充分利用CPU、多核环境下的硬件优势，使用多个CPU（CPU或者CPU核心）来缩短stop-The-World停顿时间。部分其他收集器原本需要停顿<a href="http://lib.csdn.net/base/java">Java</a>线程执行的GC动作，G1收集器仍然可以通过并发的方式让java程序继续执行。</p><p>2、分代收集：虽然G1可以不需要其他收集器配合就能独立管理整个GC堆，但是还是保留了分代的概念。它能够采用不同的方式去处理新创建的对象和已经存活了一段时间，熬过多次GC的旧对象以获取更好的收集效果。</p><p>3、空间整合：与CMS的“标记–清理”算法不同，G1从整体来看是基于“标记整理”算法实现的收集器；从局部上来看是基于“复制”算法实现的。</p><p>4、可预测的停顿：这是G1相对于CMS的另一个大优势，降低停顿时间是G1和ＣＭＳ共同的关注点，但Ｇ１除了追求低停顿外，还能建立可预测的停顿时间模型，能让使用者明确指定在一个长度为M毫秒的时间片段内，</p><p>上面几个步骤的运作过程和CMS有很多相似之处。初始标记阶段仅仅只是标记一下GC Roots能直接关联到的对象，并且修改TAMS的值，让下一个阶段用户程序并发运行时，能在正确可用的Region中创建新对象，这一阶段需要停顿线程，但是耗时很短，并发标记阶段是从GC Root开始对堆中对象进行可达性分析，找出存活的对象，这阶段时耗时较长，但可与用户程序并发执行。而最终标记阶段则是为了修正在并发标记期间因用户程序继续运作而导致标记产生变动的那一部分标记记录，虚拟机将这段时间对象变化记录在线程Remenbered Set Logs里面，最终标记阶段需要把Remembered Set Logs的数据合并到Remembered Set Logs里面，最终标记阶段需要把Remembered Set Logs的数据合并到Remembered Set中，这一阶段需要停顿线程，但是可并行执行。最后在筛选回收阶段首先对各个Region的回收价值和成本进行排序，根据用户所期望的GC停顿时间来制定回收计划。</p>]]></content>
<summary type="html"><p>G1运作步骤：</p>
<ol>
<li>初始标记(stop the world事件CPU停顿只处理垃圾)；</li>
<li>并发标记(与用户线程并发执行)；</li>
<li>最终标记(stop the world事件，CPU停顿处理垃圾)；</li>
<li>筛选回收(stop the world事件根据用户期望的GC停顿时间回收)(注意：CMS在这一步不需要stop the world)（阿里问为何停顿时间可以设置，参考：<a href="https://www.cnblogs.com/aspirant/p/8663872.html">G1 垃圾收集器架构和如何做到可预测的停顿(阿里)</a></li>
</ol></summary>
<category term="java" scheme="https://idelin.github.io/categories/java/"/>
<category term="java" scheme="https://idelin.github.io/tags/java/"/>
</entry>
<entry>
<title>通用WEB框架的一些思考</title>
<link href="https://idelin.github.io/2022/08/05/%E9%80%9A%E7%94%A8WEB%E6%A1%86%E6%9E%B6%E7%9A%84%E4%B8%80%E4%BA%9B%E6%80%9D%E8%80%83/"/>
<id>https://idelin.github.io/2022/08/05/%E9%80%9A%E7%94%A8WEB%E6%A1%86%E6%9E%B6%E7%9A%84%E4%B8%80%E4%BA%9B%E6%80%9D%E8%80%83/</id>
<published>2022-08-05T08:39:06.000Z</published>
<updated>2023-04-19T09:23:36.239Z</updated>
<content type="html"><![CDATA[<p>基础技术框架的研究和推广往往是很重要的动作，跟同事聊的过程中，产生了一些想法，分享如下。</p><span id="more"></span><h3 id="一、所需的功能点"><a href="#一、所需的功能点" class="headerlink" title="一、所需的功能点"></a>一、所需的功能点</h3><p>1、权限集成，登录登出、用户管理、角色管理、组织架构、认证安全（例如：验证码、多因子认证、二步验证、限制 IP 登录等）、单点登录、第三方认证接入认证（例如：oAuth3）、支持LDAP、支持应用授权（用于开放 OPENAPI 或其他对接） 等； </p><p>2、接口规范类（如：Response），先有规范再有相关类，建议使用 标准 Restful 规范来定义标准接口相关类；</p><p>3、migration管理，建议使用 flyway 做sql文件的版本管理；</p><p>4、操作日志记录，安全产品必备功能点，可以考虑用注解形式来做；</p><p>5、升级管理，支持全量、增量等升级模式，支持拓展升级类来适应不同产品自己的升级业务（例如：规则包的升级）；</p><p>6、许可管理，各产品强需求，可以直接对接公司新的许可管理平台，一方面更好推广新的公司许可标准推广，另一方面新产品可以无需重复开发许可模块；</p><p>7、配置管理，支持系统参数配置； </p><p>8、OPENAPI框架，支持管理 OPEANAPI，监控调用次数、健康度、熔断等情况，支持开放动态接口文档（可以借用 Swagger，但安全性原因，建议自行开发页面）； </p><p>9、常用工具类集成，集成IPUtils、HttpUtils、EhcacheUtils、Md5Utils 基础工具类–必备，操作ES、Kafka、HBase、Zookeeper等大数据组件的操作工具类–可选； </p><p>10、集成脚本，集成 构建、安装、启停 等基础脚本。 </p><h3 id="二、形态思考"><a href="#二、形态思考" class="headerlink" title="二、形态思考"></a>二、形态思考</h3><p>1、开发框架选型 </p><p>大概率是不同语言会提供多个版本，Java 建议是基于 Springboot（流行、简单、易集成），Python建议是基于 Sanic（别问为什么不是 Django，因为它快）。 </p><p>2、开发及部署模式 </p><p>建议兼容微服务模式和单体模式。在如今的产品选型里，有相当多的产品采用微服务开发，支持微服务模式和单体模式 两种开发部署是很有必要的，代码维护可以是一套，注册中心可以支持几款主流（Eureka、ZooKeeper、Consul、Nacos） </p><p>3、权限注册中心独立 </p><p>功能点里有提到权限集成，这个是此基础框架是必要的且第一优先级的功能点，建议可以独立起来，一方面可以独立维护权限相关代码逻辑，一方面当各个产品做集成的时候可以直接切换到同一个权限服务（公司内产品集成痛点）。无论是否微服务形态，都可以是独立权限服务。</p><h3 id="三、迭代开发模式"><a href="#三、迭代开发模式" class="headerlink" title="三、迭代开发模式"></a>三、迭代开发模式</h3><p>建议由公共研发团队来主导公司内开源+持续维护推进，业务部门研发提需求+提 issue+提 PR 做贡献。</p><p>说实话到公司这个体量，各开发语言没有形成一个统一的开发框架确实也不多见，先从 基础WEB框架 入手也好的，也很期待公司可以统一牵头做这个事情，让大家快速实现产品业务。 </p>]]></content>
<summary type="html"><p>基础技术框架的研究和推广往往是很重要的动作，跟同事聊的过程中，产生了一些想法，分享如下。</p></summary>
<category term="开发" scheme="https://idelin.github.io/categories/%E5%BC%80%E5%8F%91/"/>
<category term="web" scheme="https://idelin.github.io/tags/web/"/>
</entry>
<entry>
<title>git patch使用</title>
<link href="https://idelin.github.io/2022/08/05/git%20patch%E4%BD%BF%E7%94%A8/"/>
<id>https://idelin.github.io/2022/08/05/git%20patch%E4%BD%BF%E7%94%A8/</id>
<published>2022-08-05T08:00:52.000Z</published>
<updated>2023-04-19T09:23:36.233Z</updated>
<content type="html"><![CDATA[<p>Git 提供了两种补丁方案，一种是通过 git diff 生成的 .diff 文件，第二种是通过 git format-patch 生成的 .patch 文件。</p><span id="more"></span><p>通过 git diff 生成的文件不含有 commit 信息，可以指定文件生成 diff，也可以指定单个 commit， 多个 commit 生成 。</p><p>通过 git format-patch 生成的 .patch 文件 含有 commmit 信息。一个 commit 对应一个 patch 文件，也可以整合到一个 patch 文件里。</p><p>此方案只针对某一个 branch 的代码的迁移。</p><h2 id="1-git-format-patch"><a href="#1-git-format-patch" class="headerlink" title="1. git format-patch"></a>1. git format-patch</h2><p>此命令用于生成 patch 文件。</p><p>某次提交以后的所有 patch:</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">git format-patch [commitId]<br></code></pre></td></tr></table></figure><h2 id="2-git-am"><a href="#2-git-am" class="headerlink" title="2. git am"></a>2. git am</h2><p>此命令用于打 patch 文件入分支。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">git am xxx.patch<br></code></pre></td></tr></table></figure><h2 id="3-Usage"><a href="#3-Usage" class="headerlink" title="3. Usage"></a>3. Usage</h2><p>使用流程：start -> 云桌面查看待同步分支最新 commitId -> 内网切换到需要同步的分支 -> 内网 git format-patch -> 打包 patch 文件内传外（包含代码，需要有高权限的人才可以传） -> 云桌面解包 -> git am -> over。</p>]]></content>
<summary type="html"><p>Git 提供了两种补丁方案，一种是通过 git diff 生成的 .diff 文件，第二种是通过 git format-patch 生成的 .patch 文件。</p></summary>
<category term="git" scheme="https://idelin.github.io/categories/git/"/>
<category term="git" scheme="https://idelin.github.io/tags/git/"/>
</entry>
<entry>
<title>AiLPHA REST API指南</title>
<link href="https://idelin.github.io/2022/03/26/AiLPHA-REST-API%E6%8C%87%E5%8D%97/"/>
<id>https://idelin.github.io/2022/03/26/AiLPHA-REST-API%E6%8C%87%E5%8D%97/</id>
<published>2022-03-26T07:42:44.000Z</published>
<updated>2023-04-19T09:23:36.232Z</updated>
<content type="html"><![CDATA[<h2 id="1-摘要"><a href="#1-摘要" class="headerlink" title="1. 摘要"></a>1. 摘要</h2><p>AiLPHA REST API指南作为一种设计规范，鼓励开发同学通过RESTful HTTP接口访问资源。</p><p>本文档建立了AiLPHA REST API应该遵循的指导原则，以便开发统一一致的RESTful接口。</p><span id="more"></span><h2 id="2-引言"><a href="#2-引言" class="headerlink" title="2. 引言"></a>2. 引言</h2><p>为了给开发同学提供更流畅的开发体验，让API遵循统一的设计准则很重要，这些准则能让API简单且直观。</p><p>API一致性的好处在于使团队拥有统一的代码、模式、文档风格和设计策略。</p><p>这些准则旨在达成如下目标：</p><ul><li>尽可能地遵循行业普遍接受的 REST/HTTP 最佳实践；</li><li>内部接口层面，让前后端对接更为顺畅；</li><li>对外接口层面，让第三方开发同学都可以轻松的通过REST接口访问的AiLPHA OPEN API；</li><li>接口测试层面，让测试同学可以按照标准更容易地去进行自动化接口测试；</li><li>让其他部门、合作伙伴使用这些准则来设计自己的REST API。</li></ul><h3 id="2-1-推荐阅读"><a href="#2-1-推荐阅读" class="headerlink" title="2.1 推荐阅读"></a>2.1 推荐阅读</h3><p>了解REST架构风格背后的理念，更有助于开发优秀的基于 HTTP 的服务。 如果您对 RESTful 设计不熟悉，请参阅以下优秀资源：</p><ul><li><a href="https://tools.ietf.org/html/rfc7231">RFC7231</a> – 定义HTTP/1.1 语义规范的权威文档</li><li><a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm">REST论文</a> – Roy Fielding网络架构论文中关于REST的章节，“架构风格与基于网络的软件体系结构设计”</li><li><a href="http://www.ruanyifeng.com/blog/2011/09/restful.html">理解RESTful架构</a> – 一篇介绍RESTful架构较为准确的文章</li><li><a href="https://www.zhihu.com/question/27785028">生动理解REST</a> – 一篇介绍REST较为生动易懂的文章</li><li><a href="https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#1444-example-response">微软API规范</a> – 本指南的主要参考规范</li></ul><h2 id="3-指南解读"><a href="#3-指南解读" class="headerlink" title="3. 指南解读"></a>3. 指南解读</h2><h3 id="3-1-应用指南"><a href="#3-1-应用指南" class="headerlink" title="3.1 应用指南"></a>3.1 应用指南</h3><p>这些准则适用于AiLPHA基础研发部（或任何公司内其他部门、公司外合作伙伴）开发服务里的公开REST API。私有或内部API也应该尝试遵循这些准则，保证一致性不仅对外部用户有价值，对内部服务使用者也很有价值，而这些准则为任何服务都提供了最佳实践。 </p><p>有合理理由可不遵循这些准则，如：</p><p>一些服务与某些外部定义的REST API互操作时必须与某些外部的API兼容，允许其无法遵循这些准则。</p><p>一些服务可能具有需要特殊功能上或性能上的需求，允许其无法遵循这些准则。</p><h3 id="3-2-历史接口"><a href="#3-2-历史接口" class="headerlink" title="3.2 历史接口"></a>3.2 历史接口</h3><p>不建议仅仅为了遵从指南而对这些指南之前的旧服务或模块进行重大更改。</p><h3 id="3-3-“要求”关键字"><a href="#3-3-“要求”关键字" class="headerlink" title="3.3 “要求”关键字"></a>3.3 “要求”关键字</h3><p>本文档中的 ”必须”（MUST、REQUIRED、SHALL）、”禁止”（MUST NOT、SHALL NOT）、”推荐”（SHOULD、RECOMMENDED）、 ”可以”（MAY）、”可选“（OPTIONAL） 等关键字的详细解释见 <a href="https://tools.ietf.org/html/rfc2119">RFC2119</a>。</p><h2 id="4-分类"><a href="#4-分类" class="headerlink" title="4. 分类"></a>4. 分类</h2><p>服务必须符合下面定义的分类法。</p><h3 id="4-1-错误（Error）"><a href="#4-1-错误（Error）" class="headerlink" title="4.1 错误（Error）"></a>4.1 错误（Error）</h3><p>错误，定义为因客户端向服务传递错误数据，导致服务端拒绝该请求。客户端传递错误的或者不合法的数据的情况通常返回 “4xx” 的 HTTP错误码。</p><p>错误不会影响API的整体可用性。</p><p>例如 无效凭证、错误的参数、未知的版本ID等。</p><h3 id="4-2-故障（Faults）"><a href="#4-2-故障（Faults）" class="headerlink" title="4.2 故障（Faults）"></a>4.2 故障（Faults）</h3><p>故障，定义为服务无法正确返回数据以响应有效的客户端请求。通常会返回“5xx”HTTP错误码。</p><p>故障会影响整体 API 的可用性。故障意味着服务端代码出现故障，可能会影响整体的API使用。</p><p>例如 数据库连接超时。</p><h2 id="5-客户端要求"><a href="#5-客户端要求" class="headerlink" title="5. 客户端要求"></a>5. 客户端要求</h2><p>为确保客户端更好的接入REST服务，客户端应遵循以下最佳实践:</p><h3 id="5-1-字段忽略原则"><a href="#5-1-字段忽略原则" class="headerlink" title="5.1 字段忽略原则"></a>5.1 字段忽略原则</h3><p>客户端<strong>必须</strong>安全地忽略未约定的字段。</p><p>在产品迭代的过程中，有些服务接口可能在不更改版本号的情况下向响应数据中添加字段。此类服务接口<strong>必须</strong>在其文档中注明，而客户端<strong>必须</strong>可以忽略这些未知字段。</p><h3 id="5-2-字段排序忽略原则"><a href="#5-2-字段排序忽略原则" class="headerlink" title="5.2 字段排序忽略原则"></a>5.2 字段排序忽略原则</h3><p>客户端处理响应数据时<strong>禁止</strong>依赖服务端JSON响应数据字段的顺序。</p><p>当服务器返回的 JSON 对象中的字段顺序发生变化，客户端应当能够正确进行解析处理。</p><p>服务端<strong>可以</strong>在返回值中显式说明指定某些元素按特定方式进行排序，客户端处理数据时<strong>可以</strong>依赖于服务端明确指定的排序行为。</p><h3 id="5-3-无声失效原则"><a href="#5-3-无声失效原则" class="headerlink" title="5.3 无声失效原则"></a>5.3 无声失效原则</h3><p>当客户端请求带可选功能参数的服务时（包含带可选的头部信息），<strong>必须</strong>对服务端的返回格式有一定兼容性，<strong>可以</strong>忽略这些特定功能。</p><p>例如 分页数、排序、国际化等自定义参数的支持和返回格式的兼容。</p><h2 id="6-基础原则"><a href="#6-基础原则" class="headerlink" title="6. 基础原则"></a>6. 基础原则</h2><h3 id="6-1-URL结构☆"><a href="#6-1-URL结构☆" class="headerlink" title="6.1 URL结构☆"></a>6.1 URL结构☆</h3><p>结构约束：</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs http">{schema}://{serviceRoot}/{collection}[/{id}]<br>{schema}://{host}:{port}/{prefix}/{version}/{collection}[/{id}]<br></code></pre></td></tr></table></figure><ul><li>schema - API接口支持的协议（http或https）</li><li>{serviceRoot} – 站点URL (site URL) + 服务的根路径的组合<ul><li>host - 服务的域名或IP</li><li>port - 服务的端口</li><li>prefix - 接口的标识前缀：”/api”</li><li>version - 接口的版本</li></ul></li><li>{collection} – 集合的名称，复数</li><li>{id} – 唯一标识属性的值，唯一标识中原则上不允许使用”/“，如果业务需要，一定要带”\“转义。</li></ul><p>API URL路径结构应该是友好的易于理解的。甚至用户无需通过阅读API文档能够猜出相关结构和路径。</p><p>正例：<a href="https://ailpha.com/api/v1.0/agents/JHF8UE6H5W34D/memorys">https://ailpha.com/api/v1.0/agents/JHF8UE6H5W34D/memorys</a></p><p>显而易见，这是一个： 版本为v1.0的 获取标识为“JHF8UE6H5W34D”探针内存数据资源 的API。</p><p>反例：<a href="https://ailpha.com/api/EWS/OData/agents('JHF8UE6H5W34D')/Folders('MEM')">https://ailpha.com/api/EWS/OData/agents('JHF8UE6H5W34D')/Folders('MEM')</a></p><p>虽然也能理解部分含义，但结构混乱，解析困难且不友好。</p><p>URL结尾为<strong>资源（resource）</strong>，既然是资源，必须是名词，必须为复数形式。</p><p>反例：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-built_in">GET</span> /api/agent/list?<span class="hljs-attribute">agentType</span>=ainta<br><span class="hljs-built_in">GET</span> /api/agent/getAgentById?<span class="hljs-attribute">id</span>=JHF8UE6H5W34D<br>POST /api/agent/deleteAgentById?<span class="hljs-attribute">id</span>=JHF8UE6H5W34D<br></code></pre></td></tr></table></figure><p>正例：</p><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs gradle">GET <span class="hljs-regexp">/api/</span>v1.<span class="hljs-number">0</span>/agents?agentType=ainta<br>GET <span class="hljs-regexp">/api/</span>v1.<span class="hljs-number">0</span><span class="hljs-regexp">/agents/</span>JHF8UE6H5W34D<br><span class="hljs-keyword">DELETE</span> <span class="hljs-regexp">/api/</span>v1.<span class="hljs-number">0</span><span class="hljs-regexp">/agents/</span>JHF8UE6H5W34D<br></code></pre></td></tr></table></figure><h3 id="6-2-URL长度"><a href="#6-2-URL长度" class="headerlink" title="6.2 URL长度"></a>6.2 URL长度</h3><p>HTTP/1.1 标准中并未对请求长度做限制，其中包含URL。<br>RFC 7230 原文：</p><blockquote><p>HTTP does not place a predefined limit on the length of a request-line. […] A server that receives a request-target longer than any URI it wishes to parse MUST respond with a 414 (URI Too Long) status code.</p></blockquote><p>对应翻译：</p><blockquote><p>HTTP不会对请求行的长度设置预定义的限制。[…]如果服务器接收的请求目标比它希望解析的任何URI都长，则必须使用414（URI过长）状态码进行响应。</p></blockquote><p>此处约定URL长度不超过2,000个字符，约定依据是IE所能接收的URL长度为2083（其他浏览器也有限定但大于这个值）。</p><p>不同客户端支持的最长 URL 长度参见以下资料：</p><ul><li><a href="http://stackoverflow.com/a/417184">http://stackoverflow.com/a/417184</a></li><li><a href="https://blogs.msdn.microsoft.com/ieinternals/2014/08/13/url-length-limits/">https://blogs.msdn.microsoft.com/ieinternals/2014/08/13/url-length-limits/</a></li></ul><h3 id="6-3-规范标识符"><a href="#6-3-规范标识符" class="headerlink" title="6.3 规范标识符"></a>6.3 规范标识符</h3><p>在URL中，针对具体的资源，必须提供一个唯一且稳定的标识符。</p><p>包含规范标识符的URL的一个例子依然是6.1中的正例：</p><p><a href="https://ailpha.com/api/v1.0/agents/JHF8UE6H5W34D/memory">https://ailpha.com/api/v1.0/agents/JHF8UE6H5W34D/memory</a></p><p>标识符推荐使用有含义的唯一性索引字段（IP、邮箱），也可以是暴露给客户端的主键。</p><h3 id="6-4-操作方法☆"><a href="#6-4-操作方法☆" class="headerlink" title="6.4 操作方法☆"></a>6.4 操作方法☆</h3><p>对于资源的具体操作类型，由HTTP Method表示。</p><table><thead><tr><th>方法</th><th>描述</th><th>是否幂等</th></tr></thead><tbody><tr><td><strong>GET</strong></td><td>返回资源的当前值<br />成功状态码：200（OK）</td><td>✔</td></tr><tr><td><strong>POST</strong></td><td>根据提供的数据创建一个新资源，或者提交一个操作<br />成功状态码：201（Created）</td><td></td></tr><tr><td><strong>PUT</strong></td><td>更新资源（客户端提供改变后的完整资源，或者称：替换资源）<br />成功状态码：200（OK）</td><td>✔</td></tr><tr><td><strong>PATCH</strong></td><td>更新资源（客户端提供改变的属性）<br />成功状态码：200（OK）<br />针对不存在的资源的 PATCH 调用必须使用409（Conflict）响应错误</td><td></td></tr><tr><td><strong>DELETE</strong></td><td>删除资源<br />成功状态码：204（No Content）</td><td>✔</td></tr><tr><td>HEAD</td><td>返回GET响应的资源的元数据，不常用<br />成功状态码：200（OK）</td><td>✔</td></tr><tr><td>OPTIONS</td><td>获取关于请求的信息，不常用<br />成功状态码：200（OK）</td><td>✔</td></tr></tbody></table><p>例子：</p><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs gradle">GET /agents：列出所有探针<br>POST /agents：新建一个探针<br>GET <span class="hljs-regexp">/agents/</span>{agentCode}：获取某个指定探针的信息<br>PUT <span class="hljs-regexp">/agents/</span>{agentCode}：更新某个指定探针的信息（提供该探针的全部信息）<br>PATCH <span class="hljs-regexp">/agents/</span>{agentCode}：更新某个指定探针的的信息（提供该探针的部分信息）<br><span class="hljs-keyword">DELETE</span> <span class="hljs-regexp">/agents/</span>{agentCode}：删除某个指定探针<br>GET <span class="hljs-regexp">/agents/</span>{agentCode}/monitors：列出某个指定探针的所有监控数据<br><span class="hljs-keyword">DELETE</span> <span class="hljs-regexp">/agents/</span>{agentCode}<span class="hljs-regexp">/monitors/</span>{monitorId}：删除某个指定探针的指定监控数据<br></code></pre></td></tr></table></figure><h3 id="6-5-标准请求头"><a href="#6-5-标准请求头" class="headerlink" title="6.5 标准请求头"></a>6.5 标准请求头</h3><p>当前有规范的请求头参数如下所示，使用这些请求头参数不是强制性的，但如果使用它们则必须始终一致地遵循 AiLPHA REST API指南服务使用。</p><table><thead><tr><th>请求头参数</th><th>类型</th><th>描述</th></tr></thead><tbody><tr><td>Accept</td><td>Content type</td><td>请求的响应内容类型<br />一般为 Accept: application/json</td></tr><tr><td>Accept-Language</td><td>“en”, “cn”, etc.</td><td>指定响应的首选语言。<br />AiLPHA平台支持国际化（i18n），无特殊情况必须使用Accept-Language来指定语言。</td></tr><tr><td>Accept-Charset</td><td>Charset type like “UTF-8”</td><td>指定响应的编码。默认值是UTF-8。</td></tr><tr><td>Content-Type</td><td>Content type</td><td>请求正文的媒体（Mime）类型 (PUT/POST/PATCH)<br />媒体类型在<a href="https://www.rfc-editor.org/rfc/rfc2854.txt">RFC2854</a>有明确定义</td></tr><tr><td>Prefer</td><td>return=minimal, return=representation</td><td>如果指定了return = minimal首选项，则服务应该返回一个空主体（empty body）以响应一次成功的插入或更新。<br />如果指定了return = representation，则服务应该返回创建或更新的资源。<br />使用场景：客户端有时会需要从响应中获得资源，但有时响应会对带宽造成太大的影响而不想获得资源。</td></tr></tbody></table><h3 id="6-6-标准响应头"><a href="#6-6-标准响应头" class="headerlink" title="6.6 标准响应头"></a>6.6 标准响应头</h3><p>服务应该返回以下响应头参数。</p><table><thead><tr><th>响应头参数</th><th>类型</th><th>描述</th></tr></thead><tbody><tr><td>Content-Type</td><td>Content type</td><td>请求正文的媒体（Mime）类型 (PUT/POST/PATCH)<br />媒体类型在<a href="https://www.rfc-editor.org/rfc/rfc2854.txt">RFC2854</a>有明确定义</td></tr><tr><td>X-Frame-Options</td><td>DENY、SAMEORIGIN、ALLOW-FROM uri、NONE</td><td>指示允许一个页面可否在 <frame>,\ </iframe> 或者 <object> 中展现的标记。<br />在AiLPHA中为全局配置，无特殊需要接口开发不指定。</td></tr></tbody></table><h3 id="6-7-状态码☆"><a href="#6-7-状态码☆" class="headerlink" title="6.7 状态码☆"></a>6.7 状态码☆</h3><p>应使用标准HTTP状态码作为响应状态码</p><p>HTTP 状态码就是一个三位数，分成五个类别。</p><ul><li><code>1xx</code>：相关信息</li><li><code>2xx</code>：操作成功</li><li><code>3xx</code>：重定向</li><li><code>4xx</code>：客户端错误</li><li><code>5xx</code>：服务器错误</li></ul><p>这五大类总共包含<a href="https://en.wikipedia.org/wiki/List_of_HTTP_status_codes">100多种</a>状态码，覆盖了绝大部分可能遇到的情况。每一种状态码都有标准的（或者约定的）解释，客户端只需查看状态码，就可以判断出发生了什么情况，所以服务器应该返回尽可能精确的状态码。</p><p>API 不需要<code>1xx</code>状态码，下面介绍其他四类状态码的精确含义。</p><h4 id="6-7-1-常用状态码☆"><a href="#6-7-1-常用状态码☆" class="headerlink" title="6.7.1 常用状态码☆"></a>6.7.1 常用状态码☆</h4><p>以下列举了常用的状态码，表述其使用场景及对应使用说明。</p><h5 id="2xx状态码"><a href="#2xx状态码" class="headerlink" title="2xx状态码"></a>2xx状态码</h5><p><code>200</code>状态码表示操作成功，但是不同的方法可以返回更精确的状态码。</p><blockquote><ul><li>GET: 200 OK</li><li>POST: 201 Created</li><li>PUT: 200 OK</li><li>PATCH: 200 OK</li><li>DELETE: 204 No Content</li></ul></blockquote><p>上面代码中，<code>POST</code>返回<code>201</code>状态码，表示生成了新的资源；<code>DELETE</code>返回<code>204</code>状态码，表示资源已经不存在。</p><p>此外，<code>202 Accepted</code>状态码表示服务器已经收到请求，但还未进行处理，会在未来再处理，通常用于异步操作。<br>下面是一个例子。</p> <figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs http"><span class="hljs-meta">HTTP/1.1</span> <span class="hljs-number">202</span> Accepted<br>{<br> "task": {<br> "href": "/api/update-jobs/12345",<br> "id": "12345"<br> }<br>}<br></code></pre></td></tr></table></figure><h5 id="4xx-状态码"><a href="#4xx-状态码" class="headerlink" title="4xx 状态码"></a>4xx 状态码</h5><p><code>4xx</code>状态码表示客户端错误，主要有下面几种。</p><p><code>400 Bad Request</code>：服务器不理解客户端的请求，未做任何处理。</p><p><code>401 Unauthorized</code>：用户未提供身份验证凭据，或者没有通过身份验证。</p><p><code>403 Forbidden</code>：用户通过了身份验证，但是不具有访问资源所需的权限。</p><p><code>404 Not Found</code>：所请求的资源不存在，或不可用。</p><p><code>405 Method Not Allowed</code>：用户已经通过身份验证，但是所用的 HTTP 方法不在他的权限之内。</p><p><code>415 Unsupported Media Type</code>：客户端要求的返回格式不支持。比如，API 只能返回 JSON 格式，但是客户端要求返回 XML 格式。</p><p><code>422 Unprocessable Entity</code> ：客户端上传的附件无法处理，导致请求失败。</p><p><code>429 Too Many Requests</code>：客户端的请求次数超过限额。</p><h5 id="5xx-状态码"><a href="#5xx-状态码" class="headerlink" title="5xx 状态码"></a>5xx 状态码</h5><p><code>5xx</code>状态码表示服务端错误。一般来说，API 不会向用户透露服务器的详细信息，所以只要两个状态码就够了。</p><p><code>500 Internal Server Error</code>：客户端请求有效，服务器处理时发生了意外。</p><p><code>503 Service Unavailable</code>：服务器无法处理请求，一般用于网站维护状态。</p><h4 id="6-7-2-状态码表"><a href="#6-7-2-状态码表" class="headerlink" title="6.7.2 状态码表"></a>6.7.2 状态码表</h4><table><thead><tr><th align="left">状态码</th><th align="left">状态码英文名称</th><th align="left">中文描述</th></tr></thead><tbody><tr><td align="left">100</td><td align="left">Continue</td><td align="left">继续。<a href="http://www.dreamdu.com/webbuild/client_vs_server/">客户端</a>应继续其请求</td></tr><tr><td align="left">101</td><td align="left">Switching Protocols</td><td align="left">切换协议。服务器根据客户端的请求切换协议。只能切换到更高级的协议，例如，切换到HTTP的新版本协议</td></tr><tr><td align="left"></td><td align="left"></td><td align="left"></td></tr><tr><td align="left">200</td><td align="left">OK</td><td align="left">请求成功。一般用于GET与POST请求</td></tr><tr><td align="left">201</td><td align="left">Created</td><td align="left">已创建。成功请求并创建了新的资源</td></tr><tr><td align="left">202</td><td align="left">Accepted</td><td align="left">已接受。已经接受请求，但未处理完成</td></tr><tr><td align="left">203</td><td align="left">Non-Authoritative Information</td><td align="left">非授权信息。请求成功。但返回的meta信息不在原始的服务器，而是一个副本</td></tr><tr><td align="left">204</td><td align="left">No Content</td><td align="left">无内容。服务器成功处理，但未返回内容。在未更新网页的情况下，可确保浏览器继续显示当前文档</td></tr><tr><td align="left">205</td><td align="left">Reset Content</td><td align="left">重置内容。服务器处理成功，用户终端（例如：浏览器）应重置文档视图。可通过此返回码清除浏览器的表单域</td></tr><tr><td align="left">206</td><td align="left">Partial Content</td><td align="left">部分内容。服务器成功处理了部分GET请求</td></tr><tr><td align="left"></td><td align="left"></td><td align="left"></td></tr><tr><td align="left">300</td><td align="left">Multiple Choices</td><td align="left">多种选择。请求的资源可包括多个位置，相应可返回一个资源特征与地址的列表用于用户终端（例如：浏览器）选择</td></tr><tr><td align="left">301</td><td align="left">Moved Permanently</td><td align="left">永久移动。请求的资源已被永久的移动到新URI，返回信息会包括新的URI，浏览器会自动定向到新URI。今后任何新的请求都应使用新的URI代替</td></tr><tr><td align="left">302</td><td align="left">Found</td><td align="left">临时移动。与301类似。但资源只是临时被移动。客户端应继续使用原有URI</td></tr><tr><td align="left">303</td><td align="left">See Other</td><td align="left">查看其它地址。与301类似。使用GET和POST请求查看</td></tr><tr><td align="left">304</td><td align="left">Not Modified</td><td align="left">未修改。所请求的资源未修改，服务器返回此状态码时，不会返回任何资源。客户端通常会缓存访问过的资源，通过提供一个头信息指出客户端希望只返回在指定日期之后修改的资源</td></tr><tr><td align="left">305</td><td align="left">Use Proxy</td><td align="left">使用代理。所请求的资源必须通过代理访问</td></tr><tr><td align="left">306</td><td align="left">Unused</td><td align="left">已经被废弃的HTTP状态码</td></tr><tr><td align="left">307</td><td align="left">Temporary Redirect</td><td align="left">临时重定向。与302类似。使用GET请求重定向</td></tr><tr><td align="left"></td><td align="left"></td><td align="left"></td></tr><tr><td align="left">400</td><td align="left">Bad Request</td><td align="left">客户端请求的语法错误，服务器无法理解</td></tr><tr><td align="left">401</td><td align="left">Unauthorized</td><td align="left">用户未提供身份验证凭据，或者没有通过身份验证</td></tr><tr><td align="left">402</td><td align="left">Payment Required</td><td align="left">保留，将来使用</td></tr><tr><td align="left">403</td><td align="left">Forbidden</td><td align="left">用户通过了身份验证，但是不具有访问资源所需的权限</td></tr><tr><td align="left">404</td><td align="left">Not Found</td><td align="left">所请求的资源不存在，或不可用</td></tr><tr><td align="left">405</td><td align="left">Method Not Allowed</td><td align="left">用户已经通过身份验证，但是所用的 HTTP 方法不在他的权限之内</td></tr><tr><td align="left">406</td><td align="left">Not Acceptable</td><td align="left">服务器无法根据客户端请求的内容特性完成请求</td></tr><tr><td align="left">407</td><td align="left">Proxy Authentication Required</td><td align="left">请求要求代理的身份认证，与401类似，但请求者应当使用代理进行授权</td></tr><tr><td align="left">408</td><td align="left">Request Time-out</td><td align="left">服务器等待客户端发送的请求时间过长，超时</td></tr><tr><td align="left">409</td><td align="left">Conflict</td><td align="left">服务器完成客户端的 PUT 请求时可能返回此代码，服务器处理请求时发生了冲突</td></tr><tr><td align="left">410</td><td align="left">Gone</td><td align="left">客户端请求的资源已经不存在。410不同于404，如果资源以前有现在被永久删除了可使用410代码，网站设计人员可通过301代码指定资源的新位置</td></tr><tr><td align="left">411</td><td align="left">Length Required</td><td align="left">服务器无法处理客户端发送的不带Content-Length的请求信息</td></tr><tr><td align="left">412</td><td align="left">Precondition Failed</td><td align="left">客户端请求信息的先决条件错误</td></tr><tr><td align="left">413</td><td align="left">Request Entity Too Large</td><td align="left">由于请求的实体过大，服务器无法处理，因此拒绝请求。为防止客户端的连续请求，服务器可能会关闭连接。如果只是服务器暂时无法处理，则会包含一个Retry-After的响应信息</td></tr><tr><td align="left">414</td><td align="left">Request-URI Too Large</td><td align="left">请求的URI过长（URI通常为网址），服务器无法处理</td></tr><tr><td align="left">415</td><td align="left">Unsupported Media Type</td><td align="left">服务器无法处理请求附带的媒体格式</td></tr><tr><td align="left">416</td><td align="left">Requested range not satisfiable</td><td align="left">客户端请求的范围无效</td></tr><tr><td align="left">417</td><td align="left">Expectation Failed</td><td align="left">服务器无法满足Expect的请求头信息</td></tr><tr><td align="left"></td><td align="left"></td><td align="left"></td></tr><tr><td align="left">500</td><td align="left">Internal Server Error</td><td align="left">服务器内部错误，无法完成请求</td></tr><tr><td align="left">501</td><td align="left">Not Implemented</td><td align="left">服务器不支持请求的功能，无法完成请求</td></tr><tr><td align="left">502</td><td align="left">Bad Gateway</td><td align="left">作为网关或者代理工作的服务器尝试执行请求时，从远程服务器接收到了一个无效的响应</td></tr><tr><td align="left">503</td><td align="left">Service Unavailable</td><td align="left">由于超载或系统维护，服务器暂时的无法处理客户端的请求。延时的长度可包含在服务器的Retry-After头信息中</td></tr><tr><td align="left">504</td><td align="left">Gateway Time-out</td><td align="left">充当网关或代理的服务器，未及时从远端服务器获取请求</td></tr><tr><td align="left">505</td><td align="left">HTTP Version not supported</td><td align="left">服务器不支持请求的HTTP协议的版本，无法完成处理</td></tr></tbody></table><p>更多状态码详情，参见：<a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html">RFC2616-sec10</a></p><h3 id="6-8-响应格式☆"><a href="#6-8-响应格式☆" class="headerlink" title="6.8 响应格式☆"></a>6.8 响应格式☆</h3><p>此章节为重点，此处规定了可读性较好并且一致的响应结果，开发人员使用 HTTP工具方法处理响应（TODO）。</p><p>没有特殊需求，推荐使用JSON作为传输格式。JSON属性名应该采用驼峰命名规范 (<a href="https://tools.ietf.org/html/rfc3864">RFC3864</a>)。</p><h4 id="6-8-1-响应格式指定"><a href="#6-8-1-响应格式指定" class="headerlink" title="6.8.1 响应格式指定"></a>6.8.1 响应格式指定</h4><p>客户端应该使用Accept参数请求响应格式。服务端可以选择性地忽略，如客户端发送多个Accept参数值，服务可以选择其中一个格式进行响应。</p><p>默认的响应格式（没有指定Accept）应该是application/json。</p><p><code>Accept: application/json</code></p><table><thead><tr><th>请求参数类型</th><th>响应类型</th></tr></thead><tbody><tr><td>application/json</td><td>必须是返回json格式</td></tr></tbody></table><h4 id="6-8-2-标准响应规范☆"><a href="#6-8-2-标准响应规范☆" class="headerlink" title="6.8.2 标准响应规范☆"></a>6.8.2 标准响应规范☆</h4><p>对于调用成功的情况，规定统一的成功响应的字段规范。</p><h5 id="SuccessResponse"><a href="#SuccessResponse" class="headerlink" title="SuccessResponse"></a>SuccessResponse</h5><p>对象类型</p><table><thead><tr><th>字段</th><th>类型</th><th>是否必填</th><th>描述</th></tr></thead><tbody><tr><td><code>data</code></td><td>Object、Object[] …</td><td>✔</td><td>一般为操作的资源，也可以是客户端约定的标识</td></tr><tr><td>…</td><td>…</td><td></td><td>其他业务需要的拓展字段</td></tr></tbody></table><p>成功响应<strong>必须</strong>是单个JSON对象。为保证一致性，该对象必须有一个名为“data”的键，其值类型按不同情况返回。</p><p>如果是单个资源的获取，“data”必须是单个JSON对象。</p><p>如果是多个资源的获取，“data”必须是包含0-n个对象的JSON数组。</p><h5 id="成功响应样例"><a href="#成功响应样例" class="headerlink" title="成功响应样例"></a>成功响应样例</h5><p>例子「单个对象」：</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs http">GET /api/agent-configs/soc<br>{<br> "data": {<br> "agentType":"soc",<br> "softVersion":"3.0.1",<br> "config":"",<br> "version":1611054006000<br> }<br>}<br></code></pre></td></tr></table></figure><p>例子「多个对象」</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><code class="hljs http">GET /api/agent-configs<br>{<br> "data": [<br> {<br> "agentType":"soc",<br> "softVersion":"3.0.1",<br> "config":"",<br> "version":1611054006000<br> },<br> {<br> "agentType":"ainta",<br> "softVersion":"1.1.3",<br> "config":"",<br> "version":1611054006000<br> }]<br>}<br></code></pre></td></tr></table></figure><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs http">GET /api/agent-configs<br>{<br> "data": []<br>}<br></code></pre></td></tr></table></figure><p>例子「成功执行」</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs http">DELETE /api/agents/HI6J5OEJ23FKH83W<br>{<br> "data":1<br>}<br></code></pre></td></tr></table></figure><h4 id="6-8-3-错误响应规范☆"><a href="#6-8-3-错误响应规范☆" class="headerlink" title="6.8.3 错误响应规范☆"></a>6.8.3 错误响应规范☆</h4><p>对于调用不成功的情况，规定统一的错误响应的字段规范。</p><h5 id="ErrorResponse"><a href="#ErrorResponse" class="headerlink" title="ErrorResponse"></a>ErrorResponse</h5><p>对象类型</p><table><thead><tr><th>字段</th><th>类型</th><th>是否必填</th><th>描述</th></tr></thead><tbody><tr><td><code>error</code></td><td>Error</td><td>✔</td><td>错误对象</td></tr><tr><td>…</td><td>…</td><td></td><td>其他业务需要的拓展字段</td></tr></tbody></table><h5 id="Error"><a href="#Error" class="headerlink" title="Error"></a>Error</h5><p>对象类型</p><table><thead><tr><th>字段</th><th>类型</th><th>是否必填</th><th>描述</th></tr></thead><tbody><tr><td><code>code</code></td><td>String</td><td>✔</td><td>服务器定义的固定错误代码，禁止随意定义，必须跟随版本发布。</td></tr><tr><td><code>message</code></td><td>String</td><td>✔</td><td><strong>必须</strong>是可读且易于理解，可作为面向用户的提示消息。</td></tr><tr><td><code>target</code></td><td>String</td><td></td><td>具体的目标。</td></tr><tr><td><code>details</code></td><td>Error[]</td><td></td><td>更细分的多个错误用错误组表示，Error对象数组类型。</td></tr><tr><td><code>innererror</code></td><td>InnerError</td><td></td><td>包含比当前对象更具体的错误信息的对象，InnerError对象类型。</td></tr></tbody></table><p>错误响应<strong>必须</strong>是单个JSON对象。该对象必须有一个名为“error”的键，其值<strong>必须</strong>是JSON对象。</p><p>error对象<strong>必须</strong>包含名称“code”和“message”的键值对，并且<strong>推荐</strong>包含譬如“target”、“details”和 “innererror” 的键值对。</p><p>“code”值是该服端务定义的错误代码，应该简单可读。与响应中指定的HTTP错误代码相比，此代码用作定位更具体的错误。</p><h5 id="InnerError"><a href="#InnerError" class="headerlink" title="InnerError"></a>InnerError</h5><p>对象类型</p><table><thead><tr><th>Property</th><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td><code>code</code></td><td>String</td><td>✔</td><td>比包含错误的提供的更具体的错误代码，允许自行拓展，区别于Error对象中的code。</td></tr><tr><td><code>innererror</code></td><td>InnerError</td><td></td><td>包含比当前对象更具体的错误信息的对象，InnerError对象类型。<br />最重要的是支持嵌套。</td></tr></tbody></table><p>如果要指明更为具体的错误，<strong>应该</strong>在“innererror” 对象中定义。</p><p>“message” 键值对的值<strong>必须</strong>是错误提示消息，<strong>必须</strong>是可读且易于理解，可作为面向用户的提示消息。</p><p>“target” 键值对的值 是指向错误的具体的目标（例如，错误中属性的名称）。</p><p>“details”键值对的值 必须是JSON对象数组，数组中的对象通常表示请求期间发生的不同的、相关的错误。</p><h5 id="Error-code"><a href="#Error-code" class="headerlink" title="Error.code"></a>Error.code</h5><p>通用的错误code（待完善）</p><table><thead><tr><th align="left">错误代码</th><th>响应码</th><th align="left">描述</th></tr></thead><tbody><tr><td align="left">BadArgument</td><td>400</td><td align="left">提供的参数无效</td></tr><tr><td align="left">BadUserArgument</td><td>400</td><td align="left">提供的用户参数无效</td></tr><tr><td align="left">ErrorUnsupportedOrderBy</td><td>400</td><td align="left">资源不支持排序 或 不支持使用指定的字段排序</td></tr><tr><td align="left">ErrorUnsupportedPaging</td><td>400</td><td align="left">资源不支持分页请求</td></tr><tr><td align="left">InvalidOperation</td><td>400</td><td align="left">请求在当前上下文中无效<br />例子：<br />JOB提交中，不可修改。<br />模型与请求版本不兼容无法更新。</td></tr><tr><td align="left">Unauthorized</td><td>401</td><td align="left">请求未获授权，无法访问资源。</td></tr><tr><td align="left">InvalidApiKey</td><td>401</td><td align="left">无效apiKey。</td></tr><tr><td align="left">NotFound</td><td>404</td><td align="left">找不到资源。</td></tr><tr><td align="left">RequestTimeout</td><td>408</td><td align="left">无法在允许的时间内完成该操作。</td></tr><tr><td align="left">Conflict</td><td>409</td><td align="left">请求资源冲突，POST已存在资源 或 PATCH不存在资源。</td></tr><tr><td align="left">Overlimit</td><td>409</td><td align="left">请求资源超出系统限制。</td></tr><tr><td align="left">InternalError</td><td>500</td><td align="left">执行时遇到内部错误。<br />具体异常类型自定义，放入innererror</td></tr><tr><td align="left">ServiceUnavailable</td><td>503</td><td align="left">服务暂时不可用。</td></tr><tr><td align="left">GatewayTimeout</td><td>504</td><td align="left">无法在允许的时间内完成该操作。</td></tr></tbody></table><h5 id="错误响应样例"><a href="#错误响应样例" class="headerlink" title="错误响应样例"></a>错误响应样例</h5><p>例子「带有”innererror“」</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><code class="hljs json"><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"error"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"code"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"BadArgument"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"message"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Previous passwords may not be reused"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"target"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"password"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"innererror"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"code"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"PasswordError"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"innererror"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"code"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"PasswordDoesNotMeetPolicy"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"minLength"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"6"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"maxLength"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"64"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"characterTypes"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><span class="hljs-string">"lowerCase"</span><span class="hljs-punctuation">,</span><span class="hljs-string">"upperCase"</span><span class="hljs-punctuation">,</span><span class="hljs-string">"number"</span><span class="hljs-punctuation">,</span><span class="hljs-string">"symbol"</span><span class="hljs-punctuation">]</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"minDistinctCharacterTypes"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"2"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"innererror"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"code"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"PasswordReuseNotAllowed"</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">}</span><br><span class="hljs-punctuation">}</span><br></code></pre></td></tr></table></figure><p>在本例中，基本的错误码是“BadArgument”，而“innererror”中提供了更具体的错误代码。 “PasswordReuseNotAllowed”代码可能是在之后的迭代中新增的，之前只返回“PasswordDoesNotMeetPolicy”。 这种增量型的添加方式并不会破坏老的客户端的处理过程，而又可以给开发者一些更详细的信息。</p><p>例子「带有“details”」</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><code class="hljs json"><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"error"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"code"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"BadArgument"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"message"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Multiple errors in ContactInfo data"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"target"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"ContactInfo"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"details"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span><br> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"code"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"NullValue"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"target"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"PhoneNumber"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"message"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Phone number must not be null"</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"code"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"NullValue"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"target"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"LastName"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"message"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Last name must not be null"</span><br> <span class="hljs-punctuation">}</span><span class="hljs-punctuation">,</span><br> <span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"code"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"MalformedValue"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"target"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Address"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"message"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Address is not valid"</span><br> <span class="hljs-punctuation">}</span><br> <span class="hljs-punctuation">]</span><br> <span class="hljs-punctuation">}</span><br><span class="hljs-punctuation">}</span><br></code></pre></td></tr></table></figure><p>在本例中，请求存在多处问题，每个错误都列在 “details” 字段中进行返回了。</p><h4 id="6-8-4-不支持的请求"><a href="#6-8-4-不支持的请求" class="headerlink" title="6.8.4 不支持的请求"></a>6.8.4 不支持的请求</h4><p>客户端可以请求当前不受支持的功能。 但服务端必须响应与本节一致的有效但不受支持的请求。</p><p>如果服务不支持一些功能，则必须在客户端请求API时返回错误响应。 </p><p>举例：</p><ul><li>用key在集合中检索，例如：<code>https://ailpha.com/api/v1.0/agents/JHF8UE6H5W34D</code></li><li>用条件过滤集合，例如：<code>https://ailpha.com/api/v1.0/agents?agentType=soc</code></li><li>通过_$page_和_$size_进行分页，例如：<code>https://ailpha.com/api/v1.0/agents?$page=5&$size=10</code></li><li>按_$orderBy_排序，例如：<code>https://ailpha.com/api/v1.0/agents?$orderBy=createTime desc</code></li></ul><p>错误响应<strong>必须</strong>是来自4xx系列的HTTP状态码，表示无法满足请求。</p><p>除非更具体的错误状态适用于给定的请求，否则服务应该返回“400 Bad Request”。服务应该在响应消息中包含足够的详细信息，以便开发人员明确不支持请求的哪个部分。</p><p>例子：</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs http"><span class="hljs-keyword">GET</span> <span class="hljs-string">https://ailpha.com/api/v1.0/agents?$orderBy=name</span> <span class="hljs-meta">HTTP/1.1</span><br><span class="hljs-attribute">Accept</span><span class="hljs-punctuation">: </span>application/json<br></code></pre></td></tr></table></figure><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs http"><span class="hljs-meta">HTTP/1.1</span> <span class="hljs-number">400</span> Bad Request<br><span class="hljs-attribute">Content-Type</span><span class="hljs-punctuation">: </span>application/json<br>{<br> "error": {<br> "code": "ErrorUnsupportedOrderBy",<br> "message": "Ordering by name is not supported."<br> }<br>}<br></code></pre></td></tr></table></figure><h2 id="7-资源集合"><a href="#7-资源集合" class="headerlink" title="7. 资源集合"></a>7. 资源集合</h2><h3 id="7-1-资源标识"><a href="#7-1-资源标识" class="headerlink" title="7.1 资源标识"></a>7.1 资源标识</h3><p>资源集合中每个具体资源推荐使用持久标识符（主键），含义等同于6.3章节的定义。</p><h3 id="7-2-序列化"><a href="#7-2-序列化" class="headerlink" title="7.2 序列化"></a>7.2 序列化</h3><p>使用JSON作为序列化标准格式。</p><h3 id="7-3-集合的URL"><a href="#7-3-集合的URL" class="headerlink" title="7.3 集合的URL"></a>7.3 集合的URL</h3><p>URL中集合直接定义在根路径之后，或者在另一个具体资源之后。</p><p>集合元素本质是 <strong>资源（resource）</strong>，既然是资源，必须是名词，必须为复数形式。</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">GET https://{serviceRoot}/{collection}/{id}<br></code></pre></td></tr></table></figure><ul><li>{serviceRoot} – 站点URL (site URL) + 服务的根路径的组合</li><li>{collection} – 集合的名称，复数</li><li>{id} – 唯一标识属性的值，唯一标识中原则上不允许使用”/“，如果业务需要，一定要带”\“转义。</li></ul><p>例子：</p><p><strong>常规资源集合</strong></p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">GET https://ailpha.com/api/v1.0/agents<br></code></pre></td></tr></table></figure><p><strong>嵌套资源集合和属性</strong></p><p>集合项可以包含其他集合。例如，探针集合可能包含多个内存数据资源：</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">GET https://ailpha.com/api/v1.0/agents/JHF8UE6H5W34D/memorys<br></code></pre></td></tr></table></figure><h3 id="7-3-通用子资源"><a href="#7-3-通用子资源" class="headerlink" title="7.3 通用子资源"></a>7.3 通用子资源</h3><p>资源集合根据业务场景及具体数据定义，但部分子资源对多数资源来说通用的，此处定义的通用子资源如涉及，每个资源定义必须遵守。</p><h4 id="7-4-1-files（文件子资源）"><a href="#7-4-1-files（文件子资源）" class="headerlink" title="7.4.1 files（文件子资源）"></a>7.4.1 files（文件子资源）</h4><p><em>files</em>表述的是一个资源集合或具体资源的文件子资源，常用于 导入、导出 场景。</p><p>使用可选过滤参数<em>type</em>来区分不同文件类型，服务端<strong>必须</strong>定义默认type，当未指定type时使用默认文件类型。服务端也<strong>应该</strong>在只支持一种默认type的时候，可以忽略客户端发来的type参数。</p><p>导出或下载使用GET，表述意义为获取资源的文件子资源。</p><p>导入或上传使用POST，表述意义为创建资源的文件子资源。</p><p>例子：</p><p><strong>获取资源集合的文件子资源（导出资源集合）</strong></p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs http">GET https://ailpha.com/api/v1.0/agents/files<br><span class="hljs-attribute">Accept</span><span class="hljs-punctuation">: </span>application/json<br><span class="hljs-meta">HTTP/1.1</span> <span class="hljs-number">200</span> OK<br><span class="hljs-attribute">Content-Type</span><span class="hljs-punctuation">: </span>application/octet-stream<br></code></pre></td></tr></table></figure><p><strong>获取特定资源的文件子资源（导出特定资源）</strong></p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs http">GET https://ailpha.com/api/v1.0/agents/JHF8UE6H5W34D/files?type=docx<br><span class="hljs-attribute">Accept</span><span class="hljs-punctuation">: </span>application/json<br><span class="hljs-meta">HTTP/1.1</span> <span class="hljs-number">200</span> OK<br><span class="hljs-attribute">Content-Type</span><span class="hljs-punctuation">: </span>application/octet-stream<br></code></pre></td></tr></table></figure><p><strong>创建资源集合的文件子资源（导入资源集合）</strong></p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">POST https://ailpha.com/api/v1.0/agents/files<br></code></pre></td></tr></table></figure><p><strong>创建特定资源的文件子资源（导入特定资源）</strong></p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">POST https://ailpha.com/api/v1.0/agents/JHF8UE6H5W34D/files?type=docx<br></code></pre></td></tr></table></figure><h4 id="7-4-2-synchronizations（同步子资源）"><a href="#7-4-2-synchronizations（同步子资源）" class="headerlink" title="7.4.2 synchronizations（同步子资源）"></a>7.4.2 synchronizations（同步子资源）</h4><p><em>synchronizations</em>表述的是从一个资源集合同步全部或部分数据到另一个资源集合。资源集合可以是内部的也可以是外部的。</p><p>一般情况下，创建同步子资源是异步的，响应码应该返回202。</p><p>例子：</p> <figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs http">POST https://ailpha.com/api/v1.0/assets/synchronizations<br><span class="hljs-attribute">Accept</span><span class="hljs-punctuation">: </span>application/json<br>{<br> "from":"https://soc.com/api/assets"<br>}<br><br><span class="hljs-meta">HTTP/1.1</span> <span class="hljs-number">202</span> Accepted<br>{<br> "task": {<br> "href": "/api/v1.0/assets/synchronizations/12345",<br> "id": "12345"<br> }<br>}<br></code></pre></td></tr></table></figure><h4 id="7-4-3-connectivities（连通性子资源）"><a href="#7-4-3-connectivities（连通性子资源）" class="headerlink" title="7.4.3 connectivities（连通性子资源）"></a>7.4.3 connectivities（连通性子资源）</h4><p><em>connectivities</em>表述的是一个资源集合或具体资源的连通性子资源，常用于检测资源（通常是设备类资源）的连通性。</p><p>连通性的检测包含几类：系统 -> 资源，资源 -> 资源，资源 -> 外部实体。</p><p>例子：</p><p><strong>创建特定资源的连通性子资源（检测特定资源的连通性 - 常用）</strong></p><ul><li>测试系统对特定资产连通性。系统 -> 资源。</li></ul><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">POST https://ailpha.com/api/v1.0/assets/JHF8UE6H5W34D/connectivities<br></code></pre></td></tr></table></figure><ul><li>测试邮件服务资源对外部邮箱的连通性。资源 -> 外部实体。</li></ul><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs http">POST https://ailpha.com/api/v1.0/email-servers/JHF8UE6H5W34D/connectivities<br>{<br>"to":"[email protected]"<br>}<br></code></pre></td></tr></table></figure><p><strong>创建资源集合的连通性子资源（检测资源集合的连通性）</strong></p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">POST https://ailpha.com/api/v1.0/assets/connectivities<br></code></pre></td></tr></table></figure><h3 id="7-5-排序"><a href="#7-5-排序" class="headerlink" title="7.5 排序"></a>7.5 排序</h3><p><em>$orderBy</em> 参数允许客户端对集合查询的结果进行排序。</p><p><em>$orderBy</em> 参数的值包含可以是多个，用逗号分隔的。 可以指定排序方式（asc 升序、desc 降序），空格分隔，默认为升序。</p><p>空值（NULL）排序必须“小于”非空值。</p><p>如果不支持排序字段，需统一返回 400 Bad Request，响应不支持的参数类型。</p><p>例子：</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">GET https://ailpha.com/api/v1.0/agents?$orderBy=name<br></code></pre></td></tr></table></figure><p>按名称正序排。</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">GET https://ailpha.com/api/v1.0/agents?$orderBy=name desc<br></code></pre></td></tr></table></figure><p>按名称倒序排。</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">GET https://ailpha.com/api/v1.0/agents?$orderBy=name desc,createTime<br></code></pre></td></tr></table></figure><p>按名称倒序排，再按创建时间排。</p><p>响应内容中，资源必须放在data内，同时<strong>必须</strong>返回*$orderBy*字段。</p><p>响应举例：</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs http">GET https://ailpha.com/api/v1.0/agents?$orderBy=name desc,createTime<br><span class="hljs-attribute">Accept</span><span class="hljs-punctuation">: </span>application/json<br><span class="hljs-meta">HTTP/1.1</span> <span class="hljs-number">200</span> OK<br><span class="hljs-attribute">Content-Type</span><span class="hljs-punctuation">: </span>application/json<br>{<br> ...,<br> "$orderBy": "name desc,createTime",<br> "data": [...],<br> "total":100<br>}<br></code></pre></td></tr></table></figure><h3 id="7-6-分页"><a href="#7-6-分页" class="headerlink" title="7.6 分页"></a>7.6 分页</h3><p>客户端可以使用 <em>$page</em>和*$size*查询参数来指定页数和每页的数量。</p><p> <em>$page</em>和 <em>$size</em>参数的值是正整数（1-N）。</p><p>如果不支持分页字段，需统一返回 400 Bad Request，响应不支持的参数类型。</p><p>响应内容中，资源必须放在data内，同时<strong>必须</strong>返回*$page<em>、</em>$size*字段，<strong>必须</strong>返回total字段。</p><p>例子：</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs http"><span class="hljs-keyword">GET</span> <span class="hljs-string">https://ailpha.com/api/v1.0/agents?$page=5&$size=10</span> <span class="hljs-meta">HTTP/1.1</span><br><span class="hljs-attribute">Accept</span><span class="hljs-punctuation">: </span>application/json<br>HTTP/1.1 200 OK<br><span class="hljs-attribute">Content-Type</span><span class="hljs-punctuation">: </span>application/json<br>{<br> ...,<br> "$page": 5,<br> "$size": 10,<br> "data": [...],<br> "total":100<br>}<br></code></pre></td></tr></table></figure><h3 id="7-7-批量操作"><a href="#7-7-批量操作" class="headerlink" title="7.7 批量操作"></a>7.7 批量操作</h3><p>对 资源的批量操作 准确的理解应该是 对资源集合的操作。以下几个例子来助于理解。</p><p>例子：</p><p>批量获取</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">GET https://ailpha.com/api/v1.0/agents<br></code></pre></td></tr></table></figure><p>批量新增</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs http">POST https://ailpha.com/api/v1.0/agents<br>{<br>"data":[]<br>}<br></code></pre></td></tr></table></figure><p>批量更新（提供改变的资源属性）</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs http">PATCH https://ailpha.com/api/v1.0/agents<br>{<br>"data":[]<br>}<br></code></pre></td></tr></table></figure><p>批量替换（提供改变后的完整资源）</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs http">PUT https://ailpha.com/api/v1.0/agents<br>{<br>"data":[]<br>}<br></code></pre></td></tr></table></figure><p>批量删除</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs http"># 根据IDs删除<br>DELETE https://ailpha.com/api/v1.0/agents/JHF8UE6H5W34D,GDSRUEJHFR87K,HJHAS45SDAOL<br># 根据条件删除<br>DELETE https://ailpha.com/api/v1.0/agents?agentType="ainta"<br></code></pre></td></tr></table></figure><h2 id="8-JSON标准化"><a href="#8-JSON标准化" class="headerlink" title="8. JSON标准化"></a>8. JSON标准化</h2><h3 id="8-1-基本类型"><a href="#8-1-基本类型" class="headerlink" title="8.1 基本类型"></a>8.1 基本类型</h3><p>必须按照<a href="https://www.rfc-editor.org/rfc/rfc8259.txt">RFC8259</a>的规则将原始值序列化为JSON。JSON文本必须使用UTF-8 <a href="https://www.rfc-editor.org/info/rfc3629">RFC3629</a> 编码。</p><p>JSON值必须是字符串、数字、布尔值、对象、数组、null，取值<strong>禁止</strong>超过这个范畴。</p><table><thead><tr><th><strong>数据类型</strong></th><th><strong>描述</strong></th><th><strong>示例</strong></th></tr></thead><tbody><tr><td><strong>String</strong></td><td>字符串的表示类似于C系列编程语言中使用的约定。<strong>字符串以双引号开头和结尾</strong>。所有Unicode字符都可以放在引号中，但必须转义的字符除外：引号、反斜线分隔符\和控制字符(U+0000到U+001 F)。</td><td>“hello” “\u4f60\u597d”</td></tr><tr><td><strong>Number</strong></td><td>数字的表示与大多数编程语言中使用的表示类似。<strong>一个数字用十进制数字以基数10表示</strong>。它包含一个整数分量，其前缀可以是可选减号，后面可以是分数部分和/或指数部分。不允许前导零。</td><td>1234 123.22</td></tr><tr><td><strong>Boolean</strong></td><td><strong>值为 true 或 false</strong> 。</td><td>true false</td></tr><tr><td><strong>Object</strong></td><td><strong>对象结构表示为围绕零或多个名称/值对(或多个成员)的一对花括号。名称是字符串。在每个名称之后加上一个冒号，将名称与值分隔开来。一个逗号将值与接下来的名称分隔开来。对象中的名称应该是唯一的</strong>。</td><td>{“a”:1, “b”:true}</td></tr><tr><td><strong>Array</strong></td><td><strong>数组结构表示为围绕零个或多个值(或多个元素)的方括号。元素用逗号分隔</strong>。<br />不要求数组中的值具有相同的类型。</td><td>[1,2,3,4]</td></tr><tr><td><strong>Null</strong></td><td>空。</td><td>null</td></tr></tbody></table><p>注意：</p><p><strong>64位整数的重要说明：</strong> JavaScript将静默截断大于“Number.MAX_SAFE_INTEGER”（2 ^ 53-1）的整数或小于“Number.MIN_SAFE_INTEGER”（-2 ^ 53 + 1）的数字。 <strong>如果数值类型值过大，必须将该值作为字符串返回</strong>。</p><h3 id="8-2-空值"><a href="#8-2-空值" class="headerlink" title="8.2 空值"></a>8.2 空值</h3><p>为了保证接口的一致性，对空值做如下约束：</p><table><thead><tr><th><strong>数据类型</strong></th><th>空值约束</th></tr></thead><tbody><tr><td><strong>String</strong></td><td>“”</td></tr><tr><td><strong>Number</strong></td><td>0</td></tr><tr><td><strong>Boolean</strong></td><td>不允许存在空值</td></tr><tr><td><strong>Object</strong></td><td>{}</td></tr><tr><td><strong>Array</strong></td><td>[]</td></tr></tbody></table><h3 id="8-3-时间类型"><a href="#8-3-时间类型" class="headerlink" title="8.3 时间类型"></a>8.3 时间类型</h3><p>时间字段在各个接口中的定义都不要显得混乱且不统一，<strong>必须</strong>按本节一致的约定执行。</p><h4 id="8-3-1-日期-时间约定"><a href="#8-3-1-日期-时间约定" class="headerlink" title="8.3.1 日期\时间约定"></a>8.3.1 日期\时间约定</h4><p>通常情况下，对于日期\时间在不同的国家、不同的业务场景下会有相当多种类的不同表示。</p><p>为保证接口的一致性，对日期\时间做如下约束：</p><ul><li>接口层面<strong>必须</strong>使用Unix时间戳传递日期\时间，<strong>必须</strong>毫秒级。例如：1611469304000。</li><li>建议后端时间存储也采用Unix时间戳存储。</li><li>前端根据用户的时区、业务场景自行处理呈现。</li></ul><h4 id="8-3-2-持续时长"><a href="#8-3-2-持续时长" class="headerlink" title="8.3.2 持续时长"></a>8.3.2 持续时长</h4><p>按照[ISO 8601] <a href="http://en.wikipedia.org/wiki/ISO_8601#Durations">wikipedia-iso8601-durations</a>进行序列化。</p><p>持续时间“由格式’P [n] Y [n] M [n] DT [n] H [n] M [n] S`表示。”</p><ul><li>P是在持续时间表示开始时放置的持续时间指示符（“period”）。</li><li>Y是年份指示符，它遵循年数值。</li><li>M是月份指示符，它遵循月数值。</li><li>W是周指示符，它遵循周数值。</li><li>D是遵循天数值的日期指示符。</li><li>T是在表示的时间分量之前的时间指示符。</li><li>H是小时指示符，它遵循小时数值。</li><li>M是遵循分钟数值的分钟指示符。</li><li>S是秒指示符，它遵循秒数的值。</li></ul><p>例如，</p><p>“P3Y6M4DT12H30M5S”表示“：”三年，六个月，四天，十二个小时，三十分钟加五秒” 的持续时间。</p><h2 id="9-版本号"><a href="#9-版本号" class="headerlink" title="9. 版本号"></a>9. 版本号</h2><p>**所有符合AiLPHA REST API准则的API都必须支持显式版本控制。 **</p><h3 id="9-1-版本定义"><a href="#9-1-版本定义" class="headerlink" title="9.1 版本定义"></a>9.1 版本定义</h3><p>使用major.minor格式进行版本定义。</p><p>版本定义在/api路径之后。</p><figure class="highlight http"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs http">https://ailpha.com/api/v1.0/agents<br></code></pre></td></tr></table></figure><h3 id="9-2-版本更新说明"><a href="#9-2-版本更新说明" class="headerlink" title="9.2 版本更新说明"></a>9.2 版本更新说明</h3><p><strong>必须</strong>保留历史版本接口，除非你能确认该接口已无任何客户端在使用。</p><p>不更新版本的情况下，允许修改接口返回，但只允许新增属性，<strong>禁止</strong>删除和修改原属性（参见5.1章节「字段忽略原则」）。</p><h2 id="10-命名准则"><a href="#10-命名准则" class="headerlink" title="10. 命名准则"></a>10. 命名准则</h2><h3 id="10-1-建议"><a href="#10-1-建议" class="headerlink" title="10.1 建议"></a>10.1 建议</h3><p>命名准则有助于开发同学避免经常查阅资料思考如何去命名字段，可以花更多时间专注研发。使用通用和标准的约定可以帮助开发同学正确猜测到公共属性名称及其含义。 应该使用详细的单词命名，并且不应该使用非通用的缩写词，通用缩写词指的是API范畴内大家公认的(比如Url)。</p><h3 id="10-2-格式"><a href="#10-2-格式" class="headerlink" title="10.2 格式"></a>10.2 格式</h3><ul><li>缩写<strong>应该</strong>遵循大小写惯例，就像它们是普通单词一样（例如Url）。</li><li>所有标识符包括 名称空间（namespaces）、实体类型（entityTypes）、实体集合（entitySets）、属性（properties）、动作（actions）、函数（functions）和枚举值（enumeration）<strong>应该</strong>使用驼峰法（lowerCamelCase）。</li><li>HTTP头属性是例外，其<strong>应该</strong>使用Capitalized-Hyphenated-Terms的标准HTTP约定命名。</li></ul><h3 id="10-3-避免命名"><a href="#10-3-避免命名" class="headerlink" title="10.3 避免命名"></a>10.3 避免命名</h3><p>某些专有词汇的命名应该避免使用，在某些API使用场景里可能会导致冲突。</p><p>接口禁止使用以下单词命名：</p><ul><li>Context</li><li>Scope</li><li>Resource</li></ul><h3 id="10-4-复合名称"><a href="#10-4-复合名称" class="headerlink" title="10.4 复合名称"></a>10.4 复合名称</h3><ul><li><strong>应该</strong>避免使用诸如’a’，’the’，’of’之类的单词，除非需要有业务必要。<ul><li>例如 不应该使用诸如 a User，theAccount，countOfBooks 之类的名称，而应该首选user，account，bookCount。</li></ul></li><li><strong>应该</strong>向属性名添加类型，如果不这样做会导致数据的表示方式不明确。<ul><li>向属性名添加类型时，<strong>必须</strong>在末尾添加类型，例如createdDateTime。</li></ul></li></ul><h3 id="10-5-标识字段"><a href="#10-5-标识字段" class="headerlink" title="10.5 标识字段"></a>10.5 标识字段</h3><ul><li><strong>必须</strong>使用字符串类型定义标识字段。</li><li><strong>可以</strong>使用简单的“id”来表示资源主键值。</li><li><strong>应该</strong>使用后缀为“Id”的关系名称来表示另一个资源的外键，例如：subscriptionId。</li><li>此字段的内容应该是引用资源的规范ID。</li></ul><h3 id="10-6-时间字段"><a href="#10-6-时间字段" class="headerlink" title="10.6 时间字段"></a>10.6 时间字段</h3><ul><li>对于同时需要日期和时间的字段，<strong>必须</strong>使用后缀“DateTime”，例如：createDateTime。</li><li>对于只需要日期信息而无需时间的字段，<strong>必须</strong>使用后缀“date”，例如：birthDate。</li><li>对于只需要时间信息而无需日期的字段，<strong>必须</strong>使用后缀“time”，例如：startTime。</li></ul><h3 id="10-7-名称属性"><a href="#10-7-名称属性" class="headerlink" title="10.7 名称属性"></a>10.7 名称属性</h3><ul><li>对于通用命名字段，<strong>必须</strong>使用后缀”Name“，例如：agentName。</li></ul><h3 id="10-8-集合和计数"><a href="#10-8-集合和计数" class="headerlink" title="10.8 集合和计数"></a>10.8 集合和计数</h3><ul><li>集合命名必须为复数名词或复数名词短语。</li><li><strong>可以</strong>使用简化英语来表示不常用的复数名词。<ul><li>例如<strong>可以</strong>用schemas来替代schemata。</li></ul></li><li><strong>必须</strong>使用后缀为“Count”的名词或名词短语来命名资源的计数。</li></ul><h3 id="10-9-常用字段名称"><a href="#10-9-常用字段名称" class="headerlink" title="10.9 常用字段名称"></a>10.9 常用字段名称</h3><p>如果服务有属性的含义跟下面的命名匹配，则服务必须使用此表中的命名。此表将随着服务添加更常用的术语而更新。所有有相关建议的同学都<strong>可以</strong>一起维护这个表格。</p><table><thead><tr><th></th></tr></thead><tbody><tr><td>ailpha</td></tr><tr><td>createdDateTime</td></tr><tr><td>contentUrl</td></tr><tr><td>country</td></tr><tr><td>createdBy</td></tr><tr><td>displayName</td></tr><tr><td>errorUrl</td></tr><tr><td>event</td></tr><tr><td>givenName</td></tr><tr><td>jobTitle</td></tr><tr><td>id</td></tr><tr><td>lastModifiedDateTime</td></tr><tr><td>location</td></tr><tr><td>memberOf</td></tr><tr><td>message</td></tr><tr><td>name</td></tr><tr><td>owner</td></tr><tr><td>people</td></tr><tr><td>person</td></tr><tr><td>postalCode</td></tr><tr><td>photo</td></tr><tr><td>properties</td></tr><tr><td>signInName</td></tr><tr><td>tags</td></tr><tr><td>userPrincipalName</td></tr><tr><td>updatedDateTime</td></tr><tr><td>webUrl</td></tr></tbody></table>]]></content>
<summary type="html"><h2 id="1-摘要"><a href="#1-摘要" class="headerlink" title="1. 摘要"></a>1. 摘要</h2><p>AiLPHA REST API指南作为一种设计规范，鼓励开发同学通过RESTful HTTP接口访问资源。</p>
<p>本文档建立了AiLPHA REST API应该遵循的指导原则，以便开发统一一致的RESTful接口。</p></summary>
<category term="编程" scheme="https://idelin.github.io/categories/%E7%BC%96%E7%A8%8B/"/>
<category term="接口" scheme="https://idelin.github.io/categories/%E7%BC%96%E7%A8%8B/%E6%8E%A5%E5%8F%A3/"/>
<category term="restapi" scheme="https://idelin.github.io/tags/restapi/"/>
<category term="接口规范" scheme="https://idelin.github.io/tags/%E6%8E%A5%E5%8F%A3%E8%A7%84%E8%8C%83/"/>
<category term="代码" scheme="https://idelin.github.io/tags/%E4%BB%A3%E7%A0%81/"/>
</entry>
<entry>
<title>202110-记录</title>
<link href="https://idelin.github.io/2021/10/08/202110-%E8%AE%B0%E5%BD%95/"/>
<id>https://idelin.github.io/2021/10/08/202110-%E8%AE%B0%E5%BD%95/</id>
<published>2021-10-08T10:16:00.000Z</published>
<updated>2022-08-03T12:45:23.383Z</updated>
<content type="html"><![CDATA[<p>说实话我也没想到今年的十一那么的不一样。</p><span id="more"></span><p>第一次的星空露营。假期的第一天跟朋友一起计划了一次不错的露营活动⛺️，到达地点支起帐篷，当夜晚来临，我们一行四人躺在帐篷里头向外仰望着满天繁星，不禁欢呼。<img src="https://tva1.sinaimg.cn/large/e6c9d24ely1h4lm9bsxs5j20n40hcq39.jpg" alt="image-20220727173254552"></p><p>第一次的环山公里自驾。四明山确实是个风景不错的地方，无需特意找景点，把车随意在某个临停出或公共停车位上停下都能看到不一样的景致，当然还有个原因是标注景点我们也没找到地方。</p><p><img src="https://tva1.sinaimg.cn/large/e6c9d24ely1h4lm9k33doj20n40hctc0.jpg" alt="image-20220727173310816"></p><p>第一次的雪窦山爬行及弥勒大佛道场礼佛。四明山下一站是溪口，不过没预料到是，溪口风景区特别大…特别大，留给我们时间并不多，所以我们挑了其中的雪窦山景区。其中蒋氏故居什么倒是没太多感觉，就是觉得国庆去参观常公故居总感觉怪怪的。给我最大印象的是弥勒道场的弥勒大佛，真的非常地……大。上香拜佛，祈福许愿。</p><p><img src="https://tva1.sinaimg.cn/large/e6c9d24ely1h4lm9qpvi6j20n40hcdib.jpg" alt="image-20220727173321444"></p><p>第一次的嵊州打卡之旅。嵊州是一个中转点，没有找景点，只是为美食而来。不得不说豆腐小笼简直了，中毒了，下次去可能会为了这口直接去嵊州。</p><p><img src="https://tva1.sinaimg.cn/large/e6c9d24ely1h4lm9vmpiwj20n40hc77a.jpg" alt="image-20220727173329977"></p><p>以及第一次……的高速车祸。本来计划中的下一站是温岭的石塘镇，第三天下午从嵊州出发上高速，在台金高速临海段的地方遭遇车祸。事情经过是这样，我当时在高速快车道以120自适应巡航在行驶，突然有辆比亚迪suv变道到我前面，不到十秒后，我只看到他紧急刹车，我随即也踩死刹车，但根本刹不住，直接就撞上了。后来才明白这是一个三车追尾的事故，比亚迪suv前面还有一辆大众轿车，大众车先发现前面高速堵车，他刹住了。但变道过来的比亚迪没刹住把大众撞到前面去了车撞停了，我随即追尾了suv。现在细想，还是很后怕，当时快撞上的时候还是很清醒，又快速撇了下右后视镜，看够不够能右变道避过去，但看到一辆半挂车当时就根本不敢打方向，副驾我老婆还在睡着。首先最最重要的是，人都没事，除了惊吓。车损方面，大众最轻，但屁股和后车牌也都凹了进去，suv则前后都有点凹进去，但最严重的是我的车，因为有点撞到suv尾部侧面，整个右前方掀起来一大部分。索性索性的是，虽然车看起来撞得很严重，但车内包括我和我老婆都感觉没有很大冲击力。在接下来的事情我都是懵了，包括后面拖车拖到临近高速口，到后面交警事故认责，全程都懵。以至于接下来的一两天里，都感觉有ptsd的症状。</p><p><img src="https://tva1.sinaimg.cn/large/e6c9d24ely1h4lma0r3khj20n40hcadl.jpg" alt="image-20220727173338351"></p><p>车祸发生后的详情。当我发生车祸后，第一反应是快速检查人有没事情，然后后备箱打开三脚架放车近百米后面，再就是被救援人员将车推到应急车道，接着按救援人员提示拨打12122报警，随即拖车来了我们跟着车一起上了拖车。之后我联系了保险公司保险人员，我简单说了情况，他让我不要着急他那边联系杭州的4s店派拖车到临近高速口。之后事故认定，3个小时候拖车和4s店的人来了，拖车拖车，4s店的车拖我们和我们的一堆旅行准备的东西。这本来是一件悲伤得不能再悲伤的事情，可里面最喜剧的部分是我没想到的，我在联系完保险人员的时候，手机开始不停地响，有 杭州那边其他区的4s店、临海的4s店、台州的4s店、温岭的4s店、好几个当地修车厂接连联系我，还主动说给返点，我当时都怒了，合着我出事故他们争抢生意，后面想想真是欲哭无泪。</p><p><img src="https://tva1.sinaimg.cn/large/e6c9d24ely1h4lma5ysbnj20n40hc76r.jpg" alt="image-20220727173346042"></p><p>总之总之，在没有超速没有疲劳驾驶的情况下都出了如此惨痛的教训，我以后一定会更加谨慎，大家也引以为戒，高速行车千万要小心再小心，生命健康比什么都重要。</p><p>接下来的几天，还有过半的假期时间没了车，我反而有不一样的体会。参加了大学室友的婚礼，坐高铁的。去了啦喜街打卡，坐地铁的。逛了周边公园感受生活气息，用双脚的。我感觉节奏慢了下了，我也感受到一些不一样的人和事，和好友畅饮和陌生人擦肩。人生总是这样，得失无常，凡是路过，都是风景。</p>]]></content>
<summary type="html"><p>说实话我也没想到今年的十一那么的不一样。</p></summary>
<category term="生活" scheme="https://idelin.github.io/categories/%E7%94%9F%E6%B4%BB/"/>
<category term="记录" scheme="https://idelin.github.io/tags/%E8%AE%B0%E5%BD%95/"/>
</entry>
<entry>
<title>kafka的相关操作脚本</title>
<link href="https://idelin.github.io/2019/12/21/kafka%E7%9A%84%E7%9B%B8%E5%85%B3%E6%93%8D%E4%BD%9C%E8%84%9A%E6%9C%AC/"/>
<id>https://idelin.github.io/2019/12/21/kafka%E7%9A%84%E7%9B%B8%E5%85%B3%E6%93%8D%E4%BD%9C%E8%84%9A%E6%9C%AC/</id>
<published>2019-12-21T08:14:54.000Z</published>
<updated>2022-08-05T08:16:29.499Z</updated>
<content type="html"><![CDATA[<p>总结最近用到的kafka相关命令和脚本。</p><span id="more"></span><p>1、创建Topic<br>./kafka-topics.sh –zookeeper cc13-141:2182 –topic mytopic –replication-factor 1 –partitions 1 –create</p><p>2、Topic列表<br>./kafka-topics.sh –zookeeper cc13-138:2181 –list</p><p>3、创建生产者<br>./kafka-console-producer.sh –broker-list cc13-141:9092 –topic mytopic</p><p>4、创建消费者<br>./kafka-console-consumer.sh –zookeeper c13-138:2181 –topic mytopic<br>–from-beginning 每次都是从开始位置消费，在生产环境下不建议这样使用。</p><p>5、查看指定topic<br>./kafka-topics.sh –describe –zookeeper c13-138:2181 –topic mytopic</p><p>6、删除topic<br>./kafka-topics.sh –delete –zookeeper c13-138:2181–topic mytopic</p><p>7、显示出Consumer的Group、Topic、分区ID、分区对应已经消费的Offset、logSize大小，Lag以及Owner等信息。<br>使用脚本：kafka-consumer-offset-checker.sh<br>./kafka-consumer-offset-checker.sh –zookeeper c13-138:2181,c13-139:2181,c13-141:2181 –topic mytopic –group xb_id –broker-info</p><p>8、有时候我们需要验证日志索引是否正确，或者仅仅想从log文件中直接打印消息。<br>使用脚本：kafka-run-class.sh<br>./kafka-run-class.sh kafka.tools.DumpLogSegments<br>./kafka-run-class.sh kafka.tools.DumpLogSegments /nodedata/kafka/kafka-logs/xb_topic-0/00000000000000000033.log<br>./kafka-run-class.sh kafka.tools.DumpLogSegments –files /nodedata/kafka/kafka-logs/xb_topic-0/00000000000000000033.log –print-data-log</p><p>9、导出Zookeeper中Group相关的偏移量。有时候我们需要导出某个Consumer group各个分区的偏移量。<br>使用脚本：kafka-run-class.sh<br>./kafka-run-class.sh kafka.tools.ExportZkOffsets<br>./kafka-run-class.sh kafka.tools.ExportZkOffsets –group xb_id –zkconnect c13-138:2181,c13-139:2181,c13-141:2181 –output-file ~/offset<br>vim ~/offset</p><p>10、这个工具主要作用是从一个Kafka集群里面读取指定Topic的消息，并将这些消息发送到其他集群的指定topic中。<br>使用脚本：./kafka-replay-log-producer.sh</p><p>11、kafka-simple-consumer-shell.sh工具主要是使用Simple Consumer API从指定Topic的分区读取数据并打印在终端。<br>使用脚本：./kafka-simple-consumer-shell.sh –broker-list c13-141:9092 –topic mytopic –partition 0</p><p>12、kafka.tools.UpdateOffsetsInZK工具可以更新Zookeeper中指定Topic所有分区的偏移量，可以指定成 earliest或者latest<br>使用工具：./kafka-run-class.sh kafka.tools.UpdateOffsetsInZK</p><p>最后再注意kafka的启动和停止。<br>启动kafka： ./kafka-server-start.sh /kafka/config/server.properties >/dev/null 2>&1 &<br>停止kafka： 直接kill掉进程就行。</p>]]></content>
<summary type="html"><p>总结最近用到的kafka相关命令和脚本。</p></summary>
<category term="kafka" scheme="https://idelin.github.io/categories/kafka/"/>
<category term="kafka" scheme="https://idelin.github.io/tags/kafka/"/>
</entry>
<entry>
<title>渗透学习：SQL盲注</title>
<link href="https://idelin.github.io/2019/03/18/%E6%B8%97%E9%80%8F%E5%AD%A6%E4%B9%A0%EF%BC%9ASQL%E7%9B%B2%E6%B3%A8/"/>
<id>https://idelin.github.io/2019/03/18/%E6%B8%97%E9%80%8F%E5%AD%A6%E4%B9%A0%EF%BC%9ASQL%E7%9B%B2%E6%B3%A8/</id>
<published>2019-03-18T08:35:54.000Z</published>
<updated>2022-08-03T12:31:43.421Z</updated>
<content type="html"><![CDATA[<h2 id="SQL-盲注"><a href="#SQL-盲注" class="headerlink" title="SQL 盲注"></a>SQL 盲注</h2><p>可能会查看、修改或删除数据库条目和表</p><h3 id="可能原因"><a href="#可能原因" class="headerlink" title="可能原因"></a>可能原因</h3><p>未对用户输入正确执行危险字符清理</p><span id="more"></span><h3 id="技术描述"><a href="#技术描述" class="headerlink" title="技术描述"></a>技术描述</h3><p>该软件使用受外部影响的输入来构造 SQL 命令的全部或一部分，但是它未能对可能在 SQL 命令发送到数据库时修改该命令的元素进行无害化处理。如果在用户可控制的输入中没有对 SQL 语法充分地除去或加上引号，那么生成的 SQL 查询可能会导致将这些输入解释为 SQL 而不是普通用户数据。这可用于修改查询逻辑以绕过安全性检查，或者插入其他用于修改后端数据库的语句，可能包括执行系统命令。<br>例如，假设有一个带有登录表单的 HTML 页面，该页面最终使用用户输入对数据库运行以下 SQL 查询： <code>SELECT * FROM accounts WHERE username='$user' AND password='$pass'</code></p><p>这两个变量（$user 和 $pass）包含了用户在登录表单中输入的用户凭证。如果用户输入“jsmith”作为用户名，并输入“Demo1234”作为密码，那么 SQL 查询将如下所示：<br> <code>SELECT * FROM accounts WHERE username='jsmith' AND password='Demo1234'</code></p><p>但如果用户输入“’”（单引号）作为用户名，输入“’”（单引号）作为密码，那么 SQL 查询将如下所示：<br> <code>SELECT * FROM accounts WHERE username=''' AND password='''</code></p><p>当然，这是格式错误的 SQL 查询，并将调用错误消息，而 HTTP 响应中可能会返回此错误消息。通过此类错误，攻击者会知道 SQL 注入已成功，这样攻击者就会尝试进一步的攻击媒介。SQL 盲注类似于 SQL 注入。不同之处在于，要利用该攻击，攻击者无需寻找响应中的 SQL 错误。因此，AppScan 用于识别该攻击的方法也不同。AppScan 会查找易受 SQL 注入（通过多个请求来操纵应用程序的逻辑，而不是尝试调用 SQL 错误）影响的脚本。</p><p>该技巧需要发送特定请求，其中易受攻击的参数（嵌入在 SQL 查询中的参数）进行了相应修改，以便响应中会指示是否在 SQL 查询上下文中使用数据。该修改涉及将 AND 布尔表达式与原始字符串一起使用，使其一时求值为 True，一时求值为 False。在一种情况下，净结果应该与原始结果相同（登录成功），而在另一种情况下，结果应该完全不同（登录失败）。在某些少见的情况下，求值为 True 的 OR 表达式也可能很有用。如果原始数据是数字，可以使用更简单的花招。假设原始数据为 123。此数据可以在一个请求中替换为 0+123，而在另一个请求中替换为 456+123。第一个请求的结果应该与原始结果相同，第二个请求的结果应该不同（因为得出的数字是 579）。在某些情况中，我们仍需要上面所说明的攻击版本（使用 AND 和 OR），但并不转义字符串上下文。</p><p>SQL 盲注背后的概念是，即使不直接从数据库接收数据（以错误消息或泄漏的信息的形式），也可能从数据库中抽取数据（每次一个比特），或以恶意方式修改查询。其原理在于，应用程序的行为（返回与原始响应相同或不同的响应）可以提供有关所求值的（已修改）查询的单比特信息，也就是说，攻击者有可能设计出一个 SQL 布尔表达式，其求值（单比特）通过应用程序行为（与原始行为相同/不同）来造成破坏。</p><h3 id="引用和相关链接"><a href="#引用和相关链接" class="headerlink" title="引用和相关链接"></a>引用和相关链接</h3><blockquote><p>“Web Application Disassembly with ODBC Error Messages”（作者：David Litchfield）<br> “Using Binary Search with SQL Injection”（作者：Sverre H. Huseby）<br> Blind SQL Injection Training Module </p></blockquote>]]></content>
<summary type="html"><h2 id="SQL-盲注"><a href="#SQL-盲注" class="headerlink" title="SQL 盲注"></a>SQL 盲注</h2><p>可能会查看、修改或删除数据库条目和表</p>
<h3 id="可能原因"><a href="#可能原因" class="headerlink" title="可能原因"></a>可能原因</h3><p>未对用户输入正确执行危险字符清理</p></summary>
<category term="网络安全 渗透" scheme="https://idelin.github.io/categories/%E7%BD%91%E7%BB%9C%E5%AE%89%E5%85%A8-%E6%B8%97%E9%80%8F/"/>
</entry>
<entry>
<title>jekins出现无法打包</title>
<link href="https://idelin.github.io/2018/06/13/jekins%E5%87%BA%E7%8E%B0%E6%97%A0%E6%B3%95%E6%89%93%E5%8C%85/"/>
<id>https://idelin.github.io/2018/06/13/jekins%E5%87%BA%E7%8E%B0%E6%97%A0%E6%B3%95%E6%89%93%E5%8C%85/</id>
<published>2018-06-13T07:33:42.000Z</published>
<updated>2022-08-03T12:45:23.656Z</updated>
<content type="html"><![CDATA[<p>jekins出现无法打包<br>出现错误：</p><figure class="highlight livecodeserver"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs livecodeserver"><span class="hljs-keyword">stderr</span>: error: Your <span class="hljs-built_in">local</span> changes <span class="hljs-built_in">to</span> <span class="hljs-keyword">the</span> following <span class="hljs-built_in">files</span> would be overwritten <span class="hljs-keyword">by</span> checkout:<br>src/routes.js<br></code></pre></td></tr></table></figure><span id="more"></span><p>进入对应git目录丢弃文件即可：<br><code>git checkout . && git clean -xdf</code></p>]]></content>
<summary type="html"><p>jekins出现无法打包<br>出现错误：</p>
<figure class="highlight livecodeserver"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs livecodeserver"><span class="hljs-keyword">stderr</span>: error: Your <span class="hljs-built_in">local</span> changes <span class="hljs-built_in">to</span> <span class="hljs-keyword">the</span> following <span class="hljs-built_in">files</span> would be overwritten <span class="hljs-keyword">by</span> checkout:<br>src/routes.js<br></code></pre></td></tr></table></figure></summary>
</entry>
<entry>
<title>Storm单机安装</title>
<link href="https://idelin.github.io/2017/11/11/Storm%E5%8D%95%E6%9C%BA%E5%AE%89%E8%A3%85/"/>
<id>https://idelin.github.io/2017/11/11/Storm%E5%8D%95%E6%9C%BA%E5%AE%89%E8%A3%85/</id>
<published>2017-11-11T13:38:59.000Z</published>
<updated>2022-08-03T12:45:23.676Z</updated>
<content type="html"><![CDATA[<h2 id="一、下载安装包"><a href="#一、下载安装包" class="headerlink" title="一、下载安装包"></a>一、下载安装包</h2><p><a href="http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz">http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz</a></p><span id="more"></span><p>##二、修改配置文件<br>vim storm.yaml</p><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs stylus">#<span class="hljs-selector-id">#zookeeper</span><br>storm<span class="hljs-selector-class">.zookeeper</span><span class="hljs-selector-class">.servers</span>:<br> - <span class="hljs-string">"localhost"</span><br>##nimbus所在节点<br>nimbus<span class="hljs-selector-class">.seeds</span>: <span class="hljs-selector-attr">[<span class="hljs-string">"localhost"</span>]</span><br>##可用端口号配置，每个对应一个worker，可配置多个<br>supervisor<span class="hljs-selector-class">.slots</span><span class="hljs-selector-class">.ports</span>:<br> - <span class="hljs-number">6700</span><br></code></pre></td></tr></table></figure><p>##三、启动组件<br>###1.启动zookeeper<br>Storm中使用Zookeeper主要用于Storm集群各节点的分布式协调工作，所以最先启动zookeeper，当然zookeeper也是需要下载，自行下载启动>><a href="http://zookeeper.apache.org/">http://zookeeper.apache.org/</a></p><figure class="highlight clean"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs clean">##先启动zookeeper<br>zkServer.sh start<br></code></pre></td></tr></table></figure><p>###2.启动nimbus<br>主控节点运行Nimbus守护进程，类似于Hadoop中的jobtracker，负责在集群中分发代码，对节点分配任务，并监视主机故障。所以再启动nimbus。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs awk"><span class="hljs-comment">##启动nimbus</span><br>.<span class="hljs-regexp">/bin/</span>storm nimbus &<br></code></pre></td></tr></table></figure><p>###3.启动Storm UI</p><p>storm UI是一个监控storm运行状态的网页，bin/storm ui，访问地址: http://{nimbus.host}:ui.port/。默认端口8080。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">.<span class="hljs-regexp">/bin/</span>storm ui &<br></code></pre></td></tr></table></figure><p>###4.Supervisor<br>Supervisor负责监听工作节点上已经分配的主机作业，启动和停止Nimbus已经分配的工作进程。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs awk"><span class="hljs-comment">##启动supervisor</span><br>.<span class="hljs-regexp">/bin/</span>storm supervisor &<br></code></pre></td></tr></table></figure><p>##四、测试使用<br>###1.提交storm任务</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">.<span class="hljs-regexp">/bin/</span>storm jar <span class="hljs-regexp">/Users/</span>delin<span class="hljs-regexp">/stromdemo/</span>target/strom-demo-<span class="hljs-number">1.0</span>-SNAPSHOT.jar com.ailpha.SimpleTopology wordcount<br></code></pre></td></tr></table></figure><p>###2.杀死任务<br>格式：storm kill name -w seconds</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">storm</span> kill wordcount -w <span class="hljs-number">10</span><br></code></pre></td></tr></table></figure><p>###3.停用任务命令<br>格式：storm deactivte name</p><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">storm deactivte wordcount</span><br></code></pre></td></tr></table></figure><p>###4.启用任务命令<br>格式：storm activate name</p><figure class="highlight applescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs applescript">storm <span class="hljs-built_in">activate</span> wordcount<br></code></pre></td></tr></table></figure><p>###5.平衡任务命令<br>格式：storm rebalance name</p><figure class="highlight ebnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs ebnf"><span class="hljs-attribute">storm rebalance wordcount</span><br></code></pre></td></tr></table></figure>]]></content>
<summary type="html"><h2 id="一、下载安装包"><a href="#一、下载安装包" class="headerlink" title="一、下载安装包"></a>一、下载安装包</h2><p><a href="http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz">http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz</a></p></summary>
<category term="storm" scheme="https://idelin.github.io/categories/storm/"/>
<category term="storm" scheme="https://idelin.github.io/tags/storm/"/>
</entry>
<entry>
<title>sqoop使用</title>
<link href="https://idelin.github.io/2017/10/28/sqoop%E4%BD%BF%E7%94%A8/"/>
<id>https://idelin.github.io/2017/10/28/sqoop%E4%BD%BF%E7%94%A8/</id>
<published>2017-10-28T13:54:33.000Z</published>
<updated>2022-08-03T12:45:23.675Z</updated>
<content type="html"><![CDATA[<ol><li>list-databases 列出数据库<br>sqoop list-databases –connect jdbc:oracle:thin:@172.21.202.4:1521:TJZHCSDEV –username tjzhcs –password tjzhcs</li></ol><span id="more"></span><ol start="2"><li><p>list-tables 列出某数据库下的表<br>sqoop list-tables–connect jdbc:oracle:thin:@172.21.202.4:1521:TJZHCSDEV –username tjzhcs –password tjzhcs</p></li><li><p>eval 可以快速地使用SQL语句对关系数据库进行操作，这可以使得在使用import这种工具进行数据导入的时候，可以预先了解相关的SQL语句是否正确，并能将结果显示在控制台<br>sqoop eval –connect jdbc:oracle:thin:@172.21.202.4:1521:TJZHCSDEV –username tjzhcs –password tjzhcs -e “select * from b_code”</p></li><li><p>create-hive-table 生成与关系数据库表的表结构对应的HIVE表<br>sqoop create-hive-table –connect jdbc:oracle:thin:@172.21.202.4:1521:TJZHCSDEV –username tjzhcs –password tjzhcs -table B_CODE -hive-table h_b_code(注：oracle导入的时候表名需要使用大写！！！)</p></li><li><p>导入oracle表到hive中<br>sqoop import –connect jdbc:oracle:thin:@172.21.202.4:1521:TJZHCSDEV –username tjzhcs –password tjzhcs –table B_CODE –hive-table h_b_code –hive-import<br>Hive arguments: <code> ArgumentDescription --hive-home <dir>Override $HIVE_HOME --hive-importImport tables into Hive (Uses Hive’s default delimiters if none are set.) --hive-overwriteOverwrite existing data in the Hive table. --create-hive-tableIf set, then the job will fail if the target hive table exits. By default this property is false. --hive-table <table-name>Sets the table name to use when importing to Hive. --hive-drop-import-delimsDrops \n, \r, and \01 from string fields when importing to Hive. --hive-delims-replacementReplace \n, \r, and \01 from string fields with user defined string when importing to Hive. --hive-partition-keyName of a hive field to partition are sharded on --hive-partition-value <v>String-value that serves as partition key for this imported into hive in this job. --map-column-hive <map>Override default mapping from SQL type to Hive type for configured columns.</code></p></li><li><p>sqoop list-databases –connect jdbc:mysql://172.21.80.123:3306/ –username root –password 1234</p><figure class="highlight sas"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs sas">sqoop import --connect jdbc:mysql://172.21.80.123:3306/hadoop --username root --password 1234 --<span class="hljs-keyword">table</span> lsemp --hbase-<span class="hljs-keyword">table</span> lsemp --hbase-<span class="hljs-keyword">create</span>-<span class="hljs-keyword">table</span> --hbase-row-<span class="hljs-keyword">key</span> empno --column-family info<br>HBase arguments:```<br><br>ArgumentDescription<br>--column-family <family>Sets the target column family for the import<br>--hbase-<span class="hljs-keyword">create</span>-<span class="hljs-keyword">table</span><span class="hljs-keyword">If</span> specified, <span class="hljs-keyword">create</span> <span class="hljs-keyword">missing</span> HBase tables<br>--hbase-row-<span class="hljs-keyword">key</span> <col>Specifies which <span class="hljs-keyword">input</span> column to use <span class="hljs-keyword">as</span> the row <span class="hljs-keyword">key</span><br><span class="hljs-keyword">In</span> case, <span class="hljs-keyword">if</span> <span class="hljs-keyword">input</span> <span class="hljs-keyword">table</span> contains composite<br><span class="hljs-keyword">key</span>, <span class="hljs-keyword">then</span> <col> must be <span class="hljs-keyword">in</span> the form of a<br>comma-separated <span class="hljs-keyword">list</span> of composite <span class="hljs-keyword">key</span><br>attributes<br>--hbase-<span class="hljs-keyword">table</span> <<span class="hljs-keyword">table</span>-name>Specifies an HBase <span class="hljs-keyword">table</span> to use <span class="hljs-keyword">as</span> the target instead of HDFS<br>--hbase-bulkloadEnables bulk loading<br></code></pre></td></tr></table></figure></li></ol><figure class="highlight dsconfig"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs dsconfig"><span class="hljs-string">mysql</span>相关<br><span class="hljs-string">1</span>.执行<span class="hljs-string">sql</span><br><span class="hljs-string">sqoop</span> <span class="hljs-string">eval</span> <span class="hljs-built_in">--connect</span> <span class="hljs-string">jdbc:mysql:</span>//<span class="hljs-string">localhost:3306/</span><span class="hljs-string">hive</span> <span class="hljs-built_in">--username</span> <span class="hljs-string">root</span> <span class="hljs-built_in">--password</span> <span class="hljs-string">123456</span> -<span class="hljs-string">e</span> <span class="hljs-string">'show tables'</span><br><span class="hljs-string">2</span>.列出库<br><span class="hljs-string">sqoop</span> <span class="hljs-built_in">list-databases</span> <span class="hljs-built_in">--connect</span> <span class="hljs-string">jdbc:mysql:</span>//<span class="hljs-string">localhost:3306/</span> <span class="hljs-built_in">--username</span> <span class="hljs-string">root</span> <span class="hljs-built_in">--password</span> <span class="hljs-string">123456</span><br><span class="hljs-string">3</span>.导入<span class="hljs-string">mysql</span>数据到<span class="hljs-string">hive</span><br><span class="hljs-string">sqoop</span> <span class="hljs-string">import</span> <span class="hljs-built_in">--connect</span> <span class="hljs-string">jdbc:mysql:</span>//<span class="hljs-string">localhost:3306/</span><span class="hljs-string">test</span> <span class="hljs-built_in">--username</span> <span class="hljs-string">root</span> <span class="hljs-built_in">--password</span> <span class="hljs-string">123456</span> <span class="hljs-built_in">--table</span> <span class="hljs-string">student</span> <span class="hljs-built_in">--hive-table</span> <span class="hljs-string">student</span> <span class="hljs-built_in">--hive-import</span> <span class="hljs-built_in">--hive-database</span> <span class="hljs-string">test</span><br><span class="hljs-string">4</span>.导出<span class="hljs-string">hive</span>数据到<span class="hljs-string">mysql</span><br><span class="hljs-string">sqoop</span> <span class="hljs-string">export</span> <span class="hljs-built_in">--connect</span> <span class="hljs-string">jdbc:mysql:</span>//<span class="hljs-string">localhost:3306/</span><span class="hljs-string">test</span> <span class="hljs-built_in">--username</span> <span class="hljs-string">root</span> <span class="hljs-built_in">--password</span> <span class="hljs-string">123456</span> <span class="hljs-built_in">--table</span> <span class="hljs-string">word_count</span> -<span class="hljs-string">export-dir</span> /<span class="hljs-string">user</span>/<span class="hljs-string">hive</span>/<span class="hljs-string">warehouse</span>/<span class="hljs-string">test</span>.<span class="hljs-string">db</span>/<span class="hljs-string">word_count</span> <span class="hljs-built_in">--input-fields-terminated-by</span> <span class="hljs-string">'\001'</span><br></code></pre></td></tr></table></figure>]]></content>
<summary type="html"><ol>
<li>list-databases 列出数据库<br>sqoop list-databases –connect jdbc:oracle:thin:@172.21.202.4:1521:TJZHCSDEV –username tjzhcs –password tjzhcs</li>
</ol></summary>
<category term="sqoop" scheme="https://idelin.github.io/categories/sqoop/"/>
</entry>
<entry>
<title>hive2.1.1安装部署</title>
<link href="https://idelin.github.io/2017/10/25/hive2.1.1%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/"/>
<id>https://idelin.github.io/2017/10/25/hive2.1.1%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/</id>
<published>2017-10-25T14:34:00.000Z</published>
<updated>2022-08-03T12:45:23.676Z</updated>
<content type="html"><![CDATA[<h2 id="二、Hive-运行模式" style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:-25px 0px 0.83em; font-size:2.15em; padding:40px 0px 0px"><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">转：</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">http://blog.csdn.net/reesun/article/details/8556078<br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">http://blog.csdn.net/zhongguozhichuang/article/details/52702476</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">http://blog.csdn.net/huoyuyan/article/details/50769104</p></h2><h2 id="二、Hive-运行模式" style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:-25px 0px 0.83em; font-size:2.15em; padding:40px 0px 0px">一、Hive 运行模式</h2><span id="more"></span><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">与 <a target="_blank" target="_blank" href="http://lib.csdn.net/base/hadoop" class="replace_word" title="Hadoop知识库" style="background:transparent; color:rgb(223,52,52); margin:0px; padding:0px; font-weight:bold; outline:none">Hadoop</a> 类似，<a target="_blank" target="_blank" href="http://lib.csdn.net/base/hive" class="replace_word" title="Hive知识库" style="background:transparent; color:rgb(223,52,52); margin:0px; padding:0px; font-weight:bold; outline:none">Hive</a> 也有 3 种运行模式：</p><blockquote style="padding:10px 20px; margin:0px 0px 20px; border-left:5px solid rgb(238,238,238); color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px; font-size:17.5px; line-height:1.25">1. 内嵌模式</p><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px; font-size:17.5px; line-height:1.25">将元数据保存在本地内嵌的 Derby <a target="_blank" target="_blank" href="http://lib.csdn.net/base/mysql" class="replace_word" title="MySQL知识库" style="background:transparent; color:rgb(223,52,52); margin:0px; padding:0px; font-weight:bold; outline:none">数据库</a>中，这是使用 Hive 最简单的方式。但是这种方式缺点也比较明显，因为一个内嵌的 Derby 数据库每次只能访问一个数据文件，这也就意味着它不支持多会话连接。</p><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px; font-size:17.5px; line-height:1.25">2. 本地模式</p><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px; font-size:17.5px; line-height:1.25">这种模式是将元数据保存在本地独立的数据库中（一般是 <a target="_blank" target="_blank" href="http://lib.csdn.net/base/mysql" class="replace_word" title="MySQL知识库" style="background:transparent; color:rgb(223,52,52); margin:0px; padding:0px; font-weight:bold; outline:none">MySQL</a>），这用就可以支持多会话和多用户连接了。</p><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px; font-size:17.5px; line-height:1.25">3. 远程模式</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; font-size:17.5px; line-height:1.25">此模式应用于 Hive 客户端较多的情况。把 MySQL 数据库独立出来，将元数据保存在远端独立的 MySQL 服务中，避免了在每个客户端都安装 MySQL 服务从而造成冗余浪费的情况。</p></blockquote><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">二、下载安装 Hive</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">http://hive.apache.org/downloads.html<br style=""></p><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">tar -xzvf apache-hive-2.1.1-bin.tar.gz ##解压<br style=""></p><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><a target="_blank" target="_blank" name="t2" style="background:transparent; color:rgb(79,161,219); margin:0px; padding:0px; outline:none"></a><span style="margin:0px; padding:0px; font-size:18px">三、配置系统环境变量</span></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px">修改 /etc/profile 文件 vi /etc/profile 来修改（root用户操作）：<br style=""></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:857px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">设置 Hive环境变量 </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"># Hive environment </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">export <span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">HIVE_HOME</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=/home/hadoop/cloud/apache-hive-2.1.1-bin </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">export <span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">PATH</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=$HIVE_HOME/bin:$HIVE_HOME/conf:$PATH </span></span></li></ol></div></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:978px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">使环境变量生效: </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">source /etc/profile </span></li></ol></div><br style=""><br style=""></h6><h2 id="五、内嵌模式" style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:-25px 0px 0.83em; font-size:2.15em; padding:40px 0px 0px"><a target="_blank" target="_blank" name="t1" style="background:transparent; color:rgb(79,161,219); margin:0px; padding:0px; outline:none"></a>四、内嵌模式</h2><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">（1）修改 Hive 配置文件</p><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">$HIVE_HOME/conf 对应的是 Hive 的配置文件路径，类似于之前学习的<a target="_blank" target="_blank" href="http://lib.csdn.net/base/hbase" class="replace_word" title="Hbase知识库" style="background:transparent; color:rgb(223,52,52); margin:0px; padding:0px; font-weight:bold; outline:none">Hbase</a>, 该路径下的 hive-site.xml 是 Hive 工程的配置文件。默认情况下，该文件并不存在，我们需要拷贝它的模版来实现：</p><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; color:rgb(69,69,69); margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:1281px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">cp hive-default.xml.template hive-site.xml </span></span></li></ol></div><br style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">hive-site.xml 的主要配置有：</span><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px"><a target="_blank" target="_blank" href="https://dn-anything-about-doc.qbox.me/userid46108labid766time1427423801808" title="图片描述信息" class="fancybox" rel="gallery0" style="background:transparent; color:rgb(202,12,22); margin:0px; padding:0px; outline:none"><img src="https://dn-anything-about-doc.qbox.me/userid46108labid766time1427423801808" alt="图片描述信息" style="border:0px; vertical-align:middle; outline:none; max-width:97.5%; margin:15px auto; display:block; height:auto"></a></p><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px"><a target="_blank" target="_blank" href="https://dn-anything-about-doc.qbox.me/userid46108labid766time1427423813828" title="图片描述信息" class="fancybox" rel="gallery0" style="background:transparent; color:rgb(202,12,22); margin:0px; padding:0px; outline:none"><img src="https://dn-anything-about-doc.qbox.me/userid46108labid766time1427423813828" alt="图片描述信息" style="border:0px; vertical-align:middle; outline:none; max-width:97.5%; margin:15px auto; display:block; height:auto"></a></p><blockquote style="padding:10px 20px; margin:0px 0px 20px; border-left:5px solid rgb(238,238,238)"><ul style="margin:0px; padding:0px; list-style-position:initial"><li style="margin:0px 0px 0px 40px; padding:0px; list-style:disc"><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; font-size:17.5px; line-height:1.25">hive.metastore.warehouse.dir<br style="">该参数指定了 Hive 的数据存储目录，默认位置在 HDFS 上面的 /user/hive/warehouse 路径下。</p></li><li style="margin:0px 0px 0px 40px; padding:0px; list-style:disc"><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; font-size:17.5px; line-height:1.25">hive.exec.scratchdir<br style="">该参数指定了 Hive 的数据临时文件目录，默认位置为 HDFS 上面的 /tmp/hive 路径下。</p></li></ul></blockquote>同时我们还要修改 Hive 目录下 /conf/hive-env.sh 文件（请根据自己的实际路径修改），该文件默认也不存在，同样是拷贝它的模版来修改：<br style=""></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px">cp hive-env.sh.template hive-env.sh<br style=""></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:1922px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit"># Set HADOOP_HOME to point to a specific hadoop install directory </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">HADOOP_HOME</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=/home/hadoop/cloud/hadoop-2.7.3 </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> </span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"># Hive Configuration Directory can be controlled by: </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">export <span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">HIVE_CONF_DIR</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=/home/hadoop/cloud/apache-hive-2.1.1-bin/conf </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"># Folder containing extra ibraries required for hive compilation/execution can be controlled by: </span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">export <span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">HIVE_AUX_JARS_PATH</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=/home/hadoop/cloud/apache-hive-2.1.1-bin/lib </span></span></li></ol></div><br style="">（2）创建必要目录<br style=""><br style=""><br style="">前面我们看到 hive-site.xml 文件中有两个重要的路径，切换到 hadoop 用户下查看 HDFS 是否有这些路径：<br style=""></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:2203px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">hadoop fs -ls / </span></span></li></ol></div></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px">没有发现上面提到的路径，因此我们需要自己新建这些目录，并且给它们赋予用户写（W）权限。<br style=""></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:2302px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">$HADOOP_HOME/bin/hadoop fs -mkdir -p /user/hive/warehouse </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">$HADOOP_HOME/bin/hadoop fs -mkdir -p /tmp/hive/ </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">hadoop fs -chmod 777 /user/hive/warehouse </span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">hadoop fs -chmod 777 /tmp/hive </span></li></ol></div><br style="">检查是否新建成功 hadoop fs -ls / 以及 hadoop fs -ls /user/hive/ ：<br style=""></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><br style=""></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px">（3）修改 io.tmpdir/user.name</p><p style="margin-top:0px; margin-bottom:10px; padding-top:0px; padding-bottom:0px">同时，要修改 hive-site.xml 中所有包含 <span style="margin:0px; padding:0px; font-family:Menlo,"Bitstream Vera Sans Mono","DejaVu Sans Mono",Monaco,Consolas,monospace; color:rgb(199,37,78)"><span style="margin:0px 2px; padding:0px 5px; font-size:15.12px; background:rgb(238,238,238); word-break:break-all">${system:j<a target="_blank" target="_blank" href="http://lib.csdn.net/base/javase" class="replace_word" title="Java SE知识库" style="background:transparent; color:rgb(223,52,52); margin:0px; padding:0px; font-weight:bold; outline:none">ava</a>.io.tmpdir}</span></span> 字段的 value 即路径（vim下 / 表示搜索，后面跟你的关键词，比如搜索 hello，则为 <span style="margin:0px; padding:0px; font-family:Menlo,"Bitstream Vera Sans Mono","DejaVu Sans Mono",Monaco,Consolas,monospace; color:rgb(199,37,78)"><span style="margin:0px 2px; padding:0px 5px; font-size:15.12px; background:rgb(238,238,238); word-break:break-all">/hello</span></span> , 再回车即可），你可以自己新建一个目录来替换它，例如 <span style="margin:0px; padding:0px; font-family:Menlo,"Bitstream Vera Sans Mono","DejaVu Sans Mono",Monaco,Consolas,monospace; color:rgb(199,37,78)"><span style="margin:0px 2px; padding:0px 5px; font-size:15.12px; background:rgb(238,238,238); word-break:break-all">/home/hadoop/cloud/apache-hive-2.1.1-bin/iotmp</span></span></p></h6><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; color:rgb(69,69,69); margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:2575px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">mkdir /home/hadoop/cloud/apache-hive-2.1.1-bin/iotmp </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">chmod 777 /home/hadoop/cloud/apache-hive-2.1.1-bin/iotmp </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">把hive-site.xml 中所有包含 ${system:Java.io.tmpdir}替换成/home/hadoop/cloud/apache-hive-2.1.1-bin/iotmp </span></li></ol></div><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px">全局替换命令 先按Esc键 再同时按shift+:把以下替换命令粘贴按回车即可全局替换</h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:2710px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">%s#${system:java.io.tmpdir}#/home/hadoop/cloud/apache-hive-2.1.1-bin/iotmp#g </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit"></span></span><p class="p1">%s#${system:user.name}#delin#g</p></li></ol></div><br style="">（4）运行 Hive<br style=""></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:2823px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">./bin/hive </span></span></li></ol></div></h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px">报错</h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><img src="https://img-blog.csdn.net/20170121180043170?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""></h6><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">解决办法：./bin/schematool -initSchema -dbType derby</span><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px">报错</h6><h6 style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0px 0px 10px; font-size:0.8em; padding:0px"><img src="https://img-blog.csdn.net/20170121175046059?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""></h6><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">解决方法：删除/home/hadoop/cloud/apache-hive-2.1.1-bin目录下 rm -rf metastore_db/ 在初始化：</span><span style="margin:0px; padding:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><strong>./bin/schematool -initSchema -dbType derby</strong></span><br style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">重新运行</span><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">./bin/hive<br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">报错</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><img src="https://img-blog.csdn.net/20170121180226141?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">/tem/hive 没写的权限</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">Hive本身自带一个<a target="_blank" target="_blank" href="http://lib.csdn.net/base/mysql" class="replace_word" title="MySQL知识库" style="background:transparent; color:rgb(223,52,52); margin:0px; padding:0px; font-weight:bold; outline:none">数据库</a>，但是有弊端，hive本身数据库，每次只允许一个用户登录<br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">mysql安装：http://blog.csdn.net/u014695188/article/details/51532410</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><h2 id="设置mysql关联hive" style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0.8em 0px; font-size:2.15em; padding:0px"><a target="_blank" target="_blank" name="t2" style="background:transparent; color:rgb(79,161,219); margin:0px; padding:0px; outline:none"></a>设置mysql关联hive</h2><h3 id="修改配置文件" style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0.8em 0px; font-size:1.7em; padding:0px"><a target="_blank" target="_blank" name="t3" style="background:transparent; color:rgb(79,161,219); margin:0px; padding:0px; outline:none"></a><a target="_blank" target="_blank" name="t18" style="background:transparent; color:rgb(79,161,219); margin:0px; padding:0px; outline:none"></a>修改配置文件</h3><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">### 创建hive-site.xml文件 </span><br style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">在hive/conf/目录下创建hive-site.xml文件</span><br style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; color:rgb(69,69,69); margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:3916px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><?</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">xml</span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span><span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">version</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=</span><span class="attribute-value" style="margin:0px; padding:0px; border:none; color:blue; background-color:inherit">"1.0"</span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span><span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">encoding</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=</span><span class="attribute-value" style="margin:0px; padding:0px; border:none; color:blue; background-color:inherit">"UTF-8"</span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span><span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">standalone</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=</span><span class="attribute-value" style="margin:0px; padding:0px; border:none; color:blue; background-color:inherit">"no"</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">?></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><?</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">xml-stylesheet</span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span><span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">type</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=</span><span class="attribute-value" style="margin:0px; padding:0px; border:none; color:blue; background-color:inherit">"text/xsl"</span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span><span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">href</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=</span><span class="attribute-value" style="margin:0px; padding:0px; border:none; color:blue; background-color:inherit">"configuration.xsl"</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">?></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">configuration</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">javax.jdo.option.ConnectionURL</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">jdbc:mysql://localhost:3306/hive?</span></span>useSSL=false&amp<span class="s1" style="color:rgb(69,69,69); background-color:inherit">;</span><span class="attribute" style="background-color:inherit; margin:0px; padding:0px; border:none; color:red">createDatabaseIfNotExist</span><span style="background-color:inherit; color:black; margin:0px; padding:0px; border:none">=</span><span class="attribute-value" style="background-color:inherit; margin:0px; padding:0px; border:none; color:blue">true</span><span class="tag" style="background-color:inherit; margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none"></</span><span class="tag-name" style="background-color:inherit; margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0)">value</span><span class="tag" style="background-color:inherit; margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none">></span><span style="background-color:inherit; color:black; margin:0px; padding:0px; border:none"> </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">javax.jdo.option.ConnectionDriverName</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">com.mysql.jdbc.Driver</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">javax.jdo.option.ConnectionUserName</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">root</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">javax.jdo.option.ConnectionPassword</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">123456</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">hive.metastore.schema.verification</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">false</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">description</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> Enforce metastore schema version consistency. </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> True: Verify that version information stored in metastore matches with one from Hive jars. Also disable automatic </span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> proper metastore schema migration. (Default) </span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> False: Warn if the version information stored in metastore doesn't match with one from in Hive jars. </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">description</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">configuration</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li></ol></div><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">报错：Caused by: MetaException(message:Version information not found in metastore. )</span><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">解决：hive-site.xml加入</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; color:rgb(69,69,69); margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:4604px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">hive.metastore.schema.verification</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">name</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit">false</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">value</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"><</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">description</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> Enforce metastore schema version consistency. </span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> True: Verify that version information stored in metastore matches with one from Hive jars. Also disable automatic </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures </span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> proper metastore schema migration. (Default) </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> False: Warn if the version information stored in metastore doesn't match with one from in Hive jars. </span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">description</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> <span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit"></</span><span class="tag-name" style="margin:0px; padding:0px; font-weight:bold; border:none; color:rgb(153,51,0); background-color:inherit">property</span><span class="tag" style="margin:0px; padding:0px; font-weight:bold; color:rgb(153,51,0); border:none; background-color:inherit">></span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> </span></span></li></ol></div><br style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><br style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">报错：缺少mysql jar包</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">解决：将其（如mysql-connector-java-5.1.15-bin.jar）拷贝到$HIVE_HOME/lib下即可。</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">报错：</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; color:rgb(69,69,69); margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:5004px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">Exception in thread "main" java.lang.RuntimeException: Hive metastore database is not initialized. </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?<span class="attribute" style="margin:0px; padding:0px; border:none; color:red; background-color:inherit">createDatabaseIfNotExist</span><span style="margin:0px; padding:0px; border:none; background-color:inherit">=</span><span class="attribute-value" style="margin:0px; padding:0px; border:none; color:blue; background-color:inherit">true</span><span style="margin:0px; padding:0px; border:none; background-color:inherit"> for mysql) </span></span></li></ol></div><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">解决：</span><br style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; color:rgb(69,69,69); margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:5185px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">#数据库的初始化。 </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"> bin/schematool -initSchema -dbType mysql </span></li></ol></div><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">报错：[ERROR] Terminal initialization failed; falling back to unsupported<br style="">java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected<br style=""> at jline.TerminalFactory.create(TerminalFactory.java:101)<br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">解决：http://www.68idc.cn/help/buildlang/ask/20151006570946.html</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">${HADOOP_HOME}/share/hadoop/yarn/lib 目录下存在老版本jline<br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">解决：把${HIVE_HOME}/lib/下的 jline-x.xx拷贝得到${HADOOP_HOME}/share/hadoop/yarn/lib</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">cp /hive/apache-hive-1.1.0-bin/lib/jline-2.12.jar /hadoop-2.5.2/share/hadoop/yarn/lib</p><br style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">启动：</p><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; color:rgb(69,69,69); margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:5552px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">bin/hive </span></span></li></ol></div><br style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">启动后mysql 多了hive 数据库</span><br style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><img src="https://img-blog.csdn.net/20170122145434927?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">测试</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><h3 id="创建数据库" style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0.8em 0px; font-size:1.7em; padding:0px"><a target="_blank" target="_blank" name="t4" style="background:transparent; color:rgb(79,161,219); margin:0px; padding:0px; outline:none"></a><a target="_blank" target="_blank" name="t22" style="background:transparent; color:rgb(79,161,219); margin:0px; padding:0px; outline:none"></a>创建数据库</h3><div style="margin:0px; padding:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">create database db_hive_test;<br style=""></div><div style="margin:0px; padding:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><img src="https://img-blog.csdn.net/20170122145641806?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""></div><div style="margin:0px; padding:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><br style=""></div><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">创建测试表</span><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">use db_hive_test;<br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">create table student(id int,name string) row format delimited fields terminated by '\t';</p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><img src="https://img-blog.csdn.net/20170122151518519?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><img src="https://img-blog.csdn.net/20170122145825134?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""><br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><h4 id="加载数据到表中" style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0.8em 0px; font-size:1.25em; padding:0px">加载数据到表中</h4><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">新建student.txt 文件写入数据(id，name 按tab键分隔)</p><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">vi student.txt</span><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><div class="dp-highlighter bg_html" style="padding:1px 0px 0px; font-family:Consolas,"Courier New",Courier,mono,serif; background-color:rgb(231,229,220); width:851.391px; overflow-x:auto; overflow-y:hidden; position:relative; color:rgb(69,69,69); margin:18px 0px!important"><div class="bar" style="margin:0px; padding:0px 0px 0px 45px"><div class="tools" style="margin:0px; padding:3px 8px 10px 10px; font-size:9px; line-height:normal; font-family:Verdana,Geneva,Arial,Helvetica,sans-serif; color:silver; background-color:rgb(248,248,248); border-left:3px solid rgb(108,226,108); border-right:1px solid rgb(231,229,220)"><span style="">[html]</span> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="ViewSource" title="view plain" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">view plain</a><span class="tracking-ad" style="margin:0px; padding:0px"> <a target="_blank" target="_blank" href="http://blog.csdn.net/u014695188/article/details/54646575#" class="CopyToClipboard" title="copy" style="background-color:inherit; color:rgb(202,12,22); margin:0px 10px 0px 0px; padding:1px; outline:none; border:none; font-size:9px; display:inline-block; width:16px; height:16px; text-indent:-2000px">copy</a></span><div style="margin:0px; padding:0px; position:absolute; left:259px; top:6410px; width:16px; height:16px; z-index:99"></div><span class="tracking-ad" style="margin:0px; padding:0px"></span></div></div><ol start="1" class="dp-xml" style="padding:0px; list-style-position:initial; border-top:none; border-right:1px solid rgb(231,229,220); border-bottom:none; border-left:none; background-color:rgb(255,255,255); color:rgb(92,92,92); margin:0px 0px 1px 45px!important"><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit"><span style="margin:0px; padding:0px; border:none; background-color:inherit">1001 zhangsan </span></span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">1002 lisi </span></li><li class="alt" style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); color:inherit; line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">1003 wangwu </span></li><li style="margin-left:40px; list-style:decimal; border-top:none; border-right:none; border-bottom:none; border-left:3px solid rgb(108,226,108); background-color:rgb(248,248,248); line-height:18px; margin-top:0px!important; margin-right:0px!important; margin-bottom:0px!important; padding:0px 3px 0px 10px!important"><span style="margin:0px; padding:0px; border:none; color:black; background-color:inherit">1004 zhaoli </span></li></ol></div><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px">load data local inpath '/home/hadoop/student.txt' into table db_hive_test.student</span><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><img src="https://img-blog.csdn.net/20170122151730987?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><h5 id="查询表信息" style="font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:0.8em 0px; font-size:1em; padding:0px">查询表信息</h5><code style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace; font-size:14.4px; padding:2px 4px; color:rgb(199,37,78); white-space:nowrap; background-color:rgb(249,242,244)">select * from student;</code><span style="color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></span><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace"><br style=""></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace">查看表的详细信息</span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace"><code style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace; font-size:14.4px; padding:2px 4px; color:rgb(199,37,78); white-space:nowrap; background-color:rgb(249,242,244)">desc formatted student;</code></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace"><br style=""></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace">通过ui页面查看创建的数据位置</span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace">http://192.168.169.132:50070/explorer.html#/user/hive/warehouse/db_hive_test.db<br style=""></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace"><img src="https://img-blog.csdn.net/20170122151909697?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace">通过Mysql查看创建的表</span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace"><img src="https://img-blog.csdn.net/20170122152504896?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace"><br style=""></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace">查看hive的函数 <br style=""></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace"><code style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace; font-size:14.4px; padding:2px 4px; color:rgb(199,37,78); white-space:nowrap; background-color:rgb(249,242,244)">show functions;</code><br style=""></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace"><code style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace; font-size:14.4px; padding:2px 4px; color:rgb(199,37,78); white-space:nowrap; background-color:rgb(249,242,244)">查看函数详细信息 <br style=""></code></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace"><code style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace; font-size:14.4px; padding:2px 4px; color:rgb(199,37,78); white-space:nowrap; background-color:rgb(249,242,244)"><code style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace; font-size:12.96px; padding:2px 4px">desc function sum; <br style="">desc function extended</code><br style=""></code></span></p><p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:"PingFang SC","Microsoft YaHei",SimHei,Arial,SimSun; font-size:16px"><span style="margin:0px; padding:0px; font-family:monospace"><code style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace; font-size:14.4px; padding:2px 4px; color:rgb(199,37,78); white-space:nowrap; background-color:rgb(249,242,244)"><code style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace; font-size:12.96px; padding:2px 4px"><img src="https://img-blog.csdn.net/20170122152947399?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvdTAxNDY5NTE4OA==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center" alt="" style="border:0px; vertical-align:middle; outline:none; max-width:100%"><br style=""></code></code></span></p><div><span style="margin:0px; padding:0px; font-family:monospace"><code style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace; font-size:14.4px; padding:2px 4px; color:rgb(199,37,78); white-space:nowrap; background-color:rgb(249,242,244)"><code style="font-family:Monaco,Menlo,Consolas,"Courier New",monospace; font-size:12.96px; padding:2px 4px"><br></code></code></span></div>]]></content>
<summary type="html"><h2 id="二、Hive-运行模式" style="font-family:&quot;PingFang SC&quot;,&quot;Microsoft YaHei&quot;,SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:-25px 0px 0.83em; font-size:2.15em; padding:40px 0px 0px">
<p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:&quot;PingFang SC&quot;,&quot;Microsoft YaHei&quot;,SimHei,Arial,SimSun; font-size:16px">
转：</p>
<p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:&quot;PingFang SC&quot;,&quot;Microsoft YaHei&quot;,SimHei,Arial,SimSun; font-size:16px">
http://blog.csdn.net/reesun/article/details/8556078<br style="">
</p>
<p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:&quot;PingFang SC&quot;,&quot;Microsoft YaHei&quot;,SimHei,Arial,SimSun; font-size:16px">
http://blog.csdn.net/zhongguozhichuang/article/details/52702476</p>
<p style="margin-top:0px; margin-bottom:0px; padding-top:0px; padding-bottom:0px; color:rgb(69,69,69); font-family:&quot;PingFang SC&quot;,&quot;Microsoft YaHei&quot;,SimHei,Arial,SimSun; font-size:16px">
http://blog.csdn.net/huoyuyan/article/details/50769104</p>
</h2>
<h2 id="二、Hive-运行模式" style="font-family:&quot;PingFang SC&quot;,&quot;Microsoft YaHei&quot;,SimHei,Arial,SimSun; font-weight:100; line-height:1.1; color:rgb(69,69,69); margin:-25px 0px 0.83em; font-size:2.15em; padding:40px 0px 0px">
一、Hive 运行模式</h2></summary>
<category term="hive" scheme="https://idelin.github.io/categories/hive/"/>
</entry>
<entry>
<title>Hadoop安装教程_伪分布式配置</title>
<link href="https://idelin.github.io/2017/10/06/Hadoop%E5%AE%89%E8%A3%85%E6%95%99%E7%A8%8B_%E4%BC%AA%E5%88%86%E5%B8%83%E5%BC%8F%E9%85%8D%E7%BD%AE/"/>
<id>https://idelin.github.io/2017/10/06/Hadoop%E5%AE%89%E8%A3%85%E6%95%99%E7%A8%8B_%E4%BC%AA%E5%88%86%E5%B8%83%E5%BC%8F%E9%85%8D%E7%BD%AE/</id>
<published>2017-10-06T06:18:41.000Z</published>
<updated>2022-08-03T12:45:23.441Z</updated>
<content type="html"><![CDATA[<h2 style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; line-height:1.1; color:rgb(51,51,51); margin-top:20px; margin-bottom:10px; font-size:30px; border-left:10px solid rgb(52,152,219); padding:10px 0px 10px 14px; background-color:rgb(247,247,247)">创建hadoop用户</h2><span id="more"></span><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">如果你安装 CentOS 的时候不是用的 "hadoop" 用户，那么需要增加一个名为 hadoop 的用户。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">首先点击左上角的 “应用程序” -> "系统工具" -> "终端"，首先在终端中输入 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">su</code> ，按回车，输入 root 密码以 root 用户登录，接着执行命令创建新用户 hadoop:</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">su </span><span class="com" style="font-style:italic"># 上述提到的以 root 用户登录</span></li><li class="L1" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">useradd </span><span class="pln" style="color:rgb(255,255,255)">-m hadoop -s /bin/bash </span><span class="com" style="font-style:italic"># 创建新用户hadoop</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">如下图所示，这条命令创建了可以登陆的 hadoop 用户，并使用 /bin/bash 作为shell。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">接着使用如下命令修改密码，按提示输入两次密码，可简单的设为 "hadoop"（密码随意指定，若提示“无效的密码，过于简单”则再次输入确认就行）:</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">passwd </span><span class="pln" style="color:rgb(255,255,255)">hadoop</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><img src="" alt=""><br></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">可为 hadoop 用户增加管理员权限，方便部署，避免一些对新手来说比较棘手的权限问题，执行：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="color:rgb(117,113,94); margin-top:0px; margin-bottom:0px; padding-left:40px; list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="pln" style="color:rgb(255,255,255)">visudo</span></li></ol></pre><div class="code-pretty-toolbar" style="color:rgb(221,221,221); position:absolute; top:0px; left:0px; font-size:12px"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">如下图，找到 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">root ALL=(ALL) ALL</code> 这行（应该在第98行，可以先按一下键盘上的 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">ESC</code> 键，然后输入 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">:98</code> (按一下冒号，接着输入98，再按回车键)，可以直接跳到第98行 ），然后在这行下面增加一行内容：<code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">hadoop ALL=(ALL) ALL</code>（当中的间隔为tab）。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><span style="text-align:center">添加上一行内容后，先按一下键盘上的 </span><code style="text-align:center; font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">ESC</code><span style="text-align:center"> 键，然后输入 </span><code style="text-align:center; font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">:wq</code><span style="text-align:center"> (输入冒号还有wq，这是vi/vim编辑器的保存方法)，再按回车键保存退出就可以了。</span></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">最后注销当前用户(点击屏幕右上角的用户名，选择退出->注销)，在登陆界面使用刚创建的 hadoop 用户进行登陆。（如果已经是 hadoop 用户，且在终端中使用 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">su</code> 登录了 root 用户，那么需要执行 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">exit</code> 退出 root 用户状态）</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"></p><h2 style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; line-height:1.1; color:rgb(51,51,51); margin-top:20px; margin-bottom:10px; font-size:30px; border-left:10px solid rgb(52,152,219); padding:10px 0px 10px 14px; background-color:rgb(247,247,247)">安装SSH、配置SSH无密码登陆</h2><p></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">集群、单节点模式都需要用到 SSH 登陆（类似于远程登陆，你可以登录某台 Linux 主机，并且在上面运行命令），一般情况下，CentOS 默认已安装了 SSH client、SSH server。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">若需要安装，则可以通过 yum 进行安装（安装过程中会让你输入 [y/N]，输入 y 即可）：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">sudo </span><span class="pln" style="color:rgb(255,255,255)">yum install openssh-clients</span></li><li class="L1" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">sudo </span><span class="pln" style="color:rgb(255,255,255)">yum install openssh-server</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">接着执行如下命令测试一下 SSH 是否可用：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">ssh </span><span class="pln" style="color:rgb(255,255,255)">localhost</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">此时会有如下提示(SSH首次登陆提示)，输入 yes 。然后按提示输入密码 hadoop，这样就登陆到本机了。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">但这样登陆是需要每次输入密码的，我们需要配置成SSH无密码登陆比较方便。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">首先输入 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">exit</code> 退出刚才的 ssh，就回到了我们原先的终端窗口，然后利用 ssh-keygen 生成密钥，并将密钥加入到授权中：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="pln" style="color:rgb(255,255,255)">exit </span><span class="com" style="font-style:italic"># 退出刚才的 ssh localhost</span></li><li class="L1" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">cd </span><span class="pln" style="color:rgb(255,255,255)">~/.ssh/ </span><span class="com" style="font-style:italic"># 若没有该目录，请先执行一次ssh localhost</span></li><li class="L2" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="pln" style="color:rgb(255,255,255)">ssh-keygen -t rsa </span><span class="com" style="font-style:italic"># 会有提示，都按回车就可以</span></li><li class="L3" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">cat </span><span class="pln" style="color:rgb(255,255,255)">id_rsa.pub >> authorized_keys </span><span class="com" style="font-style:italic"># 加入授权</span></li><li class="L4" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">chmod </span><span class="pln" style="color:rgb(255,255,255)">600 ./authorized_keys </span><span class="com" style="font-style:italic"># 修改文件权限</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><div class="callout callout-note" style="margin:20px 0px 20px 20px; padding:14px 12px 0px 14px; border-width:1px 1px 1px 8px; border-style:solid; border-color:rgb(238,238,238) rgb(238,238,238) rgb(238,238,238) rgb(71,173,9); background:rgb(252,252,252); color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px"><span class="callout-title" style="font-size:20px; color:rgb(0,136,0)">~的含义</span><p style="margin-top:0px; margin-bottom:15px">在 Linux 系统中，~ 代表的是用户的主文件夹，即 "/home/用户名" 这个目录，如你的用户名为 hadoop，则 ~ 就代表 "/home/hadoop/"。 此外，命令中的 # 后面的文字是注释。</p></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">此时再用 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">ssh localhost</code> 命令，无需输入密码就可以直接登陆了，如下图所示。</p><div><h2 style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; line-height:1.1; color:rgb(51,51,51); margin-top:20px; margin-bottom:10px; font-size:30px; border-left:10px solid rgb(52,152,219); padding:10px 0px 10px 14px; background-color:rgb(247,247,247)">安装Java环境</h2></div><p></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"></p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">sudo </span><span class="pln" style="color:rgb(255,255,255)">yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">通过上述命令安装 OpenJDK，默认安装位置为 /usr/lib/jvm/java-1.8.0（该路径可以通过执行 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">rpm -ql java-1.7.0-openjdk-devel | grep '/bin/javac'</code> 命令确定，执行后会输出一个路径，除去路径末尾的 “/bin/javac”，剩下的就是正确的路径了）。OpenJDK 安装后就可以直接使用 java、javac 等命令了。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">接着需要配置一下 JAVA_HOME 环境变量，为方便，我们在 ~/.bashrc 中进行设置：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">vim </span><span class="pln" style="color:rgb(255,255,255)">~/.bashrc</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">在文件最后面添加如下单独一行（指向 JDK 的安装位置），并保存：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-shell prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px"><span class="kwd" style="color:rgb(102,217,239)">export </span><span class="pln" style="color:rgb(255,255,255)">JAVA_HOME=<span style="color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">/usr/lib/jvm/java-1.8.0</span></span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">source <span style="color:rgb(255,255,255); font-family:consolas,menlo,courier,monospace; font-size:16px; white-space:pre-wrap; background-color:rgb(39,40,34)">~/.bashrc</span></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><span style="color:rgb(255,255,255); font-family:consolas,menlo,courier,monospace; font-size:16px; white-space:pre-wrap; background-color:rgb(39,40,34)"><br></span></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><span style="color:rgb(255,255,255); font-family:consolas,menlo,courier,monospace; font-size:16px; white-space:pre-wrap; background-color:rgb(39,40,34)">cd $JAVA_HOME</span></p><h2 style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; line-height:1.1; color:rgb(51,51,51); margin-top:20px; margin-bottom:10px; font-size:30px; border-left:10px solid rgb(52,152,219); padding:10px 0px 10px 14px; background-color:rgb(247,247,247)">Hadoop伪分布式配置</h2><p></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">Hadoop 可以在单节点上以伪分布式的方式运行，Hadoop 进程以分离的 Java 进程来运行，节点既作为 NameNode 也作为 DataNode，同时，读取的是 HDFS 中的文件。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">在设置 Hadoop 伪分布式配置前，我们还需要设置 HADOOP 环境变量，执行如下命令在 ~/.bashrc 中设置：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="pln" style="color:rgb(255,255,255)">vim ~/.bashrc</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">这次我们选择用 gedit 而不是 vim 来编辑。gedit 是文本编辑器，类似于 Windows 中的记事本，会比较方便。保存后记得关掉整个 gedit 程序，否则会占用终端。在文件最后面增加如下内容：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-shell prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px"><span class="com" style="font-style:italic"># Hadoop Environment Variables</span></li><li class="L1" style="list-style:decimal; margin:0px; padding:0px"><span class="kwd" style="color:rgb(102,217,239)">export </span><span class="pln" style="color:rgb(255,255,255)">HADOOP_HOME=/usr/local/hadoop </span></li><li class="L2" style="list-style:decimal; margin:0px; padding:0px"><span class="kwd" style="color:rgb(102,217,239)">export </span><span class="pln" style="color:rgb(255,255,255)">HADOOP_INSTALL=$HADOOP_HOME</span></li><li class="L3" style="list-style:decimal; margin:0px; padding:0px"><span class="kwd" style="color:rgb(102,217,239)">export </span><span class="pln" style="color:rgb(255,255,255)">HADOOP_MAPRED_HOME=$HADOOP_HOME</span></li><li class="L4" style="list-style:decimal; margin:0px; padding:0px"><span class="kwd" style="color:rgb(102,217,239)">export </span><span class="pln" style="color:rgb(255,255,255)">HADOOP_COMMON_HOME=$HADOOP_HOME</span></li><li class="L5" style="list-style:decimal; margin:0px; padding:0px"><span class="kwd" style="color:rgb(102,217,239)">export </span><span class="pln" style="color:rgb(255,255,255)">HADOOP_HDFS_HOME=$HADOOP_HOME</span></li><li class="L6" style="list-style:decimal; margin:0px; padding:0px"><span class="kwd" style="color:rgb(102,217,239)">export </span><span class="pln" style="color:rgb(255,255,255)">YARN_HOME=$HADOOP_HOME</span></li><li class="L7" style="list-style:decimal; margin:0px; padding:0px"><span class="kwd" style="color:rgb(102,217,239)">export </span><span class="pln" style="color:rgb(255,255,255)">HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native</span></li><li class="L8" style="list-style:decimal; margin:0px; padding:0px"><span class="kwd" style="color:rgb(102,217,239)">export </span><span class="pln" style="color:rgb(255,255,255)">PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">保存后，不要忘记执行如下命令使配置生效：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">source </span><span class="pln" style="color:rgb(255,255,255)">~/.bashrc</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">这些变量在启动 Hadoop 进程时需要用到，不设置的话可能会报错（这些变量也可以通过修改 ./etc/hadoop/hadoop-env.sh 实现）。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">hadoop version</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">Hadoop 的配置文件位于 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">/usr/local/hadoop/etc/hadoop/</code> 中，伪分布式需要修改2个配置文件 <span style="">core-site.xml</span> 和 <span style="">hdfs-site.xml</span> 。Hadoop的配置文件是 xml 格式，每个配置以声明 property 的 name 和 value 的方式来实现。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">修改配置文件 <span style="">core-site.xml</span> (通过 gedit 编辑会比较方便: <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">gedit ./etc/hadoop/core-site.xml</code>)，将当中的</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-xml prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px"><span class="tag" style="color:rgb(249,38,114)"><configuration></span></li><li class="L1" style="list-style:decimal; margin:0px; padding:0px"><span class="tag" style="color:rgb(249,38,114)"></configuration></span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">XML</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">修改为下面配置：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-xml prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; list-style-position:inside!important"><li class="L0" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="tag" style="color:rgb(249,38,114)"><configuration></span></li><li class="L1" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><property></span></li><li class="L2" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><name></span><span class="pln" style="color:rgb(255,255,255)">hadoop.tmp.dir</span><span class="tag" style="color:rgb(249,38,114)"></name></span></li><li class="L3" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> <value>/usr/local/hadoop/tmp</span><span class="tag" style="color:rgb(249,38,114)"></value></span></li><li class="L4" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><description></span><span class="pln" style="color:rgb(255,255,255)">Abase for other temporary directories.</span><span class="tag" style="color:rgb(249,38,114)"></description></span></li><li class="L5" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"></property></span></li><li class="L6" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><property></span></li><li class="L7" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><name></span><span class="pln" style="color:rgb(255,255,255)">fs.defaultFS</span><span class="tag" style="color:rgb(249,38,114)"></name></span></li><li class="L8" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><value></span><span class="pln" style="color:rgb(255,255,255)">hdfs://localhost:9000</span><span class="tag" style="color:rgb(249,38,114)"></value></span></li><li class="L9" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"></property></span></li><li class="L0" style="color:rgb(117,113,94); list-style:decimal; margin:0px; padding:0px"><span class="tag" style="color:rgb(249,38,114)"></configuration></span></li></ol></pre><div class="code-pretty-toolbar" style="color:rgb(221,221,221); position:absolute; top:0px; left:0px; font-size:12px"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">XML</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">同样的，修改配置文件 <span style="">hdfs-site.xml</span>：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-xml prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px"><span class="tag" style="color:rgb(249,38,114)"><configuration></span></li><li class="L1" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><property></span></li><li class="L2" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><name></span><span class="pln" style="color:rgb(255,255,255)">dfs.replication</span><span class="tag" style="color:rgb(249,38,114)"></name></span></li><li class="L3" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><value></span><span class="pln" style="color:rgb(255,255,255)">1</span><span class="tag" style="color:rgb(249,38,114)"></value></span></li><li class="L4" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"></property></span></li><li class="L5" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><property></span></li><li class="L6" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><name></span><span class="pln" style="color:rgb(255,255,255)">dfs.namenode.name.dir</span><span class="tag" style="color:rgb(249,38,114)"></name></span></li><li class="L7" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><value></span><span class="pln" style="color:rgb(255,255,255)">file:/usr/local/hadoop/tmp/dfs/name</span><span class="tag" style="color:rgb(249,38,114)"></value></span></li><li class="L8" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"></property></span></li><li class="L9" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><property></span></li><li class="L0" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><name></span><span class="pln" style="color:rgb(255,255,255)">dfs.datanode.data.dir</span><span class="tag" style="color:rgb(249,38,114)"></name></span></li><li class="L1" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><value></span><span class="pln" style="color:rgb(255,255,255)">file:/usr/local/hadoop/tmp/dfs/data</span><span class="tag" style="color:rgb(249,38,114)"></value></span></li><li class="L2" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"></property></span></li><li class="L3" style="list-style:decimal; margin:0px; padding:0px"><span class="tag" style="color:rgb(249,38,114)"></configuration></span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">XML</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">配置完成后，执行 NameNode 的格式化:</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="pln" style="color:rgb(255,255,255)">hadoop namenode -format</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">成功的话，会看到 "successfully formatted" 和 "Exitting with status 0" 的提示，若为 "Exitting with status 1" 则是出错。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">接着开启 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">NaneNode</code> 和 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">DataNode</code> 守护进程：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="pln" style="color:rgb(255,255,255)">./sbin/start-dfs.sh</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">若出现如下 SSH 的提示 "Are you sure you want to continue connecting"，输入 yes 即可。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">启动时可能会有 WARN 提示 "WARN util.NativeCodeLoader..." 如前面提到的，这个提示不会影响正常使用。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">启动完成后，可以通过命令 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">jps</code> 来判断是否成功启动，若成功启动则会列出如下进程: "NameNode"、"DataNode"和<code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">SecondaryNameNode</code>（如果 SecondaryNameNode 没有启动，请运行 sbin/stop-dfs.sh 关闭进程，然后再次尝试启动尝试）。如果没有 NameNode 或 DataNode ，那就是配置不成功，请仔细检查之前步骤，或通过查看启动日志排查原因。</p><div class="callout callout-tips" style="margin:20px 0px 20px 20px; padding:14px 12px 0px 14px; border-width:1px 1px 1px 8px; border-style:solid; border-color:rgb(238,238,238) rgb(238,238,238) rgb(238,238,238) rgb(71,173,9); background:rgb(252,252,252); color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px"><span class="callout-title" style="font-weight:bold; font-size:20px; color:rgb(0,136,0)">通过查看启动日志分析启动失败原因</span><p style="margin-top:0px; margin-bottom:15px">有时 Hadoop 无法正确启动，如 NameNode 进程没有顺利启动，这时可以查看启动日志来排查原因，注意几点：</p><ul style="margin:0px 0px 15px 20px; list-style:none; padding:0px"><li style="list-style:disc; margin:0px; padding:0px">启动时会提示形如 "dblab: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-dblab.out"，其中 dblab 对应你的主机名，但启动的日志信息是记录在 /usr/local/hadoop/logs/hadoop-hadoop-namenode-dblab.log 中，所以应该查看这个后缀为 <span style="">.log</span> 的文件；</li><li style="list-style:disc; margin:0px; padding:0px">每一次的启动日志都是追加在日志文件之后，所以得拉到最后面看，看下记录的时间就知道了。</li><li style="list-style:disc; margin:0px; padding:0px">一般出错的提示在最后面，也就是写着 Fatal、Error 或者 Java Exception 的地方。</li><li style="list-style:disc; margin:0px; padding:0px">可以在网上搜索一下出错信息，看能否找到一些相关的解决方法。</li></ul></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">成功启动后，可以访问 Web 界面 <a target="_blank" target="_blank" href="http://localhost:50070/" style="background:0px 0px; color:rgb(32,128,223)">http://localhost:50070</a> 查看 NameNode 和 Datanode 信息，还可以在线查看 HDFS 中的文件。</p><h2 style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; line-height:1.1; color:rgb(51,51,51); margin-top:20px; margin-bottom:10px; font-size:30px; border-left:10px solid rgb(52,152,219); padding:10px 0px 10px 14px; background-color:rgb(247,247,247)">启动YARN</h2><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">（伪分布式不启动 YARN 也可以，一般不会影响程序执行）</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">有的读者可能会疑惑，怎么启动 Hadoop 后，见不到书上所说的 JobTracker 和 TaskTracker，这是因为新版的 Hadoop 使用了新的 MapReduce 框架（MapReduce V2，也称为 YARN，Yet Another Resource Negotiator）。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">YARN 是从 MapReduce 中分离出来的，负责资源管理与任务调度。YARN 运行于 MapReduce 之上，提供了高可用性、高扩展性，YARN 的更多介绍在此不展开，有兴趣的可查阅相关资料。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">上述通过 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">./sbin/start-dfs.sh</code> 启动 Hadoop，仅仅是启动了 MapReduce 环境，我们可以启动 YARN ，让 YARN 来负责资源管理与任务调度。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">首先修改配置文件 <span style="">mapred-site.xml</span>，这边需要先进行重命名：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="kwd" style="color:rgb(102,217,239)">mv </span><span class="pln" style="color:rgb(255,255,255)">./etc/hadoop/mapred-site.xml.template ./etc/hadoop/mapred-site.xml</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">然后再进行编辑，同样使用 gedit 编辑会比较方便些 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">gedit ./etc/hadoop/mapred-site.xml</code> ：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-xml prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px"><span class="tag" style="color:rgb(249,38,114)"><configuration></span></li><li class="L1" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><property></span></li><li class="L2" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><name></span><span class="pln" style="color:rgb(255,255,255)">mapreduce.framework.name</span><span class="tag" style="color:rgb(249,38,114)"></name></span></li><li class="L3" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><value></span><span class="pln" style="color:rgb(255,255,255)">yarn</span><span class="tag" style="color:rgb(249,38,114)"></value></span></li><li class="L4" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"></property></span></li><li class="L5" style="list-style:decimal; margin:0px; padding:0px"><span class="tag" style="color:rgb(249,38,114)"></configuration></span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">XML</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">接着修改配置文件 <span style="">yarn-site.xml</span>：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-xml prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px"><span class="tag" style="color:rgb(249,38,114)"><configuration></span></li><li class="L1" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><property></span></li><li class="L2" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><name></span><span class="pln" style="color:rgb(255,255,255)">yarn.nodemanager.aux-services</span><span class="tag" style="color:rgb(249,38,114)"></name></span></li><li class="L3" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"><value></span><span class="pln" style="color:rgb(255,255,255)">mapreduce_shuffle</span><span class="tag" style="color:rgb(249,38,114)"></value></span></li><li class="L4" style="list-style:decimal; margin:0px; padding:0px"><span class="pln" style="color:rgb(255,255,255)"> </span><span class="tag" style="color:rgb(249,38,114)"></property></span></li><li class="L5" style="list-style:decimal; margin:0px; padding:0px"><span class="tag" style="color:rgb(249,38,114)"></configuration></span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">XML</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">然后就可以启动 YARN 了（需要先执行过 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">./sbin/start-dfs.sh</code>）：</p><div class="code-pretty-container" style="position:relative; padding-top:9px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><pre class="prettyprint linenums lang-bash prettyprinted" style="overflow:auto; font-family:consolas,menlo,courier,monospace; padding:1em; margin:1em auto; line-height:20px; word-break:break-all; word-wrap:break-word; background-color:rgb(39,40,34); border:1px solid rgb(11,12,10); white-space:pre-wrap; width:847.5px"><ol class="linenums" style="margin-top:0px; margin-bottom:0px; padding-left:40px; color:rgb(117,113,94); list-style-position:inside!important"><li class="L0" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="pln" style="color:rgb(255,255,255)">./sbin/start-yarn.sh $ 启动YARN</span></li><li class="L1" style="list-style:decimal; margin:0px; padding:0px 0px 0px 20px"><span class="pln" style="color:rgb(255,255,255)">./sbin/mr-jobhistory-daemon.sh start historyserver </span><span class="com" style="font-style:italic"># 开启历史服务器，才能在Web中查看任务运行情况</span></li></ol></pre><div class="code-pretty-toolbar" style="position:absolute; top:0px; left:0px; font-size:12px; color:rgb(221,221,221)"><span class="title" style="display:inline-block; vertical-align:top; background-color:rgb(30,31,26); border-top:1px solid rgb(11,12,10); border-right:1px solid rgb(11,12,10); border-left:1px solid rgb(11,12,10); padding:4px 14px 2px; border-bottom:none; margin-right:12px">Shell 命令</span><a target="_blank" target="_blank" title="复制代码" class="tool clipboard" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-files-o" style=""></span></a><a target="_blank" target="_blank" title="查看纯文本代码" class="tool view-source" style="background:0px 0px; color:rgb(119,119,119); display:inline-block; vertical-align:top; font-size:16px; width:24px; height:24px; margin:-1px 3px 0px 0px"><span class="fa fa-code" style=""></span></a><span class="msg" style="display:inline-block; vertical-align:top; color:rgb(39,40,34); line-height:28px; margin-left:10px"></span></div></div><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">开启后通过 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">jps</code> 查看，可以看到多了 NodeManager 和 ResourceManager 两个后台进程。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"></p><h2 style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; line-height:1.1; color:rgb(51,51,51); margin-top:20px; margin-bottom:10px; font-size:30px; border-left:10px solid rgb(52,152,219); padding:10px 0px 10px 14px; background-color:rgb(247,247,247)">附加教程: 配置PATH环境变量</h2><p></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">在这里额外讲一下 PATH 这个环境变量（可执行 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">echo $PATH</code> 查看，当中包含了多个目录）。例如我们在主文件夹 ~ 中执行 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">ls</code> 这个命令时，实际执行的是 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">/bin/ls</code> 这个程序，而不是 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">~/ls</code> 这个程序。系统是根据 PATH 这个环境变量中包含的目录位置，逐一进行查找，直至在这些目录位置下找到匹配的程序（若没有匹配的则提示该命令不存在）。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">上面的教程中，我们都是先进入到 /usr/local/hadoop 目录中，再执行 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">./sbin/hadoop</code>，实际上等同于运行 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">/usr/local/hadoop/sbin/hadoop</code>。我们可以将 Hadoop 命令的相关目录加入到 PATH 环境变量中，这样就可以直接通过 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">start-dfs.sh</code> 开启 Hadoop，也可以直接通过 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">hdfs</code> 访问 HDFS 的内容，方便平时的操作。</p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">在前面我们设置 HADOOP 环境变量时，我们已经顺便设置了 PATH 变量（即 "export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin"），那么以后我们在任意目录中都可以直接通过执行 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">start-dfs.sh</code> 来启动 Hadoop 或者执行 <code style="font-family:Menlo,Monaco,Consolas,"Courier New",monospace; font-size:14.4px; padding:4px 6px; background-color:rgb(254,233,204); vertical-align:middle">hdfs dfs -ls input</code> 查看 HDFS 文件了。</p><br><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><span style="color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)">在平时的学习中，我们使用伪分布式就足够了。</span><br></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><span style="color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><br></span></p><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><span style="color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"></span></p><h2 style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; line-height:1.1; color:rgb(51,51,51); margin-top:20px; margin-bottom:10px; font-size:30px; border-left:10px solid rgb(52,152,219); padding:10px 0px 10px 14px; background-color:rgb(247,247,247)">参考资料</h2><ul style="margin:0px 0px 15px 20px; list-style:none; padding:0px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><li style="list-style:disc; margin:0px; padding:0px"><a target="_blank" target="_blank" href="http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html" style="background:0px 0px; color:rgb(32,128,223)">Hadoop: Setting up a Single Node Cluster</a></li><li style="list-style:disc; margin:0px; padding:0px"><a target="_blank" target="_blank" href="http://tecadmin.net/setup-hadoop-single-node-cluster-on-centos-redhat/" style="background:0px 0px; color:rgb(32,128,223)">How to Setup Hadoop 2.7.1 on CentOS, Ubuntu & LinuxMint</a></li><li style="list-style:disc; margin:0px; padding:0px"><a target="_blank" target="_blank" href="http://blog.chinaunix.net/uid-28311809-id-4383551.html" style="background:0px 0px; color:rgb(32,128,223)">Yarn简单介绍及内存配置</a></li></ul><p></p><br><p style="margin-top:0px; margin-bottom:15px; color:rgb(51,51,51); font-family:"Helvetica Neue",Helvetica,Arial,sans-serif; font-size:16px; background-color:rgb(254,254,254)"><br></p>]]></content>
<summary type="html"><h2 style="font-family:&quot;Helvetica Neue&quot;,Helvetica,Arial,sans-serif; line-height:1.1; color:rgb(51,51,51); margin-top:20px; margin-bottom:10px; font-size:30px; border-left:10px solid rgb(52,152,219); padding:10px 0px 10px 14px; background-color:rgb(247,247,247)">
创建hadoop用户</h2></summary>
</entry>
<entry>
<title>elasticsearch UpdateByQuery的使用</title>
<link href="https://idelin.github.io/2017/08/29/elasticsearch%20UpdateByQuery%E7%9A%84%E4%BD%BF%E7%94%A8/"/>
<id>https://idelin.github.io/2017/08/29/elasticsearch%20UpdateByQuery%E7%9A%84%E4%BD%BF%E7%94%A8/</id>
<published>2017-08-29T08:10:43.000Z</published>
<updated>2022-08-03T12:45:23.441Z</updated>
<content type="html"><![CDATA[<p>上一篇发了解决elasticsearch UpdateByQuery的问题，后来遇到好多小伙伴问我，为什么他使用不了UpdateByQuery，遂科普下。<br>在于你可能没有引入reindex包：</p><span id="more"></span><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs xml"><span class="hljs-tag"><<span class="hljs-name">dependency</span>></span><br><span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>org.elasticsearch.module<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span><br> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>reindex<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span><br> <span class="hljs-tag"><<span class="hljs-name">version</span>></span>2.4.3<span class="hljs-tag"></<span class="hljs-name">version</span>></span><br><span class="hljs-tag"></<span class="hljs-name">dependency</span>></span><br></code></pre></td></tr></table></figure><p>使用方法也很简单，举个栗子，这个栗子的作用是把所有满足条件的session_id记录都使用对应script更新：</p><figure class="highlight reasonml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><code class="hljs reasonml">public static void update<span class="hljs-constructor">ByQuery(String <span class="hljs-params">index</span>, String <span class="hljs-params">type</span>, Set<Map<String, Object>> <span class="hljs-params">docs</span>, String <span class="hljs-params">scriptStr</span>)</span> {<br> <span class="hljs-keyword">if</span> (docs<span class="hljs-operator"> == </span>null<span class="hljs-operator"> || </span>docs.is<span class="hljs-constructor">Empty()</span>){<br> <span class="hljs-module-access"><span class="hljs-module"><span class="hljs-identifier">LOG</span>.</span></span>info(<span class="hljs-string">"No data can updateByQuery to es! index:{}."</span>, index);<br> return;<br> }<br> UpdateByQueryRequestBuilder ubqrb = <span class="hljs-module-access"><span class="hljs-module"><span class="hljs-identifier">UpdateByQueryAction</span>.</span><span class="hljs-module"><span class="hljs-identifier">INSTANCE</span>.</span></span><span class="hljs-keyword">new</span><span class="hljs-constructor">RequestBuilder(<span class="hljs-params">client</span>)</span>;<br> <span class="hljs-keyword">for</span> (Map<String, Object> doc : docs) {<br> <span class="hljs-keyword">if</span> (doc==null<span class="hljs-operator"> || </span>doc.is<span class="hljs-constructor">Empty()</span>){<br> return;<br> }<br> Script script = <span class="hljs-keyword">new</span> <span class="hljs-constructor">Script(<span class="hljs-params">scriptStr</span>)</span>;<br> BulkIndexByScrollResponse scrollResponse =<br> ubqrb.source(index)<br> .script(script)<br> .filter(<span class="hljs-module-access"><span class="hljs-module"><span class="hljs-identifier">QueryBuilders</span>.</span></span>term<span class="hljs-constructor">Query(<span class="hljs-string">"session_id"</span>, <span class="hljs-params">doc</span>.<span class="hljs-params">get</span>(<span class="hljs-string">"orgin_session_id"</span>)</span>))<br> .abort<span class="hljs-constructor">OnVersionConflict(<span class="hljs-params">false</span>)</span>.get<span class="hljs-literal">()</span>;<br> <span class="hljs-keyword">for</span> (BulkItemResponse.Failure failure : scrollResponse.get<span class="hljs-constructor">IndexingFailures()</span>) {<br> <span class="hljs-module-access"><span class="hljs-module"><span class="hljs-identifier">LOG</span>.</span></span>error(failure.get<span class="hljs-constructor">Message()</span>);<br> }<br> }<br> }<br></code></pre></td></tr></table></figure>]]></content>
<summary type="html"><p>上一篇发了解决elasticsearch UpdateByQuery的问题，后来遇到好多小伙伴问我，为什么他使用不了UpdateByQuery，遂科普下。<br>在于你可能没有引入reindex包：</p></summary>
<category term="elasticsearch" scheme="https://idelin.github.io/categories/elasticsearch/"/>
<category term="elasticsearch" scheme="https://idelin.github.io/tags/elasticsearch/"/>
</entry>
<entry>
<title>es UpdateByQuery 出现 VersionConflictEngineException</title>
<link href="https://idelin.github.io/2017/08/25/es%20%20UpdateByQuery%20%E5%87%BA%E7%8E%B0%20VersionConflictEngineException/"/>
<id>https://idelin.github.io/2017/08/25/es%20%20UpdateByQuery%20%E5%87%BA%E7%8E%B0%20VersionConflictEngineException/</id>
<published>2017-08-25T07:05:08.000Z</published>
<updated>2022-08-03T12:45:23.441Z</updated>
<content type="html"><![CDATA[<p>问题出在多次更新统一doc出现版本不一致，抛出异常，修改不成功。<br>解决方法：<br>确认你的操作不会造成数据不一致的情况下，<br>UpdateByQueryRequestBuilder实现类增加执行方法abortOnVersionConflict(false)</p><span id="more"></span><p>like this</p><figure class="highlight reasonml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs reasonml">UpdateByQueryRequestBuilder ubqrb = <span class="hljs-module-access"><span class="hljs-module"><span class="hljs-identifier">UpdateByQueryAction</span>.</span><span class="hljs-module"><span class="hljs-identifier">INSTANCE</span>.</span></span><span class="hljs-keyword">new</span><span class="hljs-constructor">RequestBuilder(<span class="hljs-params">client</span>)</span>;<br>Script script = <span class="hljs-keyword">new</span> <span class="hljs-constructor">Script(<span class="hljs-params">scriptStr</span>)</span>;<br>BulkIndexByScrollResponse scrollResponse = ubqrb.source(index).script(script)<br>.filter(<span class="hljs-module-access"><span class="hljs-module"><span class="hljs-identifier">QueryBuilders</span>.</span></span>term<span class="hljs-constructor">Query(<span class="hljs-string">"xxx"</span>,<span class="hljs-params">xxx</span>)</span>))<br>.abort<span class="hljs-constructor">OnVersionConflict(<span class="hljs-params">false</span>)</span><br>.get<span class="hljs-literal">()</span>;<br></code></pre></td></tr></table></figure>]]></content>
<summary type="html"><p>问题出在多次更新统一doc出现版本不一致，抛出异常，修改不成功。<br>解决方法：<br>确认你的操作不会造成数据不一致的情况下，<br>UpdateByQueryRequestBuilder实现类增加执行方法abortOnVersionConflict(false)</p></summary>
<category term="elasticsearch" scheme="https://idelin.github.io/categories/elasticsearch/"/>
<category term="异常" scheme="https://idelin.github.io/tags/%E5%BC%82%E5%B8%B8/"/>
</entry>
<entry>
<title>disruptor高性能框架</title>
<link href="https://idelin.github.io/2017/08/25/disruptor%E9%AB%98%E6%80%A7%E8%83%BD%E6%A1%86%E6%9E%B6/"/>
<id>https://idelin.github.io/2017/08/25/disruptor%E9%AB%98%E6%80%A7%E8%83%BD%E6%A1%86%E6%9E%B6/</id>
<published>2017-08-25T06:58:53.000Z</published>
<updated>2022-08-03T12:45:23.441Z</updated>
<content type="html"><![CDATA[<p>最近项目里需要写模型，然后数据是从es取，不同人负责写不同模型，这里遇到一个问题，就是数据的读其实公用，没必要各自取一次数据浪费带宽、内存、cpu…遂想是否可以提取公用部分做数据读，下发给各个模型，然后开发老大说用disruptor试试，调研了下确实可行，就稍加记录下。</p><span id="more"></span><p>disruptor项目地址：<a href="https://github.com/LMAX-Exchange/disruptor">https://github.com/LMAX-Exchange/disruptor</a><br>简单来说disruptor是一个高性能的异步的消息处理框架。</p><p>disruptor最核心的概念是ringbuffer（环形缓冲队列），下图解释所有↓<br><img src="http://img.blog.csdn.net/20170828102904174?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvamFja2xpbjkyOQ==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast" alt="这里写图片描述"><br>ringbuffer就如上一个环形队列，生产者往队列写数据，消费者读取数据处理，如果生产者追到消费者，那么生产者会停止生产，阻塞，等待它完成；同样的，消费者消费完了所有数据，消费者还未来得及消费，前者一样阻塞等待。</p><p>disruptor采用的就是ringbuffer数据结构，之所以采用这种结构，官网是这样描述的：<br><em>之所以采用ringbuffer，是因为它在可靠消息传递方面有很好的性能。这就够了，不过它还有一些其他的优点。<br>首先，因为它是数组，所以要比链表快，而且有一个容易预料到的数据访问的内存地址。这是对CPU缓存友好的—也就是说，在硬件级别，数组中的元素是会被预加载的，因此在ringbuffer当中，cpu无需时不时去主存加载数组中的下一个元素。（校对注：因为只要一个元素被加载到缓存行，其他相邻的几个元素也会被加载进同一个缓存行）<br>其次，你可以为数组预先分配内存，使得数组对象一直存在（除非程序终止）。这就意味着不需要花大量的时间用于垃圾回收。此外，不像链表那样，需要为每一个添加到其上面的对象创造节点对象—对应的，当删除节点时，需要执行相应的内存清理操作。</em></p><p>其他概念就不详细描述了，直接照例代码上。<br>引入包</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs xml"><span class="hljs-tag"><<span class="hljs-name">dependency</span>></span><br> <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>com.lmax<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span><br> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>disruptor<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span><br> <span class="hljs-tag"><<span class="hljs-name">version</span>></span>3.3.6<span class="hljs-tag"></<span class="hljs-name">version</span>></span><br><span class="hljs-tag"></<span class="hljs-name">dependency</span>></span><br></code></pre></td></tr></table></figure><p>Event类获取用于填充数据，data类型按需定义。</p><figure class="highlight typescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs typescript"><span class="hljs-keyword">public</span> <span class="hljs-keyword">class</span> <span class="hljs-title class_">MessageEvent</span> {<br> <span class="hljs-keyword">private</span> <span class="hljs-title class_">List</span><<span class="hljs-title class_">Map</span><<span class="hljs-title class_">String</span>, <span class="hljs-title class_">Object</span>>> data;<br><br> <span class="hljs-keyword">public</span> <span class="hljs-built_in">void</span> <span class="hljs-title function_">setData</span>(<span class="hljs-params">List<<span class="hljs-built_in">Map</span><<span class="hljs-built_in">String</span>, <span class="hljs-built_in">Object</span>>> data</span>) {<br> <span class="hljs-variable language_">this</span>.<span class="hljs-property">data</span> = data;<br> }<br><br> <span class="hljs-keyword">public</span> <span class="hljs-title class_">List</span><<span class="hljs-title class_">Map</span><<span class="hljs-title class_">String</span>, <span class="hljs-title class_">Object</span>>> <span class="hljs-title function_">getData</span>(<span class="hljs-params"></span>) {<br> <span class="hljs-keyword">return</span> data;<br> }<br>}<br></code></pre></td></tr></table></figure><p>数据工厂类，实现EventFactory接口</p><figure class="highlight haxe"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs haxe"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">MessageEventFactory</span> <span class="hljs-keyword"><span class="hljs-keyword">implements</span> <span class="hljs-type">EventFactory</span></span><<span class="hljs-title">MessageEvent</span>> </span>{<br> @Override<br> <span class="hljs-keyword">public</span> MessageEvent <span class="hljs-keyword">new</span><span class="hljs-type">Instance</span>() {<br> <span class="hljs-keyword">return</span> <span class="hljs-keyword">new</span> <span class="hljs-type">MessageEvent</span>();<br> }<br>}<br></code></pre></td></tr></table></figure><p>生产者类</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs csharp"><span class="hljs-keyword">public</span> <span class="hljs-keyword">class</span> <span class="hljs-title">MessageEventProducer</span> {<br> <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> Logger logger = LoggerFactory.getLogger(MessageEventProducer.<span class="hljs-keyword">class</span>);<br><br> <span class="hljs-keyword">private</span> final RingBuffer<MessageEvent> ringBuffer;<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-title">MessageEventProducer</span>(<span class="hljs-params">RingBuffer<MessageEvent> ringBuffer</span>)</span> {<br> <span class="hljs-keyword">this</span>.ringBuffer = ringBuffer;<br> }<br><br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">onData</span>(<span class="hljs-params">List<Map<String, Object>> datas</span>)</span> {<br> <span class="hljs-built_in">long</span> sequence = ringBuffer.next(); <br> <span class="hljs-keyword">try</span> {<br> MessageEvent <span class="hljs-keyword">event</span> = ringBuffer.<span class="hljs-keyword">get</span>(sequence);<br> <span class="hljs-keyword">event</span>.setData(datas);<br> } <span class="hljs-keyword">finally</span> {<br> ringBuffer.publish(sequence);<br> }<br> }<br>}<br></code></pre></td></tr></table></figure><p>消费者类，handler，具体逻辑数据类，实现EventHandler接口即可，实现其onEvent方法，可通过event.getData()拿到数据。</p><figure class="highlight csharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs csharp"><span class="hljs-keyword">public</span> <span class="hljs-keyword">class</span> <span class="hljs-title">MessageEventHandler</span> <span class="hljs-title">implements</span> <span class="hljs-title">EventHandler</span><<span class="hljs-title">MessageEvent</span>><br>{<br> <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">onEvent</span>(<span class="hljs-params">MessageEvent <span class="hljs-keyword">event</span>, <span class="hljs-built_in">long</span> sequence, boolean endOfBatch</span>)</span><br> {<br> System.<span class="hljs-keyword">out</span>.println(<span class="hljs-string">"Event: "</span> + <span class="hljs-keyword">event</span>);<br> }<br>}<br></code></pre></td></tr></table></figure><p>主类，启动类</p><figure class="highlight reasonml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><code class="hljs reasonml">public <span class="hljs-keyword">class</span> MessageEventMain extends Thread {<br> <span class="hljs-keyword">private</span> static Logger logger = <span class="hljs-module-access"><span class="hljs-module"><span class="hljs-identifier">LoggerFactory</span>.</span></span>get<span class="hljs-constructor">Logger(MessageEventMain.<span class="hljs-params">class</span>)</span>;<br><br> @Override<br> public void run<span class="hljs-literal">()</span> {<br> ThreadFactory threadFactory = <span class="hljs-module-access"><span class="hljs-module"><span class="hljs-identifier">Executors</span>.</span></span>default<span class="hljs-constructor">ThreadFactory()</span>;<br><br> <span class="hljs-comment">// 创建消息工厂类</span><br> MessageEventFactory factory = <span class="hljs-keyword">new</span> <span class="hljs-constructor">MessageEventFactory()</span>;<br><br> <span class="hljs-comment">// ringbuffer size，一般为2的指数倍</span><br> <span class="hljs-built_in">int</span> bufferSize = <span class="hljs-number">1024</span>;<br><br> <span class="hljs-comment">// 定义 Disruptor</span><br> Disruptor<MessageEvent> disruptor =<br> <span class="hljs-keyword">new</span> Disruptor<>(factory, bufferSize, threadFactory, ProducerType.SINGLE, <span class="hljs-keyword">new</span> <span class="hljs-constructor">BlockingWaitStrategy()</span>);<br><br> <span class="hljs-comment">// 关联消费者</span><br> disruptor.handle<span class="hljs-constructor">EventsWith(<span class="hljs-params">new</span> MessageHandler()</span>);<br> disruptor.handle<span class="hljs-constructor">EventsWith(<span class="hljs-params">new</span> <span class="hljs-params">xxxxHandler</span>()</span>);<br><br> <span class="hljs-comment">// 启动disruptor框架</span><br> disruptor.start<span class="hljs-literal">()</span>;<br><br> <span class="hljs-comment">// 获得ringbuffer用于消息发布</span><br> RingBuffer<MessageEvent> ringBuffer = disruptor.get<span class="hljs-constructor">RingBuffer()</span>;<br> MessageEventProducer producer = <span class="hljs-keyword">new</span> <span class="hljs-constructor">MessageEventProducer(<span class="hljs-params">ringBuffer</span>)</span>;<br> <span class="hljs-keyword">while</span> (<span class="hljs-literal">true</span>) {<br> <span class="hljs-keyword">try</span> {<br> List <span class="hljs-built_in">list</span> = get<span class="hljs-constructor">Data()</span>;<span class="hljs-comment">//自己的取数逻辑</span><br> producer.on<span class="hljs-constructor">Data(<span class="hljs-params">list</span>)</span>;<br> <span class="hljs-module-access"><span class="hljs-module"><span class="hljs-identifier">Thread</span>.</span></span>sleep(<span class="hljs-number">60</span><span class="hljs-operator"> * </span><span class="hljs-number">10</span><span class="hljs-operator"> * </span><span class="hljs-number">1000</span>);<br> } catch (Exception e) {<br> e.print<span class="hljs-constructor">StackTrace()</span>;<br> }<br> }<br> }<br> }<br></code></pre></td></tr></table></figure><p>打完收工<br>另外讲几点：<br>1.handler会在producer.onData后对框架通知消费数据；<br>2.消费数据最好按最简单类型处理，性能更好，我上面用list实在是业务框定，所有后续造成了一个严重问题；<br>3.handler消费完对应ringbuffer数据并不会马上被清理，是直到生产者填充环形队列再到这个位置才会覆盖数据，所以问题2就是这样出现，如果定义复杂类型，一格存储大量数据，会操作jvm内存不够，程序运行一段数据会出现大量full fc或者oom，解决办法是，要么换成一条条数据发送，要么减小ringbuffer size，我这里业务限定，只能改小了ringbuffer size；<br>4.框架有阻塞等待机制，具体使用算法可以自行查，也就是说，如果某一个消费者太慢，生产者太快，到生产者追上消费者的那一格，生产者会阻塞，直到最慢的那个消费者消费完成，同样的，如果消费者很快，消费到了最新的一份数据，一样会阻塞等待生成者再次生成数据。所以这里要注意，如果某一个handler一直太慢，你可能要处理下这个handler效率，必要时撤掉，起码不能影响其他handler正常运行。</p>]]></content>
<summary type="html"><p>最近项目里需要写模型，然后数据是从es取，不同人负责写不同模型，这里遇到一个问题，就是数据的读其实公用，没必要各自取一次数据浪费带宽、内存、cpu…遂想是否可以提取公用部分做数据读，下发给各个模型，然后开发老大说用disruptor试试，调研了下确实可行，就稍加记录下。</p></summary>
<category term="java" scheme="https://idelin.github.io/categories/java/"/>
<category term="高性能" scheme="https://idelin.github.io/tags/%E9%AB%98%E6%80%A7%E8%83%BD/"/>
</entry>
<entry>
<title>【Storm初探】wordcount demo</title>
<link href="https://idelin.github.io/2017/06/26/%E3%80%90Storm%E5%88%9D%E6%8E%A2%E3%80%91wordcount%20demo/"/>
<id>https://idelin.github.io/2017/06/26/%E3%80%90Storm%E5%88%9D%E6%8E%A2%E3%80%91wordcount%20demo/</id>
<published>2017-06-26T08:33:48.000Z</published>
<updated>2022-08-03T12:45:23.189Z</updated>
<content type="html"><![CDATA[<p>目前工作下需要接触storm，遂写个demo练练手。<br>代码主要借鉴的是storm官方的github项目：<a href="https://github.com/apache/storm">https://github.com/apache/storm</a></p><span id="more"></span><p>略做修改。<br>1.pom.xml引入</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs xml"><span class="hljs-tag"><<span class="hljs-name">dependency</span>></span><br> <span class="hljs-tag"><<span class="hljs-name">groupId</span>></span>org.apache.storm<span class="hljs-tag"></<span class="hljs-name">groupId</span>></span><br> <span class="hljs-tag"><<span class="hljs-name">artifactId</span>></span>storm-core<span class="hljs-tag"></<span class="hljs-name">artifactId</span>></span><br> <span class="hljs-tag"><<span class="hljs-name">type</span>></span>jar<span class="hljs-tag"></<span class="hljs-name">type</span>></span><br> <span class="hljs-tag"><<span class="hljs-name">version</span>></span>1.0.0<span class="hljs-tag"></<span class="hljs-name">version</span>></span><br><span class="hljs-tag"></<span class="hljs-name">dependency</span>></span><br></code></pre></td></tr></table></figure><p>2.Spout<br>Spout是strom里产生元数据（tuple）流的类，通常情况下Spout会读取外部数据，然后转换为Topology内部的元数据（tuple），主动角色。<br>nextTuple方法不断执行，数据在此方法内产生。<br>此处用在数组中随机获得句子作为元数据流做示例。</p><figure class="highlight typescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><code class="hljs typescript"><span class="hljs-keyword">public</span> <span class="hljs-keyword">class</span> <span class="hljs-title class_">RandomSentenceSpout</span> <span class="hljs-keyword">extends</span> <span class="hljs-title class_ inherited__">BaseRichSpout</span> {<br> <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> final <span class="hljs-title class_">Logger</span> <span class="hljs-variable constant_">LOG</span> = <span class="hljs-title class_">LoggerFactory</span>.<span class="hljs-title function_">getLogger</span>(<span class="hljs-title class_">RandomSentenceSpout</span>.<span class="hljs-property">class</span>);<br><br> <span class="hljs-title class_">SpoutOutputCollector</span> _collector;<br> <span class="hljs-title class_">Random</span> _rand;<br><br> <span class="hljs-meta">@Override</span><br> <span class="hljs-keyword">public</span> <span class="hljs-built_in">void</span> <span class="hljs-title function_">open</span>(<span class="hljs-params"><span class="hljs-built_in">Map</span> conf, TopologyContext context, SpoutOutputCollector collector</span>) {<br> _collector = collector;<br> _rand = <span class="hljs-keyword">new</span> <span class="hljs-title class_">Random</span>();<br> }<br><br> <span class="hljs-meta">@Override</span><br> <span class="hljs-keyword">public</span> <span class="hljs-built_in">void</span> <span class="hljs-title function_">nextTuple</span>(<span class="hljs-params"></span>) {<br> <span class="hljs-title class_">Utils</span>.<span class="hljs-title function_">sleep</span>(<span class="hljs-number">100</span>);<br> <span class="hljs-title class_">String</span>[] sentences = <span class="hljs-keyword">new</span> <span class="hljs-title class_">String</span>[]{<span class="hljs-title function_">sentence</span>(<span class="hljs-string">"the cow jumped over the moon"</span>), <span class="hljs-title function_">sentence</span>(<span class="hljs-string">"an apple a day keeps the doctor away"</span>),<br> <span class="hljs-title function_">sentence</span>(<span class="hljs-string">"four score and seven years ago"</span>), <span class="hljs-title function_">sentence</span>(<span class="hljs-string">"snow white and the seven dwarfs"</span>), <span class="hljs-title function_">sentence</span>(<span class="hljs-string">"i am at two with nature"</span>)};<br> final <span class="hljs-title class_">String</span> sentence = sentences[_rand.<span class="hljs-title function_">nextInt</span>(sentences.<span class="hljs-property">length</span>)];<br><br> <span class="hljs-variable constant_">LOG</span>.<span class="hljs-title function_">info</span>(<span class="hljs-string">"Emitting tuple: {}"</span>, sentence);<br><br> _collector.<span class="hljs-title function_">emit</span>(<span class="hljs-keyword">new</span> <span class="hljs-title class_">Values</span>(sentence));<br> }<br><br> <span class="hljs-keyword">protected</span> <span class="hljs-title class_">String</span> <span class="hljs-title function_">sentence</span>(<span class="hljs-params"><span class="hljs-built_in">String</span> input</span>) {<br> <span class="hljs-keyword">return</span> input;<br> }<br><br> <span class="hljs-meta">@Override</span><br> <span class="hljs-keyword">public</span> <span class="hljs-built_in">void</span> <span class="hljs-title function_">ack</span>(<span class="hljs-params"><span class="hljs-built_in">Object</span> id</span>) {<br> }<br><br> <span class="hljs-meta">@Override</span><br> <span class="hljs-keyword">public</span> <span class="hljs-built_in">void</span> <span class="hljs-title function_">fail</span>(<span class="hljs-params"><span class="hljs-built_in">Object</span> id</span>) {<br> }<br><br> <span class="hljs-meta">@Override</span><br> <span class="hljs-keyword">public</span> <span class="hljs-built_in">void</span> <span class="hljs-title function_">declareOutputFields</span>(<span class="hljs-params">OutputFieldsDeclarer declarer</span>) {<br> declarer.<span class="hljs-title function_">declare</span>(<span class="hljs-keyword">new</span> <span class="hljs-title class_">Fields</span>(<span class="hljs-string">"word"</span>));<br> }<br>}<br></code></pre></td></tr></table></figure><p>3.Bolt<br>在一个Topology中接收数据并处理的组件，被动角色。<br>execute函数处理接收到的数据，并生成新元组转发（emit）出去。<br>此处定义了两个bolt，第一个blot先把得到Sentence分隔成word，然后下一个bolt对word计数。</p><figure class="highlight arduino"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><code class="hljs arduino"><span class="hljs-keyword">public</span> <span class="hljs-keyword">class</span> <span class="hljs-title class_">SplitSentenceBlot</span> extends BaseBasicBolt {<br><br> @<span class="hljs-function">Override</span><br><span class="hljs-function"> <span class="hljs-keyword">public</span> <span class="hljs-type">void</span> <span class="hljs-title">execute</span><span class="hljs-params">(Tuple tuple, BasicOutputCollector collector)</span> </span>{<br> <span class="hljs-type">String</span> <span class="hljs-type">word</span> = tuple.<span class="hljs-built_in">getString</span>(<span class="hljs-number">0</span>);<br> <span class="hljs-type">String</span>[] words = <span class="hljs-type">word</span>.<span class="hljs-built_in">split</span>(<span class="hljs-string">" "</span>);<br> <span class="hljs-keyword">for</span> (<span class="hljs-type">String</span> s : words) {<br> System.out.<span class="hljs-built_in">println</span>(<span class="hljs-string">"=========="</span> + s);<br> collector.<span class="hljs-built_in">emit</span>(<span class="hljs-keyword">new</span> <span class="hljs-built_in">Values</span>(s));<br> }<br> }<br><br> @<span class="hljs-function">Override</span><br><span class="hljs-function"> <span class="hljs-keyword">public</span> <span class="hljs-type">void</span> <span class="hljs-title">declareOutputFields</span><span class="hljs-params">(OutputFieldsDeclarer declarer)</span> </span>{<br> declarer.<span class="hljs-built_in">declare</span>(<span class="hljs-keyword">new</span> <span class="hljs-built_in">Fields</span>(<span class="hljs-string">"word"</span>));<br> }<br>}<br></code></pre></td></tr></table></figure><figure class="highlight arduino"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><code class="hljs arduino"><span class="hljs-keyword">public</span> <span class="hljs-keyword">class</span> <span class="hljs-title class_">WordCountBlot</span> extends BaseBasicBolt {<br> Map<<span class="hljs-type">String</span>, Integer> counts = <span class="hljs-keyword">new</span> <span class="hljs-built_in">HashMap</span><<span class="hljs-type">String</span>, Integer>();<br><br> @<span class="hljs-function">Override</span><br><span class="hljs-function"> <span class="hljs-keyword">public</span> <span class="hljs-type">void</span> <span class="hljs-title">execute</span><span class="hljs-params">(Tuple tuple, BasicOutputCollector collector)</span> </span>{<br> <span class="hljs-type">String</span> <span class="hljs-type">word</span> = tuple.<span class="hljs-built_in">getString</span>(<span class="hljs-number">0</span>);<br> Integer count = counts.<span class="hljs-built_in">get</span>(<span class="hljs-type">word</span>);<br> <span class="hljs-keyword">if</span> (count == null)<br> count = <span class="hljs-number">0</span>;<br> count++;<br> counts.<span class="hljs-built_in">put</span>(<span class="hljs-type">word</span>, count);<br> System.out.<span class="hljs-built_in">println</span>(<span class="hljs-string">"==============["</span> + <span class="hljs-type">word</span> + <span class="hljs-string">"]:"</span> + count);<br> collector.<span class="hljs-built_in">emit</span>(<span class="hljs-keyword">new</span> <span class="hljs-built_in">Values</span>(<span class="hljs-type">word</span>, count));<br> }<br><br> @<span class="hljs-function">Override</span><br><span class="hljs-function"> <span class="hljs-keyword">public</span> <span class="hljs-type">void</span> <span class="hljs-title">declareOutputFields</span><span class="hljs-params">(OutputFieldsDeclarer declarer)</span> </span>{<br> declarer.<span class="hljs-built_in">declare</span>(<span class="hljs-keyword">new</span> <span class="hljs-built_in">Fields</span>(<span class="hljs-string">"word"</span>, <span class="hljs-string">"count"</span>));<br> }<br>}<br></code></pre></td></tr></table></figure><p>4.Topology<br>拓扑，storm里的关键性概念，本质上就是在storm里运行的一个实时应用程序，也就是此处要定义的主类。<br>具体看注释即可…</p><figure class="highlight reasonml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><code class="hljs reasonml">public <span class="hljs-keyword">class</span> SimpleTopology {<br> public static void main(String<span class="hljs-literal">[]</span> args) throws Exception{<br> <span class="hljs-comment">// 实例化TopologyBuilder类。</span><br> TopologyBuilder topologyBuilder = <span class="hljs-keyword">new</span> <span class="hljs-constructor">TopologyBuilder()</span>;<br> <span class="hljs-comment">// 设置喷发节点并分配并发数，该并发数将会控制该对象在集群中的线程数。</span><br> topologyBuilder.set<span class="hljs-constructor">Spout(<span class="hljs-string">"spout"</span>, <span class="hljs-params">new</span> RandomSentenceSpout()</span>, <span class="hljs-number">1</span>);<br> <span class="hljs-comment">// 设置数据处理节点并分配并发数。指定该节点接收喷发节点的策略为随机方式。</span><br> topologyBuilder.set<span class="hljs-constructor">Bolt(<span class="hljs-string">"split"</span>, <span class="hljs-params">new</span> SplitSentenceBlot()</span>, <span class="hljs-number">3</span>).shuffle<span class="hljs-constructor">Grouping(<span class="hljs-string">"spout"</span>)</span>;<br> topologyBuilder.set<span class="hljs-constructor">Bolt(<span class="hljs-string">"count"</span>, <span class="hljs-params">new</span> WordCountBlot()</span>, <span class="hljs-number">3</span>).fields<span class="hljs-constructor">Grouping(<span class="hljs-string">"split"</span>, <span class="hljs-params">new</span> Fields(<span class="hljs-string">"word"</span>)</span>);<br> String topologyName = <span class="hljs-string">"word-count"</span>;<br><br> Config config = <span class="hljs-keyword">new</span> <span class="hljs-constructor">Config()</span>;<br><span class="hljs-comment">// config.setDebug(true);</span><br> <span class="hljs-keyword">if</span> (args != null<span class="hljs-operator"> && </span>args.length > <span class="hljs-number">0</span>) {<br> config.set<span class="hljs-constructor">NumWorkers(1)</span>;<br> <span class="hljs-module-access"><span class="hljs-module"><span class="hljs-identifier">StormSubmitter</span>.</span></span>submit<span class="hljs-constructor">Topology(<span class="hljs-params">args</span>[0], <span class="hljs-params">config</span>, <span class="hljs-params">topologyBuilder</span>.<span class="hljs-params">createTopology</span>()</span>);<br> } <span class="hljs-keyword">else</span> {<br> <span class="hljs-comment">// 这里是本地模式下运行的启动代码。</span><br> config.set<span class="hljs-constructor">MaxTaskParallelism(1)</span>;<br> LocalCluster cluster = <span class="hljs-keyword">new</span> <span class="hljs-constructor">LocalCluster()</span>;<br> cluster.submit<span class="hljs-constructor">Topology(<span class="hljs-params">topologyName</span>, <span class="hljs-params">config</span>, <span class="hljs-params">topologyBuilder</span>.<span class="hljs-params">createTopology</span>()</span>);<br> }<br> }<br>}<br></code></pre></td></tr></table></figure>]]></content>
<summary type="html"><p>目前工作下需要接触storm，遂写个demo练练手。<br>代码主要借鉴的是storm官方的github项目：<a href="https://github.com/apache/storm">https://github.com/apache/storm</a></p></summary>
<category term="storm" scheme="https://idelin.github.io/categories/storm/"/>
<category term="storm" scheme="https://idelin.github.io/tags/storm/"/>
</entry>
</feed>