Skip to content

Commit

Permalink
refine ptn script download
Browse files Browse the repository at this point in the history
  • Loading branch information
yonghua-sun committed Dec 12, 2024
1 parent 027d545 commit f8fe789
Show file tree
Hide file tree
Showing 6 changed files with 69 additions and 9 deletions.
1 change: 1 addition & 0 deletions .vitepress/siderbars/getOscompatibleDemo.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ function getSidebar() {
text: 'ES兼容',
items: [
{ text: 'Kibana Dashboard迁移', link: '/oscompatibledemo/kibana_dashboard_transfer' },
{ text: 'Kibana Index Pattern批量删除', link: '/oscompatibledemo/kibana_pattern_batch_delete' },
{ text: '最佳实践', link: '/oscompatibledemo/es_best_practice' },
],
},
Expand Down
22 changes: 13 additions & 9 deletions src/oscompatibledemo/kibana_dashboard_transfer.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@

本文介绍在数据迁移到SLS后,如何将原来Kibana中的dashboard迁移到SLS对接的Kibana中

# 操作思路
# 思路

![](/img/oscompatibledemo/dashboard10.png)


* 从旧的Kibana导出dashboard的export.ndjson文件
* 新的Kibana中准备好index pattern
* 把export.ndjson中的旧的index pattern id替换成新的Kibana index patter Id
* 完成替换后,把export.ndjson导入到新的kibana
* 把export.ndjson中的旧的index Pattern id替换成新的Kibana Index Patter Id
* 完成替换后,把export.ndjson导入到新的Kibana中



# Dashboard迁移演示
Expand Down Expand Up @@ -64,7 +64,7 @@ POST people/_bulk

* 查看老的es中的Index Pattern的ID

准备好 kibana_config_1.json,然后使用 [ptn_list.py 点击下载](ptn_list.py) 来list Index Pattern
准备好 kibana_config_1.json,然后使用 [ptn_list.py](ptn_list.py.md){target="_blank"} 来list Index Pattern
```
{
"url" : "http://xxx:5601",
Expand All @@ -75,7 +75,7 @@ POST people/_bulk
```
执行
```
/tmp python ptn_list.py kibana_config_1.json
➜ python ptn_list.py kibana_config_1.json
f06fc2b0-b82d-11ef-88c6-15adf26175c7 people
```

Expand All @@ -87,12 +87,12 @@ f06fc2b0-b82d-11ef-88c6-15adf26175c7 这个就是旧的people这个index pattern

* 在SLS对接的Kibana中找到新的Index Pattern的Id

同样使用 [ptn_list.py 点击下载](ptn_list.py) 来list 新的Kibana中Index Pattern的ID
同样使用 [ptn_list.py](ptn_list.py.md){target="_blank"} 来list 新的Kibana中Index Pattern的ID

```
# 准备好kibana_config_2.json
/tmp python ptn_list.py kibana_config_2.json
➜ python ptn_list.py kibana_config_2.json
ef710470-b83a-11ef-bb2b-ad198b7b763d etl.people
```

Expand All @@ -116,4 +116,8 @@ sed -i 's/f06fc2b0-b82d-11ef-88c6-15adf26175c7/ef710470-b83a-11ef-bb2b-ad198b7b7
![](/img/oscompatibledemo/dashboard8.png)

打开新的dashboard是否符合预期
![](/img/oscompatibledemo/dashboard9.png)
![](/img/oscompatibledemo/dashboard9.png)


### 注意事项
* 老的ES中的字段 和 SLS中写入的字段需要一致,不然可能导致Dashboard迁移后打开报错(比如报字段不存在)
48 changes: 48 additions & 0 deletions src/oscompatibledemo/kibana_pattern_batch_delete.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# 概要

[SLS对接Kibana](https://help.aliyun.com/zh/sls/user-guide/use-kibana-to-access-the-elasticsearch-compatible-api-of-log-service) 方案中,会自动创建Index Pattern。可能一开始没有设置好logstore过滤条件,导致创建了一些不需要的logstore pattern,这时候 可以通过脚本来批量删除一些不需要的Index Pattern


# 操作步骤

## List出需要删除的Index Pattern

准备好 kibana_config.json,然后使用 [ptn_list.py](ptn_list.py.md){target="_blank"} 来list Index Pattern
```
{
"url" : "http://xxx:5601",
"user" : "elastic",
"password" : "",
"space" : "default"
}
```
执行
```
➜ python ptn_list.py kibana_config.json > /tmp/ptnlist.txt
```

/tmp/ptnlist.txt可能是长这样的

```
54c0d6c0-b83b-11ef-88c6-15adf26175c7 etl-dev.batch_test52
54266b80-b83b-11ef-88c6-15adf26175c7 etl-dev.batch_test51
52f369c0-b83b-11ef-88c6-15adf26175c7 etl-dev.batch_test49
538ceaa0-b83b-11ef-88c6-15adf26175c7 etl-dev.batch_test50
```

编辑这个/tmp/ptnlist.txt文件,只留下需要删除的index pattern 的行

## 执行Index Pattern删除

**注意,Index Pattern删除后对应的Dashboard等都会无法使用,请确保这些Index Pattern是没有用的。**

使用 [ptn_delete.py](ptn_delete.py.md){target="_blank"} 删除 Index Pattern


```
➜ cat /tmp/ptnlist.txt # 再看一眼 /tmp/ptnlist.txt,确认都是要删除的Index Pattern
# 执行删除
➜ python ptn_delete.py kibana_config.json /tmp/ptnlist.txt
```
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ptn_delete.py
```python
import sys
import json
import requests
Expand Down Expand Up @@ -75,3 +77,4 @@ def print_help():
for indexPatternId in indexPatternIds:
#print_with_time("start to delete index pattern %s" % indexPatternId)
deleteKibanaIndexPattern(kibanaSpace, indexPatternId)
```
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
ptn_list.py
```python
import sys
import json
import requests
Expand Down Expand Up @@ -91,3 +93,5 @@ def print_help():

for name, id in existsPatterns.items():
print("%s\t%s" %(id, name))

```
Binary file modified src/public/img/oscompatibledemo/dashboard10.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit f8fe789

Please sign in to comment.