Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

离线导入3亿带索引关系边报bad_alloc #406

Closed
yzjzgr opened this issue Feb 1, 2024 · 2 comments
Closed

离线导入3亿带索引关系边报bad_alloc #406

yzjzgr opened this issue Feb 1, 2024 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@yzjzgr
Copy link

yzjzgr commented Feb 1, 2024

Environment:

OS: [centos 7]
CPU: [Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz 8 cores]
MEM: 32G
TuGraph-DB Version [e.g. 4.1.0]
standalone

Describe the bug
图schema:
{
"schema": [
{
"label": "entity",
"type": "VERTEX",
"properties": [
{ "name": "primary_key", "type": "STRING" },
{ "name": "entity_type", "type": "STRING" },
{ "name": "entity_properties", "type": "STRING","optional": true }
],
"primary": "primary_key"
},
{
"label": "relation",
"type": "EDGE",
"properties": [
{ "name": "relation_type", "type": "STRING", "optional": false },
{ "name": "relation_unique_id", "type": "STRING", "optional": false, "index":true, "unique":false },
{ "name": "relation_properties", "type": "STRING", "optional": true }
],
"constraints": [["entity", "entity"]]
}
]
}

复现步骤:

  1. 向以上schema图离线写入1.5亿顶点3亿关系边
    离线导入过程报bad_alloc。
    另外将边属性relation_unique_id改为无索引,数据可以被正常导入。
    也希望关系边属性有唯一索引的场景下,支持离线导入
@qishipengqsp qishipengqsp added the bug Something isn't working label Feb 1, 2024
@spasserby spasserby self-assigned this Feb 21, 2024
@ljcui
Copy link
Collaborator

ljcui commented Mar 8, 2024

可以测试下在边的json schema里面加上 "detach_property": true 配置项,这个会优化加索引的速度和内存消耗,类似下面

{
	"label": "relation",
	"type": "EDGE",
	"detach_property": true,
	"properties": [{
			"name": "relation_type",
			"type": "STRING",
			"optional": false
		},
		{
			"name": "relation_unique_id",
			"type": "STRING",
			"optional": false,
			"index": true,
			"unique": false
		},
		{
			"name": "relation_properties",
			"type": "STRING",
			"optional": true
		}
	],
	"constraints": [
		["entity", "entity"]
	]
}

@spasserby
Copy link
Collaborator

PR: #430

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants