Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a stand-alone load mode to DM. (WIP) #11738

Closed
wants to merge 4 commits into from
Closed

Add a stand-alone load mode to DM. (WIP) #11738

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
73620d2
Primary work to add a stand-alone load mode to DM. Load&Sync mode might
OliverS929 Nov 12, 2024
a323638
Fix format.
OliverS929 Nov 13, 2024
83ddceb
Update error.txt
OliverS929 Nov 13, 2024
e0d9a1c
Add load mode to openapi.
OliverS929 Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dm/_utils/terror_gen/errors_release.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ ErrConfigLoaderCfgConflict,[code=20016:class=config:scope=internal:level=medium]
ErrConfigSyncerCfgConflict,[code=20017:class=config:scope=internal:level=medium], "Message: syncer-config-name and syncer should only specify one, Workaround: Please check the `syncer-config-name` and `syncer` config in task configuration file."
ErrConfigReadCfgFromFile,[code=20018:class=config:scope=internal:level=medium], "Message: read config file %v"
ErrConfigNeedUniqueTaskName,[code=20019:class=config:scope=internal:level=medium], "Message: must specify a unique task name, Workaround: Please check the `name` config in task configuration file."
ErrConfigInvalidTaskMode,[code=20020:class=config:scope=internal:level=medium], "Message: please specify right task-mode, support `full`, `incremental`, `all`, Workaround: Please check the `task-mode` config in task configuration file."
ErrConfigInvalidTaskMode,[code=20020:class=config:scope=internal:level=medium], "Message: please specify right task-mode, support `full`, `incremental`, `all`, `dump`, `load`, Workaround: Please check the `task-mode` config in task configuration file."
ErrConfigNeedTargetDB,[code=20021:class=config:scope=internal:level=medium], "Message: must specify target-database, Workaround: Please check the `target-database` config in task configuration file."
ErrConfigMetadataNotSet,[code=20022:class=config:scope=internal:level=medium], "Message: mysql-instance(%s) must set meta for task-mode %s, Workaround: Please check the `meta` config in task configuration file."
ErrConfigRouteRuleNotFound,[code=20023:class=config:scope=internal:level=medium], "Message: mysql-instance(%d)'s route-rules %s not exist in routes, Workaround: Please check the `route-rules` config in task configuration file."
Expand Down
2 changes: 1 addition & 1 deletion dm/config/helper.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ func HasDump(taskMode string) bool {
// HasLoad returns true if taskMode contains load unit.
func HasLoad(taskMode string) bool {
switch taskMode {
case ModeAll, ModeFull, ModeLoadSync:
case ModeAll, ModeFull, ModeLoad, ModeLoadSync:
return true
default:
return false
Expand Down
4 changes: 4 additions & 0 deletions dm/config/helper_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ func TestTaskModeHasFunction(t *testing.T) {
require.False(t, HasLoad(ModeDump))
require.False(t, HasSync(ModeDump))

require.False(t, HasDump(ModeLoad))
require.True(t, HasLoad(ModeLoad))
require.False(t, HasSync(ModeLoad))

require.False(t, HasDump(ModeLoadSync))
require.True(t, HasLoad(ModeLoadSync))
require.True(t, HasSync(ModeLoadSync))
Expand Down
2 changes: 2 additions & 0 deletions dm/config/subtask.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ const (
ModeFull = "full"
ModeIncrement = "incremental"
ModeDump = "dump"
ModeLoad = "load"
ModeLoadSync = "load&sync"

DefaultShadowTableRules = "^_(.+)_(?:new|gho)$"
Expand Down Expand Up @@ -330,6 +331,7 @@ func (c *SubTaskConfig) Adjust(verifyDecryptPassword bool) error {
}

// adjust dir, no need to do for load&sync mode because it needs its own s3 repository
// still use dir for standalone load mode (different from the behavior of load&sync mode)
Comment on lines 333 to +334
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// adjust dir, no need to do for load&sync mode because it needs its own s3 repository
// still use dir for standalone load mode (different from the behavior of load&sync mode)
// adjust dir for modes with load unit except load&sync mode because it needs its own s3 repository

BTW because I don't know how does the new "load" mode is used, I'm not sure should DM adjust the dir or not. Maybe the new mode should be similar with "load&sync"?

Copy link
Author

@OliverS929 OliverS929 Nov 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a comment in "engine/executor/dm/worker.go" mentioned that load&sync mode has its own external storage. From my understanding,load mode usage work with LoaderConfig.Dir, so calling storage.AdjustPath is necessary in this case. also /cc @D3Hunter . Would appreciate it if you could take a look.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change this further so that load and loadsync both skip the adjust part of the code and use "dir" directly.

if HasLoad(c.Mode) && c.Mode != ModeLoadSync {
// check
isS3 := storage.IsS3Path(c.LoaderConfig.Dir)
Expand Down
6 changes: 3 additions & 3 deletions dm/config/task.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -669,7 +669,7 @@ func (c *TaskConfig) adjust() error {
return terror.ErrConfigNeedUniqueTaskName.Generate()
}
switch c.TaskMode {
case ModeFull, ModeIncrement, ModeAll, ModeDump, ModeLoadSync:
case ModeFull, ModeIncrement, ModeAll, ModeDump, ModeLoad, ModeLoadSync:
default:
return terror.ErrConfigInvalidTaskMode.Generate()
}
Expand Down Expand Up @@ -774,9 +774,9 @@ func (c *TaskConfig) adjust() error {
instanceIDs[inst.SourceID] = i

switch c.TaskMode {
case ModeFull, ModeAll, ModeDump:
case ModeFull, ModeAll, ModeDump, ModeLoad:
if inst.Meta != nil {
log.L().Warn("metadata will not be used. for Full mode, incremental sync will never occur; for All mode, the meta dumped by MyDumper will be used", zap.Int("mysql instance", i), zap.String("task mode", c.TaskMode))
log.L().Warn("metadata will not be used. for Full/Dump/Load mode, incremental sync will never occur; for All mode, the meta dumped by MyDumper will be used", zap.Int("mysql instance", i), zap.String("task mode", c.TaskMode))
}
case ModeIncrement:
if inst.Meta == nil {
Expand Down
2 changes: 1 addition & 1 deletion dm/errors.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -929,7 +929,7 @@ workaround = "Please check the `name` config in task configuration file."
tags = ["internal", "medium"]

[error.DM-config-20020]
message = "please specify right task-mode, support `full`, `incremental`, `all`"
message = "please specify right task-mode, support `full`, `incremental`, `all`, `dump`, `load`"
description = ""
workaround = "Please check the `task-mode` config in task configuration file."
tags = ["internal", "medium"]
Expand Down
Loading