Discuss about donate seatunnel as a subproject of Apache dolphinscheduler

[garyelephant]

Hello everyone,

I am the founder of SeaTunnel(was Waterdrop)[1], SeaTunnel is a distributed data synchronization solution which is a suitable framework for Apache DolphinScheduler[2] scaling data synchronization jobs.

Seatunnel is a very easy-to-use, high-performance, massive data synchronization solution, and many companies are using SeaTunnel in the production environment,

SeaTunnel mainly solves the problems encountered during data synchronization:

-Lack of data source connectors
-Data loss and duplication
-Task accumulation and delay
-Low throughput
-Long cycle when applied to the production environment
-Lack of application running status monitoring

When workflow jobs are running, we need to synchronize a lot of data from different data sources to the target warehouse. The DolphinScheduler itself is very weak in this area. SeaTunnel does a very good job in this regard, The core contributors of the SeaTunnel project and I hope to better integrate SeaTunnel's data synchronization capabilities into DolphinScheduler, We have communicated with LidongDai,PMC Chair of DolphinScheduler, and he like SeaTunnel very much. so I want to start discussing donating SeaTunnel as a sub-project of Apache DolphinScheduler.

Welcome give me some feedback, thank you very much.

[1] https://github.com/InterestingLab/seatunnel

[CalvinKirs]

SGTM, I think this will greatly promote the development of the two projects.

[davidzollo]

+1
Great news, I'm looking forward to see more integration of the two projects.

[debrachena]

That's a great idea, excited about new possibility brought by the powerful combination!

[lukairui]

1+1>2.Nice to see this .

[zhongjiajie]

What a great new! It good for both two projects ㊗️

[ruanwenjun]

Great!!!

[simon824]

Great news ！

[tinyking]

Good jobs!

[samz406]

Great!!!
The first time I used waterdrop, I choose the dolphinscheduler as the scheduler

[sanpig]

Great!!!!Hope to burst out more ways to play.

[Baoqi]

+1

Great News. I learned a lot from both dolphinscheduler and seatunnel (previously "waterdrop") before. The integration of those 2 projects will be very interesting, looking forward to it.

[brave-lee]

+1
Great. I think this is a new beginning for these two projects.

[Narcasserun]

+1

[wuchunfu]

+1

Great News. I look forward to the integration of the two projects.

[nielifeng]

+1

[caishunfeng]

+1
Great News. It wil be complement each other.

[William-GuoWei]

Great idea.
Many people use Waterdrop to sync data from Hive to ClickHouse, and DolphinScheduler is a very good tool for micro-batch and trigger and dependency. 1+1>2. Users will love to see these 2 projects can work together.

[lenboo]

+1
Great! 1+1 > 2

[jon-qj]

+1
some excellent projects hold together for warmth and enhance competitiveness

[wu-sheng]

Good to see this. There are several things should be taken care of.

1. IP Clearance must be done and sent to ASF incubator for lazy consensus.
2. We need to determine what is the final goal for project SeaTunnel(was Waterdrop) first. If it will go for an independent project, personally, I would recommend them go for the official incubator sponsored process, rather than staying as a subproject, unless there are some special reasons.

I want to be clear a little more about (2), incubating in the umbrella of DolphinScheduler actually shares the same requirements comparing to standard incubating process. So, usually, a subproject donation means this donated project will become a component(even could run independently). The original maintainers will become members of DolphinScheduler PMC, after the current PMC makes sure they could follow all ASF policies. Before that, the DolphinScheduler PMC have to act like what we did as IPMC.

[davidzollo]

Good to see this. There are several things should be taken care of.

1. IP Clearance must be done and sent to ASF incubator for lazy consensus.
2. We need to determine what is the final goal for project SeaTunnel(was Waterdrop) first. If it will go for an independent project, personally, I would recommend them go for the official incubator sponsored process, rather than staying as a subproject, unless there are some special reasons.

I want to be clear a little more about (2), incubating in the umbrella of DolphinScheduler actually shares the same requirements comparing to standard incubating process. So, usually, a subproject donation means this donated project will become a component(even could run independently). The original maintainers will become members of DolphinScheduler PMC, after the current PMC makes sure they could follow all ASF policies. Before that, the DolphinScheduler PMC have to act like what we did as IPMC.

got it, thank you, I think they need to study ASF policies for some period of time, and DolphinScheduler PMC @CalvinKirs and I will help the core contributors of SeaTunnel to follow the apache way all the time.

[wu-sheng]

If TLP is the goal, also you have been IPMC already. Please consider seriously for incubator process, because this will be asked for again when you report to incubator during IP clearance.
The whole incubator process will help that project much more than under a just relative TLP.

[garyelephant]

SeaTunnel（Waterdrop）is a data integration module. We made it opensource with 4 years, and we hope to join a relative community with DataOps to make this project more valuable. DolphinScheduler is a popular platform, and SeaTunnel plug-in code has been merge from 2020.7.30, and it win good feedback from both community. For Most use cases, data integration needs a job scheduler and a job scheduler user also need a data integration tool.We had cooperated with DolphinScheduler PMC Member for a long time. Users and Community often worked together, and we hope to be a part of DolphinScheduler Community. Thus not only the plug-in code of the SeaTunnel but also the code of the whole module can join the whole community. We hope to be a better community.

[wu-sheng]

Hi, @garyelephant Please don't get me wrong. I am exciting to see the expand of the community. I am asking from a more general perspective from ASF Incubator and ASF, about what is the recommended way for a new project.

AFIAK, SeaTunnel is forward to be an ASF TLP.

What is Seatunnel?

Seatunnel is a very easy-to-use, high-performance, massive data synchronization solution, and many companies are using SeaTunnel in the production environment,
SeaTunnel mainly solves the problems encountered during data synchronization

What is DolphinScheduler?

Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.

These 2 projects have no clear overlap, only tech dependency requirements. SeaTunnel is an independency project, which could work with or without DolphinScheduler. Working together is something we called ecosystem, crossing projects collaboration is very common. This tech connectivity should not be considered as a reason of being a sub project.

To @dailidong, I think you remembered what we proposed back to when we graduated 1 year ago. We(DS PPMC) gave one sentence to describe the project landscape, which is showing repeated in your tri-monthly report.

a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces

So, a data synchronization solution/product is out of our scope. And more importantly, they have their own core maintainers, and what they released is not for DolphinScheduler only.

From this perspective, the incubator is a better and recommended way, you could invite some of DolphinScheduler PMC to join as PPMC or mentors. The whole point of this process is that, the activities of SeaTunnel project depend on @garyelephant and other active maintainers, then, joining the ASF is more about learning the Apache Way(Culture, policy, gov. etc.), and this learning is mostly about SeaTunnel maintainers.

[davidzollo]

Thanks, I think SeaTunnel go to incubator will be a better choice, we will try to enter into the Apache Incubator

[xhmz]

+1

[Technoboy-]

Great news. Hope everything is fine.