🚀 feat(roadmap): add tools related to CDC, logging, streaming platform, data integration, and web development frameworks (#9) #9
Conversation
… tools, pipeline Orchestrations and NATS streaming platform.
|
Can anyone check and merge this please ? |
|
Hi @FaridFArab jan, thank you for your contribution. Yes, of course. Your invaluable contribution will be reviewed soon and will let you know. |
There was a problem hiding this comment.
Thanks, @FaridFArab for your contribution, I think the changes are good to be added.
| Go: | ||
| - Fiber | ||
| - Gin | ||
| Java: | ||
| - Spring | ||
| - SpringBoot |
There was a problem hiding this comment.
It's Spring Boot I think, please add a space.
There was a problem hiding this comment.
Thanks @niyushabaghayi. I did it and it is OK now.
There was a problem hiding this comment.
Thank you @FaridFArab jan for your invaluable contribution.
Your PR was reviewed by our colleagues, and it can be merged after applying a few changes.
Yours,
Mostafa
roadmap.yaml
Outdated
| ETL: | ||
| - Petl | ||
| - Pandas | ||
| - PySpark | ||
| - Requests | ||
| - Beautiful Soup |
There was a problem hiding this comment.
There is a specific section for ETL and some of these frameworks (libraries) are not ETL inherently. IMO, I suggest to remove this part, because it's not necessary to have this granularity for roadmap! You can also contribute on our wiki project.
There was a problem hiding this comment.
Yes I agree with you @mostafaghadimi but we don't mention any ETL frameworks there. I think we can keep them now and if we add more tools and libraries we divide them into the appropriate places.
There was a problem hiding this comment.
They are related somehow but not specifically developed for ETL.
As an example, beautiful soup and requests only help us with E (extract) and have nothing to do with T (transform) and L (load).
There was a problem hiding this comment.
OK I will remove Requests and Beautiful Soup from there.
There was a problem hiding this comment.
@FaridFArab, Let's remove ETL section from this part.
roadmap.yaml
Outdated
| - Flask | ||
| - Django | ||
| - FastAPI | ||
| Web Development and API: |
There was a problem hiding this comment.
this line should be removed (it is under web framework and API development section and caused redundancy)
There was a problem hiding this comment.
I separated Python frameworks and libraries.
There was a problem hiding this comment.
Now this section looks like:
Web Frameworks and API Development:
…
Web Development and API:
Which caused redundancy. It’s data engineering roadmap and we can keep related topics with higher of granularity.
Do you agree with that Farid jan? In case of any disagreement feel free to share your opinion with us.
There was a problem hiding this comment.
OK I will remove them.
roadmap.yaml
Outdated
| - Kafka Connect | ||
| - Schema Registry | ||
| - KSQL | ||
| - Kafka Rest |
There was a problem hiding this comment.
Are they the core components of the Kafka? No. I think they can be placed in another section or may be we can elaborate these components on wiki.
There was a problem hiding this comment.
Yes I think in data engineering world we have to use this tools and a company like Confluent included all of them into a Kafka cluster. I will add them into the wiki but I think we can keep them.
There was a problem hiding this comment.
Of course, about 80 percent of top 500 fortunate companies use Kafka and related tools like schema registry in their stacks. I totally agree with you on the importance of these tools. However, mentioning them under Kafka can misguide our readers.
There was a problem hiding this comment.
So what's your idea for them ?
There was a problem hiding this comment.
It can be added in a separated section or can be added and dive deeply to it on wiki!
roadmap.yaml
Outdated
| - Great Expectations | ||
| - Soda Core | ||
| - Data Security: | ||
| Hashicorp Vault |
There was a problem hiding this comment.
It should be a list, as well.
There was a problem hiding this comment.
What do you mean ? need changes ?
There was a problem hiding this comment.
It should be like the following:
- Data Security:
- Hashicorp Vault
# Conflicts: # roadmap.yaml
roadmap.yaml
Outdated
| - Connector: | ||
| - Kafka Connect |
There was a problem hiding this comment.
Is the term connector frequently used and understood in the context of messaging and streaming platforms?
There was a problem hiding this comment.
Yes connector used for connect 2 platforms together and here I mean Kafka connect
There was a problem hiding this comment.
How about the other tools like NATS and Pulsar? Is it still in common?
There was a problem hiding this comment.
and apply this one to another PR, please!
roadmap.yaml
Outdated
| - KSQL | ||
| - Kafka Stream |
There was a problem hiding this comment.
Let's review the roadmap again, which is accessible through this link. Here we don't introduce any tools. The tools are introduced in its dedicated section (data processing).
roadmap.yaml
Outdated
| - Schema Control: | ||
| - Schema Registry |
There was a problem hiding this comment.
I am of the opinion that schema control is a distinct section. From a bird's-eye view perspective, it should be positioned as an independent section. It's not only related to the messaging and streaming platform.
There was a problem hiding this comment.
Let me separate it.
There was a problem hiding this comment.
Let's finalize this PR and have it in another PR.
There was a problem hiding this comment.
Let's finalize this PR and have it in another PR.
Do you mean I make my change to another PR ?
There was a problem hiding this comment.
Hmm, I mean just remove the schema control from this PR and add it in another PR to discuss about it.
mostafaghadimi
changed the title
mostafaghadimi
changed the title
|
Well done @FaridFArab jan, welcome to DataBurst. <3 |
…m, data integration, and web development frameworks (#9) (#9) * (Improvement) - Add Webframework and correct some of them. * (Improvement) - Add CDC Debezium, DataQuality frameworks, DataLineage tools, pipeline Orchestrations and NATS streaming platform. * (Improvement) - Add Hashicorp Vault to Data Security. * Update roadmap.yaml
…m, data integration, and web development frameworks (#9) (#9) * (Improvement) - Add Webframework and correct some of them. * (Improvement) - Add CDC Debezium, DataQuality frameworks, DataLineage tools, pipeline Orchestrations and NATS streaming platform. * (Improvement) - Add Hashicorp Vault to Data Security. * Update roadmap.yaml
…m, data integration, and web development frameworks (#9) (#9) * (Improvement) - Add Webframework and correct some of them. * (Improvement) - Add CDC Debezium, DataQuality frameworks, DataLineage tools, pipeline Orchestrations and NATS streaming platform. * (Improvement) - Add Hashicorp Vault to Data Security. * Update roadmap.yaml
No description provided.