[help] Cross-projects Reference for ETL #1350
Replies: 2 comments 4 replies
-
If you need to split the workflow into 3 projects as you say, then each successive project could reference the data files from |
Beta Was this translation helpful? Give feedback.
-
I followed @wlandau's suggestion and made this verbose code. The idea is to use the external file + customized read/write function. For each external object, I need two targets
By doing this, the external object wouldn't be stored in my current project. I can still trace its change. Is this something we can add to the
|
Beta Was this translation helpful? Give feedback.
-
Help
Description
Is it possible to monitor the change of a target object from another project without loading it, and only load the object if the current target is outdated and needs an update? The use case is that I have an ETL project which has three tar projects:
Since they are defined in separate projects. The downstream projects must know if the upstream data has been changed or not. The current solution is to call
tar_load_everything
and load all data from the previous project. This method has a significant overhead and makes parallel computing not possible. Therefore, I'm hoping to have a simple and fast solution for it.I'm not a targets expert, but I can imagine two solutions for that:
tar_make
to declare dependencies(e.g.project_deps
). Therefore, the data can be loaded when needed.tar_load_project
). This can load the project's meta data only. Whentar_make
is called, it can make the required data available.Any suggestions will be appreciated.
Beta Was this translation helpful? Give feedback.
All reactions